The “Repeat” action introduced in v4.5 can be used to pull data from web APIs that require pagination. Such APIs don’t return all the requested data in one response, but instead return a limited set (page) of records (e.g. 10 records per page) and require specifying a page in requests (e.g. p=2). The calling application must sequentially pull page by page from such API until it pulls all the required data.
Unfortunately, pagination is not specified in the web standards. Therefore, in real life APIs implement various, sometimes rather exotic, pagination patterns. Some APIs require a page number, some require an offset. Some require a page size to be provided, some not. Some use sequential page numbers, some provide a next page ID provided in previous requests. Some use URL parameters for pagination, some use HTTP headers. And so on.
The “Repeat” action together with the web actions and function in EasyMorph allows arranging basically any possible pagination pattern. However, it requires a bit of workflow designing.
Below is an example of querying a free web API with pagination. The example works out of the box (assuming that the API’s author still keeps it alive). It fetches recipes from Recipe Puppy, a free public database for recipes. The API of [Recipe Puppy] is dead simple and requires 3 parameters:
q - query string, e.g. (“omelet”)
i - optional list of ingredients (comma-separated)
p - page number
The database contains millions of recipes, but its API returns only 10 recipes per request. So we need to use pagination to pull all the query results if there are more than 10 of them.
The example works as follows:
- Prepare the initial input dataset (Page = 0).
- Use the “Repeat” action to run module “Query API” until its result becomes empty
- The “Query API” module increments Page and then queries the API. If the response contains a JSON with recipes, then return the JSON and the current page number, otherwise return an empty dataset which will signal to the “Repeat” action to stop iterations
- The “Repeat” action automatically appends all the resulting JSONs into one table. Parse them using the “Parse JSON” action.
The workflow uses the ability of the “Repeat” action to pass the result of one iteration to the next iteration through the “Input” action. In the very 1st iteration, the “Input” of the iterated module is populated with the input dataset of the “Repeat” action itself (see the picture below).
The result of on an iterations looks like the one below:
Notice the page number in the result. When the result is passed to the next iteration, the page number will be incremented and a new web request will fetch the next page.
query-api-pagination.morph (9.6 KB)
