Hi - I'm still having a tough time with this concept (Using sample pokemon data)
We make the initial request:
then on the iteration I can never seem to get it to populate back for the repeat.
Pokemon - Sample API Requests.morph (8.2 KB)
Thoughts on what I'm doing wrong?
Hi Adam,
You should definitely override my reply with anything that Dmitry happens to send your way on this. But I had a API pagination item somewhat similar to this recently.
I added an action to your table that contains the loop to append the first 20 results at the end. You can definitely do other cleanup.. and I don't really know the content of this. I removed your action to write out to the csv file etc. I am just concentrating on attempting to get all the records back.
The append will bring all the records from your initial table back to the repeat table without interrupting the loop. I saw you were writing to a file in the "Get Data" module so you may not need to do this.
I think your main issue is the "Results" table. It is set to always run. And you should have it only run on condition so that the table being empty or a column being empty can signal to the loop action from the main module to exit. Take a look at my adjustment to the derive table action in image 3.
Image 2
Image 3
This configuration when not always set to run, will only run if you have at least one row with a value in it. If the upstream table is empty it skips, and this signals the loop to stop.
Here is what the first iteration through looks like (image 4)
Image 4
Here are the next 20 records coming in (image 5)
Use the export to sandbox option on the last action of your table to see the next loop iteration (image 6).
Here is the next iteration (image 7).
Here is the last iteration and there are no more values coming back. There is probably a way to handle this but I don't have the time at the moment to dig deeper (image 8).
Now if we try one more iteration, the "Input" table is empty and the "Results" table skips all the actions signaling the loop to stop in your main module (image 9).
And now looking back at your main module, all your records are there (image 10).
Hey @Twitch - Thanks for the reply. This gave me some clarity. "most" of the API's I deal with give me total count and from that I've derived the process without to repeat action.
I've cleaned it up and attaching a final example should anyone else need to see how this works as I'm believe Dmitry's example public API has been shut down
Pokemon - Sample API RequestsV2.morph (13.5 KB)
Below is a method to design workflows that query APIs with pagination using the "Repeat" action. This is a simplified and improved version of what I suggested previously. It still has a few non-obvious gotchas, so I tried to cover all of them. The method also benefits additionally from recently added features.
The example is using PokéAPI (pokeapi.co) as suggested by @adambeltz.
Here is the project. It contains just 7 actions in two modules. Its logic and the design process are explained below:
api-paging.morph (6.2 KB)
The main module:
The "Get page" module:
Start by constructing the initial dataset of the loop. Typically, it's a 1-line dataset that satisfies 2 conditions:
As we see from the API response in our example, the example API returns the URL path of the next page in the JSON property next. So, we construct our initial dataset as one column named next. In this example, I used the "Create list" action for that. In a real-life project, you will probably need to construct the initial dataset from a parameter or some other user input.
Now, let's create a new module called "Get page" with the "Input" action in it and add the "Repeat" action that calls "Get page" in the main module.
Useful tip: press the "Populate automatically" button to populate the input dataset from the parent module:
The iterated module must have a condition that returns an empty dataset when there is nothing more to fetch from the API. An empty result dataset signals to the "Repeat" action that it must stop iterating further and exit the loop.
In our case, when the last page is retrieved the API still returns the JSON property next but it's empty. So we can just check if it's empty or not in the "Skip..." action (step 2).
Important! Notice that we don't exit the loop immediately when we receive the last page because we still need its result returned to the parent module (otherwise the last page will be missing in the result of the "Repeat" action). We still do one more iteration and the "Repeat" action sends the output of the last web request to the next iteration loop and only then we exit the loop (by making the "Skip..." action produce an empty dataset), before doing another web request.
If the API didn't return any property at all, we could use the columnexists() function to check that.
Important! Configure the "Skip..." action to return an empty dataset, when the condition is not fulfilled because by default it doesn't.
Finally, let's send the API request and parse the response. Different APIs require indicating the next page in different ways, there is no standard for that. Sometimes, it's in the URL query parameters, sometimes, it's in the request body. Some creative API designers may even use request headers for that.
Some APIs use the "offset/limit" (a.k.a. "skip/take") method to specify the next page. Some APIs return a "cursor" - a semi-random ID that points the API to the next page. Luckily, EasyMorph can handle virtually any API specification.
In our case, the next page is conveniently specified by two URL parameters, offset and limit, which we can extract from the URL returned by the API in the JSON property named next.
Since our URL path is in a column, we use the "First column value" method to specify the request path:
In the last step, we parse the response using the "Parse JSON" action:
Notice that the last step produces the column next required in the input dataset in the next iteration of the loop.
To make the "Repeat" action collect and automatically append all responses, don't forget to switch it to the "Append and return all results" mode.
To see what will be in the input of your iterated module on 2nd iteration, right-click the last action in the result table and choose "Send output to sandbox/module" to send the result to the "Input" action of this module.
For debugging purposes, you can keep sending the result dataset to the input to see what's going on on the 3rd iteration, 4th, and so on...
If you need to edit data in the input dataset, send it to a sandbox (using the menu command described in the previous tip), edit the sandbox using the "Dataset editor", and then send the edited dataset from the sandbox back to the input.
In the example, our iterated module has only one table so I didn't flag it as the default result table as there is no ambiguity. If your iterated module has more than one table, mark the result table by right-clicking the table header and selecting "Flag as default result table". It will tell EasyMorph which exactly table is the result of the workflow. Otherwise, the "Repeat" action will return an error as it won't know what table contains the result.
Thanks. This really cleaned it up. I think the iterate action is very logical and repeat feels slightly more abstract for some reason, but the example helps and its now working correctly in my work scenario.
Thanks for your example, better and simpler now for "non experts like you 
)
Could you help me? I'm near to reproduce your example but I'm missing something.
I have this errors
And I do not know why in my last step I have not the "NEXT" in the parse json to select to return to the loop module.
thanks so much!
