I successfully implemented the iterate action described in this tutorial (How to pull data from web APIs with pagination) and it works great, albeit a bit slow though I attribute that to the API from our survey provider. Currently, I have a parameter that stores the survey ID that determines what survey to grab the responses for which I manually populate and then run the project. But what I need to do now is iterate through multiple surveys, changing the value in that parameter so that it will grab all the data for one survey and when finished, move onto the next survey and keep running through the table containing the survey IDs (and page number to start the API on) until it has no more surveys.
I understand the concept but the placement of where to place this additional iterate action is confusing to me and attempts at testing in various places have all failed.
There are multiple types of iterations in EasyMorph:
Iteration by rows
Iteration by rows with another table
Iteration by columns
The “Repeat” iteration (a DO…WHILE type of iteration)
Iterations can be nested in any combination. For instance, as in your case, you can iterate a table row by row, and within each iteration have a “Repeat” iteration. So it will be a two-level iteration, or iteration of iterations, if you will. The nesting can be of any depth – you can have an iteration inside iteration inside iteration, if needed.
In your current workflow that does the “Repeat” iteration, add one more module with the list of survey IDs and then use the “Iterate” action to iterate across the list, and in each iteration call the module that does the “Repeat” iteration. Assign survey ID in the “Iterate” action so that “Repeat” starts with a new survey ID in each top-level iteration.
Thank you for this. I’m going to create that new module and see about passing the information from there to the Repeat iteration and report back with how it goes. Thanks again for the support!
I’m still working on this Dmitry but I’m encountering a problem with the original solution before the “iteration of an iteration” example you provide.
In your API with pagination example, you have a parameter for {max Pages} which you use as the trigger to stop the API calls and leave the results table empty which then triggers the loop to stop as the results table is now empty. My problem is I left that condition in there and so even though my API should return only 2 pages with the item limits I provide in a parameter, it’s running it 10 times (the value in that parameter) even though after the 2nd page all the pages are “empty” with result length being 356 (presumably the length of an empty row).
If I need to have it run through until it’s no longer pulling any more pages with data without using a parameter hard coded with a page number, how should I do that? Each call gives me the number of pages in the retured JSON so if I could extract that (let’s say that number is 5) I could use that as the trigger by saying "when page number = Total Pages indicated by API +1, then stop)
I’m SUPER close on this but having it trigger when to stop sending a call is still not clicking and once that happens, I can start looking at “iterating an interaction”
You can modify the condition so that it reliably detects an empty result. For instance, replace
[Result length] > 0
with
[Result length] > 356
And/or use functions such as trim(), or squeeze() to remove whitespace in the empty result. It depends on what the empty result looks like. In any case, it should be possible to detect the empty result reliably.
not directly related, but out of curiosity: what is the squeeze() function you mention? I can’t find it in the function list (e.g. on the calculate new column action) nor on the help . I’m still on v5.1 however.
Great, I’m gonna test this out in the process I’ve built because short of that it appears to be working as expected…it’s just that it makes a ton of calls, 90% of which are pulling empty results and when we have a cap on the # of free calls we can make in a month is a little problematic
I’ll dig in and should have success. Thanks, yet again!
@dgudkov
The API I am using in this process has newest records first and I only ever need to grab the newest records from our service provider. I can store the total # of pages in a simple database table (let’s say it was 17 last time I ran the project described above) and discover the current # of records making a simple call (let’s say it it is currently 18,500) and divide that by 1000 (the # of records I can pull in a call) and then round up which tells me there are basically 19 pages of data. Subtract 17 from 19 and so there are 2 pages of new results though I’d add an extra page to be safe and pull 3 pages of results.
How do I tell the process to only pull 3 pages and then stop? The way I set it up I can tell it which page to start on but then it just runs through all subsequent pages until it returns empty results and that works great, but now I’m stuck on how to tell it to always start on page 1 and then stop at a page that is calculated.
EDIT: I noticed in the example how you used the Max Pages parameter which looks like what I’d use here, but how can I make that dynamic? If I put that call to determine # of records in the module querying the API will it run that with every loop?
If you know ahead of time the number of pages you need to pull that makes things even simpler. Just use the “Generate sequence” action to produce a table with a sequence of numbers from 1 to N, and then in that table use the “Iterate” action, instead of “Repeat”.
The general rule of thumb for loops in EasyMorph is the following:
If you know the number of iterations in advance, use “Iterate”
If you don’t know the number of iterations in advance, use “Repeat”
Thanks for this, @dgudkov . I ended up figuring out how to determine the # of pages I needed to pull by peeking the value of a Max Page column calculated by looking at the # of current responses and putting it in the table that gets sent to the API module to be iterated. Added logic to say “if the page # is <= to the Max Page then keep going, otherwise stop” and it seems to be working well. Only thing is that if the max page is say 2, it will still run a 3rd call and then discover it wasn’t supposed to do that and then stop, but that’s only a minor annoyance I can likely figure out Thanks, again!
Thanks, again!