I'm using an API endpoint to GET survey responses. The endpoint can only send back 5000 responses, so I need to create a Repeat action to get the next 5000 etc.
I think the way to do this is using the optional API parameter 'Filter', which allows me to specify a Response Date to filter from.
So I'd GET the first 5000, work out the MAX Response Date of that batch, and pass that date back into the parameter to get the next 5000, and so on.
I've used Repeat for incremental pagination but not dynamic loops using data from the batch itself, can anyone advise? Let me know if you need more info.
You're logic is pretty much correct in that you need to use the Repeat action. The steps needed would be as follows:
You need a main module which has 2 actions:
Firstly, create a list with the starting date. The field should be called whatever the field is called in the API and for which you want to get the max each time so you can use it in the API request.
Followed by the repeat action to repeatedly call another module (API Call). Set it to repeat until the returned result is empty and to append and return all results.
Then in another module which the repeat action calls (API Call) you need to do as follows:
Firstly use the Input action to receive the table of data from the current iteration of the repeat action. Note that on the first call you will receive the starting date you set in the list on the main module. For all subsequent calls, it will receive the full set of data from the initial list plus the previous iterations.
Make the web request using the web request action, setting the parameter or header field as needed by the API to "First value of column" and selecting the data column.
Do whatever you need to do to parse the result (e.g. if its JSON then parse the JSON as needed).
If no data was returned then you need to have a way of detecting that and returning an empty table. In the below, because the API I used returns an empty body if no data is available from the date passed, the Parse JSON action causes errors. So I use the Filter by type action set to filter out errors which results in a table with no rows. What you'll need to do specifically will depend on what the API you are using passes back if no records are available.
thanks Matt, I couldn't quite get it working using the date, I suspect because the date field itself doesn't live as a standalone field within the JSON, but managed to switch to an alternative parameter instead, which seems more fruitful.
