Hello, I'm struggling with this and have read several posts in the forum but I cant figure out to solve my issue. I guess the process is simple enough but what I'm missing is how to use the "nextCursor" for the iteration.
Here is my workflow:
efef
Create a list with the first cursor value that must be empty and then iterate
So the question is really how do I get the nextCursor value to the main module so the next iteration uses this and not the empty cursor from the initial start?
Please take my reply with a grain of salt. I am not an API master by any means but I was able to get API pagination working using GraphQL and an empty cursor string incorporated in my production jobs.
I think you need to use the repeat action instead of the iterate record action. I will put some screenshots below. I don't know exactly how the API works that you are using. But in my case, the empty cursor represents that there are no more records in the target endpoint that we have not received back from a previous or current request.
In my case, I don't initially query with an empty query string. I do an initial request of 1 record to get an initial cursor string because I know there will always be more than one record I am pulling back and it just feels clean.
The response to the initial request will contain the cursor string for the subsequent request. I derive my initial table and use the repeat action which calls a separate module where the main query to my endpoint lives. While the repeat action is running, any actions in the same table are not calculated until complete. That last action appends back the results from the initial query to the final dataset to get the final 100% of all the records.
You need to setup an input action in the pagination module (or whatever you end up calling your separate module if you choose separating it like that, it can be done in a separate group as well).
Next is where the real sauce lives. You need to derive a child table on condition set to check that there is at least one non-empty value in the column where you are receiving your cursor responses. If you are getting cursor responses back then it likely means there are still records available via the end point that have not been queried yet. The endpoint returns the records from the query as well as a next cursor to use which gets stamped on every returned record to be fed back through the loop. When all the records returned do not contain a value where the cursor response is coming back, it would typically mean there are no more records available from the API. When they are all empty, Easymorph runs one more loop and this time your cursor is empty on every returned record. The derived table is empty because it no longer satisfied the condition of at least one record having a non-empty cursor string. And that empty table is the real signal to the repeat action to stop.
After this is just the query to the endpoint and parsing the JSON again to be returned to the main module where the loop is running.
If your endpoint and cursor pagination works similar to this, the above methodology SHOULD mostly work for you, albeit with some possible tweaks here and there.
Thank you for the detailed response and appologies for delayed response, I've been struck with the flu for a week but now coming to life again.
I will have to read through your detailed post a couple of times and try to replicate for my use and report back
Tried adapting your approach but not there yet...is there some way to "step" or debug each iteration of the Repeat-action? So that it's possible to see what actually happens for each iteration.
To see what will be in the input of your iterated module on 2nd iteration, right-click the last action in the result table and choose "Send output to sandbox/module" to send the result to the "Input" action of this module.
