I’m creating a workflow that queries an API via the repeat action, which calls a module that runs an Iterate Web Request to return a series of records that gets parsed as JSON. Each set of returned records is associated with a JSON key called next_last_id, which is used to query the next set of records. However, on the last set of returned records, this key doesn’t exist.
Since next_last_id doesn’t exist on the last set of returned records, the repeat action fails.
To prevent that from happening, I create a column called next_last_id after the Iterate Web Request action, just so it always exists (whether it’s blank or not), and then derive a table with a condition to check whether next_last_id is empty or not. If it’s not empty, then it goes to a table called API Result (see screenshot).
The problem now is the result doesn’t include the last set of records due to that condition, but at least the project doesn’t fail. How can I set this up so all records are returned, including the last set? If I export the results to a temporary CSV file, just before the API Result table, I can then load those last page results back into my main table, but that seems rather hacky and not very efficient.
If I don’t configure any condition on this derived table, then the repeat action just runs continuously until the threshold is met, and causes the project to fail with an error saying it ran into an endless loop.
