Hi community - I hope you can help me to optimize a workflow.
I have e.g. a database result with 400.000 rows with a product gtin. (simulated in my example with the sequence)
With this gtin I have to call an API and save the result in a file or database.
Sometimes it is not clear how many requests are allowed to the API in which time (it just run in a timeout) - and there are many other possible errors, why it is not the best solution to call the API for all 400.000 rows (iterate web request) and save them afterwards. (version 1 in my example).
Another way is that I build a module for one request and direct save - and the main table iterates through the module. (version 2 in my example). But this one is much slower, because I cannot send parallel request like in version 1 with the “iterate web request”.
If i know the size of the input set and if it is more the less always the same size, I can split the input dateset manually to e.g. batch sizes with 50.000 rows and duplicate the “iterate web request” with derived tables. But if the row count isn’t always the same or a high number (like 2.000.000 row in 50.000 row batches) that solution is not good.
Do you have any idea how to split this dynamicly in batches?
no - some APIs do not return totalSize or something like pages.
for the case of “requesting an api with an id” an other user already gave me a good hint, that I can use for this case (enumerate rows with “repeat after” in batch size, concatenate ids grouped by repetition, and iterate sub module with API requests with the concatenated ids). That works fine so far.
The next level I have to solve:
There is a whole table with many rows and columns and I need in the sub module for the API call all fields - not just the id. (e.g. to create an item for a webshop). But again, I need it in batch sizes, not the entire table at once.
Take a look at the “Input” action. It receives the table from the call or iterate action that called the module. It receives the whole table so if you wish to iterate through batches for example, you need to pass a parameter with the current batch in it and then filter by this parameter after the Input action.
