I am trying to find the best way to perform the following transformation....
I am wanting to iterate through a list of rows, and pass each row as parameters into a module along with another table.
This is easy enough to do with the Iterate Table action.
However, I'm wanting to append some results to the "Other table" after each iteration, and pass the new set of results as the "other table" into the module along with the next row as new parameters, and then repeat until all rows have been iterated.
What is the best practice to achieve something like this? Do I need to resort to somehow writing these results somewhere externally or into shared memory to achieve functionality I am looking for? Or is there a better/easier way?
In this case, I'd suggest keeping the "Other table" in an external .dset file and appending new rows to it in each iteration: read the dataset in the beginning of the iterated module, append rows, and write at the end of the iterated module.
It's a simple solution but not the most performant. It can be slow if you perform a lot of iterations ( >100,000).