The Iterate action currently does not preserve data from the previous step, unless that data is passed to the Module being iterated as a parameter (using the "Iterate and append results").
iterate and append.morph (3.8 KB)
I've recently been working on the following: iterate over a list of files to...
- heuristically determine the file's encoding (not the first one dealing with this)
- heuristically determine how many header lines the file has
- count the number of data rows in the file
These steps are interdependent (step 1 is required for step 2, and steps 1 and 2 are required for step 3). So far, so good. This works, because the output of each step is a parameter for the following step.
Now I need to add other information that no longer follows this pattern of all previous output being required for the next step. I have a few options:
- maintain the pattern, meaning that the 7th steps would have 7 parameters, up to 6 of which wouldn't actually be used by the module, but are only passed on to ensure that they end up in the result dataset
- breaking up the structure of the flow to resemble the dependency structure, and using a couple of Natural Merges to combine the results
- splitting the table into column sets and recombining them by Appending
- refactoring the called modules to accept a data row instead of parameters, and having them return the original data row with an added column.
There are probably a few more workarounds, but none of them are particularly elegant. I'd propose a third Iteration mode:
- Iterate
- Iterate and append results
- Iterate and append results as new columns