Testing complex data flows


Sometimes we have complex data flows. When we have to change those data flows after a while, it is sometimes hard to predict if a change could impact the final results.

Any ideas how to integrate tests in EasyMorph to cope with that ?


For instance, you can use the “Either table” action to inject test datasets at certain points of your workflow.

In the example below, the {Use test data} parameter is either TRUE or FALSE. When it’s TRUE, the “Test data” table is not skipped and it loads a test dataset from a file. The “Either table” action inserts the dataset into the “Data” table.

When the parameter is FALSE, the “Test data” table is skipped. Its output is empty. Therefore the “Either table” uses data loaded from the text file in the previous step.

To make parts of your workflow testable, put them into modules. Then have a separate module that will run all the testable modules with the parameter “Use test data” set to TRUE and verify their outputs for correctness.

1 Like

Thanks that could indeed be an option. So you make several test input files and 1 output file with the correct results (probably manually calculated). Then run the flow with test datasets and then compare the result of the flow with the manually calculated results.