I am running an external program to process a csv.
I need the following basic execution steps:
- EasyMorph: Pre-process csv 1
- EasyMorph: Pass csv 1 to external program (I write it as a file so the external program can read it)
- EasyMorph: External program runs and generates csv 2 (writing the csv)
- EasyMorph: Wait until external program is done
- EasyMorph: read csv 2
- EasyMorph: continue execution
There are two ways this could work:
a. Some way of directly passing tables to/from external tables (for instance STDIN/STDOUT csv)
b. Some way of waiting for an external program to complete before LOADING another table.
With respect to (a), yes I can capture the output of the external script which I could then split to create a table in EasyMorph. But I have arbitrary column headers so this doesn’t work (and that’s only in one direction anyway: from the script back to EasyMorph).
I have been trying to figure out (b), but as far as I can tell there’s no way to insert a synchronize BEFORE the inputting a table. I also went down the path of updating a database to indicate execution status of the external program, but that doesn’t seem to help since, again, there’s no way of having a transformation step before inputting a table.