Synchronizing external programs

I am running an external program to process a csv.

I need the following basic execution steps:

  1. EasyMorph: Pre-process csv 1
  2. EasyMorph: Pass csv 1 to external program (I write it as a file so the external program can read it)
  3. EasyMorph: External program runs and generates csv 2 (writing the csv)
  4. EasyMorph: Wait until external program is done
  5. EasyMorph: read csv 2
  6. EasyMorph: continue execution

There are two ways this could work:

a. Some way of directly passing tables to/from external tables (for instance STDIN/STDOUT csv)
b. Some way of waiting for an external program to complete before LOADING another table.

With respect to (a), yes I can capture the output of the external script which I could then split to create a table in EasyMorph. But I have arbitrary column headers so this doesn’t work (and that’s only in one direction anyway: from the script back to EasyMorph).

I have been trying to figure out (b), but as far as I can tell there’s no way to insert a synchronize BEFORE the inputting a table. I also went down the path of updating a database to indicate execution status of the external program, but that doesn’t seem to help since, again, there’s no way of having a transformation step before inputting a table.

If the number of columns is always the same (only names are different) then “pushing” column names down into 1st row might help (see example below). “Pushing” labels up can later be done with the “Label columns” transformation.

push names down.morph (4.5 KB)

The idiomatic way in this case is to move the import transformation into a helper project (subproject). And then use the “Call” transformation to call the subproject and return the table from it. In this case it would be possible to insert Synchronize before the Call that does importing.

PS. In version 4.0 it will be possible to insert all start transformations (such as file import) anywhere in a table. It would make using subprojects in cases as described unnecessary.