We would like to have all input / output tables of our ETL-flows. With tables we mean import or export of tables from our database (not EasyMorph tables).
Note that in our flows sometimes a parameter is used as table in expressions and sometimes the dropdown has been chosen…
We also want tis for all input / output files.
Could anyone post a project to extract this from EasyMorph ?
In the project tab there is a button named “Generate Documentation.” If you click on that, it will create an HTML file that shows all the source databases as well as what outputs/exports the job creates. It also has all of your logic in there along with any parameters you setup etc.
This is the closest I can think of to what you are describing.
The “Project metadata” action produces a list of project dependencies including connector names and files. However, it doesn’t show table names.
I have made a project to capture the information though maybe it is not 100 % accurate.
For input tables I read the morph-file as xml (separate lines import mode) and then I use regex to see where there are key words like FROM, LEFT JOIN, INNER JOIN, etc. to get the tables from select statements. I also account for table parameters used in the statements.
For output tables I could use the metadata action because there’s only 1 output table per export action.
Then I merge the 2 outputs.
It would be great if you could share the project. It can be helpful for someone else. Thanks!
Here is my project. Sorry if there still some Dutch here and there in the names/descriptions.
The descriptions I translated to English.
get_input_output_tables.morph (17.0 KB)
The project is not 100 % watertight.
For tables that are literally mentioned, the schemas we have are in the regex(). So that should be parameterized or the approach should be changed .
For our input tables we only use SQL-statements, no visual query builder. So that could also be an issue for projects using a visual query builder.
I have also found that when the select sql statement is written on a single line e.g.
select * from mytable left join mytable2 on ()
that mytable2 in the example would not be identified. That’s because when we import the morph file as plain text, the statement is not split over 2 lines. We
normally do not have such SQL statements (I think) but for those who have this, it won’t work 100 % correctly. Our statements have returns between the lines…
So there are improvements to be made. Maybe you could improve it and make it more robust for various situations ?
It should be great if file imports / exports could be extracted as well.