Add path to called morph file in project metadata action [DONE]

[reynsnivea]

Hi,

I greatly support the project metadata action !

Iwould like to make a mapping file of all easymorph projects that call other easymorph projects and get their path.

It would be nice if the full path of the called morph file could be added to the output shown below.
This way I could create an overview of the relations between the different easymorph files that we use to load our data warehouse.
Is there a workaround that I can use ?

Another suggestion for this action is the following. I also want to make a mapping file of the tables of our datawarehouse (= parameter values in easymorph export actions) and the easymorph file that loads them. I can get a list of all easymorph projects that contain an “export to database” action and their parameter but the value of the parameter itself is not given in the output. However, it can be obtained doing another iteration using project metadata to call all parameters.

It would be easier if the parameter values are also shown when we list the actions. This would save a step. I don’t know if this is possible.
Also when parameters for the connection and table are used in export to database action, the two params are merged in one cell in the project metadata action. Maybe it would be better to show it in separate rows with an indicator if its a parameter for the connector or for the table ?

Thanks !
Nikolaas

[dgudkov]

Hi Nikolaas,

Version 4.6 will introduce dependency information for modules, groups, and tables. It will also be added to the “Project metadata” action. So it will be possible to see modules/projects called by a project module.

The same is true for using connectors - it will be possible to see for each module, group, table what connectors it uses.

Also when parameters for the connection and table are used in export to database action, the two params are merged in one cell in the project metadata action. Maybe it would be better to show it in separate rows with an indicator if its a parameter for the connector or for the table ?

You can split them into rows using the “Split delimited text” action, if needed.

[reynsnivea]

Hi Dmitry,

Thanks ! I think I can apply a workaround for now. Very nice that the metadata action will be elaborated further.

I haven’t tested it yet thoroughly but it would be also very useful to be able to capture the “renames path” of variables from source to target. A challenge there is that sometimes we rename using the normal rename action, but sometimes we also use an external file to rename by lookup.
The “rename path” should capture both types of rename methods.
Any plans to add this in an action ?

So if variable A is renamed to B and then to C and eventually D. We would like to see A => B => C => D

Use case: create an automatic mapping table for the fieldnames from source to target for our datawarehouse.

Thanks
Nikolaas

[reynsnivea]

@dgudkov

Hi Dmitry,

I have been playing around with the table metadata action. I can see that it is possible to capture the original names and the final names of a table as depicted below.

However, I am looking for a way to grab that kind of result for each EasyMorph file without having to run the EasyMorph file per se. This would allow us to build the whole column mapping of our data warehouse.

So what I want to do is the following:

• Create a list of all tables in all easymorph files than contain an export to database action. Those tables mark the “end” of a data flow so no renames will occur after that point. So that’s the point to make the mapping original / final column names.
• For that list of tables, I want to call a function that returns the original - final column name mapping (like depicted in the example below). With the current metadata action it is not possible to do that without running the flow.

The purpose is to put that mapping table in the datawarehouse so that we can query it to see where things come from.

• Any idea how I can do that with the current version of easyMorph?
• One problem could be the following. If explicit mapping is used in the export to database, I suggest it is not possible to make such a mapping since column names do not necessarily have the same name? In automatic mapping, normally a rename has been done in advance for the automatic mapping to succeed when exporting to the database.

[dgudkov]

Hi Nikolaas,

I’m not sure how this can be achieved in EasyMorph. As of now, column lineage in EasyMorph isn’t throughout. We track the column lineage starting from the action where the column was generated or imported. But not all actions can pass lineage through. For instance, if a column gets aggregated in the “Aggregate” action, it’s considered a new column. The previous lineage is disconnected.

As a suggestion, you can create derived tables at certain points of a workflow and use the “Table metadata” action to produce column metadata and export it into a database that collects workflow metadata. Then, after one run you will have the metadata collected. Can’t think of anything else.

[reynsnivea]

Thanks for the info !