Process a Directory of files

1

Hi,

I am new to EM and not sure if this is a feature that I haven’t seen but when reading data from a file I noticed the FileName itself must always be specified either as a parameter to a reusable project or as a hardcoded name as the source.

I was wondering if it is possible to specify a directory or path and simply process any file in that path with a specific wildcard name?

For example c:\data\etl\files*.edi

I am sure with some tinkering I could have a script file do this and call EM project with each filename in the folder but I thought it would be a nice feature if it doesn’t already exist so that there are fewer moving parts and it is more intuitive since EM is doing most of the heavy lifting.

Cheers

2

Hi Bruce,

at this point loading multiple files is done using iterations. Iterations are arranged with the help of Iterate transformation. Briefly, it works as follows:

  1. Create a project (A) that loads one file which name is defined by a parameter. Let’s say this partmeter name is File name
  2. Create another project (B) that generates a list of files in the given directory using File List transformation. Use filtering transformations to keep only particular files in the list, if necessary. That would be equivalent to using wildcards.
  3. Add Iterate transformation to project B and set it up as follows:
  • iterate project A (created in p.1);
  • assign File name parameter in that project using the column with file names in project B (the one generated by File List);
  • use “Iterate and append” mode to concatenate all loaded files into one table which becomes the result of the Iterate transformation.

Basically, project B in this example is an equivalent of the script you were mentioning. Generally speaking, EasyMorph is supposed to replace scripts, not the other way around :slight_smile:

See the example:load multiple file.zip (589.2 KB)

To read more about iterations see this tutorial chapter: Iterations (loops)

PS. We will be adding a way to load multiple files without iterations in version 3.6 planned for release in April.

3

Excellent I had a feeling that there was something I was missing. Yes, I can see the value in minimising scripts especially in this case.

I will take a better look at this approach. In our case there could be files of different formats in the same folder (eg. An FTP drop off location). Therefore, there would need to be some type of conditioning on what transformations occur after each file is retrieved and it’s format determined.

But thanks this gives me a lot of confidence that EasyMorph is flexible enough to manage some of these scenarios.

I appreciate your rapid reply

Cheers

1 Like
4

When you will look closer at Iterate transformation notice that the iterated project can also be defined by a parameter. This allows pre-calculating a path to project to run depending on data file name or some other factor. An intermediate level of iteration will be required -- i.e. not iterate directly, but iterate a helper project ("router") that will in turn iterate necessary project with precalculated path. It might sound a bit complicated, but actually it's rather simple once figured out. Let me know if/when you need an example on this.

5

Thanks. Pretty sure I follow. Will take a look. cheers