Optional data files import/skip

1

I've set up a rather complex process (for me!) which pulls in data files and processes them as you'd expect in EasyMorph!

However, for some clients some of the data files will be missing as they don't generate revenue from certain activities.

I know I can get around this by creating "empty" files (with just the headers) which are copied in to that month's data directory and used if the files don't exist, and the whole process will still work, but is there another way to basically tell EasyMorph, if this file is missing, skip this bit. I guess the end result would be similar to disabling the import?

2

First, load a list of files of this folder. Search for the file.
Derive a table and continue with opening the file only if the derived table is not empty.

3

Hi Jochen,

Thank you for the prompt reply (I love this community!) but unfortunately it looks like that's not going to work in this instance. I can build what you're suggesting, but then if the file doesn't exists I have the same issue in that any derived tables and process based on the missing file error as there is no data for them to process.

I have the possibility of more than 40 files that could be available, and if they are they should be loaded and then ingested into the various processes. Depending on what revenue was generated for a month, there could be as few as 20 files for that month, so I need a way to skip the integration if those files that are not available but still process and calculate from those that are.

I'm building a folder of blank files that have all the correct headers with no data, so the process will load those files and everything will work. I'll do this by building a list of the files in the folder and comparing these to the files that should be in the folder for the processes to work, and then pulling in the blank versions of the missing files and renaming them with the correct date for that month so the process pulls them in correctly.

Hopefully this will work!

4

A quick update for anyone interested in this!

I created a number of blank files with just headers and no data.

These files have parameters in the filename, e.g. STARTDATE, ENDDATE.

I import the filenames into a table and update the parameters based on a "setup" file which has the month required, client info etc. in it.

I then compare the list of files in a client data directory with the reworked filenames of the blank files, and filter for those only files that don't have a match.

I then copy the blank files into the client directory, and then rename the blank files in the client directory, e.g.

blank file - transaction_CLIENTMANE_STARTDATE_ENDDATE.csv

becomes - transaction_joomtree_20220101_20220131.csv

Then in my main processing file, I make sure that where there could be no data I use the Skip action where the table is empty so as to not create any errors.

Anywhere any figures from the empty table should appear where I'm appending data to another table, I'm adding in Calculate New Column actions with the column names so that any Aggregation that is calculated is still calculated if the data doesn't exist - I set the values to 0.

If the data does exist, the Calculate New Column action columns get appended with "(2)" which is then removed when the Aggregation happens!

So far this is working very nicely.

This has really highlighted how wonderfully EasyMorph surfaces errors and links directly to where the error is for you to resolve.