Create a daily flow without an external db

1

Hi all,
I have to create a daily flow, my daily steps:

  • download data from a source
  • clean, transform and select data
  • send selected data to a csv file

Every days I download all data, so every days I have to check data selected are not already sent.
To do this I thought to use an external SQL database, so that:

  • at the end of ETL process I append selected data into SQL table
  • every days I check with the SQL table if data selected are not already sent

My question: there is a way to do the same without an SQL db, using only easymorph tables and modules?

Thank you
Claudio

2

Hi Claudio,

An SQL database is optional here. Instead, you can store the processed in EasyMorph dataset files (.dset) files. The data format of .dset files is native for EasyMorph and therefore EasyMorph can read and write them very quickly.

To append a batch or rows to a .dset file, import the file into EasyMorph using the “Import dataset” action, append the rows using the “Append table” data, and export it back with the “Export dataset” action.