About EasyMorph Tutorials & Examples Web-help

Best Practice for loading a huge file


#1

Hi,
In a transformation, I have to import a very big csv file (more than 5M rows) that changes only once a month, and many other small files (100 lines max) that change frequently.
Each time, I have to wait the big csv file to be imported.
What would you recommend to avoid or limit this time consuming task (except at the beginning of each month). Should I export the csv in a MySQL database? Other solution?
Best,
Michel


#2

Hi Michel,

Exporting to a database won’t help because it would be similar performance. Load the CSV file in EasyMorph and export it into a Qlik QVD file. This is a compressed data format and EasyMorph reads it faster than regular text files.


#3

Thank you, Dmitry, for this advice.
Is an “Import a .tde file (Tableau)” transformation in the roadmap ?
Michel


#4

No, it’s not. Tableau doesn’t provide API or specification to read .tde files. We can’t do much here.

If one day they start providing a specification we will surely add it.


#5

BTW, starting from version 4.0 it will be possible to save loaded data right in EasyMorph projects. So that when you open a project its start transformations already contain last loaded data. It will also be possible to export/import to a native EasyMorph format which will be very fast to read/write.


#6

That’s what I thought, too!