Sorry if this has already been answered, but I couldn’t find anything about this…
I see that EasyMorph is using SQLite for some internal stuff, but I am wondering if that can be replaced with some other type of repository? Some of the data we want to possibly push through EM has over 100M rows and I don’t believe SQLite is the right type of Db to handle that type of volume with acceptable performance.
Can you change the backend repository in EM?
Thanks
Keith
Hi Keith, and welcome to the Community!
SQLite is only used for storing information about database connectors (e.g. server host, port, database name, etc.). Data itself is never stored or processed in SQLite.
EasyMorph doesn’t need an external database to transform data. All transformations and calculations are performed in EasyMorph’s fast in-memory engine.
Hundreds of millions of records can be processed in EasyMorph without any problems. You may only need to use some kind of partitioning and iterations to process a big dataset in chunks, rather than at once.