About EasyMorph Tutorials & Examples Web-help

Easymorph needs a lot of memory to load : correct?


#1

Hi,

I tried to load a table with 140 columns and about 800 000 records. I could not do this with easymorph because my computer had 4Go RAM left and it did not seem enough. Strange isn’t it ?

I tried the same with Excel 2016 (PowerPivot) and had no problem.

Is it surprising ?


#2

For 140 columns and 800K rows you would need to have at least 8GB of RAM to load.

It’s not strange or surprising. We just don’t have such R&D budget as Microsoft do :slight_smile: Version 4.0 of EasyMorph will have improved compression that will make it possible to load 2-3 times more data, or use 2-3 times less RAM for the same data volume. This will be on par with in-memory engines such as PowerPivot or Qlik.

Meanwhile, if you need to work with large or wide tables – install more RAM. We recommend having at least 8-16GB of RAM. On the positive side, RAM is cheap nowadays.


#3

I got excel file of 80 MB, it has ~ 18 columns, i am trying to aggregate it at 4 columns. Morph processing shows utilizing 100% RAM, later it gets hanged.
My machine has 8 GB ram, i 7 CPU… Will upgrading it to 16 GB RAM solve it?


#4

It’s hard to say. A 80MB .xlsx contains 1-2GB of uncompressed XML data, which needs to be parsed, loaded and compressed which also requires memory. Chances are that upgrading to 16GB would solve it, but I can’t guarantee it.

I would suggest renting for a couple hours a virtual machine with 32GB on Amazon or Azure. 32GB should definitely be enough. You can use the VM to measure exactly how much RAM needed to load such huge Excel file.