Hi,
(First - thanks to EasyMorph team - I love this product).
I wanted to ask a question on what are the best PC specs to get optimal/fastest performance out of EasyMorph desktop. Is there a maximum amount of Ram , after which, adding more Ram does not help? What about the number of cores? Any tips here would be most helpful - for example, to help avoid ‘out of memory’ errors.
I tried to upload a shot of my PC stats while running an EM project. Not sure if it will come up, so here are the stats with - basically - only EM running:
CPU - 90% to 100% of 4.49 GHZ
Memory - 27 of 31.9 GM
177 processes, 2309 threads,
It looks like your computer is almost out of available memory. You need more RAM for the data volumes you process in EasyMorph.
Note that EasyMorph is a full in-memory application, which means that it keeps all data loaded in memory for every project it executes. While it uses different compression algorithms it can still be a lot of data, especially on large tables (millions and tens of millions of rows).
When a project finishes and is closed, all memory is released. EasyMorph Scheduler and EasyMorph Server are a bit more memory efficient as they drop some intermediate calculations, while Desktop keeps all of them.
There is no any good methodology to calculate how much RAM actually needed because not only it depends on data volume, it also depends on the number of transformations in project, types of transformations (joins, appends, etc.), project topology and some other factors. The most effective way is to empirically measure memory consumption, and increase the amount of RAM when your computer is close to utilizing >90% of available memory (which seems to be your case).
Currently, your computer has 32GB of RAM. I would suggest increasing it to 64GB. If 64GB is enough then adding more memory won’t make things faster. But in general, there is no memory limit for EasyMorph. Some customers run EasyMorph Server on computers with 256-512GB of RAM, and that’s OK.
The number of CPU cores is secondary. It helps to have more cores, because EasyMorph automatically parallelizes certain calculations. But the dependence is not linear.
If you execute multiple ETL-jobs on the server where maybe some ETL-jobs that run simulteneously use the same files simultaneously, can this lead to errors or is EasyMorph smart enough to handle this?
Secondly, does the server do its memory management by itself so that we cannot run out of memory when we plan multiple ETL-jobs at the same time ?
It's smart enough to handle short-term file access conflicts (up to 2-3 seconds). Longer file access conflicts will lead to error because something may be wrong with the project logic.
The Server can't expand the physical memory. If you're running out of it, you're running out of it. Make sure you have enough RAM.
It depends on memory required by tasks. If you have 64GB of RAM available and two tasks that require 4GB each then it won't be a problem to run both tasks simultaneously. In any case, you should have approximate understanding of memory requirements for each task.
Just checking - the graphic card you use makes no difference to performance? My colleague here is suggesting that it might, as graphic cards are used for various processing tasks these days. My current graphics card is low grade.
