Hi,
We are exporting a dataset of 7 mln rows to DSET-format and it takes a very long time to complete in Desktop (much longer of export to CSV).
What could be the reason for this performance issue ? When we run the task on the server it runs within an acceptable time period.
Kind regards
Nikolaas
If you read this .DSET file and export into .DSET again, is it still slower than export to CSV?
**** moved to #support
I have imported the CSV and exported it to DSET an now it went very quickly (export to desktop).
On the other hand when it is embedded in a longer data transformation process, it does take way much longer for the DSET to export than the CSV. Very weird.
The file size of the DSET during export is growing with a couple of kilobytes at a time (export to network drive). But also when we run it on the server, we export to the network drive, so that cannot be the problem.
It seems like your computer is running out of memory when you run the longer process (which apparently is memory-heavy). Windows starts swapping and that slows down all running applications, including EasyMorph.
When you just load one CSV and export it, it doesn’t cause Windows run out of memory and start swapping. Therefore, it’s fast to execute.
I would suggest running memory-heavy processes in Launcher as it’s more memory-efficient than Desktop. Desktop keeps all intermediate results in memory (albeit compressed) because they are needed for instant data visualization. Launcher (as well as Server) doesn’t do data visualization and keeps in memory only some intermediate results.
Other suggestions:
Thanks for the explanation !
I believe that it is really an issue here. When I export to qvd from the same point in the flow instead of dset, it works in seconds and with dset, it is export with some kb/sec taking minutes to complete.
Could you do some performance testing on this ?