How to run iterate module without parallel

1

Hello,

I've been using EasyMorph for a few years now.
Today I'm preparing a data transfer. I'm extracting a large quantity of data to build json files.
For historical reasons, I don't have much RAM on the server and I can't get more.
I have the following processing and the fact of having parallel processing means that the memory is full.

image
image278×325 10.1 KB

To sum up, I retrieve the list of tables I want to use, and for each table a module is called up to extract the data and create a json file.

When the memory is full, I get this message.

image
image1152×696 56.7 KB

Do you have any ideas on how to avoid running the module in parallel for each table?
There are a lot of tables and doing a manual flow for each table with a "synchronize" action is too time-consuming. Especially as the list of tables is set to grow.

Thanks.

2

I found a solution. I plit SQL extractions and files by 100,000 rows, which means I don't have to fill up memory because it doesn't run enough processes in parallel to reach the limit.

image
image456×437 12.8 KB

2 Likes
3

Yes, partitioning your data and processing it chunk by chunk is the right approach to process large datasets on machines with limited RAM.