Append Table Taking Long Time

1

Hi,

I have imported 3 QVDs - quite large files.

They all import very quickly - few seconds.

But when I try to append table (to rows at bottom) it takes a very long time to execute.

Any hints?

2

How long does it take to execute?

3

It takes around 14 mins. Its strange because all I do is a) load three files and then b) append the three tables together.

The three files load in seconds. I dont understand why the append operation takes so long…

4

Hi,

I have noticed the same problem. Even if you append a small table to a very big table (and not the other way around), it takes a long time if I recall well.

Kind regards

5

14 minutes that’s too long, it shouldn’t be so.

How big are the tables (columns by rows)?

6

18million onto 18million

  • too much?
7

So I redesigned that project flow to produce the same outcome without the append action. Its definitely the append action that is causing the problem - it even produced an out of memory error. Yet I can produce the same size table (identical in all respects) with a different set of transformations.

8

Thanks for the details.

I ran some tests and I see that the Append action works correctly. Although, it can consume a lot of memory as it needs to append datasets first and them compress the result. I understand that in your case there wasn’t enough RAM on your computer which caused memory swapping on disk which in turn caused slowdown and system instability.

If you’re appending files, you can append them right on loading, by loading multiple file at once in an import action. This would be more memory efficient. You can use either the “Load multiple files” mode, or “Load list of files”. See our tutorial on how to use these modes: https://easymorph.com/learn/load-file.html