Hi extracting very large tables from Redshift through ODBC. This query takes approx two hours but works fine once it’s all there. Eventually I will need to restart my PC. Can I freeze or pause the query so when I reload EasyMorph I don’t need to update those queries and wait two hours again? My workaround is to export the large tables to a local csv and use that as the data source by cutting the link to Redshift. Wondering if there is a more obvious path that I am missing.
The only way to keep an imported dataset is to store it in a local file. But it’s better to use DSET file instead of CSV. DSET files are several times faster than CSV.
Thanks for the response Andrew! That’s what I’ll do
Also, if the database table doesn’t change often, you can create a one-off project that uses iterations saves the table by partitions (e.g. 1 date = 1 partition) into multiple .DSET files. In the case, you will be able to load only as many partitions (.DSET files) as needed.
The project would work as follows:
- Run a query to get a list of partition IDs
- Iterate: for each partition ID pull the partition (subset of rows) and save it as a .DSET file named after the partition ID