Freeze or Pause Large DB Queries

1

Hi extracting very large tables from Redshift through ODBC. This query takes approx two hours but works fine once it’s all there. Eventually I will need to restart my PC. Can I freeze or pause the query so when I reload EasyMorph I don’t need to update those queries and wait two hours again? My workaround is to export the large tables to a local csv and use that as the data source by cutting the link to Redshift. Wondering if there is a more obvious path that I am missing.

2

Hi Rick,

The only way to keep an imported dataset is to store it in a local file. But it’s better to use DSET file instead of CSV. DSET files are several times faster than CSV.

3

Thanks for the response Andrew! That’s what I’ll do

4

Also, if the database table doesn’t change often, you can create a one-off project that uses iterations saves the table by partitions (e.g. 1 date = 1 partition) into multiple .DSET files. In the case, you will be able to load only as many partitions (.DSET files) as needed.

The project would work as follows:

  • Run a query to get a list of partition IDs
  • Iterate: for each partition ID pull the partition (subset of rows) and save it as a .DSET file named after the partition ID