Limit the amount of data using "Export to Database"

1

I have a EM dataset with a large number of values. One of the values is a date field. For example:
col1 datecolumn
Row1 xyz 8-Nov-2022
Row2 abc 7-Nov-2022
Row3 def 6-Nov-2022

I’d like to load only the rows that are greater than a LOADDATE (e.g., 7 Nov would load Row1 and Row2).

So far, that’s easy … but I want to get the LOADDATE from a database table. I can easily issue a query and get the LOADDATE, but I don’t know how to remove the rows that don’t satisfy the criteria. (e.g., Row3 would not be loaded because its date of 6 Nov doesn’t satisfy the LOADDATE.

I looked at using custom SQL to load the rows but I don’t know how to put LOADDATE into the WHERE clause.

The idea is to limit the amount of data loaded to the database to just the most recent rows.

Any help would be greatly appreciated. Thank you.

2

It looks like you need to do an incremental load. I would do it as follows:

  1. Query the database table to retrieve the max date
  2. Use the “Peek” action to append the max date as a column to the exported dataset (in another table)
  3. Use the “Filter by condition” action to remove rows where [Date] < [MaxDate]
  4. Export the remaining rows into the database
3

Thank you very much for the suggestion. I had not used Peek before but now recognize its usefulness. Thanks again.