Data type check when using bulk load action

[reynsnivea]

Hi,

The new bulk load action is a great improvement in EasyMorph ! Thanks for releasing this feature.

In the standard export to database action, there’re some options available.

One of the options is to check if the data type in the dataset to be exported matches the data type in the target database as shown below.

How does that work in the new bulk export action ? Does it raise an error when types do not match or does it export the value as a null value ?
Maybe it would be better if we could have the same option like in the standard export to database action with a default value of “Fail on data type mismatch” so that errors are noticed?

There’s also a custom SQL-statement available in the standard export action. We use that often to first truncate a table before loading it. In the bulk load action, this is not available yet so we have to use a database action with custom command. That’s our workaround for the moment. Adding that custom SQL action could also be a further improvement?

Thanks
Nikolaas

[dgudkov]

@reynsnivea

How does that work in the new bulk export action ? Does it raise an error when types do not match or does it export the value as a null value ?

The action fails in case of a data type mismatch. So you won't miss an error.

@reynsnivea

There’s also a custom SQL-statement available in the standard export action. We use that often to first truncate a table before loading it. In the bulk load action, this is not available yet so we have to use a database action with custom command. That’s our workaround for the moment. Adding that custom SQL action could also be a further improvement?

That's the correct flow. We don't plan to introduce custom SQL in bulk export yet.

[reynsnivea]

@dgudkov

The action fails in case of a data type mismatch. So you won’t miss an error.

Great !

[adambeltz]

I know this is an old thread but it looks like the most relevant to my similar use case.

Scenario:
.csv file is delivered to file location, and we load to snowflake using the bulk connector.

1. I'm currently using the bulk loader to ensure that the data doesn't fail the load (effectively adding duplicates)
2. Then I'm deleting all rows (removing duplicates)
3. Then I run the bulk load

I'm doing this to avoid deleting all of the rows only to find that the file is bad/issued on data time/format ect.

Am I thinking of this correctly?

[dgudkov]

That's an interesting approach. I was thinking initially about using a temporary table, but the more I think, the more I like your way of doing bulk export. Can't think of a better way.