About EasyMorph Tutorials & Examples Web-help

Export To Database - Failure Detail Observations & Question

Hi -
When a failure with the export to DB occurs, is it possible to return additional error messaging or perhaps automagically iterate in smaller chunks to identify truly bad data as things are done in Batches?

As an example:
Ref

I’m going to attempt to export to a MySQL DB 10,000 rows per Batch
image

The Data is Largely a mess but I wanted to ensure I capture the “raw” data and make it query-able if needed at a later time.

I expect I will get at least 1 failure:
Of the 42,546 rows attempting to export 22,546 have failed “#Failed”. I assume this is just based on positioning of bad data that caused a failure.

To minimize this I could set the batch to a smaller number and have more batches, thus slower export but possibly better results in export (if more good data exists than bad)

So if I repeat the exercise and set the batch size from 10,000 to 1000 I went from 22,546 failed rows to 4,546 failed rows.

I know based on past experience that if the DB is accepting rows, then its likely something with the format of some of the data and DB constraints

If we profile it, I see there is a very large amount of data in at least 1 of the records. We know that by looking at the DB it won’t accept it


For Example Sake
SQL Error [1406] [22001]: Data truncation: Data too long for column ‘Column1’ at row 1

Long way of walking through, but it would be very nice if we had a status “OK vs Failure” and if Failure an extended column with the returned message

Not sure if that sort of thing is possible.

Hi Adam,

Thank you for the suggestions. We will replace the generic error message with the full error message - this would be more helpful, for sure.

As for “automagically iterate in smaller chunks”, the recommendation is to keep (filter) the failed batches of rows (those with errors), and then try to export them again but this time in the “Halt execution, roll back…” mode (see below). In this mode, EasyMorph “automagically” reduces batch size logarithmically and retries exporting until it singles out the exact row that fails.

image

1 Like
To learn more about EasyMorph visit easymorph.com.