I have a project for which I extract data from a database and export as DSET-file. Those files are then used in EasyMorph for further ETL.
For testing reasons, I would like to switch from DSET tot CSV because of the fact that performance on CSV has improved from version 4.0 and it allows of importing a fixed number of records for testing purposes.
When I change my source from DSET to CSV, will this impact on the ETL EasyMorph flow. Are data types going to be correctly interpreted when I switch to CSV since the data comes originally from a database ?
It depends. DSET and CSV are different data formats. Therefore there can be differences that may or may not affect correctness of calculations depending on your projects' logic and import settings in the "Import from delimited text" action.
CSV values are stored as text and then parsed into the EasyMorph data types on the fly when imported.
For instance, if you store a numerical text value in CSV as
123456
then it will be imported as a number. If you store it as
"123456"
and quoting is ignored on importing then it will be imported as text.
DSET stores values in the native EasyMorph data types. If you read a CSV file and then immediately saved it into a DSET file then importing from this DSET file would be identical to reading the original CSV file (assuming that import settings didn't change) because no parsing or data conversion happens.
Thanks for the info.
So if I run a query on my oracle database exporting de results to DSET and CSV and then import the 2 files back in EasyMorph, will they have the exact same properties ?
Note that wen exporting from the database to the 2 files, no data type transformations or other transformations are done. Maybe this could pose a problem for dates in the CSV-file that will be exported as numeric and then on import are not necessarily recognized as date ?
They may not - DSET and CSV are two different formats as I mentioned above. DSET will restore the query result verbatim. CSV converts data into text on writing and back from text on reading. It can change depending on what you are exporting to CSV, what export options you use and how you read it back. You need to analyze your import/export actions and test various cases in your data in order to get a definitive answer.