Is there a limit on number of columns when loading data?

1

Hello, I’m trying to load a csv file that has ~17 columns… For some reason only 13 columns are loading… Is this the limit?

2

No, there is no limit on number of columns.

@andrew.rybka can you please take a look here.

3

I was able to successfully import a CSV file with 30 columns.

@paulisdataviz Is it possible for you to share that file with us? You can send it to support@easymorph.com.

4

Thanks Andrew,

So, actually one quick correction, this is not a csv file, this is a “pipe delimited file”

I’ll try to get a sample file that I can share with you, but what I noticed so far is that in the underline data there are some rows with just 13 fields but some others have 17/ 18 fields but for some reason easymorph is loading all the rows with 13 columns.

I’ll try to get a sample file that I can share with you and send it to the support address.

Thanks,

Paula

5

Paula, thank you for the file.

EasyMorph detects number of columns in a delimited file from the first parsed row (usually it’s the first row of a file but it can be changed by “Skip first lines” option).

Since the first row in your file has only 13 fields, EasyMorph expects that all the rows in that file will have 13 fields. You can turn on “Show parsing errors” checkbox in “Advanced options” and see that rows with a different number of fields are marked with errors.

The possible workarounds are:

  1. Add a new row to the beginning of the file with 18 column names, delimited with pipe.

  2. Add a new row to the beginning of the file with just 17 pipes - EasyMorph will assign column names like Column 1, Column 2, etc.

  3. Move one of the actual rows which has 18 fields to the beginning of the file and check “Columns don’t have headers” option. Column names will be the same as in the previous workaround.

Thanks,

Andrew

6

Hi Andrew,

Thanks for reviewing and explaining… I haven’t seen this type of behavior on an ETL tool before but now that I know how EasyMorph works, I’ll workaround it… Probably this could be something that can be enhanced in a future release where it quickly scans all the rows to find out what the maximum number of columns is.

Well, anyways thanks for the quick response… I really like your product.

Thanks,

Paula

7

Another suggestion – load entire text as not delimited, i.e as 1 column, and with the “No headers” option checked. Then use the “Split delimited text into columns” transformation to split the column into 18 columns. In that transformation you can explicitly define the 18 columns.

8

Thanks @dgudkov,

That is an easier workaround.