Generic project for checking data types

1

Hi,

I am trying to make a project in EasyMorph that checks data type specifications based on a mapping table that gives the data type for each field.

I have created a test dataset with some columns of various types. Currently I was able to distinguish the erroneous values by separating out the fields by each data type and by using the filter by type action.

Now, I also want a check on dates. This type is not supported in the filter by type transformation. I know dates internally are represented by numbers. How can I tweak the filter by type in such a way that it can distinguish between normal numeric fields an date fields represented as numbers ?

I have put my project and data in a zip file. Could you give me some hints so that it can check on dates also and tweaking it so that I can reuse it on no matter what dataset I receive.

check_data_types.zip (16.6 KB)

Thanks in advance !

2

For dates you can use the “Filter by condition” action instead of “Filter by type”. To make it more flexible, you can define parameters {Min date} and {Max date}. In this case the filtering condition can be as follows:

if(isnumber([Data]), [Data] < {Min date} or [Data] > {Max date}, true())

This filtering condition keeps all values that are not a number, or numbers outside of the boundaries of what is considered a date.

I’ve updated the example with a check for dates.

check_data_types.zip (17.1 KB)

3

Ok thanks a lot. I will check it out and maybe post my final version to the forum.

4

Hi Dmitry,

I am not sure of having understand the condition.
What if I have an excel file with a date column. One of the cells in that column is an integer that falls in the boundaries of a date data type.

I think the condition you mentioned will not detect such cases ? Is there a way to also detect these values ?

It would be handy if there would be a checkbox for “date” in the filter by type transformation.

Kind regards

5

Dates in Excel are numbers just like in EasyMorph. The only way to check if a number from Excel is a date or not is to check if it falls into specific range.

There is no data type “date” neither in Excel, nor in EasyMorph.