Add ignore option on some actions which work with columns

Feature requests
1

Hello,

Adding an ignore option for some actions when columns do not exist would greatly help in some projects which needs flexibility:

Case 1:

For some projects we have to convert the DataType of two columns. However in some cases one of the column won't be present (it can be one or the other). It is possible to bypass this via derived tables however it adds complexities in the project.

Context: customer upload a file with data and different columns, they are not forced to add a mobile number BUT if there is a mobile number we have to format it. The same goes for the invoice number. So once they can upload a file without a column Phone and the other time it will be without the column Invoice.
Actual solution implemented: add a "calculate new column" after the import with those two columns which will be renamed 'xxxxxxx(1)' if they already exist, afterward use the function "keep columns".

Case 2:
The same issue is happening on some other actions like the "remove columns" action, in some cases we have to remove three columns but in other cases we need to remove two. And we cannot use the "keep columns" function because the ones that we have to keep will vary as well.

Context: Customers will upload a file with a phone number and an email. An API is called to make some verification (eg: the number is a mobile or not). Depending of the type of phone the API will provide different fields. If there is only one type of of phone then some columns won't be in the response. But sometimes they are present and we need to remove them afterward.
Actual Solution Implemented: Derive tables + skip actions + more derive tables + skip actions.

So in those cases we can always find solutions to fix the lack of options in some actions. But we think that having some type of rules/options which will bypass the action if the column is not found would be a great feature which would offer more flexibility in some projects.

1 Like
2

Hi @Alex,

Would the columnexists function simplify your projects? You can use it both in the Skip Actions on Condition / Derive table with condition.

Regards

1 Like
3

Hello @roberto ; yes that could work but that can make a lot of Derive Table if many different columns have to be checked. Which is why I thought that such an option would be useful.

However for the conversion part that function would work as it could be used in a Rule action. That will already help me thank you !

There is still the part of the column deletion, that would seem logic to have that option there, if the column doesn’t exist and cannot be deleted the project should not stop. (We can’t use Keep columns because they are flexible, we only have specific ones to delete).

4

Hi

I support your idea of “ignore” option. It should be a standard option in each action.
For example: when we want to debug a project and put a filter on a specific case, the flow throws an error on each aggregate or other action when columns do not exist.
Also when there’s is a variable dataset input, it would be nice to have this option. So I fully understand your question as I have encountered the problem also in several occasions.

Kind regards
Nikolaas

5

I support the idea of an ignore option when using the keep/remove option. I'm working on a dataset where the available columns vary, which causes my job to fail without manual intervention.

6

I worked around this by add column action where not exists as not all the expected columns were being returned - this resolved the issue as this forced all the columns to exist

7

@mike

The "Keep/remove columns" action now has an option to not fail if the columns to be removed are not found. The option is available starting from v5.8.

image