In previous versions the “Aggregate” action in the “Concatenate” mode concatenated values differently depending on whether grouping was applied or not. Without grouping, values were concatenated in the straight order. With grouping, values were concatenated in the reverse order.
Concatenation order before v4.4:
Such behavior was inconsistent and sometimes required additional sorting prior to aggregation. The new version changes the concatenation order to straight one in all cases to make behavior consistent. However, the change is a breaking change. If a project was dependent on concatenation order, then it needs to be reviewed and edited in order to work correctly in version 4.4.
Concatenation order starting from v4.4:
To simplify finding projects that may require editing in order to keep working correctly in v4.4, we’ve created a scanner project that scans EasyMorph projects in a given directory and finds those that can be affected by the change. Here is the project (you will need version 4.4 to open it):
Here is the scanner project:
IdentifyAggregateConcatenate.morph (20.5 KB)
Use it as follows:
- Install EasyMorph 4.4, but don’t save any old projects in it yet.
- Open the scanner project, provide the root folder for your EasyMorph projects.
- Run the scanner project
- Review all projects detected by the scanner project as potentially vulnerable to the breaking change, and edit them if necessary to compensate the change. See table “Breaking changes” to see the locations of Aggregate actions that can be affected by the change.
- Save reviewed projects in v4.4 and re-scan. The projects should disappear in the table “Breaking changes”.
Hint: you can iterate the scanner project from another project, if you have many root folders with EasyMorph projects.
Hi Dmitry,
Thanks for the announcement. I consider it a tricky change.
We have for example a project that calculates a hash for each row in a table using a concatenate of all values in the row and we then apply a hash function to it for each row. We do this by first unpivoting our dataset, then concatenate the values by rownumber (without distinct option enabled) and then calculate the hash.
The data in a row is a mixture of text data and numbers.
Could it be so that there will be an impact because the concatenated values will be in the reversed order in v4.4 when compared with the same dataset in older versions ?
This would result in a different hash.
So If I understand it correcly, we should revert the sort order in the unpivoted dataset in each row to obtain the same result as before v4.4.
How could we do that ? Maybe we could use a autonumbering by row ID for each value in that unpivoted dataset and then sort in the reverse order ?
Wouldn’t it be better to convert the old morph files automatically when they are opened in v4.4 ? So if grouping was applied in the aggregate action, keep the sort order how it was in the old version automatically ? Maybe you could provide a radio button with the type of order: straight order or reverse order with the correct radio button enabled depending on the specific morph project ?
Thanks
Nikolaas
Hi Nikolaas,
Yes, in this case v4.4 will generate a different hash on the same data, compared to previous versions.
There are 2 possible solutions: either workflow needs to be altered, or data needs to be altered. Specifically:
A) Alter workflow: add actions to reverse the row order inside groups before aggregation, or
B) Alter data: re-run hash generation in v4.4 on all old data and don't change workflow
Altering workflow would be as follows:
- Enumerate rows inside the same groups as used for aggregation.
- Sort rows by columns used for grouping in aggregation PLUS the column with row numbers in descending order. Sorting by row numbers in descending order should come last in the list.
- Aggregate as usually. The column with row numbers will be removed automatically because it's not used in the aggregation.
Below is a sample project that demonstrates how to reverse row order.
aggregate-concatenate-test.morph (5.5 KB)
We thought about this but it doesn't seem reliable. Inserting two actions automatically may interfere with the project workflow if there are other enumeration or sort actions used prior to affected aggregation. As a result, another errors can potentially be introduced without any indication or warning.
The only automatic thing happens is that EasyMorph shows a migration warning when an existing project (i.e. saved in an earlier version) with aggregation with concatenation and grouping is opened in v4.4:
We thought about this too, but such option wouldn't have much value beyond one-time migration to v4.4. Supporting until the end of time an option that was only needed once doesn't seem reasonable either.
I realize that the breaking change can be painful. We're supporting you with tools, notifications, and advice how to handle it in a least painful way. It's a one-time effort. Let's just get through it and forget about it.
Hi Dmitry,
Ok thanks for the suggestion ! I think we have the same idea as how to fix the error.
I understand that it is not always so easy to incorporate new features without breaking changes.
One other question: Can we already detect the projects with potentials problems without installing v4.4.
It would be nice if I could scan for projects using the older version of EasyMorph but maybe it’s not possible to create such a scanner project ?
Thanks for the great support so far ! Your support is greatly appreciated.
Maybe I will return with some question when we migrate our projects.
I shall already try to anticipate on the change.
Nikolaas
Thanks for the heads up @dgudkov. It’s a great idea to have a helper project to identify such issues. A good idea for future releases as well as I have been bitten by this kind of issue previously.
Any idea when the new release is landing as I need to start a new project which requires OAuth?
Thanks, Shaheed!
Version 4.4 is about to be published. It should be available within the next 48 hours.
Hi Dmitry,
I suppose that we add the actions to correct workflows should be added AFTER we have done the update to v4.4 ? If we add them before v4.4, the result would again be reversed after the upgrade to v4.4 ?
Here you go. The scanner that works in v4.3.
IdentifyAggregateConcatenate_v4.3.morph (19.4 KB)
Hi @dgudkov
Does this straight ordering only work with numbers? I have a column with text but it does not order alphabetically:
Regards,
Shaheed
@Shaheed_Fazal,
the ordering is only applied to the row order. It doesn’t sort values. Concatenation still happens from the 1st to last value in group, without sorting.
If you need concatenated values to look the same (e.g. only “No, Yes” but without “Yes, No”) then do sorting inside groups by values, not by row numbers.
What should we do with projects for which the scanner project says that it cannot scan?
We should open them manually to check if there’s an impact ?
Save them in v4.3 and then re-scan.
