Using Synchronize transformation

1

When and why use the Synchronize transformation. A recent post in our blog.

See the original blog post

Using Synchronize transformation

1/12/2024 UPDATE - reposting the whole blog post here:

Using Synchronize transformation

In order to speed up calculations EasyMorph automatically attempts to execute transformations in independent tables in parallel. Here independent means that transformations in one table don't depend on transformations in another tables. In the example below Table A is completely independent from Table B. Although, in Table B only the first three transformations are independent from Table A, because the 4th transformation (Peek) has dependency of Table A which makes all consequent transformations also dependent.

image
image846×398 14 KB

After table 'Census data' is calculated, EasyMorph will start calculating transformations in Table A and Table B in parallel. However, since the Peek transformation in Table B depends on Table A (notice the dotted line from Table A to Peek), it will calculate the first 3 transformations in Table B simultaneously with Table A, then reach Peek and wait until calculation of Table A is finished. After that it will calculate Peek and all the remaining transformations in Table B.

While automatic parallelization is good for performance it may lead to conflicts in cases when transformations in independent tables attempt to modify the same external file at the same time. Most typically this happens when a project exports data into different sheets of the same spreadsheet. In order to synchronize export transformations (or other side-effect operations) in such cases the Synchronize transformation should be used. This transformation instructs EasyMorph to wait until another table is calculated. In another words, it inserts a dependency on another table.

Below is an example where three 1-column tables are exported into three sheets of the same spreadsheet. Notice Synchronize transformations inserted before export transformations in tables "Export to Sheet2" and "Export to Sheet3". They ensure that all export transformations are executed sequentially, even if they are in different tables.

image
image910×654 27.4 KB

2

Hi,
this might be a request that is only useful for me, but anyway:

  • I often use the Synchronize action to ensure the tables are computed in the right order, and the data-dependencies between tables are certainly resolved.
  • Now, during development and debugging, I often have to manually disable the “Synchronize” action. Eitheir because I have manually resolved the inter-dependencies (or because I have ran the flows step by step) or I’m tracing an issue from the target to the source, i.e. in the reverse order of the EasyMorph flows.
  • Anyway, it would be great to have a ‘master’ switch to disable all Synchronize actions at once. For instance, with a parameter-based condition. I guess I’m looking for a ‘Run on condition’, something like in the screenshot below (combining the Synchronize action with the options of the Derive action). Is this a feature you’d consider for the roadmap?
    image
    image440×680 19.4 KB
3

“Run on condition” would be the opposite of “Skip on condition” which already exists.

4

Hi Dmitry,
I’m not sure I agree, or maybe I don’t understand your point. As I see it, I’m not looking for either “Run on condition” or “Skip on condition” - I’m looking for a “Synchronize on condition”. Basically, in my use case, the “Synchronize” action would be active when the parameter {RUNMODE} is set to ‘Production’, but would be disabled/ignored when it is set to ‘Development’. How would you implement this behaviour with the current actions?

5

Hi David,

You can put Synchronize after Skip and will be effectively like adding a condition to Synchronize.