Hi,
I have issue with json parsing and I can’t find workaround.
For simplicity, I take a minimal json.
{"magento_id":26361}
There is no option to choose the format with “parse json” action.
Converting data type does not work.
json-parsing.morph (1,5 Ko)
Can you help me ?
Hi @Florent,
EasyMorph assumes that that number represents a date, so it automatically sets the format of that column to a date format.
Please change the format of that column to “No format”:
Thanks.
I had never seen that selecting a column displayed such menu.
Help to “Parse JSON” action doesn’t talk about it.
This therefore means that for all identical actions it is necessary to go to each header to specify how the data must be interpreted. Otherwise, overnight the treatments will no longer work because an ID is in an range which is interpreted as a timestamp or a date.
JSON often arrives from a web service whose data is not always present. It’s a real problem that you don’t always treat data the same way.
In fact, I was lucky that the case come on the 1st recording to be processed when going into production.
It's not data interpretation, it's data formatting and it works for any action, not just "Parse JSON". The formatting doesn't change column value, it only changes how it's shown. Very much like cells are formatted in Excel, except in EasyMorph formats are applied to columns, not to individual cells.
Column formats are saved in the project, so once you change a column format, it remains so, no matter if it has data or not. Also, column formats are linked to lineage, so whenever the column appears, it will have the same format. Finally, you can set column format for multiple columns at once.
I would suggest using column formatting for a while and learn how it behaves. It's not that bad.
Passed from 26361 to 03-Mar-1972 is the result of an interpretation in timestamp then the formatting of this timestamp in date.
From the start, this value is handled well as an integer in EasyMorph.
I deduce that if we do not force data’s format, interpretation/formatting can change on its own depending on the data present.
In the present case, parsing JSON from Magento result to 387 columns with differents formats.
Other columns can be added dynamically according to options used by customers.
I must create tests cases for all cases to fix format to each columns.
And this is just the interface to pick up an order. We have plenty of other JSON to read like customer, items, stock or price.
In return, I create JSON send to Nav and receive another JSON.
Same ID is used several time in all these JSON. And only one is parsed in date.
It’s going to take a long time, but we can do it. I don’t understand why, only in one place, formatting change.
It would have made more sense if format used was automatically saved to avoid random operations.
Maybe this would be a problem when building the morph.
In this case, a button allowing to freeze file formats would be the easiest.
For your information, this morph have 369 actions in 10 modules, call 12 web services and call 2 other morph.
Data doesn’t work like it used to.
Now, some columns change automatically to date format from Magento.
Some tests :
EasyMorph didn’t work like this before :
Format is automatic.
I have tested to force force data to “1000”.
Now several fields integer format change in EasyMorph from “No Format” to “dd-MMM-yyyy”.
Test format.morph (2,4 Ko)
When we say “new columns” in EasyMorph it may actually mean two different things:
New columns that are created by an action that was just created. In this case, column formatting is set to auto-detection. There is certain heuristics to auto-detect dates, but it may guess format incorrectly at times. When the project is saved, the column formats are saved as well.
New columns that are created by an action that was created earlier and already existed in the project when it was opened. In this case, the column format is not auto-detected and is simply restored from the project.
It's not true. I have been working on this file since January. I only changed one input parameter and formats changed. All columns had been here for many months.
Here is an example which shows that it is not only when creating actions that the data is interpreted.
Creation of a new Morph. Parameter with value "2", action "Parameter table" and action "Calculate new column(s)"
Change paramater's value to 26360.
Save project and close EasyMorph.
Open morph file.
Final morph : TestDate.morph (1,4 Ko)
If we change bad formatting to "No format", next open, format change.
So far, the conversion action and functions have been enough to handle everything without this system.
I chose EasyMorph because it was efficient, easy to learn with rapid development.
Now I see that some minor failures are wasting time and data processing is unreliable.
The situation is now as follows.
My colleague uses EasyMorph for stats and I think he will continue.
I am using EasyMorph to make interfaces and this program does not provide data continuity. An interface may stop working due to slightly changing data.
If we would have directly connected a B2B, Magento interface would have crashed in a few days without explanation.
So it is very likely that all of my work goes in trash.
My apologies, I should’ve made it more clear. Don’t set it to “No format” because it means no formatting at all so it will cause auto-formatting again. Set it to a number format such as “1000” and save the project. Then the formatting will be saved and it won’t be auto-detected again.
Formatting doesn’t change cell values. It only applies to how EasyMorph displays numbers (also, some actions have an option to apply the formatting when exporting into text formats). Otherwise, formatting doesn’t affect workflow logic at all, so nothing will break.
For instance, when you send a web-request, even if a column is formatted as a date, the web request will still contain a number (see below). Because again formatting doesn’t affect the cell value.
I understand. I use several actions “construct json” and format is changed. I don’t want to change or select a format between read json and created json.
We were looking for a tool to allow systems using REST web services to communicate with each other. We need to read the json, do some transformations and keep other data in json file intact (including format) to send a json to another system.
In abstract, EasyMorph doesn’t know how to manage presence of optional data (we must test and create all columns not present in EasyMorph) and modifies formats.
Even if EasyMorph is very good for getting out of statistics and extracts data, it is unsuitable for managing modern interfaces.
Now I am looking for a new reliable tool to manage interfaces.
New issue. Loss of all formats in Morph file.
=> New crash in Prod for same issue.
I call web service from Magento. I get list of new orders.
When I open Morph and json, I fix format. Save morph file. Format is in Morph.
Open Morph and no new order are present. As I have modified an action, I save morph file and close it. Result:
Open Morph and new orders are present:
Not only must format be imposed, but in addition they must be reapplied.
I do not do any calculations with these fields. I am building another JSON with it to send it to another web service.
Being able to do calculations despite formatting doesn’t help because only the formatting matters.
It is therefore impossible to use EasyMorph with a REST API because of this issue.
We will look at the issue with formats. Speaking generally, I see that EasyMorph should provide better means to work with JSON. This is OK, because the tool is not static and is rapidly developing. Feedback like yours help us understand areas of improvement. We’re learning about certain things together with you.
We will introduce a few actions to simplify operating with JSON, including ways to modify parts of JSON while keeping other parts unchanged.
Also, while not mentioned in your posts, the “Construct JSON” action will get additional settings to deal with formatting.
I believe all that will eventually resolve the problem.
Meanwhile, I would suggest trying to not decompose the source JSON entirely and find ways to preserve parts of it unchanged, and then inserting these parts into the resulting JSON.
The “Construct JSON” action can detect when a table value is already a proper JSON. In this case, it inserts it as is, verbatim.
I will give a few examples to make it more meaningful. I have formatted json files to make them readable.
(This is test data. The addresses and people are fictitious.)
I get list of new orders to import from Magento.
This list can be empty.
json Magento List Order Empty.json.txt (30 Octets)
Or with data.
json Magento List Order.json.txt (597 Octets)
As you can see, result has nothing to do in EasyMorph after parsing json.
After get list of new orders, I get each order to import them. A lot of data are optionnal and not already present.
Data from Magento:
json Magento Order.json.txt (41,4 Ko)
I transform it to import into Nav. I search other data in Magento to create this file:
json Nav Create Order.json.txt (6,2 Ko)
I think that “integer” and “date” are not stored in same way in json. parsing json can set default formatting.
I don’t know how you can do this since EasyMorph needs the columns to store parameters and do some manipulation.
The easiest for you would be to add a parameter on parse action to provide a full json with all possible columns. Thus, you could build the table with these columns to preserve parameters and leave empty the not present fields in json. But I don’t think this is workable with all structures in json.
Currently I don’t see a way to not manipulate all the entire json in EasyMorph. Data are to complexe to do transformation with function and parse action doesn’t really give an option for that.
I started to rewrite part of the morph to test another method. I get all json needed and create a json with all file. After, EasyMorph will call node.js to try transformation with jsonata and send result to Nav without alteration.
EasyMorph should be able to manipulate json because this format is more and more present in interfaces. It’s a shame to have to rely on a library and an external language to do what EasyMorph is made for.
