In the next release (v5.6.1), we will introduce the ability to edit local and Server-hosted datasets (.dset files) as well as sandboxes.
(Right-click and choose "Open image in new tab" for better quality)
The list of operations in the Dataset Editor includes:
Editing cells
Inserting/deleting columns/rows
Renaming columns
Pasting data (e.g from Excel)
The Dataset Editor can be invoked from:
The "Import dataset" action
The "Sandbox" action
File Manager (for .dset files in local and Server locations)
Catalog (for static datasets)
The Dataset Editor works reasonably well with relatively large datasets - we tested it on datasets with up to 1 million rows.
In combination with the new "Append" mode of the "Export dataset" action (also coming in v5.6.1), the new feature opens a number of new use cases for EasyMorph:
Fixing data quality issues (e.g. bad rows can be exported into a temporary .dset file which is edited later manually and then re-processed)
Storing and editing various lookup and configuration tables on EasyMorph Server
Simple data store (e.g. collecting data provided through parameters in workflows - Server tasks, Catalog workflows, or webform submissions via the API Server)
How does the new capability look to you? Suggestions, ideas?
This is very useful as at the moment I have to filter the data, export to an excel file, edit the file, and then re-import and update, so this would be massively useful.
It would be useful to have a filter and/or search option to quickly find particular data to edit - I typically have millions of rows but just need to edit a dozen or so cells!
Can there be a way where if I'm looking at the data and it is in a derived table (which could be many derived tables down the process) that I can right click or similar to edit that cell and it takes me to the source, or allows me to edit in situ and updates the source? I often see errors towards the end of the process is the point I'm trying to make!
Search will definitely be added, just not in the initial release - so much to do even for the initial release.
Individual cells can't be tracked back, but columns can (to some extent). To see the column's lineage and track it back, select the column and press the "Lineage" button:
This is really interesting @dgudkov. I have been pondering on dsets, the API gateway and now this feature. Does this mean one could use EasyMorph as a backend for a fully fledged application? I need to make an app internally and was thinking of using FlutterFlow. FlutterFlow integrates with Firebase and Supabase but I am wondering whether we can get away without them. What are the pros and cons of using EM?
We're making two improvements to enable the use of EasyMorph as a simple data store for external apps:
The Dataset Editor described above.
The "Append" mode in the "Export to dataset" action.
With these two improvements, EasyMorph gets "database-like" functionality that can be used in simple low-workload, low data volume scenarios.
Pros:
Easy to use, built-in data editor, suitable for non-technical users
Native data format for EasyMorph
No need for an external database
Simple backup/restore since it's just .dset files
Cons:
Not suitable for high intensity workfloads. E.g. apps that that perform multiple reads/writes per second
Not suitable for high data volumes. Unlike in relational databases, the time required for appending to a .dset file grows proportionally to the file size (for databases it's constant). There are techniques to get around it (e.g. append to a new file every day), but it still limits the scope of use. Also, while technically possible, Dataset Editor might require a lot of RAM when editing large .dset files (e.g. >1 mln rows)
Not suitable for highly parallel use. While there is a mechanism of optimistic locking when appending to a Server .dset file, it's not as good as the mechanisms provided by databases. Also, when you open a dataset in Dataset Editor, edit it, and save back, then if any records were appended to it while it was edited will be lost (overwritten). While EasyMorph will warn you if the source .dset file has changed since opening, it still might not be convenient.
