💡 Data Catalog and roadmap in 2023: new pricing, new features

1

New licensing model

In version 5.4.2, we're introducing a new licensing model for the Data Catalog. Starting from v5.4.2, the Data Catalog is available for unlimited users in all Server editions in spaces that don't use Active Directory.

For spaces with Active Directory (only in the Enterprise edition), the Data Catalog add-on is required, and it enables unlimited users in all spaces with Active Directory. Also, the price for the Add-on has been significantly reduced. The new pricing is already available on our website.

Permissions and user roles

In spaces that don't use Active Directory for authentication, all users have full permissions. Any user can create and retrieve Catalog items of any type, which is very similar to how users can now work with Server tasks.

In spaces with Active Directory, there will be a new roles and permissions model starting from 5.6. The permissions will include not only the Catalog but all other features of EasyMorph Server: Tasks, Files, etc.

The Enterprise edition will also allow configuring user roles and identity providers. Each user role will have its own set of permissions. Also, other identity providers will be supported besides Active Directory (e.g. Google, Microsoft). Server administrators will no longer be anonymous and will always have an identity.

Metrics

Also, in version 5.6 (or a bit earlier), we will be introducing a new item type, the Metric. Metrics are intended for displaying business KPIs (key performance indicators). Unlike regular Catalog items, they don't have to be retrieved. Instead, they will display their value instantly. They will look somewhat like this:

image
image1532×502 21.6 KB

From a technical standpoint, each metric is linked to a value in the Shared Memory. It also has lower and upper thresholds to indicate "good" and "bad" ranges. Metrics are updated when the respective Shared Memory values are updated using an EasyMorph workflow (or an API call).

Since metrics get their values from the Shared Memory, it means even external applications can update or obtain metrics via an API call. In other words, the use of EasyMorph workflows for updating metrics is optional. Metrics can be calculated by external applications. Metrics can also be consumed by external applications such as BI systems.

File browser

The File browser has already been announced. It will be integrated with the Catalog -- it will be possible to add a file(e.g. in OneDrive) to the Catalog in one click. Also, the Catalog will get the "Folder" item type which will open the specified folder (e.g. in SharePoint) in the File Browser.

Queries

Queries will be another built-on application in EasyMorph Desktop, similarly to the File Browser. It will be possible to open multiple database queries in tabs, run them, and switch between them. Queries will also have the Analysis View, so the query results can be additionally filtered and analyzed.

For easy collaboration, queries can be published to the Catalog, or opened from the Catalog in a single click. Queries are on the roadmap, but the release version has not been assigned to this feature yet.

Data classification

This is not on the roadmap yet, but an interesting direction we're thinking about. In the Catalog, dataset fields can be classified to designate sensitive information (such as home addresses, phone numbers, etc.). It will be possible to configure user roles (described above) to have access to only particular data classes. Data classification will be specified in Catalog item properties. Additionally, data classes can be applied to dataset columns dynamically, using a special action "Classify columns".

Collections

Somewhere later this year, we will introduce collections of Catalog items. Users will be able to create their own collections of items picked from different directories. For instance, a collection can be created to organize work over a business project or goal. A collection can include metrics, datasets, queries, files and folders related to the project or goal.

All in all, we are making the Catalog a convenient way to organize work with business data coming from different sources. It creates actionable collaborative virtual workspaces for users with different technical skills that can work with the data they need no matter where it's located.

I personally think that we're on track to introduce an interesting alternative (or addition) to traditional BI platforms that are high-ceremony, labor-intense, expensive, and suffering from low adoption.

How does our roadmap look to you? Please feel free to post comments and questions below.

3

I think you are addressing big company challenges, like data governance and security, that means a lot to us. But it seems you also keep on working on the “ease of use” of easymorph and that’s what I appreciate most, that is definitely the signature of the tool. It is really making the difference with other big tools where people can get lost very quickly, end-users and even IT guys.

I may suggest something more : there is still a topic about cloud connectors like sharepoint and Power BI. As big companies go more and more to the cloud, there should be enhancements about this and it’s quite simple : no security token saved in projects and valid only 90 days (request token on demand) and making easymorph integration in Azure easier (documentation to explain step by step how to allow Easymorph in an Azure tenant).

3 Likes
4

You're talking about using each user's account for cloud operations instead of sharing the one in the connector, correct?

I'm assuming it's quite standard: procure a VM with Windows and include it into the virtual network (VPC). What would you like to see in such documentation?

5

The first topic is still the same since a long time ago :

  • if you are on the desktop, today you authenticate once for 90 days with token inside the project. This is not a good thing or at least you should ask the user and warn him about consequences which is the impersonation when someone sends his project to someone else. It would be nice to have one token by run => nothing stored inside the project and no security breach. We can not use cloud connectors inside our company simply because of this. That should be a major concern.

  • if you are on the server you may want to manage things by private key without token stored. We did tests with one of your developpers and it was working (there is a topic in the forum about this). But I don’t think the feature is available right now in the official desktop ?

About the second topic, it’s more about allowing Easymorph in Azure. In big companies, admin may block several apps and authorize a few ones depending on security of each. By default they block. Consequence : if you want to use a microsoft connector like sharepoint or powerbi, you receive an error message saying that the app needs an approval by admins. So the thing is : how to approve easymorph in azure tenant (screenshot of azure portal) but also how companies can be more confident about easymorph in terms of security. We can do a meeting about this if you want.

1 Like
6

Hi @RJO ,

I would like to ask you for some clarification on a few points.

Firstly, when you mention ‘security token saved to the project’, could you elaborate on this further? Because, typically, cloud actions (such as Google or Azure actions) are used with data connectors stored in the Shared Repository, either local one or on a Server. In this case, there are no tokens saved in the .morph project, and the data connector is referenced only by name.

The only scenario where I can think of tokens being embedded in the project is if you use embedded connectors. If this is the case, may I ask why you chose this particular scheme over the Shared connector repository?

On a broader concept, it is true that every authorised connector through OAuth or other interactive authorisation methods always impersonates someone (the person who authorised the connector). But this is just the intent behind the OAuth concept. Whenever other authorisation methods, more suitable for automation integration, such as API tokens or ‘client credentials’ for Power BI, are available, we try to implement them too. Unfortunately, these schemes are not universally available as a general rule. Is there any option that we may have missed in this regard?

Lastly, I want to clarify the 90-day inactivity limit - is this about a particular connector? As far as I know, this policy is enforced only by some providers (such as Microsoft Azure), but not all of them. For example, Google refresh tokens with ‘offline’ scope live forever.

7

I think we should organize a call. Maybe an open call where everyone can join and participate in the discussion about the EasyMorph roadmap.

8

Let me clarify.

  1. I was in the case where someone stores the connector inside the project (embedded). Why I choose this case ? Because I have to think of ALL cases, security requires it. As soon as I can not force my users to not store connectors inside their projects, I have to imagine the consequences and the risks

  2. I did not know about this feature storing the token inside the shared repository. Can you explain a bit further ? Does this mean that everyone in the team can retreive it ? Or is it “hidden” inside the shared repo and only the user can see it ? An individual token must remain individual. Do you have documentation about this ?

  3. I think I read the limit in one of your posts : token is saved 90 days for example to connect to power bi. Edit : it’s here
    After, the batch will fail as it’s necessary to renew it. I guess it corresponds to the microsoft “stay connected” option. What I mean is that in batch mode, you don’t want this option. You will find here the thing we imagined together with service principal :
    Don’t know if it’s available but we still need it. That’s for server side.

Lastly, I must say Easymorph is still not allowed to connect to the cloud from our big company. We have to bring justifications and insurances about this. That was my last point. Available for a call. An open call is nice, it would be even greater to organize several during the year ! Great idea.