About EasyMorph Tutorials & Examples Web-help

Data catalog features

Hi,

I read the page https://www.datacatalog.com/ about the data catalog.
Very interesting !

  • Am I right that the data catalog will also act as a data portal through which users/machines can upload/download data files ?
  • Is there a demo page or video where we can see how this will look like ?
  • Will it be possible to show the descriptions of fields/tables from databases in the catalog ? Currently we add field descriptions in our postgres databases.
  • Can the data catalog be hosted on premise ?
  • How will access rights be controlled ? Will there be an option to control access using openID connect ? We have a central access and identity management system, so we would like to onboard the data catalog onto that system.

Thanks
Nikolaas

Technically, the Data Catalog is a feature of EasyMorph Server which is installed on-premises. Depending on customers’ needs, it may obtain later a public-facing read-only internet portal provided through the EasyMorph Gateway, but not in early versions.

Not yet. But it would be great to arrange a call with EasyMorph customers and discuss UI sketches. We’ll try to arrange it in the next month or so.

The descriptions of fields will be prepopulated from column metadata of the project’s result table. Therefore it will be trivially easy to show the descriptions of database fields in the Data Catalog. Currently, it’s not clear whether it will be possible to use database table descriptions. Probably, no. But any description can always be entered and edited manually.

We’ll also provide a way to update descriptions programmatically with a special action. Although, that may come in later versions, not initially.

Access to the Data Catalog will be managed through Server spaces just like it’s done currently. A catalog is basically a part of Server space, just like Tasks or Files. We’re open to the idea of adding other authentication providers (such as Open ID or SAML) for the Server besides Active Directory, but that’s a different issue.

Hi Dmitry,

Thanks for the info !

  • Could you also answer the question about data portal functionality of the data catalog ? We are looking for an application through which internal and external users can upload files to EasyMorph server in a secure way i.e. only having access to the folders they may see (using our authentication provider service). Or can we use the current upload manager in that way and is there an easy way how machines kan upload files to folders EasyMorph server uses in ETL-flows? All info about simple solutions that can be put in place rapidly are welcome.

  • Do I understand it correctly that we could update the descriptions in the catalog in the automated way by doing some sort of query on our database to pull all descriptions from it and then through EasyMorph load it into the data catalog ?

Thanks
Nikolaas

No, the data catalog is not a cloud-based file collection/sharing service. It doesn’t operate with files. It provides datasets that are produced dynamically, on the fly.

If you need a simple way to collect files from internal and external users, why not just let them upload the files into a cloud storage folder in Google Drive or Amazon S3 or OneDrive (will be supported in v5.0.1) and then collect them with a scheduled task in EasyMorph Server?

I guess it’s better to discuss it in a separate topic.

Ok thanks. Unfortunately for privacy reasons we cannot use the cloud.

Well, then you may try to have a separate EasyMorph Server installation in DMZ for file uploading/downloading. EasyMorph Server uses the HTTP.SYS web-server on which IIS is based so it’s rather robust security-wise.

To learn more about EasyMorph visit easymorph.com.