Connector documentation : going further with metadata


Today in Easymorph you can provide a URL on a connector to get help for example and this URL appears in annotation part in actions.

I think we all need a kind of metadata dictionnary. This is a major need that is not implemented by common ETL tools. For example in Easymorph, users can see tables and views, fields inside but sometimes they would need explanations to know more about what they query.

First simple evolution that can be done fast : display connector URL in the query editor.

Second evolution more complex but why not : use the comment feature of databases. For example in oracle, you can comment tables and views fields. So why not display those comments in query editor ? Like tooltip or new information in the left pane, or by right click on it. For comments on tables, you could display them in the window of table selection. Or you provide links on “table” and each field, so that when you click on it, comments appear.

There would be another possibility to provide a kind of dictionnary that would be matched to tables and columns but maybe harder to deal with.

I think comment feature is native in Oracle, SQL Server (through description), PostgreSQL and MySQL. It’s not native in SQLite. There’s a general comment feature adoption so that’s a pity easymorph does not use them. Let’s make easymorph one of the first tools to embed a data dictionnary :slight_smile: !!

That’s a good suggestion, we’ll have it done.

Overall, we’re slowly moving towards better metadata. Even now field annotations from databases can be see when data is imported from a database. Check out this icon (below):


The data catalog we’re planning to release this year will be a better way to deal with metadata. Metadata will be searchable, editable. However, the data catalog won’t work with the Query Editor, at least not initially. Metadata in Query Editor is something that we’ll have to think about.

That’s quite annoying because query editor (for databases) is the first step for a user. He can select fields, filter, aggregate. If he does not know about the data, that’s really uncomfortable for him, he will select all the fields, less filter and less agregate and will tend to do it after, using a lot of memory with all the inconvenients that it can have.

Despite that, metadata arrival is very good news !! I’m sure looking at comments metadata and display them in the query editor would be a quick win.