Is Data Catalog right fit for us?

1

Hi @dgudkov ,

Hope you are well.

I want to understand how Data Catalog will work in practice for us. Perhaps you might have answered this question elsewhere…

We currently use a software that has inadequate reporting for our needs. There are large volumes of data that are generated by this software and the only way to create custom reports is to export the reports out as csv files.

Being a small business, we don’t have massive databases or a huge team of developers to go and create custom solutions for us. I am contemplating using something like Google Data Studio because it is free and feed the data into there.

From what I understand data catalog is not a database but structured around providing the right dset files to people who are authorised to access them.

If so, I am guessing the workflow would need to be:

  1. Get data from external system
  2. Clean it up and load it to data catalog
  3. Export from data catalog to a source that can be read by a tool such as Data Studio

Google can only read flat files with a size of up to 100mb. Therefore, given your experience, would you say that I would need to have some sort of database eventually holding this information so that we can report on it?

I feel like adding a database is adding a layer of complexity that I am not sure I want…

Regards,

Shaheed

2

Yes, that's correct. But it's designed to provide data to people. It's not possible provide data from the Catalog to Google Data Studio. From what I understand you can feed Data Studio with CSV files stored in Google Drive. So you don't need the Catalog for that. Instead, you can use EasyMorph to generate a bunch of CSV files, upload them to Google Drive and point your Data Studio reports to the csv files on the Drive.

Alternatively, if the reports are not too complex, you can use the Catalog as a reporting solution. Soon, we will be adding Analysis View to the Dataset Viewer. Therefore, when you retrieve a dataset from the Catalog, it will come already with the Analysis View which may be sufficient for your reporting needs and eliminate the need for a separate reporting tool like Data Studio.