Import files from Amazon S3

1

Is it possible to connect an AWS S3 bucket to the connectors?

2

Not possible at this point. Any operation with AWS should be done using the AWS command-line tool called using the “Run program” action in EasyMorph.

****moved to #uncategorized

3

Here is an example project that downloads and imports multiple files from S3.

DownloadFromS3.morph (7.1 KB)

It will require a bit of configuration before use:

  1. Download and install the AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html

  2. Under the same user account that is used for EasyMorph, configure the CLI as described here: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration. Use “Default output format” = text

  3. Configure the EasyMorph project. It has 3 parameters:

  • S3BucketName - well, S3 bucket name.
  • Prefix – it’s a folder in the bucket (optional). Leave it empty if you want to download all files from the bucket.
  • Target folder – a temporary folder that will be used for downloading files from S3 and loading into EasyMorph. Note that all files in this folder will be deleted before downloading, so it’s better to use a temporary folder.

When you have the project configured – run it. You can run it step by step using the “Run action button”, or run entire project by pressing “Run project”. The last step in the project is loading files. You will need to select files you need to load once all previous actions are calculated.

Еhe list of files obtained from S3 can be additionally filtered using one of the filtering actions available in EasyMorph. Just insert such action at the end of the table called “List S3 files”.

4

Hi Dmitry,

Is there an alternative solution ? For instance: is it possible to map S3 buckets (with all its subfolders) as a networkdrive on the server where EasyMorph server and EasyMorph desktop is installed ? Then if these buckets are mapped, is it possible to access the files in the folders with regular import transformations like read CSV, etc ?

Kind regards

5

Hi,

It seems that there should be some way to map S3 bucked as a network drive through AWS Storage Gateway.

Also, there are several third-party tools which support that feature.

Regards,
Andrew

6

Hi Andrew,

We have contact our AWS consultants and they say that mounting S3 is not always working well in practice.

The question is: can EasyMorph connect to the files in S3 when we map is as network drive ? I suppose the answer is yes ? Will it work like accessing the files on a regular network drive ?

Apparently there’s also another file system service on amazon: https://aws.amazon.com/efs/
I suppose that putting data in that kind of store would enable EasyMorph to connect to the data as well?

Kind regards !

7

For both S3 and EFS it depends on how closely mapped storage resembles regular network drive. If you’ll be able to read files from such drive than EasyMorph should be able to import them too. The same goes for write and EasyMorph’s export.

8

Hi There. Thanks for the information regarding importing files from S3. Are you also able to export files to S3?

9

Hi @camprice01 and welcome to the Community,

You can download and upload files from\to S3 with the “Amazon command” action.

1 Like
10

@andrew.rybka Thank you so much. It is really appreciated.