I've been trying to get EasyMorph to read input files from a private S3 bucket without success. I've tried configuring a custom connector and can't seem to connect. Any tips or tricks to try? Any pre-requisites on the S3 bucket in question to allow for the connection? thanks in advance
Hi, @doon, and welcome to the Community!
Do you have AWS IAM credentials that give you access to that private bucket?
If so, which error are you getting when you are using those credentials with the Amazon S3 connector in EasyMorph?
i do have IAM credentials to access the bucket. I can view it no problem from within the AWS S3 management console.
It seems that I am getting a generic wrong credentials error when trying to test the connection within add/edit connectors. I populated Access Key ID, and Secret Access Keys in the configure connector screen.
You should create an access key in the IAM console and then use it in the Amazon S3 connector.
Here is a guide on how to create an access key from the IAM console:
If you don't have access to the IAM console, please ask your account admin to create an access key for you.
this is helpful, thank you. My account admin is asking for further detail related to a least privilege model - S3:* is not a viable option.
Is there anywhere documented what specific S3 permissions are needed? for example:
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:GetBucketAcl",
"s3:GetObject",
"s3:GetObjectTagging",
"s3:PutObject",
"s3:PutObjectTagging",
"s3:PutObjectAcl",
"s3:DeleteObject",
"s3:CreateBucket",
"s3:PutBucketPublicAccessBlock",
"s3:PutBucketOwnershipControls"
],
"Resource": [
"arn:aws:s3:::bucketname*",
"arn:aws:s3:::bucketname*"
],
"Effect": "Allow"
Required permissions depend on the commands you are planning to use. Here is the list of the required permissions for specific actions:
s3:ListAllMyBuckets - list all the available buckets;
s3:ListBucket - list file, list folders, download folder;
s3:GetObject - download file, download folder;
s3:PutObject - upload file, upload folder, create folder;
s3:DeleteObject - delete file, delete folder;
s3:GetBucketLocation - for the bucket region autodetection to work (as an alternative, you can specify the region manually in the connector settings);
s3:AbortMultipartUpload and s3:ListBucketMultipartUploads - for the "Delete incomplete multipart uploads feature" to work.
Andrew, i've got my access key created and successfully tested the connection in the connector manager. When trying the Amazon S3 command function, I can browse my folder structure in S3 indicating the connection appears to work. However, when trying to do a simple "list files" or "list folders" or any of the other options, I'm met with a "Bucket not specified" error.
what am i missing?
Kevin, you have to specify a bucket for the action to work with.
You can specify a default bucket in the connector settings or switch the "Bucket" option in the action settings to the "Other" mode and select a bucket there.
Hi Andrew, I'm sorry for asking so many questions, but i'm just not getting it and it's frustrating me. I've specified the default bucket in my connector. Testing the connection is successful:
Now in my EasyMorph workflow, i've got one action to download a file from S3. I'm using the connector i successfully tested and created. In File path, i can use the folder icon and browse through the folders in question and see my files. i specified to download to my local C drive.
after applying and performing a run all actions, i get an error:
I feel like i'm doing something dumb. What am I doing wrong here?
thank you
Hi Kevin,
This looks like a bug on our side. Sorry for the trouble. We will look into it and fix it.
For now, you can switch the bucket selection to the "Other" mode and select the same bucket there.
That option seems to be working. If the "Apply" button won't get activated, just change some other field value, click "Apply" and change it back.
Hi Kevin,
The bug has been fixed. You can get the updated version from our download page.
1 Like