AWS - How to reference S3 bucket from EC2 qiime2 instance

[microbiomeAnalyst]

Hello, I have installed a qiime2 AMI and also created an S3 bucket to house the metadata and sequences. I am able to SSH into my EC2 instance using my local linux VM.

How do I reference my sequences and metadata within my S3 bucket from my EC2 instance? I have also created an IAM role (profile) that “authorizes” my EC2 instance to use the S3 bucket.

Can I reference the dataset in my S3 bucket as s3://bucketname while using qiime2 commands from my EC2 instance? How?

I’m really stuck.
Thanks!

[thermokarst]

Hey there @microbiomeAnalyst! This is not really a QIIME 2 question, so I moved this to a new category on the forum.

@microbiomeAnalyst

Can I reference the dataset in my S3 bucket as s3://bucketname while using qiime2 commands from my EC2 instance?

I think you could look into some FUSE filesystem to transparently mount a bucket as a filesystem, but honestly that sounds like more trouble than its worth. How about you just use awscli and sync the bucket contents locally on your EC2 instance?

[John_Chase]

Hello,

I am with @thermokarst I think the way to go is to set up the AWS command line tools. Unless your data is public you will have to setup credentials but then you can use commands like aws s3 sync s3://bucketname/path/to/files ~/local/directory to move files from the s3 bucket to your ec2 instance

[microbiomeAnalyst]

Thank you so very much for your reply!!

I was concerned about any s3 sync with my EC2 qiime2 instance since the qiime2 installation instructions are to install without a keypair, so that when I ssh qiime2@PUBLIC_AWS_IP_ADDRESS I’m using a public IP.

If I setup credentials does the EC2-S3 sync retain your data securely within the S3 buckets?

I may have a few additional questions over the course of the next few days!

[John_Chase]

I would recommend creating a key pair once the instance has been created and disabling passwords. I have to be honest security is not my expertise, so I would probably want to consult another source if your data must be confidential.

[thermokarst]

I totally agree with @John_Chase. BTW, the QIIME 2 EC2 image doesn’t really provide anything over just manually installing QIIME 2 yourself (its just provided as a quickstart convenience for users), so if you have a preferred EC2 image and setup, maybe it is easiest to just follow the native install instructions on that instance and then totally skip our pre-built EC2.