Newbie Question: How to set up Cyberducky with QIIME2 in AWS

Hi there! I'm new to using QIIME2 and I am trying to run my analyses in the cloud.

I installed qiime2 via EC2 AWS following the official qiime2 website instructions (Installing QIIME 2 using Amazon Web Services — QIIME 2 2018.6.0 documentation). I don't know how to use command line, so I was hoping to use Cyberducky to get files into and out of my instance.

I just can't seem to connect via SFTP. This is what I've tried inputting into the various fields:
Server: Public DNS or Public IP
Username: ubuntu, ec2-user, AWS account username
Password: qiime2
I've been selecting "None" for SSH Private Key because there was no key created from setting up this instance
I haven't put anything in under "More Options"

If anyone could help me troubleshoot here, I would really appreciate it. Thanks!

Hey there @jnie93!

I haven’t tested this, but it should work, since this works over SSH. Anyway, the usernames you listed above - none of those are the username that the images ship with by default: qiime2 (this is specified in the doc you linked to above). Have you tried that, with the password qiime2 (don’t forget to change that password).

:+1:

Keep us posted!

1 Like

This worked perfectly!!! Thank you!!

1 Like

Thanks for the help earlier! I have another newbie question....

I've been able to connect via SFTP, but every time I try to upload a file, it will stop before fully finishing. Below is a screenshot of my attempts to upload to /home/qiime2 and to a directory I made (qiime2-moving-pictures-tutorial). It stopped at 205 Mb in /home/qiime2 and won't even continue in the directory I made.

Am I uploading to the wrong place? Or is there a setting I need to change? I'm thinking it may be a storage issue, but I'm using m1.large on AWS, so I should have plenty of storage. I've tried in both Cyberducky and Firezilla, but I'm getting the same issue.

Ultimately, I want to understand how I actually go about uploading files to AWS and running QIIME2 analyses on them. Is there anything you can point me to? Thanks!

Just to give some further clarification:

I’ve tried uploading using wget -o and the download also stops at ~205 Mb. It seems like there’s something about the EC2 instance where I can’t upload more than ~205 Mb to it.

I’ve been able to run the QIIME2 tutorials without a hitch on EC2 instance, including downloading files, but those are all smaller than 200 Mb.

I’ve tried different AWS configurations as well as using different different versions of QIIME2 from the community AMIs, but I still can’t get this to work.

Any suggestions? Thanks!

Hey there @jnie93!

Makes sense - you are probably not working in the attached volume. I just spun up an m1.large, check this out:

13%20AM

It looks like my root filesystem has ~162MB free (this is where QIIME 2 is installed).

If you look down near the bottom, EC2 automatically attached a ~400 GB volume for me at /mnt --- if I move my analysis files there and continue to work from that location there should be plenty of space. This also has the advantage of allowing the same volume to be attached to different instances! So, if you need something bigger during feature classification, just attach this volume to a bigger node.

If you have more questions, I would suggest looking at the AWS/EC2 docs, since this isn't QIIME 2-specific, but, if you get stuck you know where to find us! :tophat: :t_rex:

1 Like

Hi thermokarst,

Sorry to keep bugging you, but I think this is the last thing holding me back from being able to use QIIME2 in the cloud. I’ve been going through the AWS documentation for hours now trying to get this to work and I just can’t.

How do I go about moving my local files to /mnt or someplace that will accept them? When I try to use Cyberduck to upload anything, it says “Permission Denied” to any folder other than /home/qiime2 (which has only ~200MB of space).

I tried making a new directory in /mnt and I got the same thing. I also tried creating a volume, mounting that under /mnt, and then uploading something to it. Permission was still denied.

I just want to be able to put files up so that I can use them for analysis. I would really appreciate any pointers you could give me here. Thanks!

Ah - figured it out! The instructions on qiime documentation (http://qiime.org/tutorials/working_with_aws.html) still apply from " Creating a volume for persistent storage across different launches of an instance (or different instances)" onwards. The only changes in the code you have to make are the following:

sudo mkfs.ext4 /dev/xvdf

sudo mount /dev/xvdf $HOME/data
sudo chown qiime2 $HOME/data
sudo chgrp qiime2 $HOME/data

2 Likes

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.