data import using Amazon aws

I am new to Qiime 2 platform. I am using Amazon Web Services.
image

I was trying to import my data. My FASTQ file and metadata is in my local computer as shown below (Under OneDrive -Personal folder under R).
image

First, I want to know how to import my data to the directory that I created above, and I also want to know how to create URL in order to follow below commands.

wget
-O "emp-single-end-sequences/barcodes.fastq.gz"
"https://data.qiime2.org/2022.2/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz"

Thank you!

Hello Choa,

There are lots of ways to move files between computers. Here's what I recommend

Get files from your local computer into a linux machine:

I like using the Remote Explorer in VS Code. This tutorials show how to set up VS Code Remote Explorer for AWS. This works great on AWS, and Azure, and local supercomputers at your university!

I also like VS Code as text editor, and it's free, get it here :point_left:

Once you get VS Code set up and Remote Explorer connected to your AWS VM, you can drag and drop you files from your desktop to your remote machine.


Get files from the internet into a linux machine:

That wget command lets you download a file from a URL. (Like, if you click on https://data.qiime2.org/2022.2/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz it will start downloading that file to your local computer.)

To download that same file to your remote unix computer, use wget.

(This only works for files already on the internet, like those in the Qiime2 tutorials. For files on your local computer, try the first option)

2 Likes

I failed to use AWS, so I started to follow the instructions for virtual box.

Now I faced different problem here. Please see below image and help me!

I am using Windows 10 64-bit operating system, x64-based processor. Thank you!

Good morning,

From inside VS Code, type >remote and add a new SSH Host with your username and IP address of your AWS machine. So in your case, this would be root and 172.31.1.88


For your VM error, try the steps listed in this troubleshooting guide:

1 Like

Thank you for your response! I tried the steps for VirtualBox, but for me, the button they require me to click was inactivated on my end. Please see below image and help me! Thank you

Did you try the next step in the tutorial?

2. Then try to start the virtual machine. If the problem persists, continue to the next method below.

Method 2. Uninstall and Re-Install VirtualBox.

Also make sure you have the newest version of VirtualBox installed, if you haven't checked recently.

Yes, I uninstalled and reinstalled the virtualbox. Then, I imported the qiime2 OVF, but still having the same problem. I also removed the qiime2 in the VM and reimported the qiime2, but having the same error message. What can I do? Please help me with this

Hello again Choa,

I'm at a bit of a loss here, as I've never seen this error when using Virtualbox. I found that troubleshooting tutorial using this google search. There may be better results on that list.

What do you think we should try next to diagnose this problem? Have you been able to get any other VMs working with Virtualbox? Would you like to pivot back to using AWS and VS Code to move files there?

Thank you so much for your assistance.

I pivoted to VS code and tried to search "remote" in VS code. I am following this link ( Developing on Remote Machines using SSH and Visual Studio Code to connect.

I typed ssh [email protected] on my PowerShell and Ubuntu. It says "ssh:connect to host 172.31.1.88 port 22: Connection timed out."

On my AWS account, I can see below image. I am not sure what to do from now on.

Please help me with this. Thank you!!

1 Like

Hi Colin,

Do you think I can use Docker? VS code is still not working :frowning: Please help me with this

1 Like

Worth a shot! Check out the Qiime2 docker install docs and this tutorial. :whale:


The Remote Explorer uses SSH, so try the SSH client tab shown on that Amazon AWS screenshot.

To start, let's see if we can get a basic SSH session running between your computer and your EC2 instance.

That's the right command! See if that SSH client tab gives other advice about connecting to your instance.


Thanks for sticking with it while we figure this out. I apologize for the scatter-shot approach. There's a bunch of ways to set up this software and connect to remote machines.

You mentioned that you were new to Qiime2, but also you are using AWS and Docker which is well beyond many traditional microbiologists. What kinds of unix software have you used before? Maybe we can use that as a starting point for qiime2.

1 Like

I have never used any types of software before. That is why I am struggling a lot. I think I completely messed up with my computer because it doesn't allow me to open docker or uninstall it. I previously downloaded the docker and I uninstalled and deleted the program in the (C:) folder, but it now seems that some folders are still there because it did not allow me to delete it. Can you help me with this..?

Good morning Choa,

Thanks for telling me more. Let's see if we can get Qiime2 installed using Windows Subsystem for Linux, as that's probably the easiest way forward.

(Before we start, you can restart your computer and try uninstalling both docker and VirtualBox, and deleting the VirtualBox image to save some space.)

First, install Windows Subsystem for as described here. Then install Ubuntu 20.04 LTS. Then install Qiime2 using the WSL commands.

As you work your way through this process, let us know if you have any questions.

Hi Colin! Thank you for your response.

I was able to install Qiime 2 using Ubuntu (I think it worked? :slight_smile: )! Thank you so much for your help.

Can I add my question from here? I have taxonomic data as shown below using 16S sequencing.

Also I have metadata in excel sheet (separated into intervention and control group in each sheet) as shown below.

As you can see it is formatted with excel sheet, and I know I have to convert excel to biom format. Can you help me how to create biom format? I have tried biom convert from other forum questions, but it did not work. Can you walk me through the process in detail as I am savvy to these software or programming?

Looks like it! You can run qiime --help to double check that it's installed correctly.

We like to keep separate questions in seperate threads. Want to open a new thread and I can help over there?

Thanks!