Documentation of an absolute failure - moving pictures tutorial

[Lacona]

Hey all,

I hope, someone can help me!

qiime2 latest version by miniconda on MacBook

After I finally succeeded in installing qiime2 I struggle with the very first steps now.
Following the moving-pictures-tutorial “Moving Pictures” tutorial — QIIME 2 2020.8.0 documentation

First I successfully ran:
mkdir qiime2-moving-pictures-tutorial
cd qiime2-moving-pictures-tutorial

Next I downloaded the sample-metadata.tsv, emp-single-end-sequences/barcodes.fastq.qz and emp-single-end-sequences/sequences.fastq.qz into my downloads and made sure the files were saved as asked for in the tutorial.

Then I ran mkdir emp-single-end-sequences

And then I completely failed to import the data with
qiime tools import \ --type EMPSingleEndSequences \ --input-path emp-single-end-sequences \ --output-path emp-single-end-sequences.qza

I tried shift+enter after every backslash and then enter, I tried enter after the whole thing, I tried to substitute --input-path emp-single-end-sequences \
with --input-path \users\myname\downloads\emp-single-end-sequences
and --input-path \users\myname\downloads\emp-single-end-sequences/sequences.fastq.qz.
Every single time I got the result "command not found".

For me as an absolute newbie to qiime and terminal these tutorials and instructions are more than frustrating.
I very hope that I can manage to install qiime studio and have a more positive experience with it.

Thank you for your understanding.

[Sreevatshan (Sreevatshan Srinivasan)]

Hey @Lacona,
first you should the activate the qiime2 conda environment,
using command conda activate qiime2-2020.8,
Then after activating the environment run the commands, it will work smoothly..

[Lacona]

Thank you!
I activated qiime2 condo environment.
I ran the code:
qiime tools import \ shift+enter
--type EMPSingleEndSequences \shift+enter
--input-path emp-single-end-sequences \ shift+enter
--output-path emp-single-end-sequences.qza enter

Failure notice: There was a problem importing emp-single-end-sequences:
Missing one or more files for EMPSingleEndDirFmt: 'sequences.fastq.gz'

Nothing worked smoothly ...

[Sreevatshan (Sreevatshan Srinivasan)]

Hey again, please follow these step,
Step 1: mkdir qiime2-moving-pictures-tutorial
cd qiime2-moving-pictures-tutorial
Step 2 : Download the metadata.tsv from the link given in tutorial
Step 3 : create an folder inside qiime2-moving-pictures-tutorial using mkdir emp-single-end-sequences.
Step 4 : Download the sequences.fastq.gz and barcodes.fastq.gz,save it in the emp-single-end-sequences
Step 5 : and then try to use the import command..

Hope it works..

[Lacona]

I already did step 1 and 2 and 3. Checked that all were done properly:

mkdir: emp-single-end-sequences: File exists

I have no idea about how to save the two datasets in the emp-single-end-sequences?
The tutorial said I have to download them, so I downloaded them into my downloads-folder.

I am so so so sorry, for all my silly questions. I can assure you, it feels pretty aweful to be so helpless.

[Sreevatshan (Sreevatshan Srinivasan)]

You have downloaded the files(barcodes.fastq.gz and sequences.fastq.gz) right.?, then copy that files & paste it inside emp-single-end-sequences folder and run the import command..

[ChrisKeefe]

No need to apologize for your questions, @Lacona. This stuff is hard for everyone at first!

I'd like to sum up some of the things we've learned so far in the great responses you've already gotten from @Sreevatshan.

1. If you install QIIME 2 in a conda environment, it's only possible to run QIIME 2 commands when that environment is active. Diagnose this with command not found errors. Fix it with conda activate your_environment_name_goes_here

2. Incorrect file paths are a very common source of errors. Diagnose this with errors like 'File not found', 'Missing one or more files', etc. Fix it by using a file browser (e.g. Finder), or command line tools like cd and ls, or find to locate your file, and either put it in the location you specified in the command, or change the "address" you specified in the command so that it points at the file you want.

3. When following tutorials, you may find it's easier to use the wget or curl download options. When run in sequence with the rest of the tutorial, they will download your files directly to the place you expect to use them. The screenshot below shows where to find those commands - depending on your system, you may have to install wget or curl before you use them.

Good luck with the rest of this tutorial! I promise the command-line gets easier with practice.

Chris

[Lacona]

Where can I find my emp-single-end-sequences folder?
I used the finder to find it, but there is no such folder.

[Lacona]

Thank you for your encouraging words.

I tried to find my emp-single-end-sequences folder.
I can't find it by using the finder.
I can't find it by using any of the other codes:

Thank you for giving me alternatives to the browser download. The problem is, that my research datasets are saved on my computer and are not available by download. So I have to learn how to import them directly from my laptop anyway.

I tried to run the import-code again and got a new failure message:

(1/1) Invalid value for '--input-path': Path 'emp-single-end-sequences' does
not exist.

After typing about 1000times emp-single-end-sequences - is there any possibility to rename that thing in something like EMP?!?
Is there any possibility to create the folder in my finder and give qiime access to it?

[Lacona]

I don't understand the import code anyway. The paths of the fastq.gz files are \Users\myname\Downloads\sequences.fastq.gz and barcodes.fastq.gz.
How does qiime know where to find them when the definition of the input path is:
--input-path emp-single-end-sequences \
I tried --input-path \Users\myname\Downloads\sequences.fastq.gz but it didn't work either.

[Lacona]

Yes, I downloaded both files into my downloads folder and then I renamed them, because the tutorial says "save as emp-single-end-sequences/sequences.fastq.gz (and barcodes.fastq.gz)", but there was no "save as" option. I was then asked if I really want to change the fastq into fastq.gz - yes.

[Lacona]

Update 1:

directory qiime2-moving-pictures-tutorial with file emp blablabla exists:

pwd /Users/lacona/qiime2-moving-pictures-tutorial/emp-single-end-sequences

It is not possible to do anything with file or directory, not even remove it!!!
I tried cd, rmdir, rm -r and every time I got No such file or directory
ls nothing happens at all.
ls ..gives me the emp-blabla file
ls -la plus the path for the directory/file, shows me, that the directory qiime2-moving-blabla contains the folder emp-blabla

So, obviously I am helplessly stuck in the emp-blabla file?!

I really need help here, please!
The situation is more than frustrating ...

Update 2:

I was able to move the folder moving-picture-tutorial into my Downloads and it is visible there.
Therefore I tried again to import sequences.fastq.gz, which is stored at /Users/lacona/Downloads/sequences.fastq.gz:

Update 3:

I deleted all existing folders manually in the finder and I rm -rf directory in qiime.
Result was that qiime shut down completely and I had to activate it again.

by mkdir /Users/lacona/Downloads/moving-pictures I opened a new folder.
cd moving-pictures is not possible No such file or directory but I can see the folder moving-pictures in the finder!!!
So, I tried cd /Users/lacona/Downloads/moving-pictures and it worked.
Next was mkdir emp and I could see a new emp folder in the finder.
Then I moved my sequences.fastq.gz in the moving-pictures folder and tried to import it into the emp folder file is uncompressed. Finder shows me a zipped file.
So. What now?

Update 4:

Now I deleted all the gzipped files in my Downloads folder and downloaded again the data by the link given in the tutorial -> barcodes.fastq and sequences.fastq
Then I used the terminal to gzip both datasets: gzip /Users/lacona/Downloads/sequences.fastqand same for barcodes.fastq.
I tried the qiime import tool with specificating the file I want to import
--input-path /Users/lacona/Downloads/sequences.fastq.gz NO!!! qiime want to have a folder = directory, it is obviously not able to import a single file!
--input-path /Users/lacona/Downloads is a directory, but NO!!! qiime tried to import every single file in the Downloads folder.
So, I moved the two gzipped files into the moving-pictures folder and ran it again:

I am so very sure, that this is again a failure, because now the two datasets barcodes and sequences are merged.

Finally I figured out how to change directory. I "only" had to type in the whole path for where I want to go: cd /Users/lacona/Downloads/moving-pictures

There are no words to describe my feelings about qiime or the terminal or this "beginner" moving-pictures-tutorial...

[Lacona]

The next step in the tutorial is demultiplexing the sequences.
The code given in the tutorial:

qiime demux em-single \
  --i-seqs emp/sequences.qza \
  --m-barcodes-file sample-metadata.tsv \
  --m-barcodes-column barcode-sequence \
  --o-per-sample-sequences demux.qza \
  --o-error-correction-details demux-details.qza

Didn't work, although I was in the moving-pictures folder and sequences.qza in the emp-subfolder.
(1/2) Missing option '--o-error-correction-details'. ("--output-dir" may also be used)
(2/2) Got unexpected extra arguments (o-error-correction-details demux-details.qza)

My solution was to change
--o-error-correction-details demux-details.qza in
--o-error-correction-details /Users/lacona/Downloads/moving-pictures/demux-details.qza
And it worked. demux.qza and demux-details.qza were both saved in my moving-pictures folder, although I didn't type in the whole path for demux.qza.
Is there anybody out there who can explain to me, why that is?

About the demux-details.qza the tutorial says "you can visualize these data using qiime metadata tabulate". I am not happy to say: no, you can't. I tried it.
qiime metadata tabulate demux-details.qza
(1/3) Missing option '--m-input-file'.
(2/3) Missing option '--o-visualization'. ("--output-dir" may also be used)
(3/3) Got unexpected extra argument (demux-details.qza)
Oh, ok. Interesting.

Next step in the tutorial is to visualize the data.

qiime demux summarize \
  --i-data demux.qza \
  --o-visualization demux.qzv

It worked!!! and made me so happy ... until I tried to see the results by
qiime tools view demux.qzv
Bad decision to run that code. Bad bad decision. A new window in Safari appeared with all the information about the dataset. I closed the tab and returned to the terminal only to face the fact, that something is wrong again:
Press the 'q' key, Control-C, or Control-D to quit. This view may no longer be accessible or work correctly after quitting.
Control-D and Control-C didn't work, so I had to quit the whole thing.
Note to myself: never try to see your results ever again ...

Dada2:
"Question
Based on the plots you see in demux.qzv, what values would you choose for --p-trunc-len and --p-trim-left in this case?" I don't dare to plot the dataset again, so no, thanks.
But I wonder how I ever will be able to work properly with my own data when I can't visualize it?!

Dada2 code:

  --i-demultiplexed-seqs demux.qza \
  --p-trim-left 0 \
  --p-trunc-len 120 \
  --o-representative-sequences rep-seqs-dada2.qza \
  --o-table table-dada2.qza \
  --o-denoising-stats stats-dada2.qza

New insight: o = output and i = input.
Question: what is p and len? My guess: p = please, len = Lenny. please trim 0 at the left, please truncate Lenny at 120.

qiime metadata tabulate \
  --m-input-file stats-dada2.qza \
  --o-visualization stats-dada2.qzv

Worked nicely. What means the m? My guess: m = my. My input file stats-dada2.qza.

Good news. I did the dada2 and deblur part without any difficulties.
I only would like to add, that instead of

mv rep-seqs-deblur.qza rep-seqs.qza
mv table-deblur.qza table.qza

It is possible to just go to the finder and rename the files.

Fortunately I will to all further analyses in R, so after I figured out, that my data is already in Casava 1.8 demultiplexed format, I will try the corresponding import data tutorial now.
Wish me luck!

[ChrisKeefe]

@Lacona, it looks like you've been working really hard to get through Moving Pictures. A lot of the things you're struggling with aren't actually parts of QIIME 2 - they are features of command-line interfaces that are hard for most people at first. There's not much QIIME 2 can do to change them, but I promise they'll get easier with practice.

You've answered many of your questions independently already. Nice work! For now, I'll share some high-level ideas and resources which I hope will help you better understand the environment you're working in.

1. A command line interface (e.g Terminal) is only another way to interact with your same computer. All of the files and folders you see in Finder are also available via Terminal, and vice versa. Here's a short intro to the command line. There are also lots of resources on youtube.

2. File paths are tricky at first. Here, you noticed that you could cd into a directory if you used the "absolute file path", but not a "relative file path".

@Lacona

So, I tried cd /Users/lacona/Downloads/moving-pictures and it worked.

Think of absolute paths as street addresses, and relative paths as directions from a location. It's often shorter to say "the tea shop is at the end of the block on the right." These directions are only true if you happen to be on the same block as the tea shop. If you're in the next town over, you'd need more complex relative directions, or a street address (e.g /Users/lacona/Downloads/moving-pictures) to get there.

Applying that, if you are in the /Users/lacona/Downloads directory and there is a subdirectory called moving-pictures, cd moving-pictures will work. If you're in /Users/lacona/, though (and not in Downloads), cd moving-pictures won't work. Why? Because the computer can only "see" the things in your present working directory, and the moving-pictures directory is inside of the Downloads subfolder. You could cd Downloads/moving-pictures - that would work just fine. If this explanation doesn't help, google "absolute vs. relative file paths linux". Mac and linux are pretty similar at the command line, and you'll find lots of other explanations.

TLDR: you can use an absolute file path anywhere. Relative paths only work when the computer can "see" the first thing in the relative path from the directory you're in. Use ls to tell you what files/directories the computer can see from where you are.

3. Leave poor Lenny alone. These are single-end sequences, so he's short enough already.

@Lacona

New insight: o = output and i = input.
Question: what is p and len? My guess: p = please, len = Lenny. please trim 0 at the left, please truncate Lenny at 120.

Good eye! These single-letter annotations can be really useful if you know what they're saying.

• i = input (specifically, an input here is a QIIME 2 Artifact: a .qza created by QIIME 2)
• p = parameter (e.g. an integer, string, etc. QIIME 2 uses to run a command how you want it run)
• m = metadata (This is data about your data - often in a tabular form, like my-sample-metadata.tsv. Some commands allow specific types of .qza to be used as metadata, too!)
• o = outputs. (Artifacts or Visualizations created by QIIME 2).

@Lacona

It is possible to just go to the finder and rename the files.

This is totally true. Remember, Finder and Terminal are just two different ways of using the same computer. I used to be much happier with Graphical interfaces. After a lot of practice, I'm much happier with command-line interfaces. They take some getting used to, but for many of the things I do, they're more ergonomic. Use whatever tools work best for you. QIIME 2 doesn't care how you do what you do. It is "interface agnostic", and mostly just cares that you give it clear directions and the right type of data for a given command.

You're clearly not afraid to try new things, and I suspect you'll get the hang of the CLI pretty quickly if you take the time to learn. If you can run QIIME 2 in a VirtualBox, there is a prototype graphical interface for QIIME 2 called Q2Studio. If I were you, though, I'd probably focus on the CLI for now - it's really powerful, and having CLI skills will help you with other, less user-friendly bioinformatics tools in the future.

Keep at it - you got this!
Chris

[Lacona]

Thank you soso much ChrisKeefe!
Your explanations are great.

Why all this information can't be found in some completely-new-to-everything tutorial? It would have been so helpful and time saving ...

I like the idea of p = please better than p = parameter

[ChrisKeefe]

Glad to hear it, @Lacona!

@Lacona

Why all this information can’t be found in some completely-new-to-everything tutorial?

Because the QIIME 2 team is small and very busy, we have to focus our documentation and support on the things that we build ourselves. "How to use a command-line interface" is just beyond the scope of what we're able to build and maintain right now.

This forum has some great tutorial topics, created by QIIME 2 users for other QIIME 2 users. As you get better at working with the command line, consider creating one that answers the the questions you've struggled to answer. I'm sure others would appreciate your work! In the meantime, keep working hard, reading the documentation, and searching this forum, and you're sure to figure it out.

All the best,
Chris