RE: Deblur Plugin Set-up

Hello @jwdebelius !

It has been some time but I've finally had the chance to look at the project I'd been planning on doing, which you previously advised me on.

I now have the raw multiplexed and mapping files (see below), and I've tried to run the data through deblur by following the steps outline in the Moving Pictures Tutorial—however, I've come across a few issues.

First, when I try to import, this is the issue I am getting:

(qiime2) [email protected] qiime2-moving-pictures-tutorial % qiime tools import \
--type EMPSingleEndSequences \
--input-path trail1stock \
--output-path trail1stock-seq.qza

There was a problem importing trail1stock:

  Missing one or more files for EMPSingleEndDirFmt: 'sequences.fastq.gz'

Do you know how I can instruct EMP to accept my data file?

Also, when I am to demultiplex my data with demux , how do I have this command read my files properly. My sequences are in .fastq format and my mapping file is just a .txt and looks like this:

Mapping File:

Sequences: .fastq format

I really appreciate your insight!
Taylor xx

Hi @Taylorr,

I made this public partially because I've been super busy recently and not on the forum as much; I apologize.

It looks like the error is that your sequences need to be named sequences.fastq.gz and they might be called sequences.fastq. Yes, it's a weird little nuaunce. And yes, I think we've all done it 1000 times.

Best,
Justine

@Taylorr,

Unfortunately, our general tutorials can't each provide information about importing all kinds of data. The EMP data imports is really just a "convenience method" that makes bringing a particular kind of data(that is, data that has been collected, sequenced, and processed so far in accordance with the Earth Microbiome Protocol) into QIIME 2. However if your data in anyway does not fulfill the expectations baked into the EMP import, using this method will not end up being more convenient :flushed:

All convenience importing methods are just pre-parameterized calls to a "manifest import", which you can do directly yourself! See the documentation on how to do so here! The manifest imports let you tell QIIME 2 exactly where your files are and which file represents which piece of data.

If your data was sequenced even remotely recently, the --input-format you will want to import as is very likely to be PairedEndFastqManifestPhred33V2. Note that you need to be very precise in specifying file locations in your manifest document that you prepare, though you can use environment variables such as $HOME to indicate where your files are.

The good news is that at least in regards to using the QIIME 2 tools, importing your data is often the most difficult part of the process!

2 Likes

Thank you both @jwdebelius and @Keegan-Evans for your kind support! And apologies for not getting back to you sooner.

I have gone ahead and attempted to import using the manifest import option as suggested. Here is what my code looks like:

(qiime2) [email protected] qiime2-moving-pictures-tutorial % qiime tools import \
     --type 'SampleData[SequencesWithQuality]' \
     --input-path se-33-manifest \
     --output-path single-end-demux.qza \
     --input-format SingleEndFastqManifestPhred33V2

And this is the error:

**There was a problem importing se-33-manifest:**

**se-33-manifest is not a file.**

Currently, I have all my .fastq files in a folder called 'se-33-manifest'

Do either of you know what might be the issue here and how to resolve it?
I understand that my folder se-33-manifest is evidently not a file; but I will need to somehow specify an input path that accepts all my .fastq files.

Any ideas or help is appreciated!

Taylor :wink:

Hi @Taylorr,

You're on the right track! But the error message you are getting is exactly correct (as you mentioned) - se-33-manifest is not a file, it's a directory. You'll want to create an actual file that contains the filepath information for all of your .fastq files - something that looks like this:

sample-id     forward-absolute-filepath               reverse-absolute-filepath
sample-1      $PWD/some/filepath/sample0_R1.fastq.gz  $PWD/some/filepath/sample1_R2.fastq.gz
sample-2      $PWD/some/filepath/sample2_R1.fastq.gz  $PWD/some/filepath/sample2_R2.fastq.gz
sample-3      $PWD/some/filepath/sample3_R1.fastq.gz  $PWD/some/filepath/sample3_R2.fastq.gz
sample-4      $PWD/some/filepath/sample4_R1.fastq.gz  $PWD/some/filepath/sample4_R2.fastq.gz

More details on how to format your manifest file can be found here, for your reference. Hope this helps!

Cheers :lizard:

2 Likes

Hello @lizgehret!

Okay, so I've cone ahead and made my manifest file. It looks like this:

However, I am now getting stuck with the following error:

I am not sure what the issue might be...it seems like I have the correct file path.. when I look at the file directly it shows that this is correct /Users/taylorrayne/qiime2-moving-pictures-tutorial/demultiplexed_data

Do you see where I might be tripped up?

Thanks in advance!

Taylor

$PWD will give the path to the location from which you are calling for this path, try using $HOME instead, if you are going to give an absolute path like that, if you wanted to keep it relative you could instead do something like $PWD/demultiplexed_data/blah...blah.fastq, if you were calling your import from inside the directory containing your demultiplexed folder and your manifest file.

1 Like

Hello @Keegan-Evans ! Thanks for following up with this suggestion. I went ahead and tried running the command with $HOME instead but still got the same issue. I'll include details below; let me know if you see anything immediately wrong or not what you were suggesting in what I have done!

Error:

Manifest File:

Directory framework:

Hi @Taylorr,

Hopefully @Keegan-Evans doesn't mind if I answer!

The path you're using is redundant. $HOME is the same location as /Users/taylorrpaye/. So, you either need to use $HOME or /Users/taylorrpaye/ at the begining, not both. On my mac, I find better performance if I dont use local variables, like $HOME and instead stick with the full path (In your case /Users/taylorrpaye/...

As a PS, do you know that if you drag a file into the command line on a mac, it will show you the path? So, if you're not sure how the computer knows the address, open up a terminal, and drop it in! It's saved me a few times when I was lost.

Best,
Justine

2 Likes

Hello @jwdebelius - thank you for this suggestion with the drop-and-drag trick - super helpful!
I have now resolved the path issue by using /Users ... instead.
However, I am now finding an issue with the files themselves, and I am wondering if I can get your insights on .fastq files.

This is the issue I am getting:

And here is a view of my files. As far as I can see, they conform to the .fastq formatting, so it is not clear what the issue might be...let me know if anything is obvious to you!

Thanks <3