Importing in qiime2 a fastq file already joined trimeed and barcode sorted

[Hajar]

Hi,
I am new to qiime2,
I recently received the demultiplexed reads from a Hiseq illumina company that looks like: AOAM-1-sample1_trimmedPR_join.fastq.gz per sample. I have so far 210 files like that.
I used this command for import in my HPC cluster:
qiime tools import
–type SampleData[JoinedSequencesWithQuality]
–input-path /mnt/beegfs/scratch/eh309/fasteries/sample_testing
–output-path fj-joined-demux.qza.
I got the following Error:
There was a problem importing /mnt/beegfs/scratch/eh309/fasteries/sample_testing:
Missing one or more files for SingleLanePerSampleSingleEndFastqDirFmt: ‘.+_.+_L[0-9][0-9][0-9]_R[12]_001\.fastq\.gz’

My question is that do I need to prepare Manifest file or not?
Is there a way to use a script to run the commands with SLURM job scheduler?

Thanks a lot for your help and support.

[thermokarst]

Hey there @Hajar!

@Hajar

I am new to qiime2,

Welcome!

@Hajar

My question is that do I need to prepare Manifest file or not?

Yes. Otherwise, you could rename your files to match the spec, but that would be a lot more work, IMO.

@Hajar

Is there a way to use a script to run the commands with SLURM job scheduler?

Yep, the same way you submit any other job or task to your scheduler, nothing special about QIIME 2 here. Check with your sysadmin for details.

[Hajar]

Thanks for your reply,
Actually, as you mentioned,I prepared the manifest file but did not work:
Below the command line used with the error got:
(/mnt/beegfs/home/eh309/qiime2) [eh309@gm-hpc-login-01 HWL-trimmed]$ qiime tools import \

--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path /mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed/
--output-path paired-end-demux.qza
--source-format PairedEndFastqManifestPhred64
There was a problem importing /mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed/:

/mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed is not a file.

(/mnt/beegfs/home/eh309/qiime2) [eh309@gm-hpc-login-01 HWL-trimmed]$

any help is highly appreciated.
Please find the attached manifest used:
manifest.csv (268 Bytes)

I tried only with one sample if it works I will do the other 210 samples.
I spend three weeks only on importing this :(.

Thanks a lot.
Hajar

[thermokarst]

Hey @Hajar!

@Hajar

I spend three weeks only on importing this :(.

Yikes! We are always here to help...

@Hajar

/mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed is not a file.

The --input-path flag should point to your manifest.csv:

qiime tools import \
  --type 'SampleData[PairedEndSequencesWithQuality]' \
  --input-path /path/to/manifest.csv \
  --output-path paired-end-demux.qza \
  --source-format PairedEndFastqManifestPhred64

Also, are you sure your quality scores are phred 64? You are using PairedEndFastqManifestPhred64 as the source format, but most reads are phred 33 these days (use PairedEndFastqManifestPhred33 instead).

Keep us posted!

[Hajar]

Hey Matthew,
Many thanks for you reply,
please see below the suggested command and the error got:

(/mnt/beegfs/home/eh309/qiime2) [eh309@gm-hpc-login-01 HWL-trimmed] pwd /mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed (/mnt/beegfs/home/eh309/qiime2) [eh309@gm-hpc-login-01 HWL-trimmed] qiime tools import \

--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path /mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed/to/manifest.csv
--output-path paired-end-demux.qza
--source-format PairedEndFastqManifestPhred33
Usage: qiime tools import [OPTIONS]

Error: Invalid value for "--input-path": Path "/mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed/to/manifest.csv" does not exist.
(/mnt/beegfs/home/eh309/qiime2) [eh309@gm-hpc-login-01 HWL-trimmed]$

Actually I tried both Phred64 and Phred33, but got the same error.
Many thanks for all your efforts.
Highly appreciated.
Hajar

[thermokarst]

@Hajar

/mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed/to/manifest.csv

is this the path to your manifest file? The error indicates that there is no file found at that location, which makes sense, since it looks like you copied-and-pasted some of my boilerplate from above (/path/to should be replaced with the path to your manifest). You will need to provide a path to the file here. Just to make sure this is clear, this requirement isn't a QIIME 2 -specific thing, this is just generally how command-line interfaces work. If you are in the same directory as your manifest, you can just provide the manifest filename there.

[Hajar]

glad to let you know that the manifest worked for me
the command used:

qiime tools import
--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path /mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed/manifest.csv
--output-path single-end-demux.qza
--source-format PairedEndFastqManifestPhred33

But when I want to import as cassava files it did not work:
here is the error:
qiime tools import \

--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path /mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed
--source-format CasavaOneEightSingleLanePerSampleDirFmt
--output-path demux-paired-end.qza

There was a problem importing /mnt/beegfs/scratch/eh309/rawdata/sampletesting/HWL-trimmed:

Missing one or more files for CasavaOneEightSingleLanePerSampleDirFmt: '.+_.+_L[0-9][0-9][0-9]_R[12]_001\.fastq\.gz'
Thanks a lot for helping me.

[thermokarst]

@Hajar

import as cassava

This format is just a convenience format for importing files that meet a specific naming convention, the data isn't implicated at all, which means that there is no difference in the final result when using CasavaOneEightSingleLanePerSampleDirFmt or a manifest format.

@Hajar

glad to let you know that the manifest worked for me

Good! I suggest you take these samples and start working with them!

[Hajar]

thanks a lot for all your efforts