unable to import my data

Yiggrite · December 9, 2023, 7:16pm

hey, I am trying to import my demultiplexed data fiile into Qiime2. I have the Manifest file, but always meet this error. Could you please help me?
(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ qiime tools import \

--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path casava-18-paired-end-demultiplexed
--input-format CasavaOneEightSingleLanePerSampleDirFmt
--output-path demux-paired-end.qza
There was a problem importing casava-18-paired-end-demultiplexed:

Missing one or more files for CasavaOneEightSingleLanePerSampleDirFmt: '.+_.+_L[0-9][0-9][0-9]_R[12]_001\.fastq\.gz'

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$
(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ ls
casava-18-paired-end-demultiplexed casava-18-paired-end-demultiplexed.zip 'manifest file.txt' 'metadata file.txt'
(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$
Here is my manifest file.

Will appreciate it very much if you can fix this problem.
Thanks

colinbrislawn · December 9, 2023, 8:04pm

Hello @Yiggrite,

Your files are missing. This makes sense because

it looks like your files are inside this zip file.

If you extract that .zip file so you can see the fastq files inside, then the files can be imported.

Let us know if that works for you or if you have questions about how to extract the .zip file.

Yiggrite · December 10, 2023, 4:00pm

I have unzipped my data. If this is what you mean.

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ ls casava-18-paired-end-demultiplexed
AG1_R1_001.fastq.gz Controlgroup3_R2_001_fastq.gz P371_R1_001_fastq.gz P553_R2_001_fastq.gz
AG1_R2_001.fastq.gz Controlgroup4_R1_001_fastq.gz P371_R2_001_fastq.gz P554_R1_001_fastq.gz
AG2_R1_001.fastq.gz Controlgroup4_R2_001_fastq.gz P372_R1_001_fastq.gz P554_R2_001_fastq.gz
AG2_R2_001.fastq.gz Controlgroup5_R1_001_fastq.gz P372_R2_001_fastq.gz P555_R1_001_fastq.gz
AG3_R1_001.fastq.gz Controlgroup5_R2_001_fastq.gz P373_R1_001_fastq.gz P555_R2_001_fastq.gz
AG3_R2_001.fastq.gz Modelgroup1_R1_001.fastq.gz P373_R2_001_fastq.gz P851_R1_001_fastq.gz
AG4_R1_001.fastq.gz Modelgroup1_R2_001.fastq.gz P374_R1_001_fastq.gz P851_R2_001_fastq.gz
AG4_R2_001.fastq.gz Modelgroup2_R1_001.fastq.gz P374_R2_001_fastq.gz P852_R1_001_fastq.gz
AG5_R1_001.fastq.gz Modelgroup2_R2_001.fastq.gz P375_R1_001_fastq.gz P852_R2_001_fastq.gz
AG5_R2_001.fastq.gz Modelgroup3_R1_001_fastq.gz P375_R2_001_fastq.gz P853_R1_001_fastq.gz
Controlgroup1_R1_001_fastq.gz Modelgroup3_R2_001_fastq.gz P551_R1_001_fastq.gz P853_R2_001_fastq.gz
Controlgroup1_R2_001_fastq.gz Modelgroup4_R1_001.fastq.gz P551_R2_001_fastq.gz P854_R1_001_fastq.gz
Controlgroup2_R1_001_fastq.gz Modelgroup4_R2_001.fastq.gz P552_R1_001_fastq.gz P854_R2_001_fastq.gz
Controlgroup2_R2_001_fastq.gz Modelgroup5_R1_001.fastq.gz P552_R2_001_fastq.gz P855_R1_001_fastq.gz
Controlgroup3_R1_001_fastq.gz Modelgroup5_R2_001.fastq.gz P553_R1_001_fastq.gz P855_R2_001_fastq.gz
(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$

colinbrislawn · December 10, 2023, 8:39pm

That looks perfect!

Now when you run the Qiime2 command, does it import them?

Yiggrite · December 10, 2023, 4:44pm

I used the manifest fill, but met the error:
(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ qiime tools import \

--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path ./manifest.tsv
--input-format CasavaOneEightSingleLanePerSampleDirFmt
--output-path demux-paired-end.qza
There was a problem importing ./manifest.tsv:

manifest.tsv is not a directory.

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$
anything wrong here?
Thanks.

colinbrislawn · December 11, 2023, 3:04am

Take a look at the example command here:
https://docs.qiime2.org/2023.9/tutorials/importing/#casava-1-8-single-end-demultiplexed-fastq

Make sure you have the right files and folder for the command.

For this format a manifest file is not needed, and --input-path should point to that folder full of .fastq.gz file listed above.

Yiggrite · December 11, 2023, 11:35pm

I am sorry, my data is paired-end-demultiplexed fastq files. so I may need the manifest file.
I have checked the format, but still met this error:

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ head manifest.tsv
sample-id       forward-absolute-filepath       reverse-absolute-filepath
Controlgroup1   $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R1_001.fastq.gz  $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R2_001.fastq.gz
Controlgroup2   $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup2_R1_001.fastq.gz  $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup2_R2_001.fastq.gz
Controlgroup3   $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup3_R1_001.fastq.gz  $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup3_R2_001.fastq.gz
Controlgroup4   $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup4_R1_001.fastq.gz  $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup4_R2_001.fastq.gz
Controlgroup5   $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup5_R1_001.fastq.gz  $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup5_R2_001.fastq.gz
P372    $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P372_R1_001.fastq.gz   $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P372_R2_001.fastq.gz
P373    $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P373_R1_001.fastq.gz   $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P373_R2_001.fastq.gz
Modelgroup3     $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Modelgroup3_R1_001.fastq.gz    $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Modelgroup3_R2_001.fastq.gz
P854    $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P854_R1_001.fastq.gz   $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P854_R2_001.fastq.gz

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ qiime tools import \
>   --type 'SampleData[PairedEndSequencesWithQuality]' \
>   --input-path manifest.tsv \
>   --output-path paired-end-demux.qza \
  --inpu>   --input-format PairedEndFastqManifestPhred64V2
There was a problem importing manifest.tsv:

  manifest.tsv is not a(n) PairedEndFastqManifestPhred64V2 file:

  Filepath on line 1 and column "forward-absolute-filepath" could not be found ($home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R1_001.fastq.gz) for sample "Controlgroup1".

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$

Can you help?
Thanks.

colinbrislawn · December 12, 2023, 4:28am

Yes, I can help you understand this error. Here is the important part:

Filepath on line 1 and column "forward-absolute-filepath" could not be found ($home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R1_001.fastq.gz) for sample "Controlgroup1".

It looks like that first file can not be found.

I would suggest replacing the file paths that start with $home/yiggrite/ with absolute file paths. You can find the absolute file path to the muxed-se-barcode-in-seq folder by running pwd inside of it.

P.S. I've edited your post to put code in a code block. You can do that with three ticks ``` before and after a code block.

Yiggrite · December 15, 2023, 11:20pm

Hi, here is my work directory,
(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ pwd
/home/yiggrite/qiime2/muxed-se-barcode-in-seq
(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$

and i changed the manifest file format, csv or txt, and always met the same error. a little confused....

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ pwd
/home/yiggrite/qiime2/muxed-se-barcode-in-seq
(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ head manifest2.txt
sample-id,absulote-filepath,direction
Controlgroup1,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R1_001.fastq.gz,forward
Controlgroup2,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup2_R1_001.fastq.gz,forward
Controlgroup3,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup3_R1_001.fastq.gz,forward
Controlgroup4,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup4_R1_001.fastq.gz,forward
Controlgroup5,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup5_R1_001.fastq.gz,forward
P372,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P372_R1_001.fastq.gz,forward
P373,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P373_R1_001.fastq.gz,forward
Modelgroup3,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Modelgroup3_R1_001.fastq.gz,forward
P854,home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P854_R1_001.fastq.gz,forward (qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq qiime tools import \

--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path manifest2.txt
--output-path paired-end-demux.qza
--input-format PairedEndFastqManifestPhred64V2
There was a problem importing manifest2.txt:

manifest2.txt is not a(n) PairedEndFastqManifestPhred64V2 file:

Found unrecognized ID column name 'sample-id,absulote-filepath,direction' while searching for header. The first column name in the header defines the ID column, and must be one of these values:

Case-insensitive: 'feature id', 'feature-id', 'featureid', 'id', 'sample id', 'sample-id', 'sampleid'

Case-sensitive: '#OTU ID', '#OTUID', '#Sample ID', '#SampleID', 'sample_name'

NOTE: Metadata files must contain tab-separated values.

There may be more errors present in the metadata file. To get a full report, sample/feature metadata files can be validated with Keemei: https://keemei.qiime2.org

Find details on QIIME 2 metadata requirements here: Metadata in QIIME 2 — QIIME 2 2020.8.0 documentation

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$
the manifest file is readable right? but what is the correct format?

Yiggrite · December 15, 2023, 11:42pm

Also, this format didn't work.

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ head manifest.tsv
sample-id,forward-absolute-filepath,reverse-absolute-filepath
Controlgroup1,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R1_001.fastq.gz,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R2_001.fastq.gz
Controlgroup2,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup2_R1_001.fastq.gz,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup2_R2_001.fastq.gz
Controlgroup3,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup3_R1_001.fastq.gz,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup3_R2_001.fastq.gz
Controlgroup4,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup4_R1_001.fastq.gz,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup4_R2_001.fastq.gz
Controlgroup5,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup5_R1_001.fastq.gz,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup5_R2_001.fastq.gz
P372,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P372_R1_001.fastq.gz,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P372_R2_001.fastq.gz
P373,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P373_R1_001.fastq.gz,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P373_R2_001.fastq.gz
Modelgroup3,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Modelgroup3_R1_001.fastq.gz,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Modelgroup3_R2_001.fastq.gz
P854,$home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P854_R1_001.fastq.gz,home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P854_R2_001.fastq.gz (qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq qiime tools import \

--type 'SampleData[PairedEndSequencesWithQuality]'
nput-pat> --input-path manifest.tsv
--output-path paired-end-demux.qza
--input-format PairedEndFastqManifestPhred64V2
There was a problem importing manifest.tsv:

manifest.tsv is not a(n) PairedEndFastqManifestPhred64V2 file:

Found unrecognized ID column name 'sample-id,forward-absolute-filepath,reverse-absolute-filepath' while searching for header. The first column name in the header defines the ID column, and must be one of these values:

Case-insensitive: 'feature id', 'feature-id', 'featureid', 'id', 'sample id', 'sample-id', 'sampleid'

Case-sensitive: '#OTU ID', '#OTUID', '#Sample ID', '#SampleID', 'sample_name'

NOTE: Metadata files must contain tab-separated values.

There may be more errors present in the metadata file. To get a full report, sample/feature metadata files can be validated with Keemei: https://keemei.qiime2.org

Find details on QIIME 2 metadata requirements here: Metadata in QIIME 2 — QIIME 2 2020.8.0 documentation

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$

colinbrislawn · December 16, 2023, 12:19am

You are heading in the right direction. Here's the important line of that new error:

NOTE: Metadata files must contain tab-separated values.

I see your metadata file has commas, like a .csv file.

Does your spreadsheet program let you 'save as' a tsv file?
(Renaming to .tsv will not work, the internal structure must have tabs instead of commas.)

Yiggrite · December 16, 2023, 5:13pm

Thank you for your help. I used the Notepad in Windows, input my data in rows and columns, separated values by tabs, then saved as .tsv. still met the error.

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ head manifest.tsv
sample-id forward-absolute-filepath reverse-absolute-filepath
Controlgroup1 $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R1_001.fastq.gz $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R2_001.fastq.gz
Controlgroup2 $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup2_R1_001.fastq.gz $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup2_R2_001.fastq.gz
Controlgroup3 $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup3_R1_001.fastq.gz $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup3_R2_001.fastq.gz
Controlgroup4 $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup4_R1_001.fastq.gz $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup4_R2_001.fastq.gz
Controlgroup5 $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup5_R1_001.fastq.gz $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup5_R2_001.fastq.gz
P372 $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P372_R1_001.fastq.gz $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P372_R2_001.fastq.gz
P373 $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P373_R1_001.fastq.gz $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P373_R2_001.fastq.gz
Modelgroup3 $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Modelgroup3_R1_001.fastq.gz $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Modelgroup3_R2_001.fastq.gz
P854 $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P854_R1_001.fastq.gz home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/P854_R2_001.fastq.gz (qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq qiime tools import \

--type 'SampleData[PairedEndSequencesWithQuality]'
nput-pat> --input-path manifest.tsv
put-path> --output-path paired-end-demux.qza
--input-format PairedEndFastqManifestPhred64V2
There was a problem importing manifest.tsv:

manifest.tsv is not a(n) PairedEndFastqManifestPhred64V2 file:

Filepath on line 1 and column "forward-absolute-filepath" could not be found ($home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R1_001.fastq.gz) for sample "Controlgroup1".

Anything wrong about the format?
Thank you.

colinbrislawn · December 16, 2023, 6:04pm

Ok great! I can see the file now has tabs.

Here's the new error:

Filepath on line 1 and column "forward-absolute-filepath" could not be found

Can you run this and post the result? This attempts to open the file and print a few lines:

gzip -dc $home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R1_001.fastq.gz | head

Yiggrite · December 16, 2023, 6:42pm

(qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq$ gzip -dc home/yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R1_001.fastq.gz | head gzip: /yiggrite/qiime2/muxed-se-barcode-in-seq/casava-18-paired-end-demultiplexed/Controlgroup1_R1_001.fastq.gz: No such file or directory (qiime2) yiggrite@MWMJ07RHBF:~/qiime2/muxed-se-barcode-in-seq

mustbe something wrong here

Yiggrite · December 16, 2023, 6:45pm

Also, it has the problem:
manifest.tsv is not a(n) PairedEndFastqManifestPhred64V2 file:

colinbrislawn · December 16, 2023, 6:55pm

Yes, the file is missing or the path to it is wrong.

Yiggrite · December 16, 2023, 7:17pm

can you see this location?

I think it is the correct path.

colinbrislawn · December 16, 2023, 7:26pm

Cool!

Can you show me the path in Linux?

Yiggrite · December 16, 2023, 7:30pm

can you see the picture? it is in Linux?

colinbrislawn · December 16, 2023, 7:36pm

That's Windows.

Linux is where you run the Qiime2 commands.