Need help importing a manifest format

[ashish02 (ASHISH VERMA)]

Hi Sir,

Even I was looking for this answer for importing the demultiplexed raw reads. I have prepared the manifest file as shown in importing tutorial for my samples and was ready to import the file using the command for paired end data as shown below:

qiime tools import
–type ‘SampleData[PairedEndSequencesWithQuality]’
–input-path pe-64-manifest
–output-path paired-end-demux.qza
–input-format PairedEndFastqManifestPhred64V2

while running this command, I’m running with errors:

Traceback (most recent call last):
File “/home/mayil/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/util.py”, line 91, in parse_format
format_record = pm.formats[format_str]
KeyError: ‘PairedEndFastqManifestPhred64V2’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/mayil/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2cli/tools.py”, line 146, in import_data
view_type=input_format)
File “/home/mayil/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 206, in import_data
view_type = qiime2.sdk.parse_format(view_type)
File “/home/mayil/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/util.py”, line 93, in parse_format
raise TypeError(“No format: %s” % format_str)
TypeError: No format: PairedEndFastqManifestPhred64V2

I may not be able to understand, but how the above command will import the R1 and R2 data for the four samples (total 8) and how is it using the manifest file which I have created for importing? I do not see them being used in the above command for analysis.

Kindly suggest.

[thermokarst]

Hey there @ashish02!

@ashish02

KeyError: ‘PairedEndFastqManifestPhred64V2’

This error is because of this:

@ashish02

qiime2-2019.1

You are running an old version of QIIME 2, which does not contain that format. You can either stay with that version of QIIME 2, but use the PairedEndFastqManifestPhred64, or, upgrade to the latest release of QIIME 2.

[ashish02 (ASHISH VERMA)]

Thank you so much Sir for the information. I have updated the version to 2019.7 and got the result as mentioned in the tutorial.
But I’m not being able to import my data. I have also seen the queries earlier shared on QIIME2 forum and came to know that I have to prepare my manifest file in .csv format. In addition to it, I have also used the manifest file in .tsv format as mentioned in tutorial. I have followed the command as shown below:

Command-line
$ qiime tools import --type ‘SampleData[SequencesWithQuality]’ --input-path manifest.csv --output-path paired-end-demux.qza --input-format PairedEndFastqManifestPhred64V2

After running the command I’m getting the following error:

Traceback (most recent call last):
File “/home/mayil/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/builtin/tools.py”, line 154, in import_data
view_type=input_format)
File “/home/mayil/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 241, in import_data
validate_level=‘max’)
File “/home/mayil/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 266, in _from_view
recorder=recorder)
File “/home/mayil/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/core/transform.py”, line 59, in make_transformation
(self._view_type, other._view_type))
Exception: No transformation from <class ‘q2_types.per_sample_sequences._format.PairedEndFastqManifestPhred64V2’> to <class ‘q2_types.per_sample_sequences._format.SingleLanePerSampleSingleEndFastqDirFmt’>

An unexpected error has occurred:

No transformation from <class ‘q2_types.per_sample_sequences._format.PairedEndFastqManifestPhred64V2’> to <class ‘q2_types.per_sample_sequences._format.SingleLanePerSampleSingleEndFastqDirFmt’>

See above for debug info.
(qiime2-2019.7)

My manifest.csv file looks like as follows:

sample-id,absolute-filepath,direction
SA,$home/mayil/training/qiime2-2019.7/reads/SA_R1.fastq,forward
SA,$home/mayil/training/qiime2-2019.7/reads/SA_R2.fastq,reverse
SAOS,$home/mayil/training/qiime2-2019.7/reads/SAOS_R1.fastq,forward
SAOS,$home/mayil/training/qiime2-2019.7/reads/SAOS_R2.fastq,reverse
OG,$home/mayil/training/qiime2-2019.7/reads/OG_R1.fastq,forward
OG,$home/mayil/training/qiime2-2019.7/reads/OG_R2.fastq,reverse
DI,$home/mayil/training/qiime2-2019.7/reads/DI_R1.fastq,forward
DI,$home/mayil/training/qiime2-2019.7/reads/DI_R2.fastq,reverse

Kindly let me know how I can solve this issue. Many thanks in advance.

[thermokarst]

Hi @ashish02! You are combing a few incompatible elements here, let's try and untangle:

@ashish02

$ qiime tools import --type ‘SampleData[SequencesWithQuality]’ --input-path manifest.csv --output-path paired-end-demux.qza --input-format PairedEndFastqManifestPhred64V2

Here you are attempting to import a paired-end format as a single-ended semantic type. Please update the --type to SampleData[PairedEndSequencesWithQuality].

@ashish02

My manifest.csv file looks like as follows:

The you are specifying the TSV format as the format above (V2), but using the CSV. This manifest should be a TSV (tab-seperated).

@ashish02

PairedEndFastqManifestPhred64V2

Not related to your current problems, but, do you really have Phred 64 reads? We generally only see those on relatively old sequencing product...

Keep us posted!

[ashish02 (ASHISH VERMA)]

Thank you Sir for the reply. After updating my command I am again running with error message as mentioned below.:

Command:

$ qiime tools import --type ‘SampleData[PairedEndSequencesWithQuality]’ --input-path final_manifest.tsv --output-path paired-end-demux.qza --input-format PairedEndFastqManifestPhred33V2

Error message:

There was a problem importing final_manifest.tsv:

final_manifest.tsv is not a(n) PairedEndFastqManifestPhred33V2 file:

Filepath on line 1 and column “forward-absolute-filepath” could not be found ($home/mayil/training/qiime2-2019.7/reads/SA_R1.fastq) for sample “SA”.

(qiime2-2019.7)

My final_manifest file looks like as (I am copying it from txt file, but have used it in tsv format in the above command). I have also validated it at Keemei google spreadsheet and it was found to be fine.

sample-id	forward-absolute-filepath	reverse-absolute-filepath
SA	$home/mayil/training/qiime2-2019.7/reads/SA_R1.fastq	$home/mayil/training/qiime2-2019.7/reads/SA_R2.fastq
SAOS	$home/mayil/training/qiime2-2019.7/reads/SAOS_R1.fastq	$home/mayil/training/qiime2-2019.7/reads/SAOS_R2.fastq
OG	$home/mayil/training/qiime2-2019.7/reads/OG_R1.fastq	$home/mayil/training/qiime2-2019.7/reads/OG_R2.fastq
DI	$home/mayil/training/qiime2-2019.7/reads/DI_R1.fastq	$home/mayil/training/qiime2-2019.7/reads/DI_R2.fastq

I have stored the raw reads under “reads” folder as (SA_R1.fastq-------DI_R2.fastq; a total of 8 demultiplexed R1 and R2 reads). Please suggest.

[colinbrislawn (Colin J Brislawn)]

Good afternoon,

I’m glad you got this file to validate. You are so close! I think there is only one more thing you need to fix.

When says absolute-filepath, this means you need to use the path starting from root, not from $home. So for example, this would work:
/Users/myaccount/data/training/qiime2-2019.7/reads/SA_R1.fastq
but this would not
$home/data/training/qiime2-2019.7/reads/SA_R1.fastq

Notice how I start from the root directory with a / slash / to make this work.

Once you replace $home with the absolute file path, your manifest file should work great.

Colin

P.S. You can find the absolute-filepath of your home directory by typing
echo $home
or
pwd ~

[ashish02 (ASHISH VERMA)]

Thank you so much Sir. It really worked.