Manifest file TSV or CSV?

[chahoos]

Hi there,

I'm just wondering whether manifest files should be in tab-delimited or comma-delimited format? I have the latest version of QIIME2 (2022.2) and I'm working within a Singularity container built from the Docker image. While the tutorial on importing data state that "The manifest file is a tab-seperated (i.e.,.tsv) text file.", I get this error when I try to import using a TSV manifest file.

There was a problem importing /piercing_final/piercing_test_manifest.tsv:

  /piercing_final/piercing_test_manifest.tsv is not a(n) PairedEndFastqManifestPhred33 file:

  Found header on line 1 with the following labels: ['sample-id\tabsolute-filepath\tdirection'], expected: ['sample-id', 'absolute-filepath', 'direction']

I thought it might be a dos2unix problem, but I tried that to no avail. The error actually looked like QIIME2 wanted CSV, so I tried that and my data imported successfully. I've seen elsewhere on the forum that the manifest should indeed be in TSV, so I'm just confused.

[colinbrislawn (Colin J Brislawn)]

Qiime2, like other posix software, does not care about file extension, just the formatting inside of the file. But that's not the issue here:

Found header on line 1 with the following labels:
['sample-id\tabsolute-filepath\tdirection'],
expected:
['sample-id', 'absolute-filepath', 'direction']

It looks like your file has the characters \ and t in it, which is the regex for 'tab', instead of the literal tab character. See if you can get the real tab character into it.

[chahoos]

Hi @colinbrislawn,

I noticed that too but after trying a number of ways (e.g. editing within terminal, exporting as TSV win LibreOffice instead of Excel, etc.), I still can't get QIIME2 to recognize the tab character instead of '\t'. Even still, why would QIIME2 successfully import my CSV if CSV is supposedly not supported?

[colinbrislawn (Colin J Brislawn)]

@chahoos

Even still, why would QIIME2 successfully import my CSV if CSV is supposedly not supported?

Strange! I wonder if Qiime2 is doing some automatic format matching, and adapting to what's inside the file, regardless if the extension is .csv of .tsv.

@chahoos

I still can't get QIIME2 to recognize the tab character instead of '\t'.

Maybe your terminal is showing that whitespace character as '\t' even though it's the correct tab character. Maybe you could post a file that worked and a file that didn't so we can take a look?

[chahoos]

@colinbrislawn

Strange! I wonder if Qiime2 is doing some automatic format matching, and adapting to what's inside the file, regardless if the extension is .csv of .tsv.

Yeah, that's what I suspect is going on. But why isn't this functionality documented anywhere though?

@colinbrislawn

Maybe your terminal is showing that whitespace character as '\t' even though it's the correct tab character. Maybe you could post a file that worked and a file that didn't so we can take a look?

I created duplicate test manifest files in TSV and CSV formats. The TSV manifest still has the issue with '\t'. The CSV is fine, it just errors out because it can't find the non-existent FASTQ file.

QIIME2_test.tsv (103 Bytes)
QIIME2_test.csv (103 Bytes)

There was a problem importing QIIME2_test.tsv:

  QIIME2_test.tsv is not a(n) PairedEndFastqManifestPhred33 file:

  Found header on line 1 with the following labels: ['sample-id\tabsolute-filepath\tdirection'], expected: ['sample-id', 'absolute-filepath', 'direction']



There was a problem importing QIIME2_test.csv:

  QIIME2_test.csv is not a(n) PairedEndFastqManifestPhred33 file:

  File referenced on line 2 could not be found (/test/testF.fastq).

[colinbrislawn (Colin J Brislawn)]

@chahoos

Yeah, that's what I suspect is going on. But why isn't this functionality documented anywhere though?

This might be a setting specific to your terminal. Some esoteric config setting could be 'helpfully' rendering white space characters.

I can confirm that both of your files have the correct white space characters inside of them.

Note that in qiime2-2022.2, the format looks like this

sample-id	forward-absolute-filepath	reverse-absolute-filepath
sample1	s1-phred64-r1.fastq.gz	s1-phred64-r2.fastq.gz
sample2	s2-phred64-r1.fastq.gz	s2-phred64-r2.fastq.gz

Maybe this is a bug with an older version of qiime. What version are you running?

[chahoos]

@colinbrislawn

This might be a setting specific to your terminal. Some esoteric config setting could be 'helpfully' rendering white space characters.

That would be super strange, but I guess possible?

@colinbrislawn

Maybe this is a bug with an older version of qiime. What version are you running?

I'm running 2022.2.1.

qiime info
System versions
Python version: 3.8.13
QIIME 2 release: 2022.2
QIIME 2 version: 2022.2.1
q2cli version: 2022.2.0

Installed plugins
alignment: 2022.2.0
composition: 2022.2.0
cutadapt: 2022.2.0
dada2: 2022.2.0
deblur: 2022.2.0
demux: 2022.2.0
diversity: 2022.2.1
diversity-lib: 2022.2.1
emperor: 2022.2.0
feature-classifier: 2022.2.0
feature-table: 2022.2.0
fragment-insertion: 2022.2.0
gneiss: 2022.2.0
longitudinal: 2022.2.0
metadata: 2022.2.0
phylogeny: 2022.2.0
quality-control: 2022.2.0
quality-filter: 2022.2.0
sample-classifier: 2022.2.0
taxa: 2022.2.0
types: 2022.2.0
vsearch: 2022.2.0

Application config directory
/home/qiime2/q2cli

Getting help
To get help with QIIME 2, visit https://qiime2.org

[Keegan-Evans]

@chahoos,

Could you post your full input command?

[chahoos]

I built the Singularity container from the Docker image.

singularity build qiime2-2022.2.sif docker://quay.io/qiime2/core:2022.2

--env PYTHONNOUSERSITE=1 is to prevent libraries from $HOME/.local leaking into the Singularity container.

module load singularity

singularity exec --env PYTHONNOUSERSITE=1 -B $PWD:/qiime qiime2-2022.2.sif qiime tools import \
  --type 'SampleData[PairedEndSequencesWithQuality]' \
  --input-path /qiime/$manifest.tsv \
  --output-path /qiime/$manifest.qza \
  --input-format PairedEndFastqManifestPhred33

[colinbrislawn (Colin J Brislawn)]

Justine just suggested this:

I think it should be PairedEndFastqManifestPhred33V2 instead of PairedEndFastqManifestPhred33, since V2 is the more permissive TSV format

See if that other format works better for you!

[chahoos]

@colinbrislawn

PairedEndFastqManifestPhred33V2

It looks like QIIME recognized the tabs! So PairedEndFastqManifestPhred33V2 recognizes my TSV manifest files while PairedEndFastqManifestPhred33 does not. But for some reason, PairedEndFastqManifestPhred33 does recognize my CSV manifest files. So strange. Anyways, thanks for everyone's input on this. I wonder if I'm the only one who has experienced this and if not does everyone just try CSV and get on with their lives. I think that's what I did when I used QIIME previously. Hopefully, future confused QIIMErs about this will find their way to this forum post.

There was a problem importing QIIME2_test.tsv:

  QIIME2_test.tsv is not a(n) PairedEndFastqManifestPhred33V2 file:

  Filepath on line 1 and column "forward-absolute-filepath" could not be found (/test/testF.fastq) for sample "test".

[Keegan-Evans]

@chahoos,
Thanks for following up, you are right, old posts are a valuable resource for others(or speaking for myself here), our future selves . In spirit, unless you have older manifest files that are still relevant, it is generally better to produce new manifests using the V2 format and tab separated files!