Error: Not a Phred33 file

prestcat · October 6, 2017, 10:25pm

Hello,
I am trying to import some data for analysis. I keep coming up on this error saying that my csv file is not a Phred33. I know for sure it's not a Phred64, but I tried that anyway. Unsurprisingly, this did not work. I am running qiime2-2017.9. Here's the command:

(qiime2-2017.9) CAS-ANTH119M:Compressed Admin$ qiime tools import
--type SampleData[PairedEndSequencesWithQuality]
--input-path Workbook3.csv
--output-path PairedEndDemux.qza
--source-format PairedEndFastqManifestPhred33
There was a problem importing Workbook3.csv:
Workbook3.csv is not a(n) PairedEndFastqManifestPhred33 file
(qiime2-2017.9) CAS-ANTH119M:Compressed Admin$

Any suggestions?

thermokarst · October 6, 2017, 10:43pm

Hi @prestcat!

The specification for the manifest formats are pretty strict, they must be a CSV with no blank rows or extra columns, and the header row must be exactly sample-id,absolute-filepath,direction:

sample-id,absolute-filepath,direction
sample-1,$PWD/some/filepath/sample1_R1.fastq.gz,forward
sample-2,$PWD/some/filepath/sample2_R1.fastq.gz,forward

Judging by your manifest filename (Workbook3.csv), it looks like you maybe used a spreadsheet program like Excel to create this file — those tools are notorious for slapping on extra rows and columns with no regard to your original data!

sample-id,absolute-filepath,direction,
sample-1,$PWD/some/filepath/sample1_R1.fastq.gz,forward,
sample-2,$PWD/some/filepath/sample2_R1.fastq.gz,forward,
,,,,
,,,,

(note the extra commas at the end, which would create a new “column” in the table, also the extra empty rows at the end)

It would be really helpful if we could get a closer look at that manifest file (Workbook3.csv). Would it be possible to send in a DM or attach here? If you are unable to share that information, could you make a minimum working example that you could share, either by copying-and-pasting (less-ideal, the forum sometimes changes the formatting) or by sending? Thanks so much!

prestcat · October 9, 2017, 10:43pm

For some reason I’m not allowed to upload a CSV or excel file. Is there another way to send this to you?

jairideout · October 10, 2017, 8:08pm

Hi @prestcat! @thermokarst is out of the office for a bit so I’ll continue helping you. Can you try sharing the file via Dropbox or Google Drive?

prestcat · October 11, 2017, 4:15pm

Thanks for the suggestion! Here’s a link to the Google Drive file: https://drive.google.com/file/d/0B1iX01dXRn3PUWZPU1AxS2Y3RGc/view?usp=sharing
This file comes from my personal computer, so the file name and the paths are different than the one mentioned in my first post. They were created in the same way and I still came across the same errors with this file.
I am not familiar with Macs (I bought one just because my lab said qiime is easier with them) so it’s completely possible that that’s the source of my problem. I’d be embarrassed if it’s something that basic, but then that means it’d be an easier fix! Thank you for looking at this!

jairideout · October 11, 2017, 7:57pm

Thanks for sharing your file! It was as @thermokarst suspected: there are trailing blank cells on each line that are probably being created by the spreadsheet program you’re using. Deleting those trailing cells should fix it.

Here’s the first few lines in the file you sent me:

sample-id,absolute-file-path,direction ,,,
RC01_K,/Users/jcothern/desktop/TaborModule/CompressedData/RC01_K_S157_R1_001.fastq.gz,forward,,,
RC01_K,/Users/jcothern/desktop/TaborModule/CompressedData/RC01_K_S157_R2_001.fastq.gz,reverse,,,
RC28_K,/Users/jcothern/desktop/TaborModule/CompressedData/RC28_K_S98_R1_001.fastq.gz,forward,,,
RC28_K,/Users/jcothern/desktop/TaborModule/CompressedData/RC28_K_S98_R2_001.fastq.gz,reverse,,,
RC37_K,/Users/jcothern/desktop/TaborModule/CompressedData/RC37_K_S110_R1_001.fastq.gz,forward,,,

Those trailing commas can be removed, resulting in:

sample-id,absolute-file-path,direction
RC01_K,/Users/jcothern/desktop/TaborModule/CompressedData/RC01_K_S157_R1_001.fastq.gz,forward
RC01_K,/Users/jcothern/desktop/TaborModule/CompressedData/RC01_K_S157_R2_001.fastq.gz,reverse
RC28_K,/Users/jcothern/desktop/TaborModule/CompressedData/RC28_K_S98_R1_001.fastq.gz,forward
RC28_K,/Users/jcothern/desktop/TaborModule/CompressedData/RC28_K_S98_R2_001.fastq.gz,reverse
RC37_K,/Users/jcothern/desktop/TaborModule/CompressedData/RC37_K_S110_R1_001.fastq.gz,forward

Let us know how it goes!

prestcat · October 12, 2017, 4:16pm

Thank you so much for taking a look at this! I’ll let you know what happens when I run this on Friday.

prestcat · October 13, 2017, 6:41pm

So I followed your advice, here's what happened:

(qiime2-2017.9) Jessicas-MacBook-Air:TaborModule jcothern$ qiime tools import --type SampleData[PairedEndSequencesWithQuality] --input-path "GradStudentRC16s.csv" --output-path "PairedEndDemux.qza" --source-format PairedEndFastqManifest"Phred33"
There was a problem importing GradStudentRC16s.csv:
GradStudentRC16s.csv is not a(n) PairedEndFastqManifestPhred33 file

Then I tried to Phred64:

(qiime2-2017.9) Jessicas-MacBook-Air:TaborModule jcothern$ qiime tools import --type SampleData[PairedEndSequencesWithQuality] --input-path "GradStudentRC16s.csv" --output-path "PairedEndDemux.qza" --source-format PairedEndFastqManifest"Phred64"
There was a problem importing GradStudentRC16s.csv:
GradStudentRC16s.csv is not a(n) PairedEndFastqManifestPhred64 file

I've run this on multiple computers, all with the same problem. Others in my lab are stumped too. One person managed to make it work, but we can't figure out what she did differently. Here's the new file: https://drive.google.com/file/d/0B1iX01dXRn3Pb2Y4R0FaV2Nmejg/view?usp=sharing

Please let me know if you have any other ideas.

jairideout · October 16, 2017, 10:41pm

The fourth line of the file has a typo, can you try changing “forwardd” to “forward”?

Note: I know this debugging process is a pain because the error message doesn’t hint at what’s wrong with the file. With the upcoming 2017.10 release we’ll have better error reporting when importing, etc. We’ll follow up here when that’s available!

prestcat · October 17, 2017, 4:34pm

Thank you so much! That was the problem, I can’t believe I missed it.
I’m looking forward to the update

system · November 17, 2017, 10:34pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.

jairideout · November 29, 2017, 8:27pm

As of the QIIME 2 2017.10 release, more informative error messages are raised when importing data using the “fastq manifest” formats. For example, with your file that contained “forwardd” instead of “forward”, here’s the new error message you’ll get:

An unexpected error has occurred:

  Directions can only be "forward" or "reverse", but observed: forwardd