FASTQ Files Importing Error

The error I receive is:

**There was a problem importing fastq_files_demux.csv:**

**fastq_files_demux.csv is not a(n) PairedEndFastqManifestPhred64 file**

I tried removing the $PWD and it still didn’t work. Any other ideas?

Hey, What code you are typing to import your files? Cause i dont think that you need a metadata file.
And as far as i know in qiime the format of the metadata file must be .tsv.

I’m trying to import FASTQ files. Based on the importing tutorial, to import multiple FASTQ files into one .qza you have to create a .csv file that points to the FASTQ files.

qiime tools import \
--type 'SampleData[PairedEndSequencesWithQuality]' \
--input-path fastq_files_demux.csv \
--output-path demux_seqs.qza \
--input-format PairedEndFastqManifestPhred64

Hi @Stephanieorch,
Assuming you’re using your 2018.8 qiime version your commands all look ok to me. And the manifest file is indeed suppose to be a .csv file so that’s not the issue either. And I don’t have an answer for you either, but while one of the devs gives this a more thorough look I just wanted to ask if you’re sure your phred score is actually 64 and not 33. I don’t think this is the source of your problem since I expect you would get a different error for that but I looked at the quality scores you provided and based on the 33 vs 64 phred scores, I think yours makes more sense as type 33. I could be wrong but worth double checking. Again, I doubt this is going to solve your pbolem, but a potential problem later:P

1 Like

Yes, I thought that may be the problem to so I tried both 33 and 64 and still received an error.

Hello again Stephanie,

I'm glad lots of folks are Qiime-ing in this morning. To help out.

I think I may have found a clue.
42%20AM

What program did you use to make the .csv file? I'm not sure if using Windows lie ending causes a problem, but I know some programs expect Unix line endings instead.

I've uploaded a version of this document using Unix line endings, which might be worth trying.
fastq_files_demux.csv (8.8 KB)

Colin

2 Likes

I originally used excel to make the document and then used the command line to edit.

I tried the document you sent me and got the error again. I used Atom to try adding $PWD and see if that would work and it didnt. I also tried both Phred33 and Phred64 for all of these.

Any other ideas? Thanks!

1 Like

I think you are doing everything right… I’m not sure what’s going wrong.

As as a guess… what happens if you comment out or remove the sequences that start with O. like O.50064. Capitol Os and .periods. should not cause problems, but I’m just looking for any potential issues.

Qiime devs, is there a script (or internal function), we can use to validate this file? Looks like Keemei does not validate FastqManifest files, but maybe it could!

Colin

I tried importing the files after removing the FASTQ files starting with Os from the .csv file and it still gave me the same errors.

I solved the issue! Turns out .csv files DO NOT work. You MUST use TestEdit.

2 Likes

So saving as a .csv file using Excel doesn’t work, but saving as a .txt file using TextEdit does? That could be really helpful for future users.

Colin

If you use Excel to make your CSV, it will insert CRLF line breaks (whether or not you are using windows, I believe). So you need to either just open in textedit and use find/replace to fix the line breaks, or use mac2unix or a program like that to convert the carriage returns.

So Excel is okay — just make sure to convert the line endings.

This will be an issue with making sample metadata files, too, not only manifest files.

1 Like

I believe @colinbrislawn tried to alleviate the line break issue when he gave me a document to try that he created using Unix line endings. However, this document did not work either.

The issue could be other special characters that Excel is importing. In general, your advice to use textedit is valid: Excel creates headaches downstream!

Just to provide some closure, I think I found the issue:

(via hexdump -C <filename>)

00000000  ef bb bf 73 61 6d 70 6c  65 2d 69 64 2c 61 62 73  |...sample-id,abs|
00000010  6f 6c 75 74 65 2d 66 69  6c 65 70 61 74 68 2c 64  |olute-filepath,d|
00000020  69 72 65 63 74 69 6f 6e  0d 0a 35 30 30 36 34 2c  |irection..50064,|

First column is position, next two are the bytes as a single hex number and finally the ASCII value (if it is a visible character, otherwise it will be a dot)

Here we see that there are 3 bytes in front of the sample-id which starts the file. It appears you have an UTF-8 byte order mark, which is actually pretty rare, as UTF-8 doesn't have byte ordering issues like UTF-16 does.

Nonetheless, it was enough for QIIME 2 to get spooked and so when you used TextEdit, it likely stripped those bytes out making it work correctly again.

This is ultimately related to this issue:

(I just listed it as a good #hacktoberfest PR if anyone is interested in trying their hand at development, the fix is actually very straight-forward)

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.