FASTQ Files Importing Error

User Support
1

I'm trying to import FASTQ files but I keep getting an error that my .csv file isn't the correct file format. Also, the sequences I'm importing have already been demultiplexed so I'm unsure if the importing method I'm using is incorrect (I'm using paired end Phred offset score 64). I'm unable to attach my FASTQ files but one of the sequences is:

CCGCCAATTTCTTTGAGTTTCAACCTTGCGGTCGTACTCCCCAGGCGGATTGCTTATTGCGTTAGCTGCGGCACTGATCCCCGGAAAGGTTCCAACCCCTAGCCTTCTTCTTTTACGGCTTGGACTACCAGGTTTCTTAATCTTTTTCCCTCCCCACGTTTTCGATCCTCACCTTCTGTTCAAGACCTGAACCCCGTTTTCCCCCCCGGTTTCCCCCCTTTTTTTTCCCCTTTTAACCGCTCCCCTTGGATTTCCCCTTCCCCCTCTTCCCTCCAATTTTGACATTTTCCAACGTAACCTT

[email protected]:[email protected]<[email protected]@+8,,CCF9<C,,@,,,,,:[email protected]@F++C,C,[email protected]=,C<E,<+B=++44,,,,,9,,<F,557+++8>@=,5,A,,5>5:,,,3:,35,:,C,,,,,[email protected]@+,3++7D>>8+,8*,@@,,,,,,,7;,,,5,62>,4,144;1=<0=5=9):+;0.5=)2*/)/))0:):F:)20(,()((9()0/4*:*)))..)(,42(-(,.))()((..4<>4)(-.).5))-4)((((((.-)

@M02352:182:000000000-BRDB5:1:1101:9865:2050 2:N:0:169

fastq_files_demux.csv (9.2 KB)

2

Good morning,

Could you post the full text of the error you are getting about your csv file? It’s hard to say more without seeing the error itself, but I wonder if using the absolute-filepath in the absolute-filepath column would work better… For example, you could try using
/Users/stephanieorch/Desktop/disaster/WhitesonTest/16S/50064_S181_L001_R1_001.fastq.gz
Which does not start with the $PDW bash variable.

Let me know if omitting the $PWD helps, or you are able to post the full error so I can continue to look for clues.

Colin

3

The error I receive is:

**There was a problem importing fastq_files_demux.csv:**

**fastq_files_demux.csv is not a(n) PairedEndFastqManifestPhred64 file**

I tried removing the $PWD and it still didn’t work. Any other ideas?

4

Hey, What code you are typing to import your files? Cause i dont think that you need a metadata file.
And as far as i know in qiime the format of the metadata file must be .tsv.

5

I’m trying to import FASTQ files. Based on the importing tutorial, to import multiple FASTQ files into one .qza you have to create a .csv file that points to the FASTQ files.

qiime tools import \
--type 'SampleData[PairedEndSequencesWithQuality]' \
--input-path fastq_files_demux.csv \
--output-path demux_seqs.qza \
--input-format PairedEndFastqManifestPhred64
6
7

Hi @Stephanieorch,
Assuming you’re using your 2018.8 qiime version your commands all look ok to me. And the manifest file is indeed suppose to be a .csv file so that’s not the issue either. And I don’t have an answer for you either, but while one of the devs gives this a more thorough look I just wanted to ask if you’re sure your phred score is actually 64 and not 33. I don’t think this is the source of your problem since I expect you would get a different error for that but I looked at the quality scores you provided and based on the 33 vs 64 phred scores, I think yours makes more sense as type 33. I could be wrong but worth double checking. Again, I doubt this is going to solve your pbolem, but a potential problem later:P

1 Like
8
9
11

Yes, I thought that may be the problem to so I tried both 33 and 64 and still received an error.

13

Hello again Stephanie,

I'm glad lots of folks are Qiime-ing in this morning. To help out.

I think I may have found a clue.
42%20AM

What program did you use to make the .csv file? I'm not sure if using Windows lie ending causes a problem, but I know some programs expect Unix line endings instead.

I've uploaded a version of this document using Unix line endings, which might be worth trying.
fastq_files_demux.csv (8.8 KB)

Colin

2 Likes
14

I originally used excel to make the document and then used the command line to edit.

I tried the document you sent me and got the error again. I used Atom to try adding $PWD and see if that would work and it didnt. I also tried both Phred33 and Phred64 for all of these.

Any other ideas? Thanks!

1 Like
15

I think you are doing everything right… I’m not sure what’s going wrong.

As as a guess… what happens if you comment out or remove the sequences that start with O. like O.50064. Capitol Os and .periods. should not cause problems, but I’m just looking for any potential issues.

Qiime devs, is there a script (or internal function), we can use to validate this file? Looks like Keemei does not validate FastqManifest files, but maybe it could!

Colin

16

I tried importing the files after removing the FASTQ files starting with Os from the .csv file and it still gave me the same errors.

17

I solved the issue! Turns out .csv files DO NOT work. You MUST use TestEdit.

2 Likes
18

So saving as a .csv file using Excel doesn’t work, but saving as a .txt file using TextEdit does? That could be really helpful for future users.

Colin

19

If you use Excel to make your CSV, it will insert CRLF line breaks (whether or not you are using windows, I believe). So you need to either just open in textedit and use find/replace to fix the line breaks, or use mac2unix or a program like that to convert the carriage returns.

So Excel is okay — just make sure to convert the line endings.

This will be an issue with making sample metadata files, too, not only manifest files.

1 Like
20
21

I believe @colinbrislawn tried to alleviate the line break issue when he gave me a document to try that he created using Unix line endings. However, this document did not work either.

22

The issue could be other special characters that Excel is importing. In general, your advice to use textedit is valid: Excel creates headaches downstream!