Issue importing Paired end sequences with quality

[lautaro.rostoll]

Hello Everyone,
I am trying to import Paired end sequences with quality using a manifest (attached here an example with just a few files. Manifest.txt (547 Bytes)
I am using putty to access the server from my university and run qiime with conda.
The files are in a folder within the server and the path to the folder where the fastq files are is: home/lrostoll/qiime2/neo
When I try to import the files with this code:
qiime tools import
--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path Manifest.txt
--output-path demux-paired-end.qza
--input-format PairedEndFastqManifestPhred33V2

I get this error: Manifest.txt is not a(n) PairedEndFastqManifestPhred33V2 file:

Filepath on line 1 and column "forward-absolute-filepath" could not be found ($home/lrostoll/qiime2/neo/155_S15_L001_R1_001.fastq.gz) for sample "1-17-IL-C".

I would really appreciate some help with this

thanks,

Lautaro

[colinbrislawn (Colin J Brislawn)]

Hello Lautaro,

I took a look at your manifest file, and it looks good, except for the line endings.

(Windows and Linux use different ways to note a
new
line
of
text.)

Once you convert to the Linux format by saving it with a text editor or using a command like
dos2unix Manifest.txt
then this should work great!

You are not the first to have this issue! More info here:

FASTQ Files Importing Error

If you use Excel to make your CSV, it will insert CRLF line breaks (whether or not you are using windows, I believe). So you need to either just open in textedit and use find/replace to fix the line breaks, or use mac2unix or a program like that to convert the carriage returns. So Excel is okay — just make sure to convert the line endings. This will be an issue with making sample metadata files, too, not only manifest files.

[lautaro.rostoll]

@colinbrislawn

dos2unix Manifest.txt

Hi Colin,
thanks so much for your response. I tried converting the file with the dos2unix command and it was converted successfully. However, I still get the same error message.
Manifest.txt (542 Bytes)
Here is the new manifest file.
Is there anything else that could be happening?

thanks,

Lautaro

[colinbrislawn (Colin J Brislawn)]

Hello Lautaro,

I have made a mistake! I missed something important in your original message:

@lautaro.rostoll

Filepath on line 1 and column "forward-absolute-filepath" could not be found ($home/lrostoll/qiime2/neo/155_S15_L001_R1_001.fastq.gz) for sample "1-17-IL-C".

This shows an issue with finding the file, not with line endings.

Double-check that file path and see if you can get it pointing to the right file. (For example, on my computer, $HOME points to my home directory, but $home is not set at all!)

Let us know how it goes,
Colin

[lautaro.rostoll]

Hi Colin,
I tried changing $home/ to $HOME/ and it didn't work either.
Here is my file path: /home/lrostoll/qiime2/neo
I tried doing HOME/lrostoll/qiime2/neo, and /HOME/lrostoll/qiime2/neo, and also the same with home in lower case. non of them seems to work.
Here is the manifest again:
Manifest.txt (547 Bytes)

And here is the error I am getting:
There was a problem importing Manifest.txt:

Manifest.txt is not a(n) PairedEndFastqManifestPhred64V2 file:

Filepath on line 1 and column "forward-absolute-filepath" could not be found ($HOME/lrostoll/qiime2/neo/155_S15_L001_R1_001.fastq.gz) for sample "1-17-IL-C".

Any ideas of what I am doing wrong?

thanks!

Lautaro

[lautaro.rostoll]

I also tried changing to Phred64V2 and also didn't work.
I also made sure that the format was UTF-8 without BOM and used a text editor to double check that.
I have no idea why it isn't working

[colinbrislawn (Colin J Brislawn)]

@lautaro.rostoll

I have no idea why it isn't working

Well, the good news is that we are still dealing with the same error. (So nothing spooky is going on )

@lautaro.rostoll

Filepath on line 1 and column "forward-absolute-filepath" could not be found

Let's take a look in these folders and see what's inside of them. Run each of these commands and post the full output:

cd $HOME; pwd

ls -lah $HOME

ls -lah $HOME/qiime2/neo/

ls -lah $HOME/lrostoll/qiime2/neo/

[lautaro.rostoll]

Here it is:

@colinbrislawn

ls -lah $HOME

(qiime2-2021.2) lrostoll@signal[11:43][~]$ ls -lah $HOME
total 91M
drwx--x--x  19 lrostoll lrostoll 4.0K Mar 15  2021 .
drwxr-xr-x 528 root     root      16K Nov  2 11:34 ..
-rw-r--r--   1 lrostoll lrostoll   18 Oct 30  2018 .bash_logout
-rw-r--r--   1 lrostoll lrostoll  193 Oct 30  2018 .bash_profile
-rw-r--r--   1 lrostoll lrostoll  919 Jan 27  2021 .bashrc
-rw-r-----   1 lrostoll lrostoll    0 Oct 25  2019 build
drwxrwxr-x   5 lrostoll lrostoll   63 Jan 27  2021 .cache
-rw-r-----   1 lrostoll lrostoll  19K Oct  5  2019 comp1.cpp
-rw-r-----   1 lrostoll lrostoll  402 Oct  5  2019 comp1.csi
-rw-r-----   1 lrostoll lrostoll  419 Oct  5  2019 comp1.csl
-rw-r-----   1 lrostoll lrostoll 1016 Oct  5  2019 comp1.rrr
drwxr-x---   2 lrostoll lrostoll   38 Jan 26  2021 .conda
drwxrwxr-x   5 lrostoll lrostoll   71 Jan 27  2021 .config
-rw-r-----   1 lrostoll lrostoll  301 Oct 25  2019 consoleError.log
-rw-r-----   1 lrostoll lrostoll  424 Oct 25  2019 consoleStatus.log
-rw-r-----   1 lrostoll lrostoll  647 Oct 25  2019 cori1
-rw-r-----   1 lrostoll lrostoll  694 Oct 25  2019 cori1.csi
-rw-r-----   1 lrostoll lrostoll  647 Oct 21  2019 cori1.csl
drwxr-x---   2 lrostoll lrostoll   83 Jan 27  2021 .electron
-rw-r--r--   1 lrostoll lrostoll  334 Oct 30  2018 .emacs
drwx------   3 lrostoll lrostoll   36 Sep 30  2019 .emacs.d
-rw-------   1 lrostoll lrostoll 111K Apr 22  2021 .history_eramosa
-rw-------   1 lrostoll lrostoll 119K Nov  6 17:03 .history_signal.aps.uoguelph.ca
-rw-------   1 lrostoll lrostoll  475 Oct 21  2019 .InterpConsole_history
drwxr-x---   2 lrostoll lrostoll   32 Feb 22  2021 .keras
drwx------   3 lrostoll lrostoll   27 Sep 30  2019 .local
-rw-r-----   1 lrostoll lrostoll   65 Oct 21  2019 Makefile
-rw-r-----   1 lrostoll lrostoll   66 Oct 21  2019 Makefile~
drwxr-x---  16 lrostoll lrostoll  302 Jan 26  2021 miniconda3
-rw-r-----   1 lrostoll lrostoll  90M Nov 23  2020 Miniconda3-latest-Linux-x86_64.sh
drwxr-xr-x   4 lrostoll lrostoll   51 Jul 20  2017 .mozilla
drwxr-x---   4 lrostoll lrostoll  119 Jan 27  2021 .npm
drwxr-x---   8 lrostoll lrostoll 4.0K Jan 27  2021 .nvm
drwxr-----   3 lrostoll lrostoll   27 Jan 27  2021 .pki
drwxr-s---   2 lrostoll apache     32 Sep 26  2019 public_html
drwxr-x---   5 lrostoll lrostoll  100 Feb 19  2021 q2studio-2020.11.0
drwxr-x---  12 lrostoll lrostoll 8.0K Nov  4 09:45 qiime2
-rw-r-----   1 lrostoll lrostoll 9.3K Jan 26  2021 qiime2-2020.11-py36-linux-conda.yml
-rw-r-----   1 lrostoll lrostoll 9.4K Mar 15  2021 qiime2-2021.2-py36-linux-conda.yml
drwxr-x---   4 lrostoll lrostoll 4.0K Jan 28  2021 qiime2-moving-pictures-tutorial
drwx------   2 lrostoll lrostoll  100 Jan 26  2021 .ssh
-rw-r--r--   1 lrostoll lrostoll 1.5K Nov 19  2004 .Xdefaults
-rw-r--r--   1 lrostoll lrostoll  607 Nov 19  2004 .Xresources

**ls -lah $HOME/qiime2/neo/**
(qiime2-2021.2) lrostoll@signal[11:44][~]$ ls -lah $HOME/qiime2/neo/
total 933M
drwxr-x---  2 lrostoll lrostoll 4.0K Nov  6 13:19 .
drwxr-x--- 12 lrostoll lrostoll 8.0K Nov  4 09:45 ..
-rw-r-----  1 lrostoll lrostoll  80M Nov  1 14:35 155_S15_L001_R1_001.fastq
-rw-r-----  1 lrostoll lrostoll  80M Nov  1 14:35 155_S15_L001_R2_001.fastq
-rw-r-----  1 lrostoll lrostoll 155M Nov  1 14:36 157_S16_L001_R1_001.fastq
-rw-r-----  1 lrostoll lrostoll 155M Nov  1 14:36 157_S16_L001_R2_001.fastq
-rw-r-----  1 lrostoll lrostoll 119M Nov  1 14:35 158_S17_L001_R1_001.fastq
-rw-r-----  1 lrostoll lrostoll 119M Nov  1 14:35 158_S17_L001_R2_001.fastq
-rw-r-----  1 lrostoll lrostoll 115M Nov  1 14:35 159_S18_L001_R1_001.fastq
-rw-r-----  1 lrostoll lrostoll 114M Nov  1 14:35 159_S18_L001_R2_001.fastq
-rw-r-----  1 lrostoll lrostoll  547 Nov  4 13:52 Manifest.csv
-rw-r-----  1 lrostoll lrostoll  542 Nov  6 16:51 Manifest.txt

**ls -lah $HOME/lrostoll/qiime2/neo/**
(qiime2-2021.2) lrostoll@signal[11:45][~]$ ls -lah $HOME/lrostoll/qiime2/neo/
ls: cannot access /home/lrostoll/lrostoll/qiime2/neo/: No such file or directory

[lautaro.rostoll]

I just try again using this path in the manifest: $HOME/qiime2/neo/ and still get the same error:

There was a problem importing Manifest.txt:

Manifest.txt is not a(n) PairedEndFastqManifestPhred33V2 file:

Filepath on line 1 and column "forward-absolute-filepath" could not be found ($HOME/qiime2/neo/155_S15_L001_R1_001.fastq.gz) for sample "1-17-IL-C".

Here is the manifest:
Manifest.txt (470 Bytes)

When I open on a note editor it tells me that the encoding is: 1252 (ANSI - Latin I)
Should it be UTF-8? Can this be the problem?

[colinbrislawn (Colin J Brislawn)]

We are so close!

First, we now have the correct path to the folder: $HOME/qiime2/neo/

@lautaro.rostoll

could not be found ($HOME/qiime2/neo/155_S15_L001_R1_001.fastq.gz

Well that makes sense; as we saw above, your files in that folder end with .fastq not .fastq.gz.

Try taking out the .gz from your manifest, and try it one more time!

[lautaro.rostoll]

IT WORKED!!!!
I removed the .gz and it worked.
Thanks so much Colin!