reference file in manifest could not be found

[Sina]

Hi.
I’m trying to import a set of forward and reverse files with the following command:

PS C:\Users\hzq128> docker run -t -i -v ${pwd}:/data qiime2/core:2020.6 qiime tools import --type ‘SampleData[PairedEndSequencesWithQuality]’ --input-path manifest.txt --output-path paired.gza --input-format PairedEndFastqManifestPhred33

I’m using docker as the Virtualbox, and I think I’m using the latest version of Qiime2.
however, I get the following error:

manifest.txt is not a(n) PairedEndFastqManifestPhred33 file:
File referenced on line 2 could not be found (C:\Users\hzq128\s-13_R1.fastq.gz).

the manifest file is in the same directory (C:\Users\hzq128) with the .fastq.gz files.

and here’s is my manifest file:

sample-id,absolute-filepath,direction
s-13,C:\Users\hzq128\s-13_R1.fastq.gz,forward
s-13,C:\Users\hzq128\s-13_R2.fastq.gz,reverse

I looked into similar issues. I’m using the absolute path and have the files in the same folder. I don’t know what I’m missing since qiime can’t find the reference file.

[the_dummy]

Hello @Sina ,

@Sina

manifest.txt is not a(n) PairedEndFastqManifestPhred33 file

I believe this is about file format.

Here is how I make my import manifest.

sample-id,forward-absolute-filepath,reverse-absolute-filepath

Input format is PairedEndFastqManifestPhred33V2 and also type is SampleData[PairedEndSequencesWithQuality].

I have no experience with virtual box. Maybe the error is related to that, I don’t know. But you can try making your manifest as I mentioned. Maybe that would work.

Best wishes.

[thermokarst (Matthew Ryan Dillon)]

Hi @Sina.

The format you specified (PairedEndFastqManifestPhred33) is fine, there is no need to change that.

The issue is this:

@Sina

File referenced on line 2 could not be found (C:\Users\hzq128\s-13_R1.fastq.gz).

Know why it can’t be found? Because QIIME 2 is running in a docker container (according to your first command shared), which uses an entirely different file system.

@Sina

the manifest file is in the same directory (C:\Users\hzq128) with the .fastq.gz files

Good, that means you have an easy workaround:

sample-id,absolute-filepath,direction
s-13,$PWD/s-13_R1.fastq.gz,forward
s-13,$PWD/s-13_R2.fastq.gz,reverse

The $PWD is a linux environment variable that contains the absolute path to the current working directory - this should work for you - let us know how it goes.

[Sina]

Thank you Matthew.

I'm still having the same issue (attached image).
I'm using PowerShell could that be a problem?

this is the manifest file. manifest.txt (330 Bytes)

Thanks,
Sina.

[thermokarst (Matthew Ryan Dillon)]

@Sina

I’m using PowerShell could that be a problem?

No.

sample-id,absolute-filepath,direction
s-13,$PWD/s-13_R1.fastq.gz,forward
s-13,$PWD/s-13_R2.fastq.gz,reverse

Please note the direction of the slashes.

[Sina]

manifest.txt is not a(n) PairedEndFastqManifestPhred33 file:

File referenced on line 2 could not be found (PWD/s-13_R1.fastq.gz).

Maybe I should try VirtualBox or Amazon Web Services.
I’ll try to see if that fixes the problem.

Thanks,
Sina.

[thermokarst (Matthew Ryan Dillon)]

@Sina

File referenced on line 2 could not be found (PWD/s-13_R1.fastq.gz).

You’re missing the $ before PWD.

@thermokarst

sample-id,absolute-filepath,direction
s-13,$PWD/s-13_R1.fastq.gz,forward
s-13,$PWD/s-13_R2.fastq.gz,reverse

[Sina]

There was a problem importing manifest.txt:

  manifest.txt is not a(n) PairedEndFastqManifestPhred33 file:

  File referenced on line 2 could not be found ($PWD/s-13_R1.fastq.gz).

PS C:\Users\hzq128> docker run -t -i -v ${pwd}:/data qiime2/core:2020.6 qiime tools import  --type 'SampleData[PairedEndSequencesWithQuality]'  --input-path manifest.txt  --output-path paired.gza  --input-format PairedEndFastqManifestPhred33
There was a problem importing manifest.txt:

  manifest.txt is not a(n) PairedEndFastqManifestPhred33 file:

  File referenced on line 2 could not be found ($pwd/s-13_R1.fastq.gz).

It’s the same thing again!

[thermokarst (Matthew Ryan Dillon)]

@Sina

$pwd/s-13_R1.fastq.gz

Again, you need to use $PWD, not $pwd, or PWD, or any other combo - the env var is pretty specific.

Let’s take a step back though - let’s confirm your files are all loaded in docker correctly:

docker run -t -i -v ${pwd}:/data qiime2/core:2020.6 ls -lah

Please copy-and-paste the results here. Put the results in a triple-backtick block to format correctly:

```
PASTE HERE
```

[Sina]

okay. here is what I see.

total 988M
drwxrwxrwx 1 root root  20K Aug  4 17:06  .
drwxr-xr-x 1 root root 4.0K Aug  4 17:28  ..
drwxrwxrwx 1 root root 4.0K Aug  4 17:06  .VirtualBox
drwxrwxrwx 1 root root    0 Jul 31 16:05  .conda
drwxrwxrwx 1 root root 4.0K Jul 20 22:24  .docker
drwxrwxrwx 1 root root    0 Jan 18  2020  .matplotlib
-rwxr-xr-x 1 root root 3.7K Aug  3 15:26  .test.tsv.un~
-rwxr-xr-x 1 root root 5.9K Aug  3 15:31  .test.txt.un~
drwxrwxrwx 1 root root    0 Jul  2 19:52 '3D Objects'
drwxrwxrwx 1 root root    0 Jun 13  2019  AppData
drwxrwxrwx 1 root root    0 Nov 12  2019  BentleyDownloads
drwxrwxrwx 1 root root    0 Jul  2 19:52  Contacts
drwxrwxrwx 1 root root  48K Aug  4 17:24  Desktop
-rwxr-xr-x 1 root root 5.9M Aug  4 17:17  DesktopActivator.log
-rwxr-xr-x 1 root root 9.8M Feb  1  2020  DesktopActivator.log.1
drwxrwxrwx 1 root root 4.0K Jul 20 16:29  Documents
drwxrwxrwx 1 root root  96K Aug  4 17:24  Downloads
drwxrwxrwx 1 root root    0 Jul  2 19:52  Favorites
drwxrwxrwx 1 root root    0 Aug  3 17:32  Harvey
drwxrwxrwx 1 root root    0 Jul  2 19:52  Links
drwxrwxrwx 1 root root    0 Jun 13  2019  MicrosoftEdgeBackups
drwxrwxrwx 1 root root    0 Jul 22 16:18  Music
drwxrwxrwx 1 root root    0 Jun 14  2019  ODBA
drwxrwxrwx 1 root root    0 Jun 13  2019  OneDrive
drwxrwxrwx 1 root root 4.0K Jul  2 19:52  Pictures
drwxrwxrwx 1 root root    0 Jul  2 19:52 'Saved Games'
drwxrwxrwx 1 root root 4.0K Jul  2 19:52  Searches
drwxrwxrwx 1 root root 4.0K Jul 30 16:54 'University of Texas at San Antonio'
-rwxr-xr-x 1 root root 370M Sep 26  2018  VA_R1.fastq
-rwxr-xr-x 1 root root 369M Sep 26  2018  VA_R2.fastq
drwxrwxrwx 1 root root    0 Jul  2 19:52  Videos
-rwxr-xr-x 1 root root 1.9K Aug  3 16:26  _viminfo
-rwxr-xr-x 1 root root    0 Jul 31 16:20  anaconda
-rwxr-xr-x 1 root root  145 Aug  3 17:32  manifest.csv
-rwxr-xr-x 1 root root  332 Aug  4 17:20  manifest.txt
-rwxr-xr-x 1 root root 101M Sep 25  2018  s-13_R1.fastq.gz
-rwxr-xr-x 1 root root 133M Sep 25  2018  s-13_R2.fastq.gz
-rwxr-xr-x 1 root root  130 Aug  3 15:26  test.tsv
-rwxr-xr-x 1 root root  119 Aug  3 23:40  test.txt
-rwxr-xr-x 1 root root  129 Aug  3 15:30  test.txt~
-rwxr-xr-x 1 root root  278 Aug  2 19:21  testManifest.txt
drwxrwxrwx 1 root root 4.0K Aug  3 15:27  vimfiles

Thank you again for walking me through this process.

[Sina]

Hi again,
I switched to VirtualBox and I was able to import my file and it looks easier to work with. thank you for your help and comments.