Error with vsearch join-pairs. Too many fastq naming conventions? Too many fastq files?

minardsmitha · April 29, 2019, 4:26pm

I am running Qiime 2019.1.

I have run qiime tools import on my demultiplexed fastq files using this command:

qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' \
--input-path /home/ITS/CBRDC/Qiime \
--input-format CasavaOneEightLanelessPerSampleDirFmt \
--output-path /home/ITS/CBRDC/Qiime/demux-paired-end.qza

Which ended with no errors.

Then I tried to run qiime vsearch join-pairs using this command and got the following error message:

qiime vsearch join-pairs --i-demultiplexed-seqs demux-paired-end.qza --o-joined-sequences demux-joined.qza --p-minovlen 100 --p-maxns 1 --p-minmergelen 75 --verbose
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-da13b51s/6d81d9f5-a327-49c3-835b-fadb846b59c8/data/18S-01-0022-ITS_S330_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-da13b51s/6d81d9f5-a327-49c3-835b-fadb846b59c8/data/18S-01-0022-ITS_S330_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-qd6nydkn/18S-01-0022-ITS_S330_0_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 100 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --fastq_maxns 1 --fastq_minmergelen 75

vsearch v2.7.0_linux_x86_64, 440.6GB RAM, 64 cores

Unable to open file for reading (/tmp/qiime2-archive-da13b51s/6d81d9f5-a327-49c3-835b-fadb846b59c8/data/18S-01-0022-ITS_S330_L001_R1_001.fastq.gz)
Traceback (most recent call last):
File "/hpc/apps/qiime2-2019.1/install/lib/python3.6/site-packages/q2cli/commands.py", line 274, in call
results = action(**arguments)
File "</hpc/apps/qiime2-2019.1/install/lib/python3.6/site-packages/decorator.py:decorator-gen-130>", line 2, in join_pairs
File "/hpc/apps/qiime2-2019.1/install/lib/python3.6/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
File "/hpc/apps/qiime2-2019.1/install/lib/python3.6/site-packages/qiime2/sdk/action.py", line 365, in callable_executor
output_views = self._callable(**view_args)
File "/hpc/apps/qiime2-2019.1/install/lib/python3.6/site-packages/q2_vsearch/_join_pairs.py", line 57, in join_pairs
qmax, qmaxout)
File "/hpc/apps/qiime2-2019.1/install/lib/python3.6/site-packages/q2_vsearch/_join_pairs.py", line 141, in _join_pairs_w_command_output
run_command(cmd)
File "/hpc/apps/qiime2-2019.1/install/lib/python3.6/site-packages/q2_vsearch/_cluster_features.py", line 33, in run_command
subprocess.run(cmd, check=True)
File "/hpc/apps/qiime2-2019.1/install/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['vsearch', '--fastq_mergepairs', '/tmp/qiime2-archive-da13b51s/6d81d9f5-a327-49c3-835b-fadb846b59c8/data/18S-01-0022-ITS_S330_L001_R1_001.fastq.gz', '--reverse', '/tmp/qiime2-archive-da13b51s/6d81d9f5-a327-49c3-835b-fadb846b59c8/data/18S-01-0022-ITS_S330_L001_R2_001.fastq.gz', '--fastqout', '/tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-qd6nydkn/18S-01-0022-ITS_S330_0_L001_R1_001.fastq', '--fastq_ascii', '33', '--fastq_minlen', '1', '--fastq_minovlen', '100', '--fastq_maxdiffs', '10', '--fastq_qmin', '0', '--fastq_qminout', '0', '--fastq_qmax', '41', '--fastq_qmaxout', '41', '--fastq_maxns', '1', '--fastq_minmergelen', '75']' returned non-zero exit status 1.

Plugin error from vsearch:

Command '['vsearch', '--fastq_mergepairs', '/tmp/qiime2-archive-da13b51s/6d81d9f5-a327-49c3-835b-fadb846b59c8/data/18S-01-0022-ITS_S330_L001_R1_001.fastq.gz', '--reverse', '/tmp/qiime2-archive-da13b51s/6d81d9f5-a327-49c3-835b-fadb846b59c8/data/18S-01-0022-ITS_S330_L001_R2_001.fastq.gz', '--fastqout', '/tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-qd6nydkn/18S-01-0022-ITS_S330_0_L001_R1_001.fastq', '--fastq_ascii', '33', '--fastq_minlen', '1', '--fastq_minovlen', '100', '--fastq_maxdiffs', '10', '--fastq_qmin', '0', '--fastq_qminout', '0', '--fastq_qmax', '41', '--fastq_qmaxout', '41', '--fastq_maxns', '1', '--fastq_minmergelen', '75']' returned non-zero exit status 1.

See above for debug info.

I am wondering if there is such as thing as too many fastq files for Qiime in one analysis. I have a MiSeq run with 384 paired end samples.

Or is there a problem with my fastq file names? I have these example file names in my directory of 768 files:

18S-01-0022-ITS_S330_L001_R1_001.fastq.gz
18S-01-0022-ITS_S330_L001_R2_001.fastq.gz
BBB-Plate83WellF5-ITS_S191_L001_R1_001.fastq.gz
BBB-Plate83WellF5-ITS_S191_L001_R2_001.fastq.gz
NTC-A-A1-ITS_S1_L001_R1_001.fastq.gz
NTC-A-A1-ITS_S1_L001_R2_001.fastq.gz
Tube120-ITS_S341_L001_R1_001.fastq.gz
Tube120-ITS_S341_L001_R2_001.fastq.gz

Or maybe it is just a permissions issue on our cluster?

thermokarst · April 30, 2019, 2:05pm

Hello @minardsmitha!

Nope, luckily that isn't something we need to worry about here!

I don't think that is the case either, but I like where you are going with this!

Okay, this line of inquiry seems to be pretty promising to me. Why do I think that? Check out this line from your error message:

If you run qiime tools validate demux-paired-end.qza what do you see? I suspect it will work just fine, but want to double check, first. Also, can you run ls -lah /home/ITS/CBRDC/Qiime and provide the results here?

minardsmitha · May 6, 2019, 2:50pm

Thank you for your reply. I was able to find a solution to my issue. It was the --input-format. When I use --input-format CasavaOneEightSingleLanePerSampleDirFmt , vsearch join-pairs works as expected.

thermokarst · May 6, 2019, 2:52pm

Hi @minardsmitha - thanks for sharing! The input format doesn’t change anything here — all the data gets normalized the same, either way. I suspect what actually happened here is that the process of reimporting sorted out the permission/corrupt data issue, above.