Error Denoising Paired End Reads

User Support
1

I am a novel user of QIIME2-2020, however when following the walk though: Amplicon SOP v2 (qiime2 2020.2) · LangilleLab/microbiome_helper Wiki · GitHub
I receive this error when trying to join my paired end reads.
(qiime2-2020.2) mh_user@MicrobiomeHelper:~$ qiime demux summarize \

--i-data reads_qza/reads_trimmed.qza
--o-visualization reads_qza/reads_trimmed_summary.qzv
Saved Visualization to: reads_qza/reads_trimmed_summary.qzv
(qiime2-2020.2) mh_user@MicrobiomeHelper:~$ qiime vsearch join-pairs
--i-demultiplexed-seqs reads_qza/reads_trimmed.qza
--o-joined-sequences reads_qza/reads_trimmed_joined.qza
Plugin error from vsearch:

Command '['vsearch', '--fastq_mergepairs', '/tmp/qiime2-archive-6er9jrh6/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W620-B_S226_L001_R1_001.fastq.gz', '--reverse', '/tmp/qiime2-archive-6er9jrh6/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W620-B_S226_L001_R2_001.fastq.gz', '--fastqout', '/tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-nvkoov5k/W620-B_9_L001_R1_001.fastq', '--fastq_ascii', '33', '--fastq_minlen', '1', '--fastq_minovlen', '10', '--fastq_maxdiffs', '10', '--fastq_qmin', '0', '--fastq_qminout', '0', '--fastq_qmax', '41', '--fastq_qmaxout', '41', '--threads', '1']' returned non-zero exit status 1.

Debug info has been saved to /tmp/qiime2-q2cli-err-gvh8rpmd.log

If anyone could help me out, it would be greatly appreciated.

Best,
Taylor

1 Like
2

Hello Taylor,

Welcome to the forums! :qiime2:

Can you rerun that command and add in the --verbose flag? That will give us more details about what went wrong and give us clues about how to fix it!

Colin

3

Hey Colin!

If I did what you asked correctly, here is the output:
There were some problems with the command:
(1/3) Missing option "--i-demultiplexed-seqs".
(2/3) Missing option "--o-joined-sequences". ("--output-dir" may also be
used)
(3/3) Got unexpected extra arguments (--i-demultiplexed-seqs
reads_qza/reads_trimmed.qza --o-joined-sequences
reads_qza/reads_trimmed_joined.qza --verbose)

I stopped working yesterday when I recieved this error but left MicroBiome helper running over night. I'm unfamiliar with the program so I am unsure if that would effect the result. My guess is it doesn't- but thought I'd mention it just incase aha.

Thank you very much for trying to help me out!
Cheers,
Taylor

1 Like
4

Hey Taylor,

This is a different issues this time, but this one at least I know how to fix.

You may notice that the command in the tutorial looks like this:
--i-demultiplexed-seqs
but your command looks like this
“–i-demultiplexed-seqs”

Word and some other programs replace "normal quotes" with “fancy quotes” and
normal -dashes- with these –em–dashes–

You have got to copy paste from the tutorial or from a text editor (not a word processor).

Here's the command should look

qiime vsearch join-pairs --verbose \
   --i-demultiplexed-seqs reads_qza/reads_trimmed.qza \
   --o-joined-sequences reads_qza/reads_trimmed_joined.qza

(That should be safe to copy paste :+1:

Colin

5

Hi Colin,

That’s odd as I was copying straight from the tutorial and when I ran the command I didn’t think I added “” around the command, however I re-ran the code you posted with the --verbose command in the correct place and got this output: Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W402-B_S203_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W402-B_S203_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W402-B_0_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 10 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --threads 1

Merging reads 100%
96 Pairs
80 Merged (83.3%)
16 Not merged (16.7%)

Pairs that failed merging due to various reasons:
11 too many differences
5 alignment score too low, or score drop to high

Statistics of all reads:
283.34 Mean read length

Statistics of merged reads:
404.09 Mean fragment length
3.33 Standard deviation of fragment length
0.59 Mean expected error in forward sequences
3.00 Mean expected error in reverse sequences
0.34 Mean expected error in merged sequences
0.57 Mean observed errors in merged region of forward sequences
2.65 Mean observed errors in merged region of reverse sequences
3.23 Mean observed errors in merged region
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: gzip /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W402-B_0_L001_R1_001.fastq

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W402-M_S286_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W402-M_S286_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W402-M_1_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 10 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --threads 1

Merging reads 100%
31 Pairs
24 Merged (77.4%)
7 Not merged (22.6%)

Pairs that failed merging due to various reasons:
5 too many differences
2 alignment score too low, or score drop to high

Statistics of all reads:
283.45 Mean read length

Statistics of merged reads:
403.21 Mean fragment length
1.85 Standard deviation of fragment length
0.20 Mean expected error in forward sequences
2.34 Mean expected error in reverse sequences
0.23 Mean expected error in merged sequences
0.08 Mean observed errors in merged region of forward sequences
2.17 Mean observed errors in merged region of reverse sequences
2.25 Mean observed errors in merged region
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: gzip /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W402-M_1_L001_R1_001.fastq

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W402-S_S274_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W402-S_S274_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W402-S_2_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 10 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --threads 1

Merging reads 100%
24 Pairs
18 Merged (75.0%)
6 Not merged (25.0%)

Pairs that failed merging due to various reasons:
5 too many differences
1 alignment score too low, or score drop to high

Statistics of all reads:
283.40 Mean read length

Statistics of merged reads:
402.00 Mean fragment length
3.27 Standard deviation of fragment length
0.48 Mean expected error in forward sequences
2.21 Mean expected error in reverse sequences
0.22 Mean expected error in merged sequences
0.44 Mean observed errors in merged region of forward sequences
1.61 Mean observed errors in merged region of reverse sequences
2.06 Mean observed errors in merged region
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: gzip /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W402-S_2_L001_R1_001.fastq

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W410-B_S239_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W410-B_S239_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W410-B_3_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 10 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --threads 1

Merging reads 100%
73 Pairs
56 Merged (76.7%)
17 Not merged (23.3%)

Pairs that failed merging due to various reasons:
11 too many differences
6 alignment score too low, or score drop to high

Statistics of all reads:
283.47 Mean read length

Statistics of merged reads:
403.86 Mean fragment length
1.56 Standard deviation of fragment length
0.46 Mean expected error in forward sequences
3.11 Mean expected error in reverse sequences
0.27 Mean expected error in merged sequences
0.43 Mean observed errors in merged region of forward sequences
2.75 Mean observed errors in merged region of reverse sequences
3.18 Mean observed errors in merged region
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: gzip /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W410-B_3_L001_R1_001.fastq

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W410-M_S215_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W410-M_S215_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W410-M_4_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 10 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --threads 1

Merging reads 100%
1217 Pairs
1026 Merged (84.3%)
191 Not merged (15.7%)

Pairs that failed merging due to various reasons:
1 too few kmers found on same diagonal
126 too many differences
64 alignment score too low, or score drop to high

Statistics of all reads:
283.66 Mean read length

Statistics of merged reads:
404.08 Mean fragment length
1.80 Standard deviation of fragment length
0.44 Mean expected error in forward sequences
2.82 Mean expected error in reverse sequences
0.23 Mean expected error in merged sequences
0.28 Mean observed errors in merged region of forward sequences
2.40 Mean observed errors in merged region of reverse sequences
2.68 Mean observed errors in merged region
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: gzip /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W410-M_4_L001_R1_001.fastq

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W410-S_S227_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W410-S_S227_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W410-S_5_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 10 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --threads 1

Merging reads 100%
438 Pairs
382 Merged (87.2%)
56 Not merged (12.8%)

Pairs that failed merging due to various reasons:
40 too many differences
16 alignment score too low, or score drop to high

Statistics of all reads:
283.64 Mean read length

Statistics of merged reads:
404.08 Mean fragment length
1.71 Standard deviation of fragment length
0.37 Mean expected error in forward sequences
2.93 Mean expected error in reverse sequences
0.25 Mean expected error in merged sequences
0.24 Mean observed errors in merged region of forward sequences
2.44 Mean observed errors in merged region of reverse sequences
2.68 Mean observed errors in merged region
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: gzip /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W410-S_5_L001_R1_001.fastq

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W611-B_S262_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W611-B_S262_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W611-B_6_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 10 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --threads 1

Merging reads 100%
34 Pairs
24 Merged (70.6%)
10 Not merged (29.4%)

Pairs that failed merging due to various reasons:
7 too many differences
3 alignment score too low, or score drop to high

Statistics of all reads:
283.35 Mean read length

Statistics of merged reads:
405.62 Mean fragment length
2.43 Standard deviation of fragment length
0.46 Mean expected error in forward sequences
3.43 Mean expected error in reverse sequences
0.28 Mean expected error in merged sequences
0.46 Mean observed errors in merged region of forward sequences
3.00 Mean observed errors in merged region of reverse sequences
3.46 Mean observed errors in merged region
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: gzip /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W611-B_6_L001_R1_001.fastq

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W611-M_S250_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W611-M_S250_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W611-M_7_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 10 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --threads 1

Merging reads 100%
242 Pairs
202 Merged (83.5%)
40 Not merged (16.5%)

Pairs that failed merging due to various reasons:
28 too many differences
12 alignment score too low, or score drop to high

Statistics of all reads:
283.81 Mean read length

Statistics of merged reads:
404.89 Mean fragment length
1.41 Standard deviation of fragment length
0.40 Mean expected error in forward sequences
2.89 Mean expected error in reverse sequences
0.23 Mean expected error in merged sequences
0.21 Mean observed errors in merged region of forward sequences
2.60 Mean observed errors in merged region of reverse sequences
2.81 Mean observed errors in merged region
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: gzip /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W611-M_7_L001_R1_001.fastq

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W611-S_S238_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W611-S_S238_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W611-S_8_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 10 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --threads 1

Merging reads 100%
2 Pairs
2 Merged (100.0%)
0 Not merged (0.0%)

Statistics of all reads:
283.75 Mean read length

Statistics of merged reads:
402.50 Mean fragment length
1.50 Standard deviation of fragment length
0.38 Mean expected error in forward sequences
2.38 Mean expected error in reverse sequences
0.11 Mean expected error in merged sequences
0.00 Mean observed errors in merged region of forward sequences
3.00 Mean observed errors in merged region of reverse sequences
3.00 Mean observed errors in merged region
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: gzip /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W611-S_8_L001_R1_001.fastq

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --fastq_mergepairs /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W620-B_S226_L001_R1_001.fastq.gz --reverse /tmp/qiime2-archive-fzmwzy18/1303fd9e-77a4-439f-94f9-b58c1e9f9b6a/data/W620-B_S226_L001_R2_001.fastq.gz --fastqout /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-6irs0mnf/W620-B_9_L001_R1_001.fastq --fastq_ascii 33 --fastq_minlen 1 --fastq_minovlen 10 --fastq_maxdiffs 10 --fastq_qmin 0 --fastq_qminout 0 --fastq_qmax 41 --fastq_qmaxout 41 --threads 1

Fatal error: File too small

Hopefully this will provide more info!

Thanks for all your help!

Cheers,
Taylor

1 Like
6

Thanks Taylor,

Now that I can see the full output, we finally found the error!

Well that explains it! :scream_cat:
How big is W620-B_S226_L001_R1_001.fastq.gz?

You might have to drop very small files before starting joining. :man_shrugging: :woman_shrugging:

Colin

7

Hi Colin!

THANK YOU for all your advice. I ended up using the dada2 pipeline and was able to continue (I imagine because I set specific cut offs).

All the best,
Taylor

1 Like
8

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.