Quality-filter paired end illumina

I am trying quality-filter paired end illumina reads.
The artefact paired-end-demux.qza was obtained as “Fastq manifest” formats, using:

qiime tools import \
  --type 'SampleData[PairedEndSequencesWithQuality]' \
  --input-path ely-manifiest.txt \
  --output-path paired-end-demux.qza \
  --source-format PairedEndFastqManifestPhred33

The resulting qza can be visualized using

qiime demux summarize --i-data paired-end-demux.qza --o-visualization demux.qzv AND qiime tools view demux.qzv 

However using the command:

qiime quality-filter q-score \
 --i-demux paired-end-demux.qza \
 --o-filtered-sequences demux-filtered.qza \
 --o-filter-stats demux-filter-stats.qza

I get this output:

Plugin error from quality-filter:
  Argument to parameter 'demux' is not a subtype of SampleData[SequencesWithQuality].
Debug info has been saved to /tmp/qiime2-q2cli-err-7xnt_6cc.log

What am I doing wrong?
Thanks in advances for any helpful information

Hi @fvalbores! Thanks for the detailed summary!

The ability to provide a SampleData[PairedEndSequencesWithQuality] as input to quality-filter q-score was added in QIIME 2 2017.12 - this error is consistent with trying to provide this type of input with an older version of QIIME 2. You should be able to upgrade to the latest version of QIIME 2 and attempt to re-run quality-filter q-score (you shouldn’t need to reimport your data) with your paired-end reads.

Let us know if that works for you! Thanks! :t_rex:

Thank you very much, qiime quality-filter works when the new version is used.
Now the problem is the next step: Sequence quality control and feature table construction.
Using:

qiime dada2 denoise-paired \
  --i-demultiplexed-seqs paired-end-demux.qza \
  --p-trunc-len-f 0 \
  --p-trunc-len-r 0 \
  --o-table table \
  --o-representative-sequences rep-seqs

I get this output:

Plugin error from dada2:
  An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
Debug info has been saved to /tmp/qiime2-q2cli-err-rdens6iu.log

or using:

qiime deblur denoise-16S \
  --i-demultiplexed-seqs paired-end-demux.qza \
  --p-trim-length -1 \
  --o-representative-sequences rep-seqs-deblur.qza \
  --o-table table-deblur.qza \
  --p-sample-stats \
  --o-stats deblur-stats.qza

it is displayed:

Plugin error from deblur:
  No sequences passed the filter. It is possible the trim_length (%d) may exceed the longest sequence, that all of the sequences are artifacts like PhiX or adapter, or that the positive reference used is not representative of the data being denoised.
Debug info has been saved to /tmp/qiime2-q2cli-err-hctf18rg.log

Could you help me? I am using Qiime2 2017.12
Thanks in advance

Hi @fvalbores!

Excellent!

Are you attempting to use your quality-filtereded sequences now? If you are still using your filenames from above, it looks like you aren’t - just wanted to double check with you what your intentions are (paired-end-demux.qza vs demux-filtered.qza.

DADA2
Generally speaking, we don’t recommend quality-filtering reads prior to processing in DADA2, because DADA2 performs error correction prior to filtering reads. If you want to use DADA2, I would use your imported sequences (paired-end-demux.qza) and run those right into q2-dada2. If you did that, please let me know (see my note above about your filenames).

deblur
If you were accidentally using your unfiltered sequences with q2-deblur (see my note above about filenames), that is likely the problem, and you just need to update your command to use the correct artifact. If the filenames are in fact correct, and you are using your quality filtered sequences as input to q2-deblur, I would suggest running demux summarize on the filtered sequences, and sharing that here.

Thanks! :t_rex:

From my data, the artifact paired-end-demux.qza was obtained as “Fastq
manifest” formats, using:
qiime tools import
–type ‘SampleData [PairedEndSequencesWithQuality]’
–input-path ely-manifiest.txt
–output-path paired-end-demux.qza
–source-format PairedEndFastqManifestPhred33

Subsequently and following your recommendation demux-filtered.qza and
demux-filter-stats.qza were obtained using the 2017.12 version.

Good, but now how can I continue? Following the tutorials the next step is:
“Sequence quality control and feature table construction”.
You said: “Generally speaking, we don’t recommend quality-filtering reads
prior to processing in DADA2, because DADA2 performs error correction prior
to filtering reads. If you want to use DADA2, I would use your imported
sequences (paired-end-demux.qza) and run those right into q2-dada2. If you
did that, please let me know (see my note above about your filenames).”

Dada2 was run using the artifact paired-end-demux.qza and I had an error.
As you indicated to me (“Generally speaking, we don’t recommend
quality-filtering reads prior to processing in DADA2, because DADA2
performs error correction prior to filtering reads. If you want to use
DADA2, I would use your imported sequences (paired-end-demux.qza) and run
those right into q2-dada2.”), I ran Dada2 with the artefact
demux-filtered.qza and it did not work either.

How can I generate the “table.qza”? to continue with “FeatureTable and
FeatureData summaries” and subsequent steps for the Alpha and beta
diversity analysis?. In addition, in the tutorials the metadata-file is required and I do
not have it. I only have the filtered, demultiplexed and clean (whitout
primers and barcodes) sequences
I appreciate all the help you are giving me …
regards

Hi @fvalbores,

sorry, it looks like there was a misunderstanding, I was recommending that you not use demux-filtered.qza as an input to dada2 denoise-paired, for the reasons I mentioned above.

Can you please post the exact command you ran (copy and paste), and the exact error (copy and paste) when run with the --verbose flag?

Using the command:
qiime dada2 denoise-paired
–i-demultiplexed-seqs paired-end-demux.qza
–verbose
–p-trunc-len-f 0
–p-trunc-len-r 0
–o-table table
–o-representative-sequences rep-seqs

these messages is displayed:

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /tmp/tmp8x0a5jkp/forward /tmp/tmp8x0a5jkp/reverse /tmp/tmp8x0a5jkp/output.tsv.biom /tmp/tmp8x0a5jkp/filt_f /tmp/tmp8x0a5jkp/filt_r 0 0 0 0 2.0 2 consensus 1.0 1 1000000

R version 3.4.1 (2017-06-30)
Loading required package: Rcpp
DADA2 R package version: 1.6.0

  1. Filtering Error in add(bin) : internal: buf !=
    Execution halted
    Traceback (most recent call last):
    File “/home/paco/anaconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 179, in denoise_paired
    run_commands([cmd])
    File “/home/paco/anaconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 35, in run_commands
    subprocess.run(cmd, check=True)
    File “/home/paco/anaconda2/envs/qiime2-2017.12/lib/python3.5/subprocess.py”, line 398, in run
    output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command ‘[‘run_dada_paired.R’, ‘/tmp/tmp8x0a5jkp/forward’, ‘/tmp/tmp8x0a5jkp/reverse’, ‘/tmp/tmp8x0a5jkp/output.tsv.biom’, ‘/tmp/tmp8x0a5jkp/filt_f’, ‘/tmp/tmp8x0a5jkp/filt_r’, ‘0’, ‘0’, ‘0’, ‘0’, ‘2.0’, ‘2’, ‘consensus’, ‘1.0’, ‘1’, ‘1000000’]’ returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/paco/anaconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/q2cli/commands.py”, line 224, in call
results = action(**arguments)
File “”, line 2, in denoise_paired
File “/home/paco/anaconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 228, in bound_callable
output_types, provenance)
File “/home/paco/anaconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 363, in callable_executor
output_views = self._callable(**view_args)
File “/home/paco/anaconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 194, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

See above for debug info.

I hope this allows you to help me in the process of analyzing my data. I look forward to your kind response
Regards & Happy New Year

Yikes! I have never seen that error before! Would you be able to share paired-end-demux.qza? If it is small enough, you could post here or in a direct message to me, otherwise you could share it via a service like Dropbox or Google Drive. If you are unable to share that file, can you please share your demux summarize visualization (I asked for that earlier, it might’ve gotten missed)? Thanks! We will get to the bottom of this! :t_rex:

Hi @fvalbores --- thanks for sending your data (in a DM). I ran demux summarize, and was pretty surprised at the results:

The reverse reads look almost identical to that plot, too.

To me, this does not look like typical Illumnia reads - these reads look like they are synthetic, or have been manipulated in some way. Can you provide some details about the sequencing technology you used (you mentioned Illumina above, but like I said, this doesn't look like any Illumina data I have ever seen), and any preprocessing steps that you might have applied before importing into QIIME 2?

Thanks! :t_rex:

I apologize for the inconveniences; after a thorough review of the data we found that by mistake a file modified by a script had been used.
We returned to the original data and after cleaning them, 70% were discarded and we will continue with high quality sequence.
We execute each one of the mentioned steps and when arriving to dada2 we obtained the same error.

The paired-end-demux.qza file has been updated in Google Drive.

The commands are:
qiime dada2 denoise-paired
–i-demultiplexed-seqs paired-end-demux.qza
–verbose
–o-table table
–o-representative-sequences rep-seqs
–p-trunc-len-f 0
–p-trunc-len-r 0

And the message received is:
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /tmp/tmpq92n6t_t/forward /tmp/tmpq92n6t_t/reverse /tmp/tmpq92n6t_t/output.tsv.biom /tmp/tmpq92n6t_t/filt_f /tmp/tmpq92n6t_t/filt_r 0 0 0 0 2.0 2 consensus 1.0 1 1000000

R version 3.4.1 (2017-06-30)
Loading required package: Rcpp
DADA2 R package version: 1.6.0

  1. Filtering Error in add(bin) : internal: buf !=
    Execution halted
    Traceback (most recent call last):
    File “/home/paco/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 179, in denoise_paired
    run_commands([cmd])
    File “/home/paco/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 35, in run_commands
    subprocess.run(cmd, check=True)
    File “/home/paco/miniconda3/envs/qiime2-2017.12/lib/python3.5/subprocess.py”, line 398, in run
    output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command ‘[‘run_dada_paired.R’, ‘/tmp/tmpq92n6t_t/forward’, ‘/tmp/tmpq92n6t_t/reverse’, ‘/tmp/tmpq92n6t_t/output.tsv.biom’, ‘/tmp/tmpq92n6t_t/filt_f’, ‘/tmp/tmpq92n6t_t/filt_r’, ‘0’, ‘0’, ‘0’, ‘0’, ‘2.0’, ‘2’, ‘consensus’, ‘1.0’, ‘1’, ‘1000000’]’ returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/paco/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2cli/commands.py”, line 224, in call
results = action(**arguments)
File “”, line 2, in denoise_paired
File “/home/paco/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 228, in bound_callable
output_types, provenance)
File “/home/paco/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 363, in callable_executor
output_views = self._callable(**view_args)
File “/home/paco/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 194, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Hi @fvalbores - it still sounds like you are modifying your raw reads prior to running them through DADA2 --- if I had to guess these modifications are responsible for the error you are seeing, because you are fundamentally changing the data, probably in some way that DADA2 is relying on when processing these reads.

These reads still looks synthetic to me, not at all like what I would expect to see from an unmodified Illumina run.

Is it possible for you to skip your preprocessing and give it a shot with unmodified reads?

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.