Demux summarize and Dada2 error

We are working on system with 16Gb RAM and 4 cores.

The files (demultiplexed fastq.gz of 27 samples - 2.9 Gb) were imported using following:
(qiime2-2018.11) [email protected]:~$ qiime tools import --type ‘SampleData[PairedEndSequencesWithQuality]’ --input-path pe-all27-manifest --output-path paired-end-demux --input-format PairedEndFastqManifestPhred33
Imported pe-all27-manifest as PairedEndFastqManifestPhred33 to paired-end-demux

While running dada2, following error occurs:
qiime dada2 denoise-paired --i-demultiplexed-seqs paired-end-demux.qza --p-trim-left-f 8 --p-trim-left-r 9 --p-trunc-len-f 275 --p-trunc-len-r 224 --o-table pedemuxtable.qza --o-representative-sequences pedemuxrepresentative-sequences.qza --o-denoising-stats pedemuxdenoising-stats.qza --verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /tmp/tmph7asg0jq/forward /tmp/tmph7asg0jq/reverse /tmp/tmph7asg0jq/output.tsv.biom /tmp/tmph7asg0jq/track.tsv /tmp/tmph7asg0jq/filt_f /tmp/tmph7asg0jq/filt_r 275 224 8 9 2.0 2 consensus 1.0 1 1000000

R version 3.4.1 (2017-06-30)
Loading required package: Rcpp

*** caught segfault ***
address 0x10006a8b8b, cause ‘memory not mapped’
Traceback (most recent call last):
File “/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 231, in denoise_paired
run_commands([cmd])
File “/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 36, in run_commands
subprocess.run(cmd, check=True)
File “/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/subprocess.py”, line 398, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘run_dada_paired.R’, ‘/tmp/tmph7asg0jq/forward’, ‘/tmp/tmph7asg0jq/reverse’, ‘/tmp/tmph7asg0jq/output.tsv.biom’, ‘/tmp/tmph7asg0jq/track.tsv’, ‘/tmp/tmph7asg0jq/filt_f’, ‘/tmp/tmph7asg0jq/filt_r’, ‘275’, ‘224’, ‘8’, ‘9’, ‘2.0’, ‘2’, ‘consensus’, ‘1.0’, ‘1’, ‘1000000’]’ returned non-zero exit status -11

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2cli/commands.py", line 274, in call
results = action(**arguments)
File "<decorator-gen-436>", line 2, in denoise_paired
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 362, in callable_executor
output_views = self._callable(**view_args)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 246, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code -11), please inspect stdout and stderr to learn more.
Plugin error from dada2:
An error was encountered while running DADA2 in R (return code -11), please inspect stdout and stderr to learn more.
See above for debug info.

ISSUE 2: While trying to run demux summarize on the same file
(qiime2-2018.11) [email protected]:~$ qiime demux summarize --i-data paired-end-demux.qza --o-visualization paired-end-demux --verbose Traceback (most recent call last):
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2cli/commands.py", line 274, in call
results = action(**arguments)
File "<decorator-gen-419>", line 2, in summarize
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 424, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_demux/_summarize/_visualizer.py", line 131, in summarize
for seq in _read_fastq_seqs(file):
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_demux/_demux.py", line 34, in _read_fastq_seqs
for seq_header, seq, qual_header, qual in itertools.zip_longest(*[fh] * 4):
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/gzip.py", line 287, in read1
return self._buffer.read1(size)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/_compression.py", line 68, in readinto
data = self.read(len(byte_view))
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/gzip.py", line 452, in read
self._read_eof()
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/gzip.py", line 499, in _read_eof
hex(self._crc)))
OSError: CRC check failed 0x2a1edf4e != 0x214264ab

Plugin error from demux:

CRC check failed 0x2a1edf4e != 0x214264ab

See above for debug info.
Segmentation fault (core dumped)

Overall, it sounds like your hard disk may be running out of space while running these commands. Make sure you have adequate hard disk space, particularly in your tmp directory.

Can you provide some details about the disks, partitioning, and where you mapped tmp to?

For your dada2 error, see the topics here for some instructions for diagnosing and debugging:



If none of that helps, please provide the system information requested of others in those topic threads.

Sounds like a partition on your hard disk is running out of space.

Dear Nicholas,
Thank you for the response, yes the system configuration was the one hindering and we have changed the system and the command has been running for past 4 days now. Hope that will work now in that as the other demux-summarize error was sorted.
Also, it seems training the Classifier (SILVA database for V3-V4) would be again a memory requiring step, could you suggest if it work in 8Gb ram? also, are there pre-trained classifier for SILVA region mentioned available, I could find a Greengenes classifier from previous forum discussions, but not SILVA.

Thanks again.

Glad to hear it — did the job ever complete? If not, look at the demultiplexed seqs you are using as input; the file may be empty.

Yes, classification is memory-intensive. The greengenes classifier will take much less RAM than SILVA and 8GB would be fine. For SILVA you should probably have more memory but you can adjust the batch size to reduce memory demand (see other posts on this forum for more advice regarding memory issues with SILVA). We have pre-trained Naive Bayes classifiers for SILVA and greengenes database here:
https://docs.qiime2.org/2018.11/data-resources/

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.