Demux summarize and Dada2 error

[Chandrashekar_M (Chandrashekar Mootapally)]

We are working on system with 16Gb RAM and 4 cores.

The files (demultiplexed fastq.gz of 27 samples - 2.9 Gb) were imported using following:
(qiime2-2018.11) npdf@npdf-Inspiron-3268:~$ qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path pe-all27-manifest --output-path paired-end-demux --input-format PairedEndFastqManifestPhred33
Imported pe-all27-manifest as PairedEndFastqManifestPhred33 to paired-end-demux

While running dada2, following error occurs:
qiime dada2 denoise-paired --i-demultiplexed-seqs paired-end-demux.qza --p-trim-left-f 8 --p-trim-left-r 9 --p-trunc-len-f 275 --p-trunc-len-r 224 --o-table pedemuxtable.qza --o-representative-sequences pedemuxrepresentative-sequences.qza --o-denoising-stats pedemuxdenoising-stats.qza --verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /tmp/tmph7asg0jq/forward /tmp/tmph7asg0jq/reverse /tmp/tmph7asg0jq/output.tsv.biom /tmp/tmph7asg0jq/track.tsv /tmp/tmph7asg0jq/filt_f /tmp/tmph7asg0jq/filt_r 275 224 8 9 2.0 2 consensus 1.0 1 1000000

R version 3.4.1 (2017-06-30)
Loading required package: Rcpp

*** caught segfault ***
address 0x10006a8b8b, cause 'memory not mapped'
Traceback (most recent call last):
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 231, in denoise_paired
run_commands([cmd])
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 36, in run_commands
subprocess.run(cmd, check=True)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/subprocess.py", line 398, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run_dada_paired.R', '/tmp/tmph7asg0jq/forward', '/tmp/tmph7asg0jq/reverse', '/tmp/tmph7asg0jq/output.tsv.biom', '/tmp/tmph7asg0jq/track.tsv', '/tmp/tmph7asg0jq/filt_f', '/tmp/tmph7asg0jq/filt_r', '275', '224', '8', '9', '2.0', '2', 'consensus', '1.0', '1', '1000000']' returned non-zero exit status -11

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2cli/commands.py", line 274, in call
results = action(**arguments)
File "<decorator-gen-436>", line 2, in denoise_paired
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 362, in callable_executor
output_views = self._callable(**view_args)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 246, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code -11), please inspect stdout and stderr to learn more.
Plugin error from dada2:
An error was encountered while running DADA2 in R (return code -11), please inspect stdout and stderr to learn more.
See above for debug info.

ISSUE 2: While trying to run demux summarize on the same file
(qiime2-2018.11) npdf@npdf-Inspiron-3268:~$ qiime demux summarize --i-data paired-end-demux.qza --o-visualization paired-end-demux --verbose Traceback (most recent call last):
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2cli/commands.py", line 274, in call
results = action(**arguments)
File "<decorator-gen-419>", line 2, in summarize
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 424, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_demux/_summarize/_visualizer.py", line 131, in summarize
for seq in _read_fastq_seqs(file):
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_demux/_demux.py", line 34, in _read_fastq_seqs
for seq_header, seq, qual_header, qual in itertools.zip_longest(*[fh] * 4):
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/gzip.py", line 287, in read1
return self._buffer.read1(size)
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/_compression.py", line 68, in readinto
data = self.read(len(byte_view))
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/gzip.py", line 452, in read
self._read_eof()
File "/home/npdf/anaconda2/envs/qiime2-2018.11/lib/python3.5/gzip.py", line 499, in _read_eof
hex(self._crc)))
OSError: CRC check failed 0x2a1edf4e != 0x214264ab

Plugin error from demux:

CRC check failed 0x2a1edf4e != 0x214264ab

See above for debug info.
Segmentation fault (core dumped)

[Nicholas_Bokulich]

Overall, it sounds like your hard disk may be running out of space while running these commands. Make sure you have adequate hard disk space, particularly in your tmp directory.

Can you provide some details about the disks, partitioning, and where you mapped tmp to?

For your dada2 error, see the topics here for some instructions for diagnosing and debugging:

Dada2 segfault in 2018.2 and 2017.12 (Mac OS X 10.13)

I've installed Qiime2 2018.2 (2018.2.0, with dada2 version 2018.2.0) in its own conda environment following the official instructions. I am getting a segfault when Qiime tries to load the dada2 R package. I came across the problem while trying to run qiime dada2 denoise-paired, but all I need to do to trigger the error is activate the conda environment, open an R session, and execute library(dada2). The error message is: Loading required package: Rcpp *** caught segfault *** address 0x1310, c…

DADA2 (2017-12) segfault, memory not mapped error

Hi again, sorry to bother you all! Hope your holidays are going well. I'm still trying to run DADA2 on my 53 unjoined, paired-end fastq files. The code I used is below. I added a --p-n-threads option to help with the processing speed: $ qiime dada2 denoise-paired --i-demultiplexed-seqs demux-paired-end.qza --p-trim-left-f 0 --p-trunc-len-f 270 --p-trim-left-r 0 --p-trunc-len-r 120 --o-representative-sequences rep-seqs.qza --p-n-threads 0 --o-table table-dada2.qza It came out with an error: R…

Error while running DADA2 command?

Hi @Soha, It's been a while since we've seen the dreaded 0x8 error. I notice you are running 2017.12 which had an early release with this issue. Could you provide the results of qiime info? You may just need to update to the latest QIIME 2 (in fact I would recommend that in either case!) Otherwise, do you have an RStudio installation alongside QIIME 2 (or any other installation of R)? The library paths are known to conflict with each other causing this issue as well. In that case use: echo "…

If none of that helps, please provide the system information requested of others in those topic threads.

@Chandrashekar_M

Segmentation fault (core dumped)

Sounds like a partition on your hard disk is running out of space.

[Chandrashekar_M (Chandrashekar Mootapally)]

Dear Nicholas,
Thank you for the response, yes the system configuration was the one hindering and we have changed the system and the command has been running for past 4 days now. Hope that will work now in that as the other demux-summarize error was sorted.
Also, it seems training the Classifier (SILVA database for V3-V4) would be again a memory requiring step, could you suggest if it work in 8Gb ram? also, are there pre-trained classifier for SILVA region mentioned available, I could find a Greengenes classifier from previous forum discussions, but not SILVA.

Thanks again.

[Nicholas_Bokulich]

@Chandrashekar_M

Thank you for the response, yes the system configuration was the one hindering and we have changed the system and the command has been running for past 4 days now. Hope that will work now in that as the other demux-summarize error was sorted.

Glad to hear it — did the job ever complete? If not, look at the demultiplexed seqs you are using as input; the file may be empty.

@Chandrashekar_M

Also, it seems training the Classifier (SILVA database for V3-V4) would be again a memory requiring step, could you suggest if it work in 8Gb ram? also, are there pre-trained classifier for SILVA region mentioned available, I could find a Greengenes classifier from previous forum discussions, but not SILVA.

Yes, classification is memory-intensive. The greengenes classifier will take much less RAM than SILVA and 8GB would be fine. For SILVA you should probably have more memory but you can adjust the batch size to reduce memory demand (see other posts on this forum for more advice regarding memory issues with SILVA). We have pre-trained Naive Bayes classifiers for SILVA and greengenes database here:
https://docs.qiime2.org/2018.11/data-resources/