DADA2 -11 exit code (paired-end reads)

Peris · December 29, 2017, 5:37pm

Hello there,

I am running Sequence quality control using DADA2 and am using my imported sequences (paired-end-demux.qza), however, I get an error.

My script is as shown below:

qiime dada2 denoise-paired
–i-demultiplexed-seqs run2_demux-paired-end.qza
–o-table run2_table.qza
–o-representative-sequences run2_rep-seqs.qza
–p-trim-left-f 0
–p-trim-left-r 0
–p-trunc-len-f 300
–p-trunc-len-r 240

But I get this error;

Plugin error from dada2:
An error was encountered while running DADA2 in R (return code -11), please inspect stdout and stderr to learn more.
Debug info has been saved to /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/qiime2-q2cli-err-io3ltpxa.log

Am using qiime2-2017.12

I thought the problem was my R version so, I uninstalled version 3.4.2 and installed version 3.3.3 but I still get the same error.

Any help will be much appreciated.
Thank you in advance

jairideout · December 29, 2017, 6:02pm

Hi @Peris! Can you please provide the contents of /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/qiime2-q2cli-err-io3ltpxa.log, or rerun the command with --verbose and paste the debug info here? Thanks!

Peris · December 29, 2017, 8:35pm

Hi Jairideout,

So, where should I include --verbose in my command? I included it at the end of my script

qiime dada2 denoise-paired
–i-demultiplexed-seqs run2_demux-paired-end.qza
–o-table run2_table.qza
–o-representative-sequences run2_rep-seqs.qza
–p-trim-left-f 0
–p-trim-left-r 0
–p-trunc-len-f 300
–p-trunc-len-r 240
–verbose

but I got a similar error as reported earlier plus ‘-bash: --verbose: command not found’

Thanks once again.

Please find the /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/qiime2-q2cli-err-io3ltpxa.log contents.

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp0nticwhu/forward /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp0nticwhu/reverse /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp0nticwhu/output.tsv.biom /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp0nticwhu/filt_f /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp0nticwhu/filt_r 300 240 0 0 2.0 2 consensus 1.0 1 1000000

R version 3.4.1 (2017-06-30)
Loading required package: Rcpp

*** caught segfault ***
address 0x18, cause ‘memory not mapped’

Traceback:
1: dyn.load(file, DLLpath = DLLpath, …)
2: library.dynam(lib, package, package.lib)
3: loadNamespace(package, lib.loc)
4: doTryCatch(return(expr), name, parentenv, handler)
5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
6: tryCatchList(expr, classes, parentenv, handlers)
7: tryCatch({ attr(package, “LibPath”) <- which.lib.loc ns <- loadNamespace(package, lib.loc) env <- attachNamespace(ns, pos = pos, deps)}, error = function(e) { P <- if (!is.null(cc <- conditionCall(e))) paste(" in", deparse(cc)[1L]) else “” msg <- gettextf(“package or namespace load failed for %s%s:\n %s”, sQuote(package), P, conditionMessage(e)) if (logical.return) message(paste(“Error:”, msg), domain = NA) else stop(msg, call. = FALSE, domain = NA)})
8: library(pkg, character.only = TRUE, logical.return = TRUE, lib.loc = lib.loc, quietly = quietly)
9: .getRequiredPackages2(pkgInfo, quietly = quietly)
10: library(dada2)
11: withCallingHandlers(expr, warning = function(w) invokeRestart(“muffleWarning”))
12: suppressWarnings(library(dada2))
An irrecoverable exception occurred. R is aborting now …
Traceback (most recent call last):
File “/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 179, in denoise_paired
run_commands([cmd])
File “/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 35, in run_commands
subprocess.run(cmd, check=True)
File “/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/subprocess.py”, line 398, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘run_dada_paired.R’, ‘/var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp0nticwhu/forward’, ‘/var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp0nticwhu/reverse’, ‘/var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp0nticwhu/output.tsv.biom’, ‘/var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp0nticwhu/filt_f’, ‘/var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp0nticwhu/filt_r’, ‘300’, ‘240’, ‘0’, ‘0’, ‘2.0’, ‘2’, ‘consensus’, ‘1.0’, ‘1’, ‘1000000’]’ returned non-zero exit status -11

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2cli/commands.py”, line 224, in call
results = action(**arguments)
File “”, line 2, in denoise_paired
File “/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 228, in bound_callable
output_types, provenance)
File “/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 363, in callable_executor
output_views = self._callable(**view_args)
File “/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 194, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code -11), please inspect stdout and stderr to learn more.

ebolyen · December 29, 2017, 8:49pm

Hi @Peris,

You mentioned that you have an existing R installation. With this latest QIIME 2, we’re now using R 3.4.1, but I’m wondering if we’re just running into an old linking issue, where the R in our QIIME 2 conda environment finds packages compiled to your other installation.

Are you using the .Rpofile hack that we’ve recommended in the past? If not, here’s a thread describing what to do. It isn’t great but it kind of works.

Peris · January 2, 2018, 5:36pm

Hi @ebolyen,

I checked and I realized that I didn't have .Rpofile in my home directory
So, I followed the suggestions on getting .Rpofile into my home directory as shown below:

Run: echo ".libPaths(.libPaths()[2])" > $HOME/.Rprofile

Run: cat ~/.Rprofile
.libPaths(.libPaths()[2])

Run: R -e ".libPaths()"

.libPaths()
[1] "/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/R/library"

After this, I run my DADA2 script again. This time round the script continued running for 30 minutes after which, I got the same error.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code -11), please inspect stdout and stderr to learn more.

Debug info has been saved to /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/qiime2-q2cli-err-u6zf12vt.log

Here is the log file:

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp00g5__lr/forward /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp00g5__lr/reverse /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp00g5__lr/output.tsv.biom /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp00g5__lr/filt_f /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp00g5__lr/filt_r 300 240 10 0 2.0 2 consensus 1.0 1 1000000

R version 3.4.1 (2017-06-30)
Loading required package: Rcpp
DADA2 R package version: 1.6.0

Filtering ........................................................................................................................................................................................................................................................................................................................
Learning Error Rates
2a) Forward Reads
Initializing error rates to maximum possible estimate.
Sample 1 - 12719 reads in 4682 unique sequences.

*** caught segfault ***
address 0x8, cause 'memory not mapped'

Traceback:
1: .Call("_dada2_dada_uniques", PACKAGE = "dada2", seqs, abundances, err, quals, score, gap, use_kmers, kdist_cutoff, band_size, omegaA, max_clust, min_fold, min_hamming, min_abund, use_quals, final_consensus, vectorized_alignment, homo_gap, multithread, verbose, SSE)
2: dada_uniques(names(derep[[i]]$uniques), unname(derep[[i]]$uniques), err, qi, opts[["SCORE_MATRIX"]], opts[["GAP_PENALTY"]], opts[["USE_KMERS"]], opts[["KDIST_CUTOFF"]], opts[["BAND_SIZE"]], opts[["OMEGA_A"]], if (initializeErr) { 1 } else { opts[["MAX_CLUST"]] }, opts[["MIN_FOLD"]], opts[["MIN_HAMMING"]], opts[["MIN_ABUNDANCE"]], TRUE, FALSE, opts[["VECTORIZED_ALIGNMENT"]], opts[["HOMOPOLYMER_GAP_PENALTY"]], multithread, (verbose >= 2), opts[["SSE"]])
3: dada(drpsF, err = NULL, selfConsist = TRUE, multithread = multithread, VECTORIZED_ALIGNMENT = FALSE, SSE = 2)
An irrecoverable exception occurred. R is aborting now ...
Traceback (most recent call last):
File "/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 179, in denoise_paired
run_commands([cmd])
File "/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 35, in run_commands
subprocess.run(cmd, check=True)
File "/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/subprocess.py", line 398, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run_dada_paired.R', '/var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp00g5__lr/forward', '/var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp00g5__lr/reverse', '/var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp00g5__lr/output.tsv.biom', '/var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp00g5__lr/filt_f', '/var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/tmp00g5__lr/filt_r', '300', '240', '10', '0', '2.0', '2', 'consensus', '1.0', '1', '1000000']' returned non-zero exit status -11

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2cli/commands.py", line 224, in call
results = action(**arguments)
File "", line 2, in denoise_paired
File "/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py", line 228, in bound_callable
output_types, provenance)
File "/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py", line 363, in callable_executor
output_views = self._callable(**view_args)
File "/Users/perismunyaka/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 194, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code -11), please inspect stdout and stderr to learn more.

ebolyen · January 2, 2018, 7:04pm

Thanks for the info @Peris!

What version of OS X are you running?

Also, just to see, does activating your environment and then running this command:

CDPATH= R -e 'source("https://bioconductor.org/biocLite.R"); biocLite("dada2")'

fix the issue?

We still don’t really know what’s going on and we’re trying to narrow it down. Thanks for your patience!

Peris · January 2, 2018, 8:02pm

Hi @ebolyen,

I am using macOS High Sierra Version 10.13.1

I finished running the code above and am currently running DADA2 again. I will let you know how it goes.

I sincerely appreciate your great help

Peris · January 2, 2018, 8:18pm

Hello @ebolyen

Unfortunately I got the same error!

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code -11), please inspect stdout and stderr to learn more.
Debug info has been saved to /var/folders/p9/31kshwj91nv0tgcrml76m11m0000gn/T/qiime2-q2cli-err-tmurmnyp.log

ebolyen · January 3, 2018, 6:21pm

Hey @Peris!

Thanks for the info. Re-reading this thread, it does look like the RProfile had some effect. Your segfault changed from 0x18 to 0x8 (matching everyone else). So I guess we “fixed” that issue .

Would you be able to run this command?

sysctl -a | grep machdep.cpu.leaf7_features

we’re specifically looking for an entry called “AVX2” hopefully that shows up.

Thanks!

Peris · January 3, 2018, 6:29pm

Hi @ebolyen

I just did!
This is what I got:

machdep.cpu.leaf7_features: SMEP ERMS RDWRFSGS TSC_THREAD_OFFSET BMI1 AVX2 BMI2 INVPCID SMAP RDSEED ADX IPT FPU_CSDS

I see ‘AVX2’ here and hopefully that will help in finding a solution!

Thank you for the continued support

ebolyen · January 3, 2018, 6:33pm

Thanks @Peris!

Would you be able to share your data (we can use the forum’s DM system)? Perhaps there’s something about the data rather than the installation/host.

Peris · January 3, 2018, 6:43pm

Hey @ebolyen,

I am not sure of how I can share my data but will try to figure it out. However, I got the same error while running the data in the “Moving Pictures” Tutorial, which kind of makes me think that the error is not related to the data at all.

Has any one else (Qiime2-2017.12 users) encountered the same problem with the “Moving Pictures” tutorial? For me everything else worked except dada2

Kind regards

ebolyen · January 3, 2018, 6:54pm

That is super helpful! Thanks so much!

ebolyen · January 3, 2018, 7:07pm

@Peris if you have a moment, could you run the following in your QIIME 2 environment:

conda install pytest

pytest --pyargs q2_dada2

This will run our unit-tests, if they fail with the same issue, then we have a very small dataset we can use to reproduce this. I kind of hope they fail since it will make our debugging simpler.

Peris · January 3, 2018, 7:16pm

Hi @ebolyen,

Please find my outputs:

(qiime2-2017.12) Periss-MacBook-Pro:~ perismunyaka$ conda install pytest

Fetching package metadata …
Solving package specifications: .

Package plan for installation in environment /Users/perismunyaka/miniconda3/envs/qiime2-2017.12:

The following NEW packages will be INSTALLED:

attrs:          17.3.0-py35hc219a11_0
coverage:       4.4.2-py35h36907ce_0 
hypothesis:     3.38.5-py35h843401b_0
pluggy:         0.6.0-py35hf57b818_0 
py:             1.5.2-py35h4d4d87d_0 
pympler:        0.5-py35hf41ee38_0   
pytest:         3.3.0-py35hfa6e833_0 
zope:           1.0-py35_0           
zope.interface: 4.4.3-py35hdbc5dd8_0

Proceed ([y]/n)? y

coverage-4.4.2 100% |########################################################| Time: 0:00:00 1.82 MB/s
pluggy-0.6.0-p 100% |########################################################| Time: 0:00:00 5.42 MB/s
py-1.5.2-py35h 100% |########################################################| Time: 0:00:00 1.48 MB/s
pympler-0.5-py 100% |########################################################| Time: 0:00:00 1.87 MB/s
zope-1.0-py35_ 100% |########################################################| Time: 0:00:00 511.70 kB/s
zope.interface 100% |########################################################| Time: 0:00:00 1.99 MB/s
attrs-17.3.0-p 100% |########################################################| Time: 0:00:00 7.48 MB/s
hypothesis-3.3 100% |########################################################| Time: 0:00:00 1.55 MB/s
pytest-3.3.0-p 100% |########################################################| Time: 0:00:00 1.47 MB/s

(qiime2-2017.12) Periss-MacBook-Pro:~ perismunyaka$ pytest --pyargs q2_dada2

========================================= test session starts ==========================================
platform darwin – Python 3.5.4, pytest-3.3.0, py-1.5.2, pluggy-0.6.0
rootdir: /Users/perismunyaka, inifile:
plugins: hypothesis-3.38.5
collected 11 items

miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/tests/test_denoise.py … [ 54%]
… [100%]

===================================== 11 passed in 185.51 seconds ======================================

ebolyen · January 3, 2018, 7:29pm

Thank you so much for all your help @Peris!

I’ve run a bunch of tests locally as well and I’m not seeing anything either. So hopefully this isn’t a stochastic issue (though it’s virtually impossible to prove that it isn’t…).

Does the DADA 2 step of the Moving Pictures tutorial consistently fail for you?

I think we’re looking at a mix of having a more realistic dataset (so not our tests) and the host. We’ve been able to run the Moving Pictures on our test hosts (macbook pro, mac mini, docker containers, and our own dev computers) and haven’t been able to see this ourselves yet .

Peris · January 3, 2018, 8:42pm

Hi @ebolyen,

I went through the Moving Pictures tutorial before I started running my own data, and that’s when I got the error for the first time (this was before I made my first post here).

Now I have repeated dada2 analysis for the Moving Tutorial data and this time round, it worked!

I got excited and decided to repeat dada2 for my data; however, I still got the error again!
I had 312 samples in my imported qiime2 artifatcs so, I decided to split the samples and tried running dada2 with fewer samples (around 59), but I still got the same error.

Attached please find part of my data that I tried dada2 on, and got the error!

thermokarst · January 3, 2018, 9:00pm

Hi @Peris!

Did you mean to attach a file? If so, it doesn't look like it worked - can you try again? Thanks!

thermokarst · January 3, 2018, 9:08pm

Was this before you made the Rprofile changes? Without that error log, we can't say for sure if you were experiencing the same error then or not, which might be a good thing (you have reported two separate errors here in this thread - I think the first one was resolved with the Rprofile fix).

If you are seeing stochastic errors with the Moving Pictures data, that type of bug is pretty hard to reproduce and diagnose, but if it is actually the case that the Moving Pictures error you experienced was an entirely different one, then maybe we can keep working the angle that this error is data-specific. To that end, as I mentioned above, getting a dataset that reproduces this error would be extremely helpful for us, since we have been unable to reproduce this specific error locally. Thanks!

ebolyen · January 3, 2018, 9:17pm

Shot in the dark here, but would you be able to run the following:

conda create -n DONOTUSE --file  https://data.qiime2.org/distro/core/qiime2-2017.12-conda-osx-64.txt
source activate DONOTUSE
wget -qO- https://forum-qiime2-org.s3-us-west-2.amazonaws.com/original/1X/ac621debfd04f561bf1075b4efa56c0e947a1928.txt > `which run_dada_single.R`
wget -qO- https://forum-qiime2-org.s3-us-west-2.amazonaws.com/original/1X/07fa5193f904022996a35a963b12b9ac79dbb1f4.txt > `which run_dada_paired.R`

This is going to overwrite some stuff which might help. Then run dada2 like normal on your dataset.

Thanks again!