DADA2 - Error Return Code 1 - Bad Lambda

image

Hi Everyone,
I seem to be getting this error every time I run a large amount of samples through DADA2.

When investigating the actual error log further, I can gather the following info:

Command: run_dada_paired.R /tmp/tmprcrf76zz/forward /tmp/tmprcrf76zz/reverse /tmp/tmprcrf76zz/output.tsv.biom /tmp/tmprcrf76zz/track.tsv /tmp/tmprcrf76zz/filt_f /tmp/tmprcrf76zz/filt_r 150 150 19 19 2.0 2.0 2 consensus 1.0 1 1000000

R version 3.5.1 (2018-07-02)
Loading required package: Rcpp
DADA2: 1.10.0 / Rcpp: 1.0.2 / RcppParallel: 4.4.3

  1. Filtering ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
  2. Learning Error Rates
    141968499 total bases in 1083729 reads from 3 samples will be used for learning the error rates.
    141968499 total bases in 1083729 reads from 3 samples will be used for learning the error rates.
  3. Denoise remaining samples .............................................................................................................................................................................................................................................................................................Error in dada_uniques(names(derep[[i]]$uniques), unname(derep[[i]]$uniques), :
    Bad lambda.
    Calls: dada -> dada_uniques
    Execution halted
    Traceback (most recent call last):
    File "/home/ec2-user/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 234, in denoise_paired
    run_commands([cmd])
    File "/home/ec2-user/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 36, in run_commands
    subprocess.run(cmd, check=True)
    File "/home/ec2-user/miniconda3/envs/qiime2-2019.7/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['run_dada_paired.R', '/tmp/tmprcrf76zz/forward', '/tmp/tmprcrf76zz/reverse', '/tmp/tmprcrf76zz/output.tsv.biom', '/tmp/tmprcrf76zz/track.tsv', '/tmp/tmprcrf76zz/filt_f', '/tmp/tmprcrf76zz/filt_r', '150', '150', '19', '19', '2.0', '2.0', '2', 'consensus', '1.0', '1', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ec2-user/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py", line 327, in call
results = action(**arguments)
File "</home/ec2-user/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-459>", line 2, in denoise_paired
File "/home/ec2-user/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 240, in bound_callable
output_types, provenance)
File "/home/ec2-user/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 383, in callable_executor
output_views = self._callable(**view_args)
File "/home/ec2-user/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 249, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Anyone else have this problem ?

Hi!
I think that you are out of RAM or free disk space. Probably you need a stronger machine

Im using an EC2 instance with 64 Gb RAM and 500 Gb HDD, would that not be enough ? I am running quite a large sample set with 897 samples though. Is there a metric for estimating what resources DADA2 will use ?

It depends on the source of your samples and the size of the dataset. I have a machine with 128 GB of RAM and 1 Tb of ROM, and with environmental samples (about 400) and in some cases it was not enough. So I suppose that your computer is not strong enough to process such big dataset.

1 Like

Alright, Im moving it to an instance with 768 GiB RAM and 2Tb HDD, I will report back if I succeed!

@Rkubinski @timanix I am not sure that this is a memory error, and 64 GB RAM should (in most cases unless if you have a massive dataset, e.g., a multi-run analysis or meta-analysis) be sufficient, though there are exceptions.

As far as I know this error has not been reported on this forum. A similar error was reported but never solved on the dada2 issue tracker. This appears to be the key line:

That error message is coming from dada2 but it is not clear to me what would cause that error, and it sounds like the report on the dada2 issue tracker had @benjjneb just as flummoxed. Perhaps you could provide the info that was requested on that issue and we can pick up from there. Sounds like @benjjneb suspected an architecture issue so perhaps you can tell us a bit more about the system you are using; and is this error reproducible, i.e. do you get the same error each time? If you have a second computer available maybe try running on that to see if the same QIIME 2, dada2, and R versions produce the same error with the same dataset.

5 Likes

I have it running right now on the more powerful instance so I guess we will find out.
In terms of finding similar bugs, I did find it surprising that this issue was not reported elsewhere.

The error is reproducible, it occurs the same way with every run. I am running all of this on an EC2 instance with the following info:
Linux kernel 4.14 tuned for optimal performance on Amazon EC2, systemd 219, GCC 7.3, Glibc 2.26, Binutils 2.29.1, and the latest software packages through extras.

If my efforts to run on this expanded memory machine don’t work, I will run it on an instance with the following architecture: Ubuntu Server 18.04 LTS (HVM).

EDIT: I forgot to mention also: I have ran smaller datasets of ~300 samples on this same architecture without issue.

1 Like

So my run failed, even on this ultra powerful instance. I am migrating over to an Ubuntu architecture and hope to see results there. Will update here once I am done.

1 Like

Hey @Rkubinski, as @Nicholas_Bokulich mentioned above, it is probably unlikely that this bug has anything to do with available memory (thanks @timanix for suggesting that, though!).

Can you please provide the information that @Nicholas_Bokulich asked for above?

I did provide this information in my answer above, the error is reproducible and many different datasets show me the same problem. The architecture is the standard kernel from AWS instances. Is there anything else I can provide ?

Any results yet? I think that was the last piece of info I was looking for.

Thanks, I saw that! I was actually looking for the first part of what @Nicholas_Bokulich asked for:

I think the answer to that is here, but looking for confirmation:

Thanks!

1 Like

Yes, that information plus the complete command — all is given above but in different posts, so @Rkubinski answered all my queries except for testing on a different system. Sorry, I should have clarified earlier!

1 Like

Still working on it but should have results soon. No worries about the confusion !

1 Like

So update:
Bad news but it gave the same error on Ubuntu 18.04 architecture. Sorry to disappoint guys.

Thanks for the update, that’s what I expected! Are you able to share these data? If so can you send a link in a DM, that way we can attempt to reproduce?

Yes no problem ! Theyre public sets.
https://gut.bmj.com/content/67/9/1743
raw file link: PRJEB23009


raw file link: PRJEB20022

**I just remembered that I ran these files previously on a Desktop computer with Ubuntu in September and DADA2 did not crash. I think this is really some sort of AWS related problem. Im going to try to run it on the Desktop and report back.

3 Likes

Hi guys, same problem on the Desktop. So I am going to give up on this until there is a solution, let me know if theres any other info I can provide.

Thanks @Rkubinski! Sorry to hear this is a persistent issue for you with this dataset, but glad you have identified a dataset that we may be able to use to replicate this issue.

Any chance you could share the QZA inputs to dada2 that you are using? That will speed up efforts to troubleshoot on this end.

Do you have a preference for where you would like it uploaded ? it is a large file