DADA2 stuck in a loop or just taking a long time?

Hi guys,
I’m trying DADA2 on an artifact made up of 32 samples (~3.15 Gig) which starts to run fine but end up getting stuck on selfConsist step 8 for now over 24hrs. I’m wondering if its just stuck on loop somehow or maybe this expected length of time?

There is 12G of Ram and 7 cores dedicated to my Virtual Machine (running Docker and qiime 2017.7) for this and about 40 Gigs of free space on the SSD partition that is running on. Everything being run from the containing folder.

qiime dada2 denoise-paired \
--i-demultiplexed-seqs paired-end-demux.qza \
--p-trunc-len-f 270 \
--p-trunc-len-r 230 \
--p-n-threads 7 \
--o-table table-dada2.qza \
--o-representative-sequences rep-seqs.qza \
--output-dir unspec-results \
--verbose

Everything up till Step 8 was being processed in pretty good speed (2-3 hours) but then it gets stuck and has been stuck for close to 24 hours now. Just wondering if this is expected behavior or somehow stuck in a loop.
The system is working pretty hard with the fan going full throttle and close to %100 of the CPU being used.

Running external command line application(s). This may print messages to stdout and/or stderr. The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist. Command: run_dada_paired.R /tmp/tmp4wxo5i3p/forward /tmp/tmp4wxo5i3p/reverse /tmp/tmp4wxo5i3p/output.tsv.biom /tmp/tmp4wxo5i3p/filt_f /tmp/tmp4wxo5i3p/filt_r 270 230 0 0 2.0 2 consensus 1.0 7 1000000 R version 3.3.1 (2016-06-21) Loading required package: Rcpp Warning messages: 1: multiple methods tables found for ‘arbind’ 2: multiple methods tables found for ‘acbind’ 3: replacing previous import ‘IRanges::arbind’ by ‘SummarizedExperiment::arbind’ when loading ‘GenomicAlignments’ 4: replacing previous import ‘IRanges::acbind’ by ‘SummarizedExperiment::acbind’ when loading ‘GenomicAlignments’ 5: multiple methods tables found for ‘left’ 6: multiple methods tables found for ‘right’ DADA2 R package version: 1.4.0 1) Filtering ............................... 2) Learning Error Rates 2a) Forward Reads Initializing error rates to maximum possible estimate. Sample 1 - 52120 reads in 25033 unique sequences. Sample 2 - 55364 reads in 26544 unique sequences. Sample 3 - 47665 reads in 22116 unique sequences. Sample 4 - 78223 reads in 35775 unique sequences. Sample 5 - 232027 reads in 92241 unique sequences. Sample 6 - 115288 reads in 48823 unique sequences. Sample 7 - 14755 reads in 8138 unique sequences. Sample 8 - 40034 reads in 20349 unique sequences. Sample 9 - 21558 reads in 10705 unique sequences. Sample 10 - 16360 reads in 8655 unique sequences. Sample 11 - 34443 reads in 15624 unique sequences. Sample 12 - 6287 reads in 4123 unique sequences. Sample 13 - 52638 reads in 20303 unique sequences. Sample 14 - 8961 reads in 4801 unique sequences. Sample 15 - 33394 reads in 15948 unique sequences. Sample 16 - 10576 reads in 6072 unique sequences. Sample 17 - 30462 reads in 15299 unique sequences. Sample 18 - 67234 reads in 28569 unique sequences. Sample 19 - 44296 reads in 21446 unique sequences. Sample 20 - 57836 reads in 26209 unique sequences. selfConsist step 2 selfConsist step 3 selfConsist step 4 selfConsist step 5 selfConsist step 6 selfConsist step 7 selfConsist step 8

When I tried to terminate the process ctrl + C, it carries on with the following

selfConsist step 9 selfConsist step 10 Warning message: In dada(drpsF, err = NULL, selfConsist = TRUE, multithread = multithread) : Self-consistency loop terminated before convergence. 2b) Reverse Reads Initializing error rates to maximum possible estimate. Sample 1 - 52120 reads in 28794 unique sequences. Sample 2 - 55364 reads in 29168 unique sequences. Sample 3 - 47665 reads in 26478 unique sequences. Sample 4 - 78223 reads in 43292 unique sequences. Sample 5 - 232027 reads in 111034 unique sequences. Sample 6 - 115288 reads in 60240 unique sequences. Sample 7 - 14755 reads in 8901 unique sequences. Sample 8 - 40034 reads in 23578 unique sequences. Sample 9 - 21558 reads in 12732 unique sequences. Sample 10 - 16360 reads in 9052 unique sequences. Sample 11 - 34443 reads in 19197 unique sequences. Sample 12 - 6287 reads in 4175 unique sequences. Sample 13 - 52638 reads in 27308 unique sequences. Sample 14 - 8961 reads in 5817 unique sequences. Sample 15 - 33394 reads in 18903 unique sequences. Sample 16 - 10576 reads in 7271 unique sequences. Sample 17 - 30462 reads in 18194 unique sequences. Sample 18 - 67234 reads in 44234 unique sequences. Sample 19 - 44296 reads in 26943 unique sequences. Sample 20 - 57836 reads in 33664 unique sequences. selfConsist step 2 selfConsist step 3 selfConsist step 4 selfConsist step 5 selfConsist step 6 selfConsist step 7 selfConsist step 8 Convergence after 8 rounds. 3) Denoise remaining samples ........... The sequences being tabled vary in length. 4) Remove chimeras (method = consensus)

and now is continuing to run.

I’m worried I just forced it to abandon some steps and skip through them.

If it helps at all the unspec-results folder it created is currently still empty as well.

Any thoughts?

And I apologize for the terrible “paste” format. It looked fine when I was posting it…

Hi @Mehrbod_Estaki,

I don’t think it’s stuck if you’ve got real utilization.

That is super strange, but since you are running docker, it’s possible the interrupt wasn’t forwarded like we would expect. Since it kept going, I am going to assume that the ctrl-c just didn’t do anything.

Just to double check, are you processing amplicon sequences?

If you had managed to kill it, the entire process would have ended, so I wouldn’t worry.

No worries! (Even I couldn’t manage to fix it for you)

1 Like

Ok, that’s promising, I’ll let it carry on. And yes they are 16S amplicons with an expected ~460 bp long paired sequences (2x300 with 140 overlap), some variation is expected but not too much I would think.

The Ctr-C threw me off as well, though I’m thinking you are right in that it didn’t do anything in docker as I thought it would. Instead, pressing “enter” may have just kind of did a refresh of the terminal and some of the newer processes that were actually running just simply became visible. I had been running it uninterrupted for close to 24 hrs, so its possible it just hadn’t refrehsed…will update as I can.
Thanks for quick reply as always!

1 Like

Update: It was indeed just taking a long time and completed without issues.

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.