Dada2 denoise-paired running time over 24hrs and taking more than 4TB temporary space on HPC

deepak · May 28, 2023, 1:27am

I am running DADA2 through QIIME and my demux.qza file is 110GB. After 24Hr I saw it consumed more than 4TB of disk space and the log file is not updating from a certain point.
Here I am curious whether it is normal or if there is any glitch and I need to restart the analysis.
Kindly suggest.
I am sharing the log file and command:

time mpirun qiime dada2 denoise-single
--i-demultiplexed-seqs coin_single-end-demux.qza --p-trim-left 0 --p-trunc-len 0 --p-n-threads 0 --o-representative-sequences rep-seqs-dada2.qza --o-table table-dada2.qza --o-denoising-stats stats-dada2.qza --verbose

log:
Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/temp/qiime2-archive-6eqn0u1c/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp41b9srao/output.tsv.biom /home/test12/scr4-test1/temp/tmp41b9srao/track.tsv /home/test12/scr4-test1/temp/tmp41b9srao 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-77rgs2vk/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp21t11680/output.tsv.biom /home/test12/scr4-test1/temp/tmp21t11680/track.tsv /home/test12/scr4-test1/temp/tmp21t11680 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-aa9mg1yb/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmplb8rfki7/output.tsv.biom /home/test12/scr4-test1/temp/tmplb8rfki7/track.tsv /home/test12/scr4-test1/temp/tmplb8rfki7 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

Filtering Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-_04iljtf/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpcf0ah59e/output.tsv.biom /home/test12/scr4-test1/temp/tmpcf0ah59e/track.tsv /home/test12/scr4-test1/temp/tmpcf0ah59e 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

Filtering Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-7tntpq9u/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp5by7zk2j/output.tsv.biom /home/test12/scr4-test1/temp/tmp5by7zk2j/track.tsv /home/test12/scr4-test1/temp/tmp5by7zk2j 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-uzqat370/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp14bzcc08/output.tsv.biom /home/test12/scr4-test1/temp/tmp14bzcc08/track.tsv /home/test12/scr4-test1/temp/tmp14bzcc08 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-h5gbfr0u/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpv4dcttbh/output.tsv.biom /home/test12/scr4-test1/temp/tmpv4dcttbh/track.tsv /home/test12/scr4-test1/temp/tmpv4dcttbh 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-wkzb__vf/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmps0q3oqpz/output.tsv.biom /home/test12/scr4-test1/temp/tmps0q3oqpz/track.tsv /home/test12/scr4-test1/temp/tmps0q3oqpz 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-rzvzazvv/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp9gwx0asc/output.tsv.biom /home/test12/scr4-test1/temp/tmp9gwx0asc/track.tsv /home/test12/scr4-test1/temp/tmp9gwx0asc 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-uj9_hgn2/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpwvcsfo7t/output.tsv.biom /home/test12/scr4-test1/temp/tmpwvcsfo7t/track.tsv /home/test12/scr4-test1/temp/tmpwvcsfo7t 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-mvsy8es8/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpzzj2awct/output.tsv.biom /home/test12/scr4-test1/temp/tmpzzj2awct/track.tsv /home/test12/scr4-test1/temp/tmpzzj2awct 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-9tbq9434/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp21t7tboh/output.tsv.biom /home/test12/scr4-test1/temp/tmp21t7tboh/track.tsv /home/test12/scr4-test1/temp/tmp21t7tboh 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-opq32q9v/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpipict_40/output.tsv.biom /home/test12/scr4-test1/temp/tmpipict_40/track.tsv /home/test12/scr4-test1/temp/tmpipict_40 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-iknv8kse/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp0jjtcyc9/output.tsv.biom /home/test12/scr4-test1/temp/tmp0jjtcyc9/track.tsv /home/test12/scr4-test1/temp/tmp0jjtcyc9 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-5wvq4xvr/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmppcmt8xrb/output.tsv.biom /home/test12/scr4-test1/temp/tmppcmt8xrb/track.tsv /home/test12/scr4-test1/temp/tmppcmt8xrb 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-fnmu2sza/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp2gg9mye1/output.tsv.biom /home/test12/scr4-test1/temp/tmp2gg9mye1/track.tsv /home/test12/scr4-test1/temp/tmp2gg9mye1 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)Loading required package: Rcpp

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-plhy_j3/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpyq76ump/output.tsv.biom /home/test12/scr4-test1/temp/tmpyq76ump_/track.tsv /home/test12/scr4-test1/temp/tmpyq76ump_ 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-u52hhet6/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpwbwq3u4h/output.tsv.biom /home/test12/scr4-test1/temp/tmpwbwq3u4h/track.tsv /home/test12/scr4-test1/temp/tmpwbwq3u4h 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-6g8q03cl/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpg42c3dw0/output.tsv.biom /home/test12/scr4-test1/temp/tmpg42c3dw0/track.tsv /home/test12/scr4-test1/temp/tmpg42c3dw0 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-ftgam1va/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp8ty_u96i/output.tsv.biom /home/test12/scr4-test1/temp/tmp8ty_u96i/track.tsv /home/test12/scr4-test1/temp/tmp8ty_u96i 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-csx4v9me/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmppqeq3n9a/output.tsv.biom /home/test12/scr4-test1/temp/tmppqeq3n9a/track.tsv /home/test12/scr4-test1/temp/tmppqeq3n9a 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-nh4ydv2l/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpi4zyyg7_/output.tsv.biom /home/test12/scr4-test1/temp/tmpi4zyyg7_/track.tsv /home/test12/scr4-test1/temp/tmpi4zyyg7_ 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-f_bitfru/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpht9gqygf/output.tsv.biom /home/test12/scr4-test1/temp/tmpht9gqygf/track.tsv /home/test12/scr4-test1/temp/tmpht9gqygf 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-s2p8jb4g/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpewcwwqn3/output.tsv.biom /home/test12/scr4-test1/temp/tmpewcwwqn3/track.tsv /home/test12/scr4-test1/temp/tmpewcwwqn3 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering

colinbrislawn · May 30, 2023, 4:21pm

Good afternoon DEEPAK,

Thank you for your patience while we got to this question.

Is this from a single, very large NovaSeq run, or is this a collection of multiple different Illumina runs you have imported into a single big demux.qza file?

I ask because DADA2 is designed to process each sequencing run separately and make stable ASV tables you can merge later. Processing your 100GB files in chunks could solve your storage issue, and will also give you better results!

deepak · May 30, 2023, 5:21pm

@colinbrislawn Yes this is a large NovaSeq run and after importing the data in qiime, it is giving me 110Gb file. I have 32 samples (~3-4gb each). I am struggling on this DADA2 step since last moth still no progress. Last time I asked (Can removal of RNA-seq human reads expedite the analysis? (Especially DADA2)) about individual sample analysis.
Can I analyze samples in 2-3 batches and later merge the feature table?

" Processing your 100GB files in chunks could solve your storage issue, and will also give you better results!"
What is the command to do this ? is it simple splitting of file?

Kindly help me to troubleshoot the analysis for this data. In upcoming months I have to analyze more batches of samples.

colinbrislawn · May 30, 2023, 6:40pm

Sort of. The recommended way is to run DADA2 one time for each Illumina run, then merge output feature tables

DADA2 is designed to produce very stable ASVs, so dividing your input data should work well even if all samples were on the same Illumina run! You could split the NovaSeq run into three separate batches, then import and run DADA2 on each.

Yes, I think you could use this plugin to filter out human reads: filter-reads: Filter demultiplexed sequences by alignment to reference database. — QIIME 2 2023.5.1 documentation

Big data may need big computers. If you are not already running DADA2 on university HPC, I would highly recommend that!

deepak · May 30, 2023, 8:45pm

Yes, I am running my jobs on HPC (36 CPU and 30GB RAM per CPU). This is a bigmam Node and time limit is 48Hr so I couldn't complete. Before 48Hr it gives me an error... OUT_OF_MEMORY.
I am hoping your idea for splitting the data may work.
Thanks

deepak · June 1, 2023, 3:11pm

Hi
I tried with two samples (~9GB total demux.gza) but still facing the same problem, For DADA2 I gave 36CPU and 30GB RAM/CPU. It ran for 3hrs and process killed due to OUT_OF_MEMORY. I have a 10TB space on drive and it was consuming 950GB as a temporary space during the process.

Error output file:

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-o60jnscy/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp8sq9f4rn/output.tsv.biom /home/test12/scr4-test17/temp/tmp8sq9f4rn/track.tsv /home/test12/scr4-test17/temp/tmp8sq9f4rn 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-iqyfa_ht/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpzj6rew2o/output.tsv.biom /home/test12/scr4-test17/temp/tmpzj6rew2o/track.tsv /home/test12/scr4-test17/temp/tmpzj6rew2o 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-efepbqff/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmphzretwc0/output.tsv.biom /home/test12/scr4-test17/temp/tmphzretwc0/track.tsv /home/test12/scr4-test17/temp/tmphzretwc0 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-mtkzh5vd/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpox49xbi1/output.tsv.biom /home/test12/scr4-test17/temp/tmpox49xbi1/track.tsv /home/test12/scr4-test17/temp/tmpox49xbi1 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-xw7v9i31/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpmwdjlpby/output.tsv.biom /home/test12/scr4-test17/temp/tmpmwdjlpby/track.tsv /home/test12/scr4-test17/temp/tmpmwdjlpby 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-46_veqoj/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpv0hf6fye/output.tsv.biom /home/test12/scr4-test17/temp/tmpv0hf6fye/track.tsv /home/test12/scr4-test17/temp/tmpv0hf6fye 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-b4zg3ayd/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpiflr8qz1/output.tsv.biom /home/test12/scr4-test17/temp/tmpiflr8qz1/track.tsv /home/test12/scr4-test17/temp/tmpiflr8qz1 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-uwufqa_g/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpwoigvkqs/output.tsv.biom /home/test12/scr4-test17/temp/tmpwoigvkqs/track.tsv /home/test12/scr4-test17/temp/tmpwoigvkqs 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-la96j7u4/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpcj68fklc/output.tsv.biom /home/test12/scr4-test17/temp/tmpcj68fklc/track.tsv /home/test12/scr4-test17/temp/tmpcj68fklc 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-xouefzx5/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmptxpq2zd6/output.tsv.biom /home/test12/scr4-test17/temp/tmptxpq2zd6/track.tsv /home/test12/scr4-test17/temp/tmptxpq2zd6 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-3964pypc/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpd2von1u9/output.tsv.biom /home/test12/scr4-test17/temp/tmpd2von1u9/track.tsv /home/test12/scr4-test17/temp/tmpd2von1u9 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-ees_iyox/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp2h2jkkyc/output.tsv.biom /home/test12/scr4-test17/temp/tmp2h2jkkyc/track.tsv /home/test12/scr4-test17/temp/tmp2h2jkkyc 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-b0rstb72/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp_dxy9_9v/output.tsv.biom /home/test12/scr4-test17/temp/tmp_dxy9_9v/track.tsv /home/test12/scr4-test17/temp/tmp_dxy9_9v 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-ix2nd_9e/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpj48dc26o/output.tsv.biom /home/test12/scr4-test17/temp/tmpj48dc26o/track.tsv /home/test12/scr4-test17/temp/tmpj48dc26o 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-_hwwo4cf/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp4213k6tn/output.tsv.biom /home/test12/scr4-test17/temp/tmp4213k6tn/track.tsv /home/test12/scr4-test17/temp/tmp4213k6tn 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-nhwo_l14/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpwpseeaab/output.tsv.biom /home/test12/scr4-test17/temp/tmpwpseeaab/track.tsv /home/test12/scr4-test17/temp/tmpwpseeaab 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-plecegvb/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpc31mr5j8/output.tsv.biom /home/test12/scr4-test17/temp/tmpc31mr5j8/track.tsv /home/test12/scr4-test17/temp/tmpc31mr5j8 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-3befo0bu/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmprckdme2b/output.tsv.biom /home/test12/scr4-test17/temp/tmprckdme2b/track.tsv /home/test12/scr4-test17/temp/tmprckdme2b 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-g__zpq7m/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpzlsmibv6/output.tsv.biom /home/test12/scr4-test17/temp/tmpzlsmibv6/track.tsv /home/test12/scr4-test17/temp/tmpzlsmibv6 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-8p2ay4m9/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp3lsl_3c6/output.tsv.biom /home/test12/scr4-test17/temp/tmp3lsl_3c6/track.tsv /home/test12/scr4-test17/temp/tmp3lsl_3c6 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-kffxfbq3/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmptv7k6rs8/output.tsv.biom /home/test12/scr4-test17/temp/tmptv7k6rs8/track.tsv /home/test12/scr4-test17/temp/tmptv7k6rs8 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-dcjbvlux/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpqbskbyj8/output.tsv.biom /home/test12/scr4-test17/temp/tmpqbskbyj8/track.tsv /home/test12/scr4-test17/temp/tmpqbskbyj8 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-l7hrz6wv/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpwmpbd2cq/output.tsv.biom /home/test12/scr4-test17/temp/tmpwmpbd2cq/track.tsv /home/test12/scr4-test17/temp/tmpwmpbd2cq 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-6hnthq42/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp9_nt4fyb/output.tsv.biom /home/test12/scr4-test17/temp/tmp9_nt4fyb/track.tsv /home/test12/scr4-test17/temp/tmp9_nt4fyb 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-leim_hto/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp9jiwcs7k/output.tsv.biom /home/test12/scr4-test17/temp/tmp9jiwcs7k/track.tsv /home/test12/scr4-test17/temp/tmp9jiwcs7k 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-nnwf82sm/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp2s70kcf2/output.tsv.biom /home/test12/scr4-test17/temp/tmp2s70kcf2/track.tsv /home/test12/scr4-test17/temp/tmp2s70kcf2 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-447nuotq/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpoomp658h/output.tsv.biom /home/test12/scr4-test17/temp/tmpoomp658h/track.tsv /home/test12/scr4-test17/temp/tmpoomp658h 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-5dt6x62t/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp26ehl5ic/output.tsv.biom /home/test12/scr4-test17/temp/tmp26ehl5ic/track.tsv /home/test12/scr4-test17/temp/tmp26ehl5ic 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-4r703kqq/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp8pe7qukh/output.tsv.biom /home/test12/scr4-test17/temp/tmp8pe7qukh/track.tsv /home/test12/scr4-test17/temp/tmp8pe7qukh 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-3fbo64jj/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmplr_4rqbs/output.tsv.biom /home/test12/scr4-test17/temp/tmplr_4rqbs/track.tsv /home/test12/scr4-test17/temp/tmplr_4rqbs 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-nh87bjlc/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpli_u58kz/output.tsv.biom /home/test12/scr4-test17/temp/tmpli_u58kz/track.tsv /home/test12/scr4-test17/temp/tmpli_u58kz 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-lrl95txe/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp18314bf3/output.tsv.biom /home/test12/scr4-test17/temp/tmp18314bf3/track.tsv /home/test12/scr4-test17/temp/tmp18314bf3 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-8978crs0/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpunh5shmw/output.tsv.biom /home/test12/scr4-test17/temp/tmpunh5shmw/track.tsv /home/test12/scr4-test17/temp/tmpunh5shmw 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-oxy217dy/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmpt45euw0p/output.tsv.biom /home/test12/scr4-test17/temp/tmpt45euw0p/track.tsv /home/test12/scr4-test17/temp/tmpt45euw0p 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-x40gwrlz/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmptun1z1co/output.tsv.biom /home/test12/scr4-test17/temp/tmptun1z1co/track.tsv /home/test12/scr4-test17/temp/tmptun1z1co 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test17/temp/qiime2-archive-9e2jqoh9/b297535a-807b-47cc-a74e-f6bd00e265a9/data /home/test12/scr4-test17/temp/tmp6ca72krn/output.tsv.biom /home/test12/scr4-test17/temp/tmp6ca72krn/track.tsv /home/test12/scr4-test17/temp/tmp6ca72krn 0 0 2.0 2 Inf independent consensus 1.0 1 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
Filtering ..
Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

..

Learning Error Rates

Traceback (most recent call last):

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 195, in _denoise_single

run_commands([cmd])

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 36, in run_commands

subprocess.run(cmd, check=True)

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/subprocess.py", line 516, in run

raise CalledProcessError(retcode, process.args,

subprocess.CalledProcessError: Command '['run_dada_single.R', '/home/test12/scr4-test17/temp/qiime2-archive-mtkzh5vd/b297535a-807b-47cc-a74e-f6bd00e265a9/data', '/home/test12/scr4-test17/temp/tmpox49xbi1/output.tsv.biom', '/home/test12/scr4-test17/temp/tmpox49xbi1/track.tsv', '/home/test12/scr4-test17/temp/tmpox49xbi1', '0', '0', '2.0', '2', 'Inf', 'independent', 'consensus', '1.0', '1', '1000000', 'NULL', '16']' died with <Signals.SIGKILL: 9>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/q2cli/commands.py", line 339, in call

results = action(**arguments)

File "", line 2, in denoise_single

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 245, in bound_callable

outputs = self.callable_executor(scope, callable_args,

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 391, in callable_executor

output_views = self._callable(**view_args)

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 218, in denoise_single

return _denoise_single(

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 204, in _denoise_single

raise Exception("An error was encountered while running DADA2"

Exception: An error was encountered while running DADA2 in R (return code -9), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code -9), please inspect stdout and stderr to learn more.

See above for debug info.

[0mTraceback (most recent call last):

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 195, in _denoise_single

run_commands([cmd])

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 36, in run_commands

subprocess.run(cmd, check=True)

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/subprocess.py", line 516, in run

raise CalledProcessError(retcode, process.args,

subprocess.CalledProcessError: Command '['run_dada_single.R', '/home/test12/scr4-test17/temp/qiime2-archive-xouefzx5/b297535a-807b-47cc-a74e-f6bd00e265a9/data', '/home/test12/scr4-test17/temp/tmptxpq2zd6/output.tsv.biom', '/home/test12/scr4-test17/temp/tmptxpq2zd6/track.tsv', '/home/test12/scr4-test17/temp/tmptxpq2zd6', '0', '0', '2.0', '2', 'Inf', 'independent', 'consensus', '1.0', '1', '1000000', 'NULL', '16']' died with <Signals.SIGKILL: 9>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/q2cli/commands.py", line 339, in call

results = action(**arguments)

File "", line 2, in denoise_single

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 245, in bound_callable

outputs = self.callable_executor(scope, callable_args,

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 391, in callable_executor

output_views = self._callable(**view_args)

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 218, in denoise_single

return _denoise_single(

File "/data/apps/extern/anaconda/envs/qiime2/2022.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 204, in _denoise_single

raise Exception("An error was encountered while running DADA2"

Exception: An error was encountered while running DADA2 in R (return code -9), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code -9), please inspect stdout and stderr to learn more.

See above for debug info.

[0m--------------------------------------------------------------------------

Primary job terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

mpirun detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

Process name: [[65029,1],9]

Exit code: 1

real 194m34.497s

user 189m3.667s

sys 2m26.437s

slurmstepd: error: Detected 1 oom-kill event(s) inStepId=16404634.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

lizgehret · June 1, 2023, 4:55pm

Hi @deepak,

This error is still due to not enough memory allocation - you may try splitting your data even further (maybe into each individual sample), and allocating more RAM on your HPC if possible.

Cheers

deepak · June 1, 2023, 5:14pm

Hi @lizgehret
Thanks for the reply,
These were only two samples ~4GB each and I gave 1 Node 36CPU with 30GB RAM per CPU. I have 32 sample, I started with these but couldn't complete because of same error earlier. At that time it took 4.5TB of temporary space and process kill due to out of memory.
Now I will do individual sample let see what will happen.

Deepak

colinbrislawn · June 1, 2023, 5:57pm

That should be plenty of RAM for 4 GB of data.

I wonder if something is wrong with the HPC config...

I've made the mistake of requesting 16 tasks per node when running --p-threads 16, only to discover that the HPC system divided that node into 16 parts and my qiime command was using 1/16 of the full node.

Changing the config to use 1 task per node gave me access to the full node and fixed the problem.

deepak · June 1, 2023, 6:20pm

I am using following slurm script for the analysis:

#!/bin/bash -l
#SBATCH --job-name="DADA2"
#SBATCH --partition=bigmem
#SBATCH --time 00-48:00:00
#SBATCH -N 1
#SBATCH --ntasks-per-node=36
#SBATCH --export=ALL
ml qiime2
time mpirun qiime dada2 denoise-single --i-demultiplexed-seqs single-end-demux.qza --p-n-threads 0 --p-trim-left 0 --p-trunc-len 0 --o-representative-sequences rep-seqs-dada2.qza --o-table table-dada2.qza --o-denoising-stats stats-dada2.qza --verbose

In the output log files it is showing processing of 36CPU (36 lines of Filtering, Error...) and 30GB RAM is by default for per CPU. Is there any issue with this command?

colinbrislawn · June 1, 2023, 6:30pm

Ah!

I'm never sure exactly how these setting work, but try this:

#!/bin/bash -l
#SBATCH --job-name="DADA2"
#SBATCH --partition=bigmem
#SBATCH --time 00-48:00:00
#SBATCH -N 1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=36
#SBATCH --export=ALL
ml qiime2
time qiime dada2 denoise-single --i-demultiplexed-seqs single-end-demux.qza --p-n-threads 0 --p-trim-left 0 --p-trunc-len 0 --o-representative-sequences rep-seqs-dada2.qza --o-table table-dada2.qza --o-denoising-stats stats-dada2.qza --verbose

I think cpus-per-task is helpful here because we are relying on Qiime2 and DADA2 to paralize this command, we don't want SLURM to run it 16 times

I've also removed mpirun as I don't think Qiime is compatible with that

deepak · June 1, 2023, 7:02pm

I will try with this command. Thanks

system · July 3, 2023, 1:02am

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.