Another Plugin Error
Hi Everyone,
I have been struggling with this code for a few months now, scouring the forums for any answers. I have a lot of data with my original file being 40GB. I ended up subsetting this data into 3 groups based on sample location. Using all the same code and version, I’ve had two run perfect, and the other fail repeatedly. I’ve tried different variations of this code and different versions of qiime2 to see what may the issue with no success. I always get some version of a plugin error.
The demultiplex step with this data ran fine. I used fastq.gz files. My sample type is --type 'SampleData[PairedEndSequencesWithQuality] and input format is PairedEndFastqManifestPhred33V2 (This may not matter I’m just trying to add at much information that may help).
This is the script of the batch that keeps failing. Out of what I’ve requested (I added more with every failure) I am only using about 50gb of memory and 17 cpus. It always fails after 4 days of running. I would like to keep the version 2025.7 as the rest of my data is processed in this version.
This is the error I received after my last failure 4/5/2026
/packages/qiime2/2025.7/lib/python3.10/site-packages/q2_demux/_summarize/_visualizer.py:11: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
R version 4.3.3 (2024-02-29)
Loading required package: Rcpp
DADA2: 1.30.0 / Rcpp: 1.1.0 / RcppParallel: 5.1.9
2) Filtering .............
3) Learning Error Rates
1048329300 total bases in 3494431 reads from 1 samples will be used for learning the error rates.
1048329300 total bases in 3494431 reads from 1 samples will be used for learning the error rates.
3) Denoise samples .............
..........Error in table(pairdf$forward, pairdf$reverse) :
attempt to make a table with >= 2^31 elements
Calls: mergePairs -> lapply -> FUN -> table
5: stop("attempt to make a table with >= 2^31 elements")
4: table(pairdf$forward, pairdf$reverse)
3: FUN(X[[i]], ...)
2: lapply(seq_along(dadaF), function(i) {
mapF <- getDerep(derepF[[i]])$map
mapR <- getDerep(derepR[[i]])$map
if (!(is.integer(mapF) && is.integer(mapR)))
stop("Incorrect format of $map in derep-class arguments.")
if (!(length(mapF) == length(mapR) && max(mapF, na.rm = TRUE) ==
length(dadaF[[i]]$map) && max(mapR, na.rm = TRUE) ==
length(dadaR[[i]]$map))) {
stop("Non-corresponding derep-class and dada-class objects.")
}
rF <- dadaF[[i]]$map[mapF]
rR <- dadaR[[i]]$map[mapR]
pairdf <- data.frame(sequence = "", abundance = 0, forward = rF,
reverse = rR)
ups <- unique(pairdf)
keep <- !is.na(ups$forward) & !is.na(ups$reverse)
ups <- ups[keep, ]
if (nrow(ups) == 0) {
outnames <- c("sequence", "abundance", "forward", "reverse",
"nmatch", "nmismatch", "nindel", "prefer", "accept")
ups <- data.frame(matrix(ncol = length(outnames), nrow = 0))
names(ups) <- outnames
if (verbose) {
message("No paired-reads (in ZERO unique pairings) successfully merged out of ",
nrow(pairdf), " pairings) input.")
}
return(ups)
}
else {
Funqseq <- unname(as.character(dadaF[[i]]$clustering$sequence[ups$forward]))
Runqseq <- rc(unname(as.character(dadaR[[i]]$clustering$sequence[ups$reverse])))
if (justConcatenate == TRUE) {
ups$sequence <- mapply(function(x, y) paste0(x, "NNNNNNNNNN",
y), Funqseq, Runqseq, SIMPLIFY = FALSE)
ups$nmatch <- 0
ups$nmismatch <- 0
ups$nindel <- 0
ups$prefer <- NA
ups$accept <- TRUE
}
else {
tmp <- getDadaOpt(c("MATCH", "MISMATCH", "GAP_PENALTY"))
if (maxMismatch == 0) {
setDadaOpt(MATCH = 1L, MISMATCH = -64L, GAP_PENALTY = -64L)
}
else {
setDadaOpt(MATCH = 1L, MISMATCH = -8L, GAP_PENALTY = -8L)
}
alvecs <- mapply(function(x, y) nwalign(x, y, band = -1,
...), Funqseq, Runqseq, SIMPLIFY = FALSE)
setDadaOpt(tmp)
outs <- t(sapply(alvecs, function(x) C_eval_pair(x[1],
x[2])))
ups$nmatch <- outs[, 1]
ups$nmismatch <- outs[, 2]
ups$nindel <- outs[, 3]
ups$prefer <- 1 + (dadaR[[i]]$clustering$n0[ups$reverse] >
dadaF[[i]]$clustering$n0[ups$forward])
ups$accept <- (ups$nmatch >= minOverlap) & ((ups$nmismatch +
ups$nindel) <= maxMismatch)
ups$sequence <- mapply(C_pair_consensus, sapply(alvecs,
`[`, 1), sapply(alvecs, `[`, 2), ups$prefer,
trimOverhang)
}
tab <- table(pairdf$forward, pairdf$reverse)
ups$abundance <- tab[cbind(ups$forward, ups$reverse)]
ups$sequence[!ups$accept] <- ""
propagateCol <- propagateCol[propagateCol %in% colnames(dadaF[[i]]$clustering)]
for (col in propagateCol) {
ups[, paste0("F.", col)] <- dadaF[[i]]$clustering[ups$forward,
col]
ups[, paste0("R.", col)] <- dadaR[[i]]$clustering[ups$reverse,
col]
}
ups <- ups[order(ups$abundance, decreasing = TRUE), ]
rownames(ups) <- NULL
if (verbose) {
message(sum(ups$abundance[ups$accept]), " paired-reads (in ",
sum(ups$accept), " unique pairings) successfully merged out of ",
sum(ups$abundance), " (in ", nrow(ups), " pairings) input.")
}
if (!returnRejects) {
ups <- ups[ups$accept, ]
}
if (any(duplicated(ups$sequence))) {
message("Duplicate sequences in merged output.")
}
return(ups)
}
})
1: mergePairs(ddsF[[j]], drpF, ddsR[[j]], drpR, minOverlap = minOverlap,
maxMismatch = maxMergeMismatch, trimOverhang = trimOverhang)
Traceback (most recent call last):
File "/packages/qiime2/2025.7/lib/python3.10/site-packages/q2_dada2/_denoise.py", line 348, in denoise_paired
run_commands([cmd])
File "/packages/qiime2/2025.7/lib/python3.10/site-packages/qiime2/plugin/util.py", line 65, in run_commands
subprocess.run(cmd, check=True)
File "/packages/qiime2/2025.7/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_dada.R', '--input_directory', '/tmp/jmc2729/27751559/tmp87crxoet/forward', '--input_directory_reverse', '/tmp/jmc2729/27751559/tmp87crxoet/reverse', '--output_path', '/tmp/jmc2729/27751559/tmp87crxoet/output.tsv.biom', '--output_track', '/tmp/jmc2729/27751559/tmp87crxoet/track.tsv', '--filtered_directory', '/tmp/jmc2729/27751559/tmp87crxoet/filt_f', '--filtered_directory_reverse', '/tmp/jmc2729/27751559/tmp87crxoet/filt_r', '--truncation_length', '300', '--truncation_length_reverse', '300', '--trim_left', '0', '--trim_left_reverse', '0', '--max_expected_errors', '2.0', '--max_expected_errors_reverse', '2.0', '--truncation_quality_score', '2', '--min_overlap', '12', '--max_merge_mismatch', '0', '--trim_overhang', 'False', '--pooling_method', 'independent', '--chimera_method', 'consensus', '--min_parental_fold', '1.0', '--allow_one_off', 'False', '--num_threads', '20', '--learn_min_reads', '1000000']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/packages/qiime2/2025.7/lib/python3.10/site-packages/q2cli/commands.py", line 529, in __call__
results = self._execute_action(
File "/packages/qiime2/2025.7/lib/python3.10/site-packages/q2cli/commands.py", line 601, in _execute_action
results = action(**arguments)
File "<decorator-gen-908>", line 2, in denoise_paired
File "/packages/qiime2/2025.7/lib/python3.10/site-packages/qiime2/sdk/action.py", line 221, in bound_callable
outputs = self._callable_executor_(
File "/packages/qiime2/2025.7/lib/python3.10/site-packages/qiime2/sdk/action.py", line 362, in _callable_executor_
output_views = self._callable(**view_args)
File "/packages/qiime2/2025.7/lib/python3.10/site-packages/q2_dada2/_denoise.py", line 361, in denoise_paired
raise Exception("An error was encountered while running DADA2"
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
Plugin error from dada2:
An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
See above for debug info.
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada.R --input_directory /tmp/jmc2729/27751559/tmp87crxoet/forward --input_directory_reverse /tmp/jmc2729/27751559/tmp87crxoet/reverse --output_path /tmp/jmc2729/27751559/tmp87crxoet/output.tsv.biom --output_track /tmp/jmc2729/27751559/tmp87crxoet/track.tsv --filtered_directory /tmp/jmc2729/27751559/tmp87crxoet/filt_f --filtered_directory_reverse /tmp/jmc2729/27751559/tmp87crxoet/filt_r --truncation_length 300 --truncation_length_reverse 300 --trim_left 0 --trim_left_reverse 0 --max_expected_errors 2.0 --max_expected_errors_reverse 2.0 --truncation_quality_score 2 --min_overlap 12 --max_merge_mismatch 0 --trim_overhang False --pooling_method independent --chimera_method consensus --min_parental_fold 1.0 --allow_one_off False --num_threads 20 --learn_min_reads 1000000
If you want to see other plugin errors I’ve received, I’ve attached their paths below:
Debug info has been saved to /tmp/jmc2729/27231086/qiime2-q2cli-err-43jmp0zb.log
Debug info has been saved to /tmp/jmc2729/27290023/qiime2-q2cli-err-3pc748z2.log
If anyone has any insight into what this error means or how I can go about running my data, I would appreciate it very much. Please reach out for more information or with more questions.
Thank you!
Jo Corrales - Speer Lab