How many nodes can QIIME2 use?

Hi everyone!

My lab mate and I are still pretty new to QIIME2, and although we have made great progress in learning, we reached a bit of a road block when my MacBook Air couldn’t handle the DADA2 step. We recently gained access to MARCC, the Maryland Advanced Research Computing Center, and we have access to various partitions with different numbers of cores available and different amounts of memory.

We have 64 fastq files being processed.

How much memory and how many cores would be recommended for this? We tried the “lrgmem” partition, which has 120 GB of memory by default (but it can go up to 1024 GB of memory). We also set it to 10 cores. This failed and gave us error code 9, which I believe means that we need more memory.

Thoughts? Thanks in advance!

1 Like

Hi @cjone228,

I will warn you straight away that I dont have the right answer, but maybe you can give more information that will help us help you better. Are 64 fastq files from 64 samples? Were they a single MiSeq run, or were they part of a larger sequencing batch that you have a subset of?

These are also important in terms of how much memory or time you need to train the model.

Best,
Justine

Hi all,

My lab mate and I are trying to run the DADA2 step using qiime2-2019.7 through the super computer (MARCC) that we have access to through our institution. Our job failed and gave us return code 1. What do we need to change about our code to get it to successfully run? We do have the option to increase memory and # of cores, but we don't think that is the issue here.

See below text for the contents of the output from the verbose command:

(base) [[email protected]@bc-login02 ~]$ cat slurm-37205833.out
R version 3.5.1 (2018-07-02)
Loading required package: Rcpp
DADA2: 1.10.0 / Rcpp: 1.0.2 / RcppParallel: 4.4.3

  1. Filtering Error in writeFastq(fqF, fout[[1]], "a", compress = compress) :
    failed to write record 55732
    Execution halted
    Traceback (most recent call last):
    File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/denoise.py", line 234, in denoise_paired
    run_commands([cmd])
    File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/denoise.py", line 36, in run_commands
    subprocess.run(cmd, check=True)
    File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['run_dada_paired.R', '/home-1/[email protected]/scratch/tmpcx5fbjm
    /forward', '/home-1/[email protected]/scratch/tmpcx5fbjm
    /reverse', '/home-1/[email protected]/scratch/tmpcx5fbjm_/output.tsv.biom', '/home-1/[email protected]/scratch/tmpcx5fbjm_/track.tsv', '/home-1/[email protected]/scratch/tmpcx5fbjm_/filt_f', '/home-1/[email protected]/scratch/tmpcx5fbjm_/filt_r', '93', '93', '3', '0', '2.0', '2.0', '2', 'consensus', '1.0', '1', '10000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py", line 327, in call
results = action(**arguments)
File "</home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-459>", line 2, in denoise_paired
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 240, in bound_callable
output_types, provenance)
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 383, in callable_executor
output_views = self._callable(**view_args)
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 249, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

See above for debug info.
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /home-1/[email protected]/scratch/tmpcx5fbjm_/forward /home-1/[email protected]/scratch/tmpcx5fbjm_/reverse /home-1/[email protected]/scratch/tmpcx5fbjm_/output.tsv.biom /home-1/[email protected]/scratch/tmpcx5fbjm_/track.tsv /home-1/[email protected]/scratch/tmpcx5fbjm_/filt_f /home-1/[email protected]/scratch/tmpcx5fbjm_/filt_r 93 93 3 0 2.0 2.0 2 consensus 1.0 1 10000


Here is the sequence quality plot that we based our trimming amounts off of:

And here is the code that we input for our job in MARCC:

#!/bin/bash
#SBATCH -p lrgmem
#SBATCH -c 10
#SBATCH -t 72:00:00
#SBATCH --mail-type=all
#SBATCH [email protected]
#SBATCH --job-name=DADA2-30-barcode-illumina-lrgmem

set -e
export TMPDIR=$HOME/scratch

qiime dada2 denoise-paired
--i-demultiplexed-seqs ~/work/Illumina_Data/paired-end-demux.qza
--p-trim-left-f 3
--p-trim-left-r 0
--p-trunc-len-f 93
--p-trunc-len-r 93
--p-n-reads-learn 10000
--o-table table.qza
--o-representative-sequences rep-seqs.qza
--o-denoising-stats denoising-stats.qza
--verbose


1 Like

Hi Justine,

Thanks for your help! They are paired end reads, with half being forward and half being reverse files. They were all completed in a single run on an Illumina HiSeq.

Best,
Carli

1 Like

Heres my sbatch header:

#!/bin/bash
#SBATCH --job-name=Name.of.job.here # Job name
#SBATCH --mail-type=BEGIN,END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH [email protected] # Where to send mail
#SBATCH --ntasks=16 # Run on a single CPU
#SBATCH --mem=64gb # Job memory request
#SBATCH --time=72:05:00 # Time limit hrs:min:sec
#SBATCH --output=serial_test_%j.log # Standard output and error log

You can try this to help. I don’t think that the cluster we use normally supports parallelization per se. Ben

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.