My lab mate and I are still pretty new to QIIME2, and although we have made great progress in learning, we reached a bit of a road block when my MacBook Air couldn’t handle the DADA2 step. We recently gained access to MARCC, the Maryland Advanced Research Computing Center, and we have access to various partitions with different numbers of cores available and different amounts of memory.
We have 64 fastq files being processed.
How much memory and how many cores would be recommended for this? We tried the “lrgmem” partition, which has 120 GB of memory by default (but it can go up to 1024 GB of memory). We also set it to 10 cores. This failed and gave us error code 9, which I believe means that we need more memory.
I will warn you straight away that I dont have the right answer, but maybe you can give more information that will help us help you better. Are 64 fastq files from 64 samples? Were they a single MiSeq run, or were they part of a larger sequencing batch that you have a subset of?
These are also important in terms of how much memory or time you need to train the model.
My lab mate and I are trying to run the DADA2 step using qiime2-2019.7 through the super computer (MARCC) that we have access to through our institution. Our job failed and gave us return code 1. What do we need to change about our code to get it to successfully run? We do have the option to increase memory and # of cores, but we don't think that is the issue here.
See below text for the contents of the output from the verbose command:
Filtering Error in writeFastq(fqF, fout[[1]], "a", compress = compress) :
failed to write record 55732
Execution halted
Traceback (most recent call last):
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/denoise.py", line 234, in denoise_paired
run_commands([cmd])
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/denoise.py", line 36, in run_commands
subprocess.run(cmd, check=True)
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run_dada_paired.R', '/home-1/[email protected]/scratch/tmpcx5fbjm/forward', '/home-1/[email protected]/scratch/tmpcx5fbjm/reverse', '/home-1/[email protected]/scratch/tmpcx5fbjm_/output.tsv.biom', '/home-1/[email protected]/scratch/tmpcx5fbjm_/track.tsv', '/home-1/[email protected]/scratch/tmpcx5fbjm_/filt_f', '/home-1/[email protected]/scratch/tmpcx5fbjm_/filt_r', '93', '93', '3', '0', '2.0', '2.0', '2', 'consensus', '1.0', '1', '10000']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py", line 327, in call
results = action(**arguments)
File "</home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-459>", line 2, in denoise_paired
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 240, in bound_callable
output_types, provenance)
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 383, in callable_executor
output_views = self._callable(**view_args)
File "/home-1/[email protected]/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 249, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
Plugin error from dada2:
An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
See above for debug info.
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Thanks for your help! They are paired end reads, with half being forward and half being reverse files. They were all completed in a single run on an Illumina HiSeq.
#!/bin/bash #SBATCH --job-name=Name.of.job.here # Job name #SBATCH --mail-type=BEGIN,END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL) #SBATCH[email protected] # Where to send mail #SBATCH --ntasks=16 # Run on a single CPU #SBATCH --mem=64gb # Job memory request #SBATCH --time=72:05:00 # Time limit hrs:min:sec #SBATCH --output=serial_test_%j.log # Standard output and error log
You can try this to help. I don't think that the cluster we use normally supports parallelization per se. Ben
