Running QIIME2 2018.11 in cluster

[Mike26]

Hello All. I wish to classify my sequences using silva 132 database. I cannot run silva on local computer since silva always needs high memory. being said, i install qiime2 in our university server usimg miniconda3. ‘Source activate’ seems not working with standard slum. I also tried to specify the directory with " ~/.conda/envs/qiime2-2018.11/bin/qiime" but were also not working with slurm.

Here is the bash script:

#!/bin/bash

#SBATCH --account=coral_resilience
#SBATCH --job-name=train_classifier
#SBATCH --qos=shortjobs
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=100000
#SBATCH --time=24:00:00
#SBATCH --workdir=/home/users/mlnada/silva
#SBATCH --output=train_classifier.out
#SBATCH --error=train_classifier.err

Load module

module load miniconda3

Activate conda environment

source activate qiime2-2018.11

Command

qiime feature-classifier fit-classifier-naive-bayes --i-reference-reads /home/users/mlnada/silva_refseqV3V4.qza --i-reference-taxonomy home/users/mlnada/silvataxonomy.qza --o-classifier silva_classifier.qza

If anyone of you has experience running qiime2 in cluster, Can you provide me a work around on this? All i need is to classify the sequences using silva in cluster, then, I can do the remaining analysis on my local computer.

Thank you very much.

[Nicholas_Bokulich]

Hi @Mike26,
You should contact your cluster admin about this if you think this is really a slurm issue — conda should work on slurm and everything you are doing “looks” fine (but I don’t know what environments are installed). You could also just use conda info --env to discover which environments are available to load… my guess is that your qiime2-2018.11 environment may have a different name?

Good luck!

[devonorourke]

Totally agree with @Nicholas_Bokulich about looking to see where and whether or not you have any conda environment ready to go.
However, I’d also add that what can be confusing is that on some compute clusters the administrator will have already set up a QIIME2 environment for you to function in, and that may not by default be accessible with a simple conda command to run. That is, if you log in and already have access to the anaconda or miniconda suite of tools, great. But if you don’t, you may need to load something that gives you access to conda and the like…
For example, my compute cluster has set up anaconda to be accessible only if you load it through a module command, so when I first log in, the way I have to get to loading QIIME is as follows:

module purge
module load anaconda
source activate qiime2-2018.11

If you were to look in my $HOME/.conda/envs directory, there would be a path to my locally installed qiime package.
If you didn’t install yours locally, you may need to look elsewhere to find out where that package is installed.

Quickest and best advice: bring coffee | beer to your admin and ask them to help you for 2 minutes. I bet it’s a quick fix.

[Mike26]

Hello All.

conda info --env gives the following:

qiime2-2018.11 /home/users/mlnada/.conda/envs/ qiime2-2018.11
base * /home/users/mlnada/anaconda3

I change the module from miniconda3 to anaconda3, and it seems to work with slurm. I am now waiting for resource allocation to run my command.