qiime2 2024.10: slow alpha rarefaction

Technical Support
,
1

Hi everyone,

I’m having an issue with qiime diversity alpha-rarefaction. I ran the same command on both my local machine and a slurm cluster. The job took only about 1.5 hours to finish on my local machine, while it took over 9 hours on slurm. I use qiime2 version 2024.10

I allocated generous resources to the slurm job, so I don’t think it is a resource limitation. I’m wondering if there could be other factors causing this significant difference in runtime.

Does anyone have any ideas about what might be causing this? Thanks in advance for any suggestions.

2

Hello @mart_rach, welcome to the qiime2 forum!

One thing to keep in mind is that allocating more cores will probably not help much for this command. As far as I understand, qiime diversity alpha-rarefaction is not parallelized, so requesting many CPUs on Slurm does not necessarily make it faster.

A common issue on clusters is that temporary files or the QIIME 2 cache are written to a shared/network filesystem, which can be much slower than local storage on the compute node. You could try putting the temporary directory and QIIME 2 cache on local scratch, for example:

export TMPDIR="${SLURM_TMPDIR:-/tmp}"
mkdir -p "$TMPDIR/qiime2-cache"

qiime diversity alpha-rarefaction \
  --i-table "$TMPDIR/table.qza" \
  --m-metadata-file "$TMPDIR/metadata.tsv" \
  --p-max-depth 10000 \
  --o-visualization "$TMPDIR/alpha-rarefaction.qzv" \
  --use-cache "$TMPDIR/qiime2-cache"

You would need to copy table.qza and metadata.tsv to $TMPDIR before running this, and then copy the resulting .qzv back to your project directory afterwards.

So I would first check whether the Slurm job is using a network filesystem for temporary/intermediate files. If yes, using node-local scratch may make a large difference.

Note that SLURM_TMPDIR is not available on all clusters, so you may need to check your cluster documentation or ask your system administrator what the correct node-local scratch variable or path is.

Let me know if this helps!
Vinzent

3

It helps, I ran succesfully. Thank a lot

4

Hi! I see that the answer worked well for you, but I wanted to mention another option. There's the National Research Platform, which is a large, inter-institutional collection of computing resources. There are several ways you can use the cluster, including a Jupyter Notebooks GUI and via command line using Kubernetes. If you go the Jupyter route, you have a mounted storage automatically connected to your workspace. If you use Kubernetes, then you create a storage to be mounted to your pods/containers. The .yaml files for Kubernetes have their own format that you get used to, but they're not that far off from SLURM jobs:

# job-cutadapt.yaml
# This job uses cutadapt to trim the seqs, running both on demuxed and filtered for comparison
apiVersion: batch/v1
kind: Job
metadata:
  name: trim-job   #name of the pod you'll see with command 'kubectl get pods'
  namespace: <your-namespace>
spec:
  backoffLimit: 1            #number of times to retry upon failure
  template:
    spec:
      containers:
      - name: job-pod        #name of container, not important
        image: quay.io/qiime2/qiime2:2026.7  #verify that this is the correct qiime2 image version
        command:                             #put the commands for the shell below
        - /bin/sh                            #open as bash shell
        - -c
        - 'cd /biovol && echo "**Using cutadapt on demux seqs on $(date)...**" && qiime cutadapt trim-paired --i-demultiplexed-sequences seqs_demux.qza --p-front-f CTTGGTCATTTAGAGGAAGTAA --p-front-r GCTGCGTTCTTCATCGATGC --p-error-rate 0.13 --p-cores 0 --o-stats stats_cutadapt.qza --o-trimmed-sequences seqs_cutadapt.qza && echo "**Trimming complete on $(date). Now visualizing...**" && qiime demux summarize --i-data seqs_cutadapt.qza --o-visualization seqs_cutadapt.qzv && echo "**Visualization complete on $(date). Now using cutadapt on filtered seqs...**" && qiime cutadapt trim-paired --i-demultiplexed-sequences seqs_filt.qza --p-front-f CTTGGTCATTTAGAGGAAGTAA --p-front-r GCTGCGTTCTTCATCGATGC --p-error-rate 0.13 --p-cores 0 --o-stats stats_cutadapt_filt.qza --o-trimmed-sequences seqs_cutadapt_filt.qza && echo "**Trimming completed on $(date). Now visualizing...**" && qiime demux summarize --i-data seqs_cutadapt_filt.qza --o-visualization seqs_cutadapt_filt.qzv && echo "**Job completed at $(date).**"'
        resources:
          limits:
            memory: 250Gi             #storage needed to do the work (Mi is MB, Gi is GB)
            cpu: 8                    #processing power needed (500m is 0.5 CPU, or 1 thread)
            ephemeral-storage: 250Gi  #storage needed for scratch (Mi is MB, Gi is GB)
          requests:
            memory: 250Gi
            cpu: 8
            ephemeral-storage: 250Gi
        volumeMounts:
        - mountPath: /biovol       #folder used to access mounted volume
          name: biovol
      volumes:
      - name: biovol
        persistentVolumeClaim:
          claimName: rxfungi-vol   #name of mounted volume
      restartPolicy: Never

Writing a temporary directory worked for your cluster, so it makes no sense for you to switch over, but knowing about the NRP is useful for anyone else who isn't able to and might be looking for an alternative, free cluster. :slight_smile: