Error incorporating QIIME 2 with snakemake

Background: I’m running qiime2 in a conda environment (qiime2-2019.4) and snakemake v5.4.0, which is installed in a separate conda env. Additional background, I’ve run other snakemake (without qiime2) and qiime2 (without snakemake) code successfully with these same conda environments.

All related files: q2_snakemake_Q.zip (9.2 KB)
I’ve run into this error running qiime2 with snakemake with any command, so it is not related to what I’m trying to do in any particular script (this is repeatable for any qiime2 command I run).
To explain, I’ll use my db import and classify snakemake script. Beforehand, I downloaded the PR2 database (both fasta and tax files) and am trying to run the attached snakemake script. I’ve copied and pasted the relevant bits of code/messages below, but these are also attached in full. Output from snakemake -n -r completes successfully, this is also attached as dryrunoutput.txt

Error message when running snakemake:

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       all
        1       import_db
        1       import_tax
        3

[Tue Jun 25 14:13:40 2019]
rule import_db:
    input: /vortexfs1/omics/huber/shu/db/pr2_version_4.11.1_mothur.fasta
    output: /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza
    log: /vortexfs1/scratch/sarahhu/pr2-db/logs/db-import.log
    jobid: 1

[Tue Jun 25 14:13:40 2019]
Error in rule import_db:
    jobid: 1
    output: /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza
    log: /vortexfs1/scratch/sarahhu/pr2-db/logs/db-import.log

RuleException:
CalledProcessError in line 34 of /vortexfs1/omics/huber/shu/db/Snakefile:
Command ' set -euo pipefail;  
   qiime tools import       --type 'FeatureData[Sequence]'       --input-path /vortexfs1/omics/huber/shu/db/pr2_version_4.11.1_mothur.fasta       --output-path /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza ' returned non-zero exit status 127.
  File "/vortexfs1/omics/huber/shu/db/Snakefile", line 34, in __rule_import_db
  File "/vortexfs1/home/sarahhu/.conda/envs/snake-18S/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /vortexfs1/omics/huber/shu/db/.snakemake/log/2019-06-25T141340.165538.snakemake.log

Snakemake code: line 34 has a comment note

rule all:
  input:
    dbartifact = expand("{base}/pr2-db/{info}.qza", base = SCRATCHDIR, info = VERSION),
    dbtaxartifact = expand("{base}/{info}_tax.qza", base = SCRATCHDIR, info = VERSION),

rule import_db:
  input:
    inputdb = INPUTDB
  output:
    dbartifact = expand("{base}/pr2-db/{info}.qza", base = SCRATCHDIR, info = VERSION)
  log:
    SCRATCHDIR + "/pr2-db/logs/db-import.log"
  conda:
    "/vortexfs1/omics/huber/shu/18S_pipeline_V4/envs/qiime2-2019.4.yaml"
  shell:
   """
   qiime tools import \  #This is line 34!
      --type 'FeatureData[Sequence]' \
      --input-path {input.inputdb} \
      --output-path {output.dbartifact}
   """

The lines from threads.py: line 56 has a comment

class _WorkItem(object):
    def __init__(self, future, fn, args, kwargs):
        self.future = future
        self.fn = fn
        self.args = args
        self.kwargs = kwargs

    def run(self):
        if not self.future.set_running_or_notify_cancel():
            return

        try:
            result = self.fn(*self.args, **self.kwargs) #This is line 56!
        except BaseException as exc:
            self.future.set_exception(exc)
            # Break a reference cycle with the exception 'exc'
            self = None
        else:
            self.future.set_result(result)

’qiime info’ output

System versions
Python version: 3.6.7
QIIME 2 release: 2019.4
QIIME 2 version: 2019.4.0
q2cli version: 2019.4.0

Installed plugins
alignment: 2019.4.0
composition: 2019.4.0
cutadapt: 2019.4.0
dada2: 2019.4.0
deblur: 2019.4.0
demux: 2019.4.1
diversity: 2019.4.0
emperor: 2019.4.0
feature-classifier: 2019.4.0
feature-table: 2019.4.0
fragment-insertion: 2019.4.0
gneiss: 2019.4.0
longitudinal: 2019.4.0
metadata: 2019.4.0
phylogeny: 2019.4.0
quality-control: 2019.4.0
quality-filter: 2019.4.0
sample-classifier: 2019.4.0
taxa: 2019.4.0
types: 2019.4.1
vsearch: 2019.4.0

Application config directory
/vortexfs1/home/sarahhu/.conda/envs/qiime2-2019.4/var/q2cli

Based on the error message and reading the snakemake FAQ, it may be something I can add to the shell command to tell snakemake to not exit - see My shell command fails with exit code != 0 from within a pipe, what’s wrong?. But I’m worried of what this will do to the qiime2 performance.
Any advice or references for implementing qiime2 commands as rules in snakemake pipelines is appreciated (esp with respect to syntax I may be missing).

2 Likes

Hey @shu251 - any chance you can get a more detailed error log out of this? Looks like the output streams aren’t all there — I can’t tell if the problem is with Q2 or Snakemake.

:t_rex:

output from snakemake --verbose should help?

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	all
	1	import_db
	1	import_tax
	3
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (2):
	import_tax
	import_db
Selected jobs (1):
	import_db
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Jun 25 15:00:30 2019]
rule import_db:
    input: /vortexfs1/omics/huber/shu/db/pr2_version_4.11.1_mothur.fasta
    output: /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza
    log: /vortexfs1/scratch/sarahhu/pr2-db/logs/db-import.log
    jobid: 1

Full Traceback (most recent call last):
  File "/vortexfs1/home/sarahhu/.conda/envs/snake-18S/lib/python3.6/site-packages/snakemake/executors.py", line 1378, in run_wrapper
    singularity_args, use_singularity, None, jobid, is_shell, None)
  File "/vortexfs1/omics/huber/shu/db/Snakefile", line 65, in __rule_import_db
  File "/vortexfs1/home/sarahhu/.conda/envs/snake-18S/lib/python3.6/site-packages/snakemake/shell.py", line 142, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' set -euo pipefail;  
   qiime tools import       --type 'FeatureData[Sequence]'       --input-path /vortexfs1/omics/huber/shu/db/pr2_version_4.11.1_mothur.fasta       --output-path /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza ' returned non-zero exit status 127.

[Tue Jun 25 15:00:30 2019]
Error in rule import_db:
    jobid: 1
    output: /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza
    log: /vortexfs1/scratch/sarahhu/pr2-db/logs/db-import.log

Full Traceback (most recent call last):
  File "/vortexfs1/home/sarahhu/.conda/envs/snake-18S/lib/python3.6/site-packages/snakemake/executors.py", line 1378, in run_wrapper
    singularity_args, use_singularity, None, jobid, is_shell, None)
  File "/vortexfs1/omics/huber/shu/db/Snakefile", line 65, in __rule_import_db
  File "/vortexfs1/home/sarahhu/.conda/envs/snake-18S/lib/python3.6/site-packages/snakemake/shell.py", line 142, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' set -euo pipefail;  
   qiime tools import       --type 'FeatureData[Sequence]'       --input-path /vortexfs1/omics/huber/shu/db/pr2_version_4.11.1_mothur.fasta       --output-path /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza ' returned non-zero exit status 127.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/vortexfs1/home/sarahhu/.conda/envs/snake-18S/lib/python3.6/site-packages/snakemake/executors.py", line 349, in _callback
    raise ex
  File "/vortexfs1/home/sarahhu/.conda/envs/snake-18S/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/vortexfs1/home/sarahhu/.conda/envs/snake-18S/lib/python3.6/site-packages/snakemake/executors.py", line 1390, in run_wrapper
    show_traceback=True))
snakemake.exceptions.RuleException: CalledProcessError in line 34 of /vortexfs1/omics/huber/shu/db/Snakefile:
Command ' set -euo pipefail;  
   qiime tools import       --type 'FeatureData[Sequence]'       --input-path /vortexfs1/omics/huber/shu/db/pr2_version_4.11.1_mothur.fasta       --output-path /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza ' returned non-zero exit status 127.
  File "/vortexfs1/omics/huber/shu/db/Snakefile", line 34, in __rule_import_db

RuleException:
CalledProcessError in line 34 of /vortexfs1/omics/huber/shu/db/Snakefile:
Command ' set -euo pipefail;  
   qiime tools import       --type 'FeatureData[Sequence]'       --input-path /vortexfs1/omics/huber/shu/db/pr2_version_4.11.1_mothur.fasta       --output-path /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza ' returned non-zero exit status 127.
  File "/vortexfs1/omics/huber/shu/db/Snakefile", line 34, in __rule_import_db
  File "/vortexfs1/home/sarahhu/.conda/envs/snake-18S/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /vortexfs1/omics/huber/shu/db/.snakemake/log/2019-06-25T150029.750987.snakemake.log
unlocking
removing lock
removing lock
removed all locks

Ah, I missed this earlier — exit code 127 typically means “command not found” — this would imply to me that your snakemake config is not adequately activating your QIIME 2 conda environment. I have no clue how to do that, but I have high hopes that there is something floating around in the Snakemake docs… Keep us posted!

Yeah, that makes sense. However, using the exact same syntax as I have here, I’ve been able to call other conda environments with no issues. So I’m thinking it may be something conda-qiime2 env specific?

What about the option of adding set +o pipefail; before the qiime shell command? I’m not sure how to properly implement it with the qiime syntax

Okay, I decided to give snakemake a try!

  1. If I install snakemake within an existing QIIME 2 conda env and run your pipeline from there, everything works as expected (I updated the pipeline to something a little more “minimal”, though)
  2. If I install snakemake in a seperate conda env, I need to run the pipeline with the --use-conda flag. Note, if I skip the --use-conda flag I see the same “not found” error you reported above.

Hope that helps!

4 Likes

Thanks! That is helpful.
I also changed my conda environment so it has both q2 and snakemake.

2 Likes