Background: I'm running qiime2 in a conda environment (qiime2-2019.4) and snakemake v5.4.0, which is installed in a separate conda env. Additional background, I've run other snakemake (without qiime2) and qiime2 (without snakemake) code successfully with these same conda environments.
All related files: q2_snakemake_Q.zip (9.2 KB)
I've run into this error running qiime2 with snakemake with any command, so it is not related to what I'm trying to do in any particular script (this is repeatable for any qiime2 command I run).
To explain, I'll use my db import and classify snakemake script. Beforehand, I downloaded the PR2 database (both fasta and tax files) and am trying to run the attached snakemake script. I've copied and pasted the relevant bits of code/messages below, but these are also attached in full. Output from snakemake -n -r completes successfully, this is also attached as dryrunoutput.txt
Error message when running snakemake:
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 all
1 import_db
1 import_tax
3
[Tue Jun 25 14:13:40 2019]
rule import_db:
input: /vortexfs1/omics/huber/shu/db/pr2_version_4.11.1_mothur.fasta
output: /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza
log: /vortexfs1/scratch/sarahhu/pr2-db/logs/db-import.log
jobid: 1
[Tue Jun 25 14:13:40 2019]
Error in rule import_db:
jobid: 1
output: /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza
log: /vortexfs1/scratch/sarahhu/pr2-db/logs/db-import.log
RuleException:
CalledProcessError in line 34 of /vortexfs1/omics/huber/shu/db/Snakefile:
Command ' set -euo pipefail;
qiime tools import --type 'FeatureData[Sequence]' --input-path /vortexfs1/omics/huber/shu/db/pr2_version_4.11.1_mothur.fasta --output-path /vortexfs1/scratch/sarahhu/pr2-db/pr2_4.11.1.qza ' returned non-zero exit status 127.
File "/vortexfs1/omics/huber/shu/db/Snakefile", line 34, in __rule_import_db
File "/vortexfs1/home/sarahhu/.conda/envs/snake-18S/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /vortexfs1/omics/huber/shu/db/.snakemake/log/2019-06-25T141340.165538.snakemake.log
Snakemake code: line 34 has a comment note
rule all:
input:
dbartifact = expand("{base}/pr2-db/{info}.qza", base = SCRATCHDIR, info = VERSION),
dbtaxartifact = expand("{base}/{info}_tax.qza", base = SCRATCHDIR, info = VERSION),
rule import_db:
input:
inputdb = INPUTDB
output:
dbartifact = expand("{base}/pr2-db/{info}.qza", base = SCRATCHDIR, info = VERSION)
log:
SCRATCHDIR + "/pr2-db/logs/db-import.log"
conda:
"/vortexfs1/omics/huber/shu/18S_pipeline_V4/envs/qiime2-2019.4.yaml"
shell:
"""
qiime tools import \ #This is line 34!
--type 'FeatureData[Sequence]' \
--input-path {input.inputdb} \
--output-path {output.dbartifact}
"""
The lines from threads.py: line 56 has a comment
class _WorkItem(object):
def __init__(self, future, fn, args, kwargs):
self.future = future
self.fn = fn
self.args = args
self.kwargs = kwargs
def run(self):
if not self.future.set_running_or_notify_cancel():
return
try:
result = self.fn(*self.args, **self.kwargs) #This is line 56!
except BaseException as exc:
self.future.set_exception(exc)
# Break a reference cycle with the exception 'exc'
self = None
else:
self.future.set_result(result)
'qiime info' output
System versions
Python version: 3.6.7
QIIME 2 release: 2019.4
QIIME 2 version: 2019.4.0
q2cli version: 2019.4.0
Installed plugins
alignment: 2019.4.0
composition: 2019.4.0
cutadapt: 2019.4.0
dada2: 2019.4.0
deblur: 2019.4.0
demux: 2019.4.1
diversity: 2019.4.0
emperor: 2019.4.0
feature-classifier: 2019.4.0
feature-table: 2019.4.0
fragment-insertion: 2019.4.0
gneiss: 2019.4.0
longitudinal: 2019.4.0
metadata: 2019.4.0
phylogeny: 2019.4.0
quality-control: 2019.4.0
quality-filter: 2019.4.0
sample-classifier: 2019.4.0
taxa: 2019.4.0
types: 2019.4.1
vsearch: 2019.4.0
Application config directory
/vortexfs1/home/sarahhu/.conda/envs/qiime2-2019.4/var/q2cli
Based on the error message and reading the snakemake FAQ, it may be something I can add to the shell command to tell snakemake to not exit - see My shell command fails with exit code != 0 from within a pipe, what’s wrong?. But I'm worried of what this will do to the qiime2 performance.
Any advice or references for implementing qiime2 commands as rules in snakemake pipelines is appreciated (esp with respect to syntax I may be missing).
