Demux emp-single Error in Moving Pictures tutorial

Fabio_Marroni · January 23, 2018, 8:53pm

Hi, am a new qiime2 user.
I am trying to familiarize with it by running the “Moving Pictures” tutorial, and I executed the following commands (copied from here).

mkdir qiime2-moving-pictures-tutorial
cd qiime2-moving-pictures-tutorial
wget -O "sample-metadata.tsv" "https://data.qiime2.org/2017.12/tutorials/moving-pictures/sample_metadata.tsv"
mkdir emp-single-end-sequences
wget -O "emp-single-end-sequences/barcodes.fastq.gz" "https://data.qiime2.org/2017.12/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz"
wget -O "emp-single-end-sequences/sequences.fastq.gz" "https://data.qiime2.org/2017.12/tutorials/moving-pictures/emp-single-end-sequences/sequences.fastq.gz"
qiime tools import \
  --type EMPSingleEndSequences \
  --input-path emp-single-end-sequences \
  --output-path emp-single-end-sequences.qza
  
qiime demux emp-single \
  --i-seqs emp-single-end-sequences.qza \
  --m-barcodes-file sample-metadata.tsv \
  --m-barcodes-category BarcodeSequence \
  --o-per-sample-sequences demux.qza

The last command gives the following error:

Plugin error from demux:

  [('/projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97/7af2d007-0f0a-462a-b0a0-016eee96e389/provenance/VERSION', '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/VERSION', "[Errno 22] Invalid argument: '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/VERSION'"), ('/projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97/7af2d007-0f0a-462a-b0a0-016eee96e389/provenance/metadata.yaml', '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/metadata.yaml', "[Errno 22] Invalid argument: '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/metadata.yaml'"), ('/projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97/7af2d007-0f0a-462a-b0a0-016eee96e389/provenance/action/action.yaml', '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/action/action.yaml', "[Errno 22] Invalid argument: '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/action/action.yaml'"), ('/projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97/7af2d007-0f0a-462a-b0a0-016eee96e389/provenance/action', '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/action', "[Errno 22] Invalid argument: '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/action'"), ('/projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97/7af2d007-0f0a-462a-b0a0-016eee96e389/provenance', '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389', "[Errno 22] Invalid argument: '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389'")]

Debug info has been saved to /projects/novabreed/share/marroni/tmp/qiime2-q2cli-err-lrkzg5mg.log

The content of the cited log file is the following:

Traceback (most recent call last):
  File "/projects/novabreed/share/anaconda-3/envs/qiime2-2017.12.1/lib/python3.5/site-packages/q2cli/commands.py", line 224, in __call__
    results = action(**arguments)
  File "<decorator-gen-336>", line 2, in emp_single
  File "/projects/novabreed/share/anaconda-3/envs/qiime2-2017.12.1/lib/python3.5/site-packages/qiime2/sdk/action.py", line 208, in bound_callable
    provenance.add_input(name, artifact)
  File "/projects/novabreed/share/anaconda-3/envs/qiime2-2017.12.1/lib/python3.5/site-packages/qiime2/core/archive/provenance.py", line 306, in add_input
    self.inputs[name] = self.add_ancestor(input)
  File "/projects/novabreed/share/anaconda-3/envs/qiime2-2017.12.1/lib/python3.5/site-packages/qiime2/core/archive/provenance.py", line 146, in add_ancestor
    ignore=shutil.ignore_patterns(self.ANCESTOR_DIR + '*'))
  File "/projects/novabreed/share/anaconda-3/envs/qiime2-2017.12.1/lib/python3.5/shutil.py", line 359, in copytree
    raise Error(errors)
shutil.Error: [('/projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97/7af2d007-0f0a-462a-b0a0-016eee96e389/provenance/VERSION', '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/VERSION', "[Errno 22] Invalid argument: '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/VERSION'"), ('/projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97/7af2d007-0f0a-462a-b0a0-016eee96e389/provenance/metadata.yaml', '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/metadata.yaml', "[Errno 22] Invalid argument: '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/metadata.yaml'"), ('/projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97/7af2d007-0f0a-462a-b0a0-016eee96e389/provenance/action/action.yaml', '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/action/action.yaml', "[Errno 22] Invalid argument: '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/action/action.yaml'"), ('/projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97/7af2d007-0f0a-462a-b0a0-016eee96e389/provenance/action', '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/action', "[Errno 22] Invalid argument: '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389/action'"), ('/projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97/7af2d007-0f0a-462a-b0a0-016eee96e389/provenance', '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389', "[Errno 22] Invalid argument: '/projects/novabreed/share/marroni/tmp/qiime2-provenance-ssd7pp15/artifacts/7af2d007-0f0a-462a-b0a0-016eee96e389'")]

Output of qiime info is the following:

System versions
Python version: 3.5.4
QIIME 2 release: 2017.12
QIIME 2 version: 2017.12.1
q2cli version: 2017.12.0

Installed plugins
alignment 2017.12.0
composition 2017.12.0
cutadapt 2017.12.0
dada2 2017.12.1
deblur 2017.12.0
demux 2017.12.0
diversity 2017.12.0
emperor 2017.12.0
feature-classifier 2017.12.0
feature-table 2017.12.0
gneiss 2017.12.0
longitudinal 2017.12.0
metadata 2017.12.0
phylogeny 2017.12.0
quality-control 2017.12.0
quality-filter 2017.12.0
sample-classifier 2017.12.0
taxa 2017.12.0
types 2017.12.0
vsearch 2017.12.0

Application config directory
/home/marroni/.config/q2cli

Getting help
To get help with QIIME 2, visit https://qiime2.org

Citing QIIME 2
If you use QIIME 2 in any published work, you should cite QIIME 2 and the plugins that you used. To display the citations for QIIME 2 and all installed plugins, run:

  qiime info --citations

Although I was sure it was not the same issue, I already tried the solution you suggested here for a different problem in demux emp-single.

I don’t know if these details are important, but:

/projects/novabreed/share/marroni/tmp/ is my (hardcoded, i.e. loaded at login) TMPDIR
The subfolders mentioned in the log, such as /projects/novabreed/share/marroni/tmp/qiime2-archive-795p6x97 do not exist.

I am pretty sure that I am making some very stupid mistake, but at present I have no clues regarding what’s causing the issue. Can you help me?

jairideout · January 24, 2018, 6:34pm

Hi @Fabio_Marroni, thanks for the detailed report! I’m not sure what exactly is causing this issue with your temporary directory. Can you please provide the output from running these commands (with your qiime2-2017.12 environment activated):

echo $TMPDIR $TEMP $TMP
python -c 'import tempfile; print(tempfile.gettempdir())'
ls -la /projects/novabreed/share/marroni/tmp/
df -Th /projects/novabreed/share/marroni/tmp/

I suspect there may be an issue with permissions on your temporary directory, or the directory is running out of disk space when the qiime2 command is running and attempting to create temporary files. Thanks!

Fabio_Marroni · January 26, 2018, 4:06pm

Hi @jairideout, thank you for answering!

I paste below the output of the commands you suggested
echo $TMPDIR $TEMP $TMP
/projects/novabreed/share/marroni/tmp/

Here the only directory which has a value is $TMPDIR, the other are not set

python -c 'import tempfile; print(tempfile.gettempdir())'
/projects/novabreed/share/marroni/tmp

ls -la /projects/novabreed/share/marroni/tmp/
total 4484
drwxrwx---. 50 root       novabreed      1578 Jan 26 16:05 .
drwxrwx---. 22 root       novabreed      1041 Dec 28 14:13 ..
-rwxrwx---.  1 epaparelli domain users   4096 Jan 20  2017 .nfs000000014af9fa5300023b0d
-rwxrwx---.  1 marroni    domain users   3044 Jan 23 18:11 qiime2-q2cli-err-1flbtggy.log
-rwxrwx---.  1 marroni    domain users   3044 Jan 23 18:25 qiime2-q2cli-err-lrkzg5mg.log
-rwxrwx---.  1 marroni    domain users   3044 Jan 23 18:21 qiime2-q2cli-err-ro4p5uh8.log
-rwxrwx---.  1 marroni    domain users   3044 Jan 23 18:17 qiime2-q2cli-err-wy75bxv0.log
drwxrwx---.  2 abertoli   domain users      0 Jan 24 15:42 Rtmp0LMueO
drwxrwx---.  2 abertoli   domain users      0 Jan 18 16:02 Rtmp2Fh6W2
drwxrwx---.  2 abertoli   domain users      0 Jan 11 12:13 Rtmp2j2d4K
drwxrwx---.  2 marroni    domain users      0 Oct 29 09:58 Rtmp3eBUKS
drwxrwx---.  3 marroni    domain users    260 Nov  4 11:36 Rtmp4j1Wa0
drwxrwx---.  2 marroni    domain users    147 Jan  1 17:25 Rtmp5g0yc8
drwxrwx---.  2 abertoli   domain users      0 Jan 23 12:27 Rtmp5VpASe
drwxrwx---.  2 abertoli   domain users      0 Jan 24 15:33 Rtmp687gow
drwxrwx---.  3 marroni    domain users    419 Nov 17 14:52 Rtmp6Uxm9L
drwxrwx---.  2 marroni    domain users      0 Nov 28 14:56 Rtmp7RhrNg
drwxrwx---.  2 marroni    domain users      0 Nov  3 13:22 RtmpAkn1cT
drwxrwx---.  2 abertoli   domain users      0 Jan 24 00:14 Rtmpb9agKt
drwxrwx---.  3 marroni    domain users    677 Jan  6 17:09 RtmpBC8mkx
drwxrwx---.  2 marroni    domain users 493295 Jan 17 01:26 Rtmpc8mcyO
drwxrwx---.  2 marroni    domain users      0 Dec  6 20:07 RtmpcZvjFl
drwxrwx---.  2 marroni    domain users      0 Jan 24 19:30 RtmpDybzcX
drwxrwx---.  2 marroni    domain users      0 Oct 29 10:46 RtmpEi87vc
drwxrwx---.  2 abertoli   domain users      0 Jan  1 15:53 RtmpfaFUD6
drwxrwx---.  2 marroni    domain users      0 Oct 20 12:00 RtmpFwtPEr
drwxrwx---.  2 abertoli   domain users      0 Jan 25 13:43 RtmpgemF35
drwxrwx---.  2 abertoli   domain users      0 Jan 23 20:43 RtmpGNtjcU
drwxrwx---.  2 marroni    domain users    147 Dec 14 16:28 RtmpGVsp3e
drwxrwx---.  2 abertoli   domain users      0 Jan  1 17:05 RtmpHjEkLr
drwxrwx---.  2 marroni    domain users      0 Oct 30 18:12 RtmpHMGppU
drwxrwx---.  2 abertoli   domain users      0 Jan 25 13:15 RtmpiIDlxz
drwxrwx---.  2 marroni    domain users      0 Oct 14 15:33 RtmpJk6Kso
drwxrwx---.  2 abertoli   domain users      0 Jan 11 15:31 RtmpJlPBA5
drwxrwx---.  2 marroni    domain users      0 Dec 14 18:05 RtmpKCqaGV
drwxrwx---.  2 marroni    domain users      0 Dec 21 12:26 RtmpldOsHp
drwxrwx---.  2 marroni    domain users    147 Dec 29 17:24 RtmpMkhHyW
drwxrwx---.  3 marroni    domain users    260 Nov  8 15:20 Rtmpmo9SgO
drwxrwx---.  2 abertoli   domain users      0 Jan 25 14:11 RtmpnLEnsI
drwxrwx---.  2 abertoli   domain users      0 Jan  4 00:56 RtmpNn10hR
drwxrwx---.  2 abertoli   domain users      0 Jan 24 12:25 RtmpNYF86N
drwxrwx---.  2 abertoli   domain users      0 Jan 23 12:02 RtmpogQtuy
drwxrwx---.  2 marroni    domain users    147 Dec 31 16:31 RtmpP2TpdD
drwxrwx---.  2 marroni    domain users    165 Jan 25 16:51 RtmpPgKld3
drwxrwx---.  2 abertoli   domain users      0 Jan 24 00:06 RtmpQ4rd3Z
drwxrwx---.  2 abertoli   domain users      0 Jan 23 12:19 RtmpqH5a4f
drwxrwx---.  2 marroni    domain users      0 Dec 21 13:26 RtmprpMip8
drwxrwx---.  3 marroni    domain users    260 Nov  3 17:49 Rtmpsitzbd
drwxrwx---.  2 marroni    domain users      0 Jan 26 10:16 RtmpTWu9Hb
drwxrwx---.  2 marroni    domain users      0 Dec 21 16:21 RtmpuUAdDT
drwxrwx---.  2 abertoli   domain users      0 Jan 25 15:07 RtmpVePNob
drwxrwx---.  2 abertoli   domain users      0 Jan 23 12:35 RtmpX8czq8
drwxrwx---.  2 abertoli   domain users      0 Jan 14 21:25 RtmpYH3JMG
drwxrwx---.  2 marroni    domain users      0 Dec 22 15:49 RtmpYmnL5N
drwxrwx---.  2 abertoli   domain users      0 Jan 24 00:02 RtmpZy7GiJ

And finally

df -Th /projects/novabreed/share/marroni/tmp/
Filesystem Type Size Used Avail Use% Mounted on
isilon-hpc-nfs.lan.appliedgenomics.org:/ifs/projects/novabreed
nfs 17T 17T 863G 96% /projects/novabreed

I hope this helps!

jairideout · January 26, 2018, 7:14pm

Thanks for the details! I’m unsure what the exact problem is, but I have a few ideas:

Your temporary directory is on an NFS file system. There is a bug in Python 3.4, 3.5, and 3.6 preventing shutil.copytree from copying files from NFS to NFS (even if it’s the same NFS mount). Is there any way you could test out the tutorial commands using a temporary directory that is not on an NFS file system? That’d let us confirm whether you’re running into this Python 3 bug or not.
Does your user account have any user disk quotas set for /projects/novabreed/? You might be running into user-specified disk quota limits, which would prevent files from being copied.
The NFS filesystem is nearly at capacity (96% used). I’ve heard about NFS file systems not functioning well when the system is near capacity (even though you have 863GB free disk space). You might try using a different temporary directory or freeing up disk space on the NFS file system.

My hunch is that you’re running into Issue 1 above, so I’d start by using a different (non-NFS) temporary directory to see if that clears up the issue. Sorry to not have a more definitive answer at this time!

Fabio_Marroni · January 30, 2018, 4:43pm

Thank you very much, it was issue Number 1.
Now I set the TMPDIR to a location outside of nfs on activation of the qiime2 environment.
I only have limited space in the TMPDIR outside NSF (80G), I hope it will be enough!

jairideout · January 30, 2018, 4:49pm

Great, thanks for reporting back! This forum topic may be helpful to other users running into NFS-related issues. Hopefully the bug will be fixed upstream in Python 3 and it'll start "just working" in a future release

Unless you have a really large set of raw sequences that exceed 80GB, I think you should be fine (e.g. processing typical MiSeq runs shouldn't pose a problem).

jairideout · January 30, 2018, 5:13pm

Also for reference I created an issue to keep this bug/workaround on our radar. We may just add a note to the QIIME 2 docs about the issue for now, unless it’s fixed in upstream Python 3 before we get around to updating the docs. Either way, we’ll follow up here when one of those solutions are put in place.

system · March 2, 2018, 11:14pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.