Error when executing "qiime tools import" script on a server


#1

Hi,

I ran into the following problem after having installed qiime2 (followed the native installation guide for linux, microconda and then qiime2) on a HPC. The problem did not occur when testing exactly the same script(s) on my laptop.

scratch is a folder where temporary server data is stored. When I have a look at the folder “/scratch/tmp/qiime2-archive-9fiqpccd/070aaa4f-f3b9-44cd-99b2-b8c38780cc78/data”, I find some of my files but I dont find the indicated file/folder (“q2-SingleLanePerSamplePairedEndFastqDirFmt-76y9vsqw”)

This seems to be a general problem that also comes up if I exucute other qiime scritps such as dada2. I get the same error msg. It seems to be unrelated to qiime tools import.

Thanks for your help in advance!

This is the command that I try to run:
qiime tools import --type ‘SampleData[PairedEndSequencesWithQuality]’ --input-path /home/mbx/data/demulti --input-format CasavaOneEightSingleLanePerSampleDirFmt --output-path /home/mbx/data/demux-paired-end2.qza

This is the error msg that I get:
Traceback (most recent call last):
File “/home/mbx/miniconda/envs/qiime2-2018.11/lib/python3.5/site-packages/q2cli/tools.py”, line 140, in import_data
view_type=input_format)
File “/home/mbx/miniconda/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/result.py”, line 240, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File “/home/mbx/miniconda/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/result.py”, line 271, in _from_view
provenance_capture=provenance_capture)
File “/home/mbx/miniconda/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/archive/archiver.py”, line 316, in from_data
Format.write(rec, type, format, data_initializer, provenance_capture)
File “/home/mbx/miniconda/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/archive/format/v5.py”, line 21, in write
provenance_capture)
File “/home/mbx/miniconda/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/archive/format/v1.py”, line 19, in write
provenance_capture)
File “/home/mbx/miniconda/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/archive/format/v0.py”, line 62, in write
data_initializer(data_dir)
File “/home/mbx/miniconda/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/path.py”, line 37, in _move_or_copy
return _ConcretePath.rename(self, other)
File “/home/mbx/miniconda/envs/qiime2-2018.11/lib/python3.5/pathlib.py”, line 1277, in rename
self._accessor.rename(self, target)
File “/home/mbx/miniconda/envs/qiime2-2018.11/lib/python3.5/pathlib.py”, line 377, in wrapped
return strfunc(str(pathobjA), str(pathobjB), *args)
FileExistsError: [Errno 17] File exists: ‘/scratch/tmp/q2-SingleLanePerSamplePairedEndFastqDirFmt-76y9vsqw’ -> ‘/scratch/tmp/qiime2-archive-9fiqpccd/070aaa4f-f3b9-44cd-99b2-b8c38780cc78/data’

An unexpected error has occurred:

[Errno 17] File exists: ‘/scratch/tmp/q2-SingleLanePerSamplePairedEndFastqDirFmt-76y9vsqw’ -> ‘/scratch/tmp/qiime2-archive-9fiqpccd/070aaa4f-f3b9-44cd-99b2-b8c38780cc78/data’

See above for debug info.


(Matthew Ryan Dillon) #2

Hi there @mbx! I think we have seen a similar error before on certain configurations of network-mounted drivers:

Any chance this applies to your situation?


#3

Yes, kind of. Thanks a lot!

I was able to solve it with setting the temporary folder to another drive with:

export TMPDIR=/home/mbx/tmp
mkdir $TMPDIR


(Matthew Ryan Dillon) #4

Excellent - your original tmpdir was probably on an NFS-esque drive that is impacted by this bug (this appears to be a bug in the Python standard library, BTW).