Alignment mask error code

User Support
#1

Hi QIIME support,

I used the following code to build tree:

qiime alignment mask --i-alignment aligned-sequences.qza --o-masked-alignment masked-aligned-sequences.qza

The error I got is the following:

    Exception ignored in: <finalize object at 0x2aaab1f287b0; dead>
Traceback (most recent call last):
  File "/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/weakref.py", line 548, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/site-packages/qiime2/core/path.py", line 94, in _destruct
    shutil.rmtree(path)
  File "/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/shutil.py", line 480, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/shutil.py", line 418, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/shutil.py", line 422, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/shutil.py", line 420, in _rmtree_safe_fd
    os.rmdir(name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'data'

But it seems masked-aligned-sequences.qza was generated with exception. Do you know where is wrong? I used 2017.7 version of QIIME2. Thanks!

Yunhu

#2

Hi @mentorwan,

That is certainly unusual. But nothing is wrong with your generated .qza.

This means that something in our cleanup code has failed, so we may have left some temporary junk on your system. (They should be cleaned up next reboot). It is safe to ignore, although it shouldn’t have happened.

What kind of OS are you using? Your installation also looks atypical, is this a cluster or institutional computer you are using? Are you executing through a queueing system, or just on the command-line like normal?

#3

Actually it works fine in 2017.4 version. But exception happens with 2017.7 version. Not sure our admin installed stable version or not for 2017.7 version in Linux system. Can you point me to correct 2017.7 stable version?

#4

@mentorwan,

That isn’t too surprising, we fixed a bug that would leave files around, so this logic changed in between then (although you seem to be experiencing the exact opposite with some files not getting cleaned up in the new version).

What does qiime info say? Your installation is probably fine, but if you wanted to check you could follow our install docs (it will just install in your home directory, so you don’t need a sysadmin)

When you have QIIME 2017.7 loaded, what does

echo $TMPDIR

say?

Also, are you executing through a queueing system, or just on the command-line like normal?

1 Like
#5

Thanks for reply. I am unable to install on cluster since some files need permission. Admin installed it in system wide. I am running on command line mode.

qiime info:

System versions
Python version: 3.5.3
QIIME 2 release: 2017.7
QIIME 2 version: 2017.7.0
q2cli version: 2017.7.0

Installed plugins
alignment 2017.7.0
dada2 2017.7.0
deblur 2017.7.0
demux 2017.7.0
diversity 2017.7.0
emperor 2017.7.0
feature-classifier 2017.7.0
feature-table 2017.7.0
gneiss 2017.7.0
metadata 2017.7.0
phylogeny 2017.7.0
quality-filter 2017.7.0
taxa 2017.7.0
types 2017.7.0

echo $TMPDIR
/spin1/scratch/ywan

#7

Thanks @mentorwan,

I would say you install looks fine, you do have a different TMPDIR set, but that should also be fine.

Do you see this error on any QIIME 2 method/action or just the alignment mask one?

Also if you were to run the above again, does it give you the exception again?

#9

The error comes up every time however the output can used in next command. So I think it is something which can be ignored. Hopefully in next official release, it can be fixed. You can close this issue.

#11

Thanks @mentorwan,

Is this only for qiime alignment mask or every command?

We aren’t really able to reproduce this, so we’d still need more details to figure out “who, what, and why” this is happening. If it is a global occurance for every command then we know it’s the framework, but if it’s just qiime alignment mask then it’s probably very easy to figure out.

I’m suspecting there may be something strange with the permissions on your temporary directory. Do you by chance know if it is a network mounted directory or anything else that might be unusual about it?

What does:

 ls -ald /spin1/scratch/ywan

say?

Thank you for your assistance, and sorry for the trouble. Hopefully we can get to the bottom of this so that other people don’t run into this either!

1 Like
#13

ls -ald /spin1/scratch/ywan

drwxrwxrwx 14 ywan ywan Aug 31 14:04 /spin1/scratch/ywan

I have duplicated those errors on several independent sets. It happened every time. For two commands it has issue:
“qiime alignment mask --i-alignment aligned-sequences.qza --o-masked-alignment masked-aligned-sequences.qza”
“qiime phylogeny midpoint-root --i-tree unrooted-tree.qza --o-rooted-tree rooted-tree.qza”

Other commands are fine.For this system, I don’t have permission with some system files. The error code is the following:

“Exception ignored in: <finalize object at 0x7f866bc217a0; dead>
Traceback (most recent call last):
File “/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/weakref.py”, line 548, in call
return info.func(*info.args, **(info.kwargs or {}))
File “/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/site-packages/qiime2/core/path.py”, line 94, in _destruct
shutil.rmtree(path)
File “/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/shutil.py”, line 480, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File “/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/shutil.py”, line 418, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File “/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/shutil.py”, line 422, in _rmtree_safe_fd
onerror(os.rmdir, fullname, sys.exc_info())
File “/usr/local/apps/QIIME/2.2017.7/envs/qiime/lib/python3.5/shutil.py”, line 420, in _rmtree_safe_fd
os.rmdir(name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: ‘data’”

#15

Thanks @mentorwan!

I’m very strongly suspecting that /spin1/scratch is a network mounted file share. It looks like we are having issues with shutil.rmtree() (a part of the Python standard library) and it doesn’t seem to be “allowed” to delete the files (which is it’s only job).

I have an idea for how we might be able to confirm this:

I’m guessing you have a metadata file that we can use qiime metadata tabulate on (which will make this test fast).
I’d like to try it two different ways, one way:

qiime metadata tabulate --m-input-file <your file> --o-visualization doesntmatter.qzv

(I would expect to see the same errors as above)

Then we can run it again, but this time setting up a different temporary directory:

mkdir $HOME/delete-this-later

(this will be our new /spin1/scratch/ywan)

TMPDIR=$HOME/delete-this-later qiime metadata tabulate --m-input-file <your file> --o-visualization doesntmatter.qzv

(this will change TMPDIR for only this command)
If you do not see an error, then it definitely has to do with what /spin1/scratch is.
However, if you do see the same error, then it’s a little inconclusive as your home directory may also be mounted the same way (pretty common), or there is still something more fundamentally wrong (and I have no idea what that might be yet).

I’d also like to see what

ls $HOME/delete-this-later

says after you have run the above (it should be empty).

Thanks again for you assistance in debugging this! I really appreciate it!

1 Like
Plugin error from phylogeny:
#17

Thanks very much for solution. I tested your suggestion. Although qiime metadadata doesn’t have errors, my issues were resolved by setting different TMPDIR folder. The following command is also work without error. “TMPDIR=./delete-this-later qiime alignment mask --i-alignment aligned-sequences.qza --o-masked-alignment masked-aligned-sequences.qza”

You found the root cause and my issue was resolved. Thanks a lot.

1 Like
#18

Just a note that we’ll be adding the ability to customize QIIME 2’s temporary directory when we have a configuration system in place. Here’s an issue tracking progress on that, we’ll follow up here when it’s available in a QIIME 2 release!

#19

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.