q2_cutadapt tests fail with FileNotFoundError under Debian

adrianheine · April 11, 2020, 7:50pm

When I run the q2_cutadapt tests under Debian with qiime 2019.10, most reliably fail with FileNotFoundError in /usr/lib/python3.8/shutil.py. This is tracked as debian bug #956220. In an answer here, it was hinted that other people have similar issues. Is there a known workaround?

thermokarst · April 13, 2020, 2:34pm

Hi @adrianheine, thanks for the post.

Just as a quick note, at present QIIME 2 is only “officially” compatible with python 3.6 - we do not support 3.7+ (yet), although we are hoping to work on this a bit later this year.

So, taking a closer look at the build log you linked to, I think the error isn’t specific to q2-cutadapt, but rather it points to a small issue in the QIIME 2 framework:

There are a few invocations of shutil.copytree in the provenance module:

github.com

qiime2/qiime2/blob/master/qiime2/core/archive/provenance.py

# ----------------------------------------------------------------------------
# Copyright (c) 2016-2020, QIIME 2 development team.
#
# Distributed under the terms of the Modified BSD License.
#
# The full license is in the file LICENSE, distributed with this software.
# ----------------------------------------------------------------------------

import time
import collections
import pkg_resources
import uuid
import copy
import shutil
import sys
from datetime import datetime, timezone

import distutils
import yaml
import tzlocal

This file has been truncated. show original

One of the things that @ebolyen built out to address these cross-device linking errors is this util:

github.com

qiime2/qiime2/blob/6a5739dcc9d1760818215a8df2ae6092bf649db6/qiime2/util.py#L70-L96


def duplicate(src, dst):
    """Alternative to shutil.copyfile, this will use os.link when possible.


    See shutil.copyfile for documention. Only `src` and `dst` are supported.
    Unlike copyfile, this will not overwrite the destination if it exists.


    """
    if os.path.isdir(src):
        # os.link will give a permission error
        raise OSError(errno.EISDIR, "Is a directory", src)
    if os.path.isdir(dst):
        # os.link will give a FileExists error
        raise OSError(errno.EISDIR, "Is a directory", dst)


    if os.path.exists(dst):
        # shutil.copyfile will overwrite the existing file
        raise OSError(errno.EEXIST, "File exists", src, "File exists", dst)


    try:
        os.link(src, dst)

This file has been truncated. show original

We should probably do a sweep of the framework and make sure we are using that util where needed.

Thanks @adrianheine! I’ll bump this up in priority for our current dev cycle - the next release of QIIME 2 is slated for late May 2020.