Install without using fully pinned environment.yaml

Developer Discussion
1

Hey guys,

how do I install a qiime-dev environment without using the fully pinned qiime2-latest-...-conda.yml?

The reason at hand is that the pinned file contains a bunch of bioconductor-xyz packages from bioconductor 3.6, all of which were build a while ago and hence pin (probably unnecessarily) glib 2.53, while I need arb-bio-tool which requires glib >=2.56 (on osx).

Yes, this is about sina :wink:

The reason it won’t work on osx is that the package resolution pulls in an old broken build of arb-bio which didn’t pin glib correctly.

2
3

Hey @epruesse,

In principle you can write a command like this:

conda install \
  -c qiime2-staging/label/r2019.4 -c conda-forge -c bioconda -c defaults \
  qiime2 q2cli q2-types <and-anything-else>

which will give you the latest everything (without most plugins such as those using bioconductor).

However! Conda may install incompatible versions if you are unlucky and one of our dependencies has just issued a release that breaks something (this happens about monthly). So you should generally be cautious about this (for development/testing this isn’t really a problem).

What we do in our staging environment file is let conda resolve everything (like it does in the above command) and then we run our documentation to make sure it still works which catches issues with incompatible versions (letting us pin in the recipe and try again). This means the environment file won’t be committed unless that particular package resolution is able to pass our tests.

A different option would be to pare down an existing environment file to have only the dependencies you want while preserving the pinning that we already know works (saving you a little bit of time).

As of literally this afternoon, I wrote a script to do just that. It’s probably not ready for prime-time, but if you’d like to try it out, just send me a DM and I can send you that link. This script is able to “remove” packages from an existing installation without impacting versions (or actually uninstalling anything). It is also handy for checking all dependencies recursively of a given package.

1 Like
4
5

Well, I’m trying to get sina to work on osx. Even a separate q2-sina plugin would have to install into an existing qiime2 environment, which will lead to the conflict problem. I was hoping to be able to reproduce the way the environment file is generated, so that I can add the right pins to end up at a working set of packages.

6

Sorry I had assumed we were just waiting for bioconda to update their bioconductor packages (which should happen soonish as they migrate to gcc7 matching conda-forge).

Until then, I think q2-sina may be incompatible with q2-dada2 (at least as installed through conda).

You can see how we build these environment files here:

Concourse CI Job
The script that is running in the job

It’s honestly not the easiest thing to follow without context on how Concourse CI works, but really all that is happening is a really long conda install command with certain channels set. In the case of integration, some of those channels are local filesystem channels which let us pipe conda packages along the CI system.

7

(puts on bioconda hat) That’s already done. The Bioconductor packages had been rebuilding via the bulk branch for a while, and were relabled from gcc7 to main a week ago (more I think). New packages are being built using the new compiler toolchain, so using gcc7 with the new C++ ABI. Plenty of packages will yet have to be rebuilt, but our dependency graph isn’t quite as problematic as the one in conda-forge. We will pursue that in an “on demand” fashion. I’ll hopefully be able to finish our bot soon, which will create appropriate PRs bumping build numbers on request. (puts hat away)

2 Likes