Package uploaded to anaconda but not installable

conda

(Claire Duvallet) #1

Hopefully one last conda question for y’all!

After the workshop, I successfully uploaded my .tar.bz2 file to anaconda.org. However, when I try to conda install the plugin, it can’t find the package and installing doesn’t work.

$ conda install -c cduvallet q2_perc_norm
PackagesNotFoundError: The following packages are not available from current channels:

  - q2_perc_norm

It seems like conda doesn’t see that I have the package uploaded. If I do conda list -c cduvallet, I get nothing back. I’ve checked that my package is set to public on anaconda, and other than that am not sure how to fix this.

$ conda list -c cduvallet

$ 

Also a note that conda list -c cduvallet returns nothing on multiple systems (including my laptop in root environment, qiime2-2018.4, 2018.2, and dev environment, and our cluster which runs on linux).

Any ideas what could be causing this?


(Matthew Ryan Dillon) #2

(Matthew Ryan Dillon) #3

Hey there @cduvallet! I wonder if this error is related to the current path you are in when running the conda install command above — if you are sitting in your q2-perc-norm repo, then there is a folder there called q2_perc_norm - is it possible conda thinks it should be looking in there to install the package from, instead of up towards the heavens (aka anaconda.org)?

On my dev machine I ran the following:

conda install -c cduvallet q2_perc_norm
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.2
  latest version: 4.5.3

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /Users/matthew/.conda/envs/q2dev

  added / updated specs:
    - q2_perc_norm


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    scipy-0.19.1               |   py35hf362045_3        14.6 MB
    numpy-1.12.1               |   py35h655160a_1         3.6 MB
    blas-1.0                   |              mkl           6 KB
    scikit-learn-0.19.1        |   py35h2b554eb_0         4.8 MB
    q2_perc_norm-1.0           |           py35_0          11 KB  cduvallet
    certifi-2018.4.16          |           py35_0         142 KB
    ------------------------------------------------------------
                                           Total:        23.1 MB

The following NEW packages will be INSTALLED:

    q2_perc_norm: 1.0-py35_0                    cduvallet

The following packages will be UPDATED:

    certifi:      2018.4.16-py35_0              conda-forge --> 2018.4.16-py35_0
    openssl:      1.0.2o-0                      conda-forge --> 1.0.2o-h26aff7b_0

The following packages will be DOWNGRADED:

    blas:         1.1-openblas                  conda-forge --> 1.0-mkl
    numpy:        1.12.1-py35_blas_openblas_200 conda-forge [blas_openblas] --> 1.12.1-py35h655160a_1
    scikit-learn: 0.19.1-py35_blas_openblas_200 conda-forge [blas_openblas] --> 0.19.1-py35h2b554eb_0
    scipy:        0.19.1-py35_blas_openblas_202 conda-forge [blas_openblas] --> 0.19.1-py35hf362045_3

Proceed ([y]/n)?


Downloading and Extracting Packages
scipy 0.19.1############################################################################################################################################################################################ | 100%
numpy 1.12.1############################################################################################################################################################################################ | 100%
blas 1.0################################################################################################################################################################################################ | 100%
scikit-learn 0.19.1##################################################################################################################################################################################### | 100%
q2_perc_norm 1.0######################################################################################################################################################################################## | 100%
certifi 2018.4.16####################################################################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Then, to check that the conda package was installed:

conda list | ag perc
q2_perc_norm              1.0                      py35_0    cduvallet

And finally, the big reveal:

$ qiime info
System versions
Python version: 3.5.5
QIIME 2 release: 2018.4
QIIME 2 version: 2018.4.0+5.g448758b
q2cli version: 2018.4.0.dev0+2.ga2e3105

Installed plugins
alignment: 2018.4.0.dev0
composition: 2018.4.0.dev0
cutadapt: 2018.4.0.dev0
dada2: 2017.12.1.dev0+10.g5e74513
deblur: 2018.4.0.dev0+1.g9d1c639
demux: 2018.4.0.dev0+1.gf854e3d
diversity: 2017.12.0.dev0+16.g3372662
emperor: 2018.4.0.dev0+7.g8e30d63.dirty
feature-classifier: 2018.4.0.dev0
feature-table: 2018.4.0.dev0
gneiss: 2018.4.0.dev0
longitudinal: 2018.4.0.dev0+15.g91a471c
metadata: 2018.4.0.dev0
perc_norm: 0.0.1-dev
phylogeny: 2018.4.0.dev0
quality-control: 2018.4.0.dev0
quality-filter: 2018.4.0.dev0+2.g5a36e50
sample-classifier: 2018.4.0.dev0
taxa: 2018.4.0.dev0
types: 2018.4.0.dev0+1.gfdf4f0e
vsearch: 2018.4.0.dev0

Application config directory
/Users/matthew/Library/Application Support/q2cli

Getting help
To get help with QIIME 2, visit https://qiime2.org

One other thing that jumps out at me is that there is only a mac build of your plugin up on anaconda.org, that means if you are on a linux machine and try to install this package, conda will fail to resolve a valid linux-formatted package. Two ways around this come to mind:

  1. Build a version of your package on a linux machine (more work, need to have a linux computer)
  2. Build a noarch version of your package on your mac (less work).

The noarch package lets you upload one build that is compatible on multiple platforms - the caveat here is that it is up to you to know if that is the case or not (it looks like you should be safe to go with this option, I think :crossed_fingers:)


Good luck and keep us posted! :t_rex:


(Matthew Ryan Dillon) #4

(Matthew Ryan Dillon) #5

PS -

It looks like the -c flag when running conda list is for displaying “canonical package names” (seriously? I think -c everywhere else in conda-land is for setting the channel…)

I think this might give you the info you are looking for:

conda search -c cduvallet q2_perc_norm                                                                                                                                                  flashcards e90e4e7 ✗
Loading channels: done
# Name                  Version           Build  Channel
q2_perc_norm                1.0          py35_0  cduvallet

(Claire Duvallet) #6

Hm, okay. So it looks like conda install works on my Mac if I’m in a qiime2 environment (qiime2-2018.2 and qiime2-dev both work).

For posterity’s sake, it seems like conda doesn’t know where to install the various qiime2 modules from if I try to install my plugin from a non-qiime2 environment. But when I’m in the qiime2 environment, those modules are already installed so the installation of my plugin works. Is that a correct interpretation of what’s going on?

Just out of curiosity, is there a way to tell conda where to install the various dependencies from? i.e. tell it to look in the qiime2, bioconda, etc channels for the various dependencies.

And I guess the lack of a linux build explains why installing failed on my cluster. I’ll look into the noarch thing, thanks!

Aughhhhhh :confounded:


(Matthew Ryan Dillon) #7

(Matthew Ryan Dillon) #8

Excellent!

I think so!

Dependency resolution and channels get into the magic inner-workings of conda, which remains a mystery to me! With that said, we have generally had great luck with using the -c $CHANNEL business that conda exposes. Apparently the newer versions of conda support an install syntax that allows specifying the the channel on a per-package basis, :cool:! I think that looks something like this:

conda install conda-forge::python cduvallet::q2_perc_norm 

But, I haven’t really played with that myself, YMMV.

We use the following third-party channels for installing packages:

  -c conda-forge
  -c defaults
  -c bioconda
  -c biocore

And the way conda uses that, is it searches through each channel (in the order listed), for the dependency requested. If it doesn’t find what it is looking for, it moves to the next channel.

Hope that helps - let us know if you need a hand with the noarch build! :t_rex:


(Matthew Ryan Dillon) #9