Qiime2 installation with dependencies downloaded locally

Hi all
Thanks for Qiime2! Its a great tool and nice to use.

I am behind an institution firewall that blocks mass storage places, so I cannot download using wget. Up to version 2017.11 I could download the dependencies externally using the provided .txt file. Move them to the local computer and install using the --offline tag. Worked perfect. I tried to upgrade to 2018.2 now. The .yml file makes it slightly more complicated but it still works to download the packages on an external machine. But when I try installing it using the --offline flag I get the following error message

Compared to previous installations the miniconda/envs/qiime folder gets created but stays empty. Would you have hint on how to solve this error?

I am using miniconda3, pyhton 3.6.4 on debian

Thanks in advance!

1 Like

Hi @atg,

It doesn’t look like there’s a good way to do this with the .yml file. However, you should still be able to use your original process! You’ll just be creating the old txt version first on the machine that wasn’t behind the firewall.

On the host that has the working environment run (inside that environment):

conda list --export > your-new-environment.txt

This is the file you’ll want to transfer to the offline host and use with the --offline tag like you’ve done before.

Let me know if that works!

1 Like

still no luck. Can easily create that file and try to install with
conda create --name qiime2_2018.12 --file Environment_qiime2_2018_2.txt --offline /home/bioinf/miniconda3/pkgs/*.tar.bz2

the error was always python related and the machine which has a working installation has python3.5 but the offline machine has python3.6. May that be a source of error? The miniconda I’m using is also python 3.6

Hi @atg,

Could you post the error? I think your command should work, so I’m not sure what’s going on.

Another avenue you can explore is creating file-system conda channels (called custom channels). It’s a little bit more involved, but if you do this often enough it is probably worthwhile. It’s also more likely to work like you would expect as it’s treated just like any other conda channel: https://conda.io/docs/user-guide/tasks/create-custom-channels.html

We use custom channels pretty extensively in our build/CI process and they do work very well. Just remove the channel list in the yaml file (or use the txt file) and pass your path to the install command (--channel /path/here).

HI Evan, thanks for your help

When I run the command: conda create --name qiime2_2018.2 --file Environment_qiime2_2018_2.txt --offline /home/bioinf/miniconda2/pkgs/*.tar.bz2

I get the error:

Preparing transaction: done
Verifying transaction: done
Executing transaction: failed
ERROR conda.core.link:_execute(481): An error occurred while installing package '::jupyter_core-4.4.0-py_0'.
OSError(2, 'No such file or directory')
Attempting to roll back.

Rolling back transaction: done

OSError(2, 'No such file or directory')

Although the jupyter package is in the pkgs folder as a tar.bz2

When I run in verbose mode I get for every package:

file exists, but clobbering: u'/home/bioinf/miniconda2/envs/qiime2_2018.2/lib/python3.5/site-packages/pip/vcs/pycache/subversion.cpython-35.pyc'

and then:

===> LINKING PACKAGE: ::jupyter_core-4.4.0-py_0 <===
prefix=/home/bioinf/miniconda2/envs/qiime2_2018.2
source=/home/bioinf/miniconda2/pkgs/jupyter_core-4.4.0-py_0

===> REVERSING PACKAGE LINK: ::jupyter_core-4.4.0-py_0 <===
prefix=/home/bioinf/miniconda2/envs/qiime2_2018.2

done
Traceback (most recent call last):
File "/home/bioinf/miniconda2/lib/python2.7/site-packages/conda/exceptions.py", line 789, in call
return func(*args, **kwargs)
File "/home/bioinf/miniconda2/lib/python2.7/site-packages/conda/cli/main.py", line 78, in _main
exit_code = do_call(args, p)
File "/home/bioinf/miniconda2/lib/python2.7/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
exit_code = getattr(module, func_name)(args, parser)
File "/home/bioinf/miniconda2/lib/python2.7/site-packages/conda/cli/main_create.py", line 11, in execute
install(args, parser, 'create')
File "/home/bioinf/miniconda2/lib/python2.7/site-packages/conda/cli/install.py", line 173, in install
explicit(args_packages, prefix, verbose=not context.quiet)
File "/home/bioinf/miniconda2/lib/python2.7/site-packages/conda/misc.py", line 90, in explicit
txn.execute()
File "/home/bioinf/miniconda2/lib/python2.7/site-packages/conda/core/link.py", line 225, in execute
self._execute(tuple(concat(interleave(itervalues(self.prefix_action_groups)))))
File "/home/bioinf/miniconda2/lib/python2.7/site-packages/conda/core/link.py", line 500, in _execute
rollback_excs,
CondaMultiError: [Errno 2] No such file or directory

It seems to me that he tries to write into the pkgs folder but doesnt want to install stuff in the envs/Qiime/

The installation file from the machine with the working installation looks like that:

folderEnvironment_qiime2_2018_2.txt (5.5 KB)

in case it helps, conda info gives me that:

active environment : None
user config file : /home/bioinf/.condarc
populated config files : /home/bioinf/.condarc
conda version : 4.4.10
conda-build version : not installed
python version : 2.7.14.final.0
base environment : /home/bioinf/miniconda2 (writable)
channel URLs : file:///home/bioinf/linux-64
file:///home/bioinf/noarch
file:///home/bioinf/miniconda2/pkgs/linux-64
file:///home/bioinf/miniconda2/pkgs/noarch
r/linux-64
r/noarch
main/linux-64
main/noarch
Anaconda packages for Linux x86_64 (64-bit)
Anaconda packages (noarch)
r/linux-64
r/noarch
Anaconda extras for Linux x86_64 (64-bit)
Anaconda extras (noarch)
conda-forge/linux-64
conda-forge/noarch
package cache : /home/bioinf/miniconda2/pkgs
/home/bioinf/.conda/pkgs
envs directories : /home/bioinf/miniconda2/envs
/home/bioinf/.conda/envs
platform : linux-64
user-agent : conda/4.4.10 requests/2.18.4 CPython/2.7.14 Linux/3.16.0-4-amd64 debian/8 glibc/2.19
UID:GID : 1000:1000
netrc file : None
offline mode : False

Hey @atg,

It sounds like all of your packages are already in the conda pkg cache? In that case, you probably don't need the last argument where you glob the files /home/bioinf/miniconda2/pkgs/*.tar.bz2.

The --offline flag on its own should be enough.

Assuming that doesn't work. I would probably give up on trying to get conda to act the way you need via offline/pkg-cache and instead create a custom channel which is pretty straight forward with conda index. Then you aren't as reliant on how conda organizes itself internally.

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.