Environment and installation issues

qiuwa · March 6, 2019, 5:13pm

[swadmin@sirius ~/qiime2]$ echo $TMPDIR
/scratch/swadmin

The following is the logile info

[swadmin@sirius ~/qiime2]$ more /scratch/swadmin/qiime2-q2cli-err-8t9fs51h.log
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /scratch/swadmin/tmpghjjbq5k/forward /scratch/swadmin/tmpghjjbq5k/reverse /scratch/swadmin/tmpghjjbq5k/output.tsv.biom /scratch/swadmin/tmpghjjbq5k/track.tsv /scratch/swad
min/tmpghjjbq5k/filt_f /scratch/swadmin/tmpghjjbq5k/filt_r 150 150 5 5 2.0 2 consensus 1.0 1 1000000

Traceback (most recent call last):
File "/gsfs0/public/qiime/miniconda3/envs/qiime2-2018.6/lib/python3.5/site-packages/q2cli/commands.py", line 274, in call
results = action(**arguments)
File "", line 2, in denoise_paired
File "/gsfs0/public/qiime/miniconda3/envs/qiime2-2018.6/lib/python3.5/site-packages/qiime2/sdk/action.py", line 232, in bound_callable
output_types, provenance)
File "/gsfs0/public/qiime/miniconda3/envs/qiime2-2018.6/lib/python3.5/site-packages/qiime2/sdk/action.py", line 367, in callable_executor
output_views = self._callable(**view_args)
File "/gsfs0/public/qiime/miniconda3/envs/qiime2-2018.6/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 229, in denoise_paired
run_commands([cmd])
File "/gsfs0/public/qiime/miniconda3/envs/qiime2-2018.6/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 36, in run_commands
subprocess.run(cmd, check=True)
File "/gsfs0/public/qiime/miniconda3/envs/qiime2-2018.6/lib/python3.5/subprocess.py", line 383, in run
with Popen(*popenargs, **kwargs) as process:
File "/gsfs0/public/qiime/miniconda3/envs/qiime2-2018.6/lib/python3.5/subprocess.py", line 676, in init
restore_signals, start_new_session)
File "/gsfs0/public/qiime/miniconda3/envs/qiime2-2018.6/lib/python3.5/subprocess.py", line 1289, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'run_dada_paired.R'

Would you point me the path of the 'run_dada_paired.R" in qiime2 package? It seems the program can't find it.

Thanks!

Wei

thermokarst · March 6, 2019, 5:17pm

This is just a very small indication of a much larger problem.

It looks like you still have an incomplete deployment. Can you please run the two following commands, inside your qiime 2 conda environment and provide the results here:

env
conda list

Thanks! :qiime2:

thermokarst · March 6, 2019, 5:19pm

Also, your most recent output indicates that you ran that command in a 2018.6 environment, while above you demonstrated that you installed 2019.1

I will add a third command to my request: conda env list. Thanks!

qiuwa · March 6, 2019, 5:37pm

I see. Even I did "conda clean -y --all" then run "conda env create -n qiime2-2019.1 --file qiime2-2019.1-py36-linux-conda.yml"

This might be a bug.

I'll reinstall it from scratch.

Thanks!

Wei

thermokarst · March 6, 2019, 5:48pm

I don't think so, this doesn't appear to be a bug, rather, the problem appears to be related to how your computation environment is set up.

Okay, but just to make it clear, that isn't what I suggested. If you can provide the output from the three commands I requested above, that would be very helpful.

qiuwa · March 6, 2019, 9:08pm

[swadmin@sirius qiime]$ conda env create -n qiime2-2019.1 --file qiime2-2019.1-py36-linux-conda.yml
Collecting package metadata: done
Solving environment: done

--
atomicwrites-1.2.1 | 9 KB | ############################################################################### | 100%
r-utf8-1.1.4 | 137 KB | ############################################################################### | 100%
bioconductor-biobase | 2.0 MB | ############################################################################### | 100%
decorator-4.3.2 | 11 KB | ############################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: - Enabling notebook extension jupyter-js-widgets/extension...
- Validating: OK
do ne

To activate this environment, use:

> source activate qiime2-2019.1

To deactivate an active environment, use:

> source deactivate

I have finished the installation.
[swadmin@sirius qiime] activate qiime2-2019.1 QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment. [swadmin@sirius qiime] which qiime2
qiime: Command not found.

qiime is not in miniconda3/bin/ directory. The python version is 3.7.1

Thanks

Wei

thermokarst · March 6, 2019, 9:17pm

Thanks @qiuwa!

There is no command shipped with the qiime 2 environment called qiime2, which is why you aren't seeing that command.

Can you please provide the information I have asked for previously? From within your new conda env:

env
conda list
conda env list

Thanks!

qiuwa · March 6, 2019, 9:21pm

sorry. I copyed wrong.

[swadmin@sirius qiime]$ which qiime
qiime: Command not found.

qiuwa · March 6, 2019, 9:23pm

[swadmin@sirius qiime]$ env
TERM=xterm
HOME=/data/swadmin
SHELL=/bin/tcsh
USER=swadmin
LOGNAME=swadmin
PATH=/usr/public/qiime/miniconda3//bin:/gsfs0/public/fsl/5.0.11/bin:/data/swadmin/bin:/cm/shared/apps/torque/bin:/cm/shared/apps/torque/sbin:/cm/shared/apps/moab/bin:/cm/shared/apps/moab/sbin:/usr/public/moaB/scripts:/cm/shared/apps/torque/6.1.1/sbin:/cm/shared/apps/torque/6.1.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/sbin:/cm/local/apps/environment-modules/3.2.10/bin:/opt/dell/srvadmin/bin:/data/swadmin/abin
XDG_SESSION_ID=38135
HOSTTYPE=x86_64-linux
VENDOR=unknown
OSTYPE=linux
MACHTYPE=x86_64
SHLVL=1
PWD=/usr/public/qiime
GROUP=swadmin
HOST=sirius
REMOTEHOST=w4wqti0558onei.bc.edu
MAIL=/var/spool/mail/swadmin
HOSTNAME=sirius
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:*.xspf=01;36:
LANG=en_US.utf8
LESSOPEN=||/usr/bin/lesspipe.sh %s
MANPATH=/cm/shared/apps/torque/current/share/man:/cm/shared/apps/torque/6.1.1/share/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man:/cm/local/apps/environment-modules/current/share/man
ENABLE_LMOD=0
MODULEPATH=/cm/local/modulefiles:/cm/shared/modulefiles
MODULE_VERSION=3.2.10
MODULE_VERSION_STACK=3.2.10
MODULESHOME=/cm/local/apps/environment-modules/3.2.10/Modules/3.2.10
LOADEDMODULES=torque/6.1.1:moab:qiime2/qiime2-p3.6.5
QT_GRAPHICSSYSTEM_CHECKED=1
LD_LIBRARY_PATH=/cm/shared/apps/torque/lib:/cm/shared/apps/torque/6.1.1/lib/
MOABHOMEDIR=/cm/shared/apps/moab
PBS_HOME=/cm/shared/apps/torque/var/spool
LMFILES=/cm/shared/modulefiles/torque/6.1.1:/cm/shared/modulefiles/moab:/cm/shared/modulefiles/qiime2/qiime2-p3.6.5
FSLDIR=/gsfs0/public/fsl/5.0.11
FSLOUTPUTTYPE=NIFTI_GZ
FSLMULTIFILEQUIT=TRUE
FSLTCLSH=/gsfs0/public/fsl/5.0.11/bin/fsltclsh
FSLWISH=/gsfs0/public/fsl/5.0.11/bin/fslwish
FSLLOCKDIR=
FSLMACHINELIST=
FSLREMOTECALL=
FSLGECUDAQ=cuda.q
LC_ALL=en_US.utf8
TMPDIR=/scratch/swadmin

[swadmin@sirius qiime]$ conda list

packages in environment at /gsfs0/public/qiime/miniconda3:

[swadmin@sirius qiime]$ conda env list

conda environments:

                     /gsfs0/public/anaconda/4.4.0-P3.6
                     /gsfs0/public/anaconda/4.4.0-P3.6/envs/pydicomenv
                     /gsfs0/public/dacosta/miniconda3/envs/shi7

base * /gsfs0/public/qiime/miniconda3
qiime2-2019.1 /gsfs0/public/qiime/miniconda3/envs/qiime2-2019.1
/usr/public/fsl/5.0.11/fslpython/envs/fslpython
/usr/public/pytorch/anaconda3
/usr/public/qiime/miniconda3

Thanks

thermokarst · March 6, 2019, 9:28pm

These commands all appear to be run from the base conda environment.

You will need to run source activate qiime2-2019.1 before running those commands.

qiuwa · March 6, 2019, 9:45pm

[swadmin@sirius qiime]$ activate  qiime2-2019.1
[swadmin@sirius qiime]$ which qiime
qiime: Command not found.
[swadmin@sirius qiime]$ conda list
# packages in environment at /gsfs0/public/qiime/miniconda3:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py37_1000    conda-forge
cffi                      1.11.5           py37he75722e_1
chardet                   3.0.4                    py37_1
conda                     4.6.7                    py37_0    conda-forge
conda-env                 2.6.0                         1
cryptography              2.4.2            py37h1ba5d50_0
idna                      2.8                      py37_0
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libstdcxx-ng              8.2.0                hdf63c60_1
ncurses                   6.1                  he6710b0_1
openssl                   1.1.1b               h14c3975_0    conda-forge
pip                       18.1                     py37_0
pycosat                   0.6.3            py37h14c3975_0
pycparser                 2.19                     py37_0
pyopenssl                 18.0.0                   py37_0
pysocks                   1.6.8                    py37_0
python                    3.7.1                h0371630_7
readline                  7.0                  h7b6447c_5
requests                  2.21.0                   py37_0
ruamel_yaml               0.15.46          py37h14c3975_0
setuptools                40.6.3                   py37_0
six                       1.12.0                   py37_0
sqlite                    3.26.0               h7b6447c_0
tk                        8.6.8                hbc83047_0
urllib3                   1.24.1                   py37_0
wget                      1.19.5               h20c2e04_0
wheel                     0.32.3                   py37_0
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zlib                      1.2.11               h7b6447c_3
[swadmin@sirius qiime]$ conda env list
# conda environments:
#
                         /gsfs0/public/anaconda/4.4.0-P3.6
                         /gsfs0/public/anaconda/4.4.0-P3.6/envs/pydicomenv
                         /gsfs0/public/dacosta/miniconda3/envs/shi7
base                  *  /gsfs0/public/qiime/miniconda3
qiime2-2019.1            /gsfs0/public/qiime/miniconda3/envs/qiime2-2019.1
                         /usr/public/fsl/5.0.11/fslpython/envs/fslpython
                         /usr/public/pytorch/anaconda3
                         /usr/public/qiime/miniconda3

It seems the same result. I just find the qiime is in /gsfs0/public/qiime/miniconda3/envs/qiime2-2019.1/bin/

Can I just set PATH to access it? What else parameters I need to set?

thermokarst · March 6, 2019, 9:59pm

Hey there @qiuwa! Please carefully double-check the command, you have a typo above.

The command is source activate qiime2-2019.1, but you wrote activate qiime2-2019.1

This configuration is handled by conda, when you run source activate ....

qiuwa · March 6, 2019, 10:14pm

Thanks for all your help. Appreciated!
I get this run by setting the PATH
I'll keep you update.

thermokarst · March 7, 2019, 3:12pm

Hey there @qiuwa! I don't recommend hacking the PATH like that --- conda is designed to handle these things for you. My concern is that by deliberately ignoring the errors you have observed so far, you might get yourself into a situation where this deployment of QIIME 2 could produce incorrect results, due to mixups in the PATH ("wrong" versions of supporting tools might get pulled into your workflow on accident without you even knowing). Also, there are some conda-related hooks that get run when you activate and deactivate your QIIME 2 environment via conda, that won't get run using this PATH hacking.

You never directly responded to my suggestion above --- what happened when you ran source activate qiime2-2019.1?

Also, to provide some narrative for other users who might stumble across this thread --- the issues encountered here aren't specific to QIIME 2, rather, they have to do with how conda is deployed in this particular computational environment.

ccing @Heather_Olins - I think you should be aware of these concerns, especially if you are using this same environment, too.

qiuwa · March 8, 2019, 7:25pm

Sorry for the delay. Thank you very much for all your help!

We are using modules to set $PATH for running application on Linux cluster system. After running "conda env list", I find the qiime was in /gsfs0/public/qiime/miniconda3/envs/qiime2-2019.1. So I did a try to set the $PATH. We are working on qiime tutorial and hope it'll go well.

Please take a look output for source activate qiime2-2019.1.
[swadmin@sirius qiime] source /usr/public/qiime/miniconda3//bin/activate qiime2-2019.1 _CONDA_ROOT=/gsfs0/public/qiime/miniconda3: Command not found. _CONDA_ROOT: Undefined variable. [swadmin@sirius qiime]

Hope it's helpful.

Thanks again!

Wei

qiuwa · March 8, 2019, 9:11pm

I post the result again as it was cut.

[swadmin@sirius qiime]$ source /usr/public/qiime/miniconda3//bin/activate qiime2-2019.1

_CONDA_ROOT=/gsfs0/public/qiime/miniconda3: Command not found.
_CONDA_ROOT: Undefined variable.

thermokarst · March 8, 2019, 9:27pm

These errors are yet more indications that your computation environment is not configured correctly --- my best guess is that there is something wrong with your PATH, but I really don't know. You also appear to have a typo in your command path (two slashes between miniconda3 and bin). We are unable to provide additional support here --- my suggestion is that you contact the conda team via their support channels to see if they can provide some guidance. I would also recommend getting your system administrator involved with this discussion.

qiuwa · March 8, 2019, 9:38pm

I would like to keep you update .. so far it works well. But I don't know if there is any other problems coming out while we are working on qiime tutorial. Greatly appreciate all your help. Have a great weekend. Wei

Environment and installation issues

[swadmin@sirius qiime]$ conda env create -n qiime2-2019.1 --file qiime2-2019.1-py36-linux-conda.yml Collecting package metadata: done Solving environment: done

To activate this environment, use:

> source activate qiime2-2019.1

To deactivate an active environment, use:

> source deactivate

packages in environment at /gsfs0/public/qiime/miniconda3:

Name Version Build Channel

conda environments:

[swadmin@sirius qiime]$ conda env create -n qiime2-2019.1 --file qiime2-2019.1-py36-linux-conda.yml
Collecting package metadata: done
Solving environment: done