How to install q2-itsxpress in a virtual machine?

@Adam_Rivers
Wow!! This looks fantastic! Great Job!
I’ve been trying to get it to work on my VM but I don’t know enough about conda environments yet to get it to work. Have you had any success getting this to run on virtual box?

I’ve never tried installing on a virtual machine. You should be able to install miniconda on your VM then use conda to install python, pip, QIIME2 and itsxpress then install Q2-itsxpress from pip. Getting conda to work should be easier that handling the dependencies for QIIME2 yourself.

Its is also entirely possible to install ITSxpress without conda. I test each build of the software using the TravisCI continuous integration server, which builds a Docker image each time. The config file for ITSxpress and Q2-ITSxpress that provide a template to get you started.

2 Likes

@Adam_Rivers thanks! I’d prefer to do it with conda as I know absolutely nothing about continuous integration servers other than they sound like the type of thing that you could find in Docs Delorian ='D

I cloned my VM and tried to follow your instructions, Ive copied in the commands I used as well as the error, I hope that’s helpful and I’m posting correctly (literally first day with an account today!)

Thanks so much for helping!!

PS. not sure if it would be of interest to you but we’re working with a nanopore sequencer (the MINIon) which spits out some very strange reads in comparison to say illumina where everythings like 250bp so I reckon it may be a little more tricky, plus its metagenomic so even worse. Anyway, if data of that sort is of interest to you I can send you a few fastq’s to look at =)

(qiime2-2018.8) [email protected]:~/Desktop/MACE_DEMUX_FASTQs/barcode09$ conda install python
Solving environment: done

Package Plan

environment location: /home/qiime2/miniconda/envs/qiime2-2018.8

added / updated specs:
- python

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
blas-1.0                   |              mkl           6 KB

The following packages will be UPDATED:

blas: 1.0-mkl conda-forge --> 1.0-mkl

Proceed ([y]/n)? y

Downloading and Extracting Packages
blas-1.0 | 6 KB | ################################################################################################################################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(qiime2-2018.8) [email protected]:~/Desktop/MACE_DEMUX_FASTQs/barcode09$ conda install -bioconda itsxpress
usage: conda [-h] [-V] command …
conda: error: unrecognized arguments: -bioconda
(qiime2-2018.8) [email protected]:~/Desktop/MACE_DEMUX_FASTQs/barcode09$ conda install bioconda itsxpress
Solving environment: \ failed

CondaError: KeyboardInterrupt

(qiime2-2018.8) [email protected]:~/Desktop/MACE_DEMUX_FASTQs/barcode09$ conda install -c bioconda itsxpress
Solving environment: done

Package Plan

environment location: /home/qiime2/miniconda/envs/qiime2-2018.8

added / updated specs:
- itsxpress

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
reportlab-3.4.0            |   py35hdd9f355_1         2.4 MB  conda-forge
hmmer-3.2.1                |       hfc679d8_0         7.4 MB  bioconda
biopython-1.70             |      np112py35_1         2.6 MB  bioconda
pillow-5.2.0               |   py35h2dc6135_1        1003 KB  conda-forge
itsxpress-1.7.1            |           py35_0         974 KB  bioconda
bbmap-36.32                |                0         4.0 MB  bioconda
java-jdk-8.0.92            |                1       122.3 MB  bioconda
olefile-0.46               |             py_0          31 KB  conda-forge
mmtf-python-1.0.2          |           py35_0          35 KB  bioconda
------------------------------------------------------------
                                       Total:       140.7 MB

The following NEW packages will be INSTALLED:

bbmap:       36.32-0              bioconda   
biopython:   1.70-np112py35_1     bioconda   
hmmer:       3.2.1-hfc679d8_0     bioconda   
itsxpress:   1.7.1-py35_0         bioconda   
java-jdk:    8.0.92-1             bioconda   
mmtf-python: 1.0.2-py35_0         bioconda   
olefile:     0.46-py_0            conda-forge
pillow:      5.2.0-py35h2dc6135_1 conda-forge
reportlab:   3.4.0-py35hdd9f355_1 conda-forge

Proceed ([y]/n)? y

Downloading and Extracting Packages
reportlab-3.4.0 | 2.4 MB | ################################################################################################################################################################# | 100%
hmmer-3.2.1 | 7.4 MB | ################################################################################################################################################################# | 100%
biopython-1.70 | 2.6 MB | ################################################################################################################################################################# | 100%
pillow-5.2.0 | 1003 KB | ################################################################################################################################################################# | 100%
itsxpress-1.7.1 | 974 KB | ################################################################################################################################################################# | 100%
bbmap-36.32 | 4.0 MB | ################################################################################################################################################################# | 100%
java-jdk-8.0.92 | 122.3 MB | ################################################################################################################################################################# | 100%
olefile-0.46 | 31 KB | ################################################################################################################################################################# | 100%
mmtf-python-1.0.2 | 35 KB | ################################################################################################################################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(qiime2-2018.8) [email protected]:~/Desktop/MACE_DEMUX_FASTQs/barcode09$ pip install q2-itsxpress
Collecting q2-itsxpress
Using cached https://files.pythonhosted.org/packages/b0/90/aec019f9e7633e641e994ce46992392cd5d17f44bb90d0fcbf4dd142ed69/q2_itsxpress-1.7.0-py3-none-any.whl
Requirement already satisfied: itsxpress>=1.7.0 in /home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages (from q2-itsxpress) (1.7.1)
Requirement already satisfied: biopython>=1.60 in /home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages (from itsxpress>=1.7.0->q2-itsxpress) (1.70)
Requirement already satisfied: numpy in /home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages (from biopython>=1.60->itsxpress>=1.7.0->q2-itsxpress) (1.12.1)
Installing collected packages: q2-itsxpress
Successfully installed q2-itsxpress-1.7.0
(qiime2-2018.8) [email protected]:~/Desktop/MACE_DEMUX_FASTQs/barcode09$ qiime itsexpress
QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2018.8/bin/qiime”, line 11, in
sys.exit(qiime())
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 1061, in invoke
cmd_name, cmd, args = self.resolve_command(ctx, args)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 1100, in resolve_command
cmd = self.get_command(ctx, cmd_name)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/commands.py”, line 99, in get_command
plugin = self._plugin_lookup[name]
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/commands.py”, line 75, in _plugin_lookup
import q2cli.cache
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/cache.py”, line 301, in
CACHE = DeploymentCache()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/cache.py”, line 61, in init
self._state = self._get_cached_state(refresh=refresh)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/cache.py”, line 107, in _get_cached_state
self._cache_current_state(current_requirements)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/cache.py”, line 200, in _cache_current_state
state = self._get_current_state()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/cache.py”, line 238, in _get_current_state
plugin_manager = qiime2.sdk.PluginManager()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/plugin_manager.py”, line 44, in new
self._init()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/plugin_manager.py”, line 59, in _init
plugin = entry_point.load()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/pkg_resources/init.py”, line 2325, in load
return self.resolve()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/pkg_resources/init.py”, line 2331, in resolve
module = import(self.module_name, fromlist=[‘name’], level=0)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_itsxpress/init.py”, line 2, in
import q2_itsxpress.plugin_setup
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_itsxpress/plugin_setup.py”, line 35, in
citations=Citations.load(‘citations.bib’, package=‘q2_itsxpress’)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/core/cite.py”, line 30, in load
with open(path) as fh:
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_itsxpress/citations.bib’

Opps. The version 1.7.0 q2-itsxpress that was uploaded to Pypi is missing the citations file. Update q2_itsxpress to version 1.7.1 and it should work.

pip install --upgrade q2-itsxpress

Sorry about that.

2 Likes

That worked perfectly… the first time, but I have a clone vm that I use and I tried it on that, got a similar error so I tried that again, didnt work so I used pip to uninstall then reinstall and it works perfectly now =)

Thanks so much for all the help!!

Hello again!

So I had some serious errors with my vm and vm clone, I dont know if it was ITSxpress related but I had to make a new vm.
I was trying to install and got these errors which look the same as before but it’s now not being fixed by anything, any ideas?

(qiime2-2018.8) [email protected]:~/Documents/pipeline_kit$ pip install --upgrade q2-itsxpress
Requirement already up-to-date: q2-itsxpress in /home/qiime2/miniconda/lib/python3.7/site-packages (1.7.1)
Requirement not upgraded as not directly required: itsxpress>=1.7.0 in /home/qiime2/miniconda/lib/python3.7/site-packages (from q2-itsxpress) (1.7.1)
Requirement not upgraded as not directly required: biopython>=1.60 in /home/qiime2/miniconda/lib/python3.7/site-packages (from itsxpress>=1.7.0->q2-itsxpress) (1.72)
Requirement not upgraded as not directly required: numpy in /home/qiime2/miniconda/lib/python3.7/site-packages (from biopython>=1.60->itsxpress>=1.7.0->q2-itsxpress) (1.15.1)
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
(qiime2-2018.8) [email protected]:~/Documents/pipeline_kit$ pip install --upgrade pip
Collecting pip
Using cached https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 10.0.1
Uninstalling pip-10.0.1:
Successfully uninstalled pip-10.0.1
Successfully installed pip-18.0
(qiime2-2018.8) [email protected]:~/Documents/pipeline_kit$ pip install --upgrade q2-itsxpress
Requirement already up-to-date: q2-itsxpress in /home/qiime2/miniconda/lib/python3.7/site-packages (1.7.1)
Requirement already satisfied, skipping upgrade: itsxpress>=1.7.0 in /home/qiime2/miniconda/lib/python3.7/site-packages (from q2-itsxpress) (1.7.1)
Requirement already satisfied, skipping upgrade: biopython>=1.60 in /home/qiime2/miniconda/lib/python3.7/site-packages (from itsxpress>=1.7.0->q2-itsxpress) (1.72)
Requirement already satisfied, skipping upgrade: numpy in /home/qiime2/miniconda/lib/python3.7/site-packages (from biopython>=1.60->itsxpress>=1.7.0->q2-itsxpress) (1.15.1)
(qiime2-2018.8) [email protected]:~/Documents/pipeline_kit$ qiime itsxpress
QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2018.8/bin/qiime”, line 11, in
sys.exit(qiime())
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 1061, in invoke
cmd_name, cmd, args = self.resolve_command(ctx, args)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 1100, in resolve_command
cmd = self.get_command(ctx, cmd_name)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/commands.py”, line 99, in get_command
plugin = self._plugin_lookup[name]
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/commands.py”, line 75, in _plugin_lookup
import q2cli.cache
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/cache.py”, line 301, in
CACHE = DeploymentCache()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/cache.py”, line 61, in init
self._state = self._get_cached_state(refresh=refresh)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/cache.py”, line 107, in _get_cached_state
self._cache_current_state(current_requirements)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/cache.py”, line 200, in _cache_current_state
state = self._get_current_state()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/cache.py”, line 238, in _get_current_state
plugin_manager = qiime2.sdk.PluginManager()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/plugin_manager.py”, line 44, in new
self._init()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/plugin_manager.py”, line 59, in _init
plugin = entry_point.load()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/pkg_resources/init.py”, line 2325, in load
return self.resolve()
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/pkg_resources/init.py”, line 2331, in resolve
module = import(self.module_name, fromlist=[‘name’], level=0)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_itsxpress/init.py”, line 2, in
import q2_itsxpress.plugin_setup
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_itsxpress/plugin_setup.py”, line 35, in
citations=Citations.load(‘citations.bib’, package=‘q2_itsxpress’)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/core/cite.py”, line 30, in load
with open(path) as fh:
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_itsxpress/citations.bib’

It looks like Q2-ITSxpress was installed in 2 places:

‘/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_itsxpress/

and:
/home/qiime2/miniconda/lib/python3.7/site-packages

Conda did not upgrade the version in your qiime environment because it could use the newer one in your base environment but then Qiime loads the old version from ‘/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_itsxpress/.

I’d try to delete the packages from both locations, restart your qiime environment then install q2_itsxpress with pip in that location.

Hey,

I am installing q2-itsxpress in a linux environment and I am getting the same error message, except Q2-ITSxpress is installed in the same place, and it is still missing the citations file.
"
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/hayden/miniconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_itsxpress/citations.bib’
"
I have tried updating, deleting and re-installing and nothing has worked so far.

@Adam_Rivers happy friyay!

I we’re nearly there with this, so exited to get it going!

heres the wee beasty of an error I get when I try to install now after removing q2_itsxpress directory from
~/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages

after deactivating and reactivating the environment
[email protected]:~/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages$ source activate qiime2-2018.8
(qiime2-2018.8) [email protected]:~/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages$ pip install q2-itxpress
Collecting q2-itxpress
Could not find a version that satisfies the requirement q2-itxpress (from versions: )
No matching distribution found for q2-itxpress

Ugh, this citations file has caused such problems! Sorry. It was missing from the Manifest file so it didn’t get included on pip but my CI server built from Github so it didn’t catch it. I just fixed this and pushed a new release. q2_itsxpress v 1.7.2 has the file.

pip install q2-itsxpress==1.7.2

I don’t know if deleting files mucked things up for you. since its a VM it may be easiest to start with a fresh image and install q2_itsxpress 1.7.1 and itsxpress 1.7.1.

3 Likes

Wow!!! Nice one!!!
I just made a new image and the first thing I ran was pip install q2-itsxpress==1.7.2
This is what I got, looks like it might be working! =D

(qiime2-2018.8) [email protected]:~ pip install q2-itsxpress==1.7.2 Collecting q2-itsxpress==1.7.2 Downloading https://files.pythonhosted.org/packages/45/c8/0f5f0fd8a21fb28aabace831a7284ecbcb31ca9e2e1637fc079dd1699109/q2_itsxpress-1.7.2-py3-none-any.whl Collecting itsxpress>=1.7.0 (from q2-itsxpress==1.7.2) Downloading https://files.pythonhosted.org/packages/16/25/e716865df5608de93db17a03ec45b280fcdea970d6bfdfe4aa9b330566f3/itsxpress-1.7.1-py3-none-any.whl (1.6MB) 100% |████████████████████████████████| 1.6MB 4.2MB/s Collecting biopython>=1.60 (from itsxpress>=1.7.0->q2-itsxpress==1.7.2) Downloading https://files.pythonhosted.org/packages/19/ce/6865387e2c04764fd9b189a06cfd1078df1e30e422ab5494bc4c09ba3838/biopython-1.72-cp35-cp35m-manylinux1_x86_64.whl (2.1MB) 100% |████████████████████████████████| 2.1MB 6.3MB/s Requirement already satisfied: numpy in ./miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages (from biopython>=1.60->itsxpress>=1.7.0->q2-itsxpress==1.7.2) (1.12.1) Installing collected packages: biopython, itsxpress, q2-itsxpress Successfully installed biopython-1.72 itsxpress-1.7.1 q2-itsxpress-1.7.2 (qiime2-2018.8) [email protected]:~ qiime itsxpress
QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
Usage: qiime itsxpress [OPTIONS] COMMAND [ARGS]…

Description: ITSxpress trims amplicon reads down to the just their ITS
region. ITSxpress is designed to support the calling of exact sequence
variants rather than OTUs. This newer method of sequence error-correction
requires quality score data from each sequence, so each input sequence
must be trimmed. ITSxpress makes this possible by taking FASTQ data, de-
replicating the sequences then identifying the start and stop sites using
HMMSearch. Results are parsed and the trimmed files are returned. The ITS
1, ITS2 or the entire ITS region including the 5.8s rRNA gene can be
selected. ITSxpress uses the hmm models from ITSx so results are
comparable.

Plugin website: https://github.com/USDA-ARS-GBRU/q2_itsxpress
ITSxpress: https://github.com/USDA-ARS-GBRU/itsxpress

Getting user support: Please post to the QIIME 2 forum for help with this
plugin: https://forum.qiime2.org

Options:
–version Show the version and exit.
–citations Show citations and exit.
–help Show this message and exit.

Commands:
trim-pair Trim paired-end reads, output merged reads for
use with Deblur
trim-pair-output-unmerged Trim paired-end reads, output unmerged reads for
use with Dada2
trim-single Trim single-end reads
(qiime2-2018.8) [email protected]:~$

1 Like

so I thought i out to do a test run so I loaded up the data from your tutorial and got this error

(qiime2-2018.8) [email protected]:~/test$ qiime itsxpress trim-pair-output-unmerged\

–i-per-sample-sequences sequences.qza
–p-region ITS1
–p-taxa F
–o-trimmed trimmed.qza

Plugin error from itsxpress:

[Errno 2] No such file or directory: ‘bbmerge.sh’

Debug info has been saved to /tmp/qiime2-q2cli-err-l0m3y7ec.log
(qiime2-2018.8) [email protected]:~/test$ qiime itsxpress trim-pair-output-unmerged\

–i-per-sample-sequences sequences.qza
–p-region ITS1
–p-taxa F
–p-cluster-id 1.0
–p-threads 2
–o-trimmed trimmed_exact.qza
Plugin error from itsxpress:

[Errno 2] No such file or directory: ‘bbmerge.sh’

Debug info has been saved to /tmp/qiime2-q2cli-err-259m0a7n.log
(qiime2-2018.8) [email protected]:~/test$

would it be useful for me to send you the debug info?

The program cannot find bbtools which is normally installed by conda as part of the itsxpress package. Likely causes are that you installed itsxpress with pip not conda or that you installed itsxpress outside of your current qiime2-2018.8 environment

Try this:

# Start your qiime environment
source activate qiime2-2018.8


# Check which versions of q2-itsxpress and itsxpress are installed.
conda list |grep "itsxpress"

# It should look like
itsxpress     1.7.1    py35_0    bioconda
q2-itsxpress  1.7.2    <pip>

#Check which version of bbtools (formerly bbmap) is available
conda list |grep "bbmap"
# it should look about like this (the version does not matter)
bbmap     38.22     h470a237_0    bioconda 

# If needed install itsxpress and q2-itsxpress
conda install -c bioconda itsxpress
pip install q2-itsxpress

If they are already installed you may need to select upgrade instead of install.

2 Likes

Thanks so much! All fixed =D

You are only using 1 thread. Use the --p-threads flag. The demo has 2 samples with 10,000 reads. Real experiments have more samples with more sequences so they will take longer.

1 Like

Ahhhh! Amazing! I dont know why I didnt think of that! thanks Adam!

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.