QIIME2 2021.11 in GoogleColab not installing

[Lauren_Anderson]

I am trying to install QIIME2 in Google Colab. I specifically need QIIME2 2021.11 to run with PICRUSt2. I was using the installation tutorial by @cdiener, and I can get it to install the most recent version, no problem. I can also install 2020.11 (I only know that because I grabbed the wrong one by mistake ). However, when I try to get 2021.11, it hangs. Specifically at the line "o, e = r.communicate()".

I can't share an error message, because it never finishes running and my Colab notebook eventually disconnects from the runtime.

Code Below. I will note the changes I made to it at the bottom:

"""Set up Qiime 2 on Google colab.
Do not use this on o local machine, especially not as an admin!
"""

import os
import sys
from subprocess import Popen, PIPE

r = Popen(["pip", "install", "rich"])
r.wait()
from rich.console import Console # noqa
con = Console()

has_conda = "conda version" in os.popen("conda info").read()
has_qiime = "QIIME 2 release:" in os.popen("qiime info").read()

MINICONDA_PATH = (
** "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh"**
)
QIIME_YAML_URL = (
** "https://data.qiime2.org/distro/core/qiime2-2021.11-py38-linux-conda.yml"**
)
QIIME_YAML = os.path.basename(QIIME_YAML_URL)

def cleanup():
** """Remove downloaded files."""**
** if os.path.exists("Miniconda3-latest-Linux-x86_64.sh"):**
** os.remove("Miniconda3-latest-Linux-x86_64.sh")**
** if os.path.exists(QIIME_YAML):**
** os.remove(QIIME_YAML)**
** con.log("Cleaned up unneeded files.")**

def run_and_check(args, check, message, failure, success, console=con):
** """Run a command and check that it worked."""**
** console.log(message)**
** r = Popen(args, env=os.environ, stdout=PIPE, stderr=PIPE,**
** universal_newlines=True)**
** o, e = r.communicate()**
** out = o + e**
** if r.returncode == 0 and check in out:**
** console.log("[blue]%s[/blue]" % success)**
** else:**
** console.log("[red]%s[/red]" % failure, out)**
** cleanup()**
** sys.exit(1)**

def _hack_in_the_plugins():
** """Add the plugins to QIIME."""**
** import qiime2.sdk as sdk**
** from importlib.metadata import entry_points**

** pm = sdk.PluginManager(add_plugins=False)**
** for entry in entry_points()["qiime2.plugins"]:**
** plugin = entry.load()**
** package = entry.value.split(':')[0].split('.')[0]**
** pm.add_plugin(plugin, package, entry.name)**

if name == "main":
** if not has_conda:**
** run_and_check(**
** ["wget", MINICONDA_PATH],**
** "saved",**
** " Downloading miniconda...",**
** "failed downloading miniconda ",**
** " Done."**
** )**

** run_and_check(**
** ["bash", "Miniconda3-latest-Linux-x86_64.sh", "-bfp", "/usr/local"],**
** "installation finished.",**
** " Installing miniconda...",**
** "could not install miniconda ",**
** " Installed miniconda to /usr/local "**
** )**
** else:**
** con.log(" Miniconda is already installed. Skipped.")**

** if not has_qiime:**
** run_and_check(**
** ["wget", QIIME_YAML_URL],**
** "saved",**
** " Downloading Qiime 2 package list...",**
** "could not download package list ",**
** " Done."**
** )**

** run_and_check(**
** ["conda", "env", "update", "-n", "base", "--file",**
** "qiime2-2021.11-py38-linux-conda.yml"],**
** "To activate this environment, use",**
** " Installing Qiime 2. This may take a little bit.\n ",**
** "could not install Qiime 2 ",**
** " Done."**
** )**
** else:**
** con.log(" Qiime 2 is already installed. Skipped.")**

** run_and_check(**
** ["qiime", "info"],**
** "QIIME 2 release:",**
** " Checking that Qiime 2 command line works...",**
** "Qiime 2 command line does not seem to work ",**
** " Qiime 2 command line looks good "**
** )**

** if sys.version_info[0:2] == (3, 8):**
** sys.path.append("/usr/local/lib/python3.8/site-packages")**
** con.log(" Fixed import paths to include Qiime 2.")**

** con.log(" Checking if Qiime 2 import works...")**
** try:**
** import qiime2 # noqa**
** except Exception:**
** con.log("[red]Qiime 2 can not be imported [/red]")**
** sys.exit(1)**
** con.log("[blue] Qiime 2 can be imported [/blue]")**

** con.log(" Setting up QIIME 2 plugins...")**
** try:**
** _hack_in_the_plugins()**
** from qiime2.plugins import feature_table # noqa**
** except Exception:**
** con.log("[red]Could not add the plugins [/red]")**
** sys.exit(1)**
** con.log("[blue] Plugins are working [/blue]")**

** cleanup()**

** con.log("[green]Everything is A-OK. "**
** "You can start using Qiime 2 now :thumbs_up:[/green]")**

\

I changed the 2 chunks below to reflect the 2021.11 version I wanted. (They were 2022.8 in the original code, I believe)

QIIME_YAML_URL = (
"https://data.qiime2.org/distro/core/qiime2-2021.11-py38-linux-conda.yml"

    run_and_check(
        ["conda", "env", "update", "-n", "base", "--file",
         "qiime2-2021.11-py38-linux-conda.yml"],
        "To activate this environment, use",
        ":mag: Installing Qiime 2. This may take a little bit.\n :clock1:",
        "could not install Qiime 2 :sob:",
        ":mag: Done."
    )
else:
    con.log(":mag: Qiime 2 is already installed. Skipped.")

[cdiener (Christian Diener)]

There is a more up to date version here. That also allows you to specify the Qiime2 version that you want to install. You can get it in your Colab notebook with

!wget https://raw.githubusercontent.com/Gibbons-Lab/isb_course_2022/main/setup_qiime2

So for 2021.11 it should be

%run setup_qiime2 2021.11

That said it's also hanging for me, and it looks it's because there are conflicts in the conda env file. If i try it on my local machine I get:

Encountered problems while solving:
  - package perl-list-moreutils-0.413-1 requires perl >=5.22.0,<5.23.0, but none of the providers can be installed

So it looks like the env file provided by Qiime2 is broken.

[Lauren_Anderson]

Thank you so much for your response. I tried 2021.11 both locally and in Colab and get the exact same problem you mentioned.

Encountered problems while solving:

• package perl-list-moreutils-0.413-1 requires perl >=5.22.0,<5.23.0, but none of the providers can be installed

I am trying the Colab download with 2020.6 (Songbird version requirement) and it has been running for 40+ min, so I don't have much hope it'll complete considering It's usually <15 min. However, 2020.6 did install locally without issue.

[cdiener (Christian Diener)]

Yeah, it looks like there are some issues with some of the package lists for older versions. That can sometimes happen as conda repositories rebuild old package versions. Maybe @lizgehret or @ebolyen have some more insights.

At least the latest version of the setup_qiime2 script should make it easier to check a few versions and see what works, since you can provide the version you want to install as an option.

[Lauren_Anderson]

Thank you for you help. I really appreciate it!