micom build error - can't pickle PyCapsule objects

I get the following error when testing micom. I am using gubori. Installation of q2_micom and gubori seems good. Any ideas?

(qiime2-2021.2) [[email protected]/bioinfo2 test]$ qiime micom build --i-abundance crc_table.qza --i-taxonomy crc_taxa.qza --i-models agora103_genus.qza --p-cutoff 0.0001 --p-threads 8 --o-community-models models.qza --verbose
Merged with the database using ranks: genus
Each community model contains 17-36 taxa (average 27+-6).
Running ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Traceback (most recent call last):
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/q2cli/commands.py", line 329, in __call__
    results = action(**arguments)
  File "<decorator-gen-247>", line 2, in build
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    output_types, provenance)
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/q2_micom/_build.py", line 74, in build
    mw.build(tax, model_folder, out_folder, cutoff, threads)
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/micom/workflows/build.py", line 93, in build
    res = workflow(build_and_save, args, threads)
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/micom/workflows/core.py", line 41, in workflow
    results = list(it)
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/rich/progress.py", line 149, in track
    sequence, total=total, description=description, update_period=update_period
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/rich/progress.py", line 700, in track
    for value in sequence:
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x2af756c00898>'. Reason: 'TypeError("can't pickle PyCapsule objects",)'

Looks like it may be related to Workflows are broken with Gurobi · Issue #24 · micom-dev/micom · GitHub. Can you try updating optlang in the environment to see if that fixes the issue?

conda install -c conda-forge -c bioconda -c defaults "optlang>=1.5.2"

What version of Gurobi are you running?

I got the same error after updating optlang
I am using Gurobi 9

Okay. I just tried with Gurobi 9.1.2 and it seemed to work on my machine, which is probably not that helpful to you :sweat_smile:
Just to check your Gurobi setup is okay, can you run the following in your terminal with the environment activated and post the output?

python -c "from cobra.test import create_test_model; mod=create_test_model(); mod.solver='gurobi'; print(mod.optimize())"

This should return something like:

<Solution 0.380 at 0x7f040a6c49a0>

This will also check if you have the right Gurobi license since that model is too large to be solved with the restricted license.

The output from running conda list in the environment would also be very helpful.

Output from running the python code:

python -c "from cobra.test import create_test_model; mod=create_test_model(); mod.solver='gurobi'; print(mod.optimize())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/cobra/util/context.py", line 111, in wrapper
    func(self, new_value)
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/cobra/core/model.py", line 148, in solver
    self._solver = interface.Model.clone(self._solver)
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/optlang/interface.py", line 1141, in clone
    new_model = cls.from_json(model.to_json())
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/optlang/interface.py", line 1633, in from_json
    model = cls()
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/optlang/interface.py", line 1191, in __init__
  File "/ddn/gs1/home/randallta2/miniconda2/envs/qiime2-2021.2/lib/python3.6/site-packages/optlang/gurobi_interface.py", line 531, in _initialize_problem
    self.problem = gurobipy.Model()
  File "src/gurobipy/env.pxi", line 67, in gurobipy.Env.__init__
  File "src/gurobipy/gurobi.pxi", line 32, in gurobipy.gurobi._getdefaultenv
  File "src/gurobipy/env.pxi", line 67, in gurobipy.Env.__init__
gurobipy.GurobiError: HostID mismatch (licensed to dae9ad33, hostid is daeb97bf)

They gave me a one year license, it was a bit of a problem getting that since am within NIH, so govt, not academic, so this could be an issue.

One line from the conda list:
gurobi 9.1.2 py36_0 gurobi

I could send the entire output (478 lines) if needed.

Gurobi licenses are only ever valid on the same equipment they have been activated. If you don't use the Gurobi license on the old machine anymore you can just reactivate it on the new machine with grbgetkey as shown here. But you don't really need that because:

  1. Since 0.24.0 MICOM can run with an open source solver (OSQP) which will be installed along with MICOM. It has lower accuracy than Gurobi but will be much faster. Sorry for not updating the docs for q2-micom yet.

  2. We recommend using CPLEX over Gurobi anyways. You can get an academic license from IBM for government organizations (I have done so before myself) and CPLEX will be equally or more accurate than Gurobi and about 100x faster in MICOM. Also, CPLEX is just a download and requires no license activation, so you can move it to a new machine.

Let me know if any of those options work for you :grinning:

I am using qiime2-2021.2, should I update that to get the 0.24.0 version of MICOM?

No need to update Qiime 2. Just need to make sure you have micom>=0.24 installed. The conda list output from before should tell you that. If it's an older version you can just upgrade it with pip install -U micom in the environment. To use OSQP with q2-micom just uninstall Gurobi from the conda env with conda uninstall gurobi after that qiime micom build will use OSQP by default.

1 Like

Great explanation. After uninstalling Gurobi micom worked on the test data. Thanks

Awesome great to hear that.