Qiime 2022.8 install broken on linux (incorrect HDF5 versions)

[cdiener (Christian Diener)]

Following the usual install of Qiime 2 2022.8 as described in the docs for a Miniconda install on Linux will lead to a broken install due to a mismatch in HDF5 libraries.

(base) [email protected] [~] conda activate qiime2-2022.8                                                                                                                        
QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
[...]
  File "/users/cdiener/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/plugin_setup.py", line 16, in <module>
    from unifrac._meta import CONSOLIDATIONS
  File "/users/cdiener/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/unifrac/__init__.py", line 30, in <module>
    from unifrac._api import ssu, faith_pd, ssu_to_file, ssu_inmem
ImportError: libhdf5_cpp.so.200: cannot open shared object file: No such file or directory

It seems like unifrac-binaries is compiled against a recent version of HDF5 (1.12.1) but the environment file pins a pretty outdated version (1.10.6) that does not include the required shared objects.

The issue can be reproduced on Google Colab as well, so it seems reproducible.

[cdiener (Christian Diener)]

The issue seem to be broken dependency pins in bioconda/unifrac 1.1.1, especially the ones derived from the pin_compatible macro.

Those are not compatible to what was used to build bioconda/unifrac-binaries 1.1.1 (h7e6beb3_2 which were built with hdf5 1.12.1. OSX should be unaffected as long as there are no updates to the bioconda build system which would break those as well, as it would rebuild against hdf5 1.12.1 as well.

[guojun_wu]

Meet same issue.
ImportError: libhdf5_cpp.so.200: cannot open shared object file: No such file or directory

[DuncanSussfeld]

I also face this problem, trying to install Qiime2 on ubuntu 20.04 after a fresh Miniconda install.

[antgonza (Antonio Gonzalez Pena)]

Adding to the report, in case it helps.

I'm currently getting the same error while building the Qiita Q2 plugin; however, it worked fine 4 days ago; so I think is something that change 3 days - like a dependency or build.

[lizgehret]

Hi all,

Thanks for sharing these reports - all of us on the QIIME 2 core team are currently investigating this, and have been able to reproduce this on our end as well. We are actively working on a fix for this, and will circle back here with details and a solution once that's available. Thanks for your patience!

[ebolyen (Evan Bolyen)]

Hey all,

Environment files are updated. Please download and reinstall like normal.
Things should be working again.

---

We identified the issue, it involved a few pieces.

The first piece was that bioconda did a reasonably routine "bulk update" of their packages, which included unifrac-binaries. This happened 3 days and 4 hours ago according to the site, which is consistent with the timeline.

The next issue, was that unifrac-binaries does not set hdf5 as a dependency in its host or run section of the recipe.

This means that while a particular version of HDF5 is pulled in during compilation of unifrac-binaries, that version is not referenced during installation. So conda allows any version of hdf5, hence:

ImportError: libhdf5_cpp.so.200: cannot open shared object file: No such file or directory

This particular libhdf5_cpp.so.200 shared-object is only available with hdf5=1.12 (the version that this new build was compiled against). However our environment is configured to use hdf5=1.10 which was the version that the previous build compiled was against.

To avoid these issues, conda provides a run_exports directive which can inflict a pinning on downstream dependencies if they set them as a host (or in some cases run) dependency. This would prevent conda from installing a new build if it knew that it required a different host dependency than what was available.

Fortunately, the hdf5 recipe also sets a run_exports which would have worked had hdf5 been included as a host dependency in unfrac-binaries.

We've applied a quick fix to pin unifrac-binaries at the last build. So this should be resolved for now. But ultimately, the unfrac-binaries recipe will need to be updated with additional dependencies to avoid this happening next time there's a recompile between releases.

(There's an additional issue of OS X builds not currently compiling which is why this situation didn't impact OS X as far as we can tell. There were no new builds to cause the HDF5 mismatch on OS X.)

[antgonza (Antonio Gonzalez Pena)]

Thank you! I can confirm that the build is now passing.

@ebolyen, trying to minimize effort, do you think it will be possible to issue a PR for the unifrac-binaries?

[wasade (Daniel McDonald)]

Recipe updated and waiting on merge, thanks @ebolyen!

[ebolyen (Evan Bolyen)]

Thanks for the update @wasade!