Heads up for conda 4.8 and OSX 14.6

jwdebelius · June 26, 2020, 8:19pm

Hi all --

I updated conda (incidentally because I was trying to figure out why travis tests for a plugin Im working on were failing...) and tried to install the qiime2-dev enviroment (basically v2020.06).

So, basically,

conda update conda
 wget -q https://raw.githubusercontent.com/qiime2/environment-files/master/latest/staging/qiime2-latest-py36-osx-conda.yml
conda env create -q -n test-env --file qiime2-latest-py36-osx-conda.yml

Which lead to a lovely error:

ImportError: dlopen(/Users/jusdeb/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libffi.6.dylib
  Referenced from: /Users/jusdeb/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so
  Reason: image not found

There's a similar github issue in the repo:

github.com/conda/conda

After running "upgrade conda", no `conda` commands work any more due to libffi

opened 12:39AM - 29 Apr 20 UTC

closed 04:01AM - 01 Mar 23 UTC

cubranic

type::bug source::community type::packaging stale stale::closed locked tag::update

## Current Behavior After conda warned me that "A newer version of conda exists…", I ran the exact recommended command: ``` conda update -n base -c defaults conda ``` This upgraded conda from 4.8.1 to 4.8.3. After running the command, it broke conda completely, and most `conda` commands (e.g., `create`, `install`, `upgrade`) print the a long traceback ending with "ImportError: dlopen(/usr/local/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libffi.6.dylib Referenced from: /usr/local/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so Reason: image not found" ### Steps to Reproduce ``` $ conda update conda Traceback (most recent call last): File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1079, in __call__ return func(*args, **kwargs) File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/cli/main.py", line 84, in _main exit_code = do_call(args, p) File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 80, in do_call module = import_module(relative_mod, __name__.rsplit('.', 1)[0]) File "/usr/local/miniconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/cli/main_update.py", line 8, in <module> from .install import install File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/cli/install.py", line 19, in <module> from ..core.index import calculate_channel_urls, get_index File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/core/index.py", line 10, in <module> from .package_cache_data import PackageCacheData File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/core/package_cache_data.py", line 15, in <module> from .path_actions import CacheUrlAction, ExtractPackageAction File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/core/path_actions.py", line 30, in <module> from ..gateways.connection.download import download File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/gateways/connection/download.py", line 13, in <module> import ctypes File "/usr/local/miniconda3/lib/python3.7/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ImportError: dlopen(/usr/local/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libffi.6.dylib Referenced from: /usr/local/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so Reason: image not found During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/miniconda3/bin/conda", line 13, in <module> sys.exit(main()) File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/cli/main.py", line 150, in main return conda_exception_handler(_main, *args, **kwargs) File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1371, in conda_exception_handler return_value = exception_handler(func, *args, **kwargs) File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1082, in __call__ return self.handle_exception(exc_val, exc_tb) File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1126, in handle_exception return self.handle_unexpected_exception(exc_val, exc_tb) File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1137, in handle_unexpected_exception self.print_unexpected_error_report(error_report) File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1207, in print_unexpected_error_report from .cli.main_info import get_env_vars_str, get_main_info_str File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/cli/main_info.py", line 19, in <module> from ..core.index import _supplement_index_with_system File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/core/index.py", line 10, in <module> from .package_cache_data import PackageCacheData File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/core/package_cache_data.py", line 15, in <module> from .path_actions import CacheUrlAction, ExtractPackageAction File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/core/path_actions.py", line 30, in <module> from ..gateways.connection.download import download File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/gateways/connection/download.py", line 13, in <module> import ctypes File "/usr/local/miniconda3/lib/python3.7/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ImportError: dlopen(/usr/local/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libffi.6.dylib Referenced from: /usr/local/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so Reason: image not found ``` ## Expected Behavior I should be able to use conda normally. ## Environment Information <details open><summary><code>`conda info`</code></summary> ``` # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<< Traceback (most recent call last): File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1079, in __call__ return func(*args, **kwargs) File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/cli/main.py", line 84, in _main exit_code = do_call(args, p) File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 80, in do_call module = import_module(relative_mod, __name__.rsplit('.', 1)[0]) File "/usr/local/miniconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/cli/main_info.py", line 19, in <module> from ..core.index import _supplement_index_with_system File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/core/index.py", line 10, in <module> from .package_cache_data import PackageCacheData File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/core/package_cache_data.py", line 15, in <module> from .path_actions import CacheUrlAction, ExtractPackageAction File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/core/path_actions.py", line 30, in <module> from ..gateways.connection.download import download File "/usr/local/miniconda3/lib/python3.7/site-packages/conda/gateways/connection/download.py", line 13, in <module> import ctypes File "/usr/local/miniconda3/lib/python3.7/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ImportError: dlopen(/usr/local/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libffi.6.dylib Referenced from: /usr/local/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so Reason: image not found `$ /usr/local/miniconda3/bin/conda info` ``` </details> <details open><summary><code>`conda config --show-sources`</code></summary> ``` # the output is empty ``` </details> <details><summary><code>`conda list --show-channel-urls`</code></summary> ``` # packages in environment at /usr/local/miniconda3: # # Name Version Build Channel appdirs 1.4.3 pypi_0 pypi arrow 0.15.5 pypi_0 pypi asn1crypto 1.3.0 py37_0 defaults attrs 19.1.0 pypi_0 pypi black 19.3b0 pypi_0 pypi ca-certificates 2020.1.1 0 defaults certifi 2020.4.5.1 py37_0 defaults cffi 1.13.2 py37hb5b8e2f_0 defaults chardet 3.0.4 py37_1003 defaults click 7.0 pypi_0 pypi conda 4.8.3 py37_0 defaults conda-package-handling 1.6.0 py37h1de35cc_0 defaults cryptography 2.8 py37ha12b0ac_0 defaults ideal-broccoli 0.1 dev_0 <develop> idna 2.9 py_1 defaults importlib-metadata 1.6.0 pypi_0 pypi libcxx 4.0.1 hcfea43d_1 defaults libcxxabi 4.0.1 hcfea43d_1 defaults libedit 3.1.20181209 hb402a30_0 defaults libffi 3.3 h0a44026_1 defaults more-itertools 8.2.0 pypi_0 pypi ncurses 6.2 h0a44026_0 defaults openssl 1.1.1g h1de35cc_0 defaults packaging 20.3 pypi_0 pypi pip 20.0.2 py37_1 defaults pluggy 0.13.1 pypi_0 pypi py 1.8.1 pypi_0 pypi pycosat 0.6.3 py37h1de35cc_0 defaults pycparser 2.20 py_0 defaults pyopenssl 19.1.0 py37_0 defaults pyparsing 2.4.7 pypi_0 pypi pysocks 1.7.1 py37_0 defaults pytest 5.4.1 pypi_0 pypi python 3.7.2 haf84260_0 defaults python-dateutil 2.8.1 pypi_0 pypi readline 7.0 h1de35cc_5 defaults requests 2.23.0 py37_0 defaults ruamel_yaml 0.15.87 py37h1de35cc_0 defaults setuptools 46.1.3 py37_0 defaults six 1.14.0 py37_0 defaults sqlite 3.31.1 h5c1f38d_1 defaults tk 8.6.8 ha441bb4_0 defaults toml 0.10.0 pypi_0 pypi tqdm 4.45.0 py_0 defaults urllib3 1.25.8 py37_0 defaults wcwidth 0.1.9 pypi_0 pypi wheel 0.34.2 py37_0 defaults xz 5.2.5 h1de35cc_0 defaults yaml 0.1.7 hc338f04_2 defaults zipp 3.1.0 pypi_0 pypi zlib 1.2.11 h1de35cc_3 defaults ``` </details>

I haven't been able to get the upgrade to work locally... but as we're releasing 2020.6 very soon, its possibly good to be aware that (a) this may be a problem and (b) how to (possibly) troubleshoot it. I'm trying to figure out how to downgrade my conda, but at this point, people may want to be cautious about upgrades and we may want to be cautious about recommending upgrades for people in weird "No you can update your OS" limbo.

Best,
Justine

thermokarst · June 26, 2020, 9:10pm

Thank you thank you thank you @jwdebelius!

I will start looking into this, too. You're a lifesaver!

thermokarst · June 26, 2020, 9:17pm

I haven't had time to read through that gigantic GH issue you linked to above, but, I am wondering if as a potential solution, can you (re)move the current installation, and drop a new installation in its place? That only works if your envs are in another location. For example, I install to /opt/miniconda3, and then all of my envs are in ~/.conda/envs, and the packages in ~/.conda/pkgs.

The other benefit to installing in /opt is any changes to the base env require sudo, which usually stops me from doing something on accident.

Anyway, I'll keep looking.

jwdebelius · June 26, 2020, 9:19pm

I'll look into that! Thanks for the suggestion.

thermokarst · June 26, 2020, 9:22pm

Credit goes to @ebolyen!