fragment-insertion ValueError: unknown locale: UTF-8

jwdebelius · May 22, 2019, 3:32pm

I’m using a QIIME2-2019.4 enviroment, although I saw this in QIIME2-2019.1. It only shows up on my swedish machines (mac and local server). Easily fixible (if I remember), but vaguely obnoxious.

I’m running the following

qiime fragment-insertion sepp \
 --i-representative-sequences ./rep_set.qza \
 --o-tree ./tree.qza \
 --o-placements ./tree_placements.qza \
 --p-threads 3 \
 --verbose \
 --p-debug

And, I get the following as output:

========= Execution of SEPP failed with exit code 1 =================
temporary working directories are NOT deleted for further inspection:
  $tmp = /var/folders/7p/rzmyxjjx6n5fn72_78dnsgmr0000gp/T/sepp-tmp-XXXXX.fe1et0L0
  $tmpssd = /var/folders/7p/rzmyxjjx6n5fn72_78dnsgmr0000gp/T/sepp-tempssd-XXXX.1UY1yPlJ
--------- Content of STDOUT -----------------------------------------
--------- Content of STDERR -----------------------------------------
Traceback (most recent call last):
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/share/fragment-insertion//sepp/run_sepp.py", line 3, in <module>
    from sepp.exhaustive import ExhaustiveAlgorithm
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/share/fragment-insertion/sepp/sepp/exhaustive.py", line 6, in <module>
    from sepp.algorithm import AbstractAlgorithm
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/share/fragment-insertion/sepp/sepp/algorithm.py", line 11, in <module>
    from sepp.tree import PhylogeneticTree
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/share/fragment-insertion/sepp/sepp/tree.py", line 22, in <module>
    from dendropy import Tree, Taxon, treecalc
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/share/fragment-insertion/sepp/dendropy/__init__.py", line 24, in <module>
    from dendropy.dataio.nexusprocessing import get_rooting_argument
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/share/fragment-insertion/sepp/dendropy/dataio/__init__.py", line 20, in <module>
    from dendropy.dataio import newickreader
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/share/fragment-insertion/sepp/dendropy/dataio/newickreader.py", line 27, in <module>
    from dendropy.utility.textprocessing import StringIO
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/share/fragment-insertion/sepp/dendropy/utility/textprocessing.py", line 40, in <module>
    ENCODING = locale.getdefaultlocale()[1]
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/lib/python3.6/locale.py", line 562, in getdefaultlocale
    return _parse_localename(localename)
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/lib/python3.6/locale.py", line 490, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: UTF-8
=====================================================================
Traceback (most recent call last):
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/lib/python3.6/site-packages/q2cli/commands.py", line 311, in __call__
    results = action(**arguments)
  File "</Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/lib/python3.6/site-packages/decorator.py:decorator-gen-299>", line 2, in sepp
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/lib/python3.6/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
    output_types, provenance)
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/lib/python3.6/site-packages/qiime2/sdk/action.py", line 365, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/lib/python3.6/site-packages/q2_fragment_insertion/_insertion.py", line 179, in sepp
    reference_alignment, reference_phylogeny, debug)
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/lib/python3.6/site-packages/q2_fragment_insertion/_insertion.py", line 137, in _run
    subprocess.run(cmd, check=True, cwd=cwd)
  File "/Users/justinedebelius/miniconda3/envs/qiime2-2019.4-docs/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run-sepp.sh', '/Users/justinedebelius/tmp/qiime2-archive-0kvs3lwu/2ce5e7fa-c767-48b5-b6ed-5c9d29a67d5c/data/dna-sequences.fasta', 'q2-fragment-insertion', '-x', '3', '-A', '1000', '-P', '5000', '-b', '1']' returned non-zero exit status 1.

Plugin error from fragment-insertion:

  Command '['run-sepp.sh', '/Users/justinedebelius/tmp/qiime2-archive-0kvs3lwu/2ce5e7fa-c767-48b5-b6ed-5c9d29a67d5c/data/dna-sequences.fasta', 'q2-fragment-insertion', '-x', '3', '-A', '1000', '-P', '5000', '-b', '1']' returned non-zero exit status 1.

See above for debug info.

My normal locale settings are

LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

The added fun part is that the locale has to be set with outside a jupyter notebook. But, once I do that, it runs no problem

Nicholas_Bokulich · May 24, 2019, 12:44pm

pinging @Stefan

thermokarst · July 8, 2019, 2:32pm

pinging @Stefan

Stefan · July 9, 2019, 5:37pm

Hi @jwdebelius,

having no Swedish machine, it is not that easy for me to debug. However, if I look env | grep -i LANG at my LANG variable it returns LANG=en_US.UTF-8 on my laptop or LANG=de_DE.UTF-8 on a German server. Thus, I would think you should set your default to LANG=sv_SE.UTF-8, compare https://github.com/zeit/hyper/issues/441

Can you please try what happens if you make this change? I would be surprised if the error you found really stems from the third party library as seen from the stack-trace or is “just” a wrong setting.

Thanks,
Stefan

jwdebelius · July 10, 2019, 7:46am

Hi @Stefan,

Thanks for the follow up. Ive set my server to specify LANG=en_US-UTF.8, but my LANG is empty by default (see above). q2-fragment-insertion is the only plug in that throws a fit. Its fixable (short term fix) but still obnoxious and it would be nice to understand why not having a LANG throws an error?

Best,
Justine

Stefan · July 10, 2019, 8:52am

I tried to set all variables you mentioned above to your values. When I do export LC_CTYPE="UTF-8" I get the following error bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8), but it works if I either set it empty export LC_CTYPE= to to export LC_CTYPE=en_US-UTF.8.

Furthermore, the qiime2 framework throws an error if I first set variable export LANG=en_US-UTF.8 and then activate the environment conda activate qiime2-2019.1:

  File "/home/sjanssen/miniconda3/envs/qiime2-2019.1/bin/qiime", line 11, in <module>
    sys.exit(qiime())
  File "/home/sjanssen/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/sjanssen/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/click/core.py", line 696, in main
    _verify_python3_env()
  File "/home/sjanssen/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/click/_unicodefun.py", line 124, in _verify_python3_env
    ' mitigation steps.' + extra
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/en/7.x/python3/ for mitigation steps.

This system supports the C.UTF-8 locale which is recommended.
You might be able to resolve your issue by exporting the
following environment variables:

    export LC_ALL=C.UTF-8
    export LANG=C.UTF-8

Thus, I still tend to think it is a misconfiguration in your system rather than a general error of qiime2. If it turns out I am wrong, we need to contact the dendropy developers to let them know, since it is this library that throws the error in the end.

jwdebelius · July 10, 2019, 10:22am

Okay, thanks for looking into it!