Core metrics analysis error - UnicodeEncodeError, 'latin-1' codec can't encode character '\u03b2'

Hi all, I attempted to do a core metrics analysis step and encountered the following error in my SLURM output:

I followed the path to the debug info that it pointed me to and got this:

I see there have been similar posts on the forum before, but I am not computer savvy enough to be able to translate their solutions to this particular error. I am also slightly confused because we have done core metrics analysis many times now using the same code essentially, and have never encountered this error yet. Can anyone point me in the right direction? Thank you in advance!

Best,
Carli

1 Like

Hello Carli,

As close as I can tell, this error has to do with special characters (not utf-8) in your metadata file. These are often sÿmbõs not commonly found in english.

You can use keemei to validate your metadata, and that should detect any unsuspected characters. https://keemei.qiime2.org/

Let me know what you find!
Colin

2 Likes

Hi Colin,

I validated the metadata file using Keemei and did not find errors or special symbols! In our sample ID column, we have the sample names listed as Sample_1 for example, using an underscore instead of a dash because this is how they are formatted in our manifest file, and this has never caused us issues before even though Keemei gives a warning on this. There were no other errors.

I tried converting our dates from using slashes (ie. 3/26/20) to dashes (ie. 3-26-20) to see if that would help, but it did not and I got the same error.

Best,
Carli

Did you look for hiden characters?
Try open the metadata file with VI or VIM and type :set invlist
That will show you all hiden charac. Maybe there is something.
Cheers

Hi Leo,

Thanks for your reply!! When I try doing that, I get this:

Screen Shot 2020-03-26 at 9.28.39 AM

Are the $ a problem? The symbol \u03b2 in the error code refers to a greek beta, which I definitely don’t see anywhere…

Thanks,
Carli

Hello Carli,

Thanks for validating with Keemei.

Well, I’m all out of ideas. Let’s see what the devs suggest!

Colin

1 Like

Small update: I tried to follow the instructions in this post, but got the same error

1 Like

Hey @colinbrislawn & @cjone228, I think we might be looking at the wrong things here:

Looking at the error traceback (thanks for sharing that, @cjone228!), it looks like the error is cropping up when QIIME 2 is attempting to save the citation data in the QZA’s provenance. That probably means the β is being introduced via the citations, not your input files.

Okay, so taking a closer look at the “meat” of the error message - it sounds like your local computational environment might be set to a latin locale. We can double check that by running the following command:

env

There are a handful of environment variables that we might need to change, but in general QIIME 2 expects you to be using a UTF-8 locale.

1 Like

Great! That post is actually right in line with where we need to get going towards. Follow up to my previous post with the output of the env command and we will go from there. Thanks!

Hi Matthew!

When I do env, I get the following for the language:
LC_ALL=en_US.utf8

(I followed the instructions from the post I linked to to get this, but before it was UTF8 anyway. I also added this to my .bashrc file)

I got the same error again when running the core metrics analysis - any ideas?

Thanks!!

Hey @cjone228, can you provide the full output from that env command? As I mentioned above:

Thanks!

Oops, sorry about that!! Here’s the full output:

(base) [[email protected]@bc-login03 ~]$ env

MARCC_COMPILER_MAJOR=5

LMOD_FAMILY_COMPILER_VERSION=5.5.0

MANPATH=/software/apps/mpi/openmpi/3.1/gcc/5.5/share/man:/software/apps/compilers/gcc/5.5.0/share/man:/software/apps/slurm/current/share/man:/software/lmod/lmod/share/man

XDG_SESSION_ID=4478

_ModuleTable003_=dWxlZmlsZXMvbXBpX2NvbXBpbGVyL2djYy81LjUvb3Blbm1waS8zLjEubHVhIixbImZ1bGxOYW1lIl09Im9wZW5tcGkvMy4xIixbImxvYWRPcmRlciJdPTQscHJvcFQ9e30sWyJzdGFja0RlcHRoIl09MCxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0ib3Blbm1waS8zLjEiLH0sc2x1cm09e1siZm4iXT0iL3NvZnR3YXJlL2xtb2QvbW9kdWxlZmlsZXMvYXBwcy9zbHVybS8uY3VycmVudC5sdWEiLFsiZnVsbE5hbWUiXT0ic2x1cm0vLmN1cnJlbnQiLFsibG9hZE9yZGVyIl09NSxwcm9wVD17fSxbInN0YWNrRGVwdGgiXT0wLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFtZSJdPSJzbHVybSIsfSx9LG1wYXRoQT17Ii9zb2Z0d2FyZS9sbW9kL21vZHVsZWZpbGVz

HOSTNAME=bc-login03

MPI_FFLAGS=-I/software/apps/mpi/openmpi/3.1/gcc/5.5/include

MPI_F90FLAGS=-I/software/apps/mpi/openmpi/3.1/gcc/5.5/include

MPICH_HOME=/software/apps/mpi/openmpi/3.1/gcc/5.5

__LMOD_REF_COUNT_MODULEPATH=/software/lmod/modulefiles/mpi_compiler/gcc/5.5:1;/software/lmod/modulefiles/compiler_and_base:1;/software/lmod/modulefiles/apps:1;/software/lmod/modulefiles/holding_apps:1;/software/lmod/modulefiles/compiled_apps/gcc/5.5:1;/software/lmod/modulefiles/mpi_compiled_apps/gcc/5.5/openmpi/3.1:1

TERM=xterm-256color

SHELL=/bin/bash

HISTSIZE=1000

TMPDIR=/home-1/[email protected]/scratch

MODULEPATH_ROOT=/software/lmod/modulefiles

LMOD_SYSTEM_DEFAULT_MODULES=MARCC/summer-2018

SSH_CLIENT=172.16.0.9 34964 22

CONDA_SHLVL=1

__LMOD_STACK_CXX=Zysr

PERL5LIB=/software/centos7/usr/share/autoconf:/software/centos7/usr/share/automake-1.13:/software/centos7/usr/lib64/perl5/vendor_perl:/software/centos7/usr/share/perl5/vendor_perl:/software/centos7/usr/lib64/perl5:/software/centos7/usr/share/perl5

LIBRARY_PATH=/software/apps/compilers/gcc/5.5.0/lib64:/software/apps/compilers/gcc/5.5.0/lib:/software/centos7/lib64:/software/centos7/lib:/software/centos7/usr/lib64:/software/centos7/usr/lib:/software/apps/slurm/current/lib/slurm:/software/apps/slurm/current/lib

LMOD_PACKAGE_PATH=/software/lmod/site

CONDA_PROMPT_MODIFIER=(base) 

LMOD_PKG=/software/lmod/lmod

LMOD_VERSION=7.8.1

SSH_TTY=/dev/pts/192

LC_ALL=en_US.utf8

__LMOD_STACK_FC=Z2ZvcnRyYW4=

__LMOD_REF_COUNT_LOADEDMODULES=centos7/current:1;MARCC/summer-2018:1;gcc/5.5.0:1;openmpi/3.1:1;slurm/.current:1

__LMOD_STACK_CC=Z2Nj

[email protected]

LMOD_sys=Linux

LD_LIBRARY_PATH=/software/apps/mpi/openmpi/3.1/gcc/5.5/lib:/software/apps/compilers/gcc/5.5.0/lib64:/software/apps/compilers/gcc/5.5.0/lib:/usr/lib:/usr/lib64:/software/centos7/lib64:/software/centos7/lib:/software/centos7/usr/lib64:/software/centos7/usr/lib:/software/apps/slurm/current/lib/slurm:/software/apps/slurm/current/lib::

CONDA_EXE=/home-1/[email protected]/miniconda3/bin/conda

MARCC_COMPILER_MINOR=5

VALGRIND_LIB=/software/centos7/lib64/valgrind

__LMOD_STACK_F77=Z2ZvcnRyYW4=

_ModuleTable004_=L21waV9jb21waWxlci9nY2MvNS41IiwiL3NvZnR3YXJlL2xtb2QvbW9kdWxlZmlsZXMvY29tcGlsZXJfYW5kX2Jhc2UiLCIvc29mdHdhcmUvbG1vZC9tb2R1bGVmaWxlcy9hcHBzIiwiL3NvZnR3YXJlL2xtb2QvbW9kdWxlZmlsZXMvaG9sZGluZ19hcHBzIiwiL3NvZnR3YXJlL2xtb2QvbW9kdWxlZmlsZXMvY29tcGlsZWRfYXBwcy9nY2MvNS41IiwiL3NvZnR3YXJlL2xtb2QvbW9kdWxlZmlsZXMvbXBpX2NvbXBpbGVkX2FwcHMvZ2NjLzUuNS9vcGVubXBpLzMuMSIsfSxbInN5c3RlbUJhc2VNUEFUSCJdPSIvc29mdHdhcmUvbG1vZC9tb2R1bGVmaWxlcy9jb21waWxlcl9hbmRfYmFzZTovc29mdHdhcmUvbG1vZC9tb2R1bGVmaWxlcy9hcHBzOi9zb2Z0d2FyZS9sbW9kL21vZHVs

CPATH=/software/apps/compilers/gcc/5.5.0/include:/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include:/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include-fixed:/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/plugin/include:/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/install-tools/include:/software/centos7/usr/local/include/:/software/centos7/usr/include/

__LMOD_REF_COUNT__LMFILES_=/software/lmod/modulefiles/compiler_and_base/centos7/current.lua:1;/software/lmod/modulefiles/compiler_and_base/MARCC/summer-2018.lua:1;/software/lmod/modulefiles/compiler_and_base/gcc/5.5.0.lua:1;/software/lmod/modulefiles/mpi_compiler/gcc/5.5/openmpi/3.1.lua:1;/software/lmod/modulefiles/apps/slurm/.current.lua:1

_CE_CONDA=

MPI_CXXFLAGS=-I/software/apps/mpi/openmpi/3.1/gcc/5.5/include

MPI_CFLAGS=-I/software/apps/mpi/openmpi/3.1/gcc/5.5/include

LMOD_FAMILY_MPI_VERSION=3.1

FONTCONFIG_PATH=/software/centos7/etc/fonts/

_ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXtbImNvbXBpbGVyIl09ImdjYyIsWyJtcGkiXT0ib3Blbm1waSIsfSxtVD17TUFSQ0M9e1siZm4iXT0iL3NvZnR3YXJlL2xtb2QvbW9kdWxlZmlsZXMvY29tcGlsZXJfYW5kX2Jhc2UvTUFSQ0Mvc3VtbWVyLTIwMTgubHVhIixbImZ1bGxOYW1lIl09Ik1BUkNDL3N1bW1lci0yMDE4IixbImxvYWRPcmRlciJdPTIscHJvcFQ9e30sWyJzdGFja0RlcHRoIl09MCxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0iTUFSQ0Mvc3VtbWVyLTIwMTgiLH0sY2VudG9zNz17WyJmbiJdPSIvc29mdHdhcmUvbG1vZC9t

MAIL=/var/spool/mail/[email protected]

PATH=/home-1/[email protected]/miniconda3/envs/qiime2-2020.2/bin:/home-1/[email protected]/miniconda3/bin:/home-1/[email protected]/miniconda3/condabin:/software/apps/mpi/openmpi/3.1/gcc/5.5/bin:/software/apps/compilers/gcc/5.5.0/bin:/software/apps/marcc/bin:/software/centos7/bin:/software/centos7/sbin:/software/apps/slurm/current/sbin:/software/apps/slurm/current/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home-1/[email protected]/.local/bin:/home-1/[email protected]/bin

CONDA_PREFIX=/home-1/[email protected]/miniconda3

PWD=/home-1/[email protected]

_LMFILES_=/software/lmod/modulefiles/compiler_and_base/centos7/current.lua:/software/lmod/modulefiles/compiler_and_base/MARCC/summer-2018.lua:/software/lmod/modulefiles/compiler_and_base/gcc/5.5.0.lua:/software/lmod/modulefiles/mpi_compiler/gcc/5.5/openmpi/3.1.lua:/software/lmod/modulefiles/apps/slurm/.current.lua

MARCC_COMPILER=gcc

LANG=en_US.UTF-8

MODULEPATH=/software/lmod/modulefiles/mpi_compiler/gcc/5.5:/software/lmod/modulefiles/compiler_and_base:/software/lmod/modulefiles/apps:/software/lmod/modulefiles/holding_apps:/software/lmod/modulefiles/compiled_apps/gcc/5.5:/software/lmod/modulefiles/mpi_compiled_apps/gcc/5.5/openmpi/3.1

LOADEDMODULES=centos7/current:MARCC/summer-2018:gcc/5.5.0:openmpi/3.1:slurm/.current

_ModuleTable_Sz_=5

F77=gfortran

_ModuleTable005_=ZWZpbGVzL2hvbGRpbmdfYXBwcyIsfQ==

LMOD_CMD=/software/lmod/lmod/libexec/lmod

LMOD_AVAIL_STYLE=en_grouped

_CE_M=

CXX=g++

HISTCONTROL=ignoredups

SHLVL=1

HOME=/home-1/[email protected]

__LMOD_REF_COUNT_PERL5LIB=/software/centos7/usr/share/autoconf:1;/software/centos7/usr/share/automake-1.13:1;/software/centos7/usr/lib64/perl5/vendor_perl:1;/software/centos7/usr/share/perl5/vendor_perl:1;/software/centos7/usr/lib64/perl5:1;/software/centos7/usr/share/perl5:1

__LMOD_REF_COUNT_PATH=/software/apps/mpi/openmpi/3.1/gcc/5.5/bin:1;/software/apps/compilers/gcc/5.5.0/bin:1;/software/apps/marcc/bin:1;/software/centos7/bin:1;/software/centos7/sbin:1;/software/apps/slurm/current/sbin:2;/software/apps/slurm/current/bin:2;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/opt/ibutils/bin:1

_ModuleTable002_=b2R1bGVmaWxlcy9jb21waWxlcl9hbmRfYmFzZS9jZW50b3M3L2N1cnJlbnQubHVhIixbImZ1bGxOYW1lIl09ImNlbnRvczcvY3VycmVudCIsWyJsb2FkT3JkZXIiXT0xLHByb3BUPXt9LFsic3RhY2tEZXB0aCJdPTEsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09ImNlbnRvczciLH0sZ2NjPXtbImZuIl09Ii9zb2Z0d2FyZS9sbW9kL21vZHVsZWZpbGVzL2NvbXBpbGVyX2FuZF9iYXNlL2djYy81LjUuMC5sdWEiLFsiZnVsbE5hbWUiXT0iZ2NjLzUuNS4wIixbImxvYWRPcmRlciJdPTMscHJvcFQ9e30sWyJzdGFja0RlcHRoIl09MCxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0iZ2NjIix9LG9wZW5tcGk9e1siZm4iXT0iL3NvZnR3YXJlL2xtb2QvbW9k

FC=gfortran

BASH_ENV=/software/lmod/lmod/init/bash

CONDA_PYTHON_EXE=/home-1/[email protected]/miniconda3/bin/python

[email protected]

OMPI_TMPDIR=/tmp

SSH_CONNECTION=172.16.0.9 34964 172.16.5.5 22

__LMOD_REF_COUNT_LIBRARY_PATH=/software/apps/compilers/gcc/5.5.0/lib64:1;/software/apps/compilers/gcc/5.5.0/lib:1;/software/centos7/lib64:1;/software/centos7/lib:1;/software/centos7/usr/lib64:1;/software/centos7/usr/lib:1;/software/apps/slurm/current/lib/slurm:1;/software/apps/slurm/current/lib:1

MODULESHOME=/software/lmod/lmod

CONDA_DEFAULT_ENV=base

PKG_CONFIG_PATH=/software/centos7/usr/share/pkgconfig:/software/centos7/usr/lib64/pkgconfig

__LMOD_REF_COUNT_LD_LIBRARY_PATH=/software/apps/mpi/openmpi/3.1/gcc/5.5/lib:1;/software/apps/compilers/gcc/5.5.0/lib64:1;/software/apps/compilers/gcc/5.5.0/lib:1;/usr/lib:1;/usr/lib64:1;/software/centos7/lib64:1;/software/centos7/lib:1;/software/centos7/usr/lib64:1;/software/centos7/usr/lib:1;/software/apps/slurm/current/lib/slurm:2;/software/apps/slurm/current/lib:2

LMOD_SETTARG_FULL_SUPPORT=no

LESSOPEN=||/usr/bin/lesspipe.sh %s

LMOD_FAMILY_COMPILER=gcc

__LMOD_REF_COUNT_PKG_CONFIG_PATH=/software/centos7/usr/share/pkgconfig:1;/software/centos7/usr/lib64/pkgconfig:1

CC=gcc

XDG_RUNTIME_DIR=/run/user/4318

__LMOD_REF_COUNT_MANPATH=/software/apps/mpi/openmpi/3.1/gcc/5.5/share/man:1;/software/apps/compilers/gcc/5.5.0/share/man:1;/software/apps/slurm/current/share/man:1;/software/lmod/lmod/share/man:1

LMOD_DIR=/software/lmod/lmod/libexec

LMOD_FAMILY_MPI=openmpi

BASH_FUNC_module()=() { eval $($LMOD_CMD bash "[email protected]") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)

}

BASH_FUNC_ml()=() { eval $($LMOD_DIR/ml_cmd "[email protected]")

}

_=/software/centos7/bin/env

OLDPWD=/home-1/[email protected]/work/IT_pilot
1 Like

Everything is fine. The $ there means “end of line” and the “^I” means a TAB character. So it’s ok. Let’s see what Thermokarst has on the output you’ve posted.
Cheers

1 Like

Thanks @cjone228!

See this env var:

This is one of those related ones I was looking for. I suspect that all you need to do is update your LC_ALL to use the same formatting as the value above (your value is formatted differently - the locale databases vary a bit from system to system, and usually are based on an “exact match” of the value):

LC_ALL=en_US.UTF-8

If that doesn’t fix it, please run the following and paste all of the results here:

locale

Thanks!
:qiime2:

Hi Matt! I changed LC_ALL back to en_US.UTF-8 (this is what it was originally), and tried running my SLURM job again. Unfortunately it failed again, but I’m not surprised. :upside_down_face:

When I run locale, I get the following:

(base) [[email protected]@bc-login02 IT_pilot]$ locale

LANG=en_US.UTF-8

LC_CTYPE="en_US.UTF-8"

LC_NUMERIC="en_US.UTF-8"

LC_TIME="en_US.UTF-8"

LC_COLLATE="en_US.UTF-8"

LC_MONETARY="en_US.UTF-8"

LC_MESSAGES="en_US.UTF-8"

LC_PAPER="en_US.UTF-8"

LC_NAME="en_US.UTF-8"

LC_ADDRESS="en_US.UTF-8"

LC_TELEPHONE="en_US.UTF-8"

LC_MEASUREMENT="en_US.UTF-8"

LC_IDENTIFICATION="en_US.UTF-8"

LC_ALL=en_US.UTF-8

Ohhh, I wonder if the locale isn’t consistent within the slurm cluster. You might need to set these env vars inside your submission script. Might also be worth a chat with your sysadmin to see what they think.

1 Like