UnicodeDecodeError: 'ascii' codec can't decode byte 0xb0 in position 37: ordinal not in range(128)

Technical Support
1

Hello Everyone! I am a QIIME1 user attempting to transition to QIIME 2. Normally I use a virtual box to run QIIME commands, however, due to an old computer and a multitude of issues I am currently using conda to try streamline the process. While I have gotten QIIME to recognize my files (which is further than I was before) I have gotten a Decode Error. I have looked through the other posts and threads here on the forum and I tried to implement their fixes, but none of those have worked for me. Any help or insight would be greatly appreciated!

(qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab$ qiime tools import
–type ‘SampleData[PairedEndSequencesWithQuality]’
–input-path /Volumes/WILBUR\ II/WOMBATS/MASTERS/Control_Manifest.csv
–output-path Control-demux.qza
–input-format PairedEndFastqManifestPhred33

Traceback (most recent call last):
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/q2cli/tools.py”, line 140, in import_data
view_type=input_format)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/result.py”, line 240, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/result.py”, line 265, in _from_view
result = transformation(view)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/transform.py”, line 73, in transformation
other.validate(new_view)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/transform.py”, line 143, in validate
view.validate(‘min’)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py”, line 171, in validate
getattr(self, field)._validate_members(collected_paths, level)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py”, line 101, in _validate_members
self.format(path, mode=‘r’).validate(level)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/plugin/model/file_format.py”, line 24, in validate
self.validate(level)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_types/per_sample_sequences/_format.py”, line 204, in validate
self._check_n_records(record_count_map[level])
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_types/per_sample_sequences/_format.py”, line 164, in check_n_records
for i, record in file:
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/encodings/ascii.py”, line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xb0 in position 37: ordinal not in range(128)

An unexpected error has occurred:

‘ascii’ codec can’t decode byte 0xb0 in position 37: ordinal not in range(128)

See above for debug info.

(qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab$ echo LC_en_US.utf8 .utf8 (qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab echo LANG en_US.UTF-8 (qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab en_US.UTF-8
-bash: en_US.UTF-8: command not found
(qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab$ export LC_ALL=en_US.utf8
(qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab$ file C3-0A_R1.fastq -bash: : command not found

Traceback (most recent call last):
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/bin/qiime”, line 11, in
sys.exit(qiime())
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/click/core.py”, line 764, in call
return self.main(*args, **kwargs)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/click/core.py”, line 696, in main
_verify_python3_env()
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/click/_unicodefun.py”, line 124, in _verify_python3_env
’ mitigation steps.’ + extra
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/en/7.x/python3/ for mitigation steps.

This system lists a couple of UTF-8 supporting locales that
you can pick from. The following suitable locales were
discovered: af_ZA.UTF-8, am_ET.UTF-8, be_BY.UTF-8, bg_BG.UTF-8, ca_ES.UTF-8, cs_CZ.UTF-8, da_DK.UTF-8, de_AT.UTF-8, de_CH.UTF-8, de_DE.UTF-8, el_GR.UTF-8, en_AU.UTF-8, en_CA.UTF-8, en_GB.UTF-8, en_IE.UTF-8, en_NZ.UTF-8, en_US.UTF-8, es_ES.UTF-8, et_EE.UTF-8, eu_ES.UTF-8, fi_FI.UTF-8, fr_BE.UTF-8, fr_CA.UTF-8, fr_CH.UTF-8, fr_FR.UTF-8, he_IL.UTF-8, hr_HR.UTF-8, hu_HU.UTF-8, hy_AM.UTF-8, is_IS.UTF-8, it_CH.UTF-8, it_IT.UTF-8, ja_JP.UTF-8, kk_KZ.UTF-8, ko_KR.UTF-8, lt_LT.UTF-8, nl_BE.UTF-8, nl_NL.UTF-8, no_NO.UTF-8, pl_PL.UTF-8, pt_BR.UTF-8, pt_PT.UTF-8, ro_RO.UTF-8, ru_RU.UTF-8, sk_SK.UTF-8, sl_SI.UTF-8, sr_YU.UTF-8, sv_SE.UTF-8, tr_TR.UTF-8, uk_UA.UTF-8, zh_CN.UTF-8, zh_HK.UTF-8, zh_TW.UTF-8

Click discovered that you exported a UTF-8 locale
but the locale system could not pick up from it because
it does not exist. The exported locale is “en_US.utf8” but it
is not supported

Thanks in advance.

2

Hi @gpgarcia,

Welcome to the forum and :qiime2:!

You want to explore your language as en_US.UTF-8; you're currently using en_US.utf8.

I set this in my bash_profile typically because it can be an issue for me on some of my systems.

Best,
Justine

3

Hi Justine,

I don’t really know what that means. I have limited understanding about the command line and associated things so if you could explain it in painfully simple terms I would greatly appreciate it. I am going to try what you suggested and thank you for getting back to me so quickly.

I tried what you suggested but I think I have a lapse in my understanding of the syntax. I tried to use the code from this post but it is still not working.

An unexpected error has occurred:

’ascii’ codec can’t decode byte 0xb0 in position 37: ordinal not in range(128)

See above for debug info.

So then I tried to re-run the fix:
(qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab$ echo $LC_ALL

(qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab$ echo $LANG

en_US.UTF-8

(qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab$ export LC_ALL=en_US.UTF-8

And then I tried to re-import my data but I got the same error as before.

Traceback (most recent call last):
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/q2cli/tools.py”, line 140, in import_data
view_type=input_format)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/result.py”, line 240, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/result.py”, line 265, in _from_view
result = transformation(view)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/transform.py”, line 73, in transformation
other.validate(new_view)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/transform.py”, line 143, in validate
view.validate(‘min’)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py”, line 171, in validate
getattr(self, field)._validate_members(collected_paths, level)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py”, line 101, in _validate_members
self.format(path, mode=‘r’).validate(level)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/plugin/model/file_format.py”, line 24, in validate
self.validate(level)
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_types/per_sample_sequences/_format.py”, line 204, in validate
self._check_n_records(record_count_map[level])
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/site-packages/q2_types/per_sample_sequences/_format.py”, line 164, in check_n_records
for i, record in file:
File “/Users/BiddleLab/anaconda3/envs/qiime2-2018.11/lib/python3.5/encodings/ascii.py”, line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xb0 in position 37: ordinal not in range(128)

An unexpected error has occurred:

‘ascii’ codec can’t decode byte 0xb0 in position 37: ordinal not in range(128)

See above for debug info.

4

Hi @gpgarcia,

Could you try running

export LC_ALL=en_US.utf-8
locale

Your .bash_profile is a file (which usually lives at ~/.bash_profile that controls the way your terminal initializes. You cna change things like the display label, the language, etc by changing it. The .bash_profile is a beast unto itself and almost everything Ive done to mine has come through StackOverflow, but, if you add

#Changes the locale
export LANG=en_US.utf-8
export LC_ALL=en_US.utf-8

to your .bash_profile file, then it will set your locale automagically for you.

I have a second thought which might be around the way your data is encoded. Can you double check and see if, when you save your manifest file (which Im assuming you made in Excel?), you saved it as UTF-8 or latin1?

I found some instructions for saving as UTF-8 here, but I occasionally run into those encoding issues.
A couple general key things there are to make sure you don’t have letters that aren’t commonly english (ö,ø, ñ, å, etc; yes this is stupid and anglocentric but most of the early computer programing was done off a US keyboard.)

Best,
Justine

1 Like
5

Hi Justine!

I just ran the code that you posted and now I have gotten this output

(qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab$ export LC_ALL=en_US.utf-8
(qiime2-2018.11) CANR-Biddle-Lab-iMac:~ BiddleLab$ locale

LANG=“en_US.UTF-8”

LC_COLLATE=“en_US.utf-8”

LC_CTYPE=“en_US.utf-8”

LC_MESSAGES=“en_US.utf-8”

LC_MONETARY=“en_US.utf-8”

LC_NUMERIC=“en_US.utf-8”

LC_TIME=“en_US.utf-8”

LC_ALL=“en_US.utf-8”

And my manifest file I actually made in Numbers and exported it as a csv. I checked the file and it was encoded using UTF-8. (I originally tried to do it in Excel but that came with a multitude of other issues).

Thank You for all of your help!
Best,
Gisselle

6

That looks good!

And, if your locale is right and your manifest is UTF-8, do you still get the same error when you re-run the command?

(Sorry, its a slow process, but we’re working through).

The next step, if it’s not your manifest or enviroment, will be to check your fastq.gz files, and take a couple to see if you can figure out which one is failing. (It may be a gzip issue, for example). I usually test in batches by adding a # infront of a sample I dont want, so like, my manifest might be

sample-id   forward-read
sample.1    $PWD/read1.fastq.gz
#sample.2  $PWD/read2.fastq.gz
#sample.3  $PWD/read3/fastq.gz

That way, I can adjust the samples that I import. (I typically do it programatically, but depending on your sample number, you could try manually.)

If your error is realted to the sequence files, you can run running the gunzip command on them. But, if that doesn’t work, you may need to go back tot he source because there’s likely a compression issue.

Best,
Justine

7

So I try to re-run the command and I did get the same error. Do you think that the issue may be related to the fact that my files are .fastq and not .fastq.gz? The fastq files that I am using came from a data set that was originally run through QIIME 1.

Best,
Gisselle

8

If you open one with a text editor (textedit on a Mac is a choice), do they look like a fastq should look?

Sorry for the seemingly basic question, but I just want to make totally sure they are exactly what you expect!

1 Like
9

So I figured out how to look at the first few lines of my files, and they were not fastq files. I am going to re-run the code with my new files and hopefully all goes well.

3 Likes
10

IT WORKED.

Thank you so much Justine, I hope you have a great weekend!

-Gisselle

3 Likes