Phred score is out of range in deblur

Technical Support
, ,
1

I'm using the human gut dataset from Yatsuenko et al. and trying to deblur it. I've downloaded the initial fastq and imported as EMPSingleEndsSequences, and demultiplexed with q2-demux emp-single without issue. However, when I try to denoise (q2-deblur denoise-16S) some of the lanes, I get the following error log:

Loading required package: Rcpp
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:531: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('deblur version %s workflow started on %s' %
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:533: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('parameters: %s' % locals())
Loading required package: Rcpp
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:531: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('deblur version %s workflow started on %s' %
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:533: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('parameters: %s' % locals())
Loading required package: Rcpp
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:531: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('deblur version %s workflow started on %s' %
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:533: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('parameters: %s' % locals())
Loading required package: Rcpp
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:531: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('deblur version %s workflow started on %s' %
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:533: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('parameters: %s' % locals())
Loading required package: Rcpp
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:531: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('deblur version %s workflow started on %s' %
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:533: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('parameters: %s' % locals())
Traceback (most recent call last):
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur", line 684, in
deblur_cmds()
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur", line 640, in workflow
parallel_deblur(input_file_list, sys.argv, pos_ref_db_fp,
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/deblur/parallel_deblur.py", line 140, in parallel_deblur
raise RuntimeError("stdout: %s\nstderr: %s\nexit: %d" % (stdout,
RuntimeError: stdout:
stderr: /gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:531: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('deblur version %s workflow started on %s' %
/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur:533: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn('parameters: %s' % locals())
Traceback (most recent call last):
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur", line 684, in
deblur_cmds()
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/bin/deblur", line 625, in workflow
deblurred_file_name = launch_workflow(
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/deblur/workflow.py", line 830, in launch_workflow
for label, seq in trim_seqs(
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/deblur/workflow.py", line 130, in trim_seqs
for label, seq in input_seqs:
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/deblur/workflow.py", line 99, in sequence_generator
for record in skbio.read(input_fp, format=format, **kw):
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/skbio/io/registry.py", line 506, in
return (x for x in itertools.chain([next(gen)], gen))
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/skbio/io/registry.py", line 531, in _read_gen
yield from reader(file, **kwargs)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/skbio/io/registry.py", line 1008, in wrapped_reader
yield from reader_function(fhs[-1], **kwargs)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/skbio/io/format/fastq.py", line 351, in _fastq_to_generator
phred_scores, seq_header = _parse_quality_scores(fh, len(seq),
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/skbio/io/format/fastq.py", line 522, in _parse_quality_scores
_decode_qual_to_phred(chunk, variant=variant,
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/skbio/io/format/_base.py", line 34, in _decode_qual_to_phred
raise ValueError("Decoded Phred score is out of range [%d, %d]."
ValueError: Decoded Phred score is out of range [0, 62].
exit: 1
Traceback (most recent call last):
File "procedure/denoise.py", line 30, in
deblur_table, deblur_rep_seqs, deblur_stats, = denoise_16S(qual_filtered_seqs, 100, sample_stats = True, jobs_to_start = 40)
File "", line 2, in denoise_16S
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/qiime2/sdk/action.py", line 244, in bound_callable
outputs = self.callable_executor(scope, callable_args,
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/qiime2/sdk/action.py", line 390, in callable_executor
output_views = self._callable(**view_args)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/q2_deblur/_denoise.py", line 89, in denoise_16S
return _denoise_helper(
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/site-packages/q2_deblur/_denoise.py", line 180, in denoise_helper
subprocess.run(cmd, check=True)
File "/gscratch/zaneveld/miniconda3/envs/qiime2-2021.4/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['deblur', 'workflow', '--seqs-fp', '/tmp/qiime2-archive-bqegozp/0007bc0a-5af6-4235-b311-9a2536bbc676/data', '--output-dir', '/tmp/tmpcyyzcsgw', '--mean-error', '0.005', '--indel-prob', '0.01', '--indel-max', '3', '--trim-length', '100', '--left-trim-length', '0', '--min-reads', '10', '--min-size', '2', '--jobs-to-start', '40', '-w', '--keep-tmp-files']' returned non-zero exit status 1.

Searching around, it seems this error normally crops up when importing - I've not found mention of it in deblur. I'm trying a couple different importing methods but it's a fairly lengthy process to get from that step to denoising so I can't iterate very quickly.

Thanks

2

Hi @dylan! Yes, you're right, this is an issue with the import format - it sounds like you imported Phred64 files as Phred33, which is why this is failing here.

Please take a look at this tutorial for details on how to import Phred64 offset files, if you run into any more issues please post your import command here (as well as a directory listing of the files you're importing, to give us a sense of what the raw file filename patterns are).

:qiime2:

3

Thanks for the help; I'm trying the import multiple ways. It looks like the only way to specify the Phred offset is with the manifest import option. However, that seems to be aimed at demultiplexed data - will it take multiplexed fastqs?

Alternatively, could I convert the original fastq from Phred64 to Phred33, then import as EMP?

Based on the filename:

170_s_16_1_withindex_sequence.fastq.gz

I'm also trying to import as MultiplexedSingleEndBarcodeInSequence - though that doesn't let me choose the offset. Will it end up with the same problem? Or does it assume less than the EMP import option?

Thanks

4

That is correct. One simple solution - import as EMP multiplexed sequences, then demux them. Then take the demuxed seqs and export them, then re-import using the manifest format.

Yes.

Phred64 offset is pretty rare these days, we mostly only see it crop up with historical datasets, so most methods in QIIME 2 plugins will assume they're working with Phred33.

5

Thanks very much for your help!

6

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.