Plugin error from deblur

User Support
1

Dear all,

I have paired end reads for which I merged using vsearch and removed primers using cutadapt (I had four base pair spike with NNNN that needed to be removed so I removed all primers). I then imported my data into qiime 2 and followed the "[Alternative methods of read-joining in QIIME 2](tutorial Alternative methods of read-joining in QIIME 2 — QIIME 2 2018.2.0 documentation)". However when I ran Deblur I had the following error message:

DeblurErrorMessage
DeblurErrorMessage952×267 108 KB

/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "/home/mark/miniconda2/envs/deblurenv/bin/deblur", line 4, in <module>
    __import__('pkg_resources').run_script('deblur==1.0.3', 'deblur')
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 750, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1527, in run_script
    exec(code, namespace, namespace)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur-1.0.3-py3.5.egg-info/scripts/deblur", line 684, in <module>
    deblur_cmds()
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur-1.0.3-py3.5.egg-info/scripts/deblur", line 641, in workflow
    neg_ref_db_fp, jobs_to_start)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur/parallel_deblur.py", line 142, in parallel_deblur
    es))
RuntimeError: stdout: 
stderr: /home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "/home/mark/miniconda2/envs/deblurenv/bin/deblur", line 4, in <module>
    __import__('pkg_resources').run_script('deblur==1.0.3', 'deblur')
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 750, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1527, in run_script
    exec(code, namespace, namespace)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur-1.0.3-py3.5.egg-info/scripts/deblur", line 684, in <module>
    deblur_cmds()
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur-1.0.3-py3.5.egg-info/scripts/deblur", line 632, in workflow
    threads_per_sample=threads_per_sample)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur/workflow.py", line 832, in launch_workflow
    left_trim_len=left_trim_length):
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur/workflow.py", line 130, in trim_seqs
    for label, seq in input_seqs:
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur/workflow.py", line 99, in sequence_generator
    for record in skbio.read(input_fp, format=format, **kw):
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/skbio/io/registry.py", line 506, in <genexpr>
    return (x for x in itertools.chain([next(gen)], gen))
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/skbio/io/registry.py", line 531, in _read_gen
    yield from reader(file, **kwargs)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/skbio/io/registry.py", line 1008, in wrapped_reader
    yield from reader_function(fhs[-1], **kwargs)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/skbio/io/format/fastq.py", line 344, in _fastq_to_generator
    seq, qual_header = _parse_sequence_data(fh, seq_header)
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/skbio/io/format/fastq.py", line 481, in _parse_sequence_data
    _blank_error("before '+'")
  File "/home/mark/miniconda2/envs/deblurenv/lib/python3.5/site-packages/skbio/io/format/fastq.py", line 473, in _blank_error
    raise FASTQFormatError(error_string)
skbio.io._exception.FASTQFormatError: Found blank or whitespace-only line before '+' in FASTQ file

exit: 1
Traceback (most recent call last):
  File "/home/mark/miniconda2/envs/qiime2-2018.2/lib/python3.5/site-packages/q2cli/commands.py", line 246, in __call__
    results = action(**arguments)
  File "<decorator-gen-344>", line 2, in denoise_16S
  File "/home/mark/miniconda2/envs/qiime2-2018.2/lib/python3.5/site-packages/qiime2/sdk/action.py", line 228, in bound_callable
    output_types, provenance)
  File "/home/mark/miniconda2/envs/qiime2-2018.2/lib/python3.5/site-packages/qiime2/sdk/action.py", line 363, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/mark/miniconda2/envs/qiime2-2018.2/lib/python3.5/site-packages/q2_deblur/_denoise.py", line 96, in denoise_16S
    hashed_feature_ids=hashed_feature_ids)
  File "/home/mark/miniconda2/envs/qiime2-2018.2/lib/python3.5/site-packages/q2_deblur/_denoise.py", line 163, in _denoise_helper
    subprocess.run(cmd, check=True)
  File "/home/mark/miniconda2/envs/qiime2-2018.2/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['deblur', 'workflow', '--seqs-fp', '/tmp/qiime2-archive-ts48smgs/b2be2ea1-0067-4e1d-a22c-ed90fcb79087/data', '--output-dir', '/tmp/tmpx9g3ucks', '--mean-error', '0.005', '--indel-prob', '0.01', '--indel-max', '3', '--trim-length', '248', '--min-reads', '10', '--min-size', '2', '--jobs-to-start', '8', '-w', '--keep-tmp-files']' returned non-zero exit status 1

I aware of the following post which redirects me here, however reinstalling python 3.5 within a conda environment for Deblur did not solve my issues.

I would be grateful for any help!!! Thanks!!

3

Hi @Mark_Gillingham

I apologize for the delayed response. This line in the traceback suggests your input sequence file is malformed, which suggests something went wrong prior to Deblur:

Do you have the specific commands that were executed before Deblur?

Best,
Daniel

5

Many thanks for your reply and no need to apologize, I understand we are all busy. The commands I ran before Deblur within Qiime 2 were:

qiime tools import
–type ‘SampleData[JoinedSequencesWithQuality]’
–input-path /home/mark/Dropbox/MicrobiomeHM/QIIME2/FinalAnalysis/PrimersCut/manifest
–source-format SingleEndFastqManifestPhred33
–output-path /home/mark/Dropbox/MicrobiomeHM/QIIME2/FinalAnalysis/qiime2output/no-primers-demux.qza

qiime demux summarize
–i-data no-primers-demux.qza
–o-visualization no-primers-demux.qzv
#Saved Visualization to: no-primers-demux.qzv

qiime tools view no-primers-demux.qzv

qiime quality-filter q-score-joined
–i-demux no-primers-demux.qza
–o-filtered-sequences demux-joined-filtered.qza
–o-filter-stats demux-joined-filter-stats.qza

I ran everything again today, including merging and cutadapt, from the beginning in case I did something wrong that I cannot trace back and I will let you know the output as soon as it is done.

best,
Mark

7

Unfortunately I get the same error message. With the same raw files (prior to merging and removing primers) I am able to run through the DADA2 pipeline (but I want to compare results using DADA2 and Deblur). So I guess something must be wrong during merging and the removing of primers using cutadapt but I cannot see where the problem is…

8

I have managed to solve my problem. Previously I was merging (using vsearch) and removing primers (using cutadapt) prior to importing my data into QIIME2 (simply because I was familiar with these tools prior to be familiar with QIIME2 and I was used to doing things this way). Since I could not find the source of the error I adopted a new strategy of importing my reads not merged and running cutapapt and vsearch within QIIME2 using the script below. Whilst running this script cutadapt within QIIME2 revealed that one of these files was somehow corrupted (blank or whitespace-only line before ‘+’). However strangely stand alone cutadapt had no issue with this file and neither did vsearch (or the DADA2 pipeline within QIIME2). When I remove this file from my dataset I am able to run Deblur until the end without error. I will now try to find out why this single file is corrupted. Thank you for your help!

qiime tools import
–type ‘SampleData[PairedEndSequencesWithQuality]’
–input-path /home/mark/Dropbox/MicrobiomeHM/QIIME2/FinalAnalysis/RawData
–source-format CasavaOneEightSingleLanePerSampleDirFmt
–output-path /home/mark/Dropbox/MicrobiomeHM/QIIME2/FinalAnalysis/qiime2output/Deblur/demux-paired-end.qza

qiime demux summarize
–i-data demux-paired-end.qza
–o-visualization demux-paired-end.qzv

qiime cutadapt trim-paired
–i-demultiplexed-sequences demux-paired-end.qza
–p-cores 8
–p-front-f NNNNGTGCCAGCNGCCGCGGTAA
–p-front-r GGACTACNVGGGTWTCTAAT
–verbose
–output-dir demux-joined-noprimers.qza

qiime vsearch join-pairs
–i-demultiplexed-seqs trimmed_sequences.qza
–verbose
–o-joined-sequences demux-joined.qza

qiime demux summarize
–i-data demux-joined.qza
–o-visualization no-primers-demux.qzv

qiime quality-filter q-score-joined
–i-demux demux-joined.qza
–o-filtered-sequences demux-joined-filtered.qza
–o-filter-stats demux-joined-filter-stats.qza

qiime deblur denoise-16S
–i-demultiplexed-seqs demux-joined-filtered.qza
–p-trim-length 248
–p-jobs-to-start 8
–p-sample-stats
–o-representative-sequences rep-seqs.qza
–o-table table.qza
–o-stats deblur-stats.qza

2 Likes
11

Thank you for the follow up and investigation @Mark_Gillingham! I believe @ebolyen is looking into the broader validations here.

Best,
Daniel

12

Hi @Mark_Gillingham!

You workflow looks great to me, let us know how it goes!

There is also a utility called qiime tools validate which you can use on artifacts whenever you suspect they might be misbehaving.

One of the things we're really trying to work on in QIIME 2, is centralizing a lot of the parsing and validation of formats. Because as you can see, every tool has a slightly different implementation, and some catch issues, and other don't (but it's really the file formats themselves that deserve the ire, they can be notoriously ambiguous at times).

14

Hi @ebolyen!

Many thanks. I was not aware of this utility. I will try it out. For some reason when I removed two sequences from the problem file everything worked perfectly. I cannot explain it for the moment… but to be honest two sequences are out of millions are not going to change my results so I don’t think I will spend much more time figuring out what went wrong :wink:

thanks again for your quick feedback and help

Best,

Mark

1 Like
15

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.