Errors when using dada2 denoise

hongwei2017 · February 23, 2018, 12:14am

Dear Qiime2 team,

I was running the same code that worked for me before, but when I ran it recently, it kept showing me errors.

Code: “qiime dada2 denoise-paired --i-demultiplexed-seqs 16S_paired-end-demux.qza --p-trunc-len-f 250 --p-trunc-len-r 220 --p-trim-left-f 17 --p-trim-left-r 21 --p-n-threads 16 --o-table 16S_table.qza --o-representative-sequences 16S_seqs.qza”

“Bad key “backened” on line 1 in
/home/30050419/.config/matplotlib/matplotlibrc.
You probably need to get an updated matplotlibrc file from

github.com

matplotlib/matplotlib/blob/master/matplotlibrc.template

#### MATPLOTLIBRC FORMAT

## This is a sample matplotlib configuration file - you can find a copy
## of it on your system in
## site-packages/matplotlib/mpl-data/matplotlibrc.  If you edit it
## there, please note that it will be overwritten in your next install.
## If you want to keep a permanent local copy that will not be
## overwritten, place it in the following location:
## unix/linux:
##      $HOME/.config/matplotlib/matplotlibrc or
##      $XDG_CONFIG_HOME/matplotlib/matplotlibrc (if $XDG_CONFIG_HOME is set)
## other platforms:
##      $HOME/.matplotlib/matplotlibrc
##
## See http://matplotlib.org/users/customizing.html#the-matplotlibrc-file for
## more details on the paths which are checked for the configuration file.
##
## This file is best viewed in a editor which supports python mode
## syntax highlighting. Blank lines, or lines starting with a comment
## symbol, are ignored, as are trailing comments.  Other lines must

This file has been truncated. show original

or from the matplotlib source distribution
Traceback (most recent call last):
File “/opt/miniconda3/envs/qiime2-2017.7/bin/qiime”, line 6, in
sys.exit(q2cli.main.qiime())
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/q2cli/commands.py”, line 189, in call
arguments, missing_in, verbose, quiet = self.handle_in_params(kwargs)
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/q2cli/commands.py”, line 281, in handle_in_params
kwargs, fallback=cmd_fallback
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/q2cli/handlers.py”, line 302, in get_value
return qiime2.Artifact.load(path)
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/qiime2/sdk/result.py”, line 61, in load
archiver = archive.Archiver.load(filepath)
File “/opt/miniconda3/envs/qiime2-2017.7/lib/python3.5/site-packages/qiime2/core/archive/archiver.py”, line 292, in load
cls._futuristic_archive_error(filepath, archive)
TypeError: _futuristic_archive_error() takes 2 positional arguments but 3 were given”

When I ran this code using qiime2.12, it gave me this error

"QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.

Bad key “backened” on line 1 in
/home/30050419/.config/matplotlib/matplotlibrc.
You probably need to get an updated matplotlibrc file from

github.com

matplotlib/matplotlib/blob/master/matplotlibrc.template

#### MATPLOTLIBRC FORMAT

## This is a sample matplotlib configuration file - you can find a copy
## of it on your system in
## site-packages/matplotlib/mpl-data/matplotlibrc.  If you edit it
## there, please note that it will be overwritten in your next install.
## If you want to keep a permanent local copy that will not be
## overwritten, place it in the following location:
## unix/linux:
##      $HOME/.config/matplotlib/matplotlibrc or
##      $XDG_CONFIG_HOME/matplotlib/matplotlibrc (if $XDG_CONFIG_HOME is set)
## other platforms:
##      $HOME/.matplotlib/matplotlibrc
##
## See http://matplotlib.org/users/customizing.html#the-matplotlibrc-file for
## more details on the paths which are checked for the configuration file.
##
## This file is best viewed in a editor which supports python mode
## syntax highlighting. Blank lines, or lines starting with a comment
## symbol, are ignored, as are trailing comments.  Other lines must

This file has been truncated. show original

or from the matplotlib source distribution"

Can you please give me some clue about how to fix this problem?

Thanks

jairideout · February 23, 2018, 5:26pm

Hi @hongwei2017! Can you please send me the following info?

The output from running qiime info
The output from running echo $SHELL
Are you running these commands directly on your computer or server, or are you executing them through a shell script or cluster queueing system?

It looks like you have a corrupted matplotlib config file. Can you also try running these commands?

rm /home/30050419/.config/matplotlib/matplotlibrc
export MPLBACKEND='Agg'

After that, try rerunning the DADA2 command to see if it’s fixed. If that solves the issue, you can put the export MPLBACKEND='Agg' command in your ~/.bashrc file for it to take effect each time you open a new terminal window.

hongwei2017 · February 25, 2018, 11:24pm

Hi @jairideout

Please see the following results by running your commands:

(qiime2-2017.7) [30050419@u0217 ~]$ qiime info

QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.

Bad key “backened” on line 1 in
/home/30050419/.config/matplotlib/matplotlibrc.
You probably need to get an updated matplotlibrc file from

github.com

matplotlib/matplotlib/blob/master/matplotlibrc.template

### MATPLOTLIBRC FORMAT

# This is a sample matplotlib configuration file - you can find a copy
# of it on your system in
# site-packages/matplotlib/mpl-data/matplotlibrc.  If you edit it
# there, please note that it will be overwritten in your next install.
# If you want to keep a permanent local copy that will not be
# overwritten, place it in the following location:
# unix/linux:
#     $HOME/.config/matplotlib/matplotlibrc or
#     $XDG_CONFIG_HOME/matplotlib/matplotlibrc (if $XDG_CONFIG_HOME is set)
# other platforms:
#     $HOME/.matplotlib/matplotlibrc
#
# See http://matplotlib.org/users/customizing.html#the-matplotlibrc-file for
# more details on the paths which are checked for the configuration file.
#
# This file is best viewed in a editor which supports python mode
# syntax highlighting. Blank lines, or lines starting with a comment
# symbol, are ignored, as are trailing comments.  Other lines must

This file has been truncated. show original

or from the matplotlib source distribution
System versions
Python version: 3.5.3
QIIME 2 release: 2017.7
QIIME 2 version: 2017.7.0
q2cli version: 2017.7.0

Installed plugins
alignment 2017.7.0
composition 2017.7.1
dada2 2017.7.0
deblur 2017.7.0
demux 2017.7.0
diversity 2017.7.0
emperor 2017.7.0
feature-classifier 2017.7.0
feature-table 2017.7.0
gneiss 2017.7.0
metadata 2017.7.0
phylogeny 2017.7.0
quality-filter 2017.7.0
taxa 2017.7.0
types 2017.7.0

Application config directory
/home/30050419/.config/q2cli

Getting help
To get help with QIIME 2, visit https://qiime2.org

Citing QIIME 2
If you use QIIME 2 in any published work, you should cite QIIME 2 and the plugins that you used. To display the citations for QIIME 2 and all installed plugins, run:

qiime info --citations

(qiime2-2017.7) [30050419@u0217 ~]$ echo $SHELL
/bin/bash
I am running these commands on a server.

(qiime2-2017.7) [30050419@u0217 ~]$ rm/home/30050419/.config/matplotlib/matplotlibrc
-bash: rm/home/30050419/.config/matplotlib/matplotlibrc: No such file or directory

It didn’t fix the problem, probably the "rm /home/30050419/.config/matplotlib/matplotlibrc
export MPLBACKEND=‘Agg’ " codes didn’t work.

hongwei2017 · February 26, 2018, 2:09am

HI @jairideout

The other thing I would like you know is that when I am running my commands of dada2 using Qiime2-2017.12, it did show errors, but took a long time (3 days), but still hasn’t completed yet. Please see below:

(qiime2-2017.12) [30050419@u0217 Hugo_16S]$ qiime dada2 denoise-paired --i-demultiplexed-seqs 16S_paired-end-demux.qza --p-trunc-len-f 250 --p-trunc-len-r 220 --p-trim-left-f 17 --p-trim-left-r 21 --p-n-threads 0 --o-table 16S_table.qza --o-representative-sequences 16S_seqs.qza

Bad key “backened” on line 1 in
/home/30050419/.config/matplotlib/matplotlibrc.
You probably need to get an updated matplotlibrc file from

github.com

matplotlib/matplotlib/blob/master/matplotlibrc.template

### MATPLOTLIBRC FORMAT

# This is a sample matplotlib configuration file - you can find a copy
# of it on your system in
# site-packages/matplotlib/mpl-data/matplotlibrc.  If you edit it
# there, please note that it will be overwritten in your next install.
# If you want to keep a permanent local copy that will not be
# overwritten, place it in the following location:
# unix/linux:
#     $HOME/.config/matplotlib/matplotlibrc or
#     $XDG_CONFIG_HOME/matplotlib/matplotlibrc (if $XDG_CONFIG_HOME is set)
# other platforms:
#     $HOME/.matplotlib/matplotlibrc
#
# See http://matplotlib.org/users/customizing.html#the-matplotlibrc-file for
# more details on the paths which are checked for the configuration file.
#
# This file is best viewed in a editor which supports python mode
# syntax highlighting. Blank lines, or lines starting with a comment
# symbol, are ignored, as are trailing comments.  Other lines must

This file has been truncated. show original

or from the matplotlib source distribution

This is very bizzare to me, it was all OK before, but suddenly doesn’t work now.

Cheers

thermokarst · February 26, 2018, 2:31pm

Hi @hongwei2017, a few notes here:

You are running a very old version of QIIME 2, is it possible for you to upgrade to the latest, 2018.2?
Please copy-and-paste the command that @jairideout provided very carefully (rm /home/30050419/.config/matplotlib/matplotlibrc). According to the new log you provided, you skipped the space between rm and /home... (-bash: rm/home/30050419/.config/matplotlib/matplotlibrc: No such file or directory)

The error you are seeing is not a QIIME 2 error, but rather a Matplotlib error - it looks like a custom matplotlibrc file was created in your user account, but it has a typo in it: backened, when it should be backend. Please remove that file, as @jairideout suggested, and look into upgrading to the latest QIIME 2 release, this should solve your issues. Thanks!

hongwei2017 · February 28, 2018, 1:37am

Dear @thermokarst

When I am using the qiime2-2017.12, it doesn’t show errors, but my commands have been running in Qiime for more than one day, and when it reach to forward reads of sample 11 (75 samples in total), it took very long time (>10 h) and still hasn’t complete yet. According to the current speed, it may take ages to finalize the whole denoise step.

(qiime2-2017.12) [30050419@u0217 Hugo_16S]$ qiime dada2 denoise-paired --i-demultiplexed-seqs 16S_paired-end-demux.qza --p-trunc-len-f 250 --p-trunc-len-r 220 --p-trim-left-f 17 --p-trim-left-r 21 --p-n-threads 8 --o-table 16S_table --o-representative-sequences 16S_seqs --verbose

R version 3.4.1 (2017-06-30)
Loading required package: Rcpp
DADA2 R package version:1.6.0

filtering…
Learning Error Rates
2a) Forward Reads
Initializing error rates to maximum possible estimate.
Sample 1- 131818 reads in 61655 unique sequences.
Sample 2-127555 reads in 59241 unique sequences.
Sample 3-118574 reads in 54449 unique sequences.
Sample 4- 83937 reads in 54449 unique sequences.
sample 5- 97920 reads in 44572 unique sequences.
sample 6- 122973 reads in in 61023 unique sequences.
sample 7 - 109869 reads in 55350 unique sequences.
sample 8 - 76782 reads in 40465 unique sequences.
sample 9- 7585 reads in 4793 unique sequences.
sample 10- 108984 reads in 51364 unique sequences.
sample 11- 162847 reads in 76556 unique sequences
selfConsist step 2
selfConsist step 3
selfConsist step 4
selfConsist step 5
selfConsist step 6
selfConsist step 7
selfConsist step 8
selfConsist step 9
selfConsist step 10

I will leave it running but would like to report this to you first, you may have an idea if this is normal.

Thanks!

thermokarst · February 28, 2018, 3:34pm

Hi @hongwei2017, it sounds like things are working! Personally, I would let this job continue on — it looks like it was maybe halfway done at the point that you posted, so maybe it will be done soon (?). Keep us posted!

PS - check out this rough time estimate on the DADA2 docs!

hongwei2017 · March 5, 2018, 11:17pm

@thermokarst

This run took almost 4 days and eventually completed. I guess the slowness of this run may have been caused by the high number of sequences of some samples.

system · April 6, 2018, 5:17am

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.