error while running DADA2 in R (return code 1), all reads filtered.

Technical Support
1

Hi,
Newbie here, trying to use QIIME 2 2019.4 in a miniconda3 environment on a HPC cluster, I am having troubles getting DADA2 to run:

qiime dada2 denoise-paired --i-demultiplexed-seqs seqs.qza --p-trim-left-f 40 --p-trim-left-r 9 --p-trunc-len-f 240 --p-trunc-len-r 220 --o-table table.qza --o-representative-sequences rep-seqs.qza --o-denoising-stats denoising-stats.qza --verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /tmp/tmpl7eitvjl/forward /tmp/tmpl7eitvjl/reverse /tmp/tmpl7eitvjl/output.tsv.biom /tmp/tmpl7eitvjl/track.tsv /tmp/tmpl7eitvjl/filt_f /tmp/tmpl7eitvjl/filt_r 240 220 40 9 2.0 2 consensus 1.0 1 1000000

R version 3.5.1 (2018-07-02)
Loading required package: Rcpp
DADA2: 1.10.0 / Rcpp: 1.0.1 / RcppParallel: 4.4.2

  1. Filtering The filter removed all reads: /tmp/tmpl7eitvjl/filt_f/AF185_S44_L001_R1_001.fastq.gz and /tmp/tmpl7eitvjl/filt_r/AF185_S44_L001_R2_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmpl7eitvjl/filt_f/AF186_S45_L001_R1_001.fastq.gz and /tmp/tmpl7eitvjl/filt_r/AF186_S45_L001_R2_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmpl7eitvjl/filt_f/AF187_S46_L001_R1_001.fastq.gz and /tmp/tmpl7eitvjl/filt_r/AF187_S46_L001_R2_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmpl7eitvjl/filt_f/AF188_S47_L001_R1_001.fastq.gz and /tmp/tmpl7eitvjl/filt_r/AF188_S47_L001_R2_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmpl7eitvjl/filt_f/AF189_S48_L001_R1_001.fastq.gz and /tmp/tmpl7eitvjl/filt_r/AF189_S48_L001_R2_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmpl7eitvjl/filt_f/AF191_S49_L001_R1_001.fastq.gz and /tmp/tmpl7eitvjl/filt_r/AF191_S49_L001_R2_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmpl7eitvjl/filt_f/AF192_S50_L001_R1_001.fastq.gz and /tmp/tmpl7eitvjl/filt_r/AF192_S50_L001_R2_001.fastq.gz not written.
    qiime tools view demux.qzaThe filter removed all reads: /tmp/tmpl7eitvjl/filt_f/AF193_S51_L001_R1_001.fastq.gz and /tmp/tmpl7eitvjl/filt_r/AF193_S51_L001_R2_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmpl7eitvjl/filt_f/AF197_S52_L001_R1_001.fastq.gz and /tmp/tmpl7eitvjl/filt_r/AF197_S52_L001_R2_001.fastq.gz not written.
    Some input samples had no reads pass the filter.
    xxxxx.xxx...x...
  2. Learning Error Rates
    15800 total bases in 79 reads from 7 samples will be used for learning the error rates.
    16669 total bases in 79 reads from 7 samples will be used for learning the error rates.
    Error in err[c(1, 6, 11, 16), ] <- 1 :
    incorrect number of subscripts on matrix
    Execution halted
    Traceback (most recent call last):
    File "/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 231, in denoise_paired
    run_commands([cmd])
    File "/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 36, in run_commands
    subprocess.run(cmd, check=True)
    File "/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['run_dada_paired.R', '/tmp/tmpl7eitvjl/forward', '/tmp/tmpl7eitvjl/reverse', '/tmp/tmpl7eitvjl/output.tsv.biom', '/tmp/tmpl7eitvjl/track.tsv', '/tmp/tmpl7eitvjl/filt_f', '/tmp/tmpl7eitvjl/filt_r', '240', '220', '40', '9', '2.0', '2', 'consensus', '1.0', '1', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:
seqs.qzv (288.5 KB)
Traceback (most recent call last):
File "/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/commands.py", line 311, in call
results = action(**arguments)
File "</home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/decorator.py:decorator-gen-451>", line 2, in denoise_paired
File "/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
File "/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 365, in callable_executor
output_views = self._callable(**view_args)
File "/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 246, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

In this thread it was suspected that trimming may have been to much (Plugin error from dada2: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more - #2 by thermokarst) but this shouldnt really be the case here?
Any thoughts? I read through the other threads but I am still not sure what is going on or why everything would be filtered out.

Thanks in advance!

2

Hi @frdmn,
Welcome to the forum!
Can you tell us what kind of data you are using here? (Illumina, shotgun, 454 etc..) and how much memory you are accessing?
You have a couple of different errors and I'm not sure which one is at the source of it:

I believe this initial problem is that you are losing most of your reads in the initial filtering step because your reverse reads are in very poor shape and so they are discarded leaving very little reads for the error model to learn from and denoise. I would just discard your reverse reads and use your forward reads only.

To me this seems like the main problem and I think I've seen this error when the wrong data type is being used for example 454 or shotgun data. Is this by chance the case for your data?

3

Hi Mehrbod, thanks so much for your response.

It’s Illumina Miseq (2x250 paired end reads) data.

Memory: I tried to run it via a script that allocated it 128 GB on the HPC cluster, but since it never worked, I just tried to directly run the one line pasted above. As I have obviously no idea what I am doing, I am not sure how much memory it would access or how to change that if I work directly in the command line?

I also tried to denoise with deblur instead of dada2, that failed too, and then as suggested I discard the reverse reads but it still fails:

qiime dada2 denoise-single --i-demultiplexed-seqs seqs.qza --o-table table-dada2 --o-representative-sequences rep-seqs --p-trim-left 40 --p-trunc-len 250 --o-denoising-stats denoising-stats.qza --verbose

(qiime2-2019.4) n9606661@lyra04:~/subset> qiime dada2 denoise-single --i-demultiplexed-seqs seqs.qza --o-table table-dada2 --o-representative-sequences rep-seqs --p-trim-left 40 --p-trunc-len 250 --o-denoising-stats denoising-stats.qza --verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /tmp/q2-SingleLanePerSampleSingleEndFastqDirFmt-w07aft4i /tmp/tmp2mquy63z/output.tsv.biom /tmp/tmp2mquy63z/track.tsv /tmp/tmp2mquy63z 250 40 2.0 2 Inf consensus 1.0 1 1000000 NULL 16

R version 3.5.1 (2018-07-02)
Loading required package: Rcpp
DADA2: 1.10.0 / Rcpp: 1.0.1 / RcppParallel: 4.4.2

  1. Filtering The filter removed all reads: /tmp/tmp2mquy63z/AF185_S44_L001_R1_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmp2mquy63z/AF186_S45_L001_R1_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmp2mquy63z/AF187_S46_L001_R1_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmp2mquy63z/AF188_S47_L001_R1_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmp2mquy63z/AF191_S49_L001_R1_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmp2mquy63z/AF192_S50_L001_R1_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmp2mquy63z/AF193_S51_L001_R1_001.fastq.gz not written.
    The filter removed all reads: /tmp/tmp2mquy63z/AF197_S52_L001_R1_001.fastq.gz not written.
    Some input samples had no reads pass the filter.
    xxxx…xxx…x…
  2. Learning Error Rates
    219278640 total bases in 1044184 reads from 8 samples will be used for learning the error rates.
  3. Denoise samples …
  4. Remove chimeras (method = consensus)
  5. Report read numbers through the pipeline
  6. Write output
    Traceback (most recent call last):
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/commands.py”, line 311, in call
    results = action(**arguments)
    File “</home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/decorator.py:decorator-gen-449>”, line 2, in denoise_single
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 231, in bound_callable
    output_types, provenance)
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 393, in callable_executor
    spec.qiime_type, output_view, spec.view_type, prov)
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 271, in _from_view
    provenance_capture=provenance_capture)
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/archiver.py”, line 316, in from_data
    Format.write(rec, type, format, data_initializer, provenance_capture)
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/format/v5.py”, line 21, in write
    provenance_capture)
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/format/v1.py”, line 26, in write
    prov_dir, [root / cls.METADATA_FILE, archive_record.version_fp])
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/provenance.py”, line 312, in finalize
    self.write_action_yaml()
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/provenance.py”, line 289, in write_action_yaml
    fh.write(yaml.dump({‘execution’: self.make_execution_section()},
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/provenance.py”, line 255, in make_execution_section
    runtime[‘start’] = start = self._ts_to_date(self.start)
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/provenance.py”, line 249, in _ts_to_date
    return datetime.fromtimestamp(ts, tzlocal.get_localzone())
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/tzlocal/unix.py”, line 131, in get_localzone
    _cache_tz = _get_localzone()
    File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/tzlocal/unix.py”, line 85, in _get_localzone
    with open(tzpath, ‘rt’) as tzfile:
    PermissionError: [Errno 13] Permission denied: ‘/etc/sysconfig/clock’

Plugin error from dada2:

[Errno 13] Permission denied: ‘/etc/sysconfig/clock’

See above for debug info.

And then since I thought maybe it is some problem with my dat, I tried to work through the moving pictures tutorial but even here I run into a problem, making me think it might not be my data?

(qiime2-2019.4) n9606661@lyra04:~/qiime2-moving-pictures-tutorial> qiime tools import --type EMPSingleEndSequences --input-path emp-single-end-sequences --output-path emp-single-end-sequences.qza
Traceback (most recent call last):
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/builtin/tools.py”, line 152, in import_data
view_type=input_format)
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 240, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 271, in _from_view
provenance_capture=provenance_capture)
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/archiver.py”, line 316, in from_data
Format.write(rec, type, format, data_initializer, provenance_capture)
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/format/v5.py”, line 21, in write
provenance_capture)
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/format/v1.py”, line 26, in write
prov_dir, [root / cls.METADATA_FILE, archive_record.version_fp])
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/provenance.py”, line 312, in finalize
self.write_action_yaml()
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/provenance.py”, line 289, in write_action_yaml
fh.write(yaml.dump({‘execution’: self.make_execution_section()},
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/provenance.py”, line 255, in make_execution_section
runtime[‘start’] = start = self._ts_to_date(self.start)
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/provenance.py”, line 249, in _ts_to_date
return datetime.fromtimestamp(ts, tzlocal.get_localzone())
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/tzlocal/unix.py”, line 131, in get_localzone
_cache_tz = _get_localzone()
File “/home/n9606661/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/tzlocal/unix.py”, line 85, in _get_localzone
with open(tzpath, ‘rt’) as tzfile:
PermissionError: [Errno 13] Permission denied: ‘/etc/sysconfig/clock’

An unexpected error has occurred:

[Errno 13] Permission denied: ‘/etc/sysconfig/clock’

See above for debug info.

4

Hi @frdmn,
This new error is different than the previous one! That’s good because it means we worked through some of the other issues.
Unrelated to the main issues here, but moving forward, I would recommended you truncate more from your forward reads as well, at 250 you are losing many reads, so much that some of your samples are lost completely. Try truncating values from 140, 160, 180 etc. you will retain more reads the more you truncate but lose a bit of a resolution obviously. Finding the sweet spot of compromise between # of reads you retain and how long you want the reads to be is up to you. Lots of other samples like this on the forum to look through.
Now to the meat of your problem:

PermissionError: [Errno 13] Permission denied: ‘/etc/sysconfig/clock’

There appears to be a permission issue related to your HPC cluster (this is why both your data and the tutorial data are not working). Unfortunately I’m not experience enough with these to help you troubleshoot so you’ll have to wait for one of the devs to get back to the office to guide you or you can try asking your own cluster admins and they should be able to troubleshoot this more thoroughly on their end.
Good luck!

5

Thanks, I will following up with our HPC support team to see what they say regarding the permission error and hope I n continue then. But why would I lose more reads when truncuating at 250? Should't there be more than when truncuating to the sequences more, i.e. to 150?

1 Like
6

Hi @frdmn,
It may seem a bit counter-intuitive but the reason is in the filtering step that occurs prior to denoising.
As read lengths increases so does the error rate due to the poor quality sequences on the 3’ end. This causes DADA2 to discard those reads all together and so those reads never get processed. On the other hand if you truncate your long reads to shorter lengths where the quality scores are still reasonable, the filtering processes retains these reads and denoises them.
Hope that makes sense, if it doesn’t try searching the forum with regards to dada2 truncating parameters and there should be lots of different examples on this topic that will be more thoroughly described.