Error while runing DADA2

Technical Support
1

Hi team,
I want to run 1035 samples in dada2 (in qiime 2 version 2029-4). but I encontered this error. I tried and reran the command more that 10 times but still getting same error. Please find below the error and the command used. The analysis was carried out but unable to write the results.
your help is highly appreciated.

qiime dada2 denoise-paired
--i-demultiplexed-seqs /mnt/beegfs/scratch/eh309/combinedanalysis/demuxCombinedLanes.qza
--p-trim-left-f 0
--p-trim-left-r 0
--p-trunc-len-f 0
--p-trunc-len-r 0
--p-chimera-method none
--p-n-threads 0
--o-table tablecombined.qza
--o-representative-sequences rep-seqscombined.qza
--o-denoising-stats denoising-stats.qza
--verbose

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /mnt/beegfs/scratch/eh309/combinedanalysis/tmpdr/tmpt8onvgon/forward /mnt/beegfs/scratch/eh309/combinedanalysis/tmpdr/tmpt8onvgon/reverse /mnt/beegfs/scratch/eh309/combinedanalysis/tmpdr/tmpt8onvgon/output.tsv.biom /mnt/beegfs/scratch/eh309/combinedanalysis/tmpdr/tmpt8onvgon/track.tsv /mnt/beegfs/scratch/eh309/combinedanalysis/tmpdr/tmpt8onvgon/filt_f /mnt/beegfs/scratch/eh309/combinedanalysis/tmpdr/tmpt8onvgon/filt_r 0 0 0 0 2.0 2 none 1.0 0 1000000

R version 3.5.1 (2018-07-02)
Loading required package: Rcpp
DADA2: 1.10.0 / Rcpp: 1.0.1 / RcppParallel: 4.4.2

  1. Filtering ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
  2. Learning Error Rates
    418595014 total bases in 1535679 reads from 5 samples will be used for learning the error rates.
    377851001 total bases in 1535679 reads from 5 samples will be used for learning the error rates.
  3. Denoise remaining samples ............................................Duplicate sequences in merged output.
    ....Duplicate sequences in merged output.
    ............................Duplicate sequences in merged output.
    ....................................Duplicate sequences in merged output.
    ............................Duplicate sequences in merged output.
    .................................Duplicate sequences in merged output.
    ............Duplicate sequences in merged output.
    .......Duplicate sequences in merged output.
    ............Duplicate sequences in merged output.
    ............Duplicate sequences in merged output.
    ............................................................Duplicate sequences in merged output.
    .....................................................Duplicate sequences in merged output.
    ...........Duplicate sequences in merged output.
    ........................................................................Duplicate sequences in merged output.
    .....Duplicate sequences in merged output.
    ........................Duplicate sequences in merged output.
    ........Duplicate sequences in merged output.
    .......Duplicate sequences in merged output.
    .....Duplicate sequences in merged output.
    .................................................................................................................................................Duplicate sequences in merged output.
    ........................................................................Duplicate sequences in merged output.
    ..............................................Duplicate sequences in merged output.
    ....................Duplicate sequences in merged output.
    ........Duplicate sequences in merged output.
    ....................Duplicate sequences in merged output.
    ........................................Duplicate sequences in merged output.
    .................Duplicate sequences in merged output.
    .......................................................Duplicate sequences in merged output.
    .....Duplicate sequences in merged output.
    ...................................Duplicate sequences in merged output.
    ............................................................................................................
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
    Duplicate sequences detected and merged.
  4. Remove chimeras (method = none)
  5. Write output
    Traceback (most recent call last):
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/commands.py", line 311, in call
    results = action(**arguments)
    File "</mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/site-packages/decorator.py:decorator-gen-451>", line 2, in denoise_paired
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
    output_types, provenance)
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 393, in callable_executor
    spec.qiime_type, output_view, spec.view_type, prov)
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/result.py", line 271, in _from_view
    provenance_capture=provenance_capture)
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/archiver.py", line 316, in from_data
    Format.write(rec, type, format, data_initializer, provenance_capture)
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/format/v5.py", line 21, in write
    provenance_capture)
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/format/v1.py", line 19, in write
    provenance_capture)
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/archive/format/v0.py", line 62, in write
    data_initializer(data_dir)
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/path.py", line 37, in _move_or_copy
    return _ConcretePath.rename(self, other)
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/pathlib.py", line 1309, in rename
    self._accessor.rename(self, target)
    File "/mnt/beegfs/home/eh309/.conda/envs/qiime2-2019.4/lib/python3.6/pathlib.py", line 393, in wrapped
    return strfunc(str(pathobjA), str(pathobjB), *args)
    FileExistsError: [Errno 17] File exists: '/mnt/beegfs/scratch/eh309/combinedanalysis/tmpdr/q2-BIOMV210DirFmt-fjdxr30i' -> '/mnt/beegfs/scratch/eh309/combinedanalysis/tmpdr/qiime2-archive-45lpmwrm/e71d3292-0b58-481a-a6e8-10d8fbe68ded/data'
2

Welcome back, @Hajar!
This looks a lot like an error message we've run into before. Take a look at this post, and let us know if it helps.

Incidentally, 2019.4 is almost a year and a half old now. It's using the same Python version as 2020.8, so I don't expect updating will solve your problem, but installing a new version will get you some newer features and better performance in a couple places.

Best of luck,
Chris :sloth:

3

Thanks Chris for your reply.

Yes, I saw this post before. I've already contacted my HPC admins at the university, but they said that I have plenty of space with no memory restriction. I changed TMPDIR to a directory requested by them. However, I am still experiencing the same issue.
It is really frustrating as I run the analysis for 2 days and in the end, the program is not able to write the output.
I would really appreciate any help on this matter.
Thanks.

4

That does sound frustrating, @Hajar . This is a rare issue, and seems to occur only on beegfs and maybe similar nfs-like file systems. It may be an incompatability between the python standard library's handling of temporary files and your university's compute resource configuration. As such, I think this really is an issue to work out with your sysadmin. Sorry I can't be more helpful.

In the meantime, here are some thoughts that might be useful:

  • I don't think this is a space/memory issue. space/memory errors have been described clearly in prior posts of yours, and this is likely something different
  • What file system is being used on the new temporary directory you're using? Is it possible to write your outputs to a non-BeeGFS file system?
  • If you need to make progress while this gets worked out, you can always rent a server. Depending on what resources you need, it can be shockingly cheap. Search this forum for "rent a computer" or "rent a server" and you'll find some resources.

Good luck, and if you learn anything useful, please let us know so that other forum users can benefit from your experience.

CK

5

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.