Error involving temporary file (?) in de-noise DADA2 step

User Support
1

Hi,

I am currently running into an error and have no idea how to troubleshoot/fix it (or really even understand why it is happening). I am roughly following the Moving Pictures tutorial to analyze V3/V4 data. I have used the same methods (Moving Pictures slightly modified) in the past to thoroughly analyze V1/V2 data successfully. In case in makes a difference, I have the R1/R2 data but am only using the R1 (in attempts to be more comparable to my V1V2 that was single-read and to make the de-noise quicker, though I have no idea if either intention is actually valid).

I did this first:

~ adrian$ qiime tools import
--type 'SampleData[SequencesWithQuality]'
--input-path singleseq
--source-format CasavaOneEightSingleLanePerSampleDirFmt
--output-path demux-single-end.qza

And then this:

adrian$ qiime dada2 denoise-single
--i-demultiplexed-seqs demux-single-end.qza
--p-trim-left 0
--p-trunc-len 200
--o-representative-sequences rep-seqs-dada2.qza
--o-table table-dada2.qza
Traceback (most recent call last):
File "/Users/adrian/miniconda3/envs/qiime2-2017.9/bin/qiime", line 6, in
sys.exit(q2cli.main.qiime())
File "/Users/adrian/miniconda3/envs/qiime2-2017.9/lib/python3.5/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/Users/adrian/miniconda3/envs/qiime2-2017.9/lib/python3.5/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/adrian/miniconda3/envs/qiime2-2017.9/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/adrian/miniconda3/envs/qiime2-2017.9/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/adrian/miniconda3/envs/qiime2-2017.9/lib/python3.5/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/adrian/miniconda3/envs/qiime2-2017.9/lib/python3.5/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/adrian/miniconda3/envs/qiime2-2017.9/lib/python3.5/site-packages/q2cli/commands.py", line 228, in call
os.remove(log.name)
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/sc/j_366psj5yx_wcf23g83z_xm0000gn/T/qiime2-q2cli-err-yxikiirl.log'`

After running for ~6 days, the error appeared.

Here are the files related to my data:
demuxsingle.qzv (280.4 KB)
and then the giant .qza file can be found here: Dropbox - demux-single-end.qza - Simplify your life
metadata.tsv (1.1 KB)

3

Hey @adrian,

It looks like you ran into a limitation of OS X. We've seen this before and what is happening is your computer is automatically deleting data from its temporary directory that is 3 days old.

The solution is to run the command with a different temporary directory set:

# Make a directory to use for temporary storage
mkdir qiime2-tmp/
# Set that directory as our temporary directory for this session
# (this variable only lasts for as long as the terminal exists)
export TMPDIR="$PWD/qiime2-tmp/" 
# Run your command again
qiime dada2 denoise-single \
  --i-demultiplexed-seqs demux-single-end.qza \
  --p-trim-left 0 \
  --p-trunc-len 200 \
  --o-representative-sequences rep-seqs-dada2.qza \
  --o-table table-dada2.qza

Unfortunately you will need to wait another 6 days for the results. But it should complete successfully since you made it to the end the first time.

Once it has completed you can delete qiime2-tmp or keep it as you see fit.

Sorry we don't have a better solution than this at the moment :disappointed:

2 Likes
5

Also, to speed things up, you can use --p-n-threads <some number> which will run some of the steps in parallel. (You can set it to 0 to use every CPU).

2 Likes
6

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.

7

In QIIME 2 2018.2 we told q2cli to stop misbehaving and write the output even if it can’t retrieve its log file :writing_hand:

(There are still situations where you might run into this issue within a plugin’s action however, so we still don’t have a general solution for OS X reaping files after 3-days.)