I have an error message in denoise-paired commands

Hi!

I have a trouble with denoise-paired process.
I have attached the code I used and error messages that I got.

CHOSUNGJOONs-MacBook-Pro:Jeong data SUNGJOONCHO$ qiime dada2 denoise-paired \
>   --i-demultiplexed-seqs demux.qza \
>   --o-table table \
>   --o-representative-sequences rep-seqs \
>   --p-trim-left-f 0 \
>   --p-trim-left-r 0 \
>   --p-trunc-len-f 300 \
>   --p-trunc-len-r 300
Traceback (most recent call last):
  File "/Users/SUNGJOONCHO/miniconda3/envs/qiime2-2017.6/bin/qiime", line 6, in <module>
    sys.exit(q2cli.__main__.qiime())
  File "/Users/SUNGJOONCHO/miniconda3/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/SUNGJOONCHO/miniconda3/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/SUNGJOONCHO/miniconda3/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/SUNGJOONCHO/miniconda3/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/SUNGJOONCHO/miniconda3/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/SUNGJOONCHO/miniconda3/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/SUNGJOONCHO/miniconda3/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/commands.py", line 241, in __call__
    os.remove(log.name)
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/xx/pghwmr2n773df2n3kq_fl_s00000gn/T/qiime2-q2cli-err-ehlm83c6.log'

Could you please tell me what I should do to deal with this issue?
Thank you very much in advance for your help!

Hey @scho73,

This is definitely the first time we’ve seen this. Here’s roughly what I suspect happened:

The QIIME 2 CLI (q2cli) captures stderr and stdout logs just in case something goes wrong and writes them to a file. When it realizes that everything went well, it will delete that file (because there’s nothing interesting in it). Otherwise, if something did go wrong, it will give you that filepath so that you can look at it (or give it to us).

However it looks like the code that is responsible for handling failure (by capturing the logs) has failed. Ironically your denoise-paired command didn’t fail. Rather it looks like your computer’s temporary directory was cleared out between starting the denoise-paired command and it finishing. So when q2cli went to clean up its mess, it couldn’t find the file and produced what you see here.

What is definitely a bummer is that it also looks like q2cli did not save any of your results after the command finished because it was too caught up in its missing log file.

I am pretty sure that this is a transient error which means if you were to re-run this command it would succeed normally.

If you’d like to be sure, you could also run a faster command first like qiime demux summarize which would give you the same error if something was actually wrong with your system (but I don’t think there is).

Are you running any other software that you might expect to clear out temporary files?

1 Like

HI ebolyen,

Thanks for your quick reponse.
Before I ran denoise flag, I had done qiime demux summarize flag which worked well.
And I have no idea of any other software that I might expect to clear out temporary… (Sorry I am still very novice int phyton and principles behind these kinds of command line…)
What I can tell you is that the data size is so big that I spent almost three days to denoise it and during that time, command temporarily stopped when mac went sleep (but I figure this out later).

Let me rerun again and I will let you know when it is working.
Thank you very much again!

I did a bit of poking around and it looks like OS X had (and still has) a background-process which will clear the temporary files after 3 days of non-use. So it is quite likely that if you were to re-run it, it would still take about 3 days and q2cli's temporary log would be cleaned up by your operating system.

This has the potential to explain a lot of weird things we've seen in OS X. Thank you so much for bringing this possibility to our attention! We'll try and figure out some work-arounds.

Until then, I don't really have a good suggestion for how to fix this in the short-term. Sorry :frowning:

1 Like

@scho73,

An update on this. You can avoid this problem by setting the TMPDIR environment variable. If it is set to a different directory, then OS X will have no reason to reap any of these temporary files.

You can create any directory you’d like to serve as a temp dir, but I might do something like this:

mkdir $HOME/qiime2-tmp

Then before you run a command that you think might take longer than 3 days, just run the following:

export TMPDIR=$HOME/qiime2-tmp

This variable is unset after you close your terminal, so you’ll need to re-export for any new session. (I would avoid putting this line in something like .bash_profile however, because it has the potential to make other programs act differently which you probably don’t want).

1 Like

Thanks for kind update!

Before I rerun, I would like to make sure that I am doing in a correct way.

Below is the step that I will do and could you please check if there is any mistake?

  1. mkdir $HOME/qiime2-tmp (i.e, make a new directory "qiime2-tmp")
  2. copy and paste demux.qza and metadata file into the "qiime2-tmp" directory.
  3. run the following flag first
    " export TMPDIR=$HOME/qiime2-tmp"
  4. And run the denoise flag as followings

qiime dada2 denoise-paired
--i-demultiplexed-seqs demux.qza
--o-table table
--o-representative-sequences rep-seqs
--p-trim-left-f 0
--p-trim-left-r 0
--p-trunc-len-f 300
--p-trunc-len-r 300\

Thank you very much!

Hi @scho73, your instructions are close, but have an extra step in there (#2, you do not need to move any of your analysis files). Revised, your steps should look like this (these are all commands you should run in your terminal):

  1. mkdir $HOME/qiime2-tmp
  2. export TMPDIR=$HOME/qiime2-tmp
  3. qiime dada2 denoise-paired ...[fill in the rest here]...

As @ebolyen mentioned above, the first two steps here change where your computer will save temporary files that are created as part of the denoising process. Hope that helps!

1 Like

Hi thermokarst,

Thanks for comments.
I will follow it and let you know whether it works.

Btw, I rerun the all flags without making $HOME/qiime2-tmp and it worked.
This time I tried not to touch my labtop during processing in order to decrease total running time which was about 70 hr (not accurate though).

Anyway, this time, I will make $HOME/qiime2-tmp and let you know the result.
Thank you!

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.

This particular situation has been fixed in QIIME 2 2018.2.

While we don't have a solution for preventing files from disappearing after 3 days on OS X, we did just change q2cli to not worry about it's missing log file if it can't find it.