–Hi,
i need to increase more temporary space disk during deblur step, so do you think i can remove
the .trim files stored in deblur_working_dir because these are the files that take up the most space.
Can I delete these files without killing the process?
thank you –
I’m unsure if that would or would not impact the runtime, sorry. Deleting temporary data while a program is running can be risky.
-Daniel
–yes i know, but the main problem with Qiime2 is the storage of temporary files. They take up a lot of space and are not necessarily useful for the progress of the steps. It would be useful to keep them if Qiime proposed a parameter to resume an interrupted process but this is not the case. For example, when.qza archives are decompressed into separate files, why are these files not automatically deleted when they are no longer used ? I think developers should consider adding this type of parameter in future releases. Resume or not an interrupted process, if yes you keep temporary files, and if not you delete them.
Hi there @lmanchon!
QIIME 2 uses your system’s temp dir to handle automatic cleanup of temp files (at the interval defined by your operating system). Are you overriding the TMPDIR env var? If so, then those files will most likely not be covered by your system’s machinery for temporary file cleanup.
At the end of the day, all temporary files generated by QIIME 2 should be deleted by your system’s temp file machinery — QIIME 2 behaves like any other unix tool in that respect.
In a way this flag exists — by leaving your TMPDIR env var setting as your system default, this is saying “let my files be cleaned up at the system’s defined interval”. If I set the TMPDIR var to a new value, this is saying “don’t delete my temporary files, let me do it on my own.”
Yep, we love that idea, and it is something that we have been stewing on for a while now. Maybe some time in 2019?
yes i know that i can keep all my temporary files after the process completed, but it’s not exactly what i thought.
During the process, qiime store extracted gzipped files from qza archive into /tmp/data directory to then process them for example in a deblur or dada2 process, these files can be automaticaly deleted after they no longer have any use, why keep them and take up space.
Exactly! That is our thought, too. That is why they are in the temp dir - that way they will get cleaned up by your system. They should be reaped at whatever interval or age criteria your system is set to use (for example, some linux distros will clean immediately, darwin will reap once a file is 3 days old).
Why wait for the system to delete the temporary files, qiime could do it as soon as it no longer needs the files and thus released instantly space !