Our research group is running the DADA2 denoise process on a Linux server. We are using QIIME2 version 2017.2. We have 56 samples with a demux-paired-end.qza size around 10 GB.
Is there any way of designating an alternate temp file directory before running the denoise process? Or is there any way of redirecting the temp files to a specific directory within the qiime dada2 denoise-paired command itself?
Last, is there an estimate of disk storage required for processing of the 10 GB paired-end artifact file? Does the demultiplexed qza file copy itself into temporary storage?
Please let m e know if there is any additional information I can provide.
We don't currently have a great way to do this in QIIME 2 -- in the future we'll have a configuration system that'll let users set a temporary directory and other host-related details. There's an open issue to track progress on the configuration system.
In the meantime, you can override the default (platform dependent) location by setting the TMPDIR environment variable to point to your temporary directory. Be sure to use an absolute path. For example:
export TMPDIR="/home/foo/my-temp-dir"
Yes, loading the demux .qza will extract the ZIP file into a temporary directory, so that'll take up another 10GB of disk space (it'll get cleaned up after the analysis finishes). DADA2 will also take up disk space writing its output and there may be some other disk usage during the analysis. Overall, I'd expect to see around ~10GB of temporary storage being used during your DADA2 analysis. Please let us know if that's not the case!
Note: we've discussed mounting ZIP files (i.e. .qza and .qzv files) in order to avoid duplicating data in temporary storage. This may be possible with FUSE but I'm not sure when this feature will be available in QIIME 2.
The DADA2 step uses the temp directory to store filtered versions of the input files, so the temp data should take up no more disk space than the input data.
As @jairideout said, if that isn't the case, please let us know!