Docker QIIME2 / No space left on device

HI there,

I’m currently setting up a Nextflow pipeline together with a colleague to run certain steps of QIIME2 in a nextflow automated script. For this, I created a Docker Image containing all tools required by QIIME2 (basically copying the QIIME2 environment.yaml) and then started running things. Most run well, but the classifier apparently runs out of disk space every time with some test data.

  OSError: [Errno 28] No space left on device
  
  Plugin error from feature-classifier:
  
[Errno 28] No space left on device

Trying to set TMPDIR etc via Nextflow didn’t help at all unfortunately - can someone maybe elaborate where this step stores temporary data in general?

One more thing here. I read up a bit on the failure and saw that this temp dir thing is just used when an appropriate backend is used. I had to use an offscreen renderer to get around qt issues, so that might be the problem why this is not used at all?

When running inside my docker container, I can run echo $TEMP/$TEMPDIR/$TMP and they all point to the correct /tmp folder on my host which is mounted inside the container appropriately.

https://scikit-learn.org/stable/modules/generated/sklearn.utils.Parallel.html

Setting JOBLIB_TEMP_FOLDER first, setting this fixed to /tmp and using nextflows docker.temp = 'auto' resolved the issue.

2 Likes

Hey there @apeltzer, sorry we didn't get back to you sooner, things have been busy trying to get the release out the door. Glad to hear you found a solution! Related, this issue came up pretty recently on the issue tracker for the Docker builds:

My plan is to incorporate this change in the 2018.11 docker image --- probably will happen some time next week. Thanks!

Makes sense - also mentioning that this is something that can be adjusted would be nice :slight_smile: