Problem with cache using qiime cutadapt

Alecs · March 24, 2026, 7:57pm

Hello!
I'm using qiime2 locally on my personal MacBook Air to analyse eDNA barcode reads.
Since I have not much of space on my device I am storing all the files and launching the command on an external hard drive.
I have managed to import my fastq creating a temporary dir with these commands:

export TMPDIR=../tmp_qiime

mkdir -p $TMPDIR

chmod 1777 $TMPDIR

and then

qiime tools import 
--type MultiplexedPairedEndBarcodeInSequence 
--input-path muxed-pe-barcode-in-seq 
--output-path multiplexed-seqs.qza

So far so good. I have paired end fastq with in sequence barcodes both for the fw and rv sequences.
I set again the tmpdir and check with python -c "import tempfile; print(tempfile.gettempdir())".

At this point I want to run cutadapt to demultiplex my sequences with this command:

sudo qiime cutadapt demux-paired 
--i-seqs multiplexed-seqs.qza 
--m-forward-barcodes-file tag.tsv 
--m-forward-barcodes-column barcode1 
--m-reverse-barcodes-file tag.tsv 
--m-reverse-barcodes-column barcode2 
--p-batch-size 3 
--use-cache=../tmp_qiime  
--o-per-sample-sequences per-sample-sequences.qza 
--o-untrimmed-sequences untrimmed-sequences.qza

But after a while it crashes and I get this warning on the log file in the tmp directory of my Mac:

Command: cutadapt -g file:/tmp/tmpjdm0xur8 --error-rate 0.1 --minimum-length 1 -o /tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-w7d0szpr/{name}.1.fastq.gz --untrimmed-output /tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-asnsre9p/forward.fastq.gz --pair-adapters -G file:/tmp/tmppdhft3se -p /tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-w7d0szpr/{name}.2.fastq.gz --untrimmed-paired-output /tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-asnsre9p/reverse.fastq.gz /tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/forward.fastq.gz /tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/reverse.fastq.gz -U 0 -u 0 -j 1

This is cutadapt 5.1 with Python 3.10.14
Command line parameters: -g file:/tmp/tmpjdm0xur8 --error-rate 0.1 --minimum-length 1 -o /tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-w7d0szpr/{name}.1.fastq.gz --untrimmed-output /tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-asnsre9p/forward.fastq.gz --pair-adapters -G file:/tmp/tmppdhft3se -p /tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-w7d0szpr/{name}.2.fastq.gz --untrimmed-paired-output /tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-asnsre9p/reverse.fastq.gz /tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/forward.fastq.gz /tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/reverse.fastq.gz -U 0 -u 0 -j 1
Adapter 'ACACACAC' (regular 5') was specified multiple times! Please make sure that this is what you want.
Adapter 'ACACACAC' (regular 5') was specified multiple times! Please make sure that this is what you want.
Processing paired-end reads on 1 core ...
[Errno 28] No space left on device
Traceback (most recent call last):
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/cli.py", line 1261, in main
    stats = runner.run(pipeline, progress, outfiles)
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/runners.py", line 430, in run
    (n, total1_bp, total2_bp) = pipeline.process_reads(
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/pipeline.py", line 147, in process_reads
    reads = step(*reads, info1, info2)  # type: ignore
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/steps.py", line 491, in __call__
    self._untrimmed_writer.write(read1, read2)
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/dnaio/pairedend.py", line 240, in write
    self._writer2.write(read2)
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/dnaio/writers.py", line 152, in _write
    self._file.write(record.fastq_bytes())
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/isal/igzip.py", line 213, in write
    self.fileobj.write(self.compress.compress(data))
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/cli.py", line 1281, in main
    sys.exit(exit_code)
SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/bin/cutadapt", line 8, in <module>
    sys.exit(main_cli())
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/cli.py", line 1166, in main_cli
    main(sys.argv[1:])
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/cli.py", line 1285, in main
    outfiles.close()
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/files.py", line 301, in close
    bf.close()
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/gzip.py", line 334, in close
    fileobj.write(self.compress.flush())
OSError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/q2cli/commands.py", line 573, in __call__
    results = self._execute_action(
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/q2cli/commands.py", line 645, in _execute_action
    results = action(**arguments)
  File "<decorator-gen-995>", line 2, in demux_paired
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/rachis/sdk/action.py", line 307, in bound_callable
    outputs = self._callable_executor_(
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/rachis/sdk/action.py", line 496, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/q2_cutadapt/_demux.py", line 330, in demux_paired
    untrimmed = _demux(
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/q2_cutadapt/_demux.py", line 240, in _demux
    run_command(cmd)
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/q2_cutadapt/_demux.py", line 38, in run_command
    subprocess.run(cmd, check=True)
  File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cutadapt', '-g', 'file:/tmp/tmpjdm0xur8', '--error-rate', '0.1', '--minimum-length', '1', '-o', '/tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-w7d0szpr/{name}.1.fastq.gz', '--untrimmed-output', '/tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-asnsre9p/forward.fastq.gz', '--pair-adapters', '-G', 'file:/tmp/tmppdhft3se', '-p', '/tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-w7d0szpr/{name}.2.fastq.gz', '--untrimmed-paired-output', '/tmp/qiime2/root/processes/5266-1774374188.316834@root/tmp/q2-OutPath-asnsre9p/reverse.fastq.gz', '/tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/forward.fastq.gz', '/tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/reverse.fastq.gz', '-U', '0', '-u', '0', '-j', '1']' returned non-zero exit status 1.

What am I doing wrong? Is there a way to force cutadapt to use the tmpdir in my external hard disk? Or do I need to run cutadapt outside qiime2 and then import my demultiplexed samples?
Thank you in advance

Alex

ebolyen · March 25, 2026, 12:56pm

Hey @Alecs,

I would say the only thing really wrong is running QIIME 2 with sudo, I definitely don’t recommend that (but I have also done similar under duress).

--use-cache=../tmp_qiime

This should have worked (assuming that path resolves from your current working directory).

It is technically possible that the cutadapt plugin will still use /tmp, but that’s a separate issue, as it does look like your actual cache is still also in /tmp.

Now thinking about it a little more, sudo is actually a different environment and so you would lose TMPDIR (again ideally –use-cache would avoid the problem in general).

Would you mind giving the output of pwd in the same location you run the command and then also trying again without sudo (so that your TMPDIR variable is available)?

EDIT: actually this would be better ls $PWD/../tmp_qiime

Unrelated, but I am so excited you found and opened that file!

Alecs · March 25, 2026, 5:02pm

Thanks for the input!
If I do not use sudo I get this error message:
ValueError: Directory '/Volumes/KESU/hydro/tmp_qiime/qiime2' already exists without proper permissions '0o41777' set. Current permissions are '0o40777.' This most likely means something other than QIIME 2 created the directory '/Volumes/KESU/hydro/tmp_qiime/qiime2' or QIIME 2 failed between creating '/Volumes/KESU/hydro/tmp_qiime/qiime2' and setting permissions on it.

While using sudo before the qiime cutadapt command leads me to the previous error.

If I set my tmpdir to tmp_qiime and use

ls $PWD/../tmp_qiime      
qiime2

so as with sudo

Even though the command crashes and the log file I have copied in my main post is located in the tmp of my Mac

Hope I made myself clear,
thank you again for your interest

ebolyen · March 26, 2026, 9:49am

Hey @Alecs,

Great, I think that does help explain the issue. After spending too long decoding the octal mask, the error is saying that we’re missing the sticky bit on the directory qiime2.

What you could do is check if there’s anything interesting in there, and if not, delete the directory and try again, or if there is something you want, I think you can use:

sudo chmod +t /Volumes/KESU/hydro/tmp_qiime/qiime2

If that doesn’t work, then it may be that the filesystem of the external device doesn’t support this, and so we’re just in an odd situation, but we can cross that bridge in a bit.

Now what might also be causing some issues is you are using tmp_qiime as both a cache and a TMPDIR. I’m not entirely sure who is going to win here, but the TMPDIR logic will try to create a qiime2/$USER folder (which is a user-specific cache under tmp), whereas the --use-cache will try to use tmp_qiime as the cache outright.

I think what you probably want to do is something like this:

export TMPDIR=/Volumes/KESU/hydro/tmp_qiime/

(just like before) and

--use-cache=/Volumes/KESU/hydro/tmp_qiime/my_cache

which uses an extra subdirectory here so that my_cache doesn’t fight with qiime2/$USER

Pinging @Oddant1,

Is there any chance that our permission masks are different between the default cache and the user cache? It seems to me that they would be, since only the tmp cache needs the sticky bit for multi-tenant systems.

I wonder if this was the original source of the issue.

Alecs · March 26, 2026, 12:47pm

Thank you again!
So, I removed the qiime2 directory and made another one and laughed the sudo chom command, again launched the export TMPDIR=/Volumes/KESU/hydro/tmp_qiime/.
I made a new dir named my_cache and used --use-cache=/Volumes/KESU/hydro/tmp_qiime/my_cache

After a while I obtain the same message:

Plugin error from cutadapt:

Command '['cutadapt', '-g', 'file:/tmp/tmprovxnt0u', '--error-rate', '0.1', '--minimum-length', '1', '-o', '/tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-60hbgk8g/{name}.1.fastq.gz', '--untrimmed-output', '/tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-2nvhoxv3/forward.fastq.gz', '--pair-adapters', '-G', 'file:/tmp/tmpnhtxei4h', '-p', '/tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-60hbgk8g/{name}.2.fastq.gz', '--untrimmed-paired-output', '/tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-2nvhoxv3/reverse.fastq.gz', '/tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/forward.fastq.gz', '/tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/reverse.fastq.gz', '-U', '0', '-u', '0', '-j', '1']' returned non-zero exit status 1.

Debug info has been saved to /tmp/qiime2-q2cli-err-xb0mb0gi.log

So I go and check the log file (in the tmp of my main device) and I get this warning:

This is cutadapt 5.1 with Python 3.10.14
Command line parameters: -g file:/tmp/tmprovxnt0u --error-rate 0.1 --minimum-length 1 -o /tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-60hbgk8g/{name}.1.fastq.gz --untrimmed-output /tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-2nvhoxv3/forward.fastq.gz --pair-adapters -G file:/tmp/tmpnhtxei4h -p /tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-60hbgk8g/{name}.2.fastq.gz --untrimmed-paired-output /tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-2nvhoxv3/reverse.fastq.gz /tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/forward.fastq.gz /tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/reverse.fastq.gz -U 0 -u 0 -j 1
Adapter 'ACACACAC' (regular 5') was specified multiple times! Please make sure that this is what you want.
Adapter 'ACACACAC' (regular 5') was specified multiple times! Please make sure that this is what you want.
Processing paired-end reads on 1 core ...
[Errno 28] No space left on device
Traceback (most recent call last):
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/cli.py", line 1261, in main
stats = runner.run(pipeline, progress, outfiles)
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/runners.py", line 430, in run
(n, total1_bp, total2_bp) = pipeline.process_reads(
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/pipeline.py", line 147, in process_reads
reads = step(*reads, info1, info2) # type: ignore
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/steps.py", line 491, in call
self._untrimmed_writer.write(read1, read2)
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/dnaio/pairedend.py", line 239, in write
self._writer1.write(read1)
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/dnaio/writers.py", line 152, in _write
self._file.write(record.fastq_bytes())
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/isal/igzip.py", line 213, in write
self.fileobj.write(self.compress.compress(data))
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/cli.py", line 1281, in main
sys.exit(exit_code)
SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/bin/cutadapt", line 8, in
sys.exit(main_cli())
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/cli.py", line 1166, in main_cli
main(sys.argv[1:])
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/cli.py", line 1285, in main
outfiles.close()
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/cutadapt/files.py", line 301, in close
bf.close()
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/gzip.py", line 334, in close
fileobj.write(self.compress.flush())
OSError: [Errno 28] No space left on device
Traceback (most recent call last):
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/q2cli/commands.py", line 573, in call
results = self._execute_action(
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/q2cli/commands.py", line 645, in _execute_action
results = action(**arguments)
File "", line 2, in demux_paired
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/rachis/sdk/action.py", line 307, in bound_callable
outputs = self.callable_executor(
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/rachis/sdk/action.py", line 496, in callable_executor
output_views = self._callable(**view_args)
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/q2_cutadapt/_demux.py", line 330, in demux_paired
untrimmed = _demux(
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/q2_cutadapt/_demux.py", line 240, in _demux
run_command(cmd)
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/site-packages/q2_cutadapt/_demux.py", line 38, in run_command
subprocess.run(cmd, check=True)
File "/opt/miniconda3/envs/qiime2-amplicon-2026.1/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cutadapt', '-g', 'file:/tmp/tmprovxnt0u', '--error-rate', '0.1', '--minimum-length', '1', '-o', '/tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-60hbgk8g/{name}.1.fastq.gz', '--untrimmed-output', '/tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-2nvhoxv3/forward.fastq.gz', '--pair-adapters', '-G', 'file:/tmp/tmpnhtxei4h', '-p', '/tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-60hbgk8g/{name}.2.fastq.gz', '--untrimmed-paired-output', '/tmp/qiime2/root/processes/2201-1774523004.29591@root/tmp/q2-OutPath-2nvhoxv3/reverse.fastq.gz', '/tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/forward.fastq.gz', '/tmp/qiime2/root/data/39e34ff8-5681-4104-aca1-70c9d518608d/data/reverse.fastq.gz', '-U', '0', '-u', '0', '-j', '1']' returned non-zero exit status 1.

For now I had to use cutadapt outside qiime2

ebolyen · March 26, 2026, 1:04pm

Sorry @Alecs

@Oddant1 could you maybe try reproducing this with an external drive when you have a bit of time? It really seems like our cache machinery is just not doing the right thing here.

Alecs · March 27, 2026, 12:35pm

Thank you @ebolyen for your patience.
After demultiplexing with cutadapt and importing my demux file I have tried to remove my primer sequences from my fastq with cutadapt again with this command

sudo qiime cutadapt trim-paired 
--i-demultiplexed-sequences demux.qza 
--p-front-f GTCGGTAAAACTCGTGCCAGC 
--p-front-r CATAGTGGGGTATCTAATCCCAGTTTG 
--p-discard-untrimmed 
--use-cache=/Volumes/KESU/hydro/tmp_qiime/my_cache 
--o-trimmed-sequences demux-trimmed.qza

But I got the same old message
Plugin error from cutadapt:

[Errno 28] No space left on device: '/tmp/qiime2/root/processes/2333-1774608161.129419@root/tmp/q2-OutPath-sl5y7ud3/TAVO-2_24_L001_R1_001.fastq.gz' -> '/tmp/qiime2/root/processes/2333-1774608161.129419@root/tmp/q2-OutPath-v42bxluf/TAVO-2_24_L001_R1_001.fastq.gz'

Debug info has been saved to /tmp/qiime2-q2cli-err-8teaoss6.log

Do you happen to know some work around? I did not think/know how to remove the primers during the demultiplexing steps with cutadapt outside qiime2.
Thank you again!

Alecs · March 27, 2026, 4:26pm

Okay I have just created a nume .dmg on my hard disk side the problem seems to be that it was formatted as expat, now I am able to launch all the command without sudo and use a tidier on my external hard disk!

Oddant1 · March 30, 2026, 5:30pm

Hello @Alecs,

You are seeing some unusual behavior here that I am still trying to fully decipher, but I believe I have a solution for you. Please do the following:

Run the command qiime tools cache-create --cache <path-to-new-dir> that will create a new directory and make it a QIIME 2 cache. You can't use just any dir as a cache, it requires a specific structure.
Rerun your command passing the cache you created in step 1 to your --use-cache argument and do NOT use an = sign. Just --use-cache <your-cache-path> the = seems to be doing something strange that I don't entirely understand.

Thank you,

EDIT: note this all but completely removes the need to set a TMPDIR. I say all but completely, because currently a directory called "qiime2" will still be created in your TMPDIR, but nothing will actually be written to it. Everything will be written to the cache you have used. I have confirmed this behavior, I am on Linux, but it should be the same for you.