Cutadapt trim-paired error 28

hjpyun · August 25, 2021, 3:01am

Hi everyone Thank you for checking my issue.

Here is procedure of my analysis, command line and error

1) Data : 573 samples, pair-ended, V3-V4 region, demultiplexed

2) Current : Trimming primer sequences (then join-end, OTU clustering) // Even ASV is recommended for qiime2, but still.. //

3) Command

qiime cutadapt trim-paired \
--i-demultiplexed-sequences demux.qza \
--p-cores 20 \
--p-front-f ~ \ # V3 primer 
--p-front-r ~ \ # V4 primer 
--o-trimmed-sequences primer-trimmed.qza \
--verbose \

4) Error : It worked for while, and there were even summary stats, but at the end, there comes an error

=== Summary ===

Total read pairs processed:            226,141
  Read 1 with adapter:                 222,905 (98.6%)
  Read 2 with adapter:                 222,344 (98.3%)
Pairs that were too short:                   0 (0.0%)
Pairs written (passing filters):       226,141 (100.0%)

Total basepairs processed:   113,522,782 bp
  Read 1:    56,761,391 bp
  Read 2:    56,761,391 bp
Total written (filtered):    105,049,167 bp (92.5%)
  Read 1:    52,966,807 bp
  Read 2:    52,082,360 bp

=== First read: Adapter 1 ===

Sequence: (V3 primer sequences) ; Type: regular 5'; Length: 50; Trimmed: 222905 times

No. of allowed errors:
1-9 bp: 0; 10-19 bp: 1; 20-29 bp: 2; 30-39 bp: 3; 40-49 bp: 4

Overview of removed sequences
length	count	expect	max.err	error counts
3	15	3533.5	0	15
4	5	883.4	0	5
5	2	220.8	0	2
9	1	0.9	0	1
13	8	0.0	1	7 1
14	7	0.0	1	1 6
15	11	0.0	1	2 9
16	572	0.0	1	156 416
17	216169	0.0	1	205816 10353
18	6077	0.0	1	1070 5007
19	38	0.0	1	0 38

=== Second read: Adapter 2 ===

Sequence: (V4 primer sequences); Type: regular 5'; Length: 55; Trimmed: 222344 times

No. of allowed errors:
1-9 bp: 0; 10-19 bp: 1; 20-29 bp: 2; 30-39 bp: 3; 40-49 bp: 4; 50-55 bp: 5

Overview of removed sequences
length	count	expect	max.err	error counts
3	17	3533.5	0	17
4	2	883.4	0	2
11	5	0.1	1	4 1
12	10	0.0	1	8 2
13	9	0.0	1	6 3
14	4	0.0	1	3 1
15	12	0.0	1	4 8
16	17	0.0	1	4 13
17	7	0.0	1	5 2
18	32	0.0	1	5 27
19	85	0.0	1	6 13 66
20	873	0.0	2	121 698 54
21	210030	0.0	2	204811 4288 931
22	10785	0.0	2	4528 5795 462
23	442	0.0	2	0 298 144
24	14	0.0	2	0 0 14

Traceback (most recent call last):
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/site-packages/q2cli/commands.py", line 329, in __call__
    results = action(**arguments)
  File "<decorator-gen-520>", line 2, in trim_paired
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 244, in bound_callable
    outputs = self._callable_executor_(scope, callable_args,
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 417, in _callable_executor_
    artifact = qiime2.sdk.Artifact._from_view(
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/site-packages/qiime2/sdk/result.py", line 267, in _from_view
    result = transformation(view, validate_level)
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/site-packages/qiime2/core/transform.py", line 70, in transformation
    new_view = transformer(view)
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/site-packages/q2_types/per_sample_sequences/_transformer.py", line 101, in _4
    return _single_lane_per_sample_fastq_helper_partial(
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/site-packages/q2_types/per_sample_sequences/_util.py", line 50, in _single_lane_per_sample_fastq_helper
    result.sequences.write_data(view, fastq_fmt, sample_id=sample_id,
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/site-packages/qiime2/plugin/model/directory_format.py", line 88, in write_data
    result.path._move_or_copy(self.path_maker(**kwargs))
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/site-packages/qiime2/core/path.py", line 44, in _move_or_copy
    return self._copy_dir_or_file(other)
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/site-packages/qiime2/core/path.py", line 34, in _copy_dir_or_file
    return shutil.copy(str(self), str(other))
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/shutil.py", line 418, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/shutil.py", line 275, in copyfile
    _fastcopy_sendfile(fsrc, fdst)
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/shutil.py", line 166, in _fastcopy_sendfile
    raise err from None
  File "/data/home/NCC_kjs/anaconda3/envs/qiime2/lib/python3.8/shutil.py", line 152, in _fastcopy_sendfile
    sent = os.sendfile(outfd, infd, offset, blocksize)
OSError: [Errno 28] No space left on device: '/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-wt6vgi4d/E1026831_519_L001_R1_001.fastq.gz' -> '/tmp/q2-SingleLanePerSamplePairedEndFastqDirFmt-vcqfcbep/E1026831_519_L001_R1_001.fastq.gz'

Plugin error from cutadapt:

[Errno 28] No space left on device: '/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-wt6vgi4d/E1026831_519_L001_R1_001.fastq.gz' -> '/tmp/q2-SingleLanePerSamplePairedEndFastqDirFmt-vcqfcbep/E1026831_519_L001_R1_001.fastq.gz'

See above for debug info.

Here, I wanted to check that directory (/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-wt6vgi4d/) there was no such directory

5) Server Space info
So I deleted /tmp files, and there are even large space left. I checked other threads solving problem with making other TMPDIR: path, but I don't understand why I should do such thing even there are enough space in /tmp..!

-df -h

/tmp : -ls -alh

Thank you, I really hope someone can give me advice

lizgehret · August 27, 2021, 11:34pm

Hi @hjpyun,

Thanks for providing all of those error logs and details!

Based on the traceback message [Errno 28] No space left on device, it appears as though there actually isn't enough space within /tmp to complete this command. Something that you can do to address this would be to modify your TMPDIR environment variable to point to a partition with more available space (such as /data, which has 40T of space).

One note I want to add about this is that /tmp is not just used by QIIME 2 - all programs on your machine are utilizing this partition - and thus, it is not recommended to delete all files within /tmp, as this may interfere with other processes you have running.

Hopefully this helps!

Cheers,
Liz

hjpyun · August 29, 2021, 2:08am

Thank you @lizgehret for your reply.

So is it something like even at the moment when I check the /tmp file, there is enough space, but as command line starts to work, /tmp space is not enough to handle...?

And thanks for your advice not to delete /tmp files. It just that the server was new and there were only files related to qiime so I deleted everything.. I won't do it again

Thanks for your help

lizgehret · August 30, 2021, 5:40pm

Hi @hjpyun,

I would disagree with your statement above. What that error is telling you is that because you are using a partition with 70G of space, the command you are running requires greater than 70G in order to successfully complete its process. Hence, my recommendation for using a partition with more than 70G of space. Hope this helps!

Cheers,
Liz

system · September 30, 2021, 11:40pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.