vsearch dereplicate problem

b87401116 · April 15, 2023, 6:26pm

Hi! I experienced an error as the following (version 2023.2):

qiime vsearch dereplicate-sequences --i-sequences demux.qza --o-dereplicated-table derep_table.qza --o-dereplicated-sequences derep_seq.qza --verbose
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --derep_fulllength /tmp/q2-QIIME1DemuxDirFmt-w5rmnaz8/seqs.fna --output /tmp/q2-DNAFASTAFormat-an93mybf --relabel_sha1 --relabel_keep --uc /tmp/tmp7s97c4ps --xsize --minseqlength 1 --minuniquesize 1 --fasta_width 0

vsearch v2.22.1_linux_x86_64, 7.7GB RAM, 4 cores
GitHub - torognes/vsearch: Versatile open-source tool for microbiome analysis

Dereplicating file /tmp/q2-QIIME1DemuxDirFmt-w5rmnaz8/seqs.fna 100%
7034804134 nt in 16586037 seqs, min 380, max 538, avg 424
Sorting 100%
8368511 unique sequences, avg cluster 2.0, median 1, max 90765
Writing FASTA output file 100%
Writing uc file, first part 100%
Writing uc file, second part 100%
Traceback (most recent call last):
File "/home/yc/mambaforge/envs/qiime2-2023.2/lib/python3.8/site-packages/q2cli/commands.py", line 352, in call
results = action(**arguments)
File "", line 2, in dereplicate_sequences
File "/home/yc/mambaforge/envs/qiime2-2023.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
outputs = self.callable_executor(scope, callable_args,
File "/home/yc/mambaforge/envs/qiime2-2023.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 381, in callable_executor
output_views = self._callable(**view_args)
File "/home/yc/mambaforge/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_vsearch/_cluster_sequences.py", line 40, in dereplicate_sequences
table = parse_uc(out_uc)
File "/home/yc/mambaforge/envs/qiime2-2023.2/lib/python3.8/site-packages/biom/parse.py", line 302, in parse_uc
observation_id = fields[9].split()[0]
IndexError: list index out of range

Plugin error from vsearch:

list index out of range

See above for debug info.

The 'demux.qza' file is imported from fastq files which had already be merged and quality controlled. Here is the content of the manifest file used for import.

sample-id absolute-filepath
0244_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/0244 saliva Q20 merged 380.R1.fastq
0244_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/0244 swap Q20 merged 380.R1.fastq
0463_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/0463saliva Q20 merged 380.R1.fastq
0463_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/0463swap Q20 merged 380.R1.fastq
1004_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1004 SALIVA Q20-merged-380.R1.fastq
1004_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1004 SWAP Q20-merged-380.R1.fastq
112_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/112saliva Q20 merged 380.R1.fastq
112_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/112swap Q20 merged 380.R1.fastq
1289_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1289 saliva Q20 merged 380.R1.fastq
1289_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1289 swap Q20 merged 380.R1.fastq
1332_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1332 SWAP Q20-merged-380.R1.fastq
1369_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1369 SALIVA Q20-merged-380.R1.fastq
1369_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1369 SWAP Q20-merged-380.R1.fastq
1618_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1618 SALIVA Q20-merged-380.R1.fastq
1618_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1618 SWAP Q20-merged-380.R1.fastq
1622_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1622 SALIVA Q20-merged-380.R1.fastq
1622_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1622 SWAP Q20-merged-380.R1.fastq
1679_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1679 SALIVA Q20-merged-380.R1.fastq
1679_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1679 SWAP Q20-merged-380.R1.fastq
1997_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1997saliva Q20 merged 380.R1.fastq
1997_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1997swap Q20 merged 380.R1.fastq
221_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/221saliva Q20 merged 380.R1.fastq
221_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/221swap Q20 merged 380.R1.fastq
2459_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/2459swap Q20 merged 380.R1.fastq
2763_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/2763 saliva Q20 merged 380.R1.fastq
2763_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/2763 swap Q20 merged 380.R1.fastq
283_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/283saliva Q20 merged 380.R1.fastq
283_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/283swap Q20 merged 380.R1.fastq
2853_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/2853 saliva Q20 merged 380.R1.fastq
2853_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/2853 swap Q20 merged 380.R1.fastq
3121_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/3121 SALIVA Q20-merged-380.R1.fastq
3121_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/3121 SWAP Q20-merged-380.R1.fastq
3347_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/3347 SWAP Q20 merged (trimmed).R1.fastq
3543_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/3543saliva Q20 merged 380.R1.fastq
3543_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/3543swap Q20 merged 380.R1.fastq
3672_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/3672 swap Q20 merged 380.R1.fastq
385_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/385 SALIVA Q20-merged-380.R1.fastq
3870_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/3870 SWAP Q20 merged (trimmed).R1.fastq
4111_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/4111saliva Q20 merged 380.R1.fastq
4111_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/4111swap Q20 merged 380.R1.fastq
425_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/425saliva Q20 merged 380.R1.fastq
425_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/425swap Q20 merged 380.R1.fastq
4309_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/4309 SALIVA Q20-merged-380.R1.fastq
4309_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/4309 SWAP Q20-merged-380.R1.fastq
4812_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/4812 SALIVA Q20-merged-380.R1.fastq
496_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/496saliva Q20 merged 380.R1.fastq
496_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/496swap Q20 merged 380.R1.fastq
1332_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/1332 SALIVA Q20-merged-380.R1.fastq
2459_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/2459saliva Q20 merged 380.R1.fastq
3672_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/3672 saliva Q20 merged 380.R1.fastq
5045_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/5045 saliva Q20 merged 380.R1.fastq
6166_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/6166 SALIVA Q20-merged-380.R1.fastq
5045_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/5045 swap Q20 merged 380.R1.fastq
522_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/522saliva Q20 merged 380.R1.fastq
522_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/522swap Q20 merged 380.R1.fastq
549_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/549saliva Q20 merged 380.R1.fastq
549_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/549swap Q20 merged 380.R1.fastq
5579_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/5579 saliva Q20 merged 380.R1.fastq
5579_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/5579 swap Q20 merged 380.R1.fastq
5654_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/5654 SALIVA Q20-merged-380.R1.fastq
5654_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/5654 SWAP Q20-merged-380.R1.fastq
5927_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/5927saliva Q20 merged 380.R1.fastq
5927_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/5927swap Q20 merged 380.R1.fastq
6135_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/6135 saliva Q20 merged 380.R1.fastq
6135_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/6135 swap Q20 merged 380.R1.fastq
6611_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/6611 saliva Q20 merged 380.R1.fastq
6611_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/6611 swap Q20 merged 380.R1.fastq
7146_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/7146saliva Q20 merged 380.R1.fastq
7146_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/7146swap Q20 merged 380.R1.fastq
7160_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/7160 SALIVA Q20-merged-380.R1.fastq
7160_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/7160 SWAP Q20-merged-380.R1.fastq
733_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/733 SALIVA Q20-merged-380.R1.fastq
7621_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/7621saliva Q20 merged 380.R1.fastq
7679_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/7679 saliva Q20 merged 380.R1.fastq
7679_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/7679 swap Q20 merged 380.R1.fastq
7961_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/7961swap Q20 merged 380.R1.fastq
802_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/802 SALIVA Q20-merged-380.R1.fastq
802_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/802 SWAP Q20-merged-380.R1.fastq
8698_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/8698 saliva Q20 merged 380.R1.fastq
8698_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/8698 swap Q20 merged 380.R1.fastq
8796_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/8796 saliva Q20 merged 380.R1.fastq
8796_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/8796 swap Q20 merged 380.R1.fastq
9285_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/9285saliva Q20 merged 380.R1.fastq
9285_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/9285swap Q20 merged 380.R1.fastq
944_saliva /mnt/microbiota/analysis/202304_CKD/CKD/fastq/944 SALIVA Q20-merged-380.R1.fastq
944_swab /mnt/microbiota/analysis/202304_CKD/CKD/fastq/944 SWAP Q20-merged-380.R1.fastq

Any idea about that?
Many thanks to anyone who may help!

b87401116 · April 16, 2023, 1:36am

Hi, I think I figured out the problem!
I rerun the command and found the temporary uc file 'tmpknfejktb' broken.
The last lines of this file look like

S 5124853 426 * * * * * 4309_saliva_9102859 *
S 5124854 429 * * * * * 4309_saliva_9102860 *
S 5124855 427 * * * * * 4309_saliva_9102861 *
S 5124856 420 * * * * * 4309_saliva_9102864 *
S 5124857 399 * * * * * 4309_saliva_9102865 *
S 5124858 426 * * * * * 4309_saliva_9102868 *
S 5124859 427 * * * * * 4309_saliva_9102869 *
S 5124860 404 * * * * * 4309_saliva_9102870 *
S 5124861 407 * * * * * 4309_saliva_9102871 *
S 5124862 428 *

I guess this might be due to insufficient disk space in the /tmp directory. Therefore, I change the TMPDIR environmental variable to another disk and the dereplication process completed smoothly. However, I still don't know why the scripts did not raise an error with the broken temporary uc file.

colinbrislawn · April 16, 2023, 1:38am

I'm glad you got this working! Thank you for posting the result on the forums!

Hopefully, when disk space is used up or there is not enough RAM, the error message will say that.

However, sometimes this causes something else to break and the error message is about this other issue and not about memory or disk space.

b87401116 · April 17, 2023, 5:30am

Thank you for the reply. More precisely, I am just wondering why it did not raise any error during the process of uc file generation and passed to the next step.

colinbrislawn · April 17, 2023, 5:35pm

This is because the .uc file is a direct output of vsearch, and I suspect this part of the pipeline finished successfully. The next step in the pipeline that uses Python failed, so you see the error there.

system · May 18, 2023, 11:36pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.