qiime2 OTU picking with hundreds of samples

[MichelaRiba (michela)]

Hi,

thanks a lot for you answer.

Good point to separate the denoto from the rest, may you be more practical to tell where I can see it in the scripts called by vsearch?
I suppose that vsearch cluster-features-open-reference does exactly those steps, if you could tell me where I can see the commands I could dissect the workflow.

Thanks a lot

Michela

joined_import_filter_derep_OTU:
echo PBS -N star -l select=2:ncpus=12:mem=(STAR_RAM)gb;\ condactivate qiime2.1 ;\ qiime vsearch cluster-features-open-reference \ --i-table table.qza \ --i-sequences rep-seqs.qza \ --i-reference-sequences 85_otus.qza \ --p-perc-identity 0.85 \ --p-threads (CORES)
–o-clustered-table table-or-85bis.qza
–o-clustered-sequences rep-seqs-or-85bis.qza
–o-new-reference-sequences new-ref-seqs-or-85bis.qza
–verbose

May I follow for example your page:

qiime2/docs/blob/master/source/tutorials/otu-clustering.rst

Here I can see

• Closed reference OTU picking with vsearch
  which produces a file of unclustered

• I would took those unclustered and apply instead open reference with 99% identity instead of 85%

Supposing (going to test) that the second phase would last a long time
I could take some additional decision

• Then How could I merge the results?

Thanks a lot

Michela

[thermokarst]

@MichelaRiba

may you be more practical to tell where I can see it in the scripts called by vsearch?

The general steps are here:

@thermokarst

• cluster_features_closed_reference
• filter_features to remove any features from feature table produced provided to step 1 that weren’t clustered in step 1
• cluster_features_de_novo using the unmatched seqs from 1 and the filtered table from 2
• merge the results

The specific steps are here:

qiime2/q2-vsearch/blob/58389bc60a87e6ce35f5f08e71c21e67360e5956/q2_vsearch/_cluster_features.py#L301-L356

Hope that helps!

[MichelaRiba (michela)]

Hi

Thanks a lot!!

I am currently following MicrobiotaMi meeting and I will be hands on again next week

Michela

[MichelaRiba (michela)]

Hi,

I am running VSEARCH closed reference OTU picking, it seems again to be very slow with the complete dataset, total of 8,119,151 sequences (after quality filtering and de-replication).
I link this post, where it seems that vsearch classification may be slower than other methodologies

Problem in assigning taxonomy using VSEARCH consensus taxonomy classifier

Thanks @ebolyen. Unfortunately, @Aishiki’s results fit with our latest test results. We are currently recommending that users avoid using classify-consensus-vsearch for more than tens of sequences. Fortunately, classify-consensus-blast gives very similar performance to classify-consensus-vsearch in terms of accuracy, but in our tests runs 50 times faster. If run time is still an issue, classify-sklearn was 500 times faster in our tests. There is a tutorial for how to use classify-sklearn here. …

"We are currently recommending that users avoid using classify-consensus-vsearch for more than tens of sequences.

Fortunately, classify-consensus-blast gives very similar performance to classify-consensus-vsearch in terms of accuracy, but in our tests runs 50 times faster. If run time is still an issue, classify-sklearn was 500 times faster in our tests. There is a tutorial for how to use classify-sklearn here ."

May be that my problem in vsearch OTU picking relates to that?

Thanks for your patience

Michela

[llenzi (Luca)]

Ciao @MichelaRiba,

I have not much experience with vsearch in QIIME2, but if I recall correctly, one of the differences between vsearch and blast is that blast keeps searching until it finds the specified number of matches above the threshold for an OTU, on the other side vsearch score an OTU against all the sequences in the database then gets the top-best hits. At least should be true for the classify-consensus-vsearch vs classy-consensus-blast, as per plug in description.

So, would make sense to me for vsearch to be more time and memory consuming for a big dataset (also depending on the size of the database).

Luca

[MichelaRiba (michela)]

Hi,
thanks a lot for your kind reply.

This helps me a lot to understand why I am experiencing that very long running time

Michela

[Nicholas_Bokulich]

Thanks for carrying on the discussion @MichelaRiba and @llenzi!

@MichelaRiba

I link this post, where it seems that vsearch classification may be slower than other methodologies

Problem in assigning taxonomy using VSEARCH consensus taxonomy classifier User Support

I am not positive this topic is relevant:

1. that is the vsearch-based taxonomic classifier, not OTU clustering. So some different things are going on under the hood and we may not be able to compare the two.
2. that topic is around 3 years old! I believe that the VSEARCH classifier in q2-feature-classifier was still being developed at that time… in any case, the runtime performance of the classifier is comparable to that of classify-consensus-blast so that comment is not actually accurate anymore: see here for a runtime comparison of these and other taxonomy classifiers. But again that probably is not relevant to OTU clustering.

So far nobody else has reported such unusually long runtimes — nor substantial differences in runtime between qiime1 (uclust) and QIIME 2 (vsearch). So as noted before I still suspect this is either due to characteristics of your dataset, or issues with utilizing resources on your cluster.

But at the end of the day this may just be a fact of life: OTU clustering on this particular dataset you have will take time. More aggressive quality filtering prior to OTU clustering will probably help cut down runtime considerably. Denoising methods will probably also be faster…

[Nicholas_Bokulich]

@MichelaRiba,
here’s another thought: maybe your tmpdir is running out of space?

this forum user mentioned uncovering an error with VSEARCH whereby it does not detect system errors correctly and just keeps running after tmpdir fills up:

Steps for taxonomy assignment on dual-indexed sequence data generated by QIIME 1.9

Hi, I am trying QIIME 2 on one of my wife’s experiments that she had processed with QIIME 1.9 in 2015. She used dual-indexed primers in her experiment as published in https://doi.org/10.1186/2049-2618-2-6 (Fadrosh, D.W., Ma, B., Gajer, P. et al.). I didn’t try QIIME 2 for the whole preprocessing steps because it was unclear to me whether it’s easy to do. Instead, I used joined paired-ends generated by QIIME 1.9 (that is, a trimmed seqs.fna). Here’s my setup: % conda --version conda 4.8.1 %…

This sounds remarkably similar to the symptoms you report.

[colinbrislawn (Colin J Brislawn)]

Long thread!

Have you been able to confirm that the worker nodes are running near 100% CPU usage after running for 595 hours? This sort of mysterious slowdown has happened to me; I start the process, everything is going full speed… then the database uses up all my ram and search speed drops to zero.

reads	hours	reads/h
3956736	1	3.9 M
11327844	17	.66 M
89199952	595	.14 M

This is exactly the kind of speed falloff I would expect due to RAM being fully used. I just want to make sure you have plenty of RAM before we move on.

If you ssh into that the node running vsearch for 595 hours, I’ll bet you that CPU usage is at 4% and you have 0.1 avail Mem

---

I think vsearch will throw error if tmp is used up, but if RAM is used up it will limp along without throwing an error.

[MichelaRiba (michela)]

@colinbrislawn

|reads|hours|reads/h|

|3956736|1|3.9 M|
|11327844|17|.66 M|
|89199952|595|.14 M|

Hi, thanks a lot again.
I am going to speak again with our System Administrator,
anyhow I would like to share that regarding tmp directory I can see

4.0K Feb 3 14:33 RtmpUnzDQd
2.0K Feb 13 11:19 tmpBDWTAW
2.0K Feb 13 11:19 tmpCaDSuc
2.0K Feb 13 11:19 tmpHBacsu
2.0K Feb 13 11:21 tmpVgwmMf
1.8K Feb 13 11:21 tmprTdcpB
2.0K Feb 13 11:23 tmp1nb1bV
1.8K Feb 13 11:23 tmp_jNVK8
2.0K Feb 13 11:23 tmpHUpmzB
2.0K Feb 13 11:26 tmpC0mgD5
2.0K Feb 13 11:27 tmpPCds0r
2.0K Feb 13 11:28 tmpWGibzD
2.0K Feb 13 11:29 tmp7BufKa
2.0K Feb 13 11:42 tmpVeL6zt
2.0K Feb 13 13:25 tmp6D2GR7
2.0K Feb 13 13:25 tmpkYASJo
2.0K Feb 13 13:26 tmpt7fF3a
2.0K Feb 13 13:31 tmpbwwhc8
1.8K Feb 13 13:32 tmpW4zVI1
4.0K Feb 13 13:33 qiime2-archive-d1d89k9_
4.0K Feb 13 13:35 qiime2-archive-f8wfe4yx
4.0K Feb 13 13:35 qiime2-provenance-wum623za
4.0K Feb 13 13:35 qiime2-archive-e2xkrnn_
3.9G Feb 13 14:09 tmpb4nhk1c4
715M Feb 13 15:19 tmplp_tfrhn
79M Feb 13 15:19 tmpar7ddpzp
78M Feb 13 15:19 q2-DNAFASTAFormat-grxixobv
0 Feb 13 15:43 q2-DNAFASTAFormat-3a4c6h1g

Those are all tmp files produced by vsearch closed reference (just following the suggestion to try closed reference).
The last file is empty from 4 days, this sound s pretty strange to me indeed. I do not have errors because I have error report only at the end of the job…
For sure the search clustering finished pretty fast and went in parallel (as we already discussed) using 12 threads in 12 cores, while the following phase, namely database mapping, which I see as “qiime” if I use top command in the node, please see afterwards, I see no parallel going (as discussed before, so fine) and the following reports in top command:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16270 mriba 20 0 12.8g 12g 31m R 99.9 1.2 5555:13 qiime

The fact that the tmp file is empty maybe refers to a problem of tmp space, as you suggested, while regarding to memory I am not able to understand if the value I see is the available memory and this is sufficient or is limiting

thanks again

Michela

[colinbrislawn (Colin J Brislawn)]

Hello Michela,

I think you are correct; this process might be progress slowly due to one of the non-parallized steps. Your System Administrator team might be able to find other clues about where and why this process is stalling.

We may have covered this already, but I want to make sure to mention another solution to your problem: using another denoising method, like DADA2.
DADA2 was designed big data, and might be a better fit for your data set than VSEARCH.

As a bonus, dada2 produces high resolution ASVs, which are arguably better than OTUs!

Colin

[ismailp]

@colinbrislawn

I think vsearch will throw error if tmp is used up, but if RAM is used up it will limp along without throwing an error.

Sorry, I don’t want to mix my own business into this thread, but I believe it’s exactly the opposite. See xmalloc fails are fatal. Return values of file system functions, however, are not handled in vsearch, except fopen. It also checks out with my own experience with full /tmp partition while running vsearch.

[colinbrislawn (Colin J Brislawn)]

Thanks for jumping in! I appreciate your insight into this

To the /tmp/ drive!

Colin

P.S. Does xmalloc fail when allocating ram disk, or would it only fail when there is no memory left of any kind?

[MichelaRiba (michela)]

Hi,

thanks a lot, I would check when our cluster will be up again

[MichelaRiba (michela)]

Thanks a lot for the additional suggestion, I will check when our cluster would be up

Michela

[ismailp]

You’re welcome. xmalloc uses a platform-specific aligned allocation function. In both POSIX and Windows case, allocation functions try to request heap space from the operating system. I am not so familiar with Windows. Allocation fails if the operating system says there is no memory. That often means there is no memory left of any kind. There are many reasons why this might fail, including no physical memory left and swapping is disabled, or swapping is enabled but there is no free swap space, user has exceeded their quota, or memory fragmentation (although there is memory available, there is no contiguous block of requested size).