I’m working with 625 MiSeq samples & trying to use vsearch to cluster against a Silva 132 16S database. There is ~30 GB RAM + 55 GB VM, & the process is running on a 2nd mounted 4 TB drive (instead of on the main SSD which is largely handed over to VM). The process chugged along for 9 days (initially using all 8 cores), then abruptly stopped. The kernel log shows:
[755528.607932] Out of memory: Kill process 10717 (qiime) score 929 or sacrifice child
[755528.607934] Killed process 10717 (qiime) total-vm:82150232kB, anon-rss:28305832kB, file-rss:16kB, shmem-rss:0kB
The input file has already been dereplicated. I could cluster against a lesser database (Greengenes), or I could go back & split the sequences into 2 smaller sets (they represent water samples & biofilm samples). Obviously, not my 1st choices but maybe necessary. Any suggestions are welcome, & thanks for the great help you have already provided!