PID TTY TIME CMD
14775 pts/26 00:00:00 bash
15368 pts/26 00:00:08 qiime
15373 pts/26 04:08:57 R
15403 pts/26 00:02:40 R
22411 pts/26 00:00:00 ps
I am processing approximately 4M reads, and I have let it run for approximately 2 days in CPU time. Looking at some documentation, it seems like this should only take 2-10 CPU hours, so I’m wondering if this other R process (15373) is waiting on the defunct R process.
It looks like you are backgrounding your job (via the ampersand at the end of the command) - is it possible that the process is being orphaned when logging out of the compute node? Maybe try again without backgrounding --- use a job queue if available, otherwise try screen or tmux.
Is it possible that running it in parallel is causing this?
I allowed it to run on a single processor overnight, backgrounded (&), and it completed successfully. This is great (it is not necessarily an immediate issue now), but I would like to be able to throw more power at this if I have larger data sets that need it.
Hmm, I don't think so, but if you wanted to pursue that you should check out the DADA2 issue tracker
More likely, in my opinion, is that the backgrounding is to blame here, and something else was different between your 12-thread and 1-thread runs, besides the thread count (for example, your shell session did end vs didn't end).
FWIW, I generally work with >1 threads from my own analyses and have had no issues, although I am using a job scheduler. I think I have generally observed success with multiple threads from other users here on the forum too, so I suspect that might be a bit of a red herring...
