I am conducting a meta-analysis involving 2903 studies, resulting in a table size of 107GB. I am attempting to create a phylogenetic tree this September. I've been running the process for two months on an HPC (High-Performance Computing) node with 80 cores and 250GB of RAM, using the command with --threads 78.
However, I encountered an error with the message 'Plugin error from fragment-insertion' stating that 'tree.nwk returned non-zero exit status.'
The command I used was:
qiime fragment-insertion sepp \
--i-representative-sequences KT_sequences5011.qza \
--i-reference-database sepp-refs-gg-13-8 \
--o-tree kt83_5011_tree.qza \
--o-placements kt83_5011_tree_placements.qza \
Do you have any suggestions on how to speed up the process and generate my phylogenetic tree more quickly?"
Hello @kulandai1. Can you please rerun the failing command with the
--p-debug flag and post the output here? Thank you.
Thank you for your reply, the command is running.
Following are the log file of previous run
Traceback (most recent call last):
File "/home/dell_emc/easybuild/software/QIIME2/2023.5/lib/python3.8/site-packages/q2cli/commands.py", line 468, in call
results = action(**arguments)
File "", line 2, in sepp
File "/home/dell_emc/easybuild/software/QIIME2/2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 274, in bound_callable
outputs = self.callable_executor(
File "/home/dell_emc/easybuild/software/QIIME2/2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 509, in callable_executor
output_views = self._callable(**view_args)
File "/home/dell_emc/easybuild/software/QIIME2/2023.5/lib/python3.8/site-packages/q2_fragment_insertion/_insertion.py", line 71, in sepp
File "/home/dell_emc/easybuild/software/QIIME2/2023.5/lib/python3.8/site-packages/q2_fragment_insertion/_insertion.py", line 53, in _run
subprocess.run(cmd, check=True, cwd=cwd)
File "/home/dell_emc/easybuild/software/QIIME2/2023.5/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run-sepp.sh', '/home/balamurugan.s/temp/qiime2/balamurugan.s/data/9deb2fb4-3dc2-4a2a-8cfc-95c40a964528/data/dna-sequences.fasta', 'q2-fragment-insertion', '-x', '80', '-A', '1000', '-P', '5000', '-a', '/home/balamurugan.s/temp/qiime2/balamurugan.s/data/a14c6180-506b-4ecb-bacb-9cb30bc3044b/data/aligned-dna-sequences.fasta', '-t', '/home/balamurugan.s/temp/qiime2/balamurugan.s/data/a14c6180-506b-4ecb-bacb-9cb30bc3044b/data/tree.nwk', '-r', '/home/balamurugan.s/temp/qiime2/balamurugan.s/data/a14c6180-506b-4ecb-bacb-9cb30bc3044b/data/raxml-info.txt']' returned non-zero exit status 1.
I will post the --p-debug as soon as possible
Hello @kulandai1. Apologies for the delay, I was out of office. Unfortunately that log file is not giving me a lot of information to work with. it's unlikely that
--p-debug would give me much more, but can you try that anyway and post the result here?
Currently, my best guess is that you are running out of RAM. Can you run the
free command and post the output here, additionally can you run the command
/usr/bin/time -v <your QIIME 2 command here> and post the output of that here too. That should let me verify if you are running out of RAM or not.
If you are running out of RAM your best bet is probably to lower the number of threads.