ANCOMBC Error Code -9

Technical Support
1

Hi everyone,

I have run ANCOMBC successfully on a few smaller datasets (few hundred samples) without problems. I'm now trying to run it on a large dataset (~10,000 samples) on a server with 64 GB memory using Qiime2-2023.2. It ran for ~ 24 hours and then gave me the following error:

An error was encountered while running ANCOM-BC in R (return code -9), please inspect stdout and stderr to learn more.

Does anyone know what this error is?

Command I used:

qiime composition ancombc \
  --i-table table_feces.qza \
  --m-metadata-file metadata_AGP_filtered.txt \
  --p-formula age_group \
  --o-differentials dataloaf_feces.qza

Error log:

── Attaching packages ─────────────────────────────────────── tidyverse 1.3.2 ──
✔ ggplot2 3.4.1     ✔ purrr   1.0.1
✔ tibble  3.1.8     ✔ dplyr   1.1.0
✔ tidyr   1.3.0     ✔ stringr 1.5.0
✔ readr   2.1.4     ✔ forcats 1.0.0
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
R version 4.2.2 (2022-10-31) 
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_ancombc.R --inp_abundances_path /tmp/tmp9z7qytuh/input.biom.tsv --inp_metadata_path /tmp/tmp9z7qytuh/input.map.txt --formula age_group 
--p_adj_method holm --prv_cut 0.1 --lib_cut 0 --reference_levels  --neg_lb False --tol 1e-05 --max_iter 100 --conserve False --alpha 0.05 --output_l
oaf /tmp/q2-DataLoafPackageDirFmt-vup5wyo_

Traceback (most recent call last):
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_composition/_ancombc.py", line 176, in _ancombc
    run_commands([cmd])
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_composition/_ancombc.py", line 31, in run_commands
    subprocess.run(cmd, check=True)
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_ancombc.R', '--inp_abundances_path', '/tmp/tmp9z7qytuh/input.biom.tsv', '--inp_metadata_path', '/tmp/tmp9z7qytuh/input.map.txt', '--formula', 'age_group', '--p_adj_method', 'holm', '--prv_cut', '0.1', '--lib_cut', '0', '--reference_levels', '', '--neg_lb', 'False', '--tol', '1e-05', '--max_iter', '100', '--conserve', 'False', '--alpha', '0.05', '--output_loaf', '/tmp/q2-DataLoafPackageDirFmt-vup5wyo_']' died with <Signals.SIGKILL: 9>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2cli/commands.py", line 352, in __call__
    results = action(**arguments)
  File "<decorator-gen-36>", line 2, in ancombc
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
    outputs = self._callable_executor_(scope, callable_args,
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 381, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_composition/_ancombc.py", line 40, in ancombc
    return _ancombc(
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_composition/_ancombc.py", line 178, in _ancombc
    raise Exception('An error was encountered while running ANCOM-BC'
Exception: An error was encountered while running ANCOM-BC in R (return code -9), please inspect stdout and stderr to learn more.

Thank you

3

Hi @tashoo,

Thanks for reaching out! You'll need to re-run your command with the --verbose flag enabled in order for us to examine where exactly the error occurred. Once that's run again (sorry, I know it's a long run-time just to receive another error message), please copy/paste the entire traceback in your response and we can go from there. Thanks!

1 Like
5

Hi @lizgehret,

I ran it again with the --verbose flag and here is the output. Hopefully you can make something of it. A few more details - this was run on a remote linux server that is not shared with anyone else.

── Attaching packages ─────────────────────────────────────── tidyverse 1.3.2 ──
✔ ggplot2 3.4.1     ✔ purrr   1.0.1
✔ tibble  3.1.8     ✔ dplyr   1.1.0
✔ tidyr   1.3.0     ✔ stringr 1.5.0
✔ readr   2.1.4     ✔ forcats 1.0.0
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
R version 4.2.2 (2022-10-31) 
Traceback (most recent call last):
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_composition/_ancombc.py", line 176, in _ancombc
    run_commands([cmd])
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_composition/_ancombc.py", line 31, in run_commands
    subprocess.run(cmd, check=True)
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_ancombc.R', '--inp_abundances_path', '/tmp/tmp9jcogd_i/input.biom.tsv', '--inp_metadata_path', '/tmp/tmp9jcogd_i/input.map.txt', '--formula', 'age_group', '--p_adj_method', 'holm', '--prv_cut', '0.1', '--lib_cut', '0', '--reference_levels', '', '--neg_lb', 'False', '--tol', '1e-05', '--max_iter', '100', '--conserve', 'False', '--alpha', '0.05', '--output_loaf', '/tmp/q2-DataLoafPackageDirFmt-6d_ko2v6']' died with <Signals.SIGKILL: 9>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2cli/commands.py", line 352, in __call__
    results = action(**arguments)
  File "<decorator-gen-36>", line 2, in ancombc
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
    outputs = self._callable_executor_(scope, callable_args,
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 381, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_composition/_ancombc.py", line 40, in ancombc
    return _ancombc(
  File "/home/victoria/miniconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_composition/_ancombc.py", line 178, in _ancombc
    raise Exception('An error was encountered while running ANCOM-BC'
Exception: An error was encountered while running ANCOM-BC in R (return code -9), please inspect stdout and stderr to learn more.

Plugin error from composition:

  An error was encountered while running ANCOM-BC in R (return code -9), please inspect stdout and stderr to learn more.

See above for debug info.
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_ancombc.R --inp_abundances_path /tmp/tmp9jcogd_i/input.biom.tsv --inp_metadata_path /tmp/tmp9jcogd_i/input.map.txt --formula age_group --p_adj_method holm --prv_cut 0.1 --lib_cut 0 --reference_levels  --neg_lb False --tol 1e-05 --max_iter 100 --conserve False --alpha 0.05 --output_loaf /tmp/q2-DataLoafPackageDirFmt-6d_ko2v6

Thanks for the help!

6

Hello Victoria,

Return code -9 usually means the process was canceled ('killed') by another program on the computer. This confirms it:

This usually happens defensively; the SLURM queue or host OS killed the task because it is using up all the memory, or because it has used up its time allocation.

Always nice to have your own server! Is this running a torque or slurm queue, and if so can you post your submission script?

1 Like
7

Hi @colinbrislawn,

I've been using nohup. This is the command I've been using:

nohup bash ancom_bc.sh &

The script contains the following:

#!/bin/bash

source activate qiime2-2023.2

# run ancom bc
qiime composition ancombc \
  --i-table table_feces.qza \
  --m-metadata-file metadata_AGP_filtered.txt \
  --p-formula age_group \
  --o-differentials dataloaf_feces.qza \
  --verbose

Something must be killing it because there's plenty of memory left... I just can't figure out what :sob: Thanks for the help!

1 Like
8

That script looks good to me. I'm out of ideas!

Is there a cluster admin you can ask about this? They would know what else is running on the server and what could have canceled this job.

9

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.