qiime2/2019.10 composition ANCOM

Hello everyone,

I have searched this forum to see if others have faced exact same problem but couldn’t find any resources. I am running the following command but get an error that I need help with to correct it.

My questions are:

  1. Does this has anything to do with the memory? The requested memory was 20 gb (qsub -l nodes=1:ppn=4,mem=20gb,walltime=1:00:00 -q interactive -I).

  2. Could this be due to some unusual things in the table?

Thank you for your time!
Irshad

[email protected] [~/rawdata/16srna/2shi7_learning/fastqs_stitched] % qiime composition ancom --i-table dada2-single-end-composition-table.qza --m-metadata-file metadata.tsv --m-metadata-column category --p-transform-function clr --p-difference-function mean_difference --o-visualization ancom-single-end-plots.qzv \

–verbose
/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/scipy/stats/stats.py:3235: RuntimeWarning: invalid value encountered in double_scalars
f = msb / msw
Traceback (most recent call last):
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/q2cli/commands.py”, line 328, in call
results = action(**arguments)
File “</panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/decorator.py:decorator-gen-478>”, line 2, in ancom
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 240, in bound_callable
output_types, provenance)
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 445, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/q2_composition/_ancom.py”, line 92, in ancom
fold_change = transformed_table.apply(diff_func, axis=0)
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/pandas/core/frame.py”, line 6928, in apply
return op.get_result()
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/pandas/core/apply.py”, line 186, in get_result
return self.apply_standard()
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/pandas/core/apply.py”, line 292, in apply_standard
self.apply_series_generator()
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/pandas/core/apply.py”, line 321, in apply_series_generator
results[i] = self.f(v)
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/q2_composition/_ancom.py”, line 86, in diff_func
args = _d_func(*[x[metadata == c] for c in cats])
TypeError: (’() takes 2 positional arguments but 3 were given’, ‘occurred at index a3df0505c856a3cf1da8123abcdb7e30’)

Plugin error from composition:

(’() takes 2 positional arguments but 3 were given’, ‘occurred at index a3df0505c856a3cf1da8123abcdb7e30’)

See above for debug info.
[email protected] [~/rawdata/16srna/2shi7_learning/fastqs_stitched] % =>> PBS: job killed: walltime 3614 exceeded limit 3600

It is true that ANCOM scales quadratically with respect to the number of species - but that error message does not look like a memory error. It looks like you have 3 categories in your metadata instead of two. If this is the case, make sure to specify --p-difference-function f_statistic.

Hi Jamie,

Thank you for your help with this. I tried the following command but it returned another error.

[email protected] [~/rawdata/16srna/2shi7_learning/fastqs_stitched] % qiime composition ancom \

–i-table dada2-single-end-composition-table.qza
–m-metadata-file metadata.tsv
–m-metadata-column category
–p-transform-function clr
–p-difference-function f_oneway
–o-visualization ancom-single-end-plot.qzv
–verbose
Traceback (most recent call last):
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/q2cli/commands.py”, line 328, in call
results = action(**arguments)
File “</panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/decorator.py:decorator-gen-478>”, line 2, in ancom
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 203, in bound_callable
self.signature.check_types(**user_input)
File “/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/qiime2/core/type/signature.py”, line 354, in check_types
% (name, parameter, spec.qiime_type))
TypeError: Parameter ‘difference_function’ received ‘f_oneway’ as an argument, which is incompatible with parameter type: Str % Choices(‘mean_difference’, ‘f_statistic’)

Plugin error from composition:

Parameter ‘difference_function’ received ‘f_oneway’ as an argument, which is incompatible with parameter type: Str % Choices(‘mean_difference’, ‘f_statistic’)

See above for debug info.

The metadata file looks like the following. category represent either Pb, Ff or NA
Thank you,

sample-id category
#q2:types categorical
1.Ff.1.16S.S114.001 Ff
BLANK.001.B04.16S.S121.001 NA
9.Ff.C.16S.S115.001 Ff
20.Pb.G.16S.S128.001 Pb
15.Pb.B.16S.S136.001 Pb
25.Pb.4.16S.S117.001 Pb
24.Pb.3.16S.S140.001 Pb
16.Pb.C.16S.S139.001 Pb
3.Ff.3.16S.S122.001 Ff
21.Pb.Z.16S.S131.001 Pb
BLANK.001.C04.16S.S125.001 NA
7.Ff.A.16S.S135.001 Ff
18.Pb.E.16S.S120.001 Pb
11.Ff.E.16S.S123.001 Ff
17.Pb.D.16S.S116.001 Pb
8.Ff.B.16S.S138.001 Ff
5.Ff.5.16S.S129.001 Ff
13.F.G.16S.S130.001 Ff
4.Ff.4.16S.S126.001 Ff
6.Ff.6.16S.S132.001 Ff
2.Ff.2.16S.S118.001 Ff
“metadata.tsv” [readonly][dos] 29L, 720C

Hi @Irshad, I think @mortonjt might’ve mispoke above - the parameter value name is 'f_statistic'.

2 Likes

Hi @thermokarst and @mortonjt,

Thanks for all your efforts in navigating me through this. Unfortunately, it didn’t work. Didn’t give me error but it crashed. Could this be something with the memory (20 GB) or do I need to filter the low abundance features in my composition table?

[email protected] [~/rawdata/16srna/2shi7_learning/fastqs_stitched] % qiime composition ancom \

–i-table dada2-single-end-composition-table.qza
–m-metadata-file metadata.tsv
–m-metadata-column category
–p-transform-function clr
–p-difference-function f_statistic
–o-visualization ancom-single-end-plots.qzv
–verbose
/panfs/roc/msisoft/qiime2/2019.10/lib/python3.6/site-packages/scipy/stats/stats.py:3235: RuntimeWarning: invalid value encountered in double_scalars
f = msb / msw
=>> PBS: job killed: walltime 7223 exceeded limit 7200
Terminated

Please see this part of the message:

That is coming from your cluster scheduler, not from QIIME 2. It is saying that your time limit for the command expired - you will need to request more time. We aren’t able to provide guidance on how to do that, though, since it is unrelated to QIIME 2. Please check with your sysadmin for help.