# Differential abundance analysis

**URL:** <https://forum.qiime2.org/t/differential-abundance-analysis/15851>\
**Category:** User Support\
**Created:** [July 25, 2020, 11:56am UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851 "2020-07-25T11:56:15Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![M\_F](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/5fc32e/32.png) [@M\_F](https://forum.qiime2.org/u/M_F)\
**Post date:** [July 25, 2020, 11:56am UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/1 "2020-07-25T11:56:15Z")

</div>

Dear all,

I performed Taxonomy classification and taxonomic analyses referring to hitdb. I performed a differential abundance analysis using ancom and i received this message :  
Plugin error from composition:

Unable to allocate 82.0 TiB for an array with shape (3357899, 3357899) and data type float64

Debug info has been saved to /tmp/qiime2-q2cli-err-1fcfq\_sh.log

Thanks

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [July 27, 2020, 9:10pm UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/2 "2020-07-27T21:10:13Z")

</div>

Hi @M_F!

> [@M\_F](#):
>
> How can i convert HITdb\_taxonomy\_qiime2.qza file to table.qza to achieve this task ?

I'm not really sure what you're trying to do here, can you please give us more details? Also, filenames don't really mean anything in QIIME 2, since you can name a file anything you want, so it would be helpful if you kept that in mind when elaborating on your description a bit more. Thanks!

---

<div class="post-metadata">

**Author:** ![M\_F](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/5fc32e/32.png) [@M\_F](https://forum.qiime2.org/u/M_F)\
**Post date:** [July 28, 2020, 10:54am UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/3 "2020-07-28T10:54:56Z")

</div>

Hello @thermokarst

Thanks for the reponse. Sorry the question is wrong. I performed a differential abundance analysis using ancom and i received this message :  
Plugin error from composition:

Unable to allocate 82.0 TiB for an array with shape (3357899, 3357899) and data type float64

Debug info has been saved to /tmp/qiime2-q2cli-err-1fcfq\_sh.log

What can i do ? Thanks for your help

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [July 30, 2020, 4:29pm UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/4 "2020-07-30T16:29:31Z")

</div>

Thanks for clarifying, @M_F.

The error message is here:

> [@M\_F](#):
>
> Unable to allocate 82.0 TiB

You have a _huge_ amount of features in your `FeatureTable`: 3,357,899. ANCOM is crashing because the amount of memory necessary for that calculation is 82.0 TiB. That kind of memory is basically impossible to come by on a single machine:

> **[What is the most RAM a computer has ever had?](https://www.quora.com/What-is-the-most-RAM-a-computer-has-ever-had?share=1)**
>
> Answer (1 of 33): Determining the single computer with the "most RAM ever" depends on various factors and interpretations. There are two main categories to consider:
> 
> FREE Preparation: Open Google, search for " Computer MCQTUBE " and visit mcqtube...

2 TB of RAM on a single computer is about as good as it gets right now, so realistically there is no way for you to perform ANCOM on 3.3 million features.

> [@M\_F](#):
>
> What can i do ?

- Filter your table: [Filtering data — QIIME 2 2020.6.0 documentation](https://docs.qiime2.org/2020.6/tutorials/filtering/)
- Collapsing your table: [“Moving Pictures” tutorial — QIIME 2 2020.6.0 documentation](https://docs.qiime2.org/2020.6/tutorials/moving-pictures/#differential-abundance-testing-with-ancom) (see the commands in the second half of this section)

Keep us posted! 🦖

---

<div class="post-metadata">

**Author:** ![M\_F](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/5fc32e/32.png) [@M\_F](https://forum.qiime2.org/u/M_F)\
**Post date:** [August 3, 2020, 11:26am UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/5 "2020-08-03T11:26:32Z")

</div>

@thermokarst

Thank you so much for the response.  
i run the job sample per sample.  
I tried to filter and collapse the table, the job was run with success. However actually i have a problem to run the ancom test with the command --m-metadata-column  
qiime composition ancom   
--i-table comp-gut-table-l6.qza   
--m-metadata-file sample-metadata.tsv   
--m-metadata-column Disease   
--o-visualization l6-ancom-subject.qzv  
i received this message when i used the aforementioned command line  
Plugin error from composition:

All values in `grouping` are unique. This method cannot operate on a grouping vector with only unique values (e.g., there are no 'within' variance because each group of samples contains only a single sample).

Debug info has been saved to /tmp/qiime2-q2cli-err-axqj1bma.log

In the table i have a column called Disease (Healthy vs Affected) and i would like that ancom test will be performed for SAM.1 since i run the job for each sample separately.  
What can i do to run Ancom test for the column Disease taking into consideration the data specific to only SAM.1 or sample 1 in the first row ?  
Thanks a lot

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [August 4, 2020, 3:05pm UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/6 "2020-08-04T15:05:23Z")

</div>



---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [August 5, 2020, 3:33pm UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/7 "2020-08-05T15:33:34Z")

</div>

Hi @M_F.

You can't run ancom on a single sample, because then there will only be a single possible value for the metadata group - make sense? That is what the error message is telling you:

> [@M\_F](#):
>
> All values in `grouping` are unique. This method cannot operate on a grouping vector with only unique values (e.g., there are no ‘within’ variance because each group of samples contains only a single sample).

The "grouping" in this case is your `Disease` column. If there is only one sample, then there can only be one value, which means that all the values in that column are unique.

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [August 5, 2020, 3:33pm UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/8 "2020-08-05T15:33:36Z")

</div>



---

<div class="post-metadata">

**Author:** ![M\_F](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/5fc32e/32.png) [@M\_F](https://forum.qiime2.org/u/M_F)\
**Post date:** [August 10, 2020, 11:26am UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/9 "2020-08-10T11:26:22Z")

</div>

Dear @thermokarst

Thank you for the message. I filtered the table and i performed differential abundance analysis using the following command lines

1. qiime feature-table filter-samples --i-table table.qza --m-metadata-file samples-to-keep.tsv --o-filtered-table H-AFF-filtered-table.qza

2. qiime composition add-pseudocount --i-table H-AFF-filtered-table.qza --o-composition-table H-AFF-comp-table.qza

3. qiime composition ancom --i-table H-AFF-comp-table.qza --m-metadata-file samples-to-keep.tsv --m-metadata-column Disease --o-visualization H-AFF-ancom-Disease.qzv

i received this message

Plugin error from composition:

Unable to allocate 43.5 TiB for an array with shape (2445046, 2445046) and data type float64

Debug info has been saved to /tmp/qiime2-q2cli-err-ntm81n4x.log

how can i analyze this huge amount of data i filtered the table and it seems that the problem remains always, i need all those information ?

Thanks

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [August 12, 2020, 7:47pm UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/10 "2020-08-12T19:47:26Z")

</div>

Hi @M_F!

> [@M\_F](#):
>
> how can i analyze this huge amount of data i filtered the table and it seems that the problem remains always, i need all those information

You will need to filter out a lot more than half the data - like I said, 2 TB of RAM is really about as good as it gets these days, so your target is \<2TB.

Taking a step back, I am pretty skeptical that you will even be able to discern any kind of meaningful signal out a dataset this large. Surely you can reduce the size of the scope somehow, right?

---

<div class="post-metadata">

**Author:** ![system](https://forum-qiime2-org.s3.dualstack.us-west-2.amazonaws.com/original/3X/2/1/21af5fe23cb6f4579467c66a9ed94e55274ca7bd.svg) [@system](https://forum.qiime2.org/u/system)\
**Post date:** [September 13, 2020, 1:47am UTC](https://forum.qiime2.org/t/differential-abundance-analysis/15851/11 "2020-09-13T01:47:30Z")

</div>

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.
