Error in feature-table rarefy with imported table

Technical Support
,
1

Hello-

I'm having trouble running qiime feature-table rarefy from a count table I have imported into qiime.

The table was a .tsv file converted into .biom format by:

biom convert -i otu-dada2-min5in5-combat.tsv -o otu-dada2-min5in5-combat. biom --table-type="OTU table" --to-hdf5

I added "#OTU ID" to the first column name as per: convert txt. into biom file · Issue #763 · biocore/biom-format · GitHub

The commands I am running in qiime to then load the .biom file and rarefy the counts:

module load miniconda/3
source activate /project/crml-bio/.conda/envs/qiime2-2019.7

qiime tools import --input-path ./otu-dada2-min5in5-combat.biom
--type 'FeatureTable[Frequency]'
--output-path feature-table-dada2-min5in5-combat.qza

qiime feature-table rarefy
--i-table ./feature-table-dada2-min5in5-combat.qza
--p-no-with-replacement
--o-rarefied-table ./feature-table-dada2-min5in5-combat-thenRarefied10K.qza
--p-sampling-depth 10000
--verbose

The error that is generated:

Traceback (most recent call last):
File "/project/crml-bio/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py", line 327, in call
results = action(**arguments)
File "</project/crml-bio/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-305>", line 2, in rarefy
File "/project/crml-bio/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 240, in bound_callable
output_types, provenance)
File "/project/crml-bio/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 383, in callable_executor
output_views = self._callable(**view_args)
File "/project/crml-bio/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_feature_table/_normalize.py", line 15, in rarefy
with_replacement=with_replacement)
File "/project/crml-bio/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/biom/table.py", line 2809, in subsample
_subsample(data, n, with_replacement)
File "biom/_subsample.pyx", line 60, in biom._subsample._subsample
File "<array_function internals>", line 6, in repeat
File "/project/crml-bio/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 481, in repeat
return _wrapfunc(a, 'repeat', repeats, axis=axis)
File "/project/crml-bio/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 61, in _wrapfunc
return bound(*args, **kwds)
ValueError: repeats may not contain negative values.

Plugin error from feature-table:

repeats may not contain negative values.

Thinking this error might be indicating I have negative counts in my table, I checked, and I do not.

My OTU table looks correct after creating the .biom file -> loading into qiime -> extracting table as .tsv, so I'm not sure it's that. I'm a little stuck on what to try next.

I'd appreciate any help! Thank you!

2

Welcome to the forum @winzietracy!

Maybe to start, could you run qiime tools validate feature-table-dada2-min5in5-combat.qza and let us know the output?

that message could be a red herring... sounds like maybe there is a different issue with the format, quite possibly because you have done something "unusual" before importing?

Did you do batch effect normalization with combat?

I have not used combat but my foggy recollection is that this may produce negatives, floats, and other quirks (unless if I am thinking of another batch effect procedure designed for microarray data)... all of this is probably incompatible with the FeatureTable[Frequency] type, which is intended for count data.

So please tell us more about the contents of your feature table and we will see how we can get this table into QIIME 2...

3

Hi Nick,

Thanks for getting back to me, and apologies for my delay in response.

Running:

qiime tools validate feature-table-dada2-min5in5-combat.qza
Result feature-table-dada2-min5in5-combat.qza appears to be valid at level=max.

Did you do batch effect normalization with combat ?

Yes- I did. I don't see any zeros in data set. There are decimals, though- perhaps Qiime2 requires integer counts? I'll try rounding my table and see if that changes anything; will report back.

Thanks!
Lindsay

4

I don't think the decimal values are the problem, rather, it is the negative values that are present:

I know you mentioned above that you don't have negative counts, but.... are you sure? Maybe you could share these results (in a DM if you don't want to publicly post).

5

Hi! Thanks for getting back to me.

I have solved the issue it seems. It was a stupid mistake on my part- the problem was related to a log transformation I took prior to to running combat, and then undoing the log transformation post combat. It seems I wasn't actually taking a log like I thought prior to running combat, and so when I undid the log, I created giant numbers in my dataframe. After I caught this, then I was able to rarefy the table in qiime2 without a problem.

There were no negative values in the table, checked in R by:

min(df.combat)
max(df.combat)
is.numeric(df.combat)

I appreciate your guidance- sorry this was just a stupid mistake on my end that I didn't notice. I am glad I noticed it now, however. It was a big mistake!

Lindsay

2 Likes
6

Thanks for the follow-up!

Looks like this is how you wound up with negative numbers! If you take a really large value and cast to a 32-bit int you can wind up with negative values. This is known as "integer overflow." Mystery solved, good sleuthing!

1 Like
7

Looks like this is how you wound up with negative numbers! If you take a really large value and cast to a 32-bit int you can wind up with negative values. This is known as “integer overflow.”

Oh wow- well that makes sense! I was so confused by this error. Thanks a bunch!

1 Like