BIOM file Validation, QIIME1 vs QIIME2

[adinasarapu (Ashok)]

My biom files were validated using http://biom-format.org package.

With QIIME1, I have a valid biom file but with (same data) QIIME2 exported (qiime tools export) invalid biom file!

A) otu_table.biom obtained with QIIME1.9

$biom validate-table -i otu_table.biom

The input file is a valid BIOM-formatted file.

B) feature-table.biom obtained with QIIME2-2017.4

$biom validate-table -i feature-table.biom

Unknown BIOM type:
The input file is not a valid BIOM-formatted file.

Why this difference?

Sincerely,
Ashok

[ebolyen (Evan Bolyen)]

Hey @adinasarapu,

That does seem strange, but it’s possible you have an older BIOM package that isn’t aware of BIOM file-format 2.1.

What environment are you running these commands in? QIIME 1 or QIIME 2?

For both environments, could you provide the following:

• What does which biom say?
• What does python -c "import biom; print(biom.__version__)" say?

Additionally:

• Can you also run file otu_table.biom and file feature-table.biom to see what the underlying formats are (e.g. JSON or HDF5), what does it say?

Thanks!

[adinasarapu (Ashok)]

I have the following output

$which biom
~/anaconda2/envs/qiime2-2017.4/bin/biom

$python -c "import biom; print(biom.__version__)"
2.1.5

QIIME2

$file feature-table.biom
feature-table.biom: Hierarchical Data Format (version 5) data

$biom validate-table -i feature-table.biom
Unknown BIOM type:
The input file is not a valid BIOM-formatted file.

QIIME1

$file otu_table.biom
otu_table.biom: Hierarchical Data Format (version 5) data

$biom validate-table -i otu_table.biom
The input file is a valid BIOM-formatted file.

[ebolyen (Evan Bolyen)]

Thanks @adinasarapu!

Would you be able to share feature-table.biom? Everything looks fine from the outside. QIIME 2 uses the biom API directly so I’m very uncertain how we would have ended up with an “invalid” file. (I wonder if the file became corrupted on disk during write somehow.)

@wasade, have you seen anything like this before?

[wasade (Daniel McDonald)]

@ebolyen, I haven’t, sorry. I’d be very curious to see the feature-table.biom file though. Note that --detailed-report can be passed to biom validate-table to get more specific information about what is invalid

[adinasarapu (Ashok)]

$biom validate-table --detailed-report -i feature-table.biom
Validating BIOM table…
Validating ‘format-url’…
Validating ‘format-version’…
Validating ‘type’…
Unknown BIOM type:
Validating ‘shape’…
Validating ‘nnz’…
Validating ‘generated-by’…
Validating ‘id’…
Validating ‘creation-date’…
Validating ‘shape’ versus number of samples and observations…
The input file is not a valid BIOM-formatted file.

[wasade (Daniel McDonald)]

Thanks! What is the provenance of the QIIME2 feature_table.biom? That is technically out of spec – thank you for catching it – but I’m not aware of any downstream tool which actually leverages that format attribute.

@ebolyen, my guess is type is not being passed to biom.Table somewhere.

[ebolyen (Evan Bolyen)]

It looks like we don’t set it at all. In later releases of BIOM it looks like there is a special codepath for "" which will give you:

Unknown table type, however that is likely okay.

Which seems fine for now since there isn’t really a good table type for QIIME 2 here.