feature-table rename-ids error: "an issue with retrieving column 'ID' from the metadata"

[wburgess]

feature-table rename-ids error: "an issue with retrieving column 'ID' from the metadata"

I need help understanding precisely how qiime feature-table rename-ids works, especially its metadata file and column parameters. Or maybe I'm using the wrong tool, and would welcome advice on a better approach. I've searched already (Merging Error Table and RepSeq was useful but not directly solving if I read it right); I've read the glossary; I'm running qiime2-2021.2 through conda.

My feature tables are full of ids that are 36-character-long alphanumeric codes (e.g., "cc7f7cd6d961f50ae6d57a7513b2818e") and I aim to replace these codes with their more intelligible taxonomy labels (e.g., "d__Bacteria; p__Proteobacteria; c__Gammaproteobacteria; o__Gammaproteobacteria_Incertae_Sedis; f__Unknown_Family; g__Acidibacter; s__uncultured_eubacterium"). Tried writing my own script to replace the 36-character codes with just the associated "s__uncultured_eubacterium" (or whatever the most specific taxon is), but such a label is not unique and cannot serve as metadata ID. Tweaked the script so I appended the long code to the useful label, and edited so it's all keemei-approved (36 approvable characters), resulting in the above example being something like "s-uncultured-eubacterium-cc7f7cd6d9". This replacement for a whole feature table truly seems impossible/impractical manually, but should be easy with cli powers ... I just need superior expertise.

I have an unintelligible feature table. I have a metadata table consisting of one column, labeled 'ID'. I run

qiime feature-table rename-ids --i-table deblur_output/deblur_table_final.qza --m-metadata-file meta_renaming --m-metadata-column 'ID' --o-renamed-table renamed_table

and find that my result is

(1/1) Invalid value for '--m-metadata-file': There was an issue with retrieving column 'ID' from the metadata.

I find this counter-intuitive to trouble-shoot. It also doesn't work without quotation marks around 'ID' in the metadata column parameter.

Following some thinking from Merging Error Table and RepSeq, I also tried

qiime metadata tabulate --m-input-file deblur_output/deblur_table_final.qza --m-input-file meta_renaming --o-visualization merged.qzv --verbose

to see how it worked out and got

`Traceback (most recent call last):
File "/home/cory/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2cli/click/option.py", line 182, in type_cast_value
return value[0].merge(*value[1:])
File "/home/cory/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/metadata/metadata.py", line 694, in merge
"Cannot merge because there are no IDs shared across metadata "
ValueError: Cannot merge because there are no IDs shared across metadata objects.

There was an issue with merging QIIME 2 Metadata:
Cannot merge because there are no IDs shared across metadata objects.
See above for debug info.`

This error is clear, but making the IDs be the same is precisely what I'm trying to do here. (This also doesn't work with a transposed table, just in case that mattered.)

I have attached all files referred to: my feature table (deblur_table_final.qza), the taxonomy.tsv table linking codes to taxonomy (taxonomy.tsv), my (keemei-approved) metadata file (meta_renaming.txt), my representative sequences (rep_seqs_final.qza), and even the horrifying script I kludged up to grab more-useful labels from taxonomy.tsv (renaming.txt).
deblur_table_final.qza (150.5 KB)
taxonomy.tsv (2.4 MB)
meta_renaming.txt (596.5 KB)
rep_seqs_final.qza (151.0 KB)
renaming.txt (489 Bytes)

As always, I've tried to be both complete and concise. Sorry for any obvious failures. And my thanks to anyone who can help.

[Edit to move a tick mark.]

[wburgess]

Under the heading of esprit d'escalier: did I miss a very obvious way of doing this? Could I (or someone more skilled than I) write a script to find-replace or otherwise modify a feature table with the taxonomic labels instead of the 36-character codes and then import the table into QIIME?

[thermokarst]

Hi there @wburgess!

@wburgess

and find that my result is

(1/1) Invalid value for '--m-metadata-file': There was an issue with retrieving column 'ID' from the metadata.

I find this counter-intuitive to trouble-shoot. It also doesn’t work without quotation marks around ‘ID’ in the metadata column parameter.

The issue here is that this ID is a special name in QIIME 2 metadata, reserved for the sample (or in your case, feature) IDs. If you run qiime metadata tabulate on this one file you should see what I mean. So yeah, the error might seem a bit confusing, but its kinda the truth, there isn't a QIIME 2 MetadataColumn in that file with the name "ID" (there is however an ID column named "ID").

You can read more about the QIIME 2 metadata spec here: Metadata in QIIME 2 — QIIME 2 2021.4.0 documentation

@wburgess

I also tried

qiime metadata tabulate --m-input-file deblur_output/deblur_table_final.qza --m-input-file meta_renaming --o-visualization merged.qzv --verbose

to see how it worked out and got

`Traceback (most recent call last):
File “/home/cory/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2cli/click/option.py”, line 182, in type_cast_value
return value[0].merge(*value[1:])
File “/home/cory/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/metadata/metadata.py”, line 694, in merge
"Cannot merge because there are no IDs shared across metadata "
ValueError: Cannot merge because there are no IDs shared across metadata objects.

This is a separate error, but very much related to the same idea around IDs. You're trying to merge Sample IDs from your feature table, and Feature IDs from your metadata file. This error makes sense here: there really isn't any overlap in the IDs, right?

To get you moving forward, the thing that you're missing is a proper ID column in meta_renaming.txt - that ID column will need to have the same Feature IDs found in your feature table. The easiest way to do that is with the qiime taxa collapse command, which handles collapsing taxa (multiple ASVs can all classify to the same taxon):

# note, I had to reimport your taxonomy data, since you provided
# the exported data, not the original qza

qiime taxa collapse \
  --i-table deblur_table_final.qza \
  --i-taxonomy taxonomy.qza \
  --p-level 7 \
  --o-collapsed-table taxa-table.qza

qiime feature-table summarize \
  --i-table taxa-table.qza \
  --o-visualization taxa-table.qzv

I hope that helps!

[wburgess]

Thanks so much for your promptness and clarity! I took a few initial steps and it seems promising. These paragraphs (and subsequent code) were especially helpful:

This is a separate error, but very much related to the same idea around IDs. You’re trying to merge Sample IDs from your feature table, and Feature IDs from your metadata file. This error makes sense here: there really isn’t any overlap in the IDs, right?

To get you moving forward, the thing that you’re missing is a proper ID column in meta_renaming.txt - that ID column will need to have the same Feature IDs found in your feature table. The easiest way to do that is with the qiime taxa collapse command, which handles collapsing taxa (multiple ASVs can all classify to the same taxon)....

Apologies for a suboptimal table.qza.

I'll carry forward and update.

[wburgess]

Yeah, this here was a dramatic step forward. Thanks again!