Error running QIIME2 studio - collapsing metadata

HI all.
In the 2019.10 release a bug was “fixed” in Qiime2 studio - " fixed a bug that prevented MetadataColumn values from being decoded properly - this was mostly an issue for q2studio". I have just had a chance to look at this fix, using the latest version of Q2S and while the initial metadata collapse seems to have been fixed and the appropriate feature table is produced, downstream analysis of this table i.e. to plot taxa is not possible.

It seems as if the command is looking for the collapsed metadata groups to appear in the Sample ID column -
"if ids_not_in_metadata:
raise ValueError(‘Sample IDs found in the table are missing in the ’
f’metadata: {ids_not_in_metadata!r}.’)

Hopefully not just an error on my part.

Hello @bmurph79,

Welcome back to the forums! :qiime2:

Can you post the command you ran and post the input artifact? We can try it and see what's going wrong. Also, are you using 2020.2 or are you still on 2019.10?

Colin

HI Colin. Apologies for the delay. Using 2020.2. Command wise, I am running the analysis through Qiime2 Studio, a blessing for us not coding people.

In terms of the feature table, I would not like to pass on the one I am using, confidential info, however I have repeated the error with a feature table from the moving pictures tutorial.

  • Use starting feature table
  • Collapse using “Group samples or features by a metadata column” using the “BodySite” column.
  • Attempted to plot a taxa bar plot on this feature table, to produce the following error message
    : ValueError: Sample IDs found in the table are missing in the metadata: {‘right palm’, ‘left palm’, ‘tongue’, ‘gut’}.
1 Like

Hello bmurph!

Thanks! Here is the main error:

I bet those sample IDs were changed during the Group samples by “BodySite" command. For this to work, you have to collapse both your feature table AND also collapse your metadata table. Because the metadata table is not collapsed, the names don’t match.

Let me know if you need help collapsing your metadata table to get matching names.

Colin

Hi Colin. Understand your point but collapsing of the metadata table was never needed before. The same data table / mapping file was able to be used pre and post collapsing based on a metadata column.

1 Like

Hi @bmurph79 -

I’m not sure I understand - q2-taxa has never been able to generate a barplot where the sample IDs don’t match, which is what happens when you collapse or group a table - you change the IDs from one thing to another thing (in this case BodySite). This design is deliberate - how will QIIME 2 know which Sample ID in your metadata belongs to which grouped “Sample ID” in your table? There is no guarantee that the same metadata is going to be used.

Can you elaborate more on the case that you have in mind where this worked for you?

1 Like

HI Matthew. In previous iterations I was able to collapse based on a metadata field, say “body site” and then using this collapsed feature table it would be possible to e.g. plot a taxonomic bar plot with the initial mapping file. The bar plot would then have, in this example, a single bar for each of the 4 body sites. To make sure I was not losing it I went back to an old training video I made to show users how to using Q2S and that’s definitely the case (ill see if I can upload the video is you forgive the commentary). It was a similar story for collapsed tables pre diversity analysis.

Note: When I say “collapse” I am referring to “Group samples or features by a metadata column”.

Thanks for the info, @bmurph79! I would love to see this video - I think its possible that you found a bug, and were possibly able to take advantage of that bug, so we should fix that up (if it hasn't been fixed already). I am the lead developer of q2-taxa, the plugin for making these bar plots, and the "rules" I mentioned above have been in place since 2016, as best I recall, so if you found a way around it, its very likely a bug. Just to clarify though, as I mentioned above, the Sample IDs in the table need to match the Sample IDs in the metadata - its totally possible to do that post-grouping (or collapsing), but requires you to manually intervene to make the metadata work. For example, if I had metadata like this:

sample-id body-site experimental-group
s1 gut case
s2 gut control
s3 tongue case
s4 tongue control

If I grouped on body-site (new sample IDs: gut, tongue), how do I handle "grouping" the experimental-group column? Maybe I just discard it, or, not. QIIME 2 can't really "know" what to do there, which is why its up to you to make the grouped metadata. Make sense?

So, a workflow for grouping and creating barplots should look like this:

# first, group the table
qiime feature-table group \
  --i-table table.qza \
  --p-axis sample \
  --p-mode sum \
  --m-metadata-file sample-metadata.tsv \
  --m-metadata-column body-site \
  --o-grouped-table grouped-table.qza

# next, generate a new sample metadata file, 
# this can be done by hand, or automated somehow.
# I will opt for scripting:

# first, transpose the table to get the sample IDs on the "other" axis
qiime feature-table transpose \
  --i-table grouped-table.qza \
  --o-transposed-feature-table grouped-transposed-table.qza

# then, initialize a new sample metadata file:
echo "sample-id" > grouped-sample-metadata.tsv

# insert the new grouped sample IDs
qiime tools inspect-metadata grouped-transposed-table.qza --tsv | awk -F '\t' '{if (NR!=1) {print $1}}' >> grouped-sample-metadata.tsv

# look at the new metadata just to make sure all is well:
cat grouped-sample-metadata.tsv

# sample-id
# gut
# left palm
# right palm
# tongue

# create bar plot
qiime taxa barplot \
  --i-table grouped-table.qza \
  --i-taxonomy taxonomy.qza \
  --m-metadata-file grouped-sample-metadata.tsv \
  --o-visualization grouped-bar-plots.qzv

And, here is the grouped barplot:

Does that make sense?

Thanks for reporting! Keep us posted! Looking forward to seeing that training video!

Thanks Matthew. Makes sense, but when a bug is useful, is it still a bug :-). In your example above, I woudl have a new column “Body-site_Exp-Group”. I was doing all of this in Q2S so maybe that’s why it slipped the net. Happy to send the video on, just cannot load here. If you ping me a mail address, happy to pass on.

1 Like

Yep, I hear yah (and IMO - yes! its still a bug!). We have open issues on this plugin to make metadata (and taxonomy) optional for this barplot visualization, which would provide a sensible alternative. If you didn’t need to provide sample (or feature) metadata, then there is no “ID mismatch” business to deal with.

Contributions would be very welcome here, so let me know when you’re ready to tackle this one!

I’ll send you a DM regarding the video, thanks so much!

1 Like