Cool things coming to Python 3.7 for QIIME 2

Every now and then I double check that our view type annotations are going to be compatible with future Python versions.

Good news! They will be.

In fact, it looks like runtime support for type introspection is more or less guaranteed for the near future (including Python 4). There were also a bunch of neat things which I thought others might be interested in:

In PEP 557 a new decorator has been added called dataclass which uses the type annotation to dynamically generate a class definition with common “dunder” methods like __eq__/__ne__ and __init__. This was actually referenced as a reason in some other typing-related PEPs to preserve the ability to introspect types at runtime. If you think about it, our transformers are very similar in concept to this new decorator, in that they add boilerplate code dynamically to transform data between formats/objects.

Also, these @dataclass classes themselves would be a nicer way to handle our lists/dicts of <Thing>Record which we have so many of inside the SDK (right now they are all collections.namedtuple instances).

In PEP 568 context variables are an async analogue to threading.local. This could be a good way to implement ctx configuration (used behind the scenes) and warrants a deeper look.

A more dramatic change in PEP 563 makes __annotations__ a string if you import from __future__ import annotations. This will become default behavior in Python 4. This might require some changes in our codebase, but it is ultimately entirely compatible with what we are doing. The worst that would happen is we would use typing.parse_type_hints to eval the strings in the __annotations__ dictionary into classes once more (according to the PEP, we’re supposed to be doing that right now, oops). Incidentally, our transformation API technically does exact matching, so we would really only be impacted if we wanted to do better things with the typing module.

Speaking of which! In PEP 560, typing is overhauled to use __class_getitem__ and __mro_entries__ which means we will be able to pickle these annotations across processes (this is what is currently preventing us from doing nice things with it). That means in the future, instead of creating transformers to something like _PlotQualView so that you can accept disparate semantic types, you could say something like Union[FormatA, FormatB] and just check at runtime which one you’ve received (and write 0 new transformers).

All in all, it may be worth jumping straight from Python 3.5 to Python 3.7 (and pinning that as minimum) as some of these changes would significantly improve how the framework operates.
Right now 3.7 isn’t around on conda :frowning: , but it’ll be there someday soon I’m sure :slight_smile: .

5 Likes

Hi, all.

Wondering what the status is on moving the QIIME 2 releases to Python 3.7. I am working on a package that I would like to QIIME-ify but one of the core dependencies requires Python 3.7 and it seems the latest release (2020.11) still uses 3.6.

3 Likes

Hey there @gibsramen - python 3.7 (or more likely 3.8+) is still high up on our todo list. We are blocked right now while working on Library, but once we wrap up v1 of that, we will have new machinery in place to aid with that jump in python versions. Hopefully later this year…

:crossed_fingers:

1 Like