We’re sorry to report that on certain CPUs1 matrix multiplication is performed incorrectly by OpenBLAS. We had initially thought the issue was a mismatch between numpy bindings and OpenBLAS, however further research has revealed the issue to be fundamental to OpenBLAS.
1. those with AVX 512 extensions, i.e. Skylake-X
Am I affected?
Probably not, unless you have a new (and relatively expensive) computer. In particular only Intel CPUs with AVX 512 have this issue. If your CPU is not a member of one of these processor families then there is nothing to worry about; OpenBLAS will do the correct thing.
If you don’t know (or don’t care) what CPU you have, we recommend just updating to the latest patch.
If you would like to learn if your CPU has AVX 512, you can run the following:
sysctl -a | grep machdep.cpu.features # Look for AVX 512 (AVX 2 or 1 is fine)
cat /proc/cpuinfo | grep flags | uniq # Look for AVX 512 (AVX 2 or 1 is fine)
What is OpenBLAS?
OpenBLAS is a widely used part of the scientific computing stack, providing fast and efficient linear algebra routines, and can be found linked from libraries like
scipy to full programming languages like
Julia. Much of what we do would not be possible without high quality and free libraries like this one.
What is being done?
Later today/tomorrow morning we will have a patch ready (
2019.4.1). We strongly recommend you update your version of QIIME 2 (ensuring that
qiime info indicates that you have q2-types 2019.4.1 installed, not 2019.4.0). This patch will specifically pin OpenBLAS to 0.3.3 which does not use AVX 512 instructions. In order to make it easy to identify which 2019.4 version was used, we’re updating the patch number for q2-types so that it is easier to spot in provenance.
We believe that only OpenBLAS versions 0.3.5 and 0.3.6 are impacted (specifically the DGEMM routine). To the best of our knowledge this issue has not been fully solved yet as we were able to reproduce the issue with both of those versions (even though 0.3.6 should have disabled the problematic code). OpenBLAS 0.3.3 was unaffected by the issue on the same hardware.