Tensorflow warnings in conda environment with Medaka

Developer Discussion
1

Hi team,

I am currently working in a qiime2-2024.2 conda environment in which I installed Medaka using pip. This tool is dedicated to Nanopore data and heavily depends on Tensorflow.

I suspect the Medaka pip installation modified the initial Tensorflow configuration of the qiime2 environment, as I receive these lines every time I run a qiime command:

2024-06-18 15:19:09.631042: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-18 15:19:09.747506: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2024-06-18 15:19:09.747535: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2024-06-18 15:19:09.773852: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-06-18 15:19:10.148058: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2024-06-18 15:19:10.148156: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2024-06-18 15:19:10.148182: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

These are actually not error messages as the qiime commands complete successfully.

Since the Medaka tool seems very sensitive to the Tensorflow configuration, I would prefer not to change it. Instead, I need QIIME2 to tolerate the Tensorflow configuration that came with the Medaka installation (since it does not affect qiime command behavior, everything still works). Maybe the easiest solution would be to simply mute these notification lines?

Could you please indicate how I could do this?

Thank you in advance for your help!

Ben

2

Hello!
Solution 1: Ignore this warning. It is not an error, but a warning that you can improve the configuration to make it faster.
Solution 2: I use it only for the scripts in which I don't want this message to be shown.

import os
os.environ["MKL_NUM_THREADS"] = "1" 
os.environ["NUMEXPR_NUM_THREADS"] = "1" 
os.environ["OMP_NUM_THREADS"] = "1" 
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '1'

It works only for the current session.
Best,

3

Thank you for your prompt reply @timanix!

However, I would like to share this environment with other people so that they can use any qiime command without warnings. The lines you provided are interesting if qiime commands are run in a script but I need a solution to remove these warnings when directly running a command in the terminal and not into a script.

Is it possible to indicate somewhere in the QIIME2 environment configuration that (i) these warnings must not be displayed, or (ii) not to check the tensorflow config?

Or maybe there is another option to circumvent these warnings?

4

This warning is not so easy to suppress, I already tried many things that worked earlier but not now and the only thing that worked for me is that lines I shared. This warning is not qiime2 related, so I don't think there is an easy way to fix it.

I prefer not to install other software into the qiime2 environment that is not tested for it since certain commands may give errors afterward. Can it be installed in another environment?

I will leave this post queued in case other mods have better suggestions.

Best,

4 Likes