Building from source for Lmod

I am trying to provide qiime2 via an Lmod module system for use on our compute servers. I know that you recommend installation via conda, but given that all of the prerequisite programs are already on our system, and that conda installation is failing because of hundreds of conflicts, I just need to know how to compile the github repo to create a functioning qiime2 binary.

So far, I can install it through pip and it exists as an importable module to Python, but I cannot figure out where the binary comes from that provides the normal qiime2 interface. There is surely a way to do this to create it for your conda repo, so can I ask for some assistance?

@cmfield,
Unfortunately I am not familiar with the Lmod system. Could you share the errors you are getting with Conda? To me it sounds like Conda is not even installed on the system, which unfortunately, we cannot officially provide support for, however, there is a big user base on here and it is likely that the issue that you are having has either come up before or is getting ready to come up quite a bit due to a change on Conda's end, either way, it would be helpful if you could post the exact issues you are having, hopefully one of us will have an answer for you.

QIIME 2 is written in Python, so there is no way to compile it to a binary. Having the dependencies installed on your machine does not really help, as there is no way to know which versions you have installed, many of which will not work with QIIME 2, its plug-ins, or another dependency. Managing these sorts of issues is why we are using Conda in the first place. Conda uses various configuration files to figure out exactly, everything it needs to install. All of this is tracked using what is known as a "metapackage".

It might be possible to use the metapackage to tell Lmod what it needs to install, but my guess is that it would be easier to get Conda installed and configured correctly than messing around with such an approach, unless you have some un-avoidable reason you cannot install Conda on your machine. Even then, I would look at using the packaged Docker install rather than trying to hack the metapackage to work with Lmod. Here is the link to the install docs.

If I have misinterpreted what is going on, let me know and I will give it another try :joy:

.

Lmod is effectively a package manager like Conda, and used with another system called EasyBuild you can load all the appropriate modules with appropriate versions to create an environment. It's quite common on HPC systems where you don't want users creating their own environments redundantly.

Anyway, I do have Miniconda3 installed within the Lmod system (4.9.2) and that has worked for a previous version of QIIME2, and a few other programs, but not for this one. I've attached the specific output of conda if you think you can solve that (side note - you don't allow upload of .log files??):

qiime2.txt (2.6 MB)

1 Like

Hi Chris,

I have installed QIIME2 before on a HPC cluster that uses LMOD modules. This is how I approached it:

  1. Loaded a python LMOD module (with env2lmod module load)
  2. Installed miniconda on the scratch space (since my HPC cluster had a limited number of files that are allowed to be uploaded to the home directory - which were exceeded when installing QIIME2 - raising a "[Errno 122] Disk quota exceeded" error).
  3. Installed QIIME2 with miniconda (making sure to use the Linux distribution):
wget https://data.qiime2.org/distro/core/qiime2-2022.8-py38-linux-conda.yml
conda env create -n qiime2-2022.8 --file qiime2-2022.8-py38-linux-conda.yml

Is this the same way you approached it? if not, what was the command you ran that led to the attached qiime2.txt output?

Best,
Anja

2 Likes

I did something very similar. I have Miniconda3 as a module already (I have an NFS drive with large limits I can use for this), and loaded it. Then ran the same conda command as you list, albeit with a slightly different -n name, but the same YAML file.

All I can think is that my base environment has stuff in it that it shouldn't (the Miniconda3 installation was fresh so this shouldn't be the case) or the system itself is somehow incompatible with something QIIME2 wants to install through conda.

Looking through that log file, it looks like existing software on the cluster conflicts with versions requested by Qiime2.

Package _openmp_mutex conflicts for:
r-ellipsis=0.3.2 -> libgcc-ng[version='>=12']       -> _openmp_mutex[version='>=4.5']
r-munsell=0.5.0 -> r-base[version='>=4.2,<4.3.0a0'] -> _openmp_mutex
python=3.8.13 -> libgcc-ng[version='>=10.3.0']      -> _openmp_mutex[version='>=4.5']
kiwisolver=1.4.4 -> libgcc-ng[version='>=12']       -> _openmp_mutex[version='>=4.5']
...
_openmp_mutex=4.5
...

So the system has got 4.5 installed, but stuff is calling for >=4.5

Wait... why is that a conflict?

There're lots of issues like this, where it tells us there is a conflict but it looks fine to me.

Package libblas conflicts for:
numpy=1.23.2 -> libblas[version='>=3.9.0,<4.0a0']
libblas=3.9.0

Where's the conflict? Am I misreading this log?

1 Like

Yeah that log doesn't make any sense.

Anyway I took a shot at trying a different version of Miniconda - 4.12.0 where before I used 4.9.2 and lo and behold it produced no errors.. so there we go.

It'd be great if in future rather than the instructions saying 'update to the latest version of Conda' a minimum version was given, like with every other software installation. I'd also still love a way to compile QIIME2, there are other advantages to being able to do so, but I doubt it's a priority.

1 Like

@cmfield,

Adding a minimum conda version seems like a great idea, I will chat with the other devs about it.

It would be great to have a compiled version, however, there are currently no plans for such an implementation. However, we always welcome contributions! You also might want to check out the Docker images, which can be particularly helpful on clusters. Definitely not the same thing, but that is the closest we have to offer right now.

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.