Issue with cmake when making EPA-NG conda recipe

gmdouglas · October 12, 2018, 7:16pm

Hi there,

I'm working on a plugin for PICRUSt2 (GitHub - picrust/picrust2: Code, unit tests, and tutorials for running PICRUSt2 - you can see rough details on the plugin here: q2 picrust2 Tutorial · picrust/picrust2 Wiki · GitHub).

I'd like to make the pre-requisites for PICRUSt2 available on conda to make the installation more straight-forward. EPA-NG (GitHub - pierrebarbera/epa-ng: Massively parallel phylogenetic placement of genetic sequences) is one of the pre-requisities that I'm having trouble writing a conda recipe for. This tool has a number of build requirements including cmake and autotools-dev (which isn't available through conda and may be related to this problem).

I'm getting this error in the circle CI report:

Running cmake
CMake Warning (dev) at CMakeLists.txt:119:
Syntax Warning in cmake code at column 71

Argument not separated from preceding token by whitespace.
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Error at /opt/conda/conda-bld/epa-ng_1539368794807/_build_env/share/cmake-3.12/Modules/CMakeDetermineCXXCompiler.cmake:47 (message):
Could not find compiler set in environment variable CXX:

g++.

Call Stack (most recent call first):
CMakeLists.txt:23 (project)

CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage

This is what I have in the meta.yaml:

package:
name: epa-ng
version: "0.2.1_beta_dev"

build:
number: 1

source:
url: https://github.com/gavinmdouglas/epa-ng/archive/0.2.1-beta-dev.tar.gz
md5: 15e7ad94596afc5cb72d90dfc0fad3eb

requirements:
build:
- {{ compiler('c') }}
- libtool
- flex
- bison
- cmake
- automake
- autoconf
run:

test:
commands:
- epa-ng -h > /dev/null

about:
home: GitHub - pierrebarbera/epa-ng: Massively parallel phylogenetic placement of genetic sequences
license: GNU Affero General Public License v3.0
summary: Massively parallel phylogenetic placement of genetic sequences

extra:
identifiers:
- doi:10.1093/sysbio/syy054

And these are the commands in build.sh:

#!/bin/bash

make

mkdir -p $PREFIX/bin

cp bin/epa-ng $PREFIX/bin

Thanks in advance!

ebolyen · October 12, 2018, 7:59pm

Hey @gmdouglas!

Are you building this locally or in the context of something like conda-forge or bioconda (I'll assume locally).

I think if you change the compiler:

To 'cxx' instead we'll get some slightly different errors, as it looks like EPA-NG uses CMake and C++. I'm not sure how CMake factors into this (if at all), but it's probably not surprising that conda isn't setting a env-var for a C++ compiler.

We can also use conda render which will template out everything in the recipe to what it is going to use. This should show us the recipe conda is actually using, which should be helpful information.

gmdouglas · October 15, 2018, 1:53pm

Hey @ebolyen, thanks for getting back to me!

I'm building this locally in the bioconda github repo using their docker environment (https://bioconda.github.io/contribute-a-recipe.html#circle-ci-client-method).

When using 'cxx' I get the same error actually:

CMake Error at /opt/conda/conda-bld/epa-ng_1539608768649/_build_env/share/cmake-3.12/Modules/CMakeDetermineCXXCompiler.cmake:47 (message):
Could not find compiler set in environment variable CXX:

g++.

Call Stack (most recent call first):
CMakeLists.txt:23 (project)

CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage

I wasn't familiar with conda render, thanks for suggesting it! Here's the output:

No handlers could be found for logger "libarchive"
--------------
Hash contents:
--------------
{'cxx_compiler': 'gxx'}
----------
meta.yaml:
----------
package:
    name: epa-ng
    version: 0.2.1_beta_dev
source:
    md5: 15e7ad94596afc5cb72d90dfc0fad3eb
    url: https://github.com/gavinmdouglas/epa-ng/archive/0.2.1-beta-dev.tar.gz
build:
    number: '1'
requirements:
    build:
        - krb5 1.16.1 hbb41f41_0
        - rhash 1.3.6 h470a237_1
        - xz 5.2.4 h470a237_1
        - binutils_impl_linux-64 2.31.1 h6176602_1
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - curl 7.61.1 h74213dd_2
        - perl 5.26.2 h470a237_0
        - binutils_linux-64 2.31.1 h6176602_3
        - bzip2 1.0.6 h470a237_2
        - cmake 3.12.3 h011004d_0
        - libcurl 7.61.1 hbdb9355_2
        - automake 1.15.1 pl526_1001
        - m4 1.4.18 h470a237_1
        - tk 8.6.8 ha92aebf_0
        - gxx_impl_linux-64 7.3.0 hdf63c60_1
        - flex 2.6.4 hfc679d8_3
        - expat 2.2.5 hfc679d8_2
        - ncurses 6.1 hfc679d8_1
        - libssh2 1.8.0 h5b517e9_2
        - gcc_linux-64 7.3.0 h553295d_3
        - ca-certificates 2018.8.24 ha4d7672_0
        - libgcc-ng 8.2.0 hdf63c60_1
        - libtool 2.4.6 h470a237_2
        - gcc_impl_linux-64 7.3.0 habb00fd_1
        - libedit 3.1.20170329 haf1bffa_1
        - gxx_linux-64 7.3.0 h553295d_3
        - zlib 1.2.11 h470a237_3
        - openssl 1.0.2p h470a237_1
        - bison 3.0.5 hfc679d8_1
        - autoconf 2.69 pl526_7
        - libuv 1.23.2 h470a237_0
    host:
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - libgcc-ng 8.2.0 hdf63c60_1
    run:
        - libstdcxx-ng >=7.3.0
        - libgcc-ng >=7.3.0
test:
    commands:
        - epa-ng -h > /dev/null
about:
    home: https://github.com/Pbdas/epa-ng
    license: GNU Affero General Public License v3.0
    summary: Massively parallel phylogenetic placement of genetic sequences
extra:
    copy_test_source_files: true
    final: true
    identifiers:
        - doi:10.1093/sysbio/syy054

Thanks for your help!

ebolyen · October 19, 2018, 8:52pm

Hey @gmdouglas,

Sorry for the delayed response.

Since you are using bioconda, that makes things a bit harder, we’ve noticed their compiler toolchain doesn’t quite work like we expect, so it’s something we’re trying to learn as well.

I’ll spend a few hours reading up on it and see what I can do.

Do you have a repo with this recipe inside (or a pull request to bioconda)? I couldn’t find anything specifically.

gmdouglas · October 21, 2018, 11:32pm

If you have the time that would be great @ebolyen, thanks!

I do have a pull-request here: https://github.com/bioconda/bioconda-recipes/pull/10902. I have mainly been testing it locally, but that was my first attempt at pushing the tool as a test. Two major problems could be that (1) EPA-NG hasn’t been released for Mac yet and (2) the requirement autotools-dev isn’t available through conda. I don’t think these issues explain this particular error message though!

Thanks again,

Gavin

ebolyen · October 23, 2018, 7:41pm

What a rabbit hole I fell into!

I was able to reproduce the problem on Circle CI. Basically, we need gcc7 during the build step because epa-ng uses C++14. conda-forge is in the middle of a complete overhaul to make this possible:

github.com/bioconda/bioconda-recipes

conda-forge needs our help

opened 03:18PM - 31 Aug 18 UTC

closed 02:07PM - 08 Mar 19 UTC

bgruening

help wanted

Hi @bioconda/all! conda-forge has started the huge migration to GCC7, Python …3.7 and R 3.5.1. Any help to get this done as quickly as possible would enable Bioconda to do the same and for example get all bioconductor packages to rebuild. You can see all the open PRs here: https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+archived%3Afalse+org%3Aconda-forge+-review%3Achanges_requested+-repo%3Aconda-forge%2Fstaged-recipes+author%3Aregro-cf-autotick-bot+Rebuild Pick one, review it or try to fix one of the faling builds. Its probably one of the largest migration which the conda community was ever facing and it looks like a lot of fun ;) join us!

However I don't know how to enable similar compilers on bioconda yet:

github.com/bioconda/bioconda-recipes

Conda build 3 migration

opened 07:30PM - 14 Jun 18 UTC

closed 02:03PM - 08 Mar 19 UTC

johanneskoester

@bioconda/all: Via PR #9402, we have migrated all recipes to Conda Build 3, and …at the same time updated our build system to conda build 3. It will take some time until all the recipes have been rebuilt. We will take care of that, but you can expect massive slowdowns of our build system until this is done. In order to let the migration go as smooth as possible, I would ask you to only create and merge PRs that are absolutely necessary until the migration is complete. Once the migration is complete, I will close this issue and let you know. # TODO * [ ] fix failing recipes * [ ] once all recipes have been built successfully uncomment `--git-range` in upload-linux and upload-macos jobs in circleci * [ ] release new version of bioconda-utils # Workflow 1. Look for the last [bulk build](https://github.com/bioconda/bioconda-recipes/commits/bulk) and select recipes to fix 2. Create a new branch from master, fix the selected failing recipes and PR the changes to master. 3. Once **all** recipes in bulk have been fixed this way, merge master into bulk again to trigger building eventually remaining recipes (this is only allowed for admins): ``` git checkout master git pull git checkout bulk git pull git merge --squash -X theirs master # fix remaining conflicts git commit -a git push ```

ebolyen · October 23, 2018, 7:43pm

I guess the best answer I have is stay tuned? Either this is possible with some kind of machinery that we don’t know about just yet, or it will soon be possible (ideally by default).

Until then, I don’t think there’s anything we can do about this particular recipe.

ebolyen · October 23, 2018, 9:10pm

Great news! I was able to get the new compilers, and after some tweaking (and trying to figure out what exit code 141 means to bison) I was able to get a local build working in docker!

Here's a PR against your branch with the changes:

gmdouglas · October 23, 2018, 10:54pm

Fantastic - thanks so much for figuring this out! I’ll test this out tomorrow!

gmdouglas · January 15, 2019, 4:33pm

Late reply, but after troubleshooting a number of syntax errors with the developers EPA-NG is now on bioconda. Thanks @ebolyen !