Core-metrics-phylogenetic crashed: free(): invalid next size

Technical Support
1

My QIIME version is 2019.01

The command is:

(qiime2-2019.1) [xzhan9@Nucleus005 another.16s]$ rm -rf core-metrics-results/ && qiime diversity core-metrics-phylogenetic --i-phylogeny rooted-tree.qza --i-table table.qza --p-sampling-depth 38005 --m-metadata-file sample-metadata.tsv --output-dir core-metrics-results --verbose

The error message is:

/home2/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/site-packages/sklearn/utils/validation.py:595: DataConversionWarning: Data with input dtype float64 was converted to bool by check_pairwise_arrays.
warnings.warn(msg, DataConversionWarning)
/home2/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/site-packages/skbio/stats/ordination/_principal_coordinate_analysis.py:188: RuntimeWarning: invalid value encountered in true_divide
proportion_explained = eigvals / sum_eigenvalues
/home2/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/site-packages/emperor/core.py:614: RuntimeWarning: invalid value encountered in true_divide
c_data = (coords / np.max(np.abs(coords)))
*** Error in `/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python': free(): invalid next size (fast): 0x0000555557a472f0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c619)[0x2aaaaaf67619]
/home2/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/site-packages/unifrac/_api.cpython-36m-x86_64-linux-gnu.so(_ZNSt10_HashtableINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES5_SaIS5_ENSt8__detail9_IdentityESt8equal_toIS5_ESt4hashIS5_ENS7_18_Mod_range_hashingENS7_20_Default_ranged_hashENS7_20_Prime_rehash_policyENS7_17_Hashtable_traitsILb1ELb1ELb1EEEE5clearEv+0x32)[0x2aaac8a4aa12]
/home2/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/site-packages/unifrac/_api.cpython-36m-x86_64-linux-gnu.so(_ZNSt10_HashtableINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES5_SaIS5_ENSt8__detail9_IdentityESt8equal_toIS5_ESt4hashIS5_ENS7_18_Mod_range_hashingENS7_20_Default_ranged_hashENS7_20_Prime_rehash_policyENS7_17_Hashtable_traitsILb1ELb1ELb1EEEED2Ev+0xa)[0x2aaac8a4aa5a]
/home2/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/site-packages/unifrac/_api.cpython-36m-x86_64-linux-gnu.so(one_off+0xaa1)[0x2aaac8a4a621]
/home2/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/site-packages/unifrac/_api.cpython-36m-x86_64-linux-gnu.so(+0x22807)[0x2aaac8a44807]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyCFunction_FastCallDict+0x154)[0x555555667754]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x1a15ac)[0x5555556f55ac]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x30a)[0x55555571a28a]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16d801)[0x5555556c1801]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16e84f)[0x5555556c284f]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x1a1685)[0x5555556f5685]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x10c7)[0x55555571b047]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(PyEval_EvalCodeEx+0x329)[0x5555556c5eb9]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x172dd6)[0x5555556c6dd6]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(PyObject_Call+0x3e)[0x55555566755e]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x184f)[0x55555571b7cf]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16e60b)[0x5555556c260b]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x1a1685)[0x5555556f5685]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x30a)[0x55555571a28a]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16dc4e)[0x5555556c1c4e]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16e84f)[0x5555556c284f]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x1a1685)[0x5555556f5685]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x30a)[0x55555571a28a]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16d801)[0x5555556c1801]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16e84f)[0x5555556c284f]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x1a1685)[0x5555556f5685]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x10c7)[0x55555571b047]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(PyEval_EvalCodeEx+0x329)[0x5555556c5eb9]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x172dd6)[0x5555556c6dd6]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(PyObject_Call+0x3e)[0x55555566755e]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x184f)[0x55555571b7cf]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16e60b)[0x5555556c260b]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x1a1685)[0x5555556f5685]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x30a)[0x55555571a28a]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16dc4e)[0x5555556c1c4e]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16e84f)[0x5555556c284f]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x1a1685)[0x5555556f5685]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x30a)[0x55555571a28a]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(PyEval_EvalCodeEx+0x329)[0x5555556c5eb9]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x172dd6)[0x5555556c6dd6]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(PyObject_Call+0x3e)[0x55555566755e]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x178fc7)[0x5555556ccfc7]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(PyObject_Call+0x3e)[0x55555566755e]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x184f)[0x55555571b7cf]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16d801)[0x5555556c1801]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyFunction_FastCallDict+0x3d7)[0x5555556c2f17]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyObject_FastCallDict+0x26f)[0x555555667b1f]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyObject_Call_Prepend+0x63)[0x55555566c7b3]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(PyObject_Call+0x3e)[0x55555566755e]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x178fc7)[0x5555556ccfc7]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(PyObject_Call+0x3e)[0x55555566755e]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x184f)[0x55555571b7cf]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16d801)[0x5555556c1801]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyFunction_FastCallDict+0x3d7)[0x5555556c2f17]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyObject_FastCallDict+0x26f)[0x555555667b1f]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyObject_Call_Prepend+0x63)[0x55555566c7b3]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(PyObject_Call+0x3e)[0x55555566755e]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x184f)[0x55555571b7cf]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16e60b)[0x5555556c260b]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x1a1685)[0x5555556f5685]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(_PyEval_EvalFrameDefault+0x30a)[0x55555571a28a]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16dc4e)[0x5555556c1c4e]
/home2/xzhan9/.conda/envs/qiime2-2019.1/bin/python(+0x16e84f)[0x5555556c284f]
======= Memory map: ========
2aaaaaaab000-2aaaaaacc000 r-xp 00000000 08:01 39109043 /usr/lib64/ld-2.17.so
2aaaaaacc000-2aaaaaace000 r-xp 00000000 00:00 0 [vdso]
2aaaaaace000-2aaaaaacf000 rw-p 00000000 00:00 0
2aaaaaacf000-2aaaaaad6000 r--s 00000000 08:01 6426948 /usr/lib64/gconv/gconv-modules.cache
2aaaaaad6000-2aaaaaad7000 r--p 00000000 00:32 580100053 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
2aaaaaad7000-2aaaaaad9000 r-xp 00001000 00:32 580100053 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
2aaaaaad9000-2aaaaaada000 r--p 00003000 00:32 580100053 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
2aaaaaada000-2aaaaaadb000 r--p 00003000 00:32 580100053 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
2aaaaaadb000-2aaaaaadd000 rw-p 00004000 00:32 580100053 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
2aaaaaadd000-2aaaaaade000 r--p 00000000 00:32 580100035 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so
2aaaaaade000-2aaaaaadf000 r-xp 00001000 00:32 580100035 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so
2aaaaaadf000-2aaaaaae0000 r--p 00002000 00:32 580100035 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so
2aaaaaae0000-2aaaaaae1000 r--p 00002000 00:32 580100035 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so
2aaaaaae1000-2aaaaaae2000 rw-p 00003000 00:32 580100035 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so
2aaaaaae2000-2aaaaaae5000 r--p 00000000 00:32 580100069 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so
2aaaaaae5000-2aaaaaaeb000 r-xp 00003000 00:32 580100069 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so
2aaaaaaeb000-2aaaaaaed000 r--p 00009000 00:32 580100069 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so
2aaaaaaed000-2aaaaaaee000 r--p 0000a000 00:32 580100069 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so
2aaaaaaee000-2aaaaaaf0000 rw-p 0000b000 00:32 580100069 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so
2aaaaaaf0000-2aaaaaaf1000 rwxp 00000000 00:00 0
2aaaaaaf1000-2aaaaacb6000 rw-p 00000000 00:00 0
2aaaaacb6000-2aaaaacb9000 r--p 00000000 00:32 580100073 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so
2aaaaacb9000-2aaaaacbf000 r-xp 00003000 00:32 580100073 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so
2aaaaacbf000-2aaaaacc1000 r--p 00009000 00:32 580100073 /work/PCDC/xzhan9/.conda/envs/qiime2-2019.1/lib/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.soAborted (core dumped)

I also shared the input files:

https://www.dropbox.com/sh/oqtffyctdnduugn/AABAJD4clJ_mx4X8tBPETaSHa?dl=0

3

Hi @zhan_xw,

Apologies for the delay.

This looks like something strange happening in the unifrac package. How many samples are left after rarefying at 38005 (based on your feature-table summary)?

Could you also supply some information about your current environment:

uname --all

Is this running on a cluster? Is there a queuing system involved? Thanks, and sorry for the vague questions, I'm not 100% sure where to start looking yet.

cc @wasade

1 Like
5

I should have 1 samples with depth of 38005 (attached the qzv file).

(qiime2-2019.1) [xzhan9@Nucleus005 another.16s]$ uname --all
Linux Nucleus005 3.10.0-693.2.2.el7.x86_64 #1 SMPtable.qzv (294.0 KB)
Sat Sep 9 03:55:24 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

yes, it is on a cluster with Python 3.6.7(Python 3.6.7 | packaged by conda-forge | (default, Nov 21 2018, 03:09:43)).no queue system involved.

6

That may be the problem: a sample size of 1 is not useful for running through core-metrics, since that pipeline performs diversity analyses comparing multiple samples. You cannot calculate a unifrac distance matrix with 1 sample, as there are no distances to calculate — distances are calculated between pairs of samples.

If you had more than 1 sample and other samples are being dropped because you are setting a high sampling depth, reduce the sampling depth. Otherwise do not bother using this pipeline. You can use qiime diversity alpha to calculate alpha diversity values from a single sample if that is meaningful to you, but none of the beta diversity metrics are meaningful or calculable if you have one sample.

I hope that helps!

2 Likes
8

Hi @Nicholas_Bokulich and @zhan_xw, just as a brief update, we've opened a new issue for q2-diversity to check for this edge case, and to report appropriately back to the user.

Best,
Daniel

3 Likes