i find ERROR in qiime diversity core-metrics-phylogenetic

Hello, I am from China, I am a newbie in qiime2, I am learning the data of qiime2, I have a problem when facing qiime diversity core-metrics-phylogenetic, I found that many people in the forum have encountered this problem It has not been solved yet. I would like to ask if there is a clear solution now.

(qiime2-2023.9) zhang_chao 16:29:12 ~/qiime2/test/mp
$ qiime diversity core-metrics-phylogenetic \
>   --i-phylogeny rooted-tree.qza \
>   --i-table table.qza \
>   --p-sampling-depth 1103 \
>   --m-metadata-file sample-metadata.tsv \
>   --output-dir core-metrics-results
Plugin error from diversity:

  Command '['ssu', '-i', '/tmp/qiime2/zhang_chao/data/7e26fa27-228d-4058-b087-fd74caee89e7/data/feature-table.biom', '-t', '/tmp/qiime2/zhang_chao/data/efd5331a-8cad-4464-b567-085caa2ade33/data/tree.nwk', '-m', 'unweighted', '-o', '/tmp/q2-LSMatFormat-zhlmh9vk']' returned non-zero exit status 1.
Debug info has been saved to /tmp/qiime2-q2cli-err-64ca1r3w.log

Below are posts with the same problem
https://forum.qiime2.org/t/diversity-core-metrics-phylogenetic-alpha-beta-diversity/24086
https://forum.qiime2.org/t/plugin-error-from-diversity-going-through-tutorial/24319
https://forum.qiime2.org/t/core-metrics-phylogenetic-plugin-error-from-diversity-returned-non-zero-exit-status-1/27786
Looking forward to your reply and help, thank you

Hello @linke0909, can you please rerun the command with the --verbose flag and post the results here? Thank you.

Hello! Thank you for your reply. Due to the time difference, I am willing to spoof you for being slower. Sorry! The following is the code information

(qiime2-2023.9) zhang_chao 08:39:30 ~/qiime2/test/mp
$ qiime diversity core-metrics-phylogenetic \
>   --i-phylogeny rooted-tree.qza \
>   --i-table table.qza \
>   --p-sampling-depth 1103 \
>   --m-metadata-file sample-metadata.tsv \
>   --output-dir core-metrics-results \
> --verbose
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command:

faithpd -i /tmp/qiime2/zhang_chao/data/7e26fa27-228d-4058-b087-fd74caee89e7/data/feature-table.biom -t /tmp/qiime2/zhang_chao/data/efd5331a-8cad-4464-b567-085caa2ade33/data/tree.nwk -o /tmp/q2-AlphaDiversityFormat-50k_jrf9

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command:

ssu -i /tmp/qiime2/zhang_chao/data/7e26fa27-228d-4058-b087-fd74caee89e7/data/feature-table.biom -t /tmp/qiime2/zhang_chao/data/efd5331a-8cad-4464-b567-085caa2ade33/data/tree.nwk -m unweighted -o /tmp/q2-LSMatFormat-9au4ecps

Failing in Thread:0
call to cuInit returned error 804: Other

Traceback (most recent call last):
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2cli/commands.py", line 520, in __call__
    results = self._execute_action(
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2cli/commands.py", line 586, in _execute_action
    results = action(**arguments)
  File "<decorator-gen-189>", line 2, in core_metrics_phylogenetic
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/qiime2/sdk/action.py", line 342, in bound_callable
    outputs = self._callable_executor_(
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/qiime2/sdk/action.py", line 647, in _callable_executor_
    outputs = self._callable(scope.ctx, **view_args)
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2_diversity/_core_metrics.py", line 66, in core_metrics_phylogenetic
    dms += unweighted_unifrac(table=cr.rarefied_table, phylogeny=phylogeny,
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/qiime2/sdk/context.py", line 140, in deferred_action
    return action_obj._bind(
  File "<decorator-gen-711>", line 2, in unweighted_unifrac
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/qiime2/sdk/action.py", line 342, in bound_callable
    outputs = self._callable_executor_(
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/qiime2/sdk/action.py", line 566, in _callable_executor_
    output_views = self._callable(**view_args)
  File "<decorator-gen-143>", line 2, in unweighted_unifrac
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 69, in _disallow_empty_tables
    return wrapped_function(*args, **kwargs)
  File "<decorator-gen-142>", line 2, in unweighted_unifrac
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 112, in _validate_requested_cpus    return wrapped_function(*bound_arguments.args, **bound_arguments.kwargs)
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2_diversity_lib/beta.py", line 221, in unweighted_unifrac
    _omp_cmd_wrapper(threads, cmd)
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 128, in _omp_cmd_wrapper
    return _run_external_cmd(cmd, verbose=verbose, env=env)
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 122, in _run_external_cmd
    return subprocess.run(cmd, check=True, env=env)
  File "/titan/zhang_chao/anaconda3/envs/qiime2-2023.9/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ssu', '-i', '/tmp/qiime2/zhang_chao/data/7e26fa27-228d-4058-b087-fd74caee89e7/data/feature-table.biom', '-t', '/tmp/qiime2/zhang_chao/data/efd5331a-8cad-4464-b567-085caa2ade33/data/tree.nwk', '-m', 'unweighted', '-o', '/tmp/q2-LSMatFormat-9au4ecps']' returned non-zero exit status 1.

Plugin error from diversity:

  Command '['ssu', '-i', '/tmp/qiime2/zhang_chao/data/7e26fa27-228d-4058-b087-fd74caee89e7/data/feature-table.biom', '-t', '/tmp/qiime2/zhang_chao/data/efd5331a-8cad-4464-b567-085caa2ade33/data/tree.nwk', '-m', 'unweighted', '-o', '/tmp/q2-LSMatFormat-9au4ecps']' returned non-zero exit status 1.
See above for debug info.

EDIT: Follow @wasade's advice first

This is a bit of a tricky one. This part of the error:

ssu -i /tmp/qiime2/zhang_chao/data/7e26fa27-228d-4058-b087-fd74caee89e7/data/feature-table.biom -t /tmp/qiime2/zhang_chao/data/efd5331a-8cad-4464-b567-085caa2ade33/data/tree.nwk -m unweighted -o /tmp/q2-LSMatFormat-9au4ecps

Failing in Thread:0
call to cuInit returned error 804: Other

suggests that this is an Nvidia related issue. cuInit stands for "cuda init" or something along those lines and cuda is Nvidia's language for programming their GPUs. Am I correct in assuming you have access to an Nvidia GPU in that machine? If so, ssu will attempt to use it by default, but that doesn't appear to be working.

I can't get it to link directly to your specific error, but if you go on this page and look for "804" you can see the error you're getting. It looks like it has something to do with an incompatibility between the physical GPU and the drivers on the machine. I would start by uninstalling the current GPU drivers and installing whatever the latest driver version is for the specific GPU model you have. If that doesn't work then you'll probably need to post on the Nvidia help forum and ask them.

1 Like

Hi @linke0909,

Could you describe the system you're using? A few things that would be helpful are the output of head /proc/cpuinfo and uname -a. Any information about the operating system version (requesting this can vary on flavors of Linux) would be helpful as well. And are there GPUs on the system where the command was performed, and if this is a cluster, is the associated submission requesting GPU resources?

Additionally, could you try running export UNIFRAC_USE_GPU=N, and then rerun the call to core-metrics-phylogenetic?

Best,
Daniel

1 Like

Thank you very much for your answer

Thanks for the answer, here is the configuration of my server

Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.14.0-1054-oem x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

105 更新可以立即应用。
这些更新中有 1 个是标准安全更新。
要查看这些附加更新,请运行:apt list --upgradable


1 updates could not be installed automatically. For more details,
see /var/log/unattended-upgrades/unattended-upgrades.log
Your Hardware Enablement Stack (HWE) is supported until April 2025.
*** System restart required ***
Last login: Fri Jan  5 13:20:24 2024 from 10.1.42.2

架构:                           x86_64
CPU 运行模式:                   32-bit, 64-bit
字节序:                         Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU:                             20
在线 CPU 列表:                  0-19
每个核的线程数:                 2
每个座的核数:                   10
座:                             1
NUMA 节点:                      1
厂商 ID:                        GenuineIntel
CPU 系列:                       6
型号:                           85
型号名称:                       Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz
步进:                           7
CPU MHz:                        2400.000
CPU 最大 MHz:                   3200.0000
CPU 最小 MHz:                   1000.0000
BogoMIPS:                       4800.00
虚拟化:                         VT-x
L1d 缓存:                       320 KiB
L1i 缓存:                       320 KiB
L2 缓存:                        10 MiB
L3 缓存:                        13.8 MiB
NUMA 节点0 CPU:                 0-19
Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; TSX disabled
标记:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm 
                                 pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid ap                                 erfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2                                 apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_                                 l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsg                                 sbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb int                                 el_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dt                                 herm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilit                                 ies
(qiime2-2023.9) zhang_chao 08:48:34 ~
$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
(qiime2-2023.9) zhang_chao 08:54:09 ~
$  lspci -vnn | grep VGA
0000:b3:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2204] (rev a1) (prog-if 00 [VGA controller])
(qiime2-2023.9) zhang_chao 08:54:35 ~
$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
(qiime2-2023.9) zhang_chao 08:55:06 ~
$ lspci | grep -i nvidia
0000:b3:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1)
0000:b3:00.1 Audio device: NVIDIA Corporation Device 1aef (rev a1)
(qiime2-2023.9) zhang_chao 08:55:23 ~
$ sudo dpkg --list | grep nvidia-*
[sudo] zhang_chao 的密码: 
zhang_chao 不在 sudoers 文件中。此事将被报告。
(qiime2-2023.9) zhang_chao 08:55:37 ~
$ head /proc/cpuinfouname -a
head: 不适用的选项 -- a
请尝试执行 "head --help" 来获取更多信息。
(qiime2-2023.9) zhang_chao 08:56:15 ~
$ export UNIFRAC_USE_GPU=Ncore-metrics-phylogenetic

1 Like

I gave up on qiime2 because it has many bugs that cannot be solved. I don't understand why so many people are still requesting to use qiime2. I switched to vserch and userch, and qiime2 is a waste of my time

To clarify here @linke0909 the problem you're seeing has nothing to do with QIIME 2 itself. Your GPU isn't configured correctly. It looks like the wrong version of your GPU drivers was installed. It's unlikely that any application attempting to use the GPU in your system will work until this is resolved.

2 Likes

Also to clarify what @wasade was requesting here. Please first run the command export UNIFRAC_USE_GPU=N as its own command. This will tell unifrac not to use your GPU because your GPU appears to be misconfigured. Then rerun your qiime core-metrics-phylogenetic command as a separate command.

2 Likes

Thank you for your reply. Yes, I have listened to your suggestion. I did not use a server with GPU configuration issues. Instead, I tried installing Ubuntu using the Linux subsystem under Windows, but encountered a new problem while using Dada2. I checked the forum and found that the r path was incorrect, but there is only one r path in my Ubuntu. I have also tried to modify it, change the version, and reinstall it, In short, during the use of qiime2, I feel that it is very unsuitable, and I have also found many unresolved similar issues while browsing forums. The above is my personal opinion.

(qiime2-2023.9) zhangchao@DESKTOP-NV32H8O:~/mp$ qiime dada2 denoise-single \
>   --i-demultiplexed-seqs demux.qza \
>   --p-trim-left 0 \
>   --p-trunc-len 120 \
>   --o-representative-sequences rep-seqs-dada2.qza \
>   --o-table table-dada2.qza \
>   --o-denoising-stats stats-dada2.qza\
> --verbose
Plugin error from dada2:

  An error was encountered while running DADA2 in R (return code 255), please inspect stdout and stderr to learn more.

Debug info has been saved to /tmp/qiime2-q2cli-err-yyak3ydm.log
(qiime2-2023.9) zhangchao@DESKTOP-NV32H8O:~/mp$ which R
/home/zhangchao/anaconda3/envs/qiime2-2023.9/bin/R

In the above command, the --verbose flag is passed incorrectly which caused the verbose output to not display.

>   --o-denoising-stats stats-dada2.qza\
> --verbose

Because there is no space before the \ your terminal interpreted this as --o-denoising-stats stats-dada2.qza--verbose. What the \ in a terminal does is basically tell the terminal "the next line should be treated as part of this same line," so in the end all the lines with \ get combined into one line, so you still need to put a space there.

If you are still interested in debugging this please run the command with the --verbose flag passed correctly.

If you do run the command with --verbose my guess is you will see a line saying something along the lines of Rscript execution error: No such file or directory. This could be due to QIIME 2 having installed incorrectly, but we most often see errors like this when there are competing versions of R installed on a system. R does not like to respect people's virtual environments and will frequently cause headaches by deciding not to let QIIME 2 run the version of R it wants to because R thinks it knows better (it doesn't).

If you run which R with your environment active you should see what you are seeing

(qiime2-2023.9) zhangchao@DESKTOP-NV32H8O:~/mp$ which R
/home/zhangchao/anaconda3/envs/qiime2-2023.9/bin/R

You should see something similar if you run which Rscript with your environment active. If you run which R or which Rscript without your environment active then, assuming you don't have R installed on your computer outside of your conda environment, you should see nothing.

If you do see nothing then I would remove your environment and recreate it making sure to follow the install instructions found here under "WIndows (via WSL)" (we frequently have people follow install instructions that were created by people other than us then have issues with their environment).

If you do see versions of R and Rscript outside of your conda environment, then remove them if you don't need them. If you do still need them then it will take a bit more fiddling to convince R to listen, and we can deal with that if needed.

3 Likes

Thank you for your reply. The result is as you guessedRscript execution error: No such file or directory. How can I solve this problem

(qiime2-2023.9) zhangchao@DESKTOP-NV32H8O:~/mp$ qiime dada2 denoise-single \
>   --i-demultiplexed-seqs demux.qza \
>   --p-trim-left 0 \
>   --p-trunc-len 120 \
>   --o-representative-sequences rep-seqs-dada2.qza \
>   --o-table table-dada2.qza \
>   --o-denoising-stats stats-dada2.qza --verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada.R --input_directory /tmp/qiime2/zhangchao/data/e59ec413-3ba5-45c7-93f7-cc03b0b607ce/data --output_path /tmp/tmpe7zj4i1g/output.tsv.biom --output_track /tmp/tmpe7zj4i1g/track.tsv --filtered_directory /tmp/tmpe7zj4i1g --truncation_length 120 --trim_left 0 --max_expected_errors 2.0 --truncation_quality_score 2 --max_length Inf --pooling_method independent --chimera_method consensus --min_parental_fold 1.0 --allow_one_off False --num_threads 1 --learn_min_reads 1000000 --homopolymer_gap_penalty NULL --band_size 16

Rscript execution error: No such file or directory
Traceback (most recent call last):
  File "/home/zhangchao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 220, in _denoise_single
    run_commands([cmd])
  File "/home/zhangchao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 36, in run_commands
    subprocess.run(cmd, check=True)
  File "/home/zhangchao/anaconda3/envs/qiime2-2023.9/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_dada.R', '--input_directory', '/tmp/qiime2/zhangchao/data/e59ec413-3ba5-45c7-93f7-cc03b0b607ce/data', '--output_path', '/tmp/tmpe7zj4i1g/output.tsv.biom', '--output_track', '/tmp/tmpe7zj4i1g/track.tsv', '--filtered_directory', '/tmp/tmpe7zj4i1g', '--truncation_length', '120', '--trim_left', '0', '--max_expected_errors', '2.0', '--truncation_quality_score', '2', '--max_length', 'Inf', '--pooling_method', 'independent', '--chimera_method', 'consensus', '--min_parental_fold', '1.0', '--allow_one_off', 'False', '--num_threads', '1', '--learn_min_reads', '1000000', '--homopolymer_gap_penalty', 'NULL', '--band_size', '16']' returned non-zero exit status 255.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zhangchao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2cli/commands.py", line 520, in __call__
    results = self._execute_action(
  File "/home/zhangchao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2cli/commands.py", line 581, in _execute_action
    results = action(**arguments)
  File "<decorator-gen-70>", line 2, in denoise_single
  File "/home/zhangchao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/qiime2/sdk/action.py", line 342, in bound_callable
    outputs = self._callable_executor_(
  File "/home/zhangchao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/qiime2/sdk/action.py", line 566, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/zhangchao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 244, in denoise_single
    return _denoise_single(
  File "/home/zhangchao/anaconda3/envs/qiime2-2023.9/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 229, in _denoise_single
    raise Exception("An error was encountered while running DADA2"
Exception: An error was encountered while running DADA2 in R (return code 255), please inspect stdout and stderr to learn more.

Plugin error from dada2:

  An error was encountered while running DADA2 in R (return code 255), please inspect stdout and stderr to learn more.

See above for debug info.

Hi @linke0909,

Can you clarify whether the issues you're running into with the DADA2 plugin are in the environment where you made these changes, or if they are in a freshly installed QIIME 2 environment?

2 Likes

Hello, thank you for your reply. I have encountered an issue with Dada2 in the newly installed environment. Currently, I have tried three methods to test the data: WSL, virtual machine, server, server graphics card issues, virtual machine and WSL, and Dada2 errors. Now I can only run it separately, using the server in the first half and importing files into WSL for analysis in the second half. The entire process runs quickly in 5 minutes, but this problem still persists. I want to know how to avoid dada2 errors because there are also many people who encounter these problems without clear answers, which will discourage many beginners and ultimately give up qiime2. This is something I don't want to see, After all, qiime2 is indeed a great invention!

Wishing you all the best!
chaozhang

Please do the following: With the failing conda environment active run both which R and which Rscript then run env and post the entire output from all three commands here. It should look something like this



Then: Deactivate the conda environment and run which R and which Rscript again and post the entire output here. It should look something like this.



I need this information to determine that

1: Your R installation in your environment is correct
2: You do not have an R installation outside of your environment that is potentially competing with the R installation inside of your environment

3 Likes

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.