Phylogenize error 1 in R code

Pablo_V · April 29, 2020, 8:12pm

Hi,

I am trying to run the phylogenize plug-in but I am having problems with it. I think that there is an error with reading the burst12 binary but I have no idea how to solve it. I have checked the permissions and they seem to be alright.

This is the output

sh: /Users/pablo/Downloads/BURST-master/embalmlets/bin/burst12: cannot execute binary file
Quitting from lines 40-110 (phylogenize-report.Rmd)
Error in pz.error(paste0("BURST failed with error code ", r)) :
BURST failed with error code 126
Calls: ... read.abd.metadata -> process.16s -> run.burst -> pz.error
In addition: Warning messages:
1: In dir.create(pz.options("out_dir")) :
'/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-2am8ess8' already exists
2: In dir.create(pz.options("out_dir")) :
'/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-2am8ess8' already exists
3: In dir.create(pz.options("out_dir")) :
'/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-2am8ess8' already exists
4: In strsplit(conditionMessage(e), "\n") :
input string 1 is invalid in this locale
5: In pz.warning(paste0("sample column not found: ", opts("sample_column"), :
sample column not found: SampleID; assuming row names are sample IDs

Execution halted
Traceback (most recent call last):
File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.17.g4354584.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 150, in _run
Rcmd], check=True)
File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['R', '-e', 'sapply(c("phylogenize", "graphics", "stats", "methods","grDevices", "biomformat"), function(.) library(character.only=TRUE, .));phylogenize::set_data_internal(); phylogenize::render.report(output_file="index.html", report_input="phylogenize-report.Rmd", input_format="biom", biom_file="input.biom", burst_dir="/Users/pablo/Downloads/BURST-master/embalmlets/bin/", in_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-2am8ess8", out_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-2am8ess8", ncl=1, type="16S", which_phenotype="prevalence", which_envir="Infested", dset_column="dataset", env_column="Diagnostic", sample_column="SampleID", burst_cutoff=0.985, assume_below_LOD=TRUE, single_dset=TRUE, minimum=3, treemin=5, relative_out_dir=".", working_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-2am8ess8")']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2cli/commands.py", line 328, in call
results = action(**arguments)
File "</Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/decorator.py:decorator-gen-500>", line 2, in run
File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
output_types, provenance)
File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 452, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.17.g4354584.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 41, in run
report_file="phylogenize-report.Rmd")
File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.17.g4354584.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 152, in _run
raise Exception("Error %d in R code!" % e.returncode)
Exception: Error 1 in R code!

Plugin error from phylogenize:

Error 1 in R code!

See above for debug info.

cheers,
Pablo

Nicholas_Bokulich · April 30, 2020, 5:55pm

Hi @Pablo_V,
I have not seen this error before, pinging @pbradz to see if he can offer guidance.

In the meantime, could you post the command you used and any other relevant information to help @pbradz diagnose?

pbradz · April 30, 2020, 9:06pm

Hi @Pablo_V,

Based on this error I think that either 1) the burst12 binary doesn't have the right permissions (needs to be set to be executable: at the command line you should be able to do this with chmod +x burst12), or 2) you have a burst12 binary for the wrong architecture (e.g. you have a Linux executable but are running OS X).

It looks like the BURST binaries for Mac haven't been updated for a while, but phylogenize should still work if you download this older version into the correct directory and rename it to burst12: https://github.com/knights-lab/BURST/releases/download/v0.99.4a/burst_mac_DB12

Let me know if this helps!

-Patrick
(Thanks for the ping @Nicholas_Bokulich!)

Pablo_V · May 3, 2020, 7:09pm

Hi @Nicholas_Bokulich and @pbradz,

This is the command I used

qiime phylogenize run --i-table feature-table-filt-noblank/feature-table4-filt-noblanks.qza --i-sequences dada2-rep-set.qza --m-metadata-file feature-table-filt-noblank/metadata_rockwool_volatility.tsv --p-which-envir Infested --p-env-column Diagnostic --o-visualization phylogenize-infested --p-burst-dir /Users/pablo/Downloads/BURST-master/embalmlets/bin/ --verbose

These are the permissions for the files:
-rwxrwxrwx@ 1 pablo staff 4285 Apr 5 17:28 Readme_utils.txt
-rwxrwxrwx@ 1 pablo staff 2796728 Apr 5 17:28 akmer94b
-rwxrwxrwx@ 1 pablo staff 1858200 Apr 5 17:28 bcov
-rwxrwxrwx@ 1 pablo staff 2132432 Apr 5 17:28 burst12
-rwxrwxrwx@ 1 pablo staff 2132432 Apr 5 17:28 burst15
-rwxrwxrwx@ 1 pablo staff 14656 Apr 5 17:28 dburst
-rwxrwxrwx@ 1 pablo staff 992384 Apr 5 17:28 embalmulate
-rwxrwxrwx@ 1 pablo staff 9236 Apr 5 17:28 linfasta
-rwxrwxrwx@ 1 pablo staff 1004896 Apr 5 17:28 lingenome
-rwxrwxrwx@ 1 pablo staff 14760 Apr 5 17:28 llsim
-rwxrwxrwx@ 1 pablo staff 118 Apr 5 17:28 readme.md
-rwxrwxrwx@ 1 pablo staff 11120 Apr 5 17:28 t2gg

I am indeed running on an OS X architecture. I have downloaded the BURST for Mac (kindly provided by @pbradz but I still get the following error:

Calling BURST with arguments: -r /Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/R/library/phylogenize/extdata/16s_renamed.frn -fr -q /private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-4jac8lzh/input_seqs.txt -i 0.985 -o /private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-4jac8lzh/output_assignments.txt
 --> Also considering the reverse complement of reads.
 --> Setting identity threshold to 0.985000.
Using up to AVX-128 with 4 threads.
Parsed 196 queries (0.000625). Calculating minMax... 
Found min 252, max 254 (0.000063).
Converting queries... Converted (0.000058)
Copying queries... Copied (0.000041)
Sorting queries... Sorted (0.077710)
Copying indices... Copied (0.000068)
Determining uniqueness... Done (0.000038). Number unique: 196
Collecting unique sequences... Done (0.000043)
Creating data structures... Done (0.000085) [maxED: 3]
Processing reverse complements... Processed (0.000141) [offset 49783 / 49783]
Re-sorting... Re-sorted (0.075603)
Calculating divergence... Calculated (0.000077) [55.028061 avg div; 150 max]
Quitting from lines 40-110 (phylogenize-report.Rmd) 
Error in pz.error(paste0("BURST failed with error code ", r)) : 
  BURST failed with error code 11
Calls: <Anonymous> ... read.abd.metadata -> process.16s -> run.burst -> pz.error
In addition: Warning messages:
1: In dir.create(pz.options("out_dir")) :
  '/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-4jac8lzh' already exists
2: In dir.create(pz.options("out_dir")) :
  '/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-4jac8lzh' already exists
3: In dir.create(pz.options("out_dir")) :
  '/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-4jac8lzh' already exists
4: In strsplit(conditionMessage(e), "\n") :
  input string 1 is invalid in this locale
5: In pz.warning(paste0("sample column not found: ", opts("sample_column"),  :
  sample column not found: SampleID; assuming row names are sample IDs

Execution halted
Traceback (most recent call last):
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.17.g4354584.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 150, in _run
    Rcmd], check=True)
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['R', '-e', 'sapply(c("phylogenize", "graphics", "stats", "methods","grDevices", "biomformat"), function(.) library(character.only=TRUE, .));phylogenize::set_data_internal(); phylogenize::render.report(output_file="index.html", report_input="phylogenize-report.Rmd", input_format="biom", biom_file="input.biom", burst_dir="/Users/pablo/Downloads/", in_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-4jac8lzh", out_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-4jac8lzh", ncl=1, type="16S", which_phenotype="prevalence", which_envir="Infested", dset_column="dataset", env_column="Diagnostic", sample_column="SampleID", burst_cutoff=0.985, assume_below_LOD=TRUE, single_dset=TRUE, minimum=3, treemin=5, relative_out_dir=".", working_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-4jac8lzh")']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2cli/commands.py", line 328, in __call__
    results = action(**arguments)
  File "</Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/decorator.py:decorator-gen-500>", line 2, in run
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    output_types, provenance)
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 452, in _callable_executor_
    ret_val = self._callable(output_dir=temp_dir, **view_args)
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.17.g4354584.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 41, in run
    report_file="phylogenize-report.Rmd")
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.17.g4354584.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 152, in _run
    raise Exception("Error %d in R code!" % e.returncode)
Exception: Error 1 in R code!

Plugin error from phylogenize:

  Error 1 in R code!

See above for debug info.

Could you please let me know what to do?

Cheers,
Pablo

pbradz · May 3, 2020, 8:05pm

Hi Pablo, so error code 11 is a segmentation fault, which means that for some reason the BURST aligner is crashing on your computer. One possibility is that this is a bug in that version of BURST itself, but I have also seen something similar with certain older computers that don’t have the complete instruction set that BURST uses. I might try a "buzzard" release of BURST, which will be slower but should work on more diverse architectures, like v0.99.2-buzzard. (You will have to rename the binary to burst12.) Let me know if that fixes it.

-Patrick

Pablo_V · May 4, 2020, 1:11pm

Hi @pbradz,

I have downloaded the BURST file you suggested and renamed it "burst12". However, I still get an error

This is the output:
2-temp-ts0lw23l/input.biom
located biom file: /private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-ts0lw23l/input.biom
Calling BURST with arguments: -r /Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/R/library/phylogenize/extdata/16s_renamed.frn -fr -q /private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-ts0lw23l/input_seqs.txt -i 0.985 -o /private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-ts0lw23l/output_assignments.txt
ERROR: Unrecognized command-line option: -fr.
See help with -h
Quitting from lines 40-110 (phylogenize-report.Rmd)
Error in pz.error(paste0("BURST failed with error code ", r)) :
BURST failed with error code 1
Calls: ... read.abd.metadata -> process.16s -> run.burst -> pz.error
In addition: Warning messages:
1: In dir.create(pz.options("out_dir")) :
'/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-ts0lw23l' already exists
2: In dir.create(pz.options("out_dir")) :
'/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-ts0lw23l' already exists
3: In dir.create(pz.options("out_dir")) :
'/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-ts0lw23l' already exists
4: In strsplit(conditionMessage(e), "\n") :
input string 1 is invalid in this locale
5: In pz.warning(paste0("sample column not found: ", opts("sample_column"), :
sample column not found: SampleID; assuming row names are sample IDs

Execution halted
Traceback (most recent call last):
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.17.g4354584.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 150, in _run
    Rcmd], check=True)
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['R', '-e', 'sapply(c("phylogenize", "graphics", "stats", "methods","grDevices", "biomformat"), function(.) library(character.only=TRUE, .));phylogenize::set_data_internal(); phylogenize::render.report(output_file="index.html", report_input="phylogenize-report.Rmd", input_format="biom", biom_file="input.biom", burst_dir="/Users/pablo/Downloads/", in_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-ts0lw23l", out_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-ts0lw23l", ncl=1, type="16S", which_phenotype="prevalence", which_envir="Infested", dset_column="dataset", env_column="Diagnostic", sample_column="SampleID", burst_cutoff=0.985, assume_below_LOD=TRUE, single_dset=TRUE, minimum=3, treemin=5, relative_out_dir=".", working_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-ts0lw23l")']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2cli/commands.py", line 328, in __call__
    results = action(**arguments)
  File "</Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/decorator.py:decorator-gen-500>", line 2, in run
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    output_types, provenance)
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 452, in _callable_executor_
    ret_val = self._callable(output_dir=temp_dir, **view_args)
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.17.g4354584.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 41, in run
    report_file="phylogenize-report.Rmd")
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.17.g4354584.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 152, in _run
    raise Exception("Error %d in R code!" % e.returncode)
Exception: Error 1 in R code!

Plugin error from phylogenize:

  Error 1 in R code!

See above for debug info.

Cheers,
Pablo

pbradz · May 4, 2020, 7:49pm

Hi @Pablo_V, rats, it looks like one of the features phylogenize uses (searching reverse complements of reads) requires a newer version of BURST.

Just to make sure I'm fixing the right problem, what errors do you get if you run the following?

burst12 -r /Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/R/library/phylogenize/extdata/16s_renamed.frn -q input_seqs.fa -o test-output.txt

Running this against both versions of BURST you've tried, the old one and the newer one, would be helpful here. It would also be good to know what your exact hardware is so that if we find a workaround I can add it to the FAQ.

pbradz · May 4, 2020, 7:55pm

Whoops, forgot to provide a link to a sample input_seqs.fa -- try downloading this into the directory where burst12 lives and try running that command.

pbradz · May 5, 2020, 1:17am

Hi @Pablo_V and @Nicholas_Bokulich,

Since it looks like most of the problems people are having with phylogenize are related to getting BURST to work on Macs or on older systems, I decided to release a versions of phylogenize and q2-phylogenize that can swap in vsearch as an alternative. vsearch has more up-to-date binaries for OS X, as well as several other platforms (and I think it actually also comes as part of the QIIME2 conda environment).

If you want to use this feature, first install the latest versions of phylogenize and q2-phylogenize within the QIIME2 environment. Then, when performing your analysis, pass the option --p-burst-bin vsearch. If vsearch is installed somewhere else other than /usr/local/bin, additionally pass the option --p-burst-dir [directory]. (Try which vsearch from the command line if it's in your conda environment somewhere but you can't find it.)

I also added support for "buzzard" versions of BURST/EMBALMER (these don't know how to search both strands so they need the reverse complements to be provided manually). I haven't tested it yet but you could also try them out by setting --p-burst-bin and --p-burst-dir as above. (In that case, don't rename it to burst12 or phylogenize won't know it doesn't have certain command line options).

Let me know if these are helpful!

Nicholas_Bokulich · May 5, 2020, 2:28am

Thanks @pbradz, that sounds great! Correct, vsearch ships with the current Q2 releases as it is used by a few plugins (feature-classifier, q2-vsearch, etc), so this sounds like a robust solution.

I have marked your message as the solution to this Topic in hopes that it makes this new solution more visible for posterity.

Pablo_V · May 6, 2020, 6:23pm

Hi @pbradz,

Thank you for your patience. I ran the command but it gives me an error. It says that the burst12 command is not found.

Cheers,
Pablo

Pablo_V · May 6, 2020, 6:25pm

Hi @pbradz,

I guess that means I need to reinstall phylogenize, right?

Cheers,
Pablo

pbradz · May 6, 2020, 6:46pm

Hi @Pablo_V, yes, if you want to use vsearch or an older version of BURST, you will need to install the latest version of phylogenize and of the q2-phylogenize plugins. You won't need to re-install all of the dependencies, though, so it should go much faster this time. Instructions are on the Bitbucket sites.

Let me know if that works for you. As I mentioned before, you should be able to get the path to vsearch by running which vsearch from the command line (in the QIIME2 conda environment).

For the test command I gave you, you would need to be in the same directory where the BURST binaries are installed, or else provide the full path to the binary (for example, "/usr/local/burst12"). I'd still be curious about the output if you have time, but it may be worth just trying it with vsearch to see if that solves your problem.

Pablo_V · May 6, 2020, 7:12pm

Hi @pbradz

So I updated phylogenize following the instructions on the bitbucket website. However, it mentions that there is no --p-burst-bin option. Any idea how to solve this?

Cheers,
Pablo

pbradz · May 6, 2020, 8:05pm

Hi @Pablo_V, did you also update the q2-phylogenize QIIME plugin?

Pablo_V · May 6, 2020, 8:23pm

Hi @pbradz

I have updated the plug-in using the following command:

devtools::install_bitbucket("pbradz/phylogenize/package/phylogenize")
phylogenize::install.data.figshare()

Then I run

quiime dev refresh-cache

But still have the problem.

Cheers,
Pablo

pbradz · May 6, 2020, 10:18pm

Hi @Pablo_V, sorry if I was unclear -- the way you installed the q2 plugin originally was almost certainly from the source code, so in order to update the q2 plugin you would need to actually download the latest version of the source and rebuild it. The easiest way to do this is probably to go to the directory where you downloaded q2-phylogenize and run (from within the qiime2 environment):

git pull
python setup.py build
python setup.py install
qiime dev refresh-cache

Pablo_V · May 7, 2020, 11:41am

Hi @pbradz

I really appreciate your help and patience!!!

I re-installed phylogenize. This time the command works!

qiime phylogenize run --i-table feature-table-filt-noblank/feature-table4-filt-noblanks.qza --i-sequences dada2-rep-set.qza --m-metadata-file feature-table-filt-noblank/metadata_rockwool_volatility.tsv --p-which-envir Infested --p-env-column Diagnostic --o-visualization phylogenize-infested --p-burst-bin vsearch --p-burst-dir /Users/pablo/opt/anaconda2/envs/qiime2-2020.2/bin/ --verbose

But it gives me an error.

115259510 nt in 99573 seqs, min 61, max 3038, avg 1158
minseqlength 32: 14 sequences discarded.
Masking 100%  
Counting k-mers 100%  
Creating k-mer index 100%  
Searching 100%  
Matching query sequences: 107 of 196 (54.59%)
2 non-singleton environment(s) found
1 non-singleton dataset(s) found
Determining which phyla to test...
tenericutes (pct): 0; (number): 0; dropped
cyanobacteria (pct): 0; (number): 0; dropped
firmicutes (pct): 0.00604229607250755; (number): 8; dropped
actinobacteria (pct): 0.00518672199170124; (number): 5; dropped
proteobacteria (pct): 0.0186335403726708; (number): 42; dropped
acidobacteria (pct): 0; (number): 0; dropped
bacteroidetes (pct): 0.0099601593625498; (number): 5; dropped
verrucomicrobia (pct): 0; (number): 0; dropped
synergistetes (pct): 0; (number): 0; dropped
chloroflexi (pct): 0; (number): 0; dropped
delta.epsilon.subdivisions (pct): 0; (number): 0; dropped
aquificae (pct): 0; (number): 0; dropped
planctomycetes (pct): 0; (number): 0; dropped
spirochaetes (pct): 0; (number): 0; dropped
chlamydiae (pct): 0; (number): 0; dropped
fusobacteria (pct): 0; (number): 0; dropped
chlorobi (pct): 0; (number): 0; dropped
deinococcus.thermus (pct): 0; (number): 0; dropped
thermotogae (pct): 0; (number): 0; dropped
Quitting from lines 40-110 (phylogenize-report.Rmd) 
Error in pz.error(paste0("All trees had less than ", opts("pctmin"), " percent of taxa observed. Either a very small ASV ",  : 
  All trees had less than 0.025 percent of taxa observed. Either a very small ASV table was provided, read depth was very shallow, the right database was not selected or very few ASVs mapped to entries in the database.
Calls: <Anonymous> ... withVisible -> eval -> eval -> adjust.db -> pz.error
In addition: Warning messages:
1: In dir.create(pz.options("out_dir")) :
  '/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-48ygxen_' already exists
2: In dir.create(pz.options("out_dir")) :
  '/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-48ygxen_' already exists
3: In dir.create(pz.options("out_dir")) :
  '/private/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-48ygxen_' already exists
4: In strsplit(conditionMessage(e), "\n") :
  input string 1 is invalid in this locale
5: In pz.warning(paste0("sample column not found: ", opts("sample_column"),  :
  sample column not found: SampleID; assuming row names are sample IDs
6: In data.table::fread(file.path(opts("data_dir"), "family.functions"),  :
  Found and resolved improper quoting out-of-sample. First healed line 5236: <<FIG00039628	"phi-Carotenoid synthase" (EC 1.3.-.- and EC 2.1.1-)>>. If the fields are not quoted (e.g. field separator does not appear within any field), try quote="" to avoid this warning.
7: In data.table::fread(file.path(opts("data_dir"), "subsys.txt"),  :
  Found and resolved improper quoting out-of-sample. First healed line 2114: <<Fatty Acids, Lipids, and Isoprenoids	Isoprenoids	Carotenoids	"phi-Carotenoid synthase" (EC 1.3.-.- and EC 2.1.1-)	>>. If the fields are not quoted (e.g. field separator does not appear within any field), try quote="" to avoid this warning.

Execution halted
Traceback (most recent call last):
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.18.gcd393dc.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 157, in _run
    Rcmd], check=True)
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['R', '-e', 'sapply(c("phylogenize", "graphics", "stats", "methods","grDevices", "biomformat"), function(.) library(character.only=TRUE, .));phylogenize::set_data_internal(); phylogenize::render.report(output_file="index.html", report_input="phylogenize-report.Rmd", input_format="biom", biom_file="input.biom", burst_dir="/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/bin/", burst_bin="vsearch", in_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-48ygxen_", out_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-48ygxen_", ncl=1, type="16S", which_phenotype="prevalence", which_envir="Infested", dset_column="dataset", env_column="Diagnostic", sample_column="SampleID", burst_cutoff=0.985, assume_below_LOD=TRUE, single_dset=TRUE, minimum=3, treemin=5, relative_out_dir=".", working_dir="/var/folders/3q/m5mw83016g1b5spvgr6444yr0000gn/T/qiime2-temp-48ygxen_")']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2cli/commands.py", line 328, in __call__
    results = action(**arguments)
  File "</Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/decorator.py:decorator-gen-500>", line 2, in run
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    output_types, provenance)
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 452, in _callable_executor_
    ret_val = self._callable(output_dir=temp_dir, **view_args)
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.18.gcd393dc.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 43, in run
    report_file="phylogenize-report.Rmd")
  File "/Users/pablo/opt/anaconda2/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_phylogenize-0+untagged.18.gcd393dc.dirty-py3.6.egg/q2_phylogenize/_phylogenize.py", line 159, in _run
    raise Exception("Error %d in R code!" % e.returncode)
Exception: Error 1 in R code!

What should I do?

Cheers,
Pablo

pbradz · May 7, 2020, 7:52pm

Hi @Pablo_V, awesome, looks like things are finally working! Glad to be able to help

Okay, so this at least is not a technical error. Instead, phylogenize is complaining that relatively few ASVs were mapped to species in its database. Buried in there is the error text "All trees had less than 0.025 percent of taxa observed. Either a very small ASV table was provided, read depth was very shallow, the right database was not selected or very few ASVs mapped to entries in the database." (Incidentally, that should actually read 2.5% -- I've pushed a fix for that.)

Because phylogenize does comparisons across trees, if a given phylum-level tree only has a small number of species detected, the sample size can get too small to make accurate predictions. You can see that for all of these phylum-level trees, fewer than 2.5% of the leaves (species) were detected in your samples, and for most of them, you only have 5 or fewer leaves detected.

This can happen for a few reasons. One is that your read depth is relatively shallow, and therefore only a small number of species are being detected. This number of ASVs (196) is a little on the low side compared to tests I've personally run. Another is that the communities you sampled are not very diverse overall, so it's hard to correct for phylogeny. Yet another possibility is that the microbes in your environment have really terrible representation in the reference database, which could happen if you were studying some really unusual environment. I don't think this is true in your case since around half of your ASVs are getting mapped, which is decent considering the strict 98.5% sequence ID cutoff.

Finally, it could just be that phylogenize is being a little over-cautious about your data. I think this is probably one of the contributing issues, since from this, it does look like you could expect reasonable results from at least the Proteobacteria. There are 42 species detected, it's just that the (alpha-/beta-/gamma-)Proteobacterial tree is so huge that percentage-wise, it's still falling below the default cutoff of 2.5%.

Unfortunately it looks like I didn't expose the option to change the minimum species percentage in the QIIME2 plugin, but I've just added it in. If you re-download and install the latest version of the plugin there should be a new option --p-pctmin, which you can set to 0 to turn it off. (If you really want to, you can also loosen the filter on the number of species detected per tree by setting --p-treemin to 2 -- I wouldn't put it any lower, and I'd be pretty skeptical about interpreting phyla with that few detected representatives.)

The good news is that you won't need to reinstall phylogenize itself, just the q2-phylogenize plugin, following the same instructions as before (unless of course you really, really want those percentages to display as percentages instead of fractions in the log...).

I took the opportunity to make a couple of other improvements as well. The default in q2-phylogenize is now vsearch instead of BURST. Also, if you leave --p-burst-dir blank (as is now the default instead of /usr/local/bin), q2-phylogenize will now automatically search your PATH for the vsearch/BURST executable, so you shouldn't have to specify it as long as you're in the QIIME2 environment. This should make things even a little more straightforward for QIIME2 users of phylogenize.

Let me know if this works for you!

Pablo_V · May 11, 2020, 12:43pm

Hi @pbradz,

Thanks a lot for your detailed response!

So my ASV table has about 196 features and the samples come from hydroponic plants. Would you consider this a rather not very diverse environment?

Also, how could I check if my read depth is shallow?

I will re-install q2-phylogenize and try and run it again with --p-pctmin 0.

Cheers,
Pablo