Fragment-insertion sepp returned non-zero exit status

Hi @nickscales,

you are using your own reference, i.e. a phylogeny and an alignment. Unfortunately, a proper reference set also needs to contain a raxml.info file (this is not yet documented, thus you could not know). Here are example commands how to properly create phylogeny and info file from alignment: https://github.com/smirarab/sepp-refs/tree/master/silva
However, the current plugin version cannot accept info-file paths as inputs :-/ I am working on an PR to change this in the future: https://github.com/qiime2/q2-fragment-insertion/pull/32 and like to make this the right way by also creating a bioconda package for the underlying SEPP program: https://github.com/bioconda/bioconda-recipes/pull/14233 which at the moment is stalled due to some osx dendropy dependency issues.

Thus, to make it work for you, you need to “hack” a little. First you need to come up with this info file. Second, you can either replace the default info file (should be found as $CONDA_PREFIX/share/fragment-insertion/ref/RAxML_info-reference-gg-raxml-bl.info) or pull my PR https://github.com/qiime2/q2-fragment-insertion/pull/32 and provide the file path as further input argument.

Hope this helps until we merge this addition into master and make it generally available.

Stefan

3 Likes

Hi @Stefan thank you SO much for your reply! I have put my RAxML info in /share/fragment-insertion/ref and I named it the same as the previous file, but I keep getting the same error:

Blockquote

Plugin error from fragment-insertion:

Reference alignment and phylogeny do not match up. Please ensure that all sequences in the alignment correspond to exactly one tip name in the phylogeny.

Blockquote

I have checked a few times and I'm fairly sure they are matching up, I even tried shortening the names but to no avail - do you know what might be happening?

RAxML_rooted.nwk (4.9 KB)
RAxML_info.txt (34.7 KB)
seqs.fastq (107.1 KB)

Hi @nickscales,

I don’t fully understand why, but when I import your tree into an Qiime2 artifact and load this with my function that checks if alignment and tree matches up, I see that the names loose their underscores, e.g. the identifier Curtobacterium_sp._MCBA15_001 is read as Curtobacterium sp. MCBA15 001 - although it has the underscores in the Newick file!!

@Nicholas_Bokulich is that change of identifiers a bug or a feature?
(Update: looks like skbio.TreeNode.read is responsible for ID transformation)

My suggestion for a workaround is that you remove all _ from the identifiers in the Newick and the Alignment file and try again.

Best
Stefan

Hi @Stefan,

Many thanks again for your help. Removing the underscores in the Newick file has indeed stopped the names not matching error. However, frustratingly, we now have the exact same error I was seeing originally:

Blockquote
Traceback (most recent call last):
File "/data/users/nscales/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2cli/commands.py", line 274, in call
results = action(**arguments)
File "</data/users/nscales/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/decorator.py:decorator-gen-294>", line 2, in sepp
File "/data/users/nscales/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
File "/data/users/nscales/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/action.py", line 365, in callable_executor
output_views = self._callable(**view_args)
File "/data/users/nscales/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_fragment_insertion/_insertion.py", line 179, in sepp
reference_alignment, reference_phylogeny, debug)
File "/data/users/nscales/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_fragment_insertion/_insertion.py", line 137, in _run
subprocess.run(cmd, check=True, cwd=cwd)
File "/data/users/nscales/miniconda3/envs/qiime2-2019.1/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run-sepp.sh', '/scratch/6520860.1.mic/qiime2-archive-knar6piz/e4ba8d00-30b4-46c9-8e40-d0b8f3820351/data/dna-sequences.fasta', 'q2-fragment-insertion', '-x', '1', '-A', '1000', '-P', '5000', '-a', '/scratch/6520860.1.mic/qiime2-archive-7ozvg0r7/b199d679-1701-4100-a352-21e0e74790ec/data/aligned-dna-sequences.fasta', '-t', '/scratch/6520860.1.mic/qiime2-archive-xgivba8f/333eb21f-35bf-41c3-aaa8-f70d5645c0a0/data/tree.nwk']' returned non-zero exit status 1.

Blockquote
Plugin error from fragment-insertion:

Blockquote
Command '['run-sepp.sh', '/scratch/6520860.1.mic/qiime2-archive-knar6piz/e4ba8d00-30b4-46c9-8e40-d0b8f3820351/data/dna-sequences.fasta', 'q2-fragment-insertion', '-x', '1', '-A', '1000', '-P', '5000', '-a', '/scratch/6520860.1.mic/qiime2-archive-7ozvg0r7/b199d679-1701-4100-a352-21e0e74790ec/data/aligned-dna-sequences.fasta', '-t', '/scratch/6520860.1.mic/qiime2-archive-xgivba8f/333eb21f-35bf-41c3-aaa8-f70d5645c0a0/data/tree.nwk']' returned non-zero exit status 1.

Does that mean the error is to do with my info file perhaps?

Kind regards,
Nick

Could you please re-run with additional command line parameters --p-debug --verbose hoping to get more verbose error messages. If you don’t mind sending me at least a part of your input sequences, I could start debugging on my end.

Looking through the error message, it looks like the key part is

Blockquote
Warning: using a statistics file directly is now deprecated. We suggest using a reference package. If you already are, then please use the latest version of taxtastic.
WARNING: your stats file is from RAxML 8.2.12; RAxML has been tested with the following versions: 7.0.4; 7.2.3; 7.2.5; 7.2.6; 7.2.7
I'm going to try parsing as if this was version 7.2.3Problem parsing info or stats file/data/users/nscales/miniconda3/envs/qiime2-2019.1/share/fragment-insertion//ref/RAxML_info-reference-gg-raxml-bl.info

Blockquote
error:Uncaught exception: Parse_stats.Stats_parsing_error("too many partitions. Only one is allowed.")
Fatal error: exception Parse_stats.Stats_parsing_error("too many partitions. Only one is allowed.")

Which seems as though it is the info file I added. I actually do have a reference package as I was trying to do this using pplacer before I saw it could be done in qiime - is it possible to incorporate that? What does the "too many partitions" error mean, and how can I fix it?

Thank you!

fragment-insertion.o6534112.txt (26.9 KB)
RAxML_info.txt (34.7 KB)

Good morning. We are now leaving my comfort zone and I’d like to refer to SEPP’s developer :-/ As a first hint, you might want to see what we have done to create a reference (tree/alignment/info) for Silva 12.8: https://github.com/smirarab/sepp-refs/tree/master/silva as said earlier.
Another idea would be to take our default info file and try to adapt necessary value to your - but I don’t know which they are, neither am I familiar with the info file format.

1 Like

I have found a solution! At https://github.com/smirarab/sepp/blame/master/tutorial/sepp-tutorial.md#L451 they mention that you need to remove a line in my info file that read: ‘Partition: 0 with name: No Name Provided’ and this has let the script finish once I put that back in share/fragment-insertion/ref/

Thanks so much for your help - have a great day!

3 Likes

That is amazing and a very good catch - reading the documentation sometimes really helps :wink: I am sure this answer will also help others struggling with similar situations. Thanks a lot!

2 Likes

Hi all, I'm running into the same error with qiime2-2019.7:

qiime fragment-insertion sepp \
  --i-representative-sequences reads_qza/rep-seqs.qza \
  --o-tree insertion-tree.qza \
  --o-placements insertion-placements.qza



Plugin error from fragment-insertion:

  Command '['run-sepp.sh', '/tmp/qiime2-archive-lhc6w0_e/1e77bc4a-a27c-435d-8dbb-f790b01da188/data/dna-sequences.fasta', 'q2-fragment-insertion', '-x', '1', '-A', '1000', '-P', '5000']' returned non-zero exit status 1.

Debug info has been saved to /tmp/qiime2-q2cli-err-k13lb247.log

I'm working on a server with 8 TB of space and 64 processors. I have tried this using a different temp folder to avoid running out of space. E.g.:

mkdir qiime2-tmp/
export TMPDIR="$PWD/qiime2-tmp/"

And I've also tried manually editing the RAxML file here as indicated in @nickscales solution:

~/miniconda3/envs/qiime2-2019.7/share/fragment-insertion/ref/RAxML_info-reference-gg-raxml-bl.info

But it didn't have the line mentioned:

Partition: 0 with name: No Name Provided

But it did have this text:

Partition: 0
Alignment Patterns: 1285
Name: No Name Provided
DataType: DNA
Substitution Matrix: GTR

Does anyone have any insights here? Am I just being silly and not editing the right file?Log_file.txt (1.9 KB)

1 Like

Here's an update:

I thought perhaps the qza file was just too large (17345 ASVs, 1.6 mb) so I filtered it down to only the ASVs we were interested in using this:

qiime feature-table filter-seqs \
	--i-data reads_qza/rep-seqs-joined.qza \
	--m-metadata-file filt_seqs.csv \
	--o-filtered-data rep-seqs-joined_for_tree.qza

The file was now down to 823 kb (7561 ASVs), but I'm still getting the same error:

Plugin error from fragment-insertion:

  Command '['run-sepp.sh', '/u1/sdm231/canola/qiime2-tmp/qiime2-archive-jj101gil/14aae9b3-4c59-43b1-aab2-9800e1986ae9/data/dna-sequences.fasta', 'q2-fragment-insertion', '-x', '1', '-A', '1000', '-P', '5000']' returned non-zero exit status 1.

Debug info has been saved to /u1/sdm231/canola/qiime2-tmp/qiime2-q2cli-err-yjif63be.log

Here's the log file:

Removing /u1/sdm231/canola/qiime2-tmp/tmp.qQPH2SqCec/sepp-tmp-OLEy3
Traceback (most recent call last):
  File "/home/sdm231/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py", line 327, in __call__
    results = action(**arguments)
  File "</home/sdm231/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-299>", line 2, in sepp
  File "/home/sdm231/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 240, in bound_callable
    output_types, provenance)
  File "/home/sdm231/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 383, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/sdm231/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_fragment_insertion/_insertion.py", line 179, in sepp
    reference_alignment, reference_phylogeny, debug)
  File "/home/sdm231/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_fragment_insertion/_insertion.py", line 137, in _run
    subprocess.run(cmd, check=True, cwd=cwd)
  File "/home/sdm231/miniconda3/envs/qiime2-2019.7/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run-sepp.sh', '/u1/sdm231/canola/qiime2-tmp/qiime2-archive-mrlgbde3/88ef22e1-906f-4338-b8fe-acd59ef74bd9/data/dna-sequences.fasta', 'q2-fragment-insertion', '-x', '1', '-A', '1000', '-P', '5000']' returned non-zero exit status 1.

I'm not sure why this is failing—I don't think it's a space issue, but can't seem to figure out the real problem. It's something to do with the run-sepp.sh, but what?

I've attached the file here in case someone can run it on their system using:

qiime fragment-insertion sepp \
  --i-representative-sequences rep-seqs-joined_for_tree.qza \
  --o-tree insertion-tree.qza \
  --o-placements insertion-placements.qza

Help?

rep-seqs-joined_for_tree.qza (822.4 KB)

1 Like

Hi @Steven_Mamet - I am working on making some updates to this plugin right now, and would be happy to take a closer look at this. It might be a few days before I get back to you - hold tight!

2 Likes

Thanks @thermokarst! I should update that I’m trying to use SILVA132 database and went down quite a rabbit hole trying to make this work (many forums, but started here: Suggestion - SILVA v132 for q2-fragment-insertion). So I realize I may be trying to do something (prematurely) that will be available in a future qiime2 release, but do appreciate any insights you have once you have time. Thanks again!

1 Like

Thanks @Steven_Mamet - just wanted to check in here and let you know this is still on my radar.

Hey @Steven_Mamet - my best guess is that this was related to an out-of-memory error, although I can't say for sure.

placements.qza (6.0 MB) tree.qza (2.7 MB)

(note: these were computed using a development version of QIIME 2, provenance will reflect that)

Good news - this is slated to land in QIIME 2 2019.10 (we are trying to release that this week). The plan is to allow for SILVA 128 or GG 13.8.

2 Likes

Hi @thermokarst, thanks SO much for running this! The tree looks great. I’m hoisting a digital beverage for you :beers:!

I’ll keep an eye out for the latest version of qiime2. We will certainly be revisiting this plugin for other datasets as they come down the pipeline.

Thanks again!

1 Like