Alignment error message

thermokarst · January 23, 2018, 11:01pm

Hi @PabloCL — I think your only option is to increase the available memory. Are you running on a Mac or Linux machine? If so, you could try to do a native install, this will give you the most available memory. Another option is to use a cloud service, like AWS. Keep us posted!

cesteban · February 16, 2018, 10:36am

Hello, Matthew.

I’m having the same problem, but in my case I tried to run mafft in a workstation with 128GB of RAM, I thought that ir’s enought to run it, but not.
Is it possible that error is caused by other issue?
Thanks in advance

thermokarst · February 18, 2018, 9:57pm

Hi @cesteban - it certainly is possible — we would need to see the following info to start tracking down your problem, though:

What version of QIIME 2 are you currently running?
What is the exact command you are running? Copy and paste please.
What is the exact error you are seeing? Please copy and paste the results when run with the --verbose flag, or, attach the error log file reported at the end of the error message when run without the --verbose flag.

Thanks!

cesteban · February 19, 2018, 3:00pm

Hello Matthew,
I updated to version qiime2-2018.2 and I got a similar error.
The commando that I used is:
qiime alignment mafft --p-n-threads 24 --i-sequences 0.mergeALLreads_gg_rep_set.fasta.qza --o-alignment 2.mergeALLreads_gg_rep_set_aligned.fasta.qza --verbose

Here, i attach the log file:

alignmentMAFFT_20180219_1136.log.txt (2.9 MB)

Thanks in advance

thermokarst · February 19, 2018, 5:57pm

Thanks @cesteban! This is a tricky one - we aren’t the developers of MAFFT, so we aren’t experts on how it runs (and more specifically, how it fails!), so it might take us some time to understand what is happening. In the meantime, perhaps the MAFFT team has a support venue you can contact? As well, can you please send along your data - we might be able to experiment with some of the runtime options MAFFT exposes for low-mem environments. Feel free to send a link in a DM to myself, or any other Forum Admin. Thanks!

juanma · March 9, 2018, 3:14pm

Hi Matthew,

Any update about this issue?

I am facing same problem with my study. I use a VM 2018.02, allocating 14Gb.

I have run:

qiime2-5_Chapter_1_the_Ithaca_experiment$ qiime alignment mafft
--i-sequences ref_hit_seq_ithaca.qza
--o-alignment 3_aligned-rep-seqs.qza
--p-n-threads -1
--verbose

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.
Command: mafft --preservecase --inputorder --thread -1 /tmp/qiime2-archive-anck6zw_/d88b53b5-c8ca-4118-b4c6-fe65b78c5c69/data/dna-sequences.fasta
OS = linux
The number of physical cores = 1
inputfile = orig
67357 x 90 - 90 d
nthread = 1
stacksize: 8192 kb->13155 kb
generating a scoring matrix for nucleotide (dist=200) ... done
Gap Penalty = -1.53, +0.00, +0.00
Making a distance matrix ..
14001 / 67357 (thread 0)/home/qiime2/miniconda/envs/qiime2-2018.2/bin/mafft: line 2440: 2819 Killed "$prefix/disttbfast" -q $npickup -E $cycledisttbfast -V "-"$gopdist -s $unalignlevel $legacygapopt $mergearg -W $tuplesize $termgapopt $outnum $addarg $add2ndhalfarg -C $numthreadstb $memopt $weightopt $treeinopt $treeoutopt $distoutopt $seqtype $model -f "-"$gop -Q $spfactor -h $aof $param_fft $algopt $treealg $scoreoutarg < infile
pre 2>> "$progressfile"
Plugin error from alignment:
Command '['mafft', '--preservecase', '--inputorder', '--thread', '-1', '/tmp/qiime2-archive-anck6zw_/d88b53b5-c8ca-4118-b4c6-fe65b78c5c69/data/dna-sequences.fasta']' returned non-zero exit status 1

My rep-seq file is 705.3 Kb.

Thanks in advace!

Juanma

thermokarst · March 9, 2018, 9:02pm

Hi @juanma!

Please see my comment above.

badhikari · March 20, 2018, 12:37pm

Dear @thermokarst

I am wondering whether you have found some solution regarding previous discussion? I have also got same type of error while doing alignment which is as follows;
qiime alignment mafft --i-sequences …/rep-seqs-dn-99.qza --o-alignment aligned_mafft-ref-seqs.qza --verbose --p-n-threads 24
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: mafft --preservecase --inputorder --thread 24 /tmp/qiime2-archive-vclzld9_/74cea942-b3a9-4bc2-9b97-65d7cc42d988/data/dna-sequences.fasta

inputfile = orig
230857 x 580 - 200 d
nthread = 24
stacksize: 8192 kb->45089 kb
generating a scoring matrix for nucleotide (dist=200) … done
Gap Penalty = -1.53, +0.00, +0.00

Making a distance matrix …
18701 / 230857 (thread 23)/home/bxa015/miniconda3/envs/qiime2-2017.12/bin/mafft: line 2440: 10107 Killed “$prefix/disttbfast” -q $npickup -E $cycledisttbfast -V “-”$gopdist -s $unalignlevel $legacygapopt $mergearg -W $tuplesize $termgapopt
$outnum $addarg $add2ndhalfarg -C $numthreadstb $memopt $weightopt $treeinopt $treeoutopt $distoutopt $seqtype $model -f “-”$gop -Q $spfactor -h $aof $param_fft $algopt $treealg $scoreoutarg < infile > pre 2>> “$progressfile”
Traceback (most recent call last):
File “/home/bxa015/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2cli/commands.py”, line 224, in call
results = action(**arguments)
File “”, line 2, in mafft
File “/home/bxa015/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 228, in bound_callable
output_types, provenance)
File “/home/bxa015/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 363, in callable_executor
output_views = self._callable(**view_args)
File “/home/bxa015/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_alignment/_mafft.py”, line 61, in mafft
run_command(cmd, aligned_fp)
File "/home/bxa015/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_alignment/mafft.py", line 27, in run_command
subprocess.run(cmd, stdout=output_f, check=True)
File “/home/bxa015/miniconda3/envs/qiime2-2017.12/lib/python3.5/subprocess.py”, line 398, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '[‘mafft’, ‘–preservecase’, ‘–inputorder’, ‘–thread’, ‘24’, '/tmp/qiime2-archive-vclzld9/74cea942-b3a9-4bc2-9b97-65d7cc42d988/data/dna-sequences.fasta’]’ returned non-zero exit status 1

Plugin error from alignment:

Command ‘[‘mafft’, ‘–preservecase’, ‘–inputorder’, ‘–thread’, ‘24’, ‘/tmp/qiime2-archive-vclzld9_/74cea942-b3a9-4bc2-9b97-65d7cc42d988/data/dna-sequences.fasta’]’ returned non-zero exit status 1

ebolyen · March 20, 2018, 5:58pm

Hi @badhikari (and others in the thread),

I don't think we have any progress to report.

One thing I did notice in your filename:

Were these sequences picked by closed-reference by chance? If so, the reference database almost certainly has a much better phylogeny than what you are going to get from MAFFT+FastTree.

I'll also plug @Stefan's plugin q2-fragment-insertion which could also be a good way to go.

At the end of the day, the goal is usually to get some kind of reasonable phylogeny (Phylogeny[Rooted]), so if MAFFT+FastTree isn't working, other strategies may work better (and be more meaningful).

Hope that helps!

badhikari · March 21, 2018, 2:36pm

Hi @ebolyen

Thank you for your response. Actually, I have used denovo otu picking instead of closed one.

thermokarst · March 27, 2018, 2:29pm

An off-topic reply has been split into a new topic: Q2-fragment-insertion runtime

Please keep replies on-topic in the future.

abarron · April 10, 2018, 1:49pm

Hi, I’m experiencing the same issue. Here is the error message:

 qiime alignment mafft --i-sequences uchime-dn-out/rep-seqs-nonchimeric-dn.qza --p-n-threads 12 --output-dir alignment_out --verbose 
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: mafft --preservecase --inputorder --thread 12 /tmp/qiime2-archive-ufnc8jas/ed294b2b-71c6-4548-85f9-55aa95c4a6d8/data/dna-sequences.fasta


inputfile = orig
236215 x 534 - 35 d
nthread = 12
stacksize: 8192 kb->46135 kb
generating a scoring matrix for nucleotide (dist=200) ... done
Gap Penalty = -1.53, +0.00, +0.00



Making a distance matrix ..
/home/aescalante/miniconda3/envs/qiime2-18.2/bin/mafft: line 2440: 21139 Killed                  "$prefix/disttbfast" -q $npickup -E $cycledisttbfast -V "-"$gopdist -s $unalignlevel $legacygapopt $mergearg -W $tuplesize $termgapopt $outnum $addarg $add2ndhalfarg -C $numthreadstb $memopt $weightopt $treeinopt $treeoutopt $distoutopt $seqtype $model -f "-"$gop -Q $spfactor -h $aof $param_fft $algopt $treealg $scoreoutarg < infile > pre 2>> "$progressfile"
Plugin error from alignment:

  Command '['mafft', '--preservecase', '--inputorder', '--thread', '12', '/tmp/qiime2-archive-ufnc8jas/ed294b2b-71c6-4548-85f9-55aa95c4a6d8/data/dna-sequences.fasta']' returned non-zero exit status 1

thermokarst · April 10, 2018, 5:04pm

Hi @abarron - sorry, we don’t have anything to report on this end — that error is coming from mafft, not from QIIME 2. I would suggest filing a bug report with the mafft team to see if they can provide some assistance. My best guess, at least according to earlier reports is that this is related to insufficient memory on the machine you are running this command on. With that said, it sounds like the memory requirements in some cases might be pretty crazy. It is possible that mafft isn’t behaving when you specify so many threads (the memory requirement might be increasing by a factor of 12, but I am not sure). I am sorry I don’t have a better answer for you, but if you want more assistance from us, do you think you could provide us with download links to your data, that way we can re-run the command you provided above? Thanks!

system · May 11, 2018, 11:05pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.