# I'm loosing too many OTUs with taxa filter-table (~80%)--is this a normal/common phenomenon?
**URL:** https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071
**Category:** General Discussion
**Tags:** taxonomy, feature-classifier, taxa, filtering
**Created:** [January 30, 2019, 1:18am UTC](https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071 "2019-01-30T01:18:10Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@halt\_BB](https://forum.qiime2.org/u/halt_BB)
#### Post date: [January 30, 2019, 1:18am UTC](https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071/1 "2019-01-30T01:18:10Z")
Dear QIIME2 users,
I have a few general questions about filtering 16S data. I understand this is not a QIIME2 software problem, but I’d appreciate insights from senior biologists/users of QIIME2 working with 16S data on this.
I have Illumina 16S data (V4 with 515-806 primers), and I’m running QIIME2-2018.11 to examine microbial abundance and diversity in my samples. I used SILVA132 (99\_16S.fna and 99\_taxonomy\_7\_levels.txt) as the reference database and taxonomy database.
$time qiime feature-classifier classify-consensus-blast
--i-query fig2a/rep-seqs.qza
--i-reference-taxonomy fig2a/silva132\_taxonomy.qza
--i-reference-reads fig2a/silva132-db.qza
--o-classification fig2a/classify2a
--p-perc-identity 0.90
--p-maxaccepts 1
--verbose
I’ve read about taxa filtering here [https://docs.qiime2.org/2018.11/tutorials/filtering/](https://docs.qiime2.org/2018.11/tutorials/filtering/) , and on this thread [https://forum.qiime2.org/t/high-yield-of-d-2-alphaproteobacteria-d-3-rickettsiales-d-4-mitochondria-in-samples-from-wild-bats-artifact/7161](https://forum.qiime2.org/t/high-yield-of-d-2-alphaproteobacteria-d-3-rickettsiales-d-4-mitochondria-in-samples-from-wild-bats-artifact/7161).
After classification I scanned my taxonomy-to-tsv file and I knew I had to filter “Unassigned” in order to improve “relative abundance” in the samples downstream. To my dismay, “ **Unassigned” were about 65%!!!** How is this possible? Any suggestions for improving taxonomic assignment? Because I thought silva\_132\_99\_16S.fna was appropriate for this (I could be wrong!). Like what is being suggested here several random posts online [https://www.researchgate.net/post/Can\_anyone\_help\_with\_pulling\_specific\_sequences\_that\_correspond\_to\_OTU\_IDs\_from\_Qiime](https://www.researchgate.net/post/Can_anyone_help_with_pulling_specific_sequences_that_correspond_to_OTU_IDs_from_Qiime)
Also I have quite other assignments (with metagenome approx.. 17%), do I need to filter these out too? Because I’m interested in Family, Genus and Species assignments yet all this stuff with metagenome (though it hits bacteria) is only classified up to Phylum or Class. If I do, I’m worried I’ll retain only 16% of the original data!
Here:
 
And lastly, I wanted to confirm if some of the OTUs which are “Unassigned” are actually Unassigned by doing a quick ncbi blast. How do I get the sequences corresponding to the specific OTU Ids from my silva\_132\_99\_16S.fna file? I’ve noted that filter\_fasta.py (in QIIME1) could do this, but I’m running QIIME2. Where would I find an equivalent of that script in QIIME2?
If any of my questions are too basic, my apologies. I am at my wits end (almost 2 weeks on this) and I appreciate all the help I can get. This forum has been my life-saver on all things QIIME2!!
Thanks for your help.
---
### Author:  [@Nicholas\_Bokulich](https://forum.qiime2.org/u/Nicholas_Bokulich)
#### Post date: [January 30, 2019, 6:41pm UTC](https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071/2 "2019-01-30T18:41:02Z")
Hi @halt_BB,
This sounds like you genuinely have a large fraction of non-target sequences and the high level of filtering is warranted. Of course the classifier impacts this, but you are using a very permissive classifier.
> [@halt\_BB](#):
>
> $time qiime feature-classifier classify-consensus-blast
> –i-query fig2a/rep-seqs.qza
> –i-reference-taxonomy fig2a/silva132\_taxonomy.qza
> –i-reference-reads fig2a/silva132-db.qza
> –o-classification fig2a/classify2a
> –p-perc-identity 0.90
> –p-maxaccepts 1
> –verbose
You are telling BLAST to classify each sequence to the taxonomy of the first reference sequence with ≥ 90% similarity. This is generally a bad idea: the classifications will not be precise, and will be very prone to over-classification. You should use a higher perc-identity and higher maxaccepts if you use the blast-LCA classifier.
But the poor classifier choice will actually make it _less_ likely for your query sequences to be unclassified... effectively, those 65% of sequences that are unclassified have no matches that are at least 90% similar.
> [@halt\_BB](#):
>
> Also I have quite other assignments (with metagenome approx… 17%), do I need to filter these out too? Because I’m interested in Family, Genus and Species assignments yet all this stuff with metagenome (though it hits bacteria) is only classified up to Phylum or Class. If I do, I’m worried I’ll retain only 16% of the original data!
Use a different classifier or different parameters. I think this particular classification stems from your parameter choices.
> [@halt\_BB](#):
>
> I wanted to confirm if some of the OTUs which are “Unassigned” are actually Unassigned by doing a quick ncbi blast.
see the tabulate command in this section:
[https://docs.qiime2.org/2018.11/tutorials/moving-pictures/#taxonomic-analysis](https://docs.qiime2.org/2018.11/tutorials/moving-pictures/#taxonomic-analysis)
> [@halt\_BB](#):
>
> How do I get the sequences corresponding to the specific OTU Ids from my silva\_132\_99\_16S.fna file? I’ve noted that filter\_fasta.py (in QIIME1) could do this, but I’m running QIIME2. Where would I find an equivalent of that script in QIIME2?
See [Evaluating and controlling data quality with q2-quality-control — QIIME 2 2018.11.0 documentation](https://docs.qiime2.org/2018.11/tutorials/quality-control/#evaluating-sequence-quality)
good luck!
---
### Author:  [@halt\_BB](https://forum.qiime2.org/u/halt_BB)
#### Post date: [February 1, 2019, 1:02pm UTC](https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071/3 "2019-02-01T13:02:17Z")
Hi @Nicholas_Bokulich,
Thank you for the detailed explanation on my current predicament, I really appreciate.
> Blockquote This sounds like you genuinely have a large fraction of non-target sequences and the high level of filtering is warranted. Of course the classifier impacts this, but you are using a very permissive classifier.
I think I agree with you on the high percent of "Unassigned", although I've been in denial 😭 hoping I was wrong! The sequencing facility characterized most of my samples as having poor quality DNA (oh..that's after they sent me totally different sequencing results which turned out to be for a colleague of mine in the same lab!). Unfortunately, i can't do anything about DNA quality since these are field samples collected at specific time points.
> Blockquote But the poor classifier choice will actually make it _less_ likely for your query sequences to be unclassified… effectively, those 65% of sequences that are unclassified have no matches that are at least 90% similar.
Indeed I was being overly permissive (still stuck in the BLAST age 🙊).
I am going to try the new qiime q2 feature-classifier with the pretrained silva-132-99-515-806-nb-classifier.qza and hopefully with more stringent filtering, this gives me better assignment ..as detailed here [https://peerj.com/preprints/3208/](https://peerj.com/preprints/3208/)
I'll look at the resources on OTU picking and evaluating quality.
Feedback in 2 days 🤞
Again, many thanks @Nicholas_Bokulich!
---
### Author:  [@thermokarst](https://forum.qiime2.org/u/thermokarst)
#### Post date: [February 1, 2019, 2:54pm UTC](https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071/4 "2019-02-01T14:54:22Z")
---
### Author:  [@Nicholas\_Bokulich](https://forum.qiime2.org/u/Nicholas_Bokulich)
#### Post date: [February 1, 2019, 3:15pm UTC](https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071/5 "2019-02-01T15:15:07Z")
Hi @halt_BB,
> [@halt\_BB](#):
>
> I agree with you on the high percent of “Unassigned”
It happens to all of us! Read quality and non-target DNA amplification/contamination are both issues.
> [@halt\_BB](#):
>
> I am going to try the new qiime q2 feature-classifier with the pretrained silva-132-99-515-806-nb-classifier.qza and hopefully with more stringent filtering
Yes, that would give more reliable assignments at the end of the day (though you will still lose many unassigned sequences — you can spot check a few with NCBI BLAST to see if these are garbage, non-target DNA, or an error, i.e., something that should be assignable in which case I can walk you through more troubleshooting).
> [@halt\_BB](#):
>
> as detailed here [Optimizing taxonomic classification of marker gene amplicon sequences [PeerJ Preprints]](https://peerj.com/preprints/3208/)
Note that's an ancient pre-print and I don't think q2-feature-classifier is even benchmarked there... this is the paper you want: [Optimizing taxonomic classification of marker-gene amplicon sequences with QIIME 2’s q2-feature-classifier plugin | Microbiome | Full Text](https://microbiomejournal.biomedcentral.com/articles/10.1186/s40168-018-0470-z)
Let us know how it turns out!
---
### Author:  [@Nicholas\_Bokulich](https://forum.qiime2.org/u/Nicholas_Bokulich)
#### Post date: [February 1, 2019, 3:15pm UTC](https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071/6 "2019-02-01T15:15:10Z")
---
### Author:  [@halt\_BB](https://forum.qiime2.org/u/halt_BB)
#### Post date: [February 5, 2019, 8:50am UTC](https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071/7 "2019-02-05T08:50:16Z")
Hi @Nicholas_Bokulich,
> [@halt\_BB](#):
>
> I am going to try the new qiime q2 feature-classifier with the pretrained silva-132-99-515-806-nb-classifier.qza
I just tried running the feature-classifier and there was a compatibility issue. see error below
\*$time qiime feature-classifier classify-sklearn \
> --i-classifier databases/silva-132-99-515-806-nb-classifier.qza
> --i-reads fig2a/rep-seqs.qza
> --o-classification fig2a/q2taxonomy.qza
> --verbose
> Traceback (most recent call last):
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2cli/commands.py", line 274, in **call**
> results = action(\*\*arguments)
> File "", line 2, in classify\_sklearn
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 225, in bound\_callable
> spec.view\_type, recorder)
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/result.py", line 287, in \_view
> result = transformation(self.\_archiver.data\_dir)
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/transform.py", line 70, in transformation
> new\_view = transformer(view)
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2\_feature\_classifier/\_taxonomic\_classifier.py", line 64, in \_1
> % (sklearn\_version, sklearn. **version** ))
> ValueError: The scikit-learn version (0.20.2) used to generate this artifact does not match the current version of scikit-learn installed (0.19.1). Please retrain your classifier for your current deployment to prevent data-corruption errors.
Plugin error from feature-classifier:
The scikit-learn version (0.20.2) used to generate this artifact does not match the current version of scikit-learn installed (0.19.1). Please retrain your classifier for your current deployment to prevent data-corruption errors.
See above for debug info.\*
I am thinking about 2 options:
1. Install qiime2 2019.1 and re-try to see if the error goes away (running the install now..)
2. Use the classifier that I trained with SILVA132 myself.
Option 2: Tried this and it failed too
\*time qiime feature-classifier classify-sklearn \
> --i-reads fig2a/rep-seqs.qza
> --i-classifier databases/silva132-taxonomy99.qza
> --o-classification fig2a/q2taxonomy.qza
> --verbose
> Traceback (most recent call last):
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2cli/commands.py", line 274, in **call**
> results = action(\*\*arguments)
> File "", line 2, in classify\_sklearn
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 199, in bound\_callable
> self.signature.check\_types(\*\*user\_input)
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/type/signature.py", line 301, in check\_types
> name, kwargs[name].type, spec.qiime\_type))
> TypeError: Parameter 'classifier' received an argument of type FeatureData[Taxonomy]. An argument of subtype TaxonomicClassifier is required.
Plugin error from feature-classifier:
Parameter 'classifier' received an argument of type FeatureData[Taxonomy]. An argument of subtype TaxonomicClassifier is required.
See above for debug info.\*
Then I thought..wait perhaps the classifier requires a FeatureData[Sequence] artifact
So I did...
\*time qiime feature-classifier classify-sklearn \
> --i-reads fig2a/rep-seqs.qza
> --i-classifier databases/silva132-db99.qza
> --o-classification fig2a/q2taxonomy.qza
> --verbose
> Traceback (most recent call last):
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/q2cli/commands.py", line 274, in **call**
> results = action(\*\*arguments)
> File "", line 2, in classify\_sklearn
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/sdk/action.py", line 199, in bound\_callable
> self.signature.check\_types(\*\*user\_input)
> File "/Users/papa/miniconda2/envs/qiime2-2018.11/lib/python3.5/site-packages/qiime2/core/type/signature.py", line 301, in check\_types
> name, kwargs[name].type, spec.qiime\_type))
> TypeError: Parameter 'classifier' received an argument of type FeatureData[Sequence]. An argument of subtype TaxonomicClassifier is required.
Plugin error from feature-classifier:
Parameter 'classifier' received an argument of type FeatureData[Sequence]. An argument of subtype TaxonomicClassifier is required.
See above for debug info.\*
And it failed too 😞
As I wait for the install to complete (I hope it uses the current scikit-learn version (0.20.2), shouldn't option 2 be able to work?
Thanks.
---
### Author:  [@halt\_BB](https://forum.qiime2.org/u/halt_BB)
#### Post date: [February 5, 2019, 8:50am UTC](https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071/8 "2019-02-05T08:50:18Z")
Hi @Nicholas_Bokulich,
My qiime 2 2019.1 install was ok 👌 I decided to keep qiime 2 2018.11 as well (in case I run into any issues).
I read up on some conflicts that might arise when one is running 2 conda environs, [https://github.com/conda/conda/issues/3580](https://github.com/conda/conda/issues/3580) and noticed it can be done. I had a terminal window of _qiime 2 2018.11_ active so I started a new session (in a new window) and activated _qiime 2 2019.1_.
Then I restarted my feature classifier ,
```
$ time qiime feature-classifier classify-sklearn \
```
> --i-reads fig2/rep-seqs.qza
> --i-classifier databases/silva-132-99-515-806-nb-classifier.qza
> --o-classification fig2/q2taxonomy.qza
> --verbose
The job seems to be running but when I _top_, I don't see _python_ anywhere in the running processes
Maybe I could be paranoid, but the same command above, with the same samples but with consensus-blast took about 17mins.
Is this behavior ok for \*classify-sklearn \*?
Please help!
---
### Author:  [@Nicholas\_Bokulich](https://forum.qiime2.org/u/Nicholas_Bokulich)
#### Post date: [February 5, 2019, 1:30pm UTC](https://forum.qiime2.org/t/im-loosing-too-many-otus-with-taxa-filter-table-80-is-this-a-normal-common-phenomenon/8071/9 "2019-02-05T13:30:15Z")
@halt_BB,
You can definitely run two parallel environments, no problem. The sklearn classifier should take about as long as (or faster than) blast-LCA, but there are exceptions to the rule. In general, an error will be produced if something bad happens, e.g., a memory error or out of disk space... so my guess is it is just still running.