Why do you want to get rid of rare taxa?
As far as I'm aware, there is no inherent value in throwing away biological data. Sometimes people filter data for resource/performance reasons - if you can get away without doing this, you should. Waste not, want not, right? Further, everyone's data is different, so "reasonable percentages" might make your data look more "normal" at the high cost of its uniqueness.
When possible, try to think about your data in biological terms
There is a certain amount of noise inherent to the data we get from sequencing technologies, and we often use tools like denoising and filtering to attempt to reduce noise, so that the data we analyze better represents the actual biological community under study. "Will process X improve the fidelity of my data?" is often a useful razor for deciding how, and how much to filter.
Like the trade-offs Colin mentions between common and rare taxa, there is an inherent tension between noise reduction and the risk of removing rare organisms. The goal is to remove data that is not meaningful to our study, without introducing new biases. A few common approaches to this idea follow, but it's ultimately up to you to decide whether any of these are appropriate to your work, and to justify your choices.
Filtering out samples with unusually low read counts
If you expect a relatively consistent sampling depth, it might be reasonable to drop samples with fewer than some threshold number of reads, with the rationale that there was a failure of some kind in sampling or processing, and they are likely to poorly represent the community sampled.
If 94 of my 96 samples have over 10k+ reads, and two have fewer than 1000 reads, I might use qiime feature-table filter-samples to drop those two because I think they are compromised.
Filtering out artifactual features
If you expect some consistency across sampled communities, it might be reasonable to drop features that appear in fewer than some threshold number of samples, with the rationale that those features are probably artifacts.
If I had 100 mouse fecal samples, from a colony of mice raised together in controlled conditions, I might use 'qiime feature-table filter-features` to filter out any features that appear in only one sample, because I think it is unlikely that only one mouse, at one period of time, would host that organism.
Filtering out contamination
Host sequences, or sequences from contaminants introduced during sampling, lab work, or sequencing, may impact your ability to draw conclusions about the microbial community you're trying to study. If you're getting good results without removing contamination, many people avoid dealing with it entirely - this is not an easy problem to solve. Try 
searching this forum for posts on contamination removal if you find you need to remove suspected contaminants, but I think that's beyond our scope here.
Good luck!
Why not look for biomarkers in ubiquitous taxa that appear in most samples, and also biomarker taxa that appear only in specific groups of interest?