To echo @timanix's comments about using cutadapt.
I'd avoid simply using the DADA2 trim options for removing primers. The reason for this is that, there can sequencing indels (insertion / deletion) within the first 20 or so bases of the read. When you trim based on a fixed length you might arbitrarily keep or loose a base or two... this has the effect of drastically and erroneously inflating the amount of unique ASVs.
This might be the reason why you have so many ASVs. I'd recommend running cutadapt, using the --discard-untrimmed and --p-match-adapter-wildcards options. This will produce a much cleaner output with far less ASVs.