Hi @askerdb,
The usual approach is to have a very simple data set that you can use for testing of edge cases and basic functionality. It's important this is small enough that you can compute the expected results manually. These tests should be very extensive, covering both valid and invalid input, ensuring that you achieve the minimum and maximum values of the metric when it's appropriate, ensuring that appropriate and helpful error messages are raised when necessary, ... (If you want some more input on what to test, check out How to Break Software - though there are likely more modern books on the topic.)
Then, you would usually have one or two real-world data sets that you run a couple of tests on. More is better - it just depends on how time consuming it is to compute the expected results, which is the hard part. If there is a paper on the metric which contains data and presents results of computing the metric on that data, you can build your "real world" tests around that data (confirm that your implementation reproduces the results generated on real-world data in the paper). Alternatively, if there is a reference implementation of your metric that is well-tested (and ideally implemented by someone else), you could generate a test data set, run it through that implementation, and then write tests that confirm that your method gets those same results.
This is a complicated process, and it's important to get right. If done right, writing unit tests typically takes longer than writing the code that is being tested. It's worth the time investment though, and you'll get quicker with more experience. When a user inevitably gets in touch with you and tells you they think they've found a bug in your software (which will happen after your group and multiple other groups have published high-impact findings based on your code), you'll be thankful for all the time you spent writing tests. It of course may still be a legitimate bug, but if you have an extensive unit test suite then there's a good chance it's not actually a bug but rather a misunderstanding of the method by the user. And, if it is a bug, at least you'll know that you did your due diligence in trying to avoid the issue up front.
You can find the UniFrac test suite here in case that is a useful reference.
Hope this helps!