I'm trying to use Songbird to find differential taxa in a dataset comparing two groups. My cross-validation plot(s) have looked good, but the loss function plots are looking like a very bad EKG. I was reading this post, but her loss function appears to be behaving normally with an exponential decline then plateauing, while mine declines (for one try) and then turns into a series of peaks and valleys. Can anyone help me figure out what I'm doing wrong that the Loss graphs don't look right?
Here is the command I ran that gave me the best plots (in my opinion): qiime songbird multinomial \ --i-table filtered_t1_cases_obstrl.qza \ --m-metadata-file NAFLD_Mapping_JZ_SNA_V3.txt \ --p-formula "Group" \ --p-epochs 10000 \ --p-differential-prior 0.5 \ --p-summary-interval 1 \ --p-num-random-test-examples 37 \ --output-dir songbird_nafld_test_ex_37/
I have a total of 247 samples, distributed almost half and half (146 cases and 101 controls). I added the random test samples flag after reading the tutorial that I can help with many samples. I don't think creating a test/training column would benefit me since the samples are pretty evenly distributed (but will if people in the know think it would help).
I also tried increasing the epochs to 20000 since the tutorial said that would help with non-plateauing, but that just seemed to make the plot worse.
Attached are the output from the command above and the output from increasing epochs to 20000.
Any help/tips would be appreciated.
@saatkinson what does your R2 / Q2 look like? That can help decide if your model is overfitting or not.
If you don't want a noisy loss, the best approach is to lower the learning rate --p-learning-rate. The default is now 1e-3, you can try 1e-4 or 1e-5 to see if that will give you a cleaner loss (although you will need to train for more epochs).
I ran the commands to look at Q2 and both instances are extremely low (0.000245 and 0.000458, for 10000 and 20000 epochs respectively). Below is the command I ran to generate the null model and the paired-summaries.
I also ran the null model without the num-random-test-examples flag because I wasn't sure if it should be there for the null model, but that made the Q2 drop to -0.064678.
Am I correct in interpreting that my model is severely overfitting the data? Looking at the FAQs there was a section about lowering the differential-prior, do you think this would help with the data I'm seeing? Do you have other suggestions for correcting the fit?
I can't completely comment on your results without seeing the loss plots. Do you see both your loss and CV curves decline?
If you see Q2 < 0, that is not a good sign (it means that your predictive power is not good). The differential prior is worth making smaller (i.e. 0.5).
Attached are my plots, the CV curve declines nicely, but the loss curve still looks like an EKG. I did not yet try decreasing the learning curve, do you recommend lowering that in conjunction with the differential prior? I ran the commands with a differential prior of 0.5, would 0.3 be an acceptable decrease?
Epochs 10000
Epochs 20000
Thanks for your help, I'm new to regression modeling.
Samantha
Hi @saatkinson, given how noisy the loss is, there is a benefit of running with a lower learning rate (try 1e-4 and 1e-5). I'd try adjust the differential prior only after getting the loss function smoothed (I'd stick with just tuning those parameters for now)
The learning rate of 1e-4 looks better to my untrained eye, is this accurate? This this as good as I'm going to get or should I keep decreasing the learning rate until it's smooth? Should I have increased the number of epochs more than 10000?
Hi @saatkinson, those fits are looking much better and are likely good enough for you downstream analysis (not sure how much further tuning will help). Good luck!
