Songbird: Straight Line and NAN Pseudo Q-squared

Hi, all!

Forgive me if this goes against the community guidelines. I've gotten Songbird to give me an output, but am confused as to how I even got the results. I was given the output of a straight line in both graphs, and my pseudo Q-squared was nan.


I'm not sure what I would have to change in order to fix this. I was previously having trouble with my metadata training column, which I reassigned in order to properly run Songbird. This is the immediate output after that fix.

I have yet to see something like this happen to anyone else! If anyone has any insight, it would be greatly appreciated. Thank you!

-Katherine

Referencing the original post for context

Its hard for me to say what is happening either -- I have not seen this error before. Its especially weird to also have the baseline model flatline, which hints at something wrong with the biom table (or perhaps your train/test splits in your metadata).

What does the output differentials look like? Is it all nan? Did you try lowering the learning rate? How many samples do you have in your study (and how many are in train/test)?

This is so weird! Both my null and baseline models are completely empty. Take a look:

My biom table is full: all my samples are accounted for. I have 39 samples, 8 of which are selected for training and 10 of which have feature counts below 200. None of the low feature count samples are trained on. I am going to filter out these samples as an experiment, to see if that makes a difference. I'm not sure how to lower the learning rate, though––if I knew what parameter that was connected to, I'd be more than happy to try that out as well.
Here's the code I've been running, for ease/context (sorry for formatting errors, I usually run my code all in one line but think this way is better to read):

qiime songbird multinomial 
--i-table table-filtered-taxa-F.qza 
--m-metadata-file FPIES_mycobiome_metadata_june2020_F.tsv 
--p-formula "1" 
--p-epochs 50000 
--p-differential-prior 0.5 
--p-training-column training_column 
--p-summary-interval 1 
--o-differentials null-diff-F1.qza 
--o-regression-stats null-stats-F1.qza 
--o-regression-biplot null-biplot-F1.qza 
--verbose
qiime songbird multinomial 
--i-table table-filtered-taxa-F.qza 
--m-metadata-file FPIES_mycobiome_metadata_june2020_F.tsv 
--p-formula "outgrown_FPIES" 
--p-epochs 50000 
--p-differential-prior 0.5 
--p-training-column training_column 
--p-summary-interval 1 
--o-differentials differentials-o1-F.qza 
--o-regression-stats regstats-o1-F.qza 
--o-regression-biplot biplot-o1-F.qza 
--verbose

yes, try filtering. You can also try setting --p-min-feature-count 1 to keep all of the samples to try to get some readings. You could also try running --p-batch-size 1 -- there is a chance that you are dropping too many samples to actually record anything (which will happen if your test set is smaller than your batch size).

Also, how long does it take to run?

I wonder if dropping too many samples is the case. We started with 40, and one sample had no fungi so it was taken out. Then removing all samples with feature counts below 100 brings us to 33 samples. Still 8 training samples, so 24%... I wouldn't think that'd be too large to skew the data, would it? Could it be that the training samples are skewed too heavily to our null samples, which tend to have more features, making the model match too well?

It doesn't take long to run. I just tried Songbird again on the table including only samples with 100+ features, and that took 1 minute and 4 seconds. It also resulted in a completely empty differential stats file.

Would it help if I shared my metadata and table files with you? Maybe there's something that I'm missing.

yea, feel free to upload the data to reproduce this error

After going through it with you, it seems to have to do with the training column and sample/feature counts. It leads me to believe that songbird must not be reading them properly or losing information somewhere along the line. I've edited my code to include specified sample and feature counts of 1, and altered my metadata file so ~78% are Training, just to see if it runs. Now, as I run in the qiime2-songbird environment I've used this code to create my null model:

qiime songbird multinomial --i-table table-filtered-taxa-F.qza --m-metadata-file FPIES_mycobiome_metadata_june2020_F_Testing.tsv --p-formula "1" --p-epochs 50000 --p-differential-prior 0.5 --p-summary-interval 1 --p-min-feature-count 1 --p-min-sample-count 1 --p-batch-size 10 --o-differentials null-diff-F100.qza --o-regression-stats null-stats-F100.qza --o-regression-biplot null-biplot-F100.qza --verbose

And I'm getting this plugin error:

# Constructed from biom file
#OTU ID	S0094-0002	S0094-0003	S0094-0004	S0094-0005	S0094-0006	S0094-0007	S0094-0008	S0094-0009	S0094-0010	S0094-0011	S0094-0012S0094-0013	S0094-0014	S0094-0015	S0094-0016	S0094-0017	S0094-0018	S0094-0019	S0094-0020	S0094-0021	S0094-0022	S0094-0023	S0094-0024	S0094-0025	S0094-0026	S0094-0027	S0094-0028	S0094-0029	S0094-0030	S0094-0031	S0094-0032	S0094-0033	S0094-0034	S0094-0035S0094-0036	S0094-0037	S0094-0038	S0094-0039	S0094-0040
7626e6fd61bb81fc75267574a7118b18	57.0	0.0	0.0	0.0	22.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	7.0	0.0	0.0	228.0	0.0	0.0	0.0	0.0	0.0
7bf1903bf313ce1ef854e992f6122cdc	0.0	23.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	82.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
650356a45463ec00382fc6c86002a8b0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	7.0	0.0	0.0	0.0	0.0.0	190.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
65db9d31367427b8bb9fe77915600fb8	0.0	18.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	64.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	370.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
602d0196cfee48f6c8115fd09de1e745	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	3.0	0.0	0.0	36.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
27c4f40edd44f13bc18e048ee6bdac71	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	3.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	22.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	2249.0	0.0	0.0
0c5644d0d4b2e9731fc89fbd7a6fc831	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	216.0	0.0	0.0	0.0	71.0	0.0	0.0	0.0	0.0	0.4.0	0.0	30.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	3018.0	0.0	0.0
c7a6c4c792db8a71fe0d409121d3eac3	0.0	0.0	36.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	61.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	40.0	0.0	0.0	13.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
4123471995e5136c826edbefc3f08d4c	0.0	0.0	0.0	7.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	6.0	0.0	0.0	0.0.0	0.0	196.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	8.0
5ada5175f138bb90637c7d2a6adc1088	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	39.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	67.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
9266705d7387b47ea6822cd12f0033d5	0.0	0.0	0.0	6.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	76.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	19.0	0.0	42.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
8188c2a662c497e041f8e3571d058646	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	140.0	803.0	0.0	0.0	0.0	0.0	0.0	0.0	6405.0	0.0	0.0
3d7e39e31adb4cdcb719dfd0000d3cb6	0.0	0.0	0.0	9.0	0.0	17.0	0.0	0.0	24.0	0.0	0.0	0.0	0.0	0.0	90.0	0.0	0.0	0.0.0	32.0	12.0	0.0	0.0	0.0	27.0	0.0	0.0	0.0	0.0	46.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
0502d174fab189fb04ee287959eaf0b6	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	4.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	11.0	0.0	0.0	0.0	0.0	0.0
005cf61e57cfb659d65bd53ca093b377	0.0	0.0	0.0	0.0	0.0	0.0	0.0	12.0	0.0	0.0	0.0	0.0	0.0	160.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	310.0	0.0	0.0	0.0	0.0
9e7d4050d132788915023da09b8955ed	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	9.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	2.0	0.0	0.0	0.0	0.0	0.0
1b3cfc1126439c7d2bb8a5c4708c986c	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	18.0	0.0	58.0	0.0	0.0	0.0	0.0	0.0.0	0.0	148.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	3485.0	44.0	0.0	13.0	0.0	0.0
80d6e95286664059b2e9866032d0f2a1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	119.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	222.0	0.0	0.0	0.0	0.0	0.0
d1f146c5adce5ea2a93160becf6e21c3	0.0	0.0	0.0	0.0	0.0	191.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	95.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
d580abdf01c129878652a9823fc71fef	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.490.0	0.0	0.0	60.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
6038b2696c43d4bf0b36f116defc174b	0.0	0.0	0.0	0.0	0.0	0.0	67.0	0.0	0.0	0.0	0.0	3.0	19.0	203.0	0.0	0.0	1672.0	20.0	0.0	21.0	0.0	0.0	0.0	0.0	83.0	0.0	37.0	58.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	32.0	3.0	0.0	1313.0
13553eeeca57b80e17967efa543012e0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	4.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	22.0	0.0	0.0	0.0	0.0
b6e42a91398b35dfeb1bb0decb748fd9	45.0	0.0	0.0	0.0	5.0	0.0	29.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
8f2bbbb0299ab46b53432cc9d95183d7	0.0	69.0	0.0	23.0	2687.0	0.0	30262.0	8.0	32.0	0.0	0.0	217.0	0.0	23.0	23433.0	0.0	1645.0	115.0	0.0	0.0	19475.0	4287.0	4575.0	8095.0	692.0	2615.0	11212.0	67.0	54.0	1734.0	65.0	40996.0	6848.0	17342.0	2024.0	6.0	30444.0	6101.0	58648.0
66464b884b47096cd2a5a0d47811b776	0.0	42.0	0.0	0.0	0.0	0.0	567.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	336.0	0.0	0.0	111.0	0.0	0.0	0.0	0.0	0.0	99.0	286.0	0.0	0.0	0.0	0.0	0.0
f51a19374f4630279c796295a1cb43d6	0.0	0.0	0.0	3.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	6.0	0.0.0	75.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
9930dfb886d12e93158745979608a0e1	6277.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	22.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	319.0	0.0	0.0	0.0	0.0
f4b35ec856455d90bfcbb4a03fb82db0	0.0	0.0	33.0	0.0	2.0	34.0	7.0	0.0	0.0	0.0	0.0	0.0	62.0	0.0	0.0	0.0	0.0	0.0.0	8.0	0.0	0.0	0.0	10.0	0.0	0.0	15.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
eb103acfe120db89e6dfa42d61e5b8a1	0.0	0.0	0.0	0.0	161.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	4.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
7f654bcc0900e9d556b37c166e705329	44.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	12.0	0.0	0.0	0.0	0.0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
7f7736d467ce99ead96878ddf3903e4e	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	11.0	0.0	0.0	0.0	0.0	0.0.0	0.0	25.0	0.0	0.0	0.0	0.0	0.0	81.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
98afcb3d48be557186c5260fa27d3706	0.0	0.0	0.0	0.0	0.0	0.0	17.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.170.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	3.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0             

[39 rows x 26 columns]             Intercept
Sample ID            
S0094-0002        1.0
S0094-0003        1.0
S0094-0004        1.0
S0094-0005        1.0
S0094-0006        1.0
S0094-0007        1.0
S0094-0008        1.0
S0094-0009        1.0
S0094-0010        1.0
S0094-0011        1.0
S0094-0012        1.0
S0094-0013        1.0
S0094-0014        1.0
S0094-0015        1.0
S0094-0016        1.0
S0094-0017        1.0
S0094-0018        1.0
S0094-0019        1.0
S0094-0020        1.0
S0094-0021        1.0
S0094-0022        1.0
S0094-0023        1.0
S0094-0024        1.0
S0094-0025        1.0
S0094-0026        1.0
S0094-0027        1.0
S0094-0028        1.0
S0094-0029        1.0
S0094-0030        1.0
S0094-0031        1.0
S0094-0032        1.0
S0094-0033        1.0
S0094-0034        1.0
S0094-0035        1.0
S0094-0036        1.0
S0094-0037        1.0
S0094-0038        1.0
S0094-0039        1.0
S0094-0040        1.0
[ True  True  True  True  True  True  True  True  True  True  True  True
  True  True False False False  True  True  True  True  True  True  True
 False  True  True  True  True  True  True  True  True  True False  True
  True  True  True]
Traceback (most recent call last):
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/q2cli/commands.py", line 328, in __call__
    results = action(**arguments)
  File "</opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/decorator.py:decorator-gen-481>", line 2, in multinomial
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/qiime2/sdk/action.py", line 240, in bound_callable
    output_types, provenance)
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/qiime2/sdk/action.py", line 383, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/songbirddev-1.0.2-py3.6.egg/songbird/q2/_method.py", line 53, in multinomial
    seed=random_seed,
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/songbirddev-1.0.2-py3.6.egg/songbird/util.py", line 193, in split_training
    print(metadata.loc[design.index, training_column])
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/pandas/core/indexing.py", line 1418, in __getitem__
    return self._getitem_tuple(key)
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/pandas/core/indexing.py", line 805, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/pandas/core/indexing.py", line 929, in _getitem_lowerdim
    section = self._getitem_axis(key, axis=i)
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/pandas/core/indexing.py", line 1850, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/pandas/core/indexing.py", line 160, in _get_label
    return self.obj._xs(label, axis=axis)
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/pandas/core/generic.py", line 3729, in xs
    return self[key]
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/pandas/core/frame.py", line 2995, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/opt/anaconda/anaconda3/envs/qiime2-songbird/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2899, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: None

Plugin error from songbird:

  None

See above for debug info.

It also repeated my metadata which I cut out for privacy. Is that my biom table in the first part of the error message? Could it be that my table had accidentally been filtered, and there is nothing for Songbird to call back to?

Hi @kbenjamin this error message is very weird. Can you provide the --verbose flag to get a more verbose read out of the error?

1 Like

Unfortunately this is all I get with the verbose flag! Not even a tmp file. I've sent you the unedited cut.

Thanks for your continued attention to this! I've gotten songbird to work in the past on other datasets, and it's been an incredible tool. Hopefully we can get to the bottom of this!

1 Like

Hi, @kbenjamin we're a bit stumped on your output and am not entirely sure how exactly to help, particularly given that we didn't run into these issues on our end. There could be something off with your install or code edits.