What prevents the temperature T from turning negative during temperature scaling?

[eugene-yh]

Thanks for your re-implementation of temperature scaling! Although I understand this work is from a few years back, I am still wondering why the positively initiated temperature T will never turn negative during the optimization of NLL.

Both your implementation and https://github.com/gpleiss/temperature_scaling (as mentioned in Guo et al., 2017) seem to have no obvious mechanism that can prevent the temperature T from turning negative.

However, if I am not mistaken, the positivity of T is essential to not changing the prediction of the original uncalibrated classifier (i.e., if T turns negative, a class with a negative logit can have a larger probability than a class with a positive logit.)

[markus93]

Hey!

Thanks for the question!

I would say that the loss function (NLL) prevents temperature T from ever reaching negative. For example, if the T is negative, the predictions would be the opposite, and the loss would get much higher. Of course, it would be interesting to see how the loss and the temperature T change if we start with the T = -1.

If You have any further questions, let me know.

Best regards,
Markus

[eugene-yh]

Hey!

Thanks for the question!

I would say that the loss function (NLL) prevents temperature T from ever reaching negative. For example, if the T is negative, the predictions would be the opposite, and the loss would get much higher. Of course, it would be interesting to see how the loss and the temperature T change if we start with the T = -1.

If You have any further questions, let me know.

Best regards, Markus

Hi Markus,
thanks for the insightful reply!

Yes, I understand intuitively that when the temperature T is negative, the loss can get higher if we ASSUME the valid set (used to tune T) and the training set (used to train the classifier) are drawn from the same distribution.

However, I have two concerns:

1. Mathematically, I tried to take the derivative of the NLL loss w.r.t T, and I found that the derivative is not necessarily >0 or <0, meaning T can go larger or smaller. So, I am not sure if T can be pushed to the negative side during tuning.
2. Practically, what if the valid set has some noisy samples that make the loss when T < 0 smaller than the loss when T > 0?

That said, I think maybe I should do some experiments on this to clarify the question and these concerns.

I am looking forward to discussing with you if I get some meaningful results!

Best regards,
Eugene