Classifier Probability Calibration

One of the desired features of a classifier is that it generates probabilities for us. So we know with what probability a sample belongs to class A or class B (in the case of a binary classification problem). However it has been shown that the probabilities that the classifiers produce are not always correct.



A correct  value means that e.g. a classification for sample s with probability 0.8 would eventually mean that among samples that have been assigned a probability of 0.8, 80% actually belong to class A and 20% of them belong to class B. This is a hand problem to prove.

Known classification algorithms have been known to have different mistakes when assigning these probabilities.

Source. Link

So there have been some approaches on calibrating these probabilitiws. Such that once these classifiers generate a probability it actually means that probability. It's not just a number between zero and one.