Andrew Gelman, whose blog is always a good read (and is also updated much more often than this one!), provoked a discussion about the different “meanings” of Bayesian statistics. You might find the comments there interesting; I admit I found the whole thing a little hair-splitting for my taste.
Christian Robert — who wrote the superb book “The Bayesian Choice” — started the whole thing off by describing his bemusement over how much “fascination for Bayes’ Theorem [there] seems to be outside Statistics”. After all, it’s just a theorem, right? He contends that the theorem itself mustn’t be confused with the interpretation of the axioms of probability, which is the contentious and “interesting” aspect of the whole endeavour.
But I feel this is pedantic. A “Bayesian” is almost always someone who believes in the subjective interpretation of probability statements, so that Bayes’ Theorem can be used as an means to update one’s beliefs about quantities, hypotheses, and so on. Frequentists don’t reject Bayes’ Theorem itself — they can’t, as it’s just a consequence of the probability axioms, and they even use it uncontentiously for calculating diagnostic test properties such as predictive value — but they do reject its use for “updating beliefs”.
Christian also seems bemused by a rather long but entertaining “justification”/explanation for Bayes’ Theorem that I’ve been meaning to link to for a long time. [It is one of the top results that come up when googling for the term “Bayesian”]. Again, if you accept the axioms of probability, the theorem is just a consequence, so you don’t need to justify it any other way, he maintains. (As he put it,
The theorem per se offers no difficulty, so this may be due to the counter-intuitive inversion of probabilities as the one found in the example of the first blog. But the fact that people often confuse probabilities of causes and probabilities of effects—i.e. the right order of conditioning—does not require a deeper explanation for Bayes’ theorem, rather a pointer at causal reasoning!
But just being told something is true, and even being convinced it is true mathematically, doesn’t help most people. To get them to understand it intuitively sometimes requires something more, like a story, or an example. Not everyone is a mathematician after all…
[I’m sorry if I misrepresented anyone’s views here. Please tell me if I’ve got something wrong and I’ll try and put it right.]