Bad Bayes still bad

Tamino, a notorious “climate change” blogger, is alleged to also be a statistician. He certainly seems to know something about time series. (Thanks to this investigation, we know that Tamino is Grant Foster, writer of “blog diatribe”-style climate papers. His affiliation in the linked paper is “Tempo Analytics, Westbrook, Maine”, but I can’t find any other reference to it online).

Unfortunately he might be somewhat off-base when it comes to other statistical principles. His discussion of Bayesian analysis is so confused that I’ll leave it to Andrew Gelman, professor of statistics at Columbia University, to summarise it for us:

Kent Holsinger sends along this statistics discussion from a climate scientist. I don’t really feel like going into the details on this one, except to note that this appears to be a discussion between two physicists about statistics. The blog in question appears to be pretty influential, with about 70 comments on most of its entries. When it comes to blogging, I suppose it’s good to have strong opinions even (especially?) when you don’t know what you’re talking about.

Update: Gelman repeated himself on his academic blog, where he elaborates on his opinion in the comments. It’s strange that when I tried commenting (twice) on “Tamino”‘s blog to refer him to Gelman’s comments, I didn’t succeed; but when someone else did the same but with the qualifier that “[Gelman] comes around to Tamino’s side” [which not actually true] in his later comments the link appears.

At the time of writing the comment thread ends with “Tamino” abusing a commenter trying to correct one of his calculations until he eventually admits he was indeed wrong. Oh dear.

Advertisements

Leave a Comment

Another sampling from the great frequentist malpractice genre in the sky

That this isn’t well-known amongst the general public is a disgrace, but the “scientific method” as carried out by academic careerists has long been only a poor substitute for real science:

It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.

From sciencenews.org. Then follows the usual errors relating to interpretation of hypothesis tests and other applied frequentist gunk. There is an interesting point made about how randomisation isn’t all that (although what the alternative should be is anyone’s guess), before… behold!

Such sad statistical situations suggest that the marriage of science and math may be desperately in need of counseling. Perhaps it could be provided by the Rev. Thomas Bayes.

A lovely line. Whether this latest example of the litany against the standard operating procedure of too many scientists from all disciplines will change anything more than the previous attempts to do so is moot.

Leave a Comment

Glymour vs Dawid: fight! fight! fight!

This delightful example of an academic hissy fit is from almost a year ago, but still has the power to shock in its lack of professionalism. I hope it shatters the all-too-common stereotype amongst some people of academics as calm intellectuals who want nothing more than to hear each other’s honestly help views. Phooey to that.

The topic of discussion (if it can be called that when the level of aggression is this high) is Phil Dawid’s article "Beware of the DAG", which makes very reasonable points about what possible causal inferences can be undertaken for different levels of "causal" assumptions on a Directed Acyclic Graph (which is more commonly known as a Bayesian Network when reflecting conditional independence properties, as is the case here), and discusses how reasonable and testable these "causal" assumptions are. In particular, he dismisses the popular but mostly deluded endeavour of "causal discovery", which is at once ill-posed and, so far, in my opinion, ill-answered. This hits right at what Clark Glymour is trying to sell, though. But that still doesn’t quite explain what caused this reaction…

NB. No-one from the mailing list replied to this directly, as far as I can tell. The archive for that month is available here.

And now, enjoy, typos included:

From: Clark Glymour <cg09>

Date: Thu, Apr 23, 2009 at 01:12
Subject: [Causality-ML] Professor Dawid’s paper
To: causality-ml

Professor Dawid ‘s worry is announced in his abstract:

“My fundamental concern is the relationship between, on the one hand, properties or concepts relating to an external reality, such as probabilistic independence or causality,which we wish to elucidate or manipulate; and, on the other hand, formal representations of such properties by means of mathematical or logical structures, such as graphs. It is important to avoid confusing the picture with the reality.”

Be at ease. Absolutely not to worry. I have never once, not once, seen someone draw a graph or write a formula when they actually thought they were manipulating what the symbols were supposed to denote. Not once. Word and object, we are ace at distinguishing those. So I thought, having solved Professor Dawid’s concern, I should stop, but I read on a little ways.

To a really important announcement:
“it is … worthy of continual repetition and emphasis, that there is absolutely no logical reason for there to be any connexion whatsoever between observations made under the different regimes of seeing and doing: a system may very well behave entirely differently when it is kicked than when it is left alone”

Good point that. I checked my logic books, no proofs of that connection. Also no proofs that the past ever was, no proofs that the future will come to be, no proofs that Professor Dawid has mental states, no proofs that an external world exists, no proofs that the so-called laws of nature will hold next week. Not much use, those logic books, unless you assume or hypothesize stuff and then want to know the consequences.

On the other hand, there is this funny literature—I wonder if Professor Dawid has read it—where people investigate when, under what various assumptions about the world, other things follow. Like, for example, there is this subject called Euclidean geometry where assumptions are make about space, and then all kinds of interesting other things are proved about space, really amazing stuff—you could use it to design buildings even. But I read that the assumptions are not always true. Pity. Also, there was this guy Newton who had these three assumptions, and then some “rules of reasoning” at the back. He got these amazing consequences, which mostly turned out to be correct, although I hear that his assumptions don’t always hold, and I sure could not find his rules of reasoning in my logic book.

I guess it can’t be the same with observing and doing. There just couldn’t be any assumptions about the connections and proofs from assumptions that you can make the kind of inferences Professor Dawid is talking about—inferences from observations to effects of actions. Or proofs that under other assumptions you can’t. Couldn’t be. So, nah…

In fact, Professor Dawid is really helpful about this. He tells us that if we make the wrong assumptions, or not enough of the right ones, we won’t get that logical connection between seeing and doing. Not a chance of it.

“We say that a DAG D with node-set V, a set of variables, represents a collection C of CIproperties over V if the relation (bunch of symbols here, way over my head) is in C if and only if S and T are d-separated by U in D. This relationship between a D and a collection of CI properties will constitute our semantic interpretation of a DAG.”

Well sure enough, one thing talks about causality, the other talks just about probability. Different terms. My logic book tells me there have to be terms in common between the premises and the conclusion—unless the conclusion is logically true. Kind of like “force” and “acceleration” or like “probability” and “unbiased,” or “perpetual” and “motion”–no logical connection. So Professor Dawid really nailed that one.

Well, I should go on reading this stuff, like how we should just talk about probability because graph theory isn’t mathematics (so many silly people who thought they were doing mathematics) and how science is all about conditional independence not causation (I knew those physicists and chemists and epidemiologists had to be crazy talking about what does and doesn’t cause what), and I am sure he has discovered a lot more stuff than those crazy causal guys who think they have methods that have discovered errors in a mass spectrometer aboard a satellite (imagine—they weren’t ever there), and how to tell what rocks are made of from the radiation bouncing off them, and how to reduce the rate of college dropouts in a college, and that acid rain caused plant die offs in an estuary, and the processes that go on in the brain in an experiment (something about fmri), and even global climate teleconnections—those guys are so crazy. But since I solved the problem Professor Dawid had at t!

he beginning, I will just have a martini.

Bye,

Clark Glymour

Leave a Comment

Oh dear indeed

From one of Tim Worstall’s “Oh dear” posts:

Maybe inequality and poverty in modern Britain are important and maybe they’re not. It’s entirely possible to argue it either way and to a large extent depends upon your Bayesian priors.

So naturally I had to reply,

Actually this is a value judgment and would be expressed through a utility function. Bayesian priors (and posteriors) are probability distributions expressing subjective degrees of certainty over parameters of interest.

No indication thus far that the blog author has taken this on board…

Is it better that Bayesian concepts are invoked incorrectly rather than not at all? I believe so, but we must continue to strive towards fuller understanding of them amongst non-statisticians. Because otherwise we have to deal with things like this:

First, what did you think was the probability of success in Afghanistan before the mission began? This is the prior probability, which we’ll call Ps. The probability of failure, Pf, is one minus this.

Second, what is the probability that we’d see the number of deaths we have, if the mission were succeeding? Call this Pd|s. One minus this gives us Pd|f. [emphasis mine, calculation thankfully not]

Again, comments to the contrary had no effect on deflecting the author in his enthusiasm on this occasion. Oh dear.

Comments (2)

“Is risk management too complicated and subtle for InfoSec?” — I think just mathematics is too complicated and subtle for some people

It’s interesting to see how knowledge of Bayesian methods exists in certain fields while ignorance of the details leads to weird conclusions concerning their usage. A good example of this phenomenon is this mangling of the two-envelope problem — supposedly a “paradox” that Bayesian decision analysis fails at — which is then used to argue that therefore Bayesian analysis of risks is actually useless and that instead

In the absence of reliable risk information, a similar approach to information security may be the best that we can do – just try different things and see which works the best. You might call this approach “experimental security.” There may be no better approach.

Yeah, just experimenting without any inferential tools makes sense… Funny how it allows the analyst to believe anything he wants without anything to back it up.

The takedown is painstakingly given here, but the only comment to it at the time of writing should make it clear just how entrenched the forces of “irrational pragmatism” are:

They Bayesian approach has many beautiful mathematical properties, but it fails to make contact with reality — it has no pragmatics. Worse, it fails to recognize that there is more than one person in the world. In the Bayesian world there is only one subjective probability, “mine”. The fact that you exist and have your own subjectivity that just might have something to do with our agreed-upon response to any particular problem is totally irrelevant. All the technical mathematical results in the world can’t get past these foundational problems.

Wouldn’t it be better to admit ignorance of the issues at hand and then give your opinions on that basis rather than just spout nonsense? There is clearly much education about Bayesian analysis to be done, starting with demolishing incorrect preconceptions that are already out there.

Comments (3)

xkcd is getting close

After an episode about significance, xkcd has now reached the intellectual stage of joking about correlation and causation and how they might — or might not — relate to each other.

So the question is: when is he [= MAP gender of comic creator] going to mention Bayes in some way? It’s only a matter of time, surely, and I’ll take that as a sign that it’s as mainstream (in geek circles, anyway — although geeks seem pretty mainstream these days) as the frequentist idea of significance.

And then I’ll sue for patent infringement. I might be joking.

Comments (1)

How many meanings can Bayesian statistics have? or When can I get off this ship?

Andrew Gelman, whose blog is always a good read (and is also updated much more often than this one!), provoked a discussion about the different “meanings” of Bayesian statistics. You might find the comments there interesting; I admit I found the whole thing a little hair-splitting for my taste.

Christian Robert — who wrote the superb book “The Bayesian Choice” — started the whole thing off by describing his bemusement over how much “fascination for Bayes’ Theorem [there] seems to be outside Statistics”. After all, it’s just a theorem, right? He contends that the theorem itself mustn’t be confused with the interpretation of the axioms of probability, which is the contentious and “interesting” aspect of the whole endeavour.

But I feel this is pedantic. A “Bayesian” is almost always someone who believes in the subjective interpretation of probability statements, so that Bayes’ Theorem can be used as an means to update one’s beliefs about quantities, hypotheses, and so on. Frequentists don’t reject Bayes’ Theorem itself — they can’t, as it’s just a consequence of the probability axioms, and they even use it uncontentiously for calculating diagnostic test properties such as predictive value — but they do reject its use for “updating beliefs”.

Christian also seems bemused by a rather long but entertaining “justification”/explanation for Bayes’ Theorem that I’ve been meaning to link to for a long time. [It is one of the top results that come up when googling for the term “Bayesian”]. Again, if you accept the axioms of probability, the theorem is just a consequence, so you don’t need to justify it any other way, he maintains. (As he put it,

The theorem per se offers no difficulty, so this may be due to the counter-intuitive inversion of probabilities as the one found in the example of the first blog. But the fact that people often confuse probabilities of causes and probabilities of effects—i.e. the right order of conditioning—does not require a deeper explanation for Bayes’ theorem, rather a pointer at causal reasoning!

)

But just being told something is true, and even being convinced it is true mathematically, doesn’t help most people. To get them to understand it intuitively sometimes requires something more, like a story, or an example. Not everyone is a mathematician after all…

[I’m sorry if I misrepresented anyone’s views here. Please tell me if I’ve got something wrong and I’ll try and put it right.]

Comments (1)

Older Posts »