Rationality: From AI to Zombies

If Bob then winks at you, that’s a new piece of evidence. In that case, it would be a mistake to stay skeptical about whether Bob is your secret admirer; the 10:1 odds

But never forget that on any question about the way things are (or should be), and in any information situation, there is always a best estimate. You are only entitled to your best honest efort to fnd that best estimate; anything else is a lie

Worst of all, prosaic beliefs—beliefs that are in principle falsifable, beliefs that do constrain what we expect to see—can still get stuck in our heads, reinforced by a network of illusions and biases

background knowledge (priors) and a new piece of evidence, probability theory uniquely defnes the best set of new beliefs (posterior) I could adopt

O]bviously it’s useful to have as much evidence as possible, in the same way it’s useful to have as much money as possible. But equally obviously it’s useful to be able to use a limited amount of evidence wisely, in the same way it’s useful to be able to use a limited amount of money wisely

Much of our reasoning process is really rationalization—story-telling that makes our current beliefs feel more coherent and justifed, without necessarily improving their accuracy

It is widely recognized that good science requires some kind of humility. What sort of humility is more controversial

If you do not seek perfection you will halt before taking your frst steps

When you encounter new information, the hard part is to update, to react, rather than just letting the information disappear down a black hole. And humility, properly misunderstood, makes a wonderful black hole—all you have to do is admit you could be wrong

To be humble is to take specifc actions in anticipation of your own errors. To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty

Believing in Santa Claus gives children a sense of wonder and encourages them to behave well in hope of receiving presents. If Santabelief is destroyed by truth, the children will lose their sense of wonder and stop behaving nicely. Terefore, even though Santa-belief is false-to-fact, it is a Noble Lie whose net beneft should be preserved for utilitarian reasons

False dilemmas are ofen presented to justify unethical policies that are, by some vast coincidence, very convenient. Lying, for example, is ofen much more convenient than telling the truth; and believing whatever you started out with is more convenient than updating. Hence the popularity of arguments for Noble Lies; it serves as a defense of a pre-existing belief

Te process of overcoming bias requires (1) frst noticing the bias, (2) analyzing the bias in detail, (3) deciding that the bias is bad, (4) fguring out a workaround, and then (5) implementing it.

Tis is why rationalists put such a heavy premium on the paradoxical-seeming claim that a belief is only really worthwhile if you could, in principle, be persuaded to believe otherwise. If your retina ended up in the same state regardless of what light entered it, you would be blind … Hence the phrase, “blind faith.” If what you believe doesn’t depend on what you see, you’ve been blinded as efectively as by poking out your eyeballs

In the ancestral environment, politics was a matter of life and death. And sex, and wealth, and allies, and reputation … When, today, you get into an argument about whether “we” ought to raise the minimum wage, you’re executing adaptations for an ancestral environment where being on the wrong side of the argument could get you killed. Being on the right side of the argument could let you kill your hated rival

Politics is the mind-killer. Arguments are soldiers. Once you know which side you’re on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it’s like stabbing your soldiers in the back. If you abide within that pattern, policy debates will also appear one-sided to you—the costs and drawbacks of your favored policy are enemy soldiers, to be attacked by any means necessary

Te correspondence bias is the tendency to draw inferences about a person’s unique and enduring dispositions from behaviors that can be entirely explained by the situations in which they occur

A car with a broken engine cannot drive backward at 200 mph, even if the engine is really really broken

Your instinctive willingness to believe something will change along with your willingness to afliate with people who are known for believing it—quite apart from whether the belief is actually true. Some people may be reluctant to believe that God does not exist, not because there is evidence that God does exist, but rather because they are reluctant to afliate with Richard Dawkins or those darned “strident” atheists who go around publicly saying “God does not exist.”

Tags: #rationality

Barry is a famous geologist. Charles is a fourteen-year-old juvenile delinquent with a long arrest record and occasional psychotic episodes. Barry flatly asserts to Arthur some counterintuitive statement about rocks, and Arthur judges it 90% probable. Ten Charles makes an equally counterintuitive flat assertion about rocks, and Arthur judges it 10% probable. Clearly, Arthur is taking the speaker’s authority into account in deciding whether to believe the speaker’s assertions. Scenario 2: David

Barry is a famous geologist. Charles is a fourteen-year-old juvenile delinquent with a long arrest record and occasional psychotic episodes. Barry flatly asserts to Arthur some counterintuitive statement about rocks, and Arthur judges it 90% probable. Ten Charles makes an equally counterintuitive flat assertion about rocks, and Arthur judges it 10% probable. Clearly, Arthur is taking the speaker’s authority into account in deciding whether to believe the speaker’s assertions

P (SlipperyjNight; Sprinkler) = P (SlipperyjSprinkler

Tere is an ineradicable legitimacy to assigning slightly higher probability to what E. T. Jaynes tells you about Bayesian probability, than you assign to Eliezer Yudkowsky making the exact same statement. Fify additional years of experience should not count for literally zero influence. But this slight strength of authority is only ceteris paribus, and can easily be overwhelmed by stronger arguments. I have a minor erratum in one of Jaynes’s books—because algebra trumps authority

Te more directly your arguments bear on a question, without intermediate inferences—the closer the observed nodes are to the queried node, in the Great Web of Causality—the more powerful the evidence. It’s a theorem of these causal graphs that you can never get more information from distant nodes, than from strictly closer nodes that screen of the distant ones

If you really want an artist’s perspective on rationality, then read Orwell

Orwell was the outraged opponent of totalitarianism and the muddy thinking in which evil cloaks itself—which is how Orwell’s writings on language ended up as classic rationalist documents on a level with Feynman, Sagan, or Dawkins

I wrote the sentence in the passive voice, without telling you who tells authors to avoid passive voice. Passive voice removes the actor, leaving only the acted-upon

My point is not to say that journal articles should be written like novels, but that a rationalist should become consciously aware of the experiences which words create

Orwell knew that muddled language is muddled thinking; he knew that human evil and muddled thinking intertwine like conjugate strands of DNA

political language has to consist largely of euphemism, question-begging and sheer cloudy vagueness

Freedom is the freedom to say that two plus two make four

Do you remember,” he went on, “writing in your diary, ‘Freedom is the freedom to say that two plus two make four’?” “Yes,” said Winston. O’Brien held up his lef hand, its back towards Winston, with the thumb hidden and the four fngers extended. “How many fngers am I holding up, Winston?” 287

Freedom is the freedom to say that two plus two make four

Stuart Chase and others have come near to claiming that all abstract words are meaningless, and have used this as a pretext for advocating a kind of political quietism. Since you don’t know what Fascism is, how can you struggle against Fascism

Stuart Chase and others have come near to claiming that all abstract words are meaningless, and have used this as a pretext for advocating a kind of political quietism. Since you don’t know what Fascism is, how can you struggle against Fascism?

Tags: #epistemology #rationality

In all human history, every great leap forward has been driven by a new clarity of thought. Except for a few natural catastrophes, every great woe has been driven by a stupidity. Our last enemy is ourselves; and this is a war, and we are soldiers

If you’re irrational to start with, having more knowledge can hurt you

If you’re irrational to start with, having more knowledge can hurt you. For a true Bayesian, information would never have negative expected utility

I’ve seen people severely messed up by their own knowledge of biases. Tey have more ammunition with which to argue against anything they don’t like. And that problem—too much ready ammunition—is one of the primary ways that people with high mental agility end up stupid, in Stanovich’s “dysrationalia” sense of stupidity

For every expectation of evidence, there is an equal and opposite expectation of counterevidence

Whatever. Rationality is not for winning debates, it is for deciding which side to join. If you’ve already decided which side to argue for, the work of rationality is done within you, whether well or poorly

Rationality is not for winning debates, it is for deciding which side to join. If you’ve already decided which side to argue for, the work of rationality is done within you, whether well or poorly

In Te Bottom Line, I presented the dilemma of two boxes, only one of which contains a diamond, with various signs and portents as evidence. I dichotomized the curious inquirer and the clever arguer. Te curious inquirer writes down all the signs and portents, and processes them, and fnally writes down “Terefore, I estimate an 85% probability that box B contains the diamond.” Te clever arguer works for the highest bidder, and begins by writing, “Terefore, box B contains the diamond,” and then selects favorable signs and portents to list on the lines above. Te frst procedure is rationality. Te second procedure is generally known as “rationalization

Rationality is the operation that we use to obtain more accuracy for our beliefs by changing them

Rationalization operates to fx beliefs in place; it would be better named “anti-rationality,” both for its pragmatic results and for its reversed algorithm

“Rationalization” is a backward flow from conclusion to selected evidence. First you write down the bottom line, which is known and fxed; the purpose of your processing is to fnd out which arguments you should write down on the lines above. Tis, not the bottom line, is the variable unknown to the running process

Curiosity is the frst virtue, without which your questioning

In Orthodox Judaism you’re allowed to notice inconsistencies and contradictions, but only for purposes of explaining them away, and whoever comes up with the most complicated explanation gets a prize