Eliezer Yudkowsky - Rationality_ From AI to Zombies-Machine Intelligence Research Institute - 2015 ![rw-book-cover](https://readwise-assets.s3.amazonaws.com/static/images/default-book-icon-7.09749d3efd49.png) ## Metadata - Author: [[2015]] - Full Title: Eliezer Yudkowsky - Rationality_ From AI to Zombies-Machine Intelligence Research Institute - Category: #books ## Highlights - If Bob then winks at you, that’s a new piece of evidence. In that case, it would be a mistake to stay skeptical about whether Bob is your secret admirer; the 10:1 odds (Page 200) - But never forget that on any question about the way things are (or should be), and in any information situation, there is always a best estimate. You are only entitled to your best honest efort to fnd that best estimate; anything else is a lie (Page 200) - Worst of all, prosaic beliefs—beliefs that are in principle falsifable, beliefs that do constrain what we expect to see—can still get stuck in our heads, reinforced by a network of illusions and biases (Page 201) - background knowledge (priors) and a new piece of evidence, probability theory uniquely defnes the best set of new beliefs (posterior) I could adopt (Page 203) - O]bviously it’s useful to have as much evidence as possible, in the same way it’s useful to have as much money as possible. But equally obviously it’s useful to be able to use a limited amount of evidence wisely, in the same way it’s useful to be able to use a limited amount of money wisely (Page 204) - Much of our reasoning process is really rationalization—story-telling that makes our current beliefs feel more coherent and justifed, without necessarily improving their accuracy (Page 205) - It is widely recognized that good science requires some kind of humility. What sort of humility is more controversial (Page 211) - If you do not seek perfection you will halt before taking your frst steps (Page 212) - When you encounter new information, the hard part is to update, to react, rather than just letting the information disappear down a black hole. And humility, properly misunderstood, makes a wonderful black hole—all you have to do is admit you could be wrong (Page 214) - To be humble is to take specifc actions in anticipation of your own errors. To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty (Page 215) - Believing in Santa Claus gives children a sense of wonder and encourages them to behave well in hope of receiving presents. If Santabelief is destroyed by truth, the children will lose their sense of wonder and stop behaving nicely. Terefore, even though Santa-belief is false-to-fact, it is a Noble Lie whose net beneft should be preserved for utilitarian reasons (Page 216) - False dilemmas are ofen presented to justify unethical policies that are, by some vast coincidence, very convenient. Lying, for example, is ofen much more convenient than telling the truth; and believing whatever you started out with is more convenient than updating. Hence the popularity of arguments for Noble Lies; it serves as a defense of a pre-existing belief (Page 218) - Te process of overcoming bias requires (1) frst noticing the bias, (2) analyzing the bias in detail, (3) deciding that the bias is bad, (4) fguring out a workaround, and then (5) implementing it. (Page 220) - Te Sophisticate: “Te world isn’t black and white. No one does pure good or pure bad. It’s all gray. Terefore, no one is better than anyone else.” Te Zetet: “Knowing only gray, you conclude that all grays are the same shade. You mock the simplicity of the two-color view, yet you replace it with a one-color view . . . (Page 227) - Tis is why rationalists put such a heavy premium on the paradoxical-seeming claim that a belief is only really worthwhile if you could, in principle, be persuaded to believe otherwise. If your retina ended up in the same state regardless of what light entered it, you would be blind . . . Hence the phrase, “blind faith.” If what you believe doesn’t depend on what you see, you’ve been blinded as efectively as by poking out your eyeballs (Page 239) - In the ancestral environment, politics was a matter of life and death. And sex, and wealth, and allies, and reputation . . . When, today, you get into an argument about whether “we” ought to raise the minimum wage, you’re executing adaptations for an ancestral environment where being on the wrong side of the argument could get you killed. Being on the right side of the argument could let you kill your hated rival (Page 255) - Politics is the mind-killer. Arguments are soldiers. Once you know which side you’re on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it’s like stabbing your soldiers in the back. If you abide within that pattern, policy debates will also appear one-sided to you—the costs and drawbacks of your favored policy are enemy soldiers, to be attacked by any means necessary (Page 258) - Te correspondence bias is the tendency to draw inferences about a person’s unique and enduring dispositions from behaviors that can be entirely explained by the situations in which they occur (Page 264) ## New highlights added June 2, 2022 at 5:15 PM - A car with a broken engine cannot drive backward at 200 mph, even if the engine is really really broken (Page 271) - Your instinctive willingness to believe something will change along with your willingness to afliate with people who are known for believing it—quite apart from whether the belief is actually true. Some people may be reluctant to believe that God does not exist, not because there is evidence that God does exist, but rather because they are reluctant to afliate with Richard Dawkins or those darned “strident” atheists who go around publicly saying “God does not exist.” (Page 272) - Barry is a famous geologist. Charles is a fourteen-year-old juvenile delinquent with a long arrest record and occasional psychotic episodes. Barry flatly asserts to Arthur some counterintuitive statement about rocks, and Arthur judges it 90% probable. Ten Charles makes an equally counterintuitive flat assertion about rocks, and Arthur judges it 10% probable. Clearly, Arthur is taking the speaker’s authority into account in deciding whether to believe the speaker’s assertions. Scenario 2: David (Page 274) - Barry is a famous geologist. Charles is a fourteen-year-old juvenile delinquent with a long arrest record and occasional psychotic episodes. Barry flatly asserts to Arthur some counterintuitive statement about rocks, and Arthur judges it 90% probable. Ten Charles makes an equally counterintuitive flat assertion about rocks, and Arthur judges it 10% probable. Clearly, Arthur is taking the speaker’s authority into account in deciding whether to believe the speaker’s assertions (Page 274) - P (SlipperyjNight; Sprinkler) = P (SlipperyjSprinkler (Page 277) - P (SlipperyjNight; Sprinkler) = P (SlipperyjSprinkler) : (Page 277) - Tere is an ineradicable legitimacy to assigning slightly higher probability to what E. T. Jaynes tells you about Bayesian probability, than you assign to Eliezer Yudkowsky making the exact same statement. Fify additional years of experience should not count for literally zero influence. But this slight strength of authority is only ceteris paribus, and can easily be overwhelmed by stronger arguments. I have a minor erratum in one of Jaynes’s books—because algebra trumps authority (Page 279) - Te more directly your arguments bear on a question, without intermediate inferences—the closer the observed nodes are to the queried node, in the Great Web of Causality—the more powerful the evidence. It’s a theorem of these causal graphs that you can never get more information from distant nodes, than from strictly closer nodes that screen of the distant ones (Page 280) - If you really want an artist’s perspective on rationality, then read Orwell (Page 282) - Orwell was the outraged opponent of totalitarianism and the muddy thinking in which evil cloaks itself—which is how Orwell’s writings on language ended up as classic rationalist documents on a level with Feynman, Sagan, or Dawkins (Page 282) - I wrote the sentence in the passive voice, without telling you who tells authors to avoid passive voice. Passive voice removes the actor, leaving only the acted-upon (Page 283) - My point is not to say that journal articles should be written like novels, but that a rationalist should become consciously aware of the experiences which words create (Page 284) - Orwell knew that muddled language is muddled thinking; he knew that human evil and muddled thinking intertwine like conjugate strands of DNA (Page 286) - political language has to consist largely of euphemism, question-begging and sheer cloudy vagueness (Page 287) - Freedom is the freedom to say that two plus two make four (Page 287) - Do you remember,” he went on, “writing in your diary, ‘Freedom is the freedom to say that two plus two make four’?” “Yes,” said Winston. O’Brien held up his lef hand, its back towards Winston, with the thumb hidden and the four fngers extended. “How many fngers am I holding up, Winston?” 287 (Page 287) - Freedom is the freedom to say that two plus two make four (Page 287) - Stuart Chase and others have come near to claiming that all abstract words are meaningless, and have used this as a pretext for advocating a kind of political quietism. Since you don’t know what Fascism is, how can you struggle against Fascism (Page 288) - Stuart Chase and others have come near to claiming that all abstract words are meaningless, and have used this as a pretext for advocating a kind of political quietism. Since you don’t know what Fascism is, how can you struggle against Fascism? (Page 288) - In all human history, every great leap forward has been driven by a new clarity of thought. Except for a few natural catastrophes, every great woe has been driven by a stupidity. Our last enemy is ourselves; and this is a war, and we are soldiers (Page 289) - If you’re irrational to start with, having more knowledge can hurt you (Page 294) - If you’re irrational to start with, having more knowledge can hurt you. For a true Bayesian, information would never have negative expected utility (Page 294) - I’ve seen people severely messed up by their own knowledge of biases. Tey have more ammunition with which to argue against anything they don’t like. And that problem—too much ready ammunition—is one of the primary ways that people with high mental agility end up stupid, in Stanovich’s “dysrationalia” sense of stupidity (Page 294) - For every expectation of evidence, there is an equal and opposite expectation of counterevidence (Page 296) - Whatever. Rationality is not for winning debates, it is for deciding which side to join. If you’ve already decided which side to argue for, the work of rationality is done within you, whether well or poorly (Page 297) - Rationality is not for winning debates, it is for deciding which side to join. If you’ve already decided which side to argue for, the work of rationality is done within you, whether well or poorly (Page 297) - In Te Bottom Line, I presented the dilemma of two boxes, only one of which contains a diamond, with various signs and portents as evidence. I dichotomized the curious inquirer and the clever arguer. Te curious inquirer writes down all the signs and portents, and processes them, and fnally writes down “Terefore, I estimate an 85% probability that box B contains the diamond.” Te clever arguer works for the highest bidder, and begins by writing, “Terefore, box B contains the diamond,” and then selects favorable signs and portents to list on the lines above. Te frst procedure is rationality. Te second procedure is generally known as “rationalization (Page 309) - Rationality is the operation that we use to obtain more accuracy for our beliefs by changing them (Page 310) - Rationalization operates to fx beliefs in place; it would be better named “anti-rationality,” both for its pragmatic results and for its reversed algorithm (Page 310) - “Rationalization” is a backward flow from conclusion to selected evidence. First you write down the bottom line, which is known and fxed; the purpose of your processing is to fnd out which arguments you should write down on the lines above. Tis, not the bottom line, is the variable unknown to the running process (Page 310) - Curiosity is the frst virtue, without which your questioning (Page 311) - In Orthodox Judaism you’re allowed to notice inconsistencies and contradictions, but only for purposes of explaining them away, and whoever comes up with the most complicated explanation gets a prize (Page 316) - Is honesty the best policy? I don’t know if I’d go that far: Even on my ethics, it’s sometimes okay to shut up. But compared to outright lies, either honesty or silence involves less exposure to recursively propagating risks you don’t know you’re taking (Page 333) - Te dangerous thing is to have a false belief that you believe should be protected as a belief—a belief-in-belief, whether or not accompanied by actual belief (Page 337) - Tis discipline I named singlethink, afer Orwell’s doublethink. In doublethink, you forget, and then forget you have forgotten. In singlethink, you notice you are forgetting, and then you remember. You hold only a single non-contradictory thought in your mind at once (Page 344) - You can’t know the consequences of being biased, until you have already debiased yourself. And then it is too late for self-deception (Page 347) - totally irrelevant “information” can influence estimates and decisions. In the feld of heuristics and biases, this more general phenomenon is known as contamination.2 Early research (Page 369) - totally irrelevant “information” can influence estimates and decisions. In the feld of heuristics and biases, this more general phenomenon is known as contamination (Page 369) - In modern civilization particularly, no one can think fast enough to think their own thoughts (Page 375) - If you want to sound deep, you can never say anything that is more than a single step of inferential distance away from your listener’s current mental state (Page 397) - I know transhumanists who are unable to seem deep because they are unable to appreciate what their listener does not already know. If you want to sound deep, you can never say anything that is more than a single step of inferential distance away from your listener’s current mental state (Page 397) - Te principle of the bottom line is that only the actual causes of your beliefs determine your efectiveness as a rationalist. Once your belief is fxed, no amount of argument will alter the truth-value; once your decision is fxed, no amount of argument will alter the consequences (Page 400) - In lists of logical fallacies, you will fnd included “the genetic fallacy”—the fallacy of attacking a belief based on someone’s causes for believing it. (Page 404) - Te afect heuristic is how an overall feeling of goodness or badness contributes to many other judgments, whether it’s logical or not, whether you’re aware of it or not. (Page 422) - Research has shown that we automatically assign to good-looking individuals such favorable traits as talent, kindness, honesty, and intelligence (Page 422) - the halo efect, which causes people to see all positive characteristics as correlated—for example, more attractive individuals are also perceived as more kindly, honest, and intelligent—causes us to admire heroes more if they’re super-strong and immune to bullets (Page 430) - Your “zombie,” in the philosophical usage of the term, is putatively a being that is exactly like you in every respect—identical behavior, identical speech, identical brain; every atom and quark in exactly the same position, moving according to the same causal laws of motion—except that your zombie is not conscious (Page 1012) - Intelligence has nothing to do with wisdom or being a good person”—oh, and does self-awareness have nothing to do with wisdom, or being a good person? Modeling yourself takes intelligence. For one thing, it takes enough intelligence to learn evolutionary psychology (Page 1536) - You can’t get to the defnition of fre if you don’t know about atoms and molecules; you’re better of saying “that orangey-bright thing.” (Page 1542)