pradeep90.github.com/Mysterious-Answers-to-Mysterious-Questions-Learning.html at master · pradeep90/pradeep90.github.com · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
	<link rel="icon" type="image/png" href="./images/favicon-32x32.png" sizes="32x32" />
	<link rel="icon" type="image/png" href="./images/favicon-16x16.png" sizes="16x16" />
        <title>Mysterious Answers to Mysterious Questions - SPK's Rationality Essays</title>
        <link rel="stylesheet" type="text/css" href="./css/default.css" />
        <link rel="stylesheet" type="text/css" href="./css/highlight.css" />

	<!-- <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script> -->
	<!-- <script type="text/javascript" src="/js/header-links.js"></script> -->
	<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>

	<link href="atom.xml" type="application/atom+xml" rel="alternate" title="Sitewide ATOM/RSS Feed" />

	<!-- Google Analytics stuff -->
	<!-- Google tag (gtag.js) -->
	<script async src="https://www.googletagmanager.com/gtag/js?id=G-DEWF2J5BG8"></script>
	<script>
	  window.dataLayer = window.dataLayer || [];
	  function gtag(){dataLayer.push(arguments);}
	  gtag('js', new Date());

	  gtag('config', 'G-DEWF2J5BG8');
	</script>

	<script type="text/javascript" src="https://fast.fonts.net/jsapi/f7f47a40-b25b-44ee-9f9c-cfdfc8bb2741.js"></script>
    </head>
    <body>
        <div id="header">
            <div id="logo">
                <a href="./">SPK's Rationality Essays</a>
            </div>
            <div id="navigation">
                <a href="./">Home</a>
                <a href="./notes.html">Notes</a>
                <!-- <a href="/about.html">About</a> -->
                <a href="./archive.html">Archive</a>
		<a href="./atom.xml" type="application/atom+xml" rel="alternate" title="Sitewide ATOM/RSS Feed">RSS</a>
            </div>
        </div>

        <div id="content">
          <h1 id="post-title">Mysterious Answers to Mysterious Questions</h1>

            <!-- <center><img src="https://fbcdn-sphotos-d-a.akamaihd.net/hphotos-ak-prn1/t31.0-8/p600x600/10257116_10202295769100492_2438594605053717342_o.jpg" height="400" width="300" class="sujeet-pic" alt="Sujeet pic" /></center> -->

<p>These are my notes for Eliezer’s Sequence on <a href>Mysterious Answers to Mysterious Questions</a>.</p>
<h1 id="make-beliefs-pay-rent-in-anticipated-consequences">Make Beliefs Pay Rent in Anticipated Consequences</h1>
<p>Talk about the Random Experiments for which you make different predictions.</p>
<p>In other words, talk about Differing Predictions for your hypotheses.</p>
<p>Your hypotheses may postulate things that you can’t sense directly, but which will make themselves felt in indirect ways. i.e., its Anticipated Consequences!</p>
<p>Humans can make a network of causes - beliefs - that are not connected to any anticipated consequences. There is no type-checking mechanism for our beliefs. This is a “floating” belief.</p>
<blockquote>
<p>The rationalist virtue of empiricism consists of constantly asking which experiences our beliefs predict - or better yet, prohibit.</p>
</blockquote>
<p><strong>Question</strong>: Is it just about getting empirical evidence instead of going by what your mind says? Or is it about asking which experiences our beliefs predict?</p>
<p>I think it is useful to do so. No point talking about “definitions”.</p>
<p>Empiricism, for me, is the opposite of Naive Realism - get evidence. Anticipation-Constraints is about asking what you predict and forbid.</p>
<p>Anticipation-Constraints: What must <em>not</em> happen to you?</p>
<blockquote>
<p>Above all, don’t ask what to believe - ask what to <em>anticipate</em>. Every question of belief should flow from a question of anticipation, and that question of anticipation should be the center of the inquiry. Every guess of belief should begin by flowing to a specific guess of anticipation, and should continue to pay rent in future anticipations. If a belief turns deadbeat, evict it.</p>
</blockquote>
<p>Have a bunch of Random Experiments that you care about. The center of the inquiry should be what the predictions should be.</p>
<h1 id="belief-in-belief">Belief in Belief</h1>
<blockquote>
<p>The claimant must have an accurate model of the situation somewhere in his mind, because he can anticipate, in advance, exactly which experimental results he’ll need to excuse.</p>
</blockquote>
<p>If you’re making correct Anticipation-Constraints, you must have the right beliefs in your head somewhere.</p>
<blockquote>
<p>If all our thoughts were deliberate verbal sentences like philosophers manipulate, the human mind would be a great deal easier for humans to understand. Fleeting mental images, unspoken flinches, desires acted upon without acknowledgement-these account for as much of ourselves as words.</p>
</blockquote>
<blockquote>
<p>But we need a wider concept of belief, not limited to verbal sentences. “Belief” should include unspoken anticipation-controllers. “Belief in belief” should include unspoken cognitive-behavior-guiders.</p>
</blockquote>
<blockquote>
<p>But it is realistic to say the dragon-claimant anticipates as if there is no dragon in his garage, and makes excuses as if he believed in the belief.</p>
</blockquote>
<p>There can be unspoken anticipation-controllers in your mind and also excuse-controllers.</p>
<blockquote>
<p>If someone believes in their belief in the dragon, and also believes in the dragon, the problem is much less severe. They will be willing to stick their neck out on experimental predictions, and perhaps even agree to give up the belief if the experimental prediction is wrong-although belief in belief can still interfere with this, if the belief itself is not absolutely confident.</p>
</blockquote>
<h1 id="bayesian-judo">Bayesian Judo</h1>
<p>Take beliefs people have and come up with forbidden actions that will falsify those beliefs.</p>
<p>Like when they say something is impossible or cannot happen.</p>
<h1 id="professing-and-cheering">Professing and Cheering</h1>
<p>Three types of beliefs (so far): anticipation-controller, belief in belief (excuse-controller), professing and cheering.</p>
<h1 id="belief-as-attire">Belief as Attire</h1>
<p>Another form of improper belief: group-identification.</p>
<blockquote>
<p>The very concept of the courage and altruism of a suicide bomber is Enemy attire - you can tell, because the Enemy talks about it. The cowardice and sociopathy of a suicide bomber is American attire. There are no quote marks you can use to talk about how the Enemy sees the world; it would be like dressing up as a Nazi for Halloween.</p>
</blockquote>
<blockquote>
<p>Identifying with a tribe is a very strong emotional force. People will die for it. And once you get people to identify with a tribe, the beliefs which are attire of that tribe will be spoken with the full passion of belonging to that tribe.</p>
</blockquote>
<h1 id="focus-your-uncertainty">Focus your Uncertainty</h1>
<p>Subjective Probability.</p>
<p>Mutually exclusive outcomes. Have to spread probability among them.</p>
<blockquote>
<p>You’re pretty sure you weren’t taught anything like that in your statistics courses. They didn’t tell you what to do when you felt so terribly uncertain. They didn’t tell you what to do when there were no little numbers handed to you.</p>
</blockquote>
<blockquote>
<p>If only there were an art of focusing your uncertainty - of squeezing as much anticipation as possible into whichever outcome will actually happen!</p>
</blockquote>
<h1 id="the-virtue-of-narrowness">The Virtue of Narrowness</h1>
<blockquote>
<p>Outside their own professions, people often commit the misstep of trying to broaden a word as widely as possible, to cover as much territory as possible.</p>
</blockquote>
<blockquote>
<p>It is a necessary part of the rationalist’s art - or even the poet’s art! - to focus narrowly on unusual pebbles which possess some special quality.</p>
</blockquote>
<blockquote>
<p>When the unenlightened ones try to be profound, they draw endless verbal comparisons between this topic, and that topic, which is like this, which is like that; until their graph is fully connected and also totally useless. The <em>remedy</em> is specific knowledge and in-depth study. When you understand things in detail, you can see how they are <em>not alike</em>, and start enthusiastically subtracting edges off your graph.</p>
</blockquote>
<blockquote>
<p>There’s nothing wrong with focusing your mind, narrowing your categories, excluding possibilities, and sharpening your propositions. Really, there isn’t! If you make your words too broad, you end up with something that isn’t true and doesn’t even make good poetry.</p>
</blockquote>
<p>Be careful about stretching your analogies and connections too far.</p>
<h1 id="your-strength-as-a-rationalist">Your Strength as a Rationalist</h1>
<blockquote>
<p>So instead, by dint of mighty straining, I forced my model of reality to explain an anomaly that never actually happened. And I knew how embarrassing this was. I knew that the usefulness of a model is not what it can explain, but what it can’t.</p>
</blockquote>
<blockquote>
<p>A hypothesis that forbids nothing, permits everything, and thereby fails to constrain anticipation.</p>
</blockquote>
<blockquote>
<p>Your strength as a rationalist is your ability to be more confused by fiction than by reality. If you are equally good at explaining any outcome, you have zero knowledge.</p>
</blockquote>
<blockquote>
<p>I had all the information I needed to arrive at the correct answer, I even noticed the problem, and then I ignored it. My feeling of confusion was a Clue, and I threw my Clue away.</p>
</blockquote>
<p>Pay attention to your feeling of surprise.</p>
<blockquote>
<p>I should have paid more attention to that sensation of <em>still feels a little forced</em>. It’s one of the most important feelings a truthseeker can have, a part of your strength as a rationalist. It is a design flaw in human cognition that this sensation manifests as a quiet strain in the back of your mind, instead of a wailing alarm siren and a glowing neon sign reading “EITHER YOUR MODEL IS FALSE OR THIS STORY IS WRONG.”</p>
</blockquote>
<h1 id="absence-of-evidence-is-evidence-of-absence">Absence of Evidence is Evidence of Absence</h1>
<p>Likelihood ratio of E = 4/1 (let’s say)</p>
<p>P(E/H1) = 0.80; P(E/H2) = 0.20</p>
<p>Likelihood ratio of ~E = 1/4</p>
<p>P(~E/H1) = 0.20; P(~E/H2) = 0.80</p>
<p>Your outcome not happening divides your prior odds by that ratio.</p>
<blockquote>
<p>The absence of an observation that is only weakly permitted (even if the alternative hypothesis does not allow it at all), is very weak evidence of absence (though it is evidence nonetheless). This is the fallacy of “gaps in the fossil record” - fossils form only rarely; it is futile to trumpet the absence of a weakly permitted observation when many strong positive observations have already been recorded.</p>
</blockquote>
<blockquote>
<p>The strength of a model is not what it can explain, but what it can’t, for only prohibitions constrain anticipation. If you don’t notice when your model makes the evidence unlikely, you might as well have no model, and also you might as well have no evidence; no brain and no eyes.</p>
</blockquote>
<h1 id="conservation-of-expected-evidence">Conservation of Expected Evidence</h1>
<p>P(H) = P(H/E) P(E) + P(H/~E) P(~E)</p>
<blockquote>
<p>Therefore, for every expectation of evidence, there is an equal and opposite expectation of counterevidence.</p>
</blockquote>
<blockquote>
<p>If you expect a strong probability of seeing weak evidence in one direction, it must be balanced by a weak expectation of seeing strong evidence in the other direction.</p>
</blockquote>
<blockquote>
<p>If you’re very confident in your theory, and therefore anticipate seeing an outcome that matches your hypothesis, this can only provide a very small increment to your belief (it is already close to 1); but the unexpected failure of your prediction would (and must) deal your confidence a <em>huge blow</em>.</p>
</blockquote>
<blockquote>
<p>On average, you must expect to be exactly as confident as when you started out. Equivalently, the mere expectation of encountering evidence - before you’ve actually seen it - should not shift your prior beliefs.</p>
</blockquote>
<p><strong>Corollary</strong>: p * (P(H/E) - P(H)) = (1 - p) * (P(H) - P(H/~E))</p>
<p>where p = P(E)</p>
<p>The <em>change</em> in prior is inversely proportional to the probability of the evidence.</p>
<p>If you want to go up a lot with 20% probability, you must go down a little with 80% probability.</p>
<p>If you want to go up a lot with 80% probability, you must go down a lot lot more with 20% probability.</p>
<p><strong>Simple Corollary</strong>: If seeing some evidence, with whatever probability, makes your posterior go up, then <em>not</em> seeing that evidence must make it go down.</p>
<blockquote>
<p>For a true Bayesian, it is impossible to seek evidence that confirms a theory. There is no possible plan you can devise, no clever strategy, no cunning device, by which you can legitimately expect your confidence in a fixed proposition to be higher (on average) than before. You can only ever seek evidence to test a theory, not to confirm it.</p>
</blockquote>
<blockquote>
<p>This realization can take quite a load off your mind. You need not worry about how to interpret every possible experimental result to confirm your theory. You needn’t bother planning how to make any given iota of evidence confirm your theory, because you know that for every expectation of evidence, there is an equal and oppositive expectation of counterevidence. If you try to weaken the counterevidence of a possible “abnormal” observation, you can only do it by weakening the support of a “normal” observation, to a precisely equal and opposite degree. It is a zero-sum game. No matter how you connive, no matter how you argue, no matter how you strategize, you can’t possibly expect the resulting game plan to shift your beliefs (on average) in a particular direction.</p>
</blockquote>
<h1 id="hindsight-bias">Hindsight Bias</h1>
<blockquote>
<p>Hindsight bias is when people who know the answer vastly overestimate its predictability or obviousness, compared to the estimates of subjects who must guess without advance knowledge. Hindsight bias is sometimes called the I-knew-it-all-along effect.</p>
</blockquote>
<p>So, if you know what happened, you will give it a much higher <em>likelihood</em> in retrospect.</p>
<p>You will tell yourself that your hypothesis predicted this more strongly than it did.</p>
<p>So, you won’t realize how bad your hypothesis really is.</p>
<blockquote>
<p>But preventing the Challenger disaster would have required, not attending to the problem with the O-rings, but attending to every warning sign which seemed as severe as the O-ring problem, without benefit of hindsight. It could have been done, but it would have required a general policy much more expensive than just fixing the O-Rings.</p>
</blockquote>
<blockquote>
<p>We don’t learn the general lesson: the cost of effective caution is very high because you must attend to problems that are not as obvious now as past problems seem in hindsight.</p>
</blockquote>
<blockquote>
<p>The test of a model is how much probability it assigns to the observed outcome. Hindsight bias systematically distorts this test; we think our model assigned much more probability than it actually did.</p>
</blockquote>
<blockquote>
<p>When we attempt to understand past events, we implicitly test the hypotheses or rules we use both to interpret and to anticipate the world around us. If, in hindsight, we systematically underestimate the surprises that the past held and holds for us, we are subjecting those hypotheses to inordinately weak tests and, presumably, finding little reason to change them.</p>
</blockquote>
<h1 id="hindsight-devalues-science">Hindsight devalues Science</h1>
<blockquote>
<p>Do your thought processes at this point, where you really don’t know the answer, feel different from the thought processes you used to rationalize either side of the “known” answer?</p>
</blockquote>
<p>When you don’t know the answer, your likelihoods for the various outcomes are much more like that in foresight - total confusion. When you do know the answer, you start falling for the hindsight bias.</p>
<blockquote>
<p>Hindsight will lead us to systematically undervalue the surprisingness of scientific findings, especially the discoveries we understand - the ones that seem real to us, the ones we can retrofit into our models of the world.</p>
</blockquote>
<blockquote>
<p>If you understand neurology or physics and read news in that topic, then you probably underestimate the surprisingness of findings in those fields too. This unfairly devalues the contribution of the researchers; and worse, will prevent you from noticing when you are seeing evidence that doesn’t fit what you really would have expected.</p>
</blockquote>
<blockquote>
<p>We need to make a conscious effort to be shocked enough.</p>
</blockquote>
<p>You need to look at it from the eyes of a person in foresight, someone who doesn’t know what actually happened.</p>
<p>What would you have predicted?</p>
<p><strong>Important Action</strong>: Predict before you learn what actually happened - everywhere.</p>
<p>And if your model takes a hit, admit it immediately. Fail Faster!</p>
<p>In this way, you will attain enlightenment.</p>
<h1 id="fake-explanations">Fake Explanations</h1>
<blockquote>
<p>Ponder that innocent little phrase, “because of”, which comes before “heat conduction”.</p>
</blockquote>
<blockquote>
<p>A true master of the art of using numbers to constrain the anticipation of material phenomena - a “physicist”</p>
</blockquote>
<blockquote>
<p>The deeper error of the students is not simply that they failed to constrain anticipation. Their deeper error is that they thought they were doing physics. They said the phrase “because of”, followed by the sort of words Spock might say on Star Trek, and thought they thereby entered the magisterium of science.</p>
</blockquote>
<p>Beware of “because”.</p>
<p>Naive Realism - You think you can Anticipation-Constrain it properly, but you really can’t.</p>
<h1 id="guessing-the-teachers-password">Guessing the Teacher’s Password</h1>
<blockquote>
<p>Words do not have intrinsic definitions. If I hear the syllables “bea-ver” and think of a large rodent, that is a fact about my own state of mind, not a fact about the syllables “bea-ver”. The sequence of syllables “made of waves” (or “because of heat conduction”) is not a hypothesis, it is a pattern of vibrations traveling through the air, or ink on paper. It can associate to a hypothesis in someone’s mind, but it is not, of itself, right or wrong.</p>
</blockquote>
<p>They are just words. You need to interpret them to come up with a hypothesis in your head that makes Anticipation-Constraints.</p>
<blockquote>
<p>In the school system, it’s all about verbal behavior, whether written on paper or spoken aloud. Verbal behavior gets you a gold star or a failing grade. Part of unlearning this bad habit is becoming consciously aware of the difference between an explanation and a password.</p>
</blockquote>
<blockquote>
<p>It happened to me when I was nine years old - not because I was stupid, but because this is what happens by default. This is how human beings think, unless they are trained not to fall into the trap. Humanity stayed stuck in holes like this for thousands of years.</p>
</blockquote>
<blockquote>
<p>Maybe, if we drill students that words don’t count, only anticipation-controllers, the student will not get stuck on “Heat conduction? No? Maybe heat convection? That’s not it either?”</p>
</blockquote>
<p>Words don’t count, only Anticipation-Constraints do.</p>
<h1 id="science-as-attire">Science as Attire</h1>
<blockquote>
<p>If evolutionary theory could actually stretch to cover Storm, it would be able to explain anything, and we all know what that would imply.</p>
</blockquote>
<blockquote>
<p>The X-Men comics use terms like “evolution”, “mutation”, and “genetic code”, purely to place themselves in what they conceive to be the literary genre of science. The part that scares me is wondering how many people, especially in the media, understand science only as a literary genre.</p>
</blockquote>
<blockquote>
<p>I encounter people who very definitely believe in evolution, who sneer at the folly of creationists. And yet they have no idea of what the theory of evolutionary biology permits and prohibits. They’ll talk about “the next step in the evolution of humanity”, as if natural selection got here by following a plan.</p>
</blockquote>
<blockquote>
<p>They understand science only as a literary genre, or in-group to belong to. The attire doesn’t look to them like a lab coat; this isn’t the football team they’re cheering for.</p>
</blockquote>
<blockquote>
<p>Is there anything in science that you are proud of believing, and yet you do not <em>use</em> the belief professionally? You had best ask yourself which future experiences your belief prohibits from happening to you. That is the sum of what you have assimilated and made a true part of yourself. Anything else is probably passwords or attire.</p>
</blockquote>
<h1 id="fake-causality">Fake Causality</h1>
<blockquote>
<p>You couldn’t even use phlogiston theory to say what you ought not to see; it could explain everything.</p>
</blockquote>
<p>There wasn’t a causal model that would let you make predictions for different configurations of the environment.</p>
<blockquote>
<p>Fake explanations don’t feel fake. That’s what makes them dangerous.</p>
</blockquote>
<p>Naive Realism.</p>
<blockquote>
<p>One of the primary inspirations for Bayesian networks was noticing the problem of <em>double-counting evidence</em> if inference resonates between an effect and a cause. For example, let’s say that I get a bit of unreliable information that the sidewalk is wet. This should make me think it’s more likely to be raining. But, if it’s more likely to be raining, doesn’t that make it more likely that the sidewalk is wet? And wouldn’t that make it more likely that the sidewalk is slippery? But if the sidewalk is slippery, it’s probably wet; and then I should again raise my probability that it’s raining…</p>
</blockquote>
<blockquote>
<p>If you learn something about whether it’s raining, from some source <em>other</em> than observing the sidewalk to be wet, this will send a forward-message from [rain] to [sidewalk wet] and raise our expectation of the sidewalk being wet. If you observe the sidewalk to be wet, this sends a backward-message to our belief that it is raining, and this message propagates from [rain] to all neighboring nodes except the [sidewalk wet] node. We count each piece of evidence exactly once; no update message ever “bounces” back and forth.</p>
</blockquote>
<p>Or, you could start from zero - create a fresh hypothesis with some prior probability (based on its algorithmic complexity) and then update on all the evidence.</p>
<blockquote>
<p>So what went wrong in phlogiston theory? When we observe that fire is hot, the [fire] node can send a backward-evidence to the [“phlogiston”] node, leading us to update our beliefs about phlogiston. But if so, we can’t count this as a successful forward-prediction of phlogiston theory. The message should go in only one direction, and not bounce back.</p>
</blockquote>
<p>Don’t double-count your evidence.</p>
<p>Either use evidence to come up with some hypothesis or update the posterior probability based on some evidence.</p>
<p>You can’t create a hypothesis based on some evidence and then count that evidence as more evidence for that hypothesis.</p>
<hr />
<p><strong>Key point</strong>: In other words, the worth of a hypothesis is decided based on the successful forward-predictions it makes. What evidence can you find for it that you did not use to create it?</p>
<p><strong>Central Question</strong>: What new Predictive Power do I get because of this theory?</p>
<hr />
<blockquote>
<p>Speaking of “hindsight bias” is just the nontechnical way of saying that humans do not rigorously separate forward and backward messages, allowing forward messages to be contaminated by backward ones.</p>
</blockquote>
<p>Because of the backward message from the evidence and our faulty algorithm, we send a forward message that increases the probability of that outcome - we put more likelihood on that outcome than we should.</p>
<p>Basically, it’s a flow of probability backward from the evidence to the hypothesis. But then, we send a flow of probability forward to the evidence, making it seem more likely from this hypothesis than it should be.</p>
<blockquote>
<p>Thanks to hindsight bias, it’s also not enough to check how well your theory “predicts” facts you already know. You’ve got to predict for tomorrow, not yesterday. It’s the only way a messy human mind can be <em>guaranteed</em> of sending a <em>pure forward message</em>.</p>
</blockquote>
<p>We go and look at the historical evidence and say, hey, my hypothesis predicts this correctly, so it’s a good hypothesis.</p>
<h1 id="semantic-stopsigns">Semantic Stopsigns</h1>
<blockquote>
<p>When seemingly unanswerable questions are answered with, say, “God did it,” that answer doesn’t resolve the question, but it does tell you to stop asking further questions; it functions as a semantic stopsign.</p>
</blockquote>
<blockquote>
<p>If you’re tempted to solve any problem or explain any event with a word like “government” or “big business” or “terrorism,” and you fail to ask the obvious next question “How exactly does [government|business|terrorism] explain this thing or solve this problem?”, then that word is a semantic stopsign for you.</p>
</blockquote>
<blockquote>
<p>What distinguishes a semantic stopsign is failure to consider the obvious next question.</p>
</blockquote>
<p>Ask how. Look for the Anticipation-Constraints. Create a simpler hypothesis. Ask how the different parts of the hypothesis Anticipation-Constrain outcomes.</p>
<p>I think this is mainly Naive Realism - you think you have got a lot of Predictive Power behind that label, but you really don’t.</p>
<h1 id="mysterious-answers-to-mysterious-questions">Mysterious Answers to Mysterious Questions</h1>
<blockquote>
<p>When you say “Elan vital!”, it feels like you know why your hand moves. You have a little causal diagram in your head that says [“Elan vital!”] -&gt; [hand moves]. But actually you know nothing you didn’t know before. You don’t know, say, whether your hand will generate heat or absorb heat, unless you have observed the fact already; if not, you won’t be able to predict it in advance. Your curiosity feels sated, but it hasn’t been fed. Since you can say “Why? Elan vital!” to any possible observation, it is equally good at explaining all outcomes, a disguised hypothesis of maximum entropy, etcetera.</p>
</blockquote>
<p>Naive Realism - you feel you are able to Anticipation-Constrain properly, but you’re not.</p>
<p><strong>Important</strong>: You don’t get extra Predictive Power.</p>
<blockquote>
<p>But ignorance exists in the map, not in the territory. If I am ignorant about a phenomenon, that is a fact about my own state of mind, not a fact about the phenomenon itself. A phenomenon can seem mysterious to some particular person. There are no phenomena which are mysterious of themselves. To worship a phenomenon because it seems so wonderfully mysterious, is to worship your own ignorance.</p>
</blockquote>
<p>The world <em>seems</em> confusing, but it isn’t really confusing. It’s your Map that is blurry and vague.</p>
<hr />
<p><strong>Question</strong>: How do we get new Predictive Power, Alfred?</p>
<p><strong>Answer</strong>: Focus your uncertainty.</p>
<p>This is the <em>only</em> way you can get more Predictive Power. Make narrow predictions.</p>
<p>By default, you have maximum-entropy predictions - equal weight on all outcomes.</p>
<p>So, to make narrow predictions, you need to take probability weight <em>off</em> certain outcomes and put it on a few narrow outcomes, hopefully one.</p>
<p>It involves two things - putting more weight on certain outcomes and therefore putting <em>less</em> weight on others. You need to forbid certain outcomes.</p>
<p><strong>Aim</strong>: Focus your Uncertainty!</p>
<p><strong>Question</strong>: What does it forbid?</p>
<p><strong>Note</strong>: For us humans, increasing Predictive Power can also be about Anticipation-Constraining Random Experiments that we hadn’t considered earlier. It is still focussing your uncertainty (the default is a random prediction), it’s just easier to visualize it as getting Predictive Power about entirely new Random Experiments.</p>
<p>So, either focus your uncertainty for Random Experiments that you’re already thinking about, or make predictions for <em>new</em> Random Experiments.</p>
<hr />
<blockquote>
<p>It is a failure of human psychology that, faced with a mysterious phenomenon, we more readily postulate mysterious inherent substances than complex underlying processes.</p>
</blockquote>
<blockquote>
<p>But the deeper failure is supposing that an answer can be mysterious. If a phenomenon feels mysterious, that is a fact about our state of knowledge, not a fact about the phenomenon itself.</p>
</blockquote>
<blockquote>
<p>This is the ultimate and fully general explanation for why, again and again in humanity’s history, people are shocked to discover that an incredibly mysterious question has a non-mysterious answer. Mystery is a property of questions, not answers.</p>
</blockquote>
<p>Naive Realism. We think that because the answer <em>seems</em> mysterious, it must <em>be</em> mysterious.</p>
<p>Mysterious Answers to Mysterious Questions = Naive Realism (Curiosity-Stopper, Wonderfully Mysterious) + Cherish their ignorance + No causal mechanism (No new Predictive Power)</p>
<h1 id="the-futility-of-emergence">The Futility of Emergence</h1>
<p>Emergence.</p>
<p>What new Predictive Power does it give us? Where does it focus our uncertainty?</p>
<p><strong>Question</strong>: What outcome does it forbid?</p>
<blockquote>
<p>“Intelligence is an emergent phenomenon!”</p>
</blockquote>
<blockquote>
<p>It feels like you believe a new fact, but you don’t anticipate any different outcomes. Your curiosity feels sated, but it has not been fed. The hypothesis has no moving parts - there’s no detailed internal model to manipulate. Those who proffer the hypothesis of “emergence” confess their ignorance of the internals, and take pride in it; they contrast the science of “emergence” to other sciences merely mundane.</p>
</blockquote>
<blockquote>
<p>Before: Life is an emergent phenomenon. After: Life is a magical phenomenon.</p>
<p>Before: Human intelligence is an emergent product of neurons firing. After: Human intelligence is a magical product of neurons firing.</p>
</blockquote>
<blockquote>
<p>“Emergence” has become very popular, just as saying “magic” used to be very popular. “Emergence” has the same deep appeal to human psychology, for the same reason. “Emergence” is such a wonderfully easy explanation, and it feels good to say it; it gives you a sacred mystery to worship. Emergence is popular because it is the junk food of curiosity. You can explain anything using emergence, and so people do just that; for it <em>feels so wonderful</em> to explain things. Humans are still humans, even if they’ve taken a few science classes in college. Once they find a way to escape the shackles of settled science, they get up to the same shenanigans as their ancestors, dressed up in the <em>literary genre</em> of “science” but still the same species psychology.</p>
</blockquote>
<h1 id="say-not-complexity">Say not “Complexity”</h1>
<blockquote>
<p>What you must avoid is skipping over the mysterious part; you must linger at the mystery to confront it directly. There are many words that can skip over mysteries, and some of them would be legitimate in other contexts - “complexity”, for example. But the essential mistake is that skip-over, regardless of what causal node goes behind it.</p>
</blockquote>
<blockquote>
<p>And when you train yourself to avoid skipping, it will become a matter of instinct, not verbal reasoning. You have to feel which parts of your map are still blank, and more importantly, pay attention to that feeling.</p>
</blockquote>
<blockquote>
<p>Marcello and I developed a convention in our AI work: when we ran into something we didn’t understand, which was often, we would say “magic” - as in, “X magically does Y” - to remind ourselves that here was an unsolved problem, a gap in our understanding. It is far better to say “magic”, than “complexity” or “emergence”; the latter words create an illusion of understanding. Wiser to say “magic”, and leave yourself a placeholder, a reminder of work you will have to do later.</p>
</blockquote>
<h1 id="positive-bias-look-into-the-dark">Positive Bias: Look into the Dark</h1>
<p>2-4-6 challenge.</p>
<p>Find a hypothesis that accepts or rejects triplets.</p>
<p>Initial hypothesis seems to be (even + increments of 2 + positive).</p>
<p>What does this forbid? Odd, or some other increments (or decrements or just any pattern), maybe negative.</p>
<p>Positive Bias: We test whether something is accepted by our hypothesis, rather than rejected.</p>
<p>We look for the existence of evidence we predict rather than the non-existence of evidence we forbid.</p>
<h1 id="my-wild-and-reckless-youth">My Wild and Reckless Youth</h1>
<blockquote>
<p>I knew about Occam’s Razor, but not the conjunction fallacy. I thought I could get away with thinking complicated thoughts myself, in the literary style of the complicated thoughts I read in science books, not realizing that correct complexity is only possible when every step is pinned down overwhelmingly. Today, one of the chief pieces of advice I give to aspiring young rationalists is “Do not attempt long chains of reasoning or complicated plans.”</p>
</blockquote>
<blockquote>
<p>But my hypothesis made no retrospective predictions. According to Traditional Science, retrospective predictions don’t count - so why bother making them? To a Bayesian, on the other hand, if a hypothesis does not today have a favorable likelihood ratio over “I don’t know”, it raises the question of why you today believe anything more complicated than “I don’t know”.</p>
</blockquote>
<blockquote>
<p>When I think about how my younger self very carefully followed the rules of Traditional Rationality in the course of getting the answer wrong, it sheds light on the question of why people who call themselves “rationalists” do not rule the world. You need <em>one whole hell of a lot</em> of rationality before it does anything but lead you into new and interesting mistakes.</p>
</blockquote>
<blockquote>
<p>Traditional Rationality doesn’t have the ideal that thinking is an exact art in which there is only one correct probability estimate given the evidence. In Traditional Rationality, you’re allowed to guess, and then test your guess. But experience has taught me that if you don’t know, and you guess, you’ll end up being wrong.</p>
</blockquote>
<p>The Dance of Bayes.</p>
<h1 id="failing-to-learn-from-history">Failing to Learn from History</h1>
<blockquote>
<p>Many failures occurred in sequence, but one mistake stands out as most critical: My younger self did not realize that solving a mystery should make it feel less confusing. I was trying to explain a Mysterious Phenomenon - which to me meant providing a cause for it, fitting it into an integrated model of reality. Why should this make the phenomenon less Mysterious, when that is its nature? I was trying to explain the Mysterious Phenomenon, not render it (by some impossible alchemy) into a mundane phenomenon, a phenomenon that wouldn’t even call out for an unusual explanation in the first place.</p>
</blockquote>
<blockquote>
<p>It was something new, something stranger, something more difficult, something that ordinary science had failed to explain for centuries -</p>
</blockquote>
<blockquote>
<p>as if stars and matter and life had not been mysteries for hundreds of years and thousands of years, from the dawn of human thought right up until science finally solved them -</p>
</blockquote>
<blockquote>
<p>It was only afterward, when I began to see the mundane structure inside the mystery, that I realized whose shoes I was standing in. Only then did I realize how reasonable vitalism had seemed at the time, how surprising and embarrassing had been the universe’s reply of, “Life is mundane, and does not need a weird explanation.</p>
</blockquote>
<p>Wow!</p>
<h1 id="making-history-available">Making History Available</h1>
<blockquote>
<p>There is an inverse error to generalizing from fictional evidence: failing to be sufficiently moved by historical evidence. The trouble with generalizing from fictional evidence is that it is fiction - it never actually happened. It’s not drawn from the same distribution as this, our real universe; fiction differs from reality in systematic ways. But history <em>has</em> happened, and <em>should</em> be available.</p>
</blockquote>
<blockquote>
<p>In our ancestral environment, there were no movies; what you saw with your own eyes was true. Is it any wonder that fictions we see in lifelike moving pictures have too great an impact on us? Conversely, things that <em>really happened</em>, we encounter as ink on paper; they happened, but we never saw them happen. We don’t <em>remember</em> them happening to us.</p>
</blockquote>
<blockquote>
<p>When I finally realized whose shoes I was standing in, there was a sudden shock of unexpected connection with the past. I realized that the invention and destruction of vitalism - which I had only read about in books - had actually happened to real people, who experienced it much the same way I experienced the invention and destruction of my own mysterious answer.</p>
</blockquote>
<blockquote>
<p>So (I thought), to feel sufficiently the force of history, I should try to approximate the thoughts of an Eliezer who had lived through history - I should try to think as if everything I read about in history books, had actually happened to me. (With appropriate reweighting for the availability bias of history books - I should remember being a thousand peasants for every ruler.) I should immerse myself in history, imagine living through eras I only saw as ink on paper</p>
</blockquote>
<blockquote>
<p>I had to overcome the <em>false amnesia</em> of being born at a particular time. I had to recall - make available - all the memories, not just the memories which, by <em>mere coincidence</em>, belonged to myself and my own era.</p>
</blockquote>
<blockquote>
<p>The Earth became older, of a sudden.</p>
</blockquote>
<blockquote>
<p>So many mistakes, made over and over and over again, because I did not remember making them, in every era I never lived…</p>
</blockquote>
<blockquote>
<p>Don’t you remember how many times your biases have killed you? You don’t? I’ve noticed that sudden amnesia often follows a fatal mistake. But take it from me, it happened. I remember; I wasn’t there.</p>
</blockquote>
<blockquote>
<p>So the next time you doubt the strangeness of the future, remember how you were born in a hunter-gatherer tribe ten thousand years ago, when no one knew of Science at all. Remember how you were <em>shocked</em>, to the depths of your being, when Science explained the <em>great and terrible sacred mysteries</em> that you once revered so highly. Remember how you once believed that you could fly by eating the right mushrooms, and then you accepted with disappointment that you would never fly, and then you flew. Remember how you had always thought that slavery was right and proper, and then you changed your mind. Don’t imagine how you could have predicted the change, for that is amnesia. Remember that, in fact, <em>you did not guess</em>. Remember how, century after century, the world changed in ways you did not guess.</p>
</blockquote>
<blockquote>
<p>Maybe then you will be less shocked by what happens next.</p>
</blockquote>
<p>So much chaotic inversion over the millenia. Live through it all.</p>
<h1 id="science-as-curiosity-stopper">“Science” as Curiosity-Stopper</h1>
<blockquote>
<p>Is there a spell that stops curiosity?</p>
</blockquote>
<blockquote>
<p>Yes indeed! Whenever anyone asks “How did you do that?”, I just say “Science!”</p>
</blockquote>
<blockquote>
<p>You don’t actually know anything more than you knew before I said the magic word. But you turn away, satisfied that nothing unusual is going on.</p>
</blockquote>
<blockquote>
<p>But what does the phrase “scientifically explicable” mean? It means that someone else knows how the light bulb works. When you are told the light bulb is “scientifically explicable”, you don’t know more than you knew earlier; you don’t know whether the light bulb will brighten or fade. But because someone else knows, it devalues the knowledge in your eyes. You become less curious.</p>
</blockquote>
<blockquote>
<p>Why should your curiosity be diminished because someone else, not you, knows how the light bulb works? Is this not spite? It’s not enough for you to know; other people must also be ignorant, or you won’t be happy?</p>
</blockquote>
<hr />
<blockquote>
<p>What I am trying to do - to fulfill HA’s request of coming out and saying everything bluntly - is reawaken the <em>delight</em> in a <em>world full of mysteries</em>, which has been sapped by the notion that they are already understood, and therefore, no longer important. It’s not a verbal belief, but a <em>way of seeing the world</em>, which I am trying to bring into clear focus with parables. If I just said, “Hey, I saw a guy pass a light switch the other day, and he didn’t look at it curiously,” this would be true real-world example but it would not make the point.</p>
</blockquote>
<blockquote>
<p>There is a <em>tremendous demand</em> for mysteries which are frankly stupid. I wish this demand could be satisfied by scientific mysteries instead. But before we can live in that world, we have to undo the idea that what is scientific is not curiosity-material, that it is already marked as “understood”.</p>
</blockquote>
<h1 id="applause-lights">Applause Lights</h1>
<blockquote>
<p>Most applause lights are much more blatant, and can be detected by a simple reversal test. For example, suppose someone says:</p>
</blockquote>
<blockquote>
<p>We need to balance the risks and opportunities of AI.</p>
</blockquote>
<blockquote>
<p>If you reverse this statement, you get:</p>
</blockquote>
<blockquote>
<p>We shouldn’t balance the risks and opportunities of AI.</p>
</blockquote>
<h1 id="truly-part-of-you">Truly part of you</h1>
<blockquote>
<p>If you substituted <em>randomized symbols</em> for all the suggestive English names, you would be completely unable to figure out what G1071(G1072, 1073) meant. Was the AI program meant to represent hamburgers? Apples? Happiness? Who knows? If you delete the suggestive English names, they don’t grow back.</p>
</blockquote>
<blockquote>
<p>How can you realize that you shouldn’t trust your seeming knowledge that “light is waves”? One test you could apply is asking, “Could I regenerate this knowledge if it were somehow deleted from my mind?”</p>
</blockquote>
<p>Instead of just acquiring hypotheses from others, make sure you have sufficient empirical evidence to build the hypothesis yourself.</p>
<blockquote>
<p>How much of your knowledge could you regenerate? From how deep a deletion? It’s not just a test to cast out insufficiently connected beliefs. It’s a way of absorbing a fountain of knowledge, not just one fact.</p>
</blockquote>
<blockquote>
<p>When you contain the <em>source</em> of a thought, that thought can <em>change</em> along with you as you acquire new knowledge and new skills. When you contain the source of a thought, it becomes truly a part of you and grows along with you.</p>
</blockquote>
<blockquote>
<p>Strive to make yourself the source of every thought worth thinking. If the thought originally came from outside, make sure it comes from inside as well. Continually ask yourself: “How would I regenerate the thought if it were deleted?” When you have an answer, imagine that knowledge being deleted as well. And when you find a fountain, see what else it can pour.</p>
</blockquote>
<h1 id="chaotic-inversion">Chaotic Inversion</h1>
<p>When you don’t know the answer, it looks chaotic and complex. When you do, it inverts and becomes simplicity itself.</p>

<div class="info">Created: June 11, 2015</div>
<div class="info">Last modified: August 21, 2015</div>
<div class="info">Status: finished</div>
<div class="info"><b>Tags</b>: Anticipation-Constraints</div>

<br />
<div id="disqus_thread"></div>
<script type="text/javascript">
    /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
    var disqus_shortname = 'spkrationalitytrainingground'; // required: replace example with your forum shortname
    var disqus_identifier = '/Mysterious-Answers-to-Mysterious-Questions-Learning.html';
    var disqus_title = 'Mysterious Answers to Mysterious Questions';

    /* * * DON'T EDIT BELOW THIS LINE * * */
    (function() {
	var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
	dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
	(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
    })();
</script>
<script type="text/javascript" src="https://fast.fonts.net/jsapi/f7f47a40-b25b-44ee-9f9c-cfdfc8bb2741.js"></script>
<noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
<a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>

        </div>
        <div id="footer">
            Site proudly generated by
            <a href="http://jaspervdj.be/hakyll">Hakyll</a>
        </div>
    </body>
</html>