forked from byuistats/Statistics-Notebook
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathLinearRegression.html
More file actions
2804 lines (2597 loc) · 232 KB
/
Copy pathLinearRegression.html
File metadata and controls
2804 lines (2597 loc) · 232 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="utf-8" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="pandoc" />
<title>Linear Regression</title>
<script src="site_libs/jquery-1.11.3/jquery.min.js"></script>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<link href="site_libs/bootstrap-3.3.5/css/cerulean.min.css" rel="stylesheet" />
<script src="site_libs/bootstrap-3.3.5/js/bootstrap.min.js"></script>
<script src="site_libs/bootstrap-3.3.5/shim/html5shiv.min.js"></script>
<script src="site_libs/bootstrap-3.3.5/shim/respond.min.js"></script>
<script src="site_libs/navigation-1.1/tabsets.js"></script>
<script src="site_libs/navigation-1.1/codefolding.js"></script>
<link href="site_libs/highlightjs-9.12.0/default.css" rel="stylesheet" />
<script src="site_libs/highlightjs-9.12.0/highlight.js"></script>
<script src="site_libs/htmlwidgets-1.2/htmlwidgets.js"></script>
<script src="site_libs/plotly-binding-4.8.0/plotly.js"></script>
<script src="site_libs/typedarray-0.1/typedarray.min.js"></script>
<link href="site_libs/crosstalk-1.0.0/css/crosstalk.css" rel="stylesheet" />
<script src="site_libs/crosstalk-1.0.0/js/crosstalk.min.js"></script>
<link href="site_libs/plotly-htmlwidgets-css-1.39.2/plotly-htmlwidgets.css" rel="stylesheet" />
<script src="site_libs/plotly-main-1.39.2/plotly-latest.min.js"></script>
<style type="text/css">code{white-space: pre;}</style>
<style type="text/css">
pre:not([class]) {
background-color: white;
}
</style>
<script type="text/javascript">
if (window.hljs) {
hljs.configure({languages: []});
hljs.initHighlightingOnLoad();
if (document.readyState && document.readyState === "complete") {
window.setTimeout(function() { hljs.initHighlighting(); }, 0);
}
}
</script>
<style type="text/css">
h1 {
font-size: 34px;
}
h1.title {
font-size: 38px;
}
h2 {
font-size: 30px;
}
h3 {
font-size: 24px;
}
h4 {
font-size: 18px;
}
h5 {
font-size: 16px;
}
h6 {
font-size: 12px;
}
.table th:not([align]) {
text-align: left;
}
</style>
<link rel="stylesheet" href="styles.css" type="text/css" />
</head>
<body>
<style type = "text/css">
.main-container {
max-width: 940px;
margin-left: auto;
margin-right: auto;
}
code {
color: inherit;
background-color: rgba(0, 0, 0, 0.04);
}
img {
max-width:100%;
height: auto;
}
.tabbed-pane {
padding-top: 12px;
}
.html-widget {
margin-bottom: 20px;
}
button.code-folding-btn:focus {
outline: none;
}
</style>
<style type="text/css">
/* padding for bootstrap navbar */
body {
padding-top: 51px;
padding-bottom: 40px;
}
/* offset scroll position for anchor links (for fixed navbar) */
.section h1 {
padding-top: 56px;
margin-top: -56px;
}
.section h2 {
padding-top: 56px;
margin-top: -56px;
}
.section h3 {
padding-top: 56px;
margin-top: -56px;
}
.section h4 {
padding-top: 56px;
margin-top: -56px;
}
.section h5 {
padding-top: 56px;
margin-top: -56px;
}
.section h6 {
padding-top: 56px;
margin-top: -56px;
}
</style>
<script>
// manage active state of menu based on current page
$(document).ready(function () {
// active menu anchor
href = window.location.pathname
href = href.substr(href.lastIndexOf('/') + 1)
if (href === "")
href = "index.html";
var menuAnchor = $('a[href="' + href + '"]');
// mark it active
menuAnchor.parent().addClass('active');
// if it's got a parent navbar menu mark it active as well
menuAnchor.closest('li.dropdown').addClass('active');
});
</script>
<div class="container-fluid main-container">
<!-- tabsets -->
<script>
$(document).ready(function () {
window.buildTabsets("TOC");
});
</script>
<!-- code folding -->
<style type="text/css">
.code-folding-btn { margin-bottom: 4px; }
</style>
<script>
$(document).ready(function () {
window.initializeCodeFolding("hide" === "show");
});
</script>
<div class="navbar navbar-default navbar-fixed-top" role="navigation">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="index.html">Statistics Notebook</a>
</div>
<div id="navbar" class="navbar-collapse collapse">
<ul class="nav navbar-nav">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
Describing Data
<span class="caret"></span>
</a>
<ul class="dropdown-menu" role="menu">
<li>
<a href="GraphicalSummaries.html">Graphical Summaries</a>
</li>
<li>
<a href="NumericalSummaries.html">Numerical Summaries</a>
</li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
R Help
<span class="caret"></span>
</a>
<ul class="dropdown-menu" role="menu">
<li>
<a href="RCommands.html">R Commands</a>
</li>
<li>
<a href="RMarkdownHints.html">R Markdown Hints</a>
</li>
<li>
<a href="RCheatSheetsAndNotes.html">R Cheatsheets & Notes</a>
</li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
Making Inference
<span class="caret"></span>
</a>
<ul class="dropdown-menu" role="menu">
<li>
<a href="MakingInference.html">Making Inference</a>
</li>
<li>
<a href="tTests.html">t Tests</a>
</li>
<li>
<a href="WilcoxonTests.html">Wilcoxon Tests</a>
</li>
<li>
<a href="ANOVA.html">ANOVA</a>
</li>
<li>
<a href="Kruskal.html">Kruskal-Wallis</a>
</li>
<li>
<a href="LinearRegression.html">Linear Regression</a>
</li>
<li>
<a href="LogisticRegression.html">Logistic Regression</a>
</li>
<li>
<a href="ChiSquaredTests.html">Chi Squared Tests</a>
</li>
<li>
<a href="PermutationTests.html">Randomization Testing</a>
</li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
Analyses
<span class="caret"></span>
</a>
<ul class="dropdown-menu" role="menu">
<li>
<a href="./Analyses/AnalysisRubric.html">Analysis Rubric</a>
</li>
<li>
<a href="./Analyses/StudentHousing.html">Good Example Analysis</a>
</li>
<li>
<a href="./Analyses/StudentHousingPOOR.html">Poor Example Analysis</a>
</li>
<li>
<a href="./Analyses/Stephanie.html">Stephanie Analysis</a>
</li>
<li>
<a href="./Analyses/Olympics.html">Olympics Analysis</a>
</li>
</ul>
</li>
</ul>
<ul class="nav navbar-nav navbar-right">
</ul>
</div><!--/.nav-collapse -->
</div><!--/.container -->
</div><!--/.navbar -->
<div class="fluid-row" id="header">
<div class="btn-group pull-right">
<button type="button" class="btn btn-default btn-xs dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"><span>Code</span> <span class="caret"></span></button>
<ul class="dropdown-menu" style="min-width: 50px;">
<li><a id="rmd-show-all-code" href="#">Show All Code</a></li>
<li><a id="rmd-hide-all-code" href="#">Hide All Code</a></li>
</ul>
</div>
<h1 class="title toc-ignore">Linear Regression</h1>
</div>
<script type="text/javascript">
function showhide(id) {
var e = document.getElementById(id);
e.style.display = (e.style.display == 'block') ? 'none' : 'block';
}
function openTab(evt, tabName) {
var i, tabcontent, tablinks;
tabcontent = document.getElementsByClassName("tabcontent");
for (i = 0; i < tabcontent.length; i++) {
tabcontent[i].style.display = "none";
}
tablinks = document.getElementsByClassName("tablinks");
for (i = 0; i < tablinks.length; i++) {
tablinks[i].className = tablinks[i].className.replace(" active", "");
}
document.getElementById(tabName).style.display = "block";
evt.currentTarget.className += " active";
}
</script>
<hr />
<p>Determine which explanatory variables have a significant effect on the mean of the quantitative response variable.</p>
<hr />
<div id="simple-linear-regression" class="section level2 tabset tabset-fade tabset-pills">
<h2>Simple Linear Regression</h2>
<div style="float:left;width:125px;" align="center">
<p><img src="Images/QuantYQuantX.png" width=58px;></p>
</div>
<p>Simple linear regression is a good analysis technique when the data consists of a single quantitative response variable <span class="math inline">\(Y\)</span> and a single quantitative explanatory variable <span class="math inline">\(X\)</span>.</p>
<div id="overview" class="section level3 tabset">
<h3>Overview</h3>
<div style="padding-left:125px;">
<p><strong>Mathematical Model</strong></p>
<p>The true regression model assumed by a regression analysis is given by <span class="math display">\[
Y_i = \underbrace{\overbrace{\beta_0}^\text{y-intercept} + \overbrace{\beta_1}^\text{slope} X_i \ }_\text{true regression relation} + \overbrace{\epsilon_i}^\text{error term} \quad \text{where} \ \overbrace{\epsilon_i \sim N(0, \sigma^2)}^\text{error term normally distributed}
\]</span></p>
<p>The estimated regression line obtained from a regression analysis, pronounced “y-hat”, is written as</p>
<p><span class="math display">\[
\hat{Y}_i = \underbrace{\overbrace{\ b_0 \ }^\text{y-intercept} + \overbrace{b_1}^\text{slope} X_i \ }_\text{estimated regression relation}
\]</span></p>
<div style="font-size:0.8em;">
<p>Note: see the <strong>Explanation</strong> tab for details about these equations.</p>
</div>
<p><strong>Hypotheses</strong></p>
<p><span class="math display">\[
\left.\begin{array}{ll}
H_0: \beta_1 = 0 \\
H_a: \beta_1 \neq 0
\end{array}
\right\} \ \text{Slope Hypotheses}^{\quad \text{(most common)}}\quad\quad
\]</span></p>
<p><span class="math display">\[
\left.\begin{array}{ll}
H_0: \beta_0 = 0 \\
H_a: \beta_0 \neq 0
\end{array}
\right\} \ \text{Intercept Hypotheses}^{\quad\text{(sometimes useful)}}
\]</span></p>
<p>If <span class="math inline">\(\beta_1 = 0\)</span>, then the model reduces to <span class="math inline">\(Y_i = \beta_0 + \epsilon_i\)</span>, which is a flat line. This means <span class="math inline">\(X\)</span> does not improve our understanding of the mean of <span class="math inline">\(Y\)</span> if the null hypothesis is true.</p>
<p>If <span class="math inline">\(\beta_0 = 0\)</span>, then the model reduces to <span class="math inline">\(Y_i = \beta_1 X + \epsilon_i\)</span>, a line going through the origin. This means the average <span class="math inline">\(Y\)</span>-value is <span class="math inline">\(0\)</span> when <span class="math inline">\(X=0\)</span> if the null hypothesis is true.</p>
<p><strong>Assumptions</strong></p>
<p>This regression model is appropriate for the data when five assumptions can be made.</p>
<ol style="list-style-type: decimal">
<li><p><strong>Linear Relation</strong>: the true regression relation between <span class="math inline">\(Y\)</span> and <span class="math inline">\(X\)</span> is linear.</p></li>
<li><p><strong>Normal Errors</strong>: the error terms <span class="math inline">\(\epsilon_i\)</span> are normally distributed with a mean of zero.</p></li>
<li><p><strong>Constant Variance</strong>: the variance <span class="math inline">\(\sigma^2\)</span> of the error terms is constant (the same) over all <span class="math inline">\(X_i\)</span> values.</p></li>
<li><p><strong>Fixed X</strong>: the <span class="math inline">\(X_i\)</span> values can be considered fixed and measured without error.</p></li>
<li><p><strong>Independent Errors</strong>: the error terms <span class="math inline">\(\epsilon_i\)</span> are independent.</p></li>
</ol>
<div style="font-size:0.8em;">
<p>Note: see the <strong>Explanation</strong> tab for details about checking the regression assumptions.</p>
</div>
<p><strong>Interpretation</strong></p>
<p>The slope is interpreted as, “the change in the average y-value for a one unit change in the x-value.” It <strong>is not</strong> the average change in y. <strong>It is</strong> the change in the average y-value.</p>
<p>The y-intercept is interpreted as, “the average y-value when x is zero.” It is often not meaningful, but is sometimes useful. It just depends if x being zero is meaningful or not. For example, knowing the average price of a car with zero miles is useful. However, pretending to know the average height of adult males that weigh zero pounds, is not useful.</p>
<hr />
</div>
</div>
<div id="r-instructions" class="section level3">
<h3>R Instructions</h3>
<div style="padding-left:125px;">
<p><strong>Console</strong> Help Command: <code>?lm()</code></p>
<p><strong>Perform the Regression</strong></p>
<p><code>mylm <- lm(Y ~ X, data=YourDataSet)</code></p>
<p><code>summary(mylm)</code></p>
<ul>
<li><code>mylm</code> is some name you come up with to store the results of the <code>lm()</code> test. Note that <code>lm()</code> stands for “linear model.”</li>
<li><code>Y</code> must be a “numeric” vector of the quantitative response variable.</li>
<li><code>X</code> is the explanatory variable. It can either be quantitative (most usual) or qualitative.</li>
<li><code>YourDataSet</code> is the name of your data set.</li>
</ul>
<p><strong>Check Assumptions 1, 2, 3, and 5</strong></p>
<p><code>par(mfrow=c(1,3))</code></p>
<p><code>plot(mylm, which=1:2)</code></p>
<p><code>plot(mylm$residuals)</code></p>
<p><strong>Plotting the Regression Line</strong></p>
<p>To add the regression line to a scatterplot use the <code>abline(...)</code> command:</p>
<p><code>plot(Y ~ X, data=YourDataSet)</code></p>
<p><code>abline(mylm)</code></p>
<p>You can customize the look of the regression line with</p>
<p><code>abline(mylm, lty=1, lwd=1, col="someColor")</code></p>
<p>where <code>lty</code> stands for line-type with options (0=blank, 1=solid (default), 2=dashed, 3=dotted, 4=dotdash, 5=longdash, 6=twodash) and <code>lwd</code> stands for line-width.</p>
<p><strong>Accessing Parts of the Regression</strong></p>
<p>Finally, note that the <code>mylm</code> object contains the <code>names(mylm)</code> of</p>
<ul>
<li><code>mylm$coefficients</code> Contains two values. The first is the estimated <span class="math inline">\(y\)</span>-intercept. The second is the estimated slope.</li>
<li><code>mylm$residuals</code> Contains the residuals from the regression in the same order as the actual dataset.</li>
<li><code>mylm$fitted.values</code> The values of <span class="math inline">\(\hat{Y}\)</span> in the same order as the original dataset.</li>
<li><code>mylm$...</code> several other things that will not be explained here.</li>
</ul>
<hr />
</div>
</div>
<div id="explanation" class="section level3">
<h3>Explanation</h3>
<div style="padding-left:125px;">
<p>Linear regression has a rich mathematical theory behind it. This is because it uses a mathematical function and a random error term to describe the regression relation between a response variable <span class="math inline">\(Y\)</span> and an explanatory variable called <span class="math inline">\(X\)</span>.</p>
<div style="padding-left:30px;color:darkgray;">
<p>Expand each element below to learn more.</p>
</div>
<p><span style="color:steelblue;font-size:.8em;padding-left:160px;">Regression Cheat Sheet</span> <a href="javascript:showhide('regressioncheatsheet')" style="font-size:.6em;color:skyblue;">(Expand)</a></p>
<div id="regressioncheatsheet" style="display:none;font-size:.7em;">
<table style="width:71%;">
<colgroup>
<col width="9%" />
<col width="23%" />
<col width="11%" />
<col width="12%" />
<col width="13%" />
</colgroup>
<thead>
<tr class="header">
<th>Term</th>
<th>Pronunciation</th>
<th>Meaning</th>
<th>Math</th>
<th>R Code</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><span class="math inline">\(Y_i\)</span></td>
<td>“why-eye”</td>
<td>The data</td>
<td><span class="math inline">\(Y_i = \beta_0 + \beta_1 X_i + \epsilon_i \quad \text{where} \ \epsilon_i \sim N(0, \sigma^2)\)</span></td>
<td><code>YourDataSet$YourYvariable</code></td>
</tr>
<tr class="even">
<td><span class="math inline">\(\hat{Y}_i\)</span></td>
<td>“why-hat-eye”</td>
<td>The fitted line</td>
<td><span class="math inline">\(\hat{Y}_i = b_0 + b_1 X_i\)</span></td>
<td><code>lmObject$fitted.values</code></td>
</tr>
<tr class="odd">
<td><span class="math inline">\(E\{Y_i\}\)</span></td>
<td>“expected value of why-eye”</td>
<td>True mean y-value</td>
<td><span class="math inline">\(E\{Y_i\} = \beta_0 + \beta_1 X_i\)</span></td>
<td><code><none></code></td>
</tr>
<tr class="even">
<td><span class="math inline">\(\beta_0\)</span></td>
<td>“beta-zero”</td>
<td>True y-intercept</td>
<td><code><none></code></td>
<td><code><none></code></td>
</tr>
<tr class="odd">
<td><span class="math inline">\(\beta_1\)</span></td>
<td>“beta-one”</td>
<td>True slope</td>
<td><code><none></code></td>
<td><code><none></code></td>
</tr>
<tr class="even">
<td><span class="math inline">\(b_0\)</span></td>
<td>“b-zero”</td>
<td>Estimated y-intercept</td>
<td><span class="math inline">\(b_0 = \bar{Y} - b_1\bar{X}\)</span></td>
<td><code>b_0 <- mean(Y) - b_1*mean(X)</code></td>
</tr>
<tr class="odd">
<td><span class="math inline">\(b_1\)</span></td>
<td>“b-one”</td>
<td>Estimated slope</td>
<td><span class="math inline">\(b_1 = \frac{\sum X_i(Y_i - \bar{Y})}{\sum(X_i - \bar{X})^2}\)</span></td>
<td><code>b_1 <- sum( X*(Y - mean(Y)) ) / sum( (X - mean(X))^2 )</code></td>
</tr>
<tr class="even">
<td><span class="math inline">\(\epsilon_i\)</span></td>
<td>“epsilon-eye”</td>
<td>Distance of dot to true line</td>
<td><span class="math inline">\(\epsilon_i = Y_i - E\{Y_i\}\)</span></td>
<td><code><none></code></td>
</tr>
<tr class="odd">
<td><span class="math inline">\(r_i\)</span></td>
<td>“r-eye” or “residual-eye”</td>
<td>Distance of dot to estimated line</td>
<td><span class="math inline">\(r_i = Y_i - \hat{Y}_i\)</span></td>
<td><code>lmObject$residuals</code></td>
</tr>
<tr class="even">
<td><span class="math inline">\(\sigma^2\)</span></td>
<td>“sigma-squared”</td>
<td>Variance of the <span class="math inline">\(\epsilon_i\)</span></td>
<td><span class="math inline">\(Var\{\epsilon_i\} = \sigma^2\)</span></td>
<td><code><none></code></td>
</tr>
<tr class="odd">
<td><span class="math inline">\(MSE\)</span></td>
<td>“mean squared error”</td>
<td>Estimate of <span class="math inline">\(\sigma^2\)</span></td>
<td><span class="math inline">\(MSE = \frac{SSE}{n-p}\)</span></td>
<td><code>sum( lmObject$res^2 ) / (n - p)</code></td>
</tr>
<tr class="even">
<td><span class="math inline">\(SSE\)</span></td>
<td>“sum of squared error” (residuals)</td>
<td>Measure of dot’s total deviation from the line</td>
<td><span class="math inline">\(SSE = \sum_{i=1}^n (Y_i - \hat{Y}_i)^2\)</span></td>
<td><code>sum( lmObject$res^2 )</code></td>
</tr>
<tr class="odd">
<td><span class="math inline">\(SSR\)</span></td>
<td>“sum of squared regression error”</td>
<td>Measure of line’s deviation from y-bar</td>
<td><span class="math inline">\(SSR = \sum_{i=1}^n (\hat{Y}_i - \bar{Y})^2\)</span></td>
<td><code>sum( (lmObject$fit - mean(YourData$Y))^2 )</code></td>
</tr>
<tr class="even">
<td><span class="math inline">\(SSTO\)</span></td>
<td>“total sum of squares”</td>
<td>Measure of total variation in Y</td>
<td><span class="math inline">\(SSR + SSE = SSTO = \sum_{i=1}^n (Y_i - \bar{Y})^2\)</span></td>
<td><code>sum( (YourData$Y - mean(YourData$Y)^2 )</code></td>
</tr>
<tr class="odd">
<td><span class="math inline">\(R^2\)</span></td>
<td>“R-squared”</td>
<td>Proportion of variation in Y explained by the regression</td>
<td><span class="math inline">\(R^2 = \frac{SSR}{SSTO} = 1 - \frac{SSE}{SSTO}\)</span></td>
<td><code>SSR/SSTO</code></td>
</tr>
<tr class="even">
<td><span class="math inline">\(\hat{Y}_h\)</span></td>
<td>“why-hat-aitch”</td>
<td>Estimated mean y-value for some x-value called <span class="math inline">\(X_h\)</span></td>
<td><span class="math inline">\(\hat{Y}_h = b_0 + b_1 X_h\)</span></td>
<td><code>predict(lmObject, data.frame(XvarName=#))</code></td>
</tr>
<tr class="odd">
<td><span class="math inline">\(X_h\)</span></td>
<td>“ex-aitch”</td>
<td>Some x-value, not necessarily one of the <span class="math inline">\(X_i\)</span> values used in the regression</td>
<td>$X_h = $ some number</td>
<td><code>Xh = #</code></td>
</tr>
</tbody>
</table>
</div>
<p><br /></p>
<div id="the-mathematical-model-expand" class="section level4">
<h4>The Mathematical Model <a href="javascript:showhide('mathmodel1')" style="font-size:.6em;color:skyblue;">(Expand)</a></h4>
<p><span class="expand-caption"><span class="math inline">\(Y_i\)</span>, <span class="math inline">\(\hat{Y}_i\)</span>, and <span class="math inline">\(E\{Y_i\}\)</span>…</span></p>
<div id="mathmodel1" style="display:none;">
<p>There are three main elements to the mathematical model of regression. Each of these three elements is pictured below in the “Regression Relation Diagram.”</p>
<div style="padding-left:60px;color:darkgray;font-size:.8em;">
<p>Study both the three bullet points and their visual representations in the plot below for a clearer understanding.</p>
</div>
<ol style="list-style-type: decimal">
<li>The <strong>true line</strong>, i.e., the regression relation:</li>
</ol>
<div style="padding-left:60px;color:darkgray;">
<div style="color:steelblue;">
<p><span class="math inline">\(\underbrace{E\{Y\}}_{\substack{\text{true mean} \\ \text{y-value}}} = \underbrace{\overbrace{\beta_0}^\text{y-intercept} + \overbrace{\beta_1}^\text{slope} X}_\text{equation of a line}\)</span></p>
</div>
<p><a href="javascript:showhide('readmoretrueline')" style="font-size:.8em;color:skyblue;">(Read more…)</a></p>
<div id="readmoretrueline" style="display:none;">
<p>The true line is shown by the dotted line in the graph pictured below. This is typically unobservable. Think of it as “natural law” or “God’s law”. It is some true line that is unknown to us.</p>
<p>The regression relation <span class="math inline">\(E\{Y\} = \beta_0 + \beta_1 X\)</span> creates the line of regression where <span class="math inline">\(\beta_0\)</span> is the <span class="math inline">\(y\)</span>-intercept of the line and <span class="math inline">\(\beta_1\)</span> is the slope of the line. The regression relationship provides the average <span class="math inline">\(Y\)</span>-value, denoted <span class="math inline">\(E\{Y_i\}\)</span>, for a given <span class="math inline">\(X\)</span>-value, denoted by <span class="math inline">\(X_i\)</span>.</p>
<p>Note: <span class="math inline">\(E\{Y\}\)</span> is pronounced “the expected value of y” because, well… the mean is the typical, average, or “expected” value.</p>
</div>
</div>
<ol start="2" style="list-style-type: decimal">
<li>The <strong>dots</strong>, i.e., the regression relation plus an error term:</li>
</ol>
<div style="padding-left:60px;color:darkgray;">
<div style="color:steelblue;">
<p><span class="math inline">\(Y_i = \underbrace{\beta_0 + \beta_1 X_i}_{E\{Y_i\}} + \underbrace{\epsilon_i}_\text{error term} \quad \text{where} \ \epsilon_i\sim N(0,\sigma^2)\)</span></p>
</div>
<p><a href="javascript:showhide('readmoredots')" style="font-size:.8em;color:skyblue;">(Read more…)</a></p>
<div id="readmoredots" style="display:none;">
<p>This is shown by the dots in the graph below. This is the data. In regression, the assumtion is that the y-value for individual <span class="math inline">\(i\)</span>, denoted by <span class="math inline">\(Y_i\)</span>, was “created” by adding an error term <span class="math inline">\(\epsilon_i\)</span> to each individual’s “expected” value <span class="math inline">\(\beta_0 + \beta_1 X_i\)</span>. Note the “order of creation” would require first knowing an indivual’s x-value, <span class="math inline">\(X_i\)</span>, then their expected value from the regression relation <span class="math inline">\(E\{Y_i\} = \beta_0 + \beta_1 X_i\)</span> and then adding their <span class="math inline">\(\epsilon_i\)</span> value to the result. The <span class="math inline">\(\epsilon_i\)</span> allows each individual to deviate from the line. Some individuals deviate dramatically, some deviate only a little, but all dots vary some distance <span class="math inline">\(\epsilon_i\)</span> from the line.</p>
<p>Note: <span class="math inline">\(Y_i\)</span> is pronounced “why-eye” because it is the y-value for individual <span class="math inline">\(i\)</span>. Sometimes also called “why-sub-eye” because <span class="math inline">\(i\)</span> is in the subscript of <span class="math inline">\(Y\)</span>.</p>
</div>
</div>
<ol start="3" style="list-style-type: decimal">
<li>The <strong>estimated line</strong>, i.e., the line we get from a sample of data.</li>
</ol>
<div style="padding-left:60px;color:darkgray;">
<div style="color:steelblue;">
<p><span class="math inline">\(\underbrace{\hat{Y}_i}_{\substack{\text{estimated mean} \\ \text{y-value}}} = \underbrace{b_0 + b_1 X_i}_\text{estimated regression equation}\)</span></p>
</div>
<p><a href="javascript:showhide('readmoreestimatedline')" style="font-size:.8em;color:skyblue;">(Read more…)</a></p>
<div id="readmoreestimatedline" style="display:none;">
<p>The estimated line is shown by the solid line in the graph below. <span class="math inline">\(\hat{Y}\)</span> is the estimated regression equation obtained from the sample of data. It is the estimator of the true regression equation <span class="math inline">\(E\{Y\}\)</span>. So <span class="math inline">\(\hat{Y}\)</span> is interpreted as the estimated average (or mean) <span class="math inline">\(Y\)</span>-value for any given <span class="math inline">\(X\)</span>-value. Thus, <span class="math inline">\(b_0\)</span> is the estimated y-intercept and <span class="math inline">\(b_1\)</span> is the estimated slope. The b’s are sample statistics, like <span class="math inline">\(\bar{x}\)</span> and the <span class="math inline">\(\beta\)</span>’s are population parameters like <span class="math inline">\(\mu\)</span>. The <span class="math inline">\(b\)</span>’s estimate the <span class="math inline">\(\beta\)</span>’s.</p>
<p>Note: <span class="math inline">\(\hat{Y}_i\)</span> is pronounced “why-hat-eye” and is known as the “estimated y-value” or “fitted y-value” because it is the y-value you get from <span class="math inline">\(b_0 + b_1 X_i\)</span>. It is always different from <span class="math inline">\(Y_i\)</span> because dots are rarely if ever exactly on the estimated regression line.</p>
</div>
</div>
<p>This graphic depicts the true, but typically unknown, regression relation (dotted line). It also shows how a sample of data from the true regression relation (the dots) can be used to obtain an estimated regression equation (solid line) that is fairly close to the truth (dotted line).</p>
<p><img src="LinearRegression_files/figure-html/unnamed-chunk-2-1.png" width="672" /></p>
<p>Something to ponder: The true line, when coupled with the error terms, “creates” the data. The estimated (or fitted) line uses the sampled data to try to “re-create” the true line.</p>
<pre class="r"><code>## Simulating Data from a Regression Model
## This R-chunk is meant to be played in your R Console.
## It allows you to explore how the various elements
## of the regression model combine together to "create"
## data and then use the data to "re-create" the line.
set.seed(101) #Allows us to always get the same "random" sample
#Change to a new number to get a new sample
n <- 30 #set the sample size
X_i <- runif(n, 15, 45) #Gives n random values from a uniform distribution between 15 to 45.
beta0 <- 3 #Our choice for the y-intercept.
beta1 <- 1.8 #Our choice for the slope.
sigma <- 2.5 #Our choice for the std. deviation of the error terms.
epsilon_i <- rnorm(n, 0, sigma) #Gives n random values from a normal distribution with mean = 0, st. dev. = sigma.
Y_i <- beta0 + beta1*X_i + epsilon_i #Create Y using the normal error regression model
fabData <- data.frame(y=Y_i, x=X_i) #Store the data as data
View(fabData)
#In the real world, we begin with data (like fabData) and try to recover the model that (we assume) was used to created it.
fab.lm <- lm(y ~ x, data=fabData) #Fit an estimated regression model to the fabData.
summary(fab.lm) #Summarize your model.
plot(y ~ x, data=fabData) #Plot the data.
abline(fab.lm) #Add the estimated regression line to your plot.
# Now for something you can't do in real life... but since we created the data...
abline(beta0, beta1, lty=2) #Add the true regression line to your plot using a dashed line (lty=2).
legend("topleft", legend=c("True Line", "Estimated Line"), lty=c(2,1), bty="n") #Add a legend to your plot specifying which line is which.</code></pre>
</div>
<p><br /></p>
</div>
<div id="interpreting-the-model-parameters-expand" class="section level4">
<h4>Interpreting the Model Parameters <a href="javascript:showhide('interpretingparameters')" style="font-size:.6em;color:skyblue;">(Expand)</a></h4>
<p><span class="expand-caption"><span class="math inline">\(\beta_0\)</span> (intercept) and <span class="math inline">\(\beta_1\)</span> (slope), estimated by <span class="math inline">\(b_0\)</span> and <span class="math inline">\(b_1\)</span>, interpreted as…</span></p>
<div id="interpretingparameters" style="display:none;">
<p>The interpretation of <span class="math inline">\(\beta_0\)</span> is only meaningful if <span class="math inline">\(X=0\)</span> is in the scope of the model. If <span class="math inline">\(X=0\)</span> is in the scope of the model, then the intercept is interpreted as the average y-value, denoted <span class="math inline">\(E\{Y\}\)</span>, when <span class="math inline">\(X=0\)</span>.</p>
<p>The interpretation of <span class="math inline">\(\beta_1\)</span> is the amount of increase (or decrease) in the average y-value, denoted <span class="math inline">\(E\{Y\}\)</span>, per unit change in <span class="math inline">\(X\)</span>. It is often misunderstood to be the “average change in y” or just “the change in y” but it is more correctly referred to as the “change in the average y”.</p>
<p>To better see this, consider the three graphics shown below.</p>
<pre class="r"><code>par(mfrow=c(1,3))
hist(mtcars$mpg, main="Gas Mileage of mtcars Vehicles", ylab="Number of Vehicles", xlab="Gas Mileage (mpg)", col="skyblue")
boxplot(mpg ~ cyl, data=mtcars, border="skyblue", boxwex=0.5, main="Gas Mileage of mtcars Vehicles", ylab="Gas Mileage (mpg)", xlab="Number of Cylinders of Engine (cyl)")
plot(mpg ~ qsec, data=subset(mtcars, am==0), pch=16, col="skyblue", main="Gas Mileage of mtcars Vehicles", ylab="Gas Mileage (mpg)", xlab="Quarter Mile Time (qsec)")
abline(lm(mpg ~ qsec, data=subset(mtcars, am==0)), col="darkgray")
mtext(side=3, text="Automatic Transmissions Only (am==0)", cex=0.5)
abline(v = seq(16,22,2), h=seq(10,30,5), lty=3, col="gray")</code></pre>
<p><img src="LinearRegression_files/figure-html/unnamed-chunk-3-1.png" width="672" /></p>
<table style="width:88%;">
<colgroup>
<col width="27%" />
<col width="27%" />
<col width="31%" />
</colgroup>
<thead>
<tr class="header">
<th>The Histogram</th>
<th>The Boxplot</th>
<th>The Scatterplot</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>The <strong>histogram</strong> on the left shows gas mileages of vehicles from the mtcars data set. The average gas mileage is 20.09.</td>
<td>The <strong>boxplot</strong> in the middle shows that if we look at gas mileage for 4, 6, and 8 cylinder vehicles separately, we find the means to be 26.66, 19.74, and 15.1, respectively. If we wanted to, we could talk about the change in the means across cylinders, and would see that the mean is decreasing, first by <span class="math inline">\(26.66 - 19.74 = 6.92\)</span> mpg, then by <span class="math inline">\(19.74 - 15.1 = 4.64\)</span> mpg.</td>
<td>The <strong>scatterplot</strong> on the right shows that the average gas mileage (for just automatic transmission vehicles) increases by a slope of 1.44 for each 1 second increase in quarter mile time. In other words, the line gives the average y-value for any x-value. Thus, the slope of the line is the change in the average y-value.</td>
</tr>
</tbody>
</table>
</div>
<p><br /></p>
</div>
<div id="residuals-and-errors-expand" class="section level4">
<h4>Residuals and Errors <a href="javascript:showhide('residualsanderrors')" style="font-size:.6em;color:skyblue;">(Expand)</a></h4>
<p><span class="expand-caption"><span class="math inline">\(r_i\)</span>, the residual, estimates <span class="math inline">\(\epsilon_i\)</span>, the true error…</span></p>
<div id="residualsanderrors" style="display:none;">
<p>Residuals are the difference between the observed value of <span class="math inline">\(Y_i\)</span> (the point) and the predicted, or estimated value, for that point called <span class="math inline">\(\hat{Y_i}\)</span>. The errors are the true distances between the observed <span class="math inline">\(Y_i\)</span> and the actual regression relation for that point, <span class="math inline">\(E\{Y_i\}\)</span>.</p>
<p>We will denote a <strong>residual</strong> for individual <span class="math inline">\(i\)</span> by <span class="math inline">\(r_i\)</span>, <span class="math display">\[
r_i = \underbrace{Y_i}_{\substack{\text{Observed} \\ \text{Y-value}}} - \underbrace{\hat{Y}_i}_{\substack{\text{Predicted} \\ \text{Y-value}}} \quad \text{(residual)}
\]</span> The residual <span class="math inline">\(r_i\)</span> estimates the true <strong>error</strong> for individual <span class="math inline">\(i\)</span>, <span class="math inline">\(\epsilon_i\)</span>, <span class="math display">\[
\epsilon_i = \underbrace{Y_i}_{\substack{\text{Observed} \\ \text{Y-value}}} - \underbrace{E\{Y_i\}}_{\substack{\text{True Mean} \\ \text{Y-value}}} \quad \text{(error)}
\]</span></p>
<p>In summary…</p>
<div style="padding-left:30px;">
<table style="width:53%;">
<colgroup>
<col width="23%" />
<col width="29%" />
</colgroup>
<thead>
<tr class="header">
<th>Residual <span class="math inline">\(r_i\)</span></th>
<th>Error <span class="math inline">\(\epsilon_i\)</span></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Distance between the dot <span class="math inline">\(Y_i\)</span> and the estimated line <span class="math inline">\(\hat{Y}_i\)</span></td>
<td>Distance between the dot <span class="math inline">\(Y_i\)</span> and the true line <span class="math inline">\(E\{Y_i\}\)</span>.</td>
</tr>
<tr class="even">
<td><span class="math inline">\(r_i = Y_i - \hat{Y}_i\)</span></td>
<td><span class="math inline">\(\epsilon_i = Y_i - E\{Y_i\}\)</span></td>
</tr>
<tr class="odd">
<td>Known</td>
<td>Typically Unknown</td>
</tr>
</tbody>
</table>
</div>
<p>As shown in the graph below, the residuals are known values and they estimate the unknown (but true) error terms.</p>
<p><img src="LinearRegression_files/figure-html/unnamed-chunk-4-1.png" width="672" /></p>
<p>Keep in mind the idea that the errors <span class="math inline">\(\epsilon_i\)</span> “created” the data and that the residuals <span class="math inline">\(r_i\)</span> are computed after using the data to “re-create” the line.</p>
<p>Residuals have many uses in regression analysis. They allow us to</p>
<ol style="list-style-type: decimal">
<li>diagnose the regression assumptions,</li>
</ol>
<div style="padding-left:60px;color:darkgray;font-size:.8em;">
<p>See the “Assumptions” section below for more details.</p>
</div>
<ol start="2" style="list-style-type: decimal">
<li>estimate the regression relation,</li>
</ol>
<div style="padding-left:60px;color:darkgray;font-size:.8em;">
<p>See the “Estimating the Model Parameters” section below for more details.</p>
</div>
<ol start="3" style="list-style-type: decimal">
<li>estimate the variance of the error terms,</li>
</ol>
<div style="padding-left:60px;color:darkgray;font-size:.8em;">
<p>See the “Estimating the Model Variance” section below for more details.</p>
</div>
<ol start="4" style="list-style-type: decimal">
<li>and assess the fit of the regression relation.</li>
</ol>
<div style="padding-left:60px;color:darkgray;font-size:.8em;">
<p>See the “Assessing the Fit of a Regression” section below for more details.</p>
</div>
</div>
<p><br /></p>
</div>
<div id="residual-plots-regression-assumptions-expand" class="section level4">
<h4>Residual Plots & Regression Assumptions <a href="javascript:showhide('assumptions1')" style="font-size:.6em;color:skyblue;">(Expand)</a></h4>
<p><span class="expand-caption">Residuals vs. fitted-values, Q-Q Plot of the residuals, and residuals vs. order plots…</span></p>
<div id="assumptions1" style="display:none;">
<p>There are five assumptions that should be met for the mathematical model of simple linear regression to be appropriate.</p>
<div style="padding-left:60px;color:darkgray;font-size:.8em;">
<p>Each assumption is labeled in the regression equation below.</p>
</div>
<ol style="list-style-type: decimal">
<li>The regression relation between <span class="math inline">\(Y\)</span> and <span class="math inline">\(X\)</span> is linear.</li>
<li>The error terms are normally distributed with <span class="math inline">\(E\{\epsilon_i\}=0\)</span>.</li>
<li>The variance of the error terms is constant over all <span class="math inline">\(X\)</span> values.</li>
<li>The <span class="math inline">\(X\)</span> values can be considered fixed and measured without error.</li>
<li>The error terms are independent.</li>
</ol>
<p><span style="color:darkgray;">Regression Equation</span> <span class="math display">\[
Y_i = \underbrace{\beta_0 + \beta_1 \overbrace{X_i}^\text{#4}}_{\text{#1}} + \epsilon_i \quad \text{where} \ \overbrace{\epsilon_i \sim}^\text{#5} \overbrace{N(0}^\text{#2}, \overbrace{\sigma^2}^\text{#3})
\]</span></p>
<p>Residuals are used to diagnose departures from the regression assumptions.</p>
<p><a href="javascript:showhide('moreassumptionsdetail')" style="font-size:.8em;color:skyblue;">(Read more…)</a></p>
<div id="moreassumptionsdetail" style="display:none;">
<p>As shown above, the regression equation makes several claims, or assumptions, about the error terms <span class="math inline">\(\epsilon_i\)</span>, specifically 2, 3, and 5 of the regression assumptions are hidden inside the statement <span class="math inline">\(\epsilon_i \sim N(0, \sigma^2)\)</span> as shown here <span class="math display">\[
\epsilon_i \underbrace{\sim}_{\substack{\text{Independent} \\ \text{Errors}}} \overbrace{N}^{\substack{\text{Normally} \\ \text{distributed}}}(\underbrace{0}_{\substack{\text{mean of} \\ \text{zero}}}, \underbrace{\sigma^2}_{\substack{\text{Constant} \\ \text{Variance}}})
\]</span></p>
<p>While the actual error terms (<span class="math inline">\(\epsilon_i\)</span>) are unknown in real life, the residuals (<span class="math inline">\(r_i\)</span>) are known. Thus, we can use the residuals to check if the assumptions of the regression appear to be satisfied or not.</p>
</div>
<p><br /></p>
<div style="padding-left:15px;">
<h5 id="residuals-versus-fitted-values-plot-checks-assumptions-1-and-3">Residuals versus Fitted-values Plot: Checks Assumptions #1 and #3</h5>
<table width="90%">
<tr>
<td with="15%">
<p><img src="LinearRegression_files/figure-html/unnamed-chunk-5-1.png" width="144" /></p>
</td>
<td width="75%">
<p>The linear relationship and constant variance assumptions can be diagnosed using a residuals versus fitted-values plot. The fitted values are the <span class="math inline">\(\hat{Y}_i\)</span>. The residuals are the <span class="math inline">\(r_i\)</span>. This plot compares the residual to the magnitude of the fitted-value. No discernable pattern in this plot is desirable.</p>
<p>| <a href="javascript:showhide('residualsvsfittedvalues')" style="font-size:.8em;color:steelblue2;">Show Examples</a> |</p>
</td>
</tr>
</table>
<div id="residualsvsfittedvalues" style="display:none;">
<p><a href="javascript:showhide('residualsvsfittedvaluesread')" style="font-size:.8em;color:skyblue;">(Read more…)</a></p>
<div id="residualsvsfittedvaluesread" style="display:none;">
<p>The residuals versus fitted values plot checks for departures from the linear relation assumption and the constant variance assumption.</p>
<ul>
<li><p>The linear relation is assumed to be satisfied if there are no apparent trends in the plot.</p></li>
<li><p>The constant variance assumption is assumed to be satisfied if the vertical spread of the residuals remains roughly consistent across all fitted values.</p></li>
</ul>
<p>The left column of plots below show scenarios that would be considered not linear. The right column of plots show scenarios that would be considered linear, but lacking constant variance. The middle column of plots shows scenarios that would satisfy both assumptions, linear and constant variance.</p>
</div>
<pre class="r"><code>set.seed(2)
X <- rnorm(30,15,3)
notLin <- data.frame(X = X, Y = 500-X^2+rnorm(30,1,8))
notLin.lm <- lm(Y~X, data=notLin)
set.seed(15)
Lin <- data.frame(X=X, Y = 5+1.8*X+rnorm(30,2,1.3))
Lin.lm <- lm(Y~X, data=Lin)
par(mfrow=c(3,3), mai=c(.25,.25,.25,.25), mgp=c(1,.75,0))
plot(notLin.lm$fitted.values,notLin.lm$residuals, pch=20,
xlab="Fitted Values", ylab="Residuals",
main="Not Linear", cex.main=0.95,
xaxt='n', yaxt='n', col="firebrick")
mycurve <- lowess(notLin.lm$fitted.values,notLin.lm$residuals)
mycurveOrder <- order(mycurve$x)
mycurve$x <- mycurve$x[mycurveOrder]
mycurve$y <- mycurve$y[mycurveOrder]
polygon(c(mycurve$x,rev(mycurve$x)), c(mycurve$y+10, rev(mycurve$y-10)), col=rgb(.7,.7,.7,.2), border=NA)
abline(h=0)
plot(Lin.lm$fitted.values,Lin.lm$residuals, pch=20,
xlab="Fitted Values", ylab="Residuals",
main="Good: Linear, Constant Variance",
cex.main=0.95, xaxt='n', yaxt='n', col="skyblue")
abline(h=0)
set.seed(6)
notCon <- data.frame(X = X, Y = 5+1.8*X + rnorm(30,2,X^1.5))
notCon.lm <- lm(Y~X, data=notCon)
LinO <- data.frame(X=X, Y = 5+1.8*X+rnorm(30,2,1.3))
LinO[1] <- LinO[1]^2
LinO.lm <- lm(Y~X, data=LinO)
plot(notCon.lm$fitted.values,notCon.lm$residuals, pch=20, xlab="Fitted Values", ylab="Residuals", main="Unconstant Variance", cex.main=0.95, yaxt='n', xaxt='n', col="firebrick")
polygon(c(rep(min(notCon.lm$fit),2), rep(max(notCon.lm$fit), 2)), c(-30,30,1.2*max(notCon.lm$res),1.2*min(notCon.lm$res)), col=rgb(.7,.7,.7,.2), border=NA)
abline(h=0)
# plot(LinO.lm$fitted.values,LinO.lm$residuals, pch=20, xlab="Fitted Values", ylab="Residuals", main="Outliers", cex.main=0.95)
# abline(h=0)
tmp <- lm(height ~ age, data=Loblolly)
plot(tmp$residuals ~ tmp$fitted.values, pch=20,
xlab="Fitted Values", ylab="Residuals",
main="", cex.main=0.95,
xaxt='n', yaxt='n', col="firebrick")
mycurve <- lowess(tmp$fitted.values,tmp$residuals)
mycurveOrder <- order(mycurve$x)
mycurve$x <- mycurve$x[mycurveOrder]
mycurve$y <- mycurve$y[mycurveOrder]
polygon(c(mycurve$x,rev(mycurve$x)), c(mycurve$y+3, rev(mycurve$y-1)), col=rgb(.7,.7,.7,.2), border=NA)
abline(h=0)
tmp <- lm(Girth ~ Volume, data=trees[-31,])
plot(tmp$residuals ~ tmp$fitted.values, pch=20,
xlab="Fitted Values", ylab="Residuals",
main="", cex.main=0.95,
xaxt='n', yaxt='n', col="skyblue")
abline(h=0)
tmp <- lm(Height ~ Volume, data=trees)
plot(tmp$residuals ~ tmp$fitted.values, pch=20,
xlab="Fitted Values", ylab="Residuals",
main="", cex.main=0.95,
xaxt='n', yaxt='n', col="firebrick")
polygon(c(rep(min(tmp$fit), 2), max(tmp$fit)), c(1.3*max(tmp$res),1.2*min(tmp$res),0), col=rgb(.8,.8,.8,.2), border=NA)
abline(h=0)
tmp <- lm(mpg ~ disp, data=mtcars)
plot(tmp$residuals ~ tmp$fitted.values, pch=20,
xlab="Fitted Values", ylab="Residuals",
main="", cex.main=0.95,
xaxt='n', yaxt='n', col="firebrick")
mycurve <- lowess(tmp$fitted.values,tmp$residuals, f=.4)
mycurveOrder <- order(mycurve$x)
mycurve$x <- mycurve$x[mycurveOrder]
mycurve$y <- mycurve$y[mycurveOrder]
polygon(c(mycurve$x,rev(mycurve$x)), c(mycurve$y+3.5, rev(mycurve$y-2)), col=rgb(.7,.7,.7,.2), border=NA)
abline(h=0)
tmp <- lm(weight ~ repwt, data=Davis[-12,])
plot(tmp$residuals ~ tmp$fitted.values, pch=20,
xlab="Fitted Values", ylab="Residuals",
main="", cex.main=0.95,
xaxt='n', yaxt='n', col="skyblue")
abline(h=0)
tmp <- lm(weight ~ repht, data=Davis[-12,])
plot(tmp$residuals ~ tmp$fitted.values, pch=20,
xlab="Fitted Values", ylab="Residuals",
main="", cex.main=0.95,
xaxt='n', yaxt='n', col="firebrick")
polygon(c(min(tmp$fit),rep(max(tmp$fit), 2)), c(2,max(tmp$res),1.6*min(tmp$res)), col=rgb(.85,.85,.85,.2), border=NA)
abline(h=0) </code></pre>
<p><img src="LinearRegression_files/figure-html/unnamed-chunk-6-1.png" width="672" /></p>
</div>
<p><br /></p>
<h5 id="q-q-plot-of-the-residuals-checks-assumption-2">Q-Q Plot of the Residuals: Checks Assumption #2</h5>
<table width="90%">
<tr>
<td with="15%">
<img src="LinearRegression_files/figure-html/unnamed-chunk-7-1.png" width="144" />
</td>
<td width="75%">
<p>The normality of the error terms can be assessed by considering a normal probability plot (Q-Q Plot) of the residuals. If the residuals appear to be normal, then the error terms are also considered to be normal. If the residuals do not appear to be normal, then the error terms are also assumed to violate the normality assumption.</p>
<p>| <a href="javascript:showhide('qqplots')" style="font-size:.8em;color:steelblue2;">Show Examples</a> |</p>
</td>
</tr>
</table>
<div id="qqplots" style="display:none;">
<p><a href="javascript:showhide('qqplotsread')" style="font-size:.8em;color:skyblue;">(Read more…)</a></p>
<div id="qqplotsread" style="display:none;">
<p>There are four main trends that occur in a normal probability plot. Examples of each are plotted below with a histogram of the data next to the normal probability plot.</p>
<p>Often the plot is called a Q-Q Plot, which stands for quantile-quantile plot. The idea is to compare the observed distribution of data to what the distribution should look like in theory if it was normal. Q-Q Plots are more general than normal probability plots because they can be used with any theoretical distribution, not just the normal distribution.</p>
</div>
<pre class="r"><code>par(mfrow=c(2,2), mai=c(.5,.5,.25,.25), mgp=c(1,.75,0))
set.seed(123)
tmp <- rnorm(100)
qqnorm(tmp, pch=20, ylab="Observed", xaxt='n', yaxt='n', col="skyblue")
qqline(tmp)
hist(tmp, xlab="", xaxt='n', yaxt='n', main="Normal", col="skyblue")
tmp <- Davis$weight
qqnorm(tmp, pch=20, ylab="Observed", xaxt='n', yaxt='n', col="firebrick")
qqline(tmp)
hist(tmp, xlab="", xaxt='n', yaxt='n', main="Right-skewed",
breaks=15, col="firebrick")</code></pre>
<p><img src="LinearRegression_files/figure-html/unnamed-chunk-8-1.png" width="672" /></p>
<pre class="r"><code>par(mfrow=c(2,2), mai=c(.5,.5,.25,.25), mgp=c(1,.75,0))
tmp <- rbeta(100, 5,1)
qqnorm(tmp, pch=20, ylab="Observed", xaxt='n', yaxt='n', col="firebrick")
qqline(tmp)
hist(tmp, xlab="", xaxt='n', yaxt='n', main="Left-skewed",
breaks=seq(min(tmp),max(tmp), length.out=13), col="firebrick")
tmp <- rbeta(100,2,2)
qqnorm(tmp, pch=20, ylab="Observed", xaxt='n', yaxt='n', col="firebrick")
qqline(tmp)
hist(tmp, xlab="", xaxt='n', yaxt='n', main="Heavy-tailed", col="firebrick")</code></pre>
<p><img src="LinearRegression_files/figure-html/unnamed-chunk-8-2.png" width="672" /></p>
</div>
<p><br /></p>
<h5 id="residuals-versus-order-plot-checks-assumption-5">Residuals versus Order Plot: Checks Assumption #5</h5>
<table width="90%">
<tr>
<td with="15%">
<img src="LinearRegression_files/figure-html/unnamed-chunk-9-1.png" width="144" />
</td>
<td width="75%">
<p>When the data is collected in a specific order, or has some other important ordering to it, then the independence of the error terms can be assessed. This is typically done by plotting the residuals against their order of occurrance. If any dramatic trends are visible in the plot, then the independence assumption is violated.</p>
<p>| <a href="javascript:showhide('resorderplots')" style="font-size:.8em;color:steelblue2;">Show Examples</a> |</p>
</td>
</tr>
</table>
<div id="resorderplots" style="display:none;">
<p><a href="javascript:showhide('resorderplotsread')" style="font-size:.8em;color:skyblue;">(Read more…)</a></p>
<div id="resorderplotsread" style="display:none;">
<p>Plotting the residuals against the order in which the data was collected provides insight as to whether or not the observations can be considered independent. If the plot shows no trend, then the error terms are considered independent and the regression assumption satisfied. If there is a visible trend in the plot, then the regression assumption is likely violated.</p>
</div>
<pre class="r"><code>par(mfrow=c(2,2), mai=c(.5,.5,.25,.25), mgp=c(1,.75,0))