Chapter 5: Matrix Approaches to Simple Linear Regression

Our New ScheduleDate Topic Chapter11/9 Matrix algebra K 511/14 Multiple regression I K 611/16 Multiple regression II K 711/21 No Class11/23 No Class - No Lab11/28 Regression models for quantitative and qualitative predictors K 811/30 Building the regression model I K 912/5 Building the regression model II K 1012/7 Building the regression model III (Final Handed Out) K 11Lecture 17 Psychology 790

Matrix AlgebraLecture 17 Psychology 790

Matrix IntroductionOverviewMatrix Algebra➤ Introduction➤ Definitions➤ Matrices➤ Matrix Computing➤ MatricesAlgebraMore MatricesGeneral LinearModelOther MatrixProducts● Imagine you are interested in studying the culturaldifferences in student achievement in high school.● Your study gets IRB approval and you work hard to getparental approval.● You go into school and collect data using many differentinstruments.● You then input the data into your favorite stat package (or MSExcel).● How do you think the data is stored?Wrapping UpLecture 17 Psychology 790

Definitions● We begin this class with some general definitions (from dictionary.com):✦ Matrix:1. A rectangular array of numeric or algebraic quantities subject tomathematical operations.2. The substrate on or within which a fungus grows.✦ Algebra:1. A branch of mathematics in which symbols, usually letters of thealphabet, represent numbers or members of a specified set and areused to represent quantities and to express general relationships thathold for all members of the set.2. A set together with a pair of binary operations defined on the set.Usually, the set and the operations include an identity element, and theoperations are commutative or associative.Lecture 17 Psychology 790

MatricesOverviewMatrix Algebra➤ Introduction➤ Definitions➤ Matrices➤ Matrix Computing➤ MatricesAlgebraMore MatricesGeneral LinearModelOther MatrixProducts● Away from the definition, a matrix is simply a rectangular wayof storing data.● Matrices can have unlimited dimensions, however for ourpurposes, all matrices will be in two dimensions:✦ Rows✦ Columns● Matrices are symbolized by boldface font in text.[ ]4 7 5A =6 6 3Wrapping UpLecture 17 Psychology 790

MATLABOverviewMatrix Algebra➤ Introduction➤ Definitions➤ Matrices➤ Matrix Computing➤ MatricesAlgebraMore MatricesGeneral LinearModelOther MatrixProductsWrapping Up● To help demonstrate the topics we will discuss today, I will beshowing examples in MATLAB.● MATLAB (Matrix Laboratory) is a scientific computingpackage typically used in other technical fields.● For many statistical applications MATLAB is very useful.● MATLAB is available in Psychology and throughout thecampus.● SPSS and SAS both have matrix computing capabilities, but(in my opinion) neither are as efficient, as user friendly, or asflexible as MATLAB.✦ It is better to leave most of the statistical computing to thecomputer scientists.Lecture 17 Psychology 790

Matrix ElementsOverviewMatrix Algebra➤ Introduction➤ Definitions➤ Matrices➤ Matrix Computing➤ MatricesAlgebraMore MatricesGeneral LinearModelOther MatrixProductsWrapping Up● A matrix is composed of a set of elements, each denoted it’srow and column position within the matrix.● For a matrix A of size r × c, each element is denoted by:a ij✦ The first subscript is the index for the rows: i = 1, . . .,r.✦ The second subscript is the index for the columns:j = 1, . . .,c.⎡⎤a 11 a 12 . . . a 1ca 21 a 22 . . . a 1cA =⎢⎣...⎥. ⎦a r1 a r2 . . . a rcLecture 17 Psychology 790

TransposeOverviewMatrix Algebra➤ Introduction➤ Definitions➤ Matrices➤ Matrix Computing➤ MatricesAlgebraMore MatricesGeneral LinearModel● The transpose of a matrix is simply the switching of theindices for rows and columns.● An element a ij in the original matrix (in the i th row and j thcolumn) would be a ji in the transposed matrix (in the j th rowand the i th column).● If the original matrix was of size i × j the transposed matrixwould be of size j × i.[ ]4 7 5A =6 6 3Other MatrixProductsWrapping UpA ′ =⎡⎢⎣4 67 65 3⎤⎥⎦Lecture 17 Psychology 790

Types of MatricesOverviewMatrix Algebra➤ Introduction➤ Definitions➤ Matrices➤ Matrix Computing➤ MatricesAlgebraMore MatricesGeneral LinearModelOther MatrixProductsWrapping Up● Square Matrix: A matrix that as the same number of rowsand columns.✦ Correlation and covariance matrices are examples ofsquare matrices.● Diagonal Matrix: A diagonal matrix is a square matrix withnon-zero elements down the diagonal and zero values for theoff-diagonal elements.A =⎡⎢⎣2.759 0 00 1.643 00 0 0.879● Symmetric Matrix: A symmetric matrix is a square matrixwhere a ij = a ji for all elements in i and j.✦ Correlation/covariance and distance matrices areexamples of symmetric matrices.⎤⎥⎦Lecture 17 Psychology 790

VectorsOverviewMatrix Algebra➤ Introduction➤ Definitions➤ Matrices➤ Matrix Computing➤ MatricesAlgebraMore MatricesGeneral LinearModelOther MatrixProductsWrapping Up● A vector is a matrix where one dimension is equal to sizeone.✦ Column vector: A column vector is a matrix of size r × 1.✦ Row vector: A row vector is a matrix of size 1 × c.● Vectors allow for geometric representations of matrices.● The Pearson correlation coefficient is a function of the anglebetween vectors.● Much of the statistical theory given in this course (andANOVA-type courses) can be conceptualized by projectionsof vectors (think of the dependent variable Y as a columnvector).Lecture 17 Psychology 790

Scalars● A scalar is a matrix of size 1 × 1.OverviewMatrix Algebra➤ Introduction➤ Definitions➤ Matrices➤ Matrix Computing➤ MatricesAlgebra● Scalars can be thought of as any single value.● The difficult concept to get used to is seeing a number as amatrix:[ ]A = 2.759More MatricesGeneral LinearModelOther MatrixProductsWrapping UpLecture 17 Psychology 790

AlgebraLecture 17 Psychology 790

Algebraic OperationsOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular MatricesMore MatricesGeneral LinearModel● As mentioned in the definition at the beginning of class,algebra is simply a set of math that defines basic operations.✦ Identity✦ Zero✦ Addition✦ Subtraction✦ Multiplication✦ Division● Matrix algebra is simply the use of these operations withmatrices.Other MatrixProductsWrapping UpLecture 17 Psychology 790

Matrix AdditionOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular Matrices● Matrix addition is very much like scalar addition, the onlyconstraint is that the two matrices must be of the same size(same number of rows and columns).● The resulting matrix contains elements that are simply theresult of adding two scalars.More MatricesGeneral LinearModelOther MatrixProductsWrapping UpLecture 17 Psychology 790

Matrix AdditionOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular Matrices⎡A = ⎢⎣⎡B = ⎢⎣⎤a 11 a 12a 21 a 22⎥a 31 a 32 ⎦a 42a 41⎤b 11 b 12b 21 b 22⎥b 31 b 32 ⎦b 42b 41More MatricesGeneral LinearModelOther MatrixProductsWrapping UpA + B =⎡⎢⎣⎤a 11 + b 11 a 12 + b 12a 21 + b 21 a 22 + b 22⎥a 31 + b 31 a 32 + b 32 ⎦a 41 + b 41 a 42 + b 42Lecture 17 Psychology 790

Matrix SubtractionOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular Matrices● Matrix subtraction is identical to matrix addition, with theexception that all elements of the new matrix are thesubtracted elements of the previous matrices.● Again, the only constraint is that the two matrices must be ofthe same size (same number of rows and columns).● The resulting matrix contains elements that are simply theresult of subtracting two scalars.More MatricesGeneral LinearModelOther MatrixProductsWrapping UpLecture 17 Psychology 790

Matrix SubtractionOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular Matrices⎡A = ⎢⎣⎡B = ⎢⎣⎤a 11 a 12a 21 a 22⎥a 31 a 32 ⎦a 42a 41⎤b 11 b 12b 21 b 22⎥b 31 b 32 ⎦b 42b 41More MatricesGeneral LinearModelOther MatrixProductsWrapping UpA − B =⎡⎢⎣⎤a 11 − b 11 a 12 − b 12a 21 − b 21 a 22 − b 22⎥a 31 − b 31 a 32 − b 32 ⎦a 41 − b 41 a 42 − b 42Lecture 17 Psychology 790

Matrix MultiplicationOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular MatricesMore MatricesGeneral LinearModelOther MatrixProducts● Unlike matrix addition and subtraction, matrix multiplication ismuch more complicated.● Matrix multiplication results in a new matrix that can be ofdiffering size from either of the two original matrices.● Matrix multiplication is defined only for matrices where thenumber of columns of the first matrix is equal to the numberof rows of the second matrix.● The resulting matrix as the same number of rows as the firstmatrix, and the same number of columns as the secondmatrix.A B = C(r × c) (c × k) (r × k)Wrapping UpLecture 17 Psychology 790

OverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular MatricesMatrix MultiplicationAB =⎡⎢⎣A =⎡⎢⎣⎤a 11 a 12 [a 21 a 22⎥a 31 a 32 ⎦ B =a 42a 41]b 11 b 12 b 13b 21 b 22 b 23⎤a 11 b 11 + a 12 b 21 a 11 b 12 + a 12 b 22 a 11 b 13 + a 12 b 23a 21 b 11 + a 22 b 21 a 21 b 12 + a 22 b 22 a 21 b 13 + a 22 b 23⎥a 31 b 11 + a 32 b 21 a 31 b 12 + a 32 b 22 a 31 b 13 + a 32 b 23 ⎦a 41 b 11 + a 42 b 21 a 41 b 12 + a 42 b 22 a 41 b 13 + a 42 b 23More MatricesGeneral LinearModelOther MatrixProductsWrapping UpLecture 17 Psychology 790

OverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular MatricesMatrix MultiplicationAB =⎡⎢⎣A =⎡⎢⎣⎤a 11 a 12 [a 21 a 22⎥a 31 a 32 ⎦ B =a 42a 41]b 11 b 12 b 13b 21 b 22 b 23⎤a 11 b 11 + a 12 b 21 a 11 b 12 + a 12 b 22 a 11 b 13 + a 12 b 23a 21 b 11 + a 22 b 21 a 21 b 12 + a 22 b 22 a 21 b 13 + a 22 b 23⎥a 31 b 11 + a 32 b 21 a 31 b 12 + a 32 b 22 a 31 b 13 + a 32 b 23 ⎦a 41 b 11 + a 42 b 21 a 41 b 12 + a 42 b 22 a 41 b 13 + a 42 b 23More MatricesGeneral LinearModelOther MatrixProductsWrapping UpLecture 17 Psychology 790

OverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular MatricesMatrix MultiplicationAB =⎡⎢⎣A =⎡⎢⎣⎤a 11 a 12 [a 21 a 22⎥a 31 a 32 ⎦ B =a 42a 41]b 11 b 12 b 13b 21 b 22 b 23⎤a 11 b 11 + a 12 b 21 a 11 b 12 + a 12 b 22 a 11 b 13 + a 12 b 23a 21 b 11 + a 22 b 21 a 21 b 12 + a 22 b 22 a 21 b 13 + a 22 b 23⎥a 31 b 11 + a 32 b 21 a 31 b 12 + a 32 b 22 a 31 b 13 + a 32 b 23 ⎦a 41 b 11 + a 42 b 21 a 41 b 12 + a 42 b 22 a 41 b 13 + a 42 b 23More MatricesGeneral LinearModelOther MatrixProductsWrapping UpLecture 17 Psychology 790

Multiplication and SummationOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular MatricesMore MatricesGeneral LinearModelOther MatrixProducts● Because of the additive nature induced by matrixmultiplication, many statistical formulas that use:∑can be expressed by matrix notation.● For instance, consider a single variable X i , with i = 1, . . .,Nobservations.● Putting the set of observations into the column vector X, ofsize N × 1, we can show that:N∑X 2 = X ′ Xi=1Wrapping UpLecture 17 Psychology 790

Matrix Multiplication by Scalar● Recall that a scalar is simply a matrix of size (1 × 1).OverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular Matrices● Matrix multiplication by a scalar causes all elements of thematrix to be multiplied by the scalar.● The resulting matrix has all elements multiplied by the scalar.⎡ ⎤ ⎡⎤a 11 a 12s × a 11 s × a 12a 21 a 22A = ⎢ ⎥⎣ a 31 a 32 ⎦s × A = s × a 21 s × a 22⎢⎥⎣ s × a 31 s × a 32 ⎦a 41 a 42 s × a 41 s × a 42More MatricesGeneral LinearModelOther MatrixProductsWrapping UpLecture 17 Psychology 790

Identity MatrixOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular MatricesMore MatricesGeneral LinearModelOther MatrixProducts● The identity matrix is defined as a matrix that when multipliedwith another matrix produces that original matrix:A I = AI A = A● The identity matrix is simply a square matrix that has alloff-diagonal elements equal to zero, and all diagonalelements equal to one.I (3×3) =⎡⎢⎣1 0 00 1 00 0 1⎤⎥⎦Wrapping UpLecture 17 Psychology 790

Zero MatrixOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular MatricesMore MatricesGeneral LinearModel● The zero matrix is defined as a matrix that when multipliedwith another matrix produces the matrix:A 0 = 00 A = 0● The zero matrix is simply a square matrix that has allelements equal to zero.⎡ ⎤0 0 0⎢ ⎥0 (3×3) = ⎣ 0 0 0 ⎦0 0 0Other MatrixProductsWrapping UpLecture 17 Psychology 790

Unity MatrixOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular MatricesMore MatricesGeneral LinearModelOther MatrixProductsWrapping Up● The Unity matrix is defined as a matrix where all of theelements are 1.● A column vector with all elements 1 will be denoted by:⎡ ⎤111 (rx1) =⎢⎥⎣ . ⎦1● A square matrix where all the elements are 1 will be denotedby J:⎡ ⎤1 · · · 1J (rxr) = ⎢⎣.⎥. ⎦1 . . . 1Lecture 17 Psychology 790

Matrix “Division”: The InverseOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular Matrices● Recall from basic math that:● And that:ab = 1 b a = b−1 aaa = 1● Matrix inverses are just like division in basic math.More MatricesGeneral LinearModelOther MatrixProductsWrapping UpLecture 17 Psychology 790

The InverseOverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular MatricesMore MatricesGeneral LinearModel● For a square matrix, an inverse matrix is simply the matrixthat when pre-multiplied with another matrix produces theidentity matrix:A −1 A = I● Matrix inverse calculation is complicated and unnecessarysince computers are much more efficient at finding inversesof matrices.● One point of emphasis: just like in regular division, divisionby zero is undefined.● By analogy - not all matrices can be inverted.Other MatrixProductsWrapping UpLecture 17 Psychology 790

Singular Matrices● A matrix that cannot be inverted is called a singular matrix.OverviewMatrix AlgebraAlgebra➤ Algebra➤ Addition➤ Subtraction➤ Multiplication➤ Identity➤ Zero➤ Unity Matrix➤ “Division”➤ Singular Matrices● In statistics, common causes of singular matrices are foundby linear dependence among the rows or columns of asquare matrix.● Linear dependence can be cause by combinations ofvariables, or by variables with extreme correlations (eithernear 1.00 or -1.00).More MatricesGeneral LinearModelOther MatrixProductsWrapping UpLecture 17 Psychology 790

Linear DependenciesOverviewMatrix AlgebraAlgebraMore Matrices➤ LinearDependencies➤ Rank of a Matrix➤ Some BasicResults forMatricesGeneral LinearModelOther MatrixProductsWrapping Up● A set of vectors are said to be linearly dependent ifa 1 , a 2 , . . .,a k exist, and:✦ a 1 x 1 + a 2 x 2 + . . . + a k x k = 0.✦ a 1 , a 2 , . . .,a k are not all zero.● Such linear dependencies occur when a linear combinationis added to the vector set.● Matrices comprised of a set of linearly dependent vectors aresingular.● A set of linearly independent vectors forms what is called abasis for the vector space.● Any vector in the vector space can then be expressed as alinear combination of the basis vectors.Lecture 17 Psychology 790

Rank of a MatrixOverviewMatrix AlgebraAlgebraMore Matrices➤ LinearDependencies➤ Rank of a Matrix➤ Some BasicResults forMatrices● The rank of a matrix is defined to be the maximum number oflinearly independent columns.● The rank of a matrix is unique and can be equivalently bedefined as the maximum number of linearly dependent rows.● The rank of an r x c matrix cannot exceed min(r, c).● When a new matrix is formed by multiplying two matrices, itsrank cannot exceed the smaller of the two ranks for thematrices that were multiplied.General LinearModelOther MatrixProductsWrapping UpLecture 17 Psychology 790

Some Basic Results for MatricesOverviewMatrix AlgebraAlgebraMore Matrices➤ LinearDependencies➤ Rank of a Matrix➤ Some BasicResults forMatricesGeneral LinearModelOther MatrixProductsWrapping UpA + B = B + A(A + B) + C = A + (B + C)(AB)C = A(BC)C(A + B) = CA + CBk(A + B) = kA + kB(A′)′ = A(A + B)′ = A′ + B′(AB)′ = B′A′(ABC)′ = C′B′A′(AB) −1 = B −1 A −1(ABC) −1 = C −1 B −1 A −1(A −1 ) −1 = A(A′) −1 = (A −1 )′Lecture 17 Psychology 790

General Linear ModelLecture 17 Psychology 790

Regression with Matrices⎡⎢⎣⎤Y 1Y 2Y 3Y 4. ⎥. ⎦Y N=⎡⎢⎣⎤1 X 111 X 211 X 311 X 41.⎥. ⎦1 X N1[]b 0b 1+⎡⎢⎣⎤e 1e 2e 3e 4. ⎥. ⎦e NY = X b + e(N × 1) (N × (1 + k)) (k × 1) (N × 1)Lecture 17 Psychology 790

Regression with Matrices● Working the matrix multiplication and addition for a single case gives:⎡ ⎤ ⎡ ⎤⎡ ⎤Y 1 1 X 11e 1Y 21 X 21e [ ]2Y 31 X 31b 0e 3Y =41 X +41b ⎢ . ⎣⎥ ⎢. ⎦ ⎣1 e 4.⎥⎢ . . ⎦⎣⎥. ⎦Y N 1 X N1 e NY 1Lecture 17 Psychology 790

Regression with Matrices● Working the matrix multiplication and addition for a single case gives:⎡ ⎤ ⎡ ⎤⎡ ⎤Y 1 1 X 11e 1Y 21 X 21e [ ]2Y 31 X 31b 0e 3Y =41 X +41b ⎢ . ⎣⎥ ⎢. ⎦ ⎣1 e 4.⎥⎢ . . ⎦⎣⎥. ⎦Y N 1 X N1 e NY 1 = b 0Lecture 17 Psychology 790

Regression with Matrices● Working the matrix multiplication and addition for a single case gives:⎡ ⎤ ⎡ ⎤⎡ ⎤Y 1 1 X 11e 1Y 21 X 21e [ ]2Y 31 X 31b 0e 3Y =41 X +41b ⎢ . ⎣⎥ ⎢. ⎦ ⎣1 e 4.⎥⎢ . . ⎦⎣⎥. ⎦Y N 1 X N1 e NY 1 = b 0 + b 1 X 11Lecture 17 Psychology 790

Regression with Matrices● Working the matrix multiplication and addition for a single case gives:⎡ ⎤ ⎡ ⎤⎡ ⎤Y 1 1 X 11e 1Y 21 X 21e [ ]2Y 31 X 31b 0e 3Y =41 X +41b ⎢ . ⎣⎥ ⎢. ⎦ ⎣1 e 4.⎥⎢ . . ⎦⎣⎥. ⎦Y N 1 X N1 e NY 1 = b 0 + b 1 X 11 + e 1Lecture 17 Psychology 790

Regression with MatricesOverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModel➤ Regression➤ Error Distribution➤ Estimation● Note that most everything is really straightforward in terms ofmatrix algebra.● The matrix of predictors, X, has the first column containingall ones.✦ This represents the intercept parameter a.✦ This is also an introduction to setting columns of the Xmatrix to represent design and or group controls (as inANOVA).Other MatrixProductsWrapping UpLecture 17 Psychology 790

Distribution of ErrorsOverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModel➤ Regression➤ Error Distribution➤ EstimationOther MatrixProductsWrapping Up● Recall from previous classes that we often placedistributional assumptions on our error terms, allowing for thedevelopment of tractable hypothesis tests.● With matrices, the distributional assumptions are no different,except for things are approached in a multivariate fashion:e ∼ N N (0, σ 2 eI N )● Having a multivariate normal distribution with uncorrelatedvariables (from I N ) is identical to saying:for all i observations.e i ∼ N(0, σ 2 e)Lecture 17 Psychology 790

Regression Estimation with MatricesOverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModel➤ Regression➤ Error Distribution➤ EstimationOther MatrixProductsWrapping Up● Regression estimates are typically found via least squares(called L 2 estimates).● In least squares regression, the estimates are found byminimizing:N∑e 2 =i=1N∑(Y i − Y i ′ ) 2 =i=1N∑(Y i − a + b 1 X i1 + . . . + b k X ik ) 2i=1● As you could guess, we could accomplish all of this viamatrices.● Equivalently:N∑e 2 =i=1N∑(Y i − x ′ ib) 2 = (Y − Xb) ′ (Y − Xb) = e ′ ei=1Lecture 17 Psychology 790

Regression Estimation with Matrices● Thankfully, there are people to figure out the equation for b that minimizese ′ e.b = (X ′ X) −1 X ′ Yb = ( X ′ X ) −1 X ′ Y((k + 1) × 1) ((k + 1) × N) (N × (k + 1)) ((k + 1) × N) (N × 1)● This equation is what I have been talking about for quite some time, theGeneral Linear Model.● For many types of data, in many differing analyses, this equation will provideestimates:✦ Multiple Regression✦ ANOVA✦ Analysis of Covariance (ANCOVA).✦ Multiple Regression with Curvilinear relationships in X.Lecture 17 Psychology 790

Other Matrix ProductsLecture 17 Psychology 790

Other Regression Terms with MatricesOverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b Variance● Clearly, getting estimates for b isn’t the only focus of thisclass.● Most everything accomplished previously can be obtain viamatrices:✦ Mean (Expectation) Vectors.✦ Fitted Values✦ Variance-Covariance Matrices.✦ Correlation Matrices.✦ Sums of Squares.Wrapping Lecture 17 UpPsychology 790

A Vector of OnesOverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b Variance● Helpful in many applications is a simple vector of ones:⎡ ⎤11 ⎥1 N =● Note that this vector is not the identity matrix (where onesare on the diagonal).● You can probably see the use for such a vector by thefollowing equation:N∑X i = x ′ 1i=1⎢⎣.1⎥⎦Wrapping Lecture 17 UpPsychology 790

The Mean VectorOverviewMatrix Algebra● To compute a vector of means for a set of variablesZ 1 , . . .,Z k :¯Z (k×1) = Z ′ 1 (k×1) (1 ′ (1×k)1 (k×1) ) −1AlgebraMore MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b Variance● Don’t be dismayed by (1 ′ (1×k)1 (k×1) ) −1 , this is simply:1NWrapping Lecture 17 UpPsychology 790

Fitted ValuesOverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModel● The Fitted Values for each Y i can also be computed usingmatrices.● We can obtain the Fitted Value matrix (vector really) by thefollowing:Ŷ = Y − XbOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b VarianceWrapping Lecture 17 UpPsychology 790

Residuals● The Residuals can also be computed using matrices.OverviewMatrix AlgebraAlgebraMore Matrices● We can obtain the residual matrix (vector really) by thefollowing:e = Y − ŶGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b VarianceWrapping Lecture 17 UpPsychology 790

Sums of Squares and Cross Products● The SSCP matrix contains:OverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b Variance✦ On the diagonal, Sums of Squares for each variable Z k :N∑(Z ik − ¯Z·k ) 2i=1✦ On the off-diagonal the Sum of Cross Products for eachpair of variables, Z k and Z k ′:N∑(Z ik − ¯Z·k )(Z ik ′ − ¯Z·k ′)i=1✦ The matrix expression for the SSCP matrix is:SSCP = (Z − ¯Z) ′ (Z − ¯Z)Wrapping Lecture 17 UpPsychology 790

Covariance MatricesOverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b Variance● Once the SSCP is obtained, the covariance matrix is easilyobtained:✦ On the diagonal, the variance for each variable Z k :∑ Ni=1 (Z ik − ¯Z·k ) 2✦ On the off-diagonal the covariance for each pair ofvariables, Z k and Z k ′:∑ Ni=1 (Z ik − ¯Z·k )(Z ik′ − ¯Z·k ′)✦ The matrix expression for the covariance matrix is:NNCOV = 1 N (Z − ¯Z) ′ (Z − ¯Z)Wrapping Lecture 17 UpPsychology 790

Correlation MatricesOverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b Variance● Once the covariance matrix is obtained, the correlationmatrix is easily obtained.● Let D be a diagonal matrix consisting of the standarddeviation of all variables (the square root of the diagonalelements of the covariance matrix).● The matrix expression for the correlation matrix is:CORR = D −1 COV D −1Wrapping Lecture 17 UpPsychology 790

Regression Sums of SquaresOverviewMatrix AlgebraAlgebra● The Sum of Squares for the Regression can be obtained bymatrix calculations:SSR = b ′ X ′ Y − (∑ Ni=1 Y )2N= b ′ X ′ Y − 1 N (Y′ 1) 2More MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b VarianceWrapping Lecture 17 UpPsychology 790

Sums of Squares ErrorOverviewMatrix AlgebraAlgebra● The Residual Sum of Squares is also obtained by matrixcalculations:SSE =N∑e 2 i = e ′ e = Y ′ Y − b ′ X ′ Yi=1More MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b VarianceWrapping Lecture 17 UpPsychology 790

Sums of Squares TotalOverviewMatrix Algebra● The Total Sum of Squares can be obtained by matrixcalculations:SSTO = Y ′ Y ′ − 1 N (Y′ JY)AlgebraMore MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b VarianceWrapping Lecture 17 UpPsychology 790

Squared CorrelationOverview● Recall that the Squared Multiple Correlation Coefficient, orR 2 , is found from:Matrix AlgebraAlgebraR 2 =SS reg∑ Ni=1 (Y i − Ȳ ) 2More MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b Variance● This is also obtainable by matrix calculations:R 2 = b′ X ′ Y − 1 N (Y′ 1) 2Y ′ Y − 1 N (Y′ 1) 2Wrapping Lecture 17 UpPsychology 790

Variance of Regression EstimatorsOverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModelOther MatrixProducts➤ More Matrices➤ The One➤ Means➤ Fitted Values➤ Residuals➤ SSCP➤ Covariance Matrix➤ CorrelationMatrices➤ Sums of Squares➤ SquaredCorrelation➤ b Variance● The covariance matrix of the regression parameters containsuseful information regarding the standard errors of theestimates (found along the diagonal).● To find the covariance matrix of the estimates, the ResidualSum of Squares is needed (dividing this term by the residualdegrees of freedom gives the error variance, or σ 2 e):var(b) = σ 2 e(X ′ X) −1 =SS resN − k − 1 (X′ X) −1 =1N − k − 1 e′ e(X ′ X) −1Wrapping Lecture 17 UpPsychology 790

Final ThoughtOverviewMatrix AlgebraAlgebraMore MatricesGeneral LinearModelOther MatrixProductsWrapping Up➤ Final Thought➤ Next Class● Matrices are prevalent inmany statisticalapplications, but do not fearthem.● You will encounter matrixnotation from time to time,but that should not because for concern.● Leave the heavy matrix lifting to the experts: let a stat ormath computer package do advanced matrix calculations.Lecture 17 Psychology 790

Next Time● Multiple Regression:OverviewMatrix AlgebraAlgebraMore Matrices✦ We finally move away from a model that has only oneindependent variable✦ We will just extend our current model to fit with more thanone IV or XGeneral LinearModelOther MatrixProductsWrapping Up➤ Final Thought➤ Next ClassLecture 17 Psychology 790