A lot can be understood about a complex system by breaking it down into its core components, exploring them individually and understanding how they interact. Just like in that, we can break down matrices and understand hidden patterns and properties which normally wouldn't be as apparent. This process is what we call decomposition. We perform decomposition (eigendecomposition & singular value decomposition) to find its core vectors and understand the key components (vectors) of this matrix.

Let's say you have a number 12. You know that the root components (in this case, prime factors) of 12 are 2x2x3. No matter in whatever way you transform the number 12, it's prime factors do not change. Similarly, if you want to break down a matrix into its root components, you can decompose it into a set of vectors, that do not change and are structurally core to the overall matrix. Normally, a vector would change its direction when transformed via a matrix multiplication, but these special vectors that we get from matrix decomposition do not change their direction. That's the reason why they are special. These set of vectors represent the "natural vibes" or the most dominant vectors in the matrix. The reason why we would like to find the most dominant vectors in a matrix, is because if we are able to find them, then we can just play around with those vectors instead of playing around with the entire matrix. This is specifically useful when the matrices we want to work with are quite large and sparse. The process of finding these "dominant" vectors in a matrix is called eigen-decomposition, and these vectors are called eigen vectors. Eigen-decomposition of a matrix actually yields us two things: a set of eigen vectors, and a vector that measures individual scores on each of these vectors as to how dominant they are for that matrix. This vector represents eigen values. Normally, eigen-decomposition only works for square matrices. For regular matrices, we can use singular value decomposition instead. The two methods vary slightly but the overall idea is the same.

Let's say you collect data to identify how each ingredient in the culinary world "partners up" or goes well with the others. The key is to be able to find key combinations that define several food bases and recipe roots. This is our matrix A:

The data in the above matrix simply tells how well each ingredient goes with the other. The value at first row, first column = 10, which says that flour goes well with itself. The values at third row, first column (3) and third row, second column (5) may suggest that sugar and eggs goes well together more than flour and eggs. Now take a moment of break and think. What will the dimensions of an individual eigen vector of this matrix A will be? It should be 3x1 as each of the row can simply denote the quantity of the above individual ingredient from matrix A, and a particular 3x1 vector will tell us a unique combination of the ingredients in quantitative proportion as stated by the values in the matrix. If the corresponding eigen value for that eigen vector is large, then that particular ingredient combination can be said to be dominant or most popular one.

The formula for eigen-decomposition of A is given by: A v = λ v where v represents an eigen-vector, denoting a unique quantitative mix of the above ingredients, and λ is an eigen-value that state the importance or “dominance” of the corresponding eigen vector. Intuitively, you can say that by performing the decomposition of a matrix you are just finding the important vectors that drives the direction of this matrix.

Now eigen-decomposition is great for square matrices where the number of rows and columns are equal. But with most of the matrices that you regularly work with may contain real world data where their matrices are rectangular in shape (number of rows != number of columns). In those cases, the technique we use slightly differs but the overall idea is the same. With singular value decomposition, the process will give us three core components:

- A matrix U known as left singular vectors
- A matrix Σ known as singular values
- A matrix VT known as right singular vectors

So overall, the singular value decomposition of a matrix will give us three individual matrices: A matrix representing relation of rows with hidden special features, a matrix of importance of these hidden special features, and a matrix representing relation of columns with these hidden special features.

Practical applications

Let's say you have a very complex high dimensional data. You can simply compress the dimensionality of this data by performing the decomposition and then taking the most important vectors in descending order of eigen values. By selecting k vectors, we are acquiring a certain variance of data from the original one, and skipping the rest, so you get a good approximation of the original data.
Recommendation engines, where the co-occurrence matrix can give information on the most commonly related or paired movies or tv shows.