The Strassen’s method of matrix multiplication is a typical divide and conquer algorithm. We’ve seen so far some divide and conquer algorithms like merge sort and the Karatsuba’s fast multiplication of large numbers. However let’s get again on what’s behind the divide and conquer approach.

Unlike the dynamic programming where we “expand” the solutions of sub-problems in order to get the final solution, here we are talking more on joining sub-solutions together. These solutions of some sub-problems of the general problem are equal and their merge is somehow well defined.

A typical example is the merge sort algorithm. In merge sort we have two sorted arrays and all we want is to get the array representing their union again sorted. Of course, the tricky part in merge sort is the merging itself. That’s because we’ve to pass through the two arrays, A and B, and we’ve to compare each “pair” of items representing an item from A and from B. A bit off topic, but this is the weak point of merge sort and although its worst-case time complexity is O(n.log(n)), quicksort is often preferred in practice because there’s no “merge”. Quicksort just concatenates the two sub-arrays. Note that in quicksort the sub-arrays aren’t with an equal length in general and although its worst-case time complexity is O(n^2) it often outperforms merge sort.

This simple example from the paragraph above shows us how sometimes merging the solutions of two sub-problems actually isn’t a trivial task to do. Thus we must be careful when applying any divide and conquer approach.

History

Volker Strassen is a German mathematician born in 1936. He is well known for his works on probability, but in the computer science and algorithms he’s mostly recognized because of his algorithm for matrix multiplication that’s still one of the main methods that outperforms the general matrix multiplication algorithm.

Strassen firstly published this algorithm in 1969 and proved that the n^3 algorithm isn’t the optimal one. Actually the given solution by Strassen is slightly better, but his contribution is enormous because this resulted in many more researches about matrix multiplication that led to some faster approaches, i.e. the Coppersmith-Winograd algorithm with O(n^2,3737).

Overview

The general algorithm on multiplying two matrices A[NxN] and B[NxN] is fairly simple. Although it’s more difficult than multiplying two numbers and also it is not commutative it’s still very simple – but slow.

Let’s first define what’s a matrix A[NxN]. As we speak about matrices NxN we usually think of a square grid with N rows and N columns. In each row and column A[i][j] we’ve a value.

Of course, as developers, we can think of a matrix as a two-dimensional array.

// PHP two-dimensional array
$a = array(
    0 => array($v1, $v2, $v3, $v4),
    1 => array($v5, $v6, $v7, $v8),
    2 => array($v9, $v10, $v11, $v12),
); 

Don’t forget that a NxN matrix is just a private case for a matrix. We can equally likely have any other size of a matrix NxM (N <> M).

However the size of a matrix is crucial in order to multiply it with another matrix. Why is that?

As I said above multiplying matrices isn’t the same as multiplying numbers. First of all this operation isn’t commutative.

And the second problem is the way you multiply two matrices A with B.

Just because this works with NxN matrices we can see the problem with multiplying rectangular matrices. Indeed, this wouldn’t be possible unless the second dimension of A isn’t exactly equal to the first dimension of B.

Hopefully we are now talking about square matrices with exactly the same dimensions.

OK, so now we know how to multiply two square matrices (with the same dimensions NxN) and now let’s evaluate the time complexity for the general purpose algorithm.

As we know A.B = C only when:

C[i][j] = sum(A[i][k] * B[k][j]) for k = 0 .. n

Thus we have n^3 operations. Let’s try to find out a divide and conquer approach.

Indeed this isn’t difficult in case of matrices because as we know we can divide in matrix in smaller sub-matrices.

Now what do we have?

Again – the same complexity – we have 8 products and 4 sums. Where’s the catch?

Of course in order to get faster solution we’ve to be looking as Strassen did in 1969. He defined P1, P2, P3, P4, P5, P6 and P7 as defined on the image below.

Complexity

As I mentioned above the Strassen’s algorithm is slightly faster than the general matrix multiplication algorithm. The general algorithm’s time complexity is O(n^3), while the Strassen’s algorithm is O(n^2.80).

You can see on the chart below how slightly faster is this even for large n.

Application

Although this algorithm seems to be more close to pure mathematics than to computer practically everywhere we use NxN arrays we can benefit from matrix multiplication.

In the other hand the algorithm of Strassen is not much faster than the general n^3 matrix multiplication algorithm. That’s very important because for small n (usually n < 45) the general algorithm is practically a better choice. However as you can see from the chart above for n > 100 the difference can be very big.

In the same time typically NxN arrays are used always when we talk about adjacency matrix of graphs |V| = n and some graph algorithms practically depend on matrix multiplication.

Here’s a classical task on graphs. We have a group of cities and we must wire them to provide them all with electricity. Out of all possible connections we can make, which one is using minimum amount of wire.

To wire N cities, it’s clear that, you need to use at least N-1 wires connecting a pair of cities. The problem is that sometimes you have more than one choice to do it. Even for small number of cities there must be more than one solution as shown on the image bellow.

Here we can wire these four nodes in several ways, but the question is, which one is the best one. By the way defining the term “best one” is also tricky. Most often this means which uses least wire, but it can be anything else depending on the circumstances.

As we talk on weighted graphs we can generally speak of a minimum weight solution through all the vertices of the graph.

By the way there might be more the one equally optimal (minimal) solutions.

Overview

Obviously we must choose those edges that are enough to connect all the vertices of the graph and whose sum of weights is minimal. Since we can’t have cycles in our final solution it must form a tree. Thus we’re speaking on a minimum weight spanning tree, as the tree spans over the whole graph.

Does each connected and weighted graph have a minimum spanning tree? The answer is yes! By removing the cycles from the graph G we get a spanning tree, since it’s connected. From all possible spanning trees one or more are minimal.

If w(u, v) is the weight of the edge (u, v), we can speak of weight of any spanning tree T – w(T) which is the sum of all the edges forming that tree.

Thus the weight of the minimum spanning tree is less than the weight of whatever other spanning tree of G.

After we’re sure that there is at least one minimum spanning tree for all connected and weighted graphs we only need to find it somehow.

We can go with an incremental approach. At the end we’ll have the minimum spanning tree (MST), but before that on each step of our algorithm we’ll have a sub-set of this final tree, which will grow and grow until it becomes the real MST. This subset of edges we’ll keep in one additional set A.

So far we know that on each step we have a subset of the final MST, but first we need to answer a couple of questions.

How do we start?

Well, we’ll start with the empty set of edges. Clearly the empty set is a subset of any other set, thus it will be also a subset of the MST.

How do we grow the tree?

Another question we must answer is how to grow the tree. Since we have a MST sub-set (A) on each step how do we add an edge to this set in order to get another (bigger than the previous one) subset of edges, which will be again a subset of the minimum spanning tree?

Clearly we must make a decision which edge to add to the growing subset and this is the tricky part of this algorithm.

Chose the lowest weight edge!

To find the minimum spanning tree on each step we must get the lowest weighted edge that connects our subset (A) with the rest of the vertices.

However can we be sure that by choosing the less weighted edge we’ll get the MST? Well, let’s assume that isn’t right in order to prove that wrong!

OK so on some step of our growing sub-tree we don’t get the lightest edge (u, v), because we somehow doubt this rule, and we get another edge – let’s say (x, y). Mind that w(x, y) >= w(u, v).

Thus our final MST will contain somewhere in its set of edges the edge (x, y), but the weight of MST w(T) is minimal, and if we get another spanning tree that contains the exact same edges as T but instead of (x, y) contains (u, v) we’ll get a smaller weight!

That isn’t possible! Thus we proved that on each step we must get the less weighted edge.

This particular approach is called “greedy”, because on each step we get the best possible choice. However greedy algorithms don’t always get the right or optimal solution. Fortunately for MST this isn’t true so we can be greedy as much as we can!

OK let’s make a summary of our algorithm in the following pseudo code.

Pseudo Code

1. We start with an  empty set (A) subset of the final MST;
2. Until A does not form T:
      a. Get the less weighted edge u from G;	
      b. Add u to A;
3. Return A

Application

Actually this algorithm is used firstly by Borůvka which started to wire Moravia in 1926. Even without knowing that the “greedy” approach will lead him to the right solution he optimally covered Moravia with electricity.

However this algorithm is too general and there are two main algorithms – the Prim’s algorithm and the Kruskal’s algorithm that we shall see in future posts.

The thing is that on each step we must get the less weighted edge and both algorithms use different approaches to do that.