We already know what’s topological sort of a directed acyclic graph. So why do we need a revision of this algorithm? First of all I never mentioned its complexity, thus to understand why we do need a revision let’s get again on the algorithm.

We have a directed acyclic graph (DAG). There are no cycles so we must go for some kind of order putting all the vertices of the graph in such an order, that if there’s a directed edge (u, v), u must precede v in that order.

The process of putting all the vertices of the DAG in such an order is called topological sorting. It’s commonly used in task scheduling or while finding the shortest paths in a DAG.

The algorithm itself is pretty simple to understand and code. We must start from the vertex (vertices) that don’t have predecessors.

We put them in our sorted list in random order. Since they don’t depend on each other we can assume they are equally sorted already. Indeed thinking of a task schedule if there are tasks that don’t have predecessors (they don’t depend on other tasks before them) and that don’t depend on each other we can put them in random order (and execute them in random order).

Once we have the vertices with no predecessors we must remove the edges starting from them. Then – go again with the vertices with no predecessors.

It’s as simple as that, so why do we need a revision of this algorithm? Well, basically because of its efficiency.

Overview

As we know most of the graph algorithms depend on the way the graph is represented in our application. We consider as the two main representations the adjacency matrix …

… and adjacency lists.

Let’s first take a look of some of the main approaches to get the topologically sorted list at the end of the algorithm.

What can we do in order to find the vertices with no predecessors? We can only scan the entire list of vertices.

Adjacency Matrix

In case we’re using adjacency matrix we need|V|^2 space to store the graph. To find the vertices with no predecessors we have to scan the entire graph, which will cost us O(|V|^2) time. And we’ll have to do that |V| times. This will be |V|^3 time consuming algorithm and for dense graphs this will be quite an ineffective algorithm.

Adjacency Lists

What about the adjacency list? There we need |E| space to store a directed graph. How fast can we find a node with no predecessor? Practically we’ll need O(|E|) time. Thus in the worst case we have again O(|V|^2) time consuming programs.

So what can be done in order to optimize this algorithm?

Practically we can start by picking up a random vertex and “go back” until we get a node with no predecessors. This approach can be very effective yet also very ineffective. First of all if we have to scan all the way back to a node with no predecessors this will cost us |V| time, but if we stuck on a node that don’t have a preceding node then we’ll have a constant speed.

This means that we can modify the algorithm a bit in order to improve a lot the algorithm. We just need to store both incoming and outgoing edges and slightly modify the adjacency lists.

What’s the algorithm now?

First we easily find the nodes with no predecessors. Then, using a queue, we can keep the nodes with no predecessors and on each dequeue we can remove the edges from the node to all other nodes.

Pseudo Code

1. Represent the graph with two lists on each vertex (incoming edges and outgoing edges)
2. Make an empty queue Q;
3. Make an empty topologically sorted list T;
4. Push all items with no predecessors in Q;
5. While Q is not empty
   a. Dequeue from Q into u;
   b. Push u in T;
   c. Remove all outgoing edges from u;
6. Return T;

This approach will give us a better performance than the “brute force” approach. The running time complexity is O(|V| + |E|). The problem is that we need additional space and an operational queue, but this approach is a perfect example of how by using additional space you can get a better performing algorithm.

A very common problem in computer programming is finding the longest increasing (decreasing) subsequence in a sequence of numbers (usually integers). Actually this is a typical dynamic programming problem.

Dynamic programming can be described as a huge area of computer science problems that can be categorized by the way they can be solved. Unlike divide and conquer, where we were able to merge the fairly equal sub-solutions in order to receive one single solution of the problem, in dynamic programming we usually try to find an optimal sub-solution and then grow it.

Once we have an optimal sub-solution on each step we try to upgrade it in order to cover the whole problem. Thus a typical member of the dynamic programming class is finding the longest subsequence.

However this problem is interesting because it can be related to graph theory. Let’s find out how.

Overview

We already know various ways to calculate the shortest paths in a graph. Indeed finding the single-source shortest path is a typical graph problem. To model such kind of solutions we definitely need a graph represented in our solution.

However the single-source shortest path isn’t a straight-forward problem. It depends on many factors. Thus for positive edges the algorithm of Edsger Dijkstra can be a perfect solution, but when the graph contains negative edges his algorithm is no longer useful.

In the presence of negative edges the Dijkstra’s algorithm doesn’t work and we’d better use the Bellman-Ford algorithm. It is interesting to note, that it was exactly Richard Bellman who first introduced the term “dynamic programming” in the 1940s.

In fact the Bellman-Ford algorithm was able to detect negative cycles. That’s too important, because in presence of negative cycles the shortest path problem is no longer well defined.

In the other hand when we’re talking about shortest paths in a DAG (Directed Acyclic Graph) we can find a faster (linear) solution. That’s because we’re sure that there are no cycles (not even negative cycles)!

Finding the shortest paths in a DAG is closely related to the topological sorting of the DAG. This gives us a linear representation of the vertices of the DAG and we can clearly calculate the distances from the starting node to all other nodes. Note that in a DAG we have one or more nodes that can be considered as starting nodes – which means they don’t have predecessors (incoming edges).

Following the words above is pretty hard to find what all these graph algorithms have to do with finding the longest increasing (decreasing) subsequence. Actually this problem is very closely related to the toplogical sort of the DAG and the problem of finding shortest paths in a DAG.

That’s because we can represent our sequence as a DAG. The only thing we must care about is to “connect” with directed edges those elements that form an increasing (decreasing) pair.

Thus the sequence from our example [1, 8, 2, 7, 3, 4, 1, 6] is going to look like this.

Another important thing to note is that we don’t search for shortest, but for longest path, since our task is to find the longest subsequence.

Pseudo Code

Our pseudo code for finding the shortest paths in a DAG was something like that.

1. Get the toplogically sorted list L of the DAG;
2. The starting node is s;
3. The distance to s equals to 0;
4. All other distances are initialized to ∞
5. For each node (v) in L\{s} do:
5.1. If dist(v) > dist(u) + w(u, v) then
5.1.1.  dist(v) := dist(u) + w(u, v)

In the above pseudo code “u” is every predecessor of “v”!

Now we must “reverse” the solution above in order to find the longest increasing subsequence. Note that we don’t care any more about the weight of the edges, thus we can simply substitute them with 1.

1. Get the sequence (L) as a toplogically sorted DAG;
2. For each (i) in L do:
2.1. S(i) := 1 + max(S(j), where (i, j) is an edge from the DAG);

Application

Finding the longest increasing subsequence can be very useful not only at the Google/Yahoo/Facebook interview, but also in various fields of statistics.

The Strassen’s method of matrix multiplication is a typical divide and conquer algorithm. We’ve seen so far some divide and conquer algorithms like merge sort and the Karatsuba’s fast multiplication of large numbers. However let’s get again on what’s behind the divide and conquer approach.

Unlike the dynamic programming where we “expand” the solutions of sub-problems in order to get the final solution, here we are talking more on joining sub-solutions together. These solutions of some sub-problems of the general problem are equal and their merge is somehow well defined.

A typical example is the merge sort algorithm. In merge sort we have two sorted arrays and all we want is to get the array representing their union again sorted. Of course, the tricky part in merge sort is the merging itself. That’s because we’ve to pass through the two arrays, A and B, and we’ve to compare each “pair” of items representing an item from A and from B. A bit off topic, but this is the weak point of merge sort and although its worst-case time complexity is O(n.log(n)), quicksort is often preferred in practice because there’s no “merge”. Quicksort just concatenates the two sub-arrays. Note that in quicksort the sub-arrays aren’t with an equal length in general and although its worst-case time complexity is O(n^2) it often outperforms merge sort.

This simple example from the paragraph above shows us how sometimes merging the solutions of two sub-problems actually isn’t a trivial task to do. Thus we must be careful when applying any divide and conquer approach.

History

Volker Strassen is a German mathematician born in 1936. He is well known for his works on probability, but in the computer science and algorithms he’s mostly recognized because of his algorithm for matrix multiplication that’s still one of the main methods that outperforms the general matrix multiplication algorithm.

Strassen firstly published this algorithm in 1969 and proved that the n^3 algorithm isn’t the optimal one. Actually the given solution by Strassen is slightly better, but his contribution is enormous because this resulted in many more researches about matrix multiplication that led to some faster approaches, i.e. the Coppersmith-Winograd algorithm with O(n^2,3737).

Overview

The general algorithm on multiplying two matrices A[NxN] and B[NxN] is fairly simple. Although it’s more difficult than multiplying two numbers and also it is not commutative it’s still very simple – but slow.

Let’s first define what’s a matrix A[NxN]. As we speak about matrices NxN we usually think of a square grid with N rows and N columns. In each row and column A[i][j] we’ve a value.

Of course, as developers, we can think of a matrix as a two-dimensional array.

// PHP two-dimensional array
$a = array(
    0 => array($v1, $v2, $v3, $v4),
    1 => array($v5, $v6, $v7, $v8),
    2 => array($v9, $v10, $v11, $v12),
); 

Don’t forget that a NxN matrix is just a private case for a matrix. We can equally likely have any other size of a matrix NxM (N <> M).

However the size of a matrix is crucial in order to multiply it with another matrix. Why is that?

As I said above multiplying matrices isn’t the same as multiplying numbers. First of all this operation isn’t commutative.

And the second problem is the way you multiply two matrices A with B.

Just because this works with NxN matrices we can see the problem with multiplying rectangular matrices. Indeed, this wouldn’t be possible unless the second dimension of A isn’t exactly equal to the first dimension of B.

Hopefully we are now talking about square matrices with exactly the same dimensions.

OK, so now we know how to multiply two square matrices (with the same dimensions NxN) and now let’s evaluate the time complexity for the general purpose algorithm.

As we know A.B = C only when:

C[i][j] = sum(A[i][k] * B[k][j]) for k = 0 .. n

Thus we have n^3 operations. Let’s try to find out a divide and conquer approach.

Indeed this isn’t difficult in case of matrices because as we know we can divide in matrix in smaller sub-matrices.

Now what do we have?

Again – the same complexity – we have 8 products and 4 sums. Where’s the catch?

Of course in order to get faster solution we’ve to be looking as Strassen did in 1969. He defined P1, P2, P3, P4, P5, P6 and P7 as defined on the image below.

Complexity

As I mentioned above the Strassen’s algorithm is slightly faster than the general matrix multiplication algorithm. The general algorithm’s time complexity is O(n^3), while the Strassen’s algorithm is O(n^2.80).

You can see on the chart below how slightly faster is this even for large n.

Application

Although this algorithm seems to be more close to pure mathematics than to computer practically everywhere we use NxN arrays we can benefit from matrix multiplication.

In the other hand the algorithm of Strassen is not much faster than the general n^3 matrix multiplication algorithm. That’s very important because for small n (usually n < 45) the general algorithm is practically a better choice. However as you can see from the chart above for n > 100 the difference can be very big.

In the same time typically NxN arrays are used always when we talk about adjacency matrix of graphs |V| = n and some graph algorithms practically depend on matrix multiplication.

Since we already know how to represent graphs, we can go further for some very simple approaches of walking through them. Passing by all the vertices of a graph is a fundamental technique for most of the graph algorithms, such as finding shortest/longest paths, etc.

First thing to note is that graphs are not trees, in most of the cases, so walking through them can’t start from a root, as we do with trees. What we must do first is to decide from where to start – in other words – choosing a starting vertex.

After that we need to know how to proceed. There are two approaches mostly known as “breadth first” and “depth first” search. While depth first search start from a vertex and goes as far as possible, then walks back and passes through vertices that haven’t been visited yet, breath first search is an approach of passing through all the neighbors of the node first, and then go to the next level.

Overview

We can thing of breadth first search as a “wave” walk through the graph. In other words we go level by level, as shown on the picture below.

Initially we mark all vertices as unvisited. A common approach is to create an empty queue where we put the vertices level by level, starting with the initial vertex.

Code

This simple approach is fairly easy to implement. Here’s the PHP implementation in few lines of code.

 array(0, 1, 1, 0, 0, 0),
    1 => array(1, 0, 0, 1, 0, 0),
    2 => array(1, 0, 0, 1, 0, 0),
    3 => array(0, 1, 1, 0, 1, 0),
    4 => array(0, 0, 0, 1, 0, 1),
    5 => array(0, 0, 0, 0, 1, 0),
);

function init(&$visited, &$graph) 
{
    foreach ($graph as $key => $vertex) {
        $visited[$key] = 0;
    }
}

function breadth_first(&$graph, $start, $visited)
{
    // create an empty queue
    $q = array();
    
    // initially enqueue only the starting vertex
    array_push($q, $start);
    $visited[$start] = 1;
    echo $start . "\n";
    
    while (count($q)) {
        $t = array_shift($q);
        
        foreach ($graph[$t] as $key => $vertex) {
            if (!$visited[$key] && $vertex == 1) {
                $visited[$key] = 1;
                array_push($q, $key);
                echo $key . "\t";
            }
        }
        echo "\n";
    }
}

$visited = array();
init($visited, $g);
breadth_first($g, 2, $visited);

Complexity

The complexity of this algorithm clearly is O(n²).

Application

As I said breadth first and depth first searches are used in many practical cases, as finding shortest/minimal paths etc. That is why understanding these basic principles of walking through a graph is crucial for other, more complex, graph algorithms.