We already know what’s topological sort of a directed acyclic graph. So why do we need a revision of this algorithm? First of all I never mentioned its complexity, thus to understand why we do need a revision let’s get again on the algorithm.

We have a directed acyclic graph (DAG). There are no cycles so we must go for some kind of order putting all the vertices of the graph in such an order, that if there’s a directed edge (u, v), u must precede v in that order.

The process of putting all the vertices of the DAG in such an order is called topological sorting. It’s commonly used in task scheduling or while finding the shortest paths in a DAG.

The algorithm itself is pretty simple to understand and code. We must start from the vertex (vertices) that don’t have predecessors.

We put them in our sorted list in random order. Since they don’t depend on each other we can assume they are equally sorted already. Indeed thinking of a task schedule if there are tasks that don’t have predecessors (they don’t depend on other tasks before them) and that don’t depend on each other we can put them in random order (and execute them in random order).

Once we have the vertices with no predecessors we must remove the edges starting from them. Then – go again with the vertices with no predecessors.

It’s as simple as that, so why do we need a revision of this algorithm? Well, basically because of its efficiency.

Overview

As we know most of the graph algorithms depend on the way the graph is represented in our application. We consider as the two main representations the adjacency matrix …

… and adjacency lists.

Let’s first take a look of some of the main approaches to get the topologically sorted list at the end of the algorithm.

What can we do in order to find the vertices with no predecessors? We can only scan the entire list of vertices.

Adjacency Matrix

In case we’re using adjacency matrix we need|V|^2 space to store the graph. To find the vertices with no predecessors we have to scan the entire graph, which will cost us O(|V|^2) time. And we’ll have to do that |V| times. This will be |V|^3 time consuming algorithm and for dense graphs this will be quite an ineffective algorithm.

Adjacency Lists

What about the adjacency list? There we need |E| space to store a directed graph. How fast can we find a node with no predecessor? Practically we’ll need O(|E|) time. Thus in the worst case we have again O(|V|^2) time consuming programs.

So what can be done in order to optimize this algorithm?

Practically we can start by picking up a random vertex and “go back” until we get a node with no predecessors. This approach can be very effective yet also very ineffective. First of all if we have to scan all the way back to a node with no predecessors this will cost us |V| time, but if we stuck on a node that don’t have a preceding node then we’ll have a constant speed.

This means that we can modify the algorithm a bit in order to improve a lot the algorithm. We just need to store both incoming and outgoing edges and slightly modify the adjacency lists.

What’s the algorithm now?

First we easily find the nodes with no predecessors. Then, using a queue, we can keep the nodes with no predecessors and on each dequeue we can remove the edges from the node to all other nodes.

Pseudo Code

1. Represent the graph with two lists on each vertex (incoming edges and outgoing edges)
2. Make an empty queue Q;
3. Make an empty topologically sorted list T;
4. Push all items with no predecessors in Q;
5. While Q is not empty
   a. Dequeue from Q into u;
   b. Push u in T;
   c. Remove all outgoing edges from u;
6. Return T;

This approach will give us a better performance than the “brute force” approach. The running time complexity is O(|V| + |E|). The problem is that we need additional space and an operational queue, but this approach is a perfect example of how by using additional space you can get a better performing algorithm.

A very common problem in computer programming is finding the longest increasing (decreasing) subsequence in a sequence of numbers (usually integers). Actually this is a typical dynamic programming problem.

Dynamic programming can be described as a huge area of computer science problems that can be categorized by the way they can be solved. Unlike divide and conquer, where we were able to merge the fairly equal sub-solutions in order to receive one single solution of the problem, in dynamic programming we usually try to find an optimal sub-solution and then grow it.

Once we have an optimal sub-solution on each step we try to upgrade it in order to cover the whole problem. Thus a typical member of the dynamic programming class is finding the longest subsequence.

However this problem is interesting because it can be related to graph theory. Let’s find out how.

Overview

We already know various ways to calculate the shortest paths in a graph. Indeed finding the single-source shortest path is a typical graph problem. To model such kind of solutions we definitely need a graph represented in our solution.

However the single-source shortest path isn’t a straight-forward problem. It depends on many factors. Thus for positive edges the algorithm of Edsger Dijkstra can be a perfect solution, but when the graph contains negative edges his algorithm is no longer useful.

In the presence of negative edges the Dijkstra’s algorithm doesn’t work and we’d better use the Bellman-Ford algorithm. It is interesting to note, that it was exactly Richard Bellman who first introduced the term “dynamic programming” in the 1940s.

In fact the Bellman-Ford algorithm was able to detect negative cycles. That’s too important, because in presence of negative cycles the shortest path problem is no longer well defined.

In the other hand when we’re talking about shortest paths in a DAG (Directed Acyclic Graph) we can find a faster (linear) solution. That’s because we’re sure that there are no cycles (not even negative cycles)!

Finding the shortest paths in a DAG is closely related to the topological sorting of the DAG. This gives us a linear representation of the vertices of the DAG and we can clearly calculate the distances from the starting node to all other nodes. Note that in a DAG we have one or more nodes that can be considered as starting nodes – which means they don’t have predecessors (incoming edges).

Following the words above is pretty hard to find what all these graph algorithms have to do with finding the longest increasing (decreasing) subsequence. Actually this problem is very closely related to the toplogical sort of the DAG and the problem of finding shortest paths in a DAG.

That’s because we can represent our sequence as a DAG. The only thing we must care about is to “connect” with directed edges those elements that form an increasing (decreasing) pair.

Thus the sequence from our example [1, 8, 2, 7, 3, 4, 1, 6] is going to look like this.

Another important thing to note is that we don’t search for shortest, but for longest path, since our task is to find the longest subsequence.

Pseudo Code

Our pseudo code for finding the shortest paths in a DAG was something like that.

1. Get the toplogically sorted list L of the DAG;
2. The starting node is s;
3. The distance to s equals to 0;
4. All other distances are initialized to ∞
5. For each node (v) in L\{s} do:
5.1. If dist(v) > dist(u) + w(u, v) then
5.1.1.  dist(v) := dist(u) + w(u, v)

In the above pseudo code “u” is every predecessor of “v”!

Now we must “reverse” the solution above in order to find the longest increasing subsequence. Note that we don’t care any more about the weight of the edges, thus we can simply substitute them with 1.

1. Get the sequence (L) as a toplogically sorted DAG;
2. For each (i) in L do:
2.1. S(i) := 1 + max(S(j), where (i, j) is an edge from the DAG);

Application

Finding the longest increasing subsequence can be very useful not only at the Google/Yahoo/Facebook interview, but also in various fields of statistics.

We saw how to find the shortest path in a graph with positive edges using the Dijkstra’s algorithm. We also know how to find the shortest paths from a given source node to all other nodes even when there are negative edges using the Bellman-Ford algorithm. Now we’ll see that there’s a faster algorithm running in linear time that can find the shortest paths from a given source node to all other reachable vertices in a directed acyclic graph, also known as a DAG.

Because the DAG is acyclic we don’t have to worry about negative cycles. As we already know it’s pointless to speak about shortest path in the presence of negative cycles because we can “loop” over these cycles and practically our path will become shorter and shorter.

Thus we have two problems to overcome with Dijkstra and the Bellman-Ford algorithms. First of all we needed only positive weights and on the second place we didn’t want cycles. Well, we can handle both cases in this algorithm.

Overview

The first thing we know about DAGs is that they can easily be topologically sorted. Topological sort can be used in many practical cases, but perhaps the mostly used one is when trying to schedule dependent tasks.

After a topological sort we end with a list of vertices of the DAG and we’re sure that if there’s an edge (u, v), u will precede v in the topologically sorted list.

This information is precious and the only thing we need to do is to pass through this sorted list and to calculate distances for a shortest paths just like the algorithm of Dijkstra.

OK, so let’s summarize this algorithm:
– First we must topologically sort the DAG;
– As a second step we set the distance to the source to 0 and infinity to all other vertices;
– Then for each vertex from the list we pass through all its neighbors and we check for shortest path;

It’s pretty much like the Dijkstra’s algorithm with the main difference that we used a priority queue then, while this time we use the list from the topological sort.

Code

This time the code is actually a pseudocode. Altough all the examples so far was in PHP, perhaps pseudocode is easier to understand and doesn’t bind you in a specific language implementation. Also if you don’t feel comforatable with the given programming language it can be more difficult for you to understand the code than by reading pseudocode.

1. Topologically sort G into L;
2. Set the distance to the source to 0;
3. Set the distances to all other vertices to infinity;
4. For each vertex u in L
5.    - Walk through all neighbors v of u;
6.    - If dist(v) > dist(u) + w(u, v) 
7.       - Set dist(v) <- dist(u) + w(u, v);

Application

It’s clear why and where we must use this algorithm. The only problem is that we must be sure that the graph doesn’t have cycles. However if we’re aware of how the graph is created we may have some additional information if there are cycles or not – then this linear time algorithm can be very applicable.

Let’s assume we have a list of tasks to accomplish. Some of the tasks depend on others, so we must be very careful with the order of their execution. If the relationship between these tasks were simple enough we could represent them as a linked list, which would be great, and we would know the exact order of their execution. The problem is that sometimes the relations between the different tasks are more complex and some tasks depend on two or more other tasks, which in their turn depend on one or more tasks, etc.

Thus we can’t model this problem using linked lists or trees. The only rational solution is to model the problem using a graph. What kind of graph do we need? Well, we definitely need a directed graph, to desribe the relations, and this graph shouldn’t have cycles. So we need the so called directed acyclic graph (DAG).

Why we don’t what a cycle in the graph? The answer of this question is simple and obvious. In case of cyclic graph, we wouldn’t be able to determine the priority of task execution, thus we won’t be able to sort the tasks properly.

Now the solution we want is to sort the vertices of the graph in some order so for each edge (u, v) u will precede v. Then we’ll have a linear order of all tasks and by starting their execution we’ll know that everything will be OK.

This kind of sort is also known as “topological” sort (or topsort) and it is one of the very basic graph algorithms.

Overview

OK, so we have an acyclic directed graph, how do we proceed to get a linked list with all the vertices sorted? Since it’s an acyclic graph we know that there is at least one vertex without predecessor. Thus at first place, we can put all the vertices without predecessors into our linked list.

This approach answers the question – is there a possibility to have more than one valid topological sort of a graph? Indeed, the only thing we’d like to do is to put all the vertices in the correct order, but since there might be vertices with no predecessors any combination of them will be a valid topological sort for a graph.

As we can see from the picture above even for vertices with predecessors our topological sort can vary. Thus [9, 6, 2, 7, 4, 1] is a valid topological sorted graph, but [6, 9, 2, 7, 4, 1] is also a valid topological sort out of the same graph!

Now we can generalize the algorithm in some basic steps.

1. Make an empty list L and an empty list S;
2. Put all the vertices with no predecessors in L;
3. While L has items in it;
3.1. Pop an item from L – n, and push it to S;
3.2. For each vertex m adjacent to n;
3.2.1. Remove (n, m);
3.2.2. If m has no predecessors – push it to L;

Code

Here’s the very basic PHP implementation. As you can see the short implementation shows us how easy this algorithm is. However its importance to computer science and programming is enormous.

class G
{
    protected $_g = array(
        array(0, 1, 1, 0, 0, 0, 0),
        array(0, 0, 0, 1, 0, 0, 0),
        array(0, 0, 0, 0, 1, 0, 0),
        array(0, 0, 0, 0, 1, 0, 0),
        array(0, 0, 0, 0, 0, 0, 1),
        array(0, 0, 0, 0, 0, 0, 1),
        array(0, 0, 0, 0, 0, 0, 0),
    );
    protected $_list = array();
    protected $_ts   = array();
    protected $_len  = null;
    
    public function __construct()
    {
        $this->_len = count($this->_g);
        
        // finds the vertices with no predecessors
        $sum = 0;
        for ($i = 0; $i < $this->_len; $i++) {
            for ($j = 0; $j < $this->_len; $j++) {
                $sum += $this->_g[$j][$i];
            }
            
            if (!$sum) {
                // append to list
                array_push($this->_list, $i);
            }
            $sum = 0;
        }
    }
    
    public function topologicalSort() 
    {
        while ($this->_list) {
            $t = array_shift($this->_list);
            array_push($this->_ts, $t);
            
            foreach ($this->_g[$t] as $key => $vertex) {
                if ($vertex == 1) {
                    $this->_g[$t][$key] = 0;
                    
                    $sum = 0;
                    for ($i = 0; $i < $this->_len; $i++) {
                        $sum += $this->_g[$i][$key];
                    }
                    
                    if (!$sum) {
                        array_push($this->_list, $key);
                    }
                }
                $sum = 0;
            }
        }
        
        print_r($this->_ts);
    }
}

$g = new G();
/*
Array
(
    [0] => 0
    [1] => 5
    [2] => 1
    [3] => 2
    [4] => 3
    [5] => 4
    [6] => 6
)*/
$g->topologicalSort();

Application

As I already mentioned above this algorithm is practically used to sort the execution of different tasks that depend on each other. However this isn’t its only use. Actually any kind of objects that depend on each other can be modeled with a graph. Indeed sometimes these graphs may be a trees, but most of the cases that isn’t true.