One of the two main algorithms in finding the minimum spanning tree algorithms is the algorithm of Kruskal. Before getting into the details, let’s get back to the principles of the minimum spanning tree.

We have a weighted graph and of all spanning trees we’d like to find the one with minimal weight. As an example on the picture above you see a spanning tree (T) on the graph (G), but that isn’t the minimum weight spanning tree!

We can think of a group of islands and the possible connections of bridges connecting them. Of course building bridges is expensive and time consuming, so we must be aware of what kind of bridges we want to build. Nevertheless there is an important question, what’s the minimum price we’d like to pay to build such set of bridges connecting all the islands.

Thus we practically need to build a minimum spanning tree, where the vertices will be the islands, while the edges will be the possible bridges between them. Every possible bridge has a weight (the price or the time we need to build it, etc.).

This scenario is only one of possible use cases of where minimum spanning trees can be used in practice.

The two main approaches – the Kruskal’s and the Prim’s algorithms however differ.

Overview

The algorithm of Kruskal starts by initializing a set of |V| trees.

During the process of building the final spanning tree we keep a forest. Obviously we start with a forest with |V| trees, where each tree is a single node tree.

On some point we have a forest of “k” trees which are all a sub-trees of the minimum spanning tree.

Finally one step before building the final MST we have two trees and we connect them with the less weighted edge left that connects them.

It’s important to note that during the process of building the tree we sort the edges in ascending order by their weight.

Than we start getting edges and check whether their ends (the two vertices making the edge) belong to a different sub-trees.

Pseudo Code

1. T (the final spanning tree) is defined to be the empty set;
2. For each vertex v of G, make the empty set out of v;
3. Sort the edges of G in ascending (non-decreasing) order;
4. For each edge (u, v) from the sored list of step 3.
      If u and v belong to different sets
         Add (u,v) to T;
         Get together u and v in one single set;
5. Return T

A great feature about the Kruskal’s algorithm is that it also work on disconnected graphs.

History

Kruskal’s algorithm is named after Joseph Kruskal, who wasn’t only computer scientist, but also prominent mathematician and statistician. Although he is best known for its algorithm for computing the minimum spanning tree, described in this post, he’s also known with his work as a statistician and his contribution to the formulation of multidimensional scaling.

Kruskal also explored the Indo-European languages contributing the studies of the linguistics along with other scientists. His “Indo-European Lexicographical List” (http://www.wordgumbo.com/ie/cmp/) is still widely used.

As we saw in the previous post, the algorithm of Dijkstra is very useful when it comes to find all the shortest paths in a weighted graph. However it has one major problem! Obviously it doesn’t work correctly when dealing with negative lengths of the edges.

We know that the algorithm works perfectly when it comes to positive edges, and that is absolutely normal because we try to optimize the inequality of the triangle.

Since Dijkstra’s algorithm make use of a priority queue normally we get first the shortest adjacent edge to the starting point. In our very basic example we’ll get first the edge with the length of 3 -> (S, A).

However when it comes to negative edges we can’t use any more priority queues, so we need a different, yet working solution.

Overview

The solution was published by Richard E. Bellman and Lester Ford, Jr. in 1958 in their publication “On a Routing Problem” and it is quite simple to explain and understand. Since we can prioritize the edges by its lengths the only thing we should do is to calculate all the paths. And to be sure that our algorithm will find all the paths correctly we repeat that N-1 times, where N is the number of vertices (|V| = N)!

In this very basic image we can see how Bellman-Ford solves the problem. First we get the distances from S to A and B, which are respectively 3 and 4, but there is a shorter path to A, which passes through B and it is (S, B) + (B, A) = 4 – 2 = 2.

Code

Here’s the code on PHP. Note that this time we use an adjacency matrix and an additional array of distances. It’s important (for directed graphs, and our graph this time is directed) to put the positive value of A[j][i] if A[i][j] is negative. Note the case for A[1][2]!

define('INFINITY', 10000000);

$matrix = array(
    0 => array( 0,  3,  4),
    1 => array( 0,  0,  2),
    2 => array( 0,  -2, 0),
);

$len = count($matrix);

$dist = array();

function BellmanFord(&$matrix, &$dist, $start)
{
    global $len;
    
    foreach (array_keys($matrix) as $vertex) {
        $dist[$vertex] = INFINITY;
        if ($vertex == $start) {
            $dist[$vertex] = 0;
        }
    }
    
    for ($k = 0; $k < $len - 1; $k++) {
        for ($i = 0; $i < $len; $i++) {
            for ($j = 0; $j < $len; $j++) {
                if ($dist[$i] > $dist[$j] + $matrix[$j][$i]) {
                    $dist[$i] = $dist[$j] + $matrix[$j][$i];
                }
            }
        }
    }
}

BellmanFord($matrix, $dist, 0);

// [0, 2, 4]
print_r($dist);

Complexity

The complexity is clearly O(n³) which follows directly from the code above.

Application

Actually this algorithm is very useful and it not only works with negative weights, but also can help us find negative cycles in the graph.

This is done with the simple check after the main loop.

    for ($i = 0; $i < $len; $i++) {
        for ($j = 0; $j < $len; $j++) {
            if ($dist[$i] > $dist[$j] + $matrix[$j][$i]) {
                echo 'The graph contains a negative cycle!';
            }
        }
    }

And here’s the full code.

$matrix = array(
    0 => array( 0,  3,  4),
    1 => array( 0,  0,  2),
    2 => array( 0,  -2, 0),
);

$len = count($matrix);

$dist = array();

function BellmanFord(&$matrix, &$dist, $start)
{
    global $len;
    
    foreach (array_keys($matrix) as $vertex) {
        $dist[$vertex] = INFINITY;
        if ($vertex == $start) {
            $dist[$vertex] = 0;
        }
    }
    
    for ($k = 0; $k < $len - 1; $k++) {
        for ($i = 0; $i < $len; $i++) {
            for ($j = 0; $j < $len; $j++) {
                if ($dist[$i] > $dist[$j] + $matrix[$j][$i]) {
                    $dist[$i] = $dist[$j] + $matrix[$j][$i];
                }
            }
        }
    }
    
    for ($i = 0; $i < $len; $i++) {
        for ($j = 0; $j < $len; $j++) {
            if ($dist[$i] > $dist[$j] + $matrix[$j][$i]) {
                echo 'The graph contains a negative cycle!';
            }
        }
    }
}

BellmanFord($matrix, $dist, 0);

// [0, 2, 4]
print_r($dist);

We already know how we can find the shortest paths in a graph starting from a given vertex. Practically we modified breadth-first search in order to calculate the distances from s to all other nodes reachable from s. We know that this works because BFS walks through the graph level by level.

Some sources give a very simple explanation of how BFS finds the shortest paths in a graph. We must just think of the graph as a set of balls connected through strings.

As we can see by lifting the ball called “S” all other balls fall down. The closest balls are directly connected to “s” and this is the first level, while the outermost balls are those with longest paths.

Clearly edges like those between A and B doesn’t matter for our BFS algorithm because they don’t make the path from S to C through B shorter. This is also known as the triangle inequality, where the sum of the lengths of two of the sides of the triangle is always greater than the length of the third side.

We must only answer the question is BFS the best algorithm that finds the shortest path between any two nodes of the graph? This is a reasonable question because as we know by using BFS we don’t find only the shortest path between given vertices i and j, but we also get the shortest paths between i and all other vertices of G. This is an information that we actually don’t need, but can we find the shortest path between i and j without that info?

The answer is simply “no”! Practically depth-first search can’t help us. Even worse – we can find paths that are far not the shortest ones.

In the image above using DFS the distance between 1 and 7 is 7 while practically there is an edge between them.

So BFS is the optimal algorithm for finding shortest paths in a graph. But there’s a catch! This algorithm works fine when we assume that all the edges are the same length. In the examples so far each edge has the value of 1. So N edges between s and i made the distance between them of a length N.

Overview

As we know in practice different edges can have different values. Exactly that was the case in weighted graphs. Going back to the road map example the distances between different cities are commonly evaluated in miles or kilometers. Of course we can associate any other meaningful value to this edges. This can be either time in hours to travel between cities, money for fuel or anything else.

Now BFS can’t help us any more. Why? Because using non-equal values for the edges the triangle inequality is no longer true. Now the edge (the direct path) between A and B can be greater than the sum of the two edges (A, C) + (C, B)!

In other words, assuming the same abstraction with balls and wires the hanging wires can’t be discarded so easily.

So now how can we solve this problem? A very dummy approach is to break apart each edge with dummy vertices in order to make BFS work again.

However this approach has several weak points. The major one is that we’ll have to keep much more information, which means more memory usage, for even small graphs. This is done in case we break each edge on too many parts.

The solution of this problem was given by Edsger Dijkstra in 1956 and published in 1959. The only thing we should do now is to be sure that even discarding the triangle inequality we have the shortest paths. The first thing to do is to keep information for the distance from s to the parent (previous) node of i in the graph in order to calculate which distance is shorter.

In BFS we used a queue in order to walk through all the ancestors of a node. This was made consecutively. Thus for the graph G on the next image the order of enqueuing the ancestors of S was A, B, C.

The Dijkstra’s algorithm make use of a priority queue, also know as a heap. This fact combined by the fact we keep info for the shortest path so far help us find shortest paths in a weighted graphs.

Why this works? To answer this question let’s see the next very basic example, assuming the graph G from the next image. As we can see the triangle inequality isn’t true.

OK, we see that the path [S, B, A] is shorter than [S, A] although the edge (S, A) exists. How the Dijkstra algorithm overcomes this problem.

First we have no information about the distances (S, A) and (S, B), the only thing we know is that S is the starting point, its distance is 0 and its path so far is the empty set. So first we enqueue in a priority the distances from S to A and B.

Now we dequeue the minimum (first in the heap) element from the queue – the closest node to S, which is B. Then all the nodes adjacent to S in the queue are tested for adjacency to B, thus if we have already the distance between S and A now we can test if its longer than (S, B) + (B, A) – the triangle inequality!

So far we know that we must change a bit BFS to get the Dijkstra algorithm. The only thing to do is to keep info for each node for the path through its parent and to use a priority queue.

Code

Implementing this algorithms isn’t much more difficult than BFS, so here’s the code in PHP. However this example make use of the standard php library SPL and the PriorityQueue data structure, but any developer can code his own heap.

Here’s the graph from the code:

class vertex
{
    public $key         = null;
    public $visited     = 0;
    public $distance    = 1000000;  // infinite
    public $parent      = null;
    public $path        = null;
    
    public function __construct($key) 
    {
        $this->key  = $key;
    }
}

class PriorityQueue extends SplPriorityQueue
{
    public function compare($a, $b)
    {
        if ($a === $b) return 0;
        return $a > $b ? -1 : 1;
    }
}

$v0 = new vertex(0);
$v1 = new vertex(1);
$v2 = new vertex(2);
$v3 = new vertex(3);
$v4 = new vertex(4);
$v5 = new vertex(5);

$list0 = new SplDoublyLinkedList();
$list0->push(array('vertex' => $v1, 'distance' => 3));
$list0->push(array('vertex' => $v3, 'distance' => 1));
$list0->rewind();

$list1 = new SplDoublyLinkedList();
$list1->push(array('vertex' => $v0, 'distance' => 3));
$list1->push(array('vertex' => $v2, 'distance' => 7));
$list1->rewind();

$list2 = new SplDoublyLinkedList();
$list2->push(array('vertex' => $v1, 'distance' => 7));
$list2->push(array('vertex' => $v3, 'distance' => 8));
$list2->push(array('vertex' => $v4, 'distance' => 12));
$list2->rewind();

$list3 = new SplDoublyLinkedList();
$list3->push(array('vertex' => $v0, 'distance' => 1));
$list3->push(array('vertex' => $v2, 'distance' => 8));
$list3->rewind();

$list4 = new SplDoublyLinkedList();
$list4->push(array('vertex' => $v2, 'distance' => 12));
$list4->push(array('vertex' => $v5, 'distance' => 3));
$list4->rewind();

$list5 = new SplDoublyLinkedList();
$list5->push(array('vertex' => $v4, 'distance' => 3));
$list5->rewind();

$adjacencyList = array(
    $list0,
    $list1,
    $list2,
    $list3,
    $list4,
    $list5,
);

function calcShortestPaths(vertex $start, &$adjLists)
{
    // define an empty queue
    $q = new PriorityQueue();
    
    // push the starting vertex into the queue
    $q->insert($start, 0);
    $q->rewind();
    
    // mark the distance to it 0
    $start->distance = 0;
    
    // the path to the starting vertex
    $start->path = array($start->key);
    
    while ($q->valid()) {
        $t = $q->extract();
        $t->visited = 1;
        
        $l = $adjLists[$t->key];
        while ($l->valid()) {
            $item = $l->current();
            
            if (!$item['vertex']->visited) {
                if ($item['vertex']->distance > $t->distance + $item['distance']) {
                    $item['vertex']->distance = $t->distance + $item['distance'];
                    $item['vertex']->parent = $t;
                }
                
                $item['vertex']->path = array_merge($t->path, array($item['vertex']->key));
                
                $q->insert($item["vertex"], $item["vertex"]->distance);
            }
            $l->next();
        }
        $q->recoverFromCorruption();
        $q->rewind();
    }
}

calcShortestPaths($v0, $adjacencyList);

// The path from node 0 to node 5
// [0, 1, 2, 4, 5]
echo '[' . implode(', ', $v5->path) . ']';

Complexity

The complexity of that code is based on the complexity of BFS with the main difference that we keep a priority queue. For BFS we knew that the complexity was O(|V| + |E|), while Dijkstra’s algorithm has running time of O((|V| + |E|).log(|V|)). That is quite natural since the heapsort’s complexity is O(n.log(n))!

Application

Since the basic BFS can’t help us for weighted graphs and there are plenty of problems designed with weighted graphs obviously Dijkstra’s algorithm can be very handy. The only thing we should be aware of is the positive values of the edges.

To find the minimum value into an array of items itsn’t difficult. There are not many options to do that. The most natural approach is to take the first item and to compare its value against the values of all other elements. Once we find a smaller element we continue the comparisons with its value. Finally we find the minimum.

First thing to note is that we pass through the array with n steps and we need exactly n-1 comparisons. It’s clear that this is the optimal solution, because we must check all the elements. For sure we can’t be sure that we’ve found the minimum (maximum) value without checking every single value.

Overview

The algorithm above is very simple and we’re sure that it is optimal. Obviously finding both the minimum and the maximum value is O(n) with n-1 comparisons, but what about combining these tasks into one single pass.

Since they both are O(n) and need n-1 comparisons it’s natural to think that combining the two tasks will be O(n) and 2n – 2 comparisons. However we can reduce the number of comparisons!

Instead of taking only one item from the array and comparing it against the minimum and maximum we can take a pair of items at each step. Thus we can first compare them and then compare the smaller value with the currently smallest value and the greater item with the currently greatest value. This will make only 3 comparisons instead of 4.

Implementation

It’s easy to implement the minimum (maximum) algorithms with a single loop.

The implementation of finding the maximum is practically the same.

Simply merging these two functions will lead us to a O(n) with 2n – 2 comparisons solution.

However we can take a pair of items on each step. First we’ll compare the items from that pair and after that we’ll compare them respectively with the minimum and the maximum value. Because on each iteration we jump by two items, in case the number of array items is even we must check for the array boundaries. This can be overcome by adding a sentinel. Thus the array items are always odd, but this will lead us to a “extra” memory solution.

Sentinel

Without sentinel

Complexity

The complexity of finding both minimum and maximum is O(n). Even after combining the both algorithms in one single pass the complexity remains O(n). However in the second case we can reduce the number of comparisons to 3 * ceil(n/2) instead of 2n – 2!

Application

This algorithm can be applied in various fields of the computer science, since its nature is so basic. However there are two reasons why this approach is so important.

First we can see how by combining two “algorithms” doesn’t mean that we combine their complexities or the number of operations. With a clever trick and with the observation that the two operations are related (minimum and maximum) we can reduce the number of comparisons.

In the other hand we see how using a sentinel can be very handy and can spare us some comparisons, just like the sequential search.