Heapsort is one of the general sorting algorithms that performs in O(n.log(n)) in the worst-case, just like merge sort and quicksort, but sorts in place – as quicksort. Although quicksort’s worst-case sorting time is O(n²) it’s often considered that it beats other sorting algorithms in practice. Thus in practice quicksort is “faster” than heapsort. In the same time developers tend to consider heapsort as more difficult to implement than other n.log(n) sorting algorithms.

In the other hand heapsort uses a special data structure, called heap, in order to sort items in place and this data structure is quite useful in some specific cases. Thus to understand heapsort we first need to understand what is a heap.

So first let’s take a look at what is a heap.

Overview

A heap is a complete binary tree, where all the parents are greater than their children (max heap). If all the children are greater than their parents it is considered to call the heap a min-heap. But first what is a complete binary tree? Well, this is a binary tree, where all the levels are full, except the last one, where all the items are placed on the left (just like on the image below).

Combined with the fact that each node contains a greater key than its children, a heap may look like the tree on the following diagram.

The thing is that if we put indices next to each node of this tree, starting from the root (index 1) and continuing from left to right on each level, we’ll get the following tree.

Now if we take a closer look to the picture above we can see that the indices of a node and its children are closely related. Thus for a node of an index i we see that its left child has the index 2*i, while its right child’s index is 2*i + 1.

This particular order gives us the possibility to store each heap in an array.

Since in a heap its greater element is in the root of the tree (for max-heap, respectively in a min-heap its smallest element is the root) we need to answer two questions.

1. How to build a heap out of an ordinary array?
2. After extracting the root, which is the greatest (smallest) item, how can we rebuild the heap in order to keep it a heap again?

First let’s try to answer the first question. How to build a heap? Well, let’s forget about the array for a while and let’s take a look on a ordinary binary tree with only three nodes.

We see that the three green nodes destroy the structure of our heap, because the root (1) is smaller than its children (4) and (5). Thus we need to fix this problem and what we’re going to do is to swap the root with its biggest child.

As you can see on the picture above the i-th item is first compared to its left child. The greater of these two items is compared to the right child. Note that we don’t swap them – we just compare them to get which one is greater. Once we find the greatest of these three values we swap them with the root in case it’s not the root value.

Although now these three elements form a heap, by swapping the root with one of its children may destroy the heap constructed out of this child. That is why we continue the same procedure with it.

This actually gives us the procedure to heapify the three nodes constructed out of the i-th item and its children. However to build a heap from an arbitrary array we should perform this operation starting from floor(len[A] / 2) down to the first item in the array.

Why? Well, a complete binary tree with a full last level contains n/2 + 1 nodes in it. They don’t have children, thus we don’t need to check them – they are “sorted”. Indeed if we start from an item on the right of floor(len[A] / 2) there won’t be items with indices 2*i and 2*i + 1.

Code

So far we know how to build the heap. Next thing is to swap the first and the last element of the array and rebuild the heap. Here’s the PHP code of how to do this.

$a = array(1, 6, 3, 8, 2, 5, 4);

function heapify(&$a, &$i, &$heap_size)
{
    $l = $i*2 + 1;
    $r = $i*2 + 2;
    
    if ($l < $heap_size && $a[$i] < $a[$l]) {
        $largest = $l;
    } else {
        $largest = $i;
    }
    
    if ($r < $heap_size && $a[$largest] < $a[$r]) {
        $largest = $r;
    }
    
    if ($largest != $i) {
        $t = $a[$i];
        $a[$i] = $a[$largest];
        $a[$largest] = $t;
        
        heapify($a, $largest, $heap_size);
    }
}

function build_heap(&$a, &$heap_size)
{
    $len = floor($heap_size / 2);
    for ($i = $len; $i > -1; $i--) {
        heapify($a, $i, $heap_size);
    }
}

function heapsort(&$a)
{
    $heap_size = count($a);
    build_heap($a, $heap_size);
    
    while ($heap_size--) {
        $t = $a[$heap_size];
        $a[$heap_size] = $a[0];
        $a[0] = $t;
        build_heap($a, $heap_size);
    }
}

// 1 2 3 4 5 6 8
heapsort($a);

Complexity

OK, the last question is – how do we know that this algorithm sorts in place in n.log(n) time? Let’s explore the algorithm one more time. The heapify worst-case is when we start from the root down to the lowest level of the tree. In these terms if the tree height is h, the time is O(h), but because the tree is balanced (complete) the time in terms of n is O(log(n)).

In the other hand to build a heap we walk from floor(len[A] / 2) to 0, which makes it run in O(n.log(n)). However there is only one case when the heapify may run in log(n), and that is when it starts from the root, so it’s not absolutely true that building the heap runs in n.log(n).

Indeed heapify depend on the level it has been started. It doesn’t run for the last ceil(n/2) items and it runs in O(1) for another 2^h-1. Thus in practice we can build a heap in O(n).

Once we have the heap built, the only thing to do is to extract its first element and rebuild – heapify from the first item. This makes the sorting algorithm run in O(n.log(n)) – just like quicksort and mergesort.

Application

As I said in the beginning of this post quicksort is often the fastest general purpose algorithm in practice. This makes both merge sort and heapsort not so popular. However heapsort introduces an interesting data structure which can help us in many other cases.

It’s initially used to implement priority queues. What is great about a heap is that after we build it, which we know how to do in linear time, we can extract the greatest value – thus taking the highest priority task. Then with rebuilding the heap we can extract the next priority and so on, without fully sorting the array.

This makes the heapsort the only sorting algorithm that can sort the first k items out of a set of n items without sorting the whole set.

Indeed let’s say we have a set of positive integers and we’d like to get the biggest sum out of three items. Obviously we can sort the array and take the greatest three numbers, but this will cost us n.log(n) time, while using heapsort we can do it much faster! And all this without extra space – in place!

We know that finding the minimum in a list of integers is a fairly simple task, but what about finding the i-th smallest element? Then the task isn’t that trivial and we have to think for a different approach.

First of all there are some very basic and intuitive approaches. Since finding the minimum is so easy, can we just find the minimum, than exclude it from the list and then search the minimum again until we find the i-th smallest element.

That is a pure brute-force-like algorithm and it is extremely slow. In this case if we’re looking for the 99-th smallest element into an array of 100 items it will be quite inefficient. In other words this isn’t the best approach.

Another fairly intuitive approach is to sort the list in first place and then search the i-th element.

This is better than the our first attempt because we’ll need the time to sort the array and then search (in linear time) the i-th element.

In this case we need to find out which of the sorting algorithms we will use. Will it be merge sort (with constant O(n.lg(n)) complexity) or quicksort (with O(n²) in the worst case, but O(n.lg(n)) average complexity) or bubble sort (O(n^n) in the best-case scenario) it’s a developer choice.

However there is one very clever and yet more efficient approach, based on some observations.

Overview

If we’re looking for the i-th element and we decided that the list must be sorted first, we don’t need to fully sort it in order to find the desired element.

In case the list is sorted it’s easy to find which is the i-th element. However if the i-th element is in its place, the only thing we need to know is that the items on the left side of the i-th element are smaller and the items on the right side are greater. We don’t need the left and the right side ordered.

In the other hand this approach looks very much like quicksort. There during the sorting process we put the items smaller than the “pivot” on its left and the items greater than the pivot on its right. After that partitioning we executed quicksort on the left and on the right sub-lists.

Here the approach is similar with very small changes. First we choose a pivot. Then we make two partitions of the list – one left sub-list with all the elements with smaller values than the pivot and one right sub-list with all the elements with a greater value than the pivot.

Now we check the length of the left sub-list. If it is greater than i we continue recursively with the left sub-list and again we’re searching for the i-th element.

In case the length of the left sub-list is smaller than i, we continue with the right sub-list. However this time we don’t search for the i-th element, but for the i – length(LEFT).

Implementation

The following implementation is in PHP. It’s important to note that at each step we need two non-empty sub-list. That is why we take the pivot (by extracting the last item of the list) and then making two sub-lists. In case one of the sub-lists is empty we append the pivot in it. Thus we’re always partitioning the list into two non-empty sub-lists.

$list = array(3,4,5,7,8,2,5,6,9,0,1);

function partition($list, $pivot)
{
	$left = $right = array();
	
	$len = count($list);
	for ($i = 0; $i < $len; $i++) {
		if ($list[$i] <= $pivot) {
			$left[] = $list[$i];
		} else {
			$right[] = $list[$i];
		}
	}

	if (count($left) == 0) {
		$left[] = $pivot;
	} else {
		$right[] = $pivot;
	} 
	
	return array($left, $right);
}

function order_statistic($list, $i)
{
	if (count($list) == 1) {
		return $list[0];
	}
	
	// ceate a non empty partitions
	// extract the pivot from the list and
	// in case one of the sub-lists is empty
	// add the pivot there!
	$pivot = array_pop($list);
	list($left, $right) = partition($list, $pivot);
	
	if (count($left) >= $i) {
		return order_statistic($left, $i);
	} else {
		return order_statistic($right, $i - count($left));
	}
}

// 4
echo order_statistic($list, 5);

Application

Finding the minimum and maximum is easy, however sometimes we don’t search for them, but for the second, third or i-th smallest element. Then our task becomes a bit more difficult. This algorithm can be useful in many practical cases and shows us how different kind of algorithms may be related – exactly as this algorithm is related to quicksort in its principles.

1. Which string searching algorithm is faster?

• Morris-Pratt correct answer (ref)
• Brute force
• Rabin-Karp

2. Can you use radix sort for sorting floats?

• Yes
• No correct answer (ref)

3. Quicksort needs additional memory space?

• Yes
• No
• Only in iterative implementation correct answer (ref)
• Only in recursive implementation

4. In the worst case scenario which is slower?

• Quicksort
• Bubble sort
• They are equally slow correct answer (ref)

5. Is merge sort faster than quicksort in general?

• Yes, its complexity is O(n.log(n)) always!
• No, in practice quicksort is often faster than merge sort correct answer (ref)

Algorithms always depend on the input. We saw that general purpose sorting algorithms as insertion sort, bubble sort and quicksort can be very efficient in some cases and inefficient in other. Indeed insertion and bubble sort are considered slow, with best-case complexity of O(n²), but they are quite effective when the input is fairly sorted. Thus when you have a sorted array and you add some “new” values to the array you can sort it quite effectively with insertion sort. On the other hand quicksort is considered one of the best general purpose sorting algorithms, but while it’s a great algorithm when the data is randomized it’s practically as slow as bubble sort when the input is almost or fully sorted.

Now we see that depending on the input algorithms may be effective or not. For almost sorted input insertion sort may be preferred instead of quicksort, which in general is a faster algorithm.

Just because the input is so important for an algorithm efficiency we may ask are there any sorting algorithms that are faster than O(n.log(n)), which is the average-case complexity for merge sort and quicksort. And the answer is yes there are faster, linear complexity algorithms, that can sort data faster than quicksort, merge sort and heapsort. But there are some constraints!

Everything sounds great but the thing is that we can’t sort any particular data with linear complexity, so the question is what rules the input must follow in order to be sorted in linear time.

Such an algorithm that is capable of sorting data in linear O(n) time is radix sort and the domain of the input is restricted – it must consist only of integers.

Overview

Let’s say we have an array of integers which is not sorted. Just because it consists only of integers and because array keys are integers in programming languages we can implement radix sort.

First for each value of the input array we put the value of “1” on the key-th place of the temporary array as explained on the following diagram.

Implementation

Implementing radix sort is very easy in fact, which is great. The thing is that old-school programming languages weren’t so flexible and we needed to initialize the entire temporary array. That leads to another problem – we must know the interval of values from the input. Fortunately nowadays programming languages and libraries are more flexible so we can initialize our temporary array even if we don’t know the interval of input values, as on the example bellow. PHP is somewhere in the middle – it’s flexible enough to build-up arrays in the memory without knowing their size in advance, but we still must ksort them.

$list = array(4, 3, 5, 9, 7, 2, 4, 1, 6, 5);
 
function radix_sort($input)
{
    $temp = $output = array();
	$len = count($input);
 
    for ($i = 0; $i < $len; $i++) {
		$temp[$input[$i]] = ($temp[$input[$i]] > 0) 
			? ++$temp[$input[$i]]
			: 1;
    }
    
    ksort($temp);
    
    foreach ($temp as $key => $val) {
		if ($val == 1) {
			$output[] = $key; 
		} else {
			while ($val--) {
				$output[] = $key;
			}
        }
    }
    
    return $output;
}
 
// 1, 2, 3, 4, 4, 5, 5, 6, 7, 9
print_r(radix_sort($list));

The problem is that PHP needs ksort – which is completely foolish as we’re trying to sort an array using “another” sorting method, but to overcome this you must know the interval of values in advance and initialize a temporary array with 0s, as on the example bellow.

define(MIN, 1);
define(MAX, 9);
$list = array(4, 3, 5, 9, 7, 2, 4, 1, 6, 5);

function radix_sort(&$input)
{
    $temp = array();
	$len = count($input);
 
	// initialize with 0s
    $temp = array_fill(MIN, MAX-MIN+1, 0);
    
    foreach ($input as $key => $val) {
    	$temp[$val]++;
    }
    
    $input = array();
    foreach ($temp as $key => $val) {
	if ($val == 1) {
		$input[] = $key;
	} else {
		while ($val--) {
			$input[] = $key;
		}
	}
    }
}

// 4, 3, 5, 9, 7, 2, 4, 1, 6, 5
var_dump($list);

radix_sort(&$list);

// 1, 2, 3, 4, 5, 5, 6, 7, 8, 9
var_dump($list);

Here the input is modified during the sorting process and it’s used as result.

Complexity

The complexity of radix sort is linear, which in terms of omega means O(n). That is a great benefit in performance compared to O(n.log(n)) or even worse with O(n²) as we can see on the following chart.

Why using radix sort

1. It’s fast

Radix sort is very fast compared to other sorting algorithms as we saw on the diagram above. This algorithm is very useful in practice because in practice we often sort sets of integers.

2. It’s easy to understand and implement

Even a beginner can understand and implement radix sort, which is great. You need no more than few loops to implement it.

Why NOT using radix sort

1. Works only with integers

If you’re not sure about the input better do not use radix sort. We may think that our input consists only of integers and we can go for radix sort, but what if in the future someone passes floats or strings to our routine.

2. Requires additional space

Radix sort needs additional space – at least as much as the input.

Final Words

Radix sort is restricted by the input’s domain, but I must say that in practice there are tons of cases where only integers are sorted. This is when we get some data from the db based on primary keys – typically primary in database tables are integers as well. So practically there are lots of cases of sorting integers, so radix sort may be one very, very useful algorithm and it is so cool that it is also easy to implement.

When it comes to sorting items by comparing them merge sort is one very natural approach. It is natural, because simply divides the list into two equal sub-lists then sort these two partitions applying the same rule. That is a typical divide and conquer algorithm and it just follows the intuitive approach of speeding up the sorting process by reducing the number of comparisons. However there are other “divide and conquer” sorting algorithms that do not follow the merge sort scheme, while they have practically the same success. Such an algorithm is quicksort.

Overview

Back in 1960 C. A. R. Hoare comes with a brilliant sorting algorithm. In general quicksort consists of some very simple steps. First we’ve to choose an element from the list (called a pivot) then we must put all the elements with value less than the pivot on the left side of the pivot and all the items with value greater than the pivot on its right side. After that we must repeat these steps for the left and the right sub-lists. That is quicksort! Simple and elegant!

It is a pure divide and conquer approach as merge sort, but while merge sort’s tricky part was merging the sorted sub-lists, in quicksort there are other things to consider.

First of all obviously the choice of a pivot is the bottleneck. Indeed it all depends on that pivot. Imagine that you choose the greatest value from the list – than you’ve to put all the other items of the list into the “left” sub-list. If you do that on each step you’ll practically go into the worst scenario and that is no good. The thing is that in the worst case quicksort is not so effective and it’s practically as slow as bubble sort and insertion sort. The good thing is that in practice with randomly generated lists there is not a high possibility to go into the worst case of quicksort.

Choosing a pivot

Of course the best pivot is the middle element from the list. Thus the list will be divided into two fairly equal sub-lists. The problem is that there’s not an easy way to get the middle element from a list and this will slow down the algorithm. So typically we can get for a pivot the first or the last item of the list.

After choosing a pivot the rest is simple. Put every item with a greater value on the right and every item with a lesser value on the left. Then we must sort the left and right sub-lists just as we did with the initial list.

It’s clear that with this algorithm naturally we’re going into a recursive solution. Typically every divide and conquer approach is easy to implement with recursion. But because recursion can be heavy, there is an iterative approach.

Implementation

As I said above recursive approach is something very natural for quicksort as it follows the divide and conquer principles. On each step we divide the list in two and we pass those sub-lists to our recursive function. But recursion is dangerous sometimes, so an iterative approach is also available. Typically iterative approaches “model” recursion with extra memory and a model of a stack, which is our case. Here we have two examples of quicksort – recursive and iterative in PHP. Let’s go first with the recursion.

Recursive Quicksort

$list = array(5,3,9,8,7,2,4,1,6,5);
 
// recursive
function quicksort($array)
{
	if (count($array) == 0) {
    	return array();
	}
 
	$pivot = $array[0];
	$left = $right = array();
 
	for ($i = 1; $i < count($array); $i++) {
		if ($array[$i] < $pivot) {
			$left[] = $array[$i];
		} else {
			$right[] = $array[$i];
		}
	}
	
	return array_merge(quicksort($left), array($pivot), quicksort($right));
}

// 1, 2, 3, 4, 5, 5, 6, 7, 8, 9
print_r(quicksort($list));

Iterative Quicksort

$list = array(5,3,9,8,7,2,4,1,6,5);

// iterative
function quicksort_iterative($array)
{
    $stack = array($array);
    $sorted = array();
 
    while (count($stack) > 0) {
 
        $temp = array_pop($stack);
 
        if (count($temp) == 1) {
            $sorted[] = $temp[0];
            continue;
        }
 
        $pivot = $temp[0];
        $left = $right = array();
 
        for ($i = 1; $i < count($temp); $i++) {
            if ($pivot > $temp[$i]) {
                $left[] = $temp[$i];
            } else {
                $right[] = $temp[$i];
            }
        }
 
        $left[] = $pivot;
 
        if (count($right))
            array_push($stack, $right);
        if (count($left))
            array_push($stack, $left);
    }
 
    return $sorted;
}

// 1, 2, 3, 4, 5, 5, 6, 7, 8, 9
print_r(quicksort_iterative($list));

Complexity

The complexity of quicksort in the average case is O(n*log(n)) - same as Merge sort. The problem is that in the worst case it is O(n²) - same as bubble sort. Obviously the worst case is when we have an already sorted list, and we constantly take for a pivot the last element of the list. But we should consider that in practice we don’t quite use sorted lists that we have to sort again, right?

Application

Quicksort is a great sorting algorithm and developers often go for it, but let's see some pros and cons of it.

Why using quicksort

1. Recursive implementation is easy
2. In general its speed is same as merge sort - O(n*log(n))
3. Elegant solution with no tricky merging as merge sort

Why not using quicksort

1. As slow as bubble sort in the worst case!
2. Iterative implementation isn't easy
3. There are faster algorithms for some sets of data types

Quicksort is beautiful because of the elegant idea behind its principles. Indeed if you have two sorted lists one with items with a greater value from a given value and the other with items smaller form that given value you can simply concatenate them and you can be sure that the resulting list will be sorted with no need of special merge.

In fact quicksort is a very elegant general purpose sorting algorithm and every developer should be familiar with its principles.

Basically sorting algorithms can be divided into two main groups. Such based on comparisons and such that are not. I already posted about some of the algorithms of the first group. Insertion sort, bubble sort and Shell sort are based on the comparison model. The problem with these three algorithms is that their complexity is O(n²) so they are very slow.

So is it possible to sort a list of items by comparing their items faster than O(n²)? The answer is yes and here’s how we can do it.

The nature of those three algorithms mentioned above is that we almost compared each two items from initial list.

This, of course, is not the best approach and we don’t need to do that. Instead we can try to divide the list into smaller lists and then sort them. After sorting the smaller lists, which is supposed to be easier than sorting the entire initial list, we can try to merge the result into one sorted list. This technique is typically known as “divide and conquer”.

Normally if a problem is too difficult to solve, we can try to break it apart into smaller sub-sets of this problem and try to solve them. Then somehow we can merge the results of the solved problems.

Overview

Merge sort is a comparison model sorting algorithm based on the “divide and conquer” principle. So far so good, so let’s say we have a very large list of data, which we want to sort. Obviously it will be better if we divide the list into two sub-lists with equal length and then sort them. If they remain too large, we can continue breaking them down until we get to something very easy to sort as shown on the diagram bellow.

The thing is that on some step of the algorithm we have two sorted lists and the tricky part is to merge them. However this is not so difficult.
We can start comparing the first items of the lists and than we can pop the smaller of them both and put it into a new list containing the merged (sorted) array.

Implementation

The good news is that this algorithm is fast, but not so difficult to implement and that sounds quite good from a developer’s point of view. Here’s the implementation in PHP. Note that every algorithm that follows the divide and conquer principles can be easily implemented in a recursive solution. However recursion can be bitter so you can go for a iterative solution. Typically recursion is “replaced” by additional memory space in iterative solutions. Here’s a recursive version of merge sort.

$input = array(6, 5, 3, 1, 8, 7, 2, 4);

function merge_sort($arr)  
{  
	if (count($arr) <= 1) {
		return $arr;  
	}

	$left = array_slice($arr, 0, (int)(count($arr)/2));  
	$right = array_slice($arr, (int)(count($arr)/2));  
	
	$left = merge_sort($left);  
	$right = merge_sort($right);  
	
	$output = merge($left, $right);  

	return $output;  
}  
      
      
function merge($left, $right)  
{  
	$result = array();  

	while (count($left) > 0 && count($right) > 0) {  
		if ($left[0] <= $right[0]) {  
			array_push($result, array_shift($left));  
		} else {  
			array_push($result, array_shift($right));  
		}  
	}  
      
	array_splice($result, count($result), 0, $left);  
	array_splice($result, count($result), 0, $right);  

	return $result;  
}  

// 1, 2, 3, 4, 5, 6, 7, 8
$output = merge_sort($input);

Complexity

It’s great that the complexity of merge sort is O(n*log(n)) even in the worst case! Note that even quicksort’s complexity can be O(n²) in the worst case. So we can be sure that merge sort is very stable no matter the input.

Two reasons why merge sort is useful

1. Fast no matter the input

Merge sort is a great sorting algorithm mainly because it’s very fast and stable. It’s complexity is the same even in the worst case and it is O(n*log(n)). Note that even quicksort's complexity is O(n²) in the worst case, which for n = 20 is about 4.6 times slower!

2. Easy implementation

Another cool reason is that merge sort is easy to implement. Indeed most of the developer consider something fast to be difficult to implement, but that's not the case of merge sort.

Three reasons why merge sort is not useful

1. Slower than non-comparison based algorithms

Merge sort is however based on the comparison model and as such can be slower than algorithms non-based on comparisons that can sort data in linear time. Of course, this depends on the input data, so we must be careful for the input.

2. Difficult to implement for beginners

Although I don’t think this can be the main reason why not to use merge sort some people say that it can be difficult to implement for beginners, especially the merge part of the algorithm.

3. Slower than insertion and bubble sort for nearly sorted input

Again it is very important to know the input data. Indeed if the input is nearly sorted the insertion sort or bubble sort can be faster. Note that in the best case insertion and bubble sort complexity is O(n), while merge sort's best case is O(n*log(n)).

As a conclusion I can say that merge sort is practically one of the best sorting algorithms because it's easy to implement and fast, so it must be considered by every developer!

Insertion sort is a great algorithm, because it’s very intuitive and it is easy to implement, but the problem is that it makes many exchanges for each “light” element in order to put it on the right place. Thus “light” elements at the end of the list may slow down the performance of insertion sort a lot. That is why in 1959 Donald Shell proposed an algorithm that tries to overcome this problem by comparing items of the list that lie far apart.

In the other hand it is obvious that by comparing items that lie apart the list can’t be sorted in one pass as insertion sort. That is why on each pass we should use a fixed gap between the items, then decrease the value on every consecutive iteration.

However it is intuitively clear that Shell sort may need even more comparisons than insertion sort. Then why should we use it?

The thing is that insertion sort is not an effective sorting algorithm at all, but in some cases, when the list is almost sorted it can be quite useful. Here’s the answer of the question above. With Shell sort once the list is sorted for gap = i, it is sorted for every gap = j, where j < i, and this is its main advantage. [caption id="attachment_2792" align="alignnone" width="620" caption="Shell sort can make less exchanges than insertion sort."][/caption]

How to choose gap size

Not a cool thing about Shell sort is that we’ve to choose “the perfect” gap sequence for our list. However this is not an easy task, because it depends a lot of the input data. The good news is that there are some gap sequences proved to be working well in the general cases.

Shell Sequence

Donald Shell proposes a sequence that follows the formula FLOOR(N/2^k), then for N = 1000, we get the following sequence: [500, 250, 125, 62, 31, 15, 7, 3, 1]

Pratt Sequence

Pratt proposes another sequence that’s growing with a slower pace than the Shell’s sequence. He proposes successive numbers of the form 2^p3^q or [1, 2, 3, 4, 6, 8, 9, 12, …].

Knuth Sequence

Knuth in other hand proposes his own sequence following the formula (3^k – 1) / 2 or [1, 4, 14, 40, 121, …]

Of course there are many other gap sequences, proposed by various developers and researchers, but the problem is that the effectiveness of the algorithm strongly depends on the input data. But before taking a look to the complexity of Shell sort, let’s see first its implementation.

Implementation

Here’s a Shell sort implementation on PHP using the Pratt gap sequence. The thing is that for this data set other gap sequences may appear to be better solution.

$input = array(6, 5, 3, 1, 8, 7, 2, 4);

function shell_sort($arr)
{
	$gaps = array(1, 2, 3, 4, 6);
	$gap  = array_pop($gaps);
	$len  = count($arr);
	
	while($gap > 0)
	{
		for($i = $gap; $i < $len; $i++) {
			
			$temp = $arr[$i];
			$j = $i;
			
			while($j >= $gap && $arr[$j - $gap] > $temp) {
				$arr[$j] = $arr[$j - $gap];
				$j -= $gap;
			}
			
			$arr[$j] = $temp;
		}
		
		$gap = array_pop($gaps);
	}
	
	return $arr;
}

// 1, 2, 3, 4, 5, 6, 7, 8
shell_sort($input);

It’s easy to change this code in order to work with Shell sequence.

$input = array(6, 5, 3, 1, 8, 7, 2, 4);

function shell_sort($arr)
{
        $len  = count($arr);
	$gap  = floor($len/2);
	
	while($gap > 0)
	{
		for($i = $gap; $i < $len; $i++) {
			
			$temp = $arr[$i];
			$j = $i;
			
			while($j >= $gap && $arr[$j - $gap] > $temp) {
				$arr[$j] = $arr[$j - $gap];
				$j -= $gap;
			}
			
			$arr[$j] = $temp;
		}
		
		$gap = floor($gap/2);
	}
	
	return $arr;
}

// 1, 2, 3, 4, 5, 6, 7, 8
shell_sort($input);

Complexity

Yet again we can’t determine the exact complexity of this algorithm, because it depends on the gap sequence. However we may say what is the complexity of Shell sort with the sequences of Knuth, Pratt and Donald Shell. For the Shell’s sequence the complexity is O(n²), while for the Pratt’s sequence it is O(n*log²(n)). The best approach is the Knuth sequence where the complexity is O(n^3/2), as you can see on the diagram bellow.

Application

Well, as insertion sort and bubble sort, Shell sort is not very effective compared to quicksort or merge sort. The good thing is that it is quite easy to implement (not easier than insertion sort), but in general it should be avoided for large data sets. Perhaps the main advantage of Shell sort is that the list can be sorted for a gap greater than 1 and thus making less exchanges than insertion sort.

It’s weird that bubble sort is the most famous sorting algorithm in practice since it is one of the worst approaches for data sorting. Why is bubble sort so famous? Perhaps because of its exotic name or because it is so easy to implement. First let’s take a look on its nature.

Bubble sort consists of comparing each pair of adjacent items. Then one of those two items is considered smaller (lighter) and if the lighter element is on the right side of its neighbour, they swap places. Thus the lightest element bubbles to the surface and at the end of each iteration it appears on the top. I’ll try to explain this simple principle with some pictures.

1. Each two adjacent elements are compared

Here “2” appears to be less than “4”, so it is considered lighter and it continues to bubble to the surface (the front of the array).

2. Swap with heavier elements

On his way to the surface the currently lightest item meets a heavier element. Then they swap places.

3. Move forward and swap with each heavier item

The problem with bubble sort is that you may have to swap a lot of elements.

4. If there is a lighter element, then this item begins to bubble to the surface

If the currently lightest element meets another item that is lighter, then the newest currently lightest element starts to bubble to the top.

5. Finally the lightest element is on its place

At the end of each iteration we can be sure that the lightest element is on the right place – at the beginning of the list.

The problem is that this algorithm needs a tremendous number of comaprisons and as we know already this can be slow.

We can easily see how ineffective bubble sort is. Now the question remains – why is it so famous? Maybe indeed the answer lies in the simplicity of its implementation. Let’s see how to implement bubble sort.

Implementation

Implementing bubble sort is easy. The question is how easy? Well, obviously after understanding the principles of this algorithm every developer, even a beginner, can implement it. Here’s a PHP implementation of bubble sort.

$input = array(6, 5, 3, 1, 8, 7, 2, 4);

function bubble_sort($arr)
{
	$length = count($arr);
	
	for ($i = 0; $i < $length; $i++) {
		for ($j = $length-1; $j > $i; $j--) {
			if ($arr[$j] < $arr[$j-1]) {
				$t = $arr[$j];
				$arr[$j] = $arr[$j-1];
				$arr[$j-1] = $t;
			}
		}
	}
	
	return $arr;
}

// 1, 2, 3, 4, 5, 6, 7, 8
$output = bubble_sort($input);

Clearly the implementation consists of few lines of code and two nested loops.

Complexity: Where's Bubble Sort Compared to Other Sorting Algorithms

Compared to other sorting algorithm, bubble sort is really slow. Indeed the complexity of this algoritm is O(n²) which can't be worse. It's weird that the most well known sorting algorithm is the slowest one.

Even for small values of n, the number of comparisons and swaps can be tremendous.

Another problem is that most of the languages (libraries) have built-in sorting functions, that they don't make use of bubble sort and are faster for sure. So why a developer should implement bubble sort at all?

Application: 3 Cool Reasons To Use Bubble Sort

We saw that bubble sort is slow and ineffective, yet it is used in practice. Why? Is there any reason to use this slow, ineffective algorithm with weird name? Yes and here are some of them that might be helpful for any developer.

1. It is easy to implement

Definitely bubble sort is easier to implement than other "complex" sorting algorithms as quicksort. Bubble sort is easy to remember and easy to code and that's great instead of learning and remembering tons of code.

2. Because the library can't help

Let's say you work with JavaScript. Great, there you get array.sort() which can help for this: [3, 1, 2].sort(). But what would happen if you'd rather like to sort more "complex" structures like ... some DOM nodes. Here we have three LI nodes and we want to sort them in some order. Obviously you can't compare them with the "<" operator, so we've to come up with some custom solution.

Click here to sort the list

node 3
node 1
node 2

Here sort() can’t help us and we’ve to code our own function. However we have only few elements (three in our case). Why not using bubble sort?

3. The list is almost sorted

One of the problems with bubble sort is that it consists of too much swapping, but what if we know that the list is almost sorted?

// almost sorted
[1, 2, 4, 3, 5]

We have to swap only 3 with 4.

Note that in the best case bubble sort’s complexity is O(n) – faster than quicksort’s best case!