What’s the fastest way to sort the following sequence [9, 3, 0, 5, 4, 1, 2, 6, 8, 7]? Well, the question is a bit tricky since the input is somehow “predefined”. First of all we have only integers, and fortunately they are all different. That’s great and we know that in practice it’s almost impossible to count on such lucky coincidence. However here we can sort the sequence very quickly.

First of all we can pass through all these integers and by using an auxiliary array we can just put them at their corresponding index. We know in advance that that is going to work really well, because they are all different.

There is only one major problem in this solution. That’s because we assume all the integers are different. If not – we can just put all them in one single corresponding index.

That is why we can use bucket sort.

Overview

Bucket sort it’s the perfect sorting algorithm for the sequence above. We must know in advance that the integers are fairly well distributed over an interval (i, j). Then we can divide this interval in N equal sub-intervals (or buckets). We’ll put each number in its corresponding bucket. Finally for every bucket that contains more than one number we’ll use some linear sorting algorithm.

The thing is that we know that the integers are well distributed, thus we expect that there won’t be many buckets with more than one number inside.

That is why the sequence [1, 2, 3, 2, 1, 2, 3, 1] won’t be sorted faster than [4, 3, 1, 2, 9, 5, 4, 8].

Pseudo Code

1. Let n be the length of the input list L;
2. For each element i from L
   2.1. If B[i] is not empty
      2.1.1. Put A[i] into B[i] using insertion sort;
      2.1.2. Else B[i] := A[i] 
3. Concatenate B[i .. n] into one sorted list;

Complexity

The complexity of bucket sort isn’t constant depending on the input. However in the average case the complexity of the algorithm is O(n + k) where n is the length of the input sequence, while k is the number of buckets.

The problem is that its worst-case performance is O(n^2) which makes it as slow as bubble sort.

Application

As the other two linear time sorting algorithms (radix sort and counting sort) bucket sort depends so much on the input. The main thing we should be aware of is the way the input data is dispersed over an interval.

Another crucial thing is the number of buckets that can dramatically improve or worse the performance of the algorithm.

This makes bucket sort ideal in cases we know in advance that the input is well dispersed.

The first question when we see the phrase “sorting in linear time” should be – where’s the catch? Indeed there’s a catch and the thing is that we can’t sort just anything in linear time. Most of the time we can speak on sorting integers in linear time, but as we can see later this is not the only case.

Since we speak about integers, we can think of a faster sorting algorithm than usual. Such an algorithm is the counting sort, which can be very fast in some cases, but also very slow in others, so it can be used carefully. Another linear time sorting algorithm is radix sort.

Introduction

Count sort is absolutely brilliant and easy to implement. In case we sort integers in the range [n, m] on the first pass we just initialize a zero filled array with length m-n. Than on the second pass we “count” the occurrence of each integer. On the third pass we just sort the integers with an ease.

However we have some problems with that algorithm. What if we have only few items to sort that are very far from each other like [2, 1, 10000000, 2]. This will result in a very large unused data. So we need a dense integer sequence. This is important because we must know in advance the nature of the sequence which is rarely sure.

That’s why we need to use another linear time sorting algorithm for integers that doesn’t have this disadvantage. Such an algorithm is the radix sort.

Overview

The idea behind the radix sort is simple. We must look at our “integer” sequence as a string sequence. OK, to become clearer let me give you an example. Our sequence is [12, 2, 23, 33, 22]. First we take the leftmost digit of each number. Thus we must compare [_2, 2, _3, _3, _2]. Clearly we can assume that since the second number “2” is only a one digit number we can fill it up with a leading “0”, to become 02 or _2 in our example: [_2, _2, _3, _3, _2]. Now we sort this sequence with a stable sort algorithm.

What is a Stable Sort Algorithm

A stable sort algorithm is an algorithm that sorts a list by preserving the positions of the elements in case they are equal. In terms of PHP this means that:

array(0 => 12, 1=> 13, 2 => 12); 

Will be sorted as follows:

array(0 => 12, 2 => 12, 1 => 13);

Thus the third element becomes second following the first element. Note that the third and the first element are equal, but the third appears later in the sequence so it remains later in the sorted sequence.

In the radix sort example, we need a stable sort algorithm, because we need to worry about only one position of digit we explore.

So what happens in our example after we sort the sequence?

As we can see we’re far from a sorted sequence, but what if we proceed with the next “position” – the decimal digit?

Than we end up with this:

Now we have a sorted sequence, so let’s summarize the algorithm in a short pseudo code.

Pseudo Code

The simple approach behind the radix sort algorithm can be described as pseudo code, assuming that we’re sorting decimal integers.

1. For each digit at position 10^0 to 10^n
1.1. Sort the numbers by this digit using a stable sort algorithm;

The thing is that here we talk about decimal, but actually this algorithm can be applied equally on any numeric systems. That is why it’s called “radix” sort.

Thus we can sort binary numbers, hexadecimals etc.

It’s important to note that this algorithm can be also used to sort strings alphabetically.

[ABC, BBC, ABA, AC]
[__C, __C, __A, __C] => [ABA, ABC, BBC, AC]
[_B_, _B_, _B_, _A_] => [AC, ABA, ABC, BBC]
[___, A__, A__, B__] => [AC, ABA, ABC, BBC]

That is simply correct because we can assume that our alphabet is another 27 digit numeric system (in case of the Latin alphabet).

Complexity

As I said in the beginning radix sort is a linear time sorting algorithm. Let’s see why. First we depend on the numeric system. Let’s assume we have a decimal numeric system – then we have N passes sorting 10 digits which is simply 10*N. In case of K digit numeric system our algorithm will be O(K*N) which is linear.

However you must note that in case we sort N numbers in an N digit numeric system the complexity will become O(N^2)!

We must also remember that in order to implement radix sort and a supporting stable sort algorithm we need an extra space.

Application

Sorting integers can be faster than sorting just anything, so any time we need to implement a sorting algorithm we must carefully investigate the input data. And that’s also the big disadvantage of this algorithm – we must know the input in advance, which is rarely the case.

Insertion sort is a great algorithm, because it’s very intuitive and it is easy to implement, but the problem is that it makes many exchanges for each “light” element in order to put it on the right place. Thus “light” elements at the end of the list may slow down the performance of insertion sort a lot. That is why in 1959 Donald Shell proposed an algorithm that tries to overcome this problem by comparing items of the list that lie far apart.

In the other hand it is obvious that by comparing items that lie apart the list can’t be sorted in one pass as insertion sort. That is why on each pass we should use a fixed gap between the items, then decrease the value on every consecutive iteration.

However it is intuitively clear that Shell sort may need even more comparisons than insertion sort. Then why should we use it?

The thing is that insertion sort is not an effective sorting algorithm at all, but in some cases, when the list is almost sorted it can be quite useful. Here’s the answer of the question above. With Shell sort once the list is sorted for gap = i, it is sorted for every gap = j, where j < i, and this is its main advantage. [caption id="attachment_2792" align="alignnone" width="620" caption="Shell sort can make less exchanges than insertion sort."][/caption]

How to choose gap size

Not a cool thing about Shell sort is that we’ve to choose “the perfect” gap sequence for our list. However this is not an easy task, because it depends a lot of the input data. The good news is that there are some gap sequences proved to be working well in the general cases.

Shell Sequence

Donald Shell proposes a sequence that follows the formula FLOOR(N/2^k), then for N = 1000, we get the following sequence: [500, 250, 125, 62, 31, 15, 7, 3, 1]

Pratt Sequence

Pratt proposes another sequence that’s growing with a slower pace than the Shell’s sequence. He proposes successive numbers of the form 2^p3^q or [1, 2, 3, 4, 6, 8, 9, 12, …].

Knuth Sequence

Knuth in other hand proposes his own sequence following the formula (3^k – 1) / 2 or [1, 4, 14, 40, 121, …]

Of course there are many other gap sequences, proposed by various developers and researchers, but the problem is that the effectiveness of the algorithm strongly depends on the input data. But before taking a look to the complexity of Shell sort, let’s see first its implementation.

Implementation

Here’s a Shell sort implementation on PHP using the Pratt gap sequence. The thing is that for this data set other gap sequences may appear to be better solution.

$input = array(6, 5, 3, 1, 8, 7, 2, 4);

function shell_sort($arr)
{
	$gaps = array(1, 2, 3, 4, 6);
	$gap  = array_pop($gaps);
	$len  = count($arr);
	
	while($gap > 0)
	{
		for($i = $gap; $i < $len; $i++) {
			
			$temp = $arr[$i];
			$j = $i;
			
			while($j >= $gap && $arr[$j - $gap] > $temp) {
				$arr[$j] = $arr[$j - $gap];
				$j -= $gap;
			}
			
			$arr[$j] = $temp;
		}
		
		$gap = array_pop($gaps);
	}
	
	return $arr;
}

// 1, 2, 3, 4, 5, 6, 7, 8
shell_sort($input);

It’s easy to change this code in order to work with Shell sequence.

$input = array(6, 5, 3, 1, 8, 7, 2, 4);

function shell_sort($arr)
{
        $len  = count($arr);
	$gap  = floor($len/2);
	
	while($gap > 0)
	{
		for($i = $gap; $i < $len; $i++) {
			
			$temp = $arr[$i];
			$j = $i;
			
			while($j >= $gap && $arr[$j - $gap] > $temp) {
				$arr[$j] = $arr[$j - $gap];
				$j -= $gap;
			}
			
			$arr[$j] = $temp;
		}
		
		$gap = floor($gap/2);
	}
	
	return $arr;
}

// 1, 2, 3, 4, 5, 6, 7, 8
shell_sort($input);

Complexity

Yet again we can’t determine the exact complexity of this algorithm, because it depends on the gap sequence. However we may say what is the complexity of Shell sort with the sequences of Knuth, Pratt and Donald Shell. For the Shell’s sequence the complexity is O(n²), while for the Pratt’s sequence it is O(n*log²(n)). The best approach is the Knuth sequence where the complexity is O(n^3/2), as you can see on the diagram bellow.

Application

Well, as insertion sort and bubble sort, Shell sort is not very effective compared to quicksort or merge sort. The good thing is that it is quite easy to implement (not easier than insertion sort), but in general it should be avoided for large data sets. Perhaps the main advantage of Shell sort is that the list can be sorted for a gap greater than 1 and thus making less exchanges than insertion sort.

It’s weird that bubble sort is the most famous sorting algorithm in practice since it is one of the worst approaches for data sorting. Why is bubble sort so famous? Perhaps because of its exotic name or because it is so easy to implement. First let’s take a look on its nature.

Bubble sort consists of comparing each pair of adjacent items. Then one of those two items is considered smaller (lighter) and if the lighter element is on the right side of its neighbour, they swap places. Thus the lightest element bubbles to the surface and at the end of each iteration it appears on the top. I’ll try to explain this simple principle with some pictures.

1. Each two adjacent elements are compared

Here “2” appears to be less than “4”, so it is considered lighter and it continues to bubble to the surface (the front of the array).

2. Swap with heavier elements

On his way to the surface the currently lightest item meets a heavier element. Then they swap places.

3. Move forward and swap with each heavier item

The problem with bubble sort is that you may have to swap a lot of elements.

4. If there is a lighter element, then this item begins to bubble to the surface

If the currently lightest element meets another item that is lighter, then the newest currently lightest element starts to bubble to the top.

5. Finally the lightest element is on its place

At the end of each iteration we can be sure that the lightest element is on the right place – at the beginning of the list.

The problem is that this algorithm needs a tremendous number of comaprisons and as we know already this can be slow.

We can easily see how ineffective bubble sort is. Now the question remains – why is it so famous? Maybe indeed the answer lies in the simplicity of its implementation. Let’s see how to implement bubble sort.

Implementation

Implementing bubble sort is easy. The question is how easy? Well, obviously after understanding the principles of this algorithm every developer, even a beginner, can implement it. Here’s a PHP implementation of bubble sort.

$input = array(6, 5, 3, 1, 8, 7, 2, 4);

function bubble_sort($arr)
{
	$length = count($arr);
	
	for ($i = 0; $i < $length; $i++) {
		for ($j = $length-1; $j > $i; $j--) {
			if ($arr[$j] < $arr[$j-1]) {
				$t = $arr[$j];
				$arr[$j] = $arr[$j-1];
				$arr[$j-1] = $t;
			}
		}
	}
	
	return $arr;
}

// 1, 2, 3, 4, 5, 6, 7, 8
$output = bubble_sort($input);

Clearly the implementation consists of few lines of code and two nested loops.

Complexity: Where's Bubble Sort Compared to Other Sorting Algorithms

Compared to other sorting algorithm, bubble sort is really slow. Indeed the complexity of this algoritm is O(n²) which can't be worse. It's weird that the most well known sorting algorithm is the slowest one.

Even for small values of n, the number of comparisons and swaps can be tremendous.

Another problem is that most of the languages (libraries) have built-in sorting functions, that they don't make use of bubble sort and are faster for sure. So why a developer should implement bubble sort at all?

Application: 3 Cool Reasons To Use Bubble Sort

We saw that bubble sort is slow and ineffective, yet it is used in practice. Why? Is there any reason to use this slow, ineffective algorithm with weird name? Yes and here are some of them that might be helpful for any developer.

1. It is easy to implement

Definitely bubble sort is easier to implement than other "complex" sorting algorithms as quicksort. Bubble sort is easy to remember and easy to code and that's great instead of learning and remembering tons of code.

2. Because the library can't help

Let's say you work with JavaScript. Great, there you get array.sort() which can help for this: [3, 1, 2].sort(). But what would happen if you'd rather like to sort more "complex" structures like ... some DOM nodes. Here we have three LI nodes and we want to sort them in some order. Obviously you can't compare them with the "<" operator, so we've to come up with some custom solution.

Click here to sort the list

node 3
node 1
node 2

Here sort() can’t help us and we’ve to code our own function. However we have only few elements (three in our case). Why not using bubble sort?

3. The list is almost sorted

One of the problems with bubble sort is that it consists of too much swapping, but what if we know that the list is almost sorted?

// almost sorted
[1, 2, 4, 3, 5]

We have to swap only 3 with 4.

Note that in the best case bubble sort’s complexity is O(n) – faster than quicksort’s best case!

This is one of the most slowest algorithms for sorting, but it’s extremely well known because of its easy to implement nature. However as I wrote past Fridays there are lots of sorting algorithms which are really fast, like the quicksort or mergesort. In the case of bubble sort the nature of the algorithm is described in its name. The smaller element goes to the top (beginning) of the array as a bubble goes to the top of the water.

There is a cool animation showing how bubble sort works in compare to the quick sort and you can practically see how slow is bubble sort because of all the comparing.

QuickSort vs. BubbleSort

Pseudo Code

Actually what I’d like to show you is how you can move from pseudo code to code in practice. Here’s the pseudo code from Wikipedia.

procedure bubbleSort( A : list of sortable items ) defined as:
  do
    swapped := false
    for each i in 0 to length(A) - 2 inclusive do:
      if A[i] > A[i+1] then
        swap( A[i], A[i+1] )
        swapped := true
      end if
    end for
  while swapped
end procedure

JavaScript Source

var a = [34, 203, 3, 746, 200, 984, 198, 764, 9];

function bubbleSort(a)
{
    var swapped;
    do {
        swapped = false;
        for (var i=0; i < a.length-1; i++) {
            if (a[i] > a[i+1]) {
                var temp = a[i];
                a[i] = a[i+1];
                a[i+1] = temp;
                swapped = true;
            }
        }
    } while (swapped);
}

bubbleSort(a);
console.log(a);

As a result you’ve a sorted array!