Relative encoding is another data compression algorithm. While run-length encoding, bitmap encoding and diagram and pattern substitution were trying to reduce repeating data, with relative encoding the goal is a bit different. Indeed run-length encoding was searching for long runs of repeating elements, while pattern substitution and bitmap encoding were trying to “map” where the repetitions happen to occur.

The only problem with these algorithms is that not always the input stream of data is constructed out of repeating elements. It is clear that if the input stream contains many repeating elements there must be some way of reducing them. However that doesn’t mean that we cannot compress data if there are no repetitions. It all depends on the data. Let’s say we have the following stream to compress.

1, 2, 3, 4, 5, 6, 7

We can hardly imagine how this stream of data can be compressed. The same problem may occur when trying to compress the alphabet. Indeed the alphabet letters the very base of the words so it is the minimal part for word construction and it’s hard to compress them.

Fortunately this isn’t true always. An algorithm that tryies to deal with non repeating data is relative encoding. Let’s see the following input stream – years from a given decade (the 90′s).

1991,1991,1999,1998,1991,1993,1992,1992

Here we have 39 characters and we can reduce them. A natural approach is to remove the leading “19” as we humans often do.

91,91,99,98,91,93,92,92

Now we have a shorter string, but we can go even further with keeping only the first year. All other years will as relative to this year.

91,0,8,7,0,2,1,1

Now the volume of transferred data is reduced a lot (from 39 to 16 – more than 50%). However there are some questions we need to answer first, because the stream wont be always formatted in such pretty way. How about the next character stream?

91,94,95,95,98,100,101,102,105,110

We see that the value 100 is somehow in the middle of the interval and it is handy to use it as a base value for the relative encoding. Thus the stream above will become:

-9,-6,-5,-5,-2,100,1,2,5,10

The problem is that we can’t decide which value will be the base value so easily. What if the data was dispersed in a different way.

96,97,98,99,100,101,102,103,999,1000,1001,1002

Now the value of “100” isn’t useful, because compressing the stream will get something like this:

-4,-3,-2,-1,100,1,2,3,899,900,901,902

To group the relative values around “some” base values will be far more handy.

(-4,-3,-2,-1,100,1,2,3)(-1,1000,1,2)

However to decide which value will be the base value isn’t that easy. Also the encoding format is not so trivial. In the other hand this type of encoding can be useful in som specific cases as we can see bellow.

Implementation

The implementation of this algorithm depends on the specific task and the format of the data stream. Assuming that we’ve to transfer the stream of years in JSON from a web server to a browser, here’s a short PHP snippet.

// JSON: [1991,1991,1999,1998,1999,1998,1995,1997,1994,1993]
$years = array(1991,1991,1999,1998,1999,1998,1995,1997,1994,1993);
 
function relative_encoding($input)
{
	$output = array();
	$inputLength = count($input);
 
	$base = $input[0];
 
	$output[] = $base;
 
	for ($i = 1; $i < $inputLength; $i++) {
		$output[] = $input[$i] - $base;
	}
 
	return $output;
}
 
// JSON: [1991,0,8,7,8,7,4,6,3,2]
echo json_encode(relative_encoding($years));

Application

This algorithm may be very useful in many cases, but here’s one of them. There are plenty of map applications around the web. Some products as Google Maps, Yahoo! Maps, Bing Maps are quite famous, while there are very useful open source projects as OpenStreetMap. The web sites using these apps are thousands.

A typical use case is to transfer lots of Geo coordinates from web server to a browser using JSON. Indeed any GEO point on Earth is relative to the point (0,0), which is located near the west coast of Africa, however on large zoom levels, when there are tons of markers we can transfer the information with relative encoding.

For instance the following diagram shows San Francisco with some markers on it. Their coordinates are be relative to the point (0,0) on Earth.

Map markers can be relative to the (0, 0) point on Earth, which can be sometimes useless.

Far more useful may be to encode those markers, relative to the center of the city, thus we can save some space.

Relative encoding can be useful for map markers on large zoom level!

However this type of compression can be tricky, for example when dragging the map and updating the marker array. In the other hand we must group markers if we have to load more than one city. That’s why we must be careful when implementing it. But in the other hand it can be very useful – for instance on initial load of the map we can reduce data and speed up the load time.

The thing is that with relative encoding we can save only changes to base value (data) – something like version control systems and thus reducing data transfer and load. Here’s a graphical example. In the first case on the diagram bellow we can see that each item is stored on its own. It doesn’t depend on the adjacent items and it can be completely independent of them.

However we can keep full info only for the first item and any other item will be relative to it, like on the diagram bellow.

Related posts:

1. Computer Algorithms: Data Compression with Run-length Encoding
2. Computer Algorithms: Data Compression with Bitmaps
3. Computer Algorithms: Data Compression with Diagram Encoding and Pattern Substitution

If you have read some preformance tests on JavaScript loops, you may have heard that “while” is faster than “for”. However the question is how faster is “while”? Here are some results, but first let’s take a look on the JavaScript code.

The for experiment

console.time('for');
for (var i = 0; i < 10000000; i++) {
	i / 2;
}
console.timeEnd('for');

The while experiment

console.time('while');
var i = 0;
while (i++ < 10000000) {
	i / 2;
}
console.timeEnd('while');

Note – these tests are performed and measured with Firebug on Firefox.

Results

It’s a fact, that you’ll get different results as many times as you run this snippet. It depends also on the enviroment and software/hardware specs. That is why I performed them 10 times and then I took the average value. Here are the values of my performance tests. Note that both for and while perform 10,000,000 iterations.

And the Winner Is

While is the winner with an average result of 83.5 milliseconds, while “for” result is 88 average milliseconds.

As the diagram bellow shows, the while loop is slightly faster. However we should be aware that these performance gains are significant for large number of iterations!

JavaScript Performance: for vs. while

Related posts:

1. Firebug’s console.time() accuracy
2. Profiling JavaScript with Firebug. console.profile() & console.time()!
3. PHP Performance: Bitwise Division

Two variants of run-length encoding are the diagram encoding and the pattern substitution algorithms. The diagram encoding is actually a very simple algorithm. Unlike run-length encoding, where the input stream must consists of many repeating elements, as “aaaaaaaa” for instance, which are very rare in a natural language, there are many so called “diagrams” in almost any natural language. In plain English there are some diagrams as “the”, “and”, “ing” (in the word “waiting” for example), “ a”, “ t”, “ e” and many doubled letters. Actually we can extend those diagrams by adding surrounding spaces. Thus we can encode not only “the”, but “ the “, which are 5 characters (2 spaces and 3 letters) with something shorter. In the other hand, as I said, in plain English there are two many doubled letters, which unfortunately aren’t something special for run-length encoding and the compression ratio will be small. Even worse the encoded text may happen to be longer than the input message. Let’s see some examples.

Let’s say we’ve to encode the message “successfully accomplished”, which consists of four doubled letters. However to compress it with run-length encoding we’ll need at least 8 characters, which doesn’t help us a lot.

// 8 chars replaced by 8 chars!?
input: 	"successfully accomplished"
output:	"su2ce2sfu2ly a2complished"

The problem is that if the input text contains numbers, “2” in particular, we’ve to chose an escape symbol (“@” for example), which we’ll use to mark where the encoded run begins. Thus if the input message is “2 successfully accomplished tasks”, it will be encoded as “2 su@2ce@2sfu@2ly a@2complished tasks”. Now the output message is longer!!! than the input string.

// the compressed message is longer!!!
input:	"2 successfully accomplished"
output:	"2 su@2ce@2sfu@2ly a@2complished tasks"

Again if the input stream contains the escape symbol, we have to find another one, and the problem is that it is often too difficult to find short escape symbol that doesn’t appear in the input text, without a full scan of the text.

That is why run-length encoding isn’t a good solution when compressing plain text, where long runs rarely appear. Well, of course, there are exceptions. For example such an exception is the lossy text compression with run-length encoding. It is intuitively clear that compressing text with loss is rarely useful, especially when you’ve to decompress exactly the same text. However there are some cases that lossy compression may be useful. Such case can be removing spaces. Indeed the text “successfully accomplished” brings us exactly the same information as “successfully accomplished”. In this case we can simply remove those spaces. Indeed we can use a marker to indicate the long run of spaces like “successfully@6 accomplished” in order to decompress the input string with absolutely no loss, but we can also throw those symbols away. This desision depends on the goal. Exactly with the same goal in mind we can remove new lines and tabs, only if we’re sure that the sense of the text is preserved. Yet again, a problem is that such long runs don’t happen to occur in random texts. That is why it’s better to use diagram encoding for plain text compression instead of run-length encoding.

Few Questions

After understanding the principles of the diagram encoding, let’s see some examples. In the example above it is better to replace doubled letters with something shorter. Let’s say # for “cc”, @ for “ss” and % for “ll”. Thus the input text will be compressed as “su#e@fu%y a#omplished”, which is shorter. But yet again what will happen if the input message contains one of the substitutions? Also we can’t say if there are many doubled letters and enough reasonable substitutions for them. A better approach is to replace patterns.

Run-length encoding isn't a good approach for text compression, because long runs rarely appear in a natural language.

Pattern Substitution

The pattern substitution algorithm is a variant of the diagram encoding. As I said above in plain English a very commonly used pattern can be “ the “, which is five characters long. We can now replace it with something like “$%” for example. In this case the message “I send the message” will become “I send$%message”. However there are some obstacles to overcome.

The first problem is that we need to know the language and somehow to define commonly used patterns in a dictionary. What would happen with a message written in some language we don’t know nothing about. Let’s say – Latin like the example bellow.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Cras venenatis, sapien eget suscipit placerat, justo quam blandit mauris, quis tempor ante sapien sodales augue. Praesent ut mauris quam. Phasellus scelerisque, ante quis consequat tristique, metus turpis consectetur leo, vitae facilisis sapien mi eu sapien. Praesent vitae ligula elit, et faucibus augue. Sed rhoncus sodales dolor ut gravida. In quis augue ac nulla auctor mattis sed sed libero. Donec eget purus eget enim tempor porta vitae eget diam. Mauris aliquet malesuada ipsum, non pulvinar urna vestibulum ac. Donec feugiat velit vitae nunc cursus imperdiet. Donec accumsan faucibus dictum. Phasellus sed mauris sapien. Maecenas mi metus, tincidunt sed rhoncus nec, sodales non sapien.

Clearly without knowing Latin it isn’t easy to define which are those commonly used patterns. The thing is that it’s better to use pattern substitution if you know in advance the set of words and characters.

The second problem is related to decompression. It is obvious that we need to define a dictionary and this dictionary must be used when decoding the message. It will be great also if we find more patterns longer than three characters. If not, the compression ratio will be low. Unfortunately such patterns aren’t very common in any natural language.

Diagram encoding and pattern substitution are far more suitable for text compression than run-length encoding. In fact, pattern substitution is very effective on compressing programming languages.

Application

It is interesting to answer the question, how to use diagram encoding or patter substitution to compress text in natural language, especially when we don’t know the language in detail? The answer hides in the question. We wont compress natural languages, but machine language. Exactly machine (programming) languages are limited to a smaller sets of words and symbols. Isn’t it true for any programing language? Like PHP, where words like “function”, “while”, “for”, “break”, “switch”, “foreach” happen to be often in use, or HTML with its defined set of tags. Perhaps the best example is CSS, where only the values of the properties can vary. CSS files also tend to have multiple new lines, tabs and spaces, which only humans read.

The question here is why should we compress those file types. It’s clear that after the compression they will be completely useless, both for humans and machines. Yes, that is true, but what if we have to store versions of those files into a DB. Kind of a backup. Imagine you’re working for a web hosting company that has to store daily versions of the sites it’s hosting. Thus the volume of stored information even for small companies hosting only few sites can be enormous. The problem is that compressing those files with some conventional compressing tool isn’t a good idea. Thus we’ve to save a copy of the entire site every day, but as we know the difference between daily versions of a site can be small. A version control system is another solution, but then you’ve to store the plain text of the files.

Perhaps a better approach is to compress the text using pattern substitution and then saving only differences – kind of version control, which can be done with “relative encoding”.

Using the above method we can save lots of disk space and in the same time we can compress/decompress easily. Another good thing is that you can save only changes to the initial files, like version control, which can also be compressed.

Implementation

The implementation of this algorithm is again on PHP and tries only to describe the main principles of compression. In this case I tried to compress a CSS file using the compression above. Although this example is quite primitive we can see some interesting facts. First of all you only need encoding and decoding dictionaries. Practically the encoding and decoding processes are equal, so you don’t need to implement two different functions. Here in this example a native PHP function is used – str_replace, because the purpose of this algorithm is not to describe pattern substitution techniques, but pattern substitution. It assumes that today’s programming languages have string manipulation functions for the purposes of this task.

$str = file_get_contents('large_style_file.css');
 
$encoding_dict = array(
	"\n" 		=> '$0',
	'text' 		=> '$1',
	'color' 	=> '$2',
	'display' 	=> '$3',
	'font' 		=> '$4',
	'width' 	=> '$5',
	'height'	=> '$6',	
	' '		=> '',
);
 
function replace_patterns($input, $dict) 
{
	foreach ($dict as $pattern => $replace) {
		$input = str_replace($pattern, $replace, $input);
	}
 
	return $input;
}
 
$result = replace_patterns($str, $encoding_dict);

By only replacing few CSS properties I achieved almost 40% of compression ratio (as shows the diagram bellow). The initial file is 202 KB, while compressed it’s only 131 KB. Of course, it all depends on the CSS file, but how about replacing all property names with shorter ones. Perhaps then the compression will be even better.

Related posts:

1. Computer Algorithms: Data Compression with Relative Encoding
2. Computer Algorithms: Data Compression with Run-length Encoding
3. Computer Algorithms: Data Compression with Bitmaps

In my previous post we saw how to compress data consisting of very long runs of repeating elements. This type of compression is known as “run-length encoding” and can be very handy when transferring data with no loss. The problem is that the data must follow a specific format. Thus the string “aaaaaaaabbbbbbbb” can be compressed as “a8b8”. Now a string with length 16 can be compressed as a string with length 4, which is 25% of its initial length without loosing any information. There will be a problem in case the characters (elements) were dispersed in a different way. What would happen if the characters are the same, but they don’t form long runs? What if the string was “abababababababab”? The same length, the same characters, but we cannot use run-length encoding! Indeed using this algorithm we’ll get at best the same string.

In this case, however, we can see another fact. The string consists of too many repeating elements, although not arranged one after another. We can compress this string with a bitmap. This means that we can save the positions of the occurrences of a given element with a sequence of bits, which can be easily converted into a decimal value. In the example above the string “abababababababab” can be compressed as “1010101010101010”, which is 43690 in decimals, and even better AAAA in hexadecimal. Thus the long string can be compressed. When decompressing (decoding) the message we can convert again from decimal/hexadecimal into binary and match the occurrences of the characters. Well, the example above is too simple, but let’s say only one of the characters is repeating and the rest of the string consists of different characters like this: “abacadaeafagahai”. Then we can use bitmap only for the character “a” – “1010101010101010” and compress it as “AAAA bcdefghi”. As you can see all the example strings are exactly 16 characters and that is a limitation. To use bitmaps with variable length of the data is a bit tricky and it is not always easy (if possible) to decompress it.

Basically bitmap compression saves the positions of an element that is repeated very often in the message!

$data = array(
	0 	=> 1991,
	1 	=> 1992,
	2 	=> 1993,
	3 	=> 1994,
	4 	=> 1991,
	5 	=> 1992,
	6 	=> 1993,
	7 	=> 1992,
	8 	=> 1991,
	9 	=> 1991,
	10 	=> 1991,
	11 	=> 1992,
	12 	=> 1992,
	13 	=> 1991,
	14 	=> 1991,
	15 	=> 1992,	
	...
);

The JSON will encoded message will be the following (a simple but yet very large javascript array).

[1991,1992,1993,1994,1991,1992,1993,1992,1991,1991,1991,1992,1992,1991,1991,1992, ...]

However if we use bitmap compression we’ll get a “shorter” array.

$data = array(
	0 => array(1991, '1000100011100110'),
	1 => array(1992, '0100010100011001'),
	2 => array(1993, '0010001000000000'),
	3 => array(1994, '0001000000000000'),
);

Now the JSON is:

[[1991,"1000100011100110"],[1992,"0100010100011001"],[1993,"0010001000000000"],[1994,"0001000000000000"]]

It is obvious that the compression ratio is getting better and better as the uncompressed data grows. In fact, most of us know bitmap compression from images, because this algorithm is largely used for image compression. We can imagine how successful it can be when compressing black and white images (as black and white can be represented as 0 and 1s). Actually it is used for more than two colors (256 for instance) and again the level of compression is very high.

Implementation

The following implementation on PHP aims only to illustrate the bitmap compressing algorithm. As we know this algorithm can be applicable for any kind of data structures.

// too many repeating "a" characters
$msg = 'aazahalavaatalawacamaahakafaaaqaaaiauaacaaxaauaxaaaaaapaayatagaaoafaawayazavaaaazaaabararaaaaakakaaqaarazacajaazavanazaaaeanaaoajauaaaaaxalaraaapabataaavaaab';
 
function bitmap($message) 
{
	$i = 0;
	$bits = $rest = '';
 
	while ($v = $message[$i]) {
		if ($v == 'a') {
			$bits .= '1';
		} else {
			$bits .= '0';
			$rest .= $v;
		}
		$i++;
	}
 
	return number_format(bindec($bits), 0, '.', '') . $rest;;
}
 
echo bitmap($msg);
 
// uncompressed: 
acaaaaadaaaabalaaeaaaaganaaxakaavawamaasavajawaaaayaauaaadalanagaeaeamaarafalaazaaaiasaanaahaaazaraxaalaahaaawaaajasamahaajaakarapanaakaoakaanawalaacamauaamaal
// compressed:
152299251941730035874325065523548237677352452096zhlvtlwcmhkfqiucxuxpytgofwyzvzbrrkkqrzcjzvnzenojuxlrpbtvb

Application

This algorithm is very useful when there is an element in our data that repeats very often, so you need to investigate the nature of the data you want to compress. Actually because of this fact this algorithm is used for image compression as PNG8 or GIF.

Related posts:

1. Computer Algorithms: Data Compression with Run-length Encoding
2. Computer Algorithms: Data Compression with Relative Encoding
3. Computer Algorithms: Data Compression with Diagram Encoding and Pattern Substitution

No matter how fast today’s computers and networks are, the users will constantly need faster and faster services. To reduce the volume of the transferred data we usually use some sort of compression. That is why this computer sciences area will be always interesting to research and develop.

There are many data compression algorithms, some of them lossless, others lossy, but their main goal aways will be to spare storage space and traffic. These algorithms are very useful when talking about data transfer between two distant places. Perhaps the best example is the transfer between a web server and a browser.

In the last few years a lot of research has been done on compressing files, executed on the client side. Such files are javascript, css, htmls and images. In fact servers and clients already have some techniques to compress data, like using GZIP for instance, that can dramatically decrease the transfer. In the other hand there are lots of tools and tricks in order to decrease the size of the data.

Actually when a file is executed by the client’s virtual machine, it doesn’t matter how “beautifully” it is formatted from a programmer’s point of view. Thus the spaces, tabs and the new lines don’t bring any significant information for the environment. That is why such compressing tools like YUI Compressor, Google Closure Compiler, etc. remove those symbols. Well, they can achieve even more in order to improve the compression rate. In this post I won’t cover this, but this shows how important data compression algorithms are.

It would be great if we could just compress data with some tool. Unfortunately this is not the case and usually the compression rate depends on the data itself. It is obvious that the choice of data compression algorithm depends mainly on the data and first of all we must explore the data.

Here I’ll cover one very simple lossless data compression algorithm called “run-length encoding” that can be very useful in some cases.

Overview

This algorithm consists of replacing large sequences of repeating data with only one item of this data followed by a counter showing how many times this item is repeated. To become clearer let’s see a string example.

aaaaaaaaaabbbaxxxxyyyzyx

This string’s length is 24 and as we can see there are lots of repetitions. Using the run-length algorithm, we replace any run with shorter string followed by a counter.

a10b3a1x4y3z1y1x1

The length of this string is 17, which is approximately 70% of the initial length. Obviously this is not the optimal way to compress the given string. For instance we don’t need to use the digit “1” when the character is repeated only once. In some cases this approach can increase the length of the initial string which is exactly the opposite of what we need. In this case we’ll get the string bellow.

a10b3ax4y3zyx

Now the length of the resulting string is 13, which is 54% of the initial length! A variation of the example above is not to keep a counter of the repetitions of the character, but their position instead. Thus the initial string will be compressed as follows.

a0b10a13x14y18z21y22x23

Which of these two approaches you’ll use depends on the goal. In the second case we can achieve a good optimization of binary search.

It is clear that this algorithm is not only applicable on strings. We can achieve very good results on arrays. A typical example is the transfer of JSON from a server to a client. Then if there are large sequences of repeating data we can achieve great results.

Implementation

The implementation bellow is assuming that we’re compressing a string and it’s written on PHP. However the nature of this algorithm doesn’t restrict us to use only strings. As I said before with slight modifications we can use it with other data structures. It is important only to understand that the run-length algorithm is very useful on large sequences of repeating elements, no matter characters or array items.

$message = 'aaaaaaaaaabbbaxxxxyyyzyx';
 
function run_length_encode($msg)
{
	$i = $j = 0;
	$prev = '';
	$output = '';
 
	while ($msg[$i]) {
		if ($msg[$i] != $prev) {
 
			if ($i) 
				$output .= $j;
 
			$output .= $msg[$i];
 
			$prev = $msg[$i];
 
			$j = 0;
		}
		$j++;
		$i++;
	}
 
	$output .= $j;
 
	return $output;
}
 
// a10b3a1x4y3z1y1x1
echo run_length_encode($message);

And slightly optimized.

$message = 'aaaaaaaaaabbbaxxxxyyyzyx';
 
function run_length_encode($msg)
{
	$i = $j = 0;
	$prev = '';
	$output = '';
 
	while ($msg[$i]) {
		if ($msg[$i] != $prev) {
 
			if ($i && $j > 1) 
				$output .= $j;
 
			$output .= $msg[$i];
 
			$prev = $msg[$i];
 
			$j = 0;
		}
		$j++;
		$i++;
	}
 
	if ($j > 1)
		$output .= $j;
 
	return $output;
}
 
// a10b3ax4y3zyx
echo run_length_encode($message);

Finally a small change – now we store the position of the character.

$message = 'aaaaaaaaaabbbaxxxxyyyzyx';
 
function run_length_encode($msg)
{
	$i = 0;
	$prev = '';
	$output = '';
 
	while ($msg[$i]) {
		if ($msg[$i] != $prev) {
 
			$output .= $msg[$i] . $i;
 
			$prev = $msg[$i];
 
		}
 
		$i++;
	}
 
	return $output;
}
 
// a0b10a13x14y18z21y22x23
echo run_length_encode($message);

Complexity and Data Compression

We’re used to talk about complexity of an algorithm measuring time and we usually try to find the fastest implementation, like in search algorithms. Here it is not so important to compress data quickly, but to compress as much as possible so the output is as small as possible without lossing data. A great feature of run-length encoding is that this algorithm is easy to implement.

Application

We can use run-length encoding in many cases. It is commonly used to compress images and is very successful when we deal only with black and white images. Here I’ll cover another use case that I only mentioned above. Let’s say we have to transfer a very large array of data to our AJAX-powered application using JSON. Let’s say also that the data are some years, for instance the years of the premiere of a movie. There are lots of movies with a premiere in the same year, thus although the data is sorted, we actually can’t have any benefit. More important is that we have large sequences of data. Here we can use run-length encoding.

$data = array(
	0 	=> 1991,
	1 	=> 1991,
	...
	2223 	=> 1991,
	2224 	=> 1992,
	...
	19298 	=> 1995,
	19299 	=> 1996,
	...
);

As you can see to transfer the whole array can be a nightmare, especially on slow networks. It is better to compress it (i.e. with PHP’s json_encode).

// {"0":1991,"1":1991, ..., "2223":1991,"2224":1992, ..., "19298":1995,"19299":1996, ...}
echo json_encode($data);

After running run-length encoding we can receive something like the following array (note that these are only sample data and it’s up to you to decide which is the best format to store data).

$data = array(
	0 => array(1991, 2224),
	1 => array(1992, 3948),
	2 => array(1995, 2398),
	3 => array(1996, 3489),
);

And the JSON output.

// [[1991,2224],[1992,3948],[1995,2398],[1996,3489]]
echo json_encode($data);

Note that if the data is sorted we can achieve great success compressing it!!! This approach can be used for images, graphics or map coordinates.

This is only one example of how data compression can be useful in our daily work. Although the communication between the server and the client can be optimized and compressed, we can improve it. In other words we’re not always sure that the opposite side supports compression.

Well, it’s true that the client has to decompress the data, which can also be slow. Now in the first case we have only the time to transfer, as on the diagram bellow.

Time to transfer data without compression!

In the second case, we should sum the time for compression, transfer and decompression.

Time to send data with compression!

All this is important, but in general data compression can be handy in many cases in our daily work.

Related posts:

1. Computer Algorithms: Data Compression with Relative Encoding
2. Computer Algorithms: Data Compression with Bitmaps
3. Computer Algorithms: Data Compression with Diagram Encoding and Pattern Substitution

Important Note

It’s very important to say that the following results are dependant from the machine and the environment!

Source Code

Here’s the PHP source code.

function divide($n = 1) 
{
	$a = microtime(true);
	for ($i = 0; $i < $n; $i++) {
		300/2;
	}
	echo microtime(true) - $a;
}
 
divide(100);
//divide(1000);
//divide(10000);
//divide(100000);
//divide(1000000);
//divide(10000000);

and bitwise …

function bitwise($n = 1)
{
	$a = microtime(true);
	for ($i = 0; $i < $n; $i++) {
		300 >> 1;
	}
	echo microtime(true) - $a;
}
 
bitwise(100);
//bitwise(1000);
//bitwise(10000);
//bitwise(100000);
//bitwise(1000000);
//bitwise(10000000);

Note that each method was called 6 times with the same parameter. This means that divide(100) was called 6 times and then I used the average value of these six times.

Results

I said back then in my binary search post, that in PHP the bitwise operator “>> 1″ is not faster than the typical division with the “/” operator. However the results tells us that using bitwise division is slightly faster, as you can see at the diagram bellow.

n		">>"			"/"
100		0.0002334912618		0.000311803817749
1000		0.001911004384359	0.007335503896078
10000		0.013423800468445	0.039460102717081
100000		0.14417803287506	0.21413381894429
1000000		1.15839115778605	1.17152162392935
10000000	10.556711634		11.0911625623705

Bitwise division is slightly faster!

Conclusion

Although bitwise division is a bit faster the difference is so small that you should work with very large data in order to gain some performance.

Related posts:

1. JavaScript Performance: for vs. while
2. Firebug’s console.time() accuracy
3. jQuery live() vs bind() performance

I wrote about binary search in my previous post, which is indeed one very fast searching algorithm, but in some cases we can achieve even faster results. Such an algorithm is the “interpolation search” – perhaps the most interesting of all searching algorithms. However we shouldn’t forget that the data must follow some limitations. In first place the array must be sorted. Also we must know the bounds of the interval.

Why is that? Well, this algorithm tries to follow the way we search a name in a phone book, or a word in the dictionary. We, humans, know in advance that in case the name we’re searching starts with a “B”, like “Bond” for instance, we should start searching near the beginning of the phone book. Thus if we’re searching the word “algorithm” in the dictionary, you know that it should be placed somewhere at the beginning. This is because we know the order of the letters, we know the interval (a-z), and somehow we intuitively know that the words are dispersed equally. These facts are enough to realize that the binary search can be a bad choice. Indeed the binary search algorithm divides the list in two equal sub-lists, which is useless if we know in advance that the searched item is somewhere in the beginning or the end of the list. Yes, we can use also jump search if the item is at the beginning, but not if it is at the end, in that case this algorithm is not so effective.

So the interpolation search is based on some simple facts. The binary search divides the interval on two equal sub-lists, as shown on the image bellow.

The binary search algorithm divides the list in two equal sub-lists!

What will happen if we don’t use the constant ½, but another more accurate constant “C”, that can lead us closer to the searched item.

The interpolation search algorithm tries to improve the binary search!

The question is how to find this value? Well, we know bounds of the interval and looking closer to the image above we can define the following formula.

C = (x-L)/(R-L)

Now we can be sure that we’re closer to the searched value.

Implementation

Here’s an implementation of interpolation search in PHP.

$list = array(201, 209, 232, 233, 332, 399, 400);
$x = 332;
 
function interpolation_search($list, $x)
{
	$l = 0;
	$r = count($list) - 1;
 
	while ($l <= $r) {
		if ($list[$l] == $list[$r]) {
			if ($list[$l] == $x) {
				return $l;
			} else {
				// not found
				return -1;
			}
		}
 
		$k = ($x - $list[$l])/($list[$r] - $list[$l]);
 
		// not found
		if ($k < 0 || $k > 1) {
			return -1;
		}
 
		$mid = round($l + $k*($r - $l));
 
		if ($x < $list[$mid]) {
			$r = $mid - 1;
		} else if ($x > $list[$mid]) {
			$l = $mid + 1;
		} else {
			// success!
			return $mid;
		}
 
		// not found
		return -1;
	}
}
 
echo interpolation_search($list, $x);

Complexity

The complexity of this algorithm is log₂(log₂(n)) + 1. While I wont cover its proof, I’ll say that this is very slowly growing function as you can see on the following chart.

Indeed when the values are equally dispersed into the interval this search algorithm can be extremely useful – way faster than the binary search. As you can see log₂(log₂(100 M)) ≈ 4.73 !!!

Application

As I said already this algorithm is extremely interesting and very appropriate in many use cases. Here’s an example where interpolation search can be used. Let’s say there’s an array with user data, sorted by their year of birth. We know in advance that all users are born in the 80’s. In this case sequential or even binary search can be slower than interpolation search.

$list = array(
	0 => array('year' => 1980, 'name' => 'John Smith', 'username' => 'John'),
	1 => array('year' => 1980, ...),
	...
	10394 => array('year' => 1981, 'name' => 'Tomas M.', ...),
	...
	348489 => array('year' => '1985', 'name' => 'James Bond', ...),
	...
	2808008 => array('year' => '1990', 'name' => 'W.A. Mozart', ...)
);

Now if we search for somebody born in 1981 a good approach is to use interpolation search.

Related posts:

1. Computer Algorithms: Binary Search
2. Computer Algorithms: Jump Search
3. Computer Algorithms: Sequential Search

The binary search is perhaps the most famous and best suitable search algorithm for sorted arrays. Indeed when the array is sorted it is useless to check every single item against the desired value. Of course a better approach is to jump straight to the middle item of the array and if the item’s value is greater than the desired one, we can jump back again to the middle of the interval. Thus the new interval is half the size of the initial one.

Basic implementation of binary search

If the searched value is greater than the one placed at the middle of the sorted array, we can jump forward. Again on each step the considered list is getting half as long as the list on the previous step, as shown on the image bellow.

Binary search - basic implementation

Implementation

Here’s a sample implementation of this algorithm on PHP. Obviously the nature of this approach is guiding us to a recursive implementation, but as we know, sometimes recursion can be dangerous. That’s why here we can see either the recursive and iterative solution.

Recursive Binary Search

$list = array(0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144);
$x = 55;
 
function binary_search($x, $list, $left, $right) 
{
	if ($left > $right)
		return -1;
 
	$mid = ($left + $right) >> 1;
 
	if ($list[$mid] == $x) {
		return $mid;
	} elseif ($list[$mid] > $x) {
		return binary_search($x, $list, $left, $mid-1);
	} elseif ($list[$mid] < $x) {
		return binary_search($x, $list, $mid+1, $right);
	}
}
 
echo binary_search($x, $list, 0, count($list)-1);

Iterative Binary Search

$list = array(0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144);
$x = 55;
 
function iterative_binary_search($x, $list) 
{
	$left = 0;
	$right = count($list)-1;
 
	while ($left <= $right) {
		$mid = ($left + $right) >> 1;
 
		if ($list[$mid] == $x) {
			return $mid;
		} elseif ($list[$mid] > $x) {
			$right = $mid - 1;
		} elseif ($list[$mid] < $x) {
			$left = $mid + 1;
		}
	}
 
	return -1;
}
 
echo iterative_binary_search($x, $list);

Caution: Optimization

Most of the optimization techniques mentioned online recommend to replace the expensive operation of dividing by 2 with its bitwise equivalent (n >> 1) == n/2. That is not always true and it is very dependant from the programming language. Thus in PHP those operations are fairly similar as PHP is written in C. You’ve to be aware of the language specific features when optimizing code.

Fibonacci Search

Every developer has heard of Fibonacci and his sequence. The Fibonacci search algorithm is practically a variation of the binary search algorithm. In fact the only difference is that the binary search algorithm divides the list into two equal parts, while the Fibonacci search divides it in two but not equal parts. In fact sometimes it is faster to search if you divide the list by such non equal sub-lists. However the length of the sub-lists is not random.

It is clear that the ratio of any two consecutive numbers in the Fibonacci sequence is practically forming the golden ratio. This can lead us to another variation of Fibonacci and binary search – the golden section search. The only different thing is that you’ve to divide the length of the list in two parts exactly by the golden ratio.

The golden section search doesn't divide the array on two equal sub-lists!

The complexity both of the Fibonacci and the golden section search algorithm is identical with the complexity of the binary search. However these two algorithms are rarely used in practice. Also it is more difficult to implement these two algorithms than the binary search and their advantage depends on specifically dispersed data.

Complexity

The complexity of the binary search algorithm is intuitively clear – O(log(n)), which makes it far more effective than the sequential search.

f(n) = log(n) compared to f(n) = n

Application

It is useless to mention examples of its use. This algorithm is easy to implement and in the same times it is very fast. Yes, indeed, this algorithm is only possible on sorted lists and this is a limitation. Also, as I said, compared to the jump search here we have more than one jump back in most of the cases, which sometimes can be more expensive than jump forward. However is this the fastest search algorithm? I’ll try to answer this question in my next article.

Related posts:

1. Computer Algorithms: Interpolation Search
2. Computer Algorithms: Jump Search
3. Computer Algorithms: Sequential Search

In my previous article I discussed how the sequential (linear) search can be used on an ordered lists, but then we were limited by the specific features of the given task. Obviously the sequential search on an ordered list is ineffective, because we consecutively check every one of its elements. Is there any way we can optimize this approach? Well, because we know that the list is sorted we can check some of its items, but not all of them. Thus when an item is checked, if it is less than the desired value, we can skip some of the following items of the list by jumping ahead and then check again. Now if the checked element is greater than the desired value, we can be sure that the desired value is hiding somewhere between the previously checked element and the currently checked element. If not, again we can jump ahead. Of course a good approach is to use a fixed step. Let’s say the list length is n and the step’s length is k. Basically we check list(0), then list(k-1), list(2k-1) etc. Once we find the interval where the value might be (m*k-1 < x <= (m+1)*k – 1), we can perform a sequential search between the last two checked positions. By choosing this approach we avoid a lot the weaknesses of the sequential search algorithm. Many comparisons from the sequential search here are eliminated.

How to choose the step’s length

We know that it is a good practice to use a fixed size step. Actually when the step is 1, the algorithm is the traditional sequential search. The question is what should be the length of the step and is there any relation between the length of the list (n) and the length of the step (k)? Indeed there is such a relation and often you can see sources directly saying that the best length k = √n. Why is that?

Well, in the worst case, we do n/k jumps and if the last checked value is greater than the desired one, we do at most k-1 comparisons more. This means n/k + k – 1 comparisons. Now the question is for what values of k this function reaches its minimum. For those of you who remember maths classes this can be found with the formula -n/(k^2) + 1 = 0. Now it’s clear that for k = √n the minimum of the function is reached.

Of course you don’t need to prove this every time you use this algorithm. Instead you can directly assign √n to be the step length. However it is good to be familiar with this approach when trying to optimize an algorithm.

Let’s cosider the following list: (0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610). Its length is 16. Jump search will find the value of 55 with the following steps.

Jump search skips some of the items of the list in order to improve performance!

Implementation

Let’s see an example of jump search, written in PHP.

$list = array();
 
for ($i = 0; $i < 1000; $i++) {
	$list[] = $i;
}
 
// now we have a sorted list: (0, 1, 2, 3, ..., 999)
 
function jump_search($x, $list)
{
	// calculate the step
	$len = count($list);
	$step = floor(sqrt($len));
	$prev = 0;
 
	while ($list[($step < $len ? $step : $len)] < $x) {
		$prev = $step;
		$step += floor(sqrt($len));
 
		if ($step >= $len) {
			return FALSE;
		}
	}
 
	while ($list[$prev] < $x) {
		$prev++;
		if ($prev == ($step < $len ? $step : $len)) {
			return FALSE;
		}
	}
 
	if ($list[$prev] == $x) {
		return $prev;
	}
 
	return FALSE;
}
 
echo (int)jump_search(674, $list);

Here we have a sorted list with 1000 elements that looks like this: (0, 1, 2, …, 999). Obviously with sequential search we’ll find the value of 674 with exactly on the 674-th iteration. Here, with jump search we can reach it on the 44-th iteration, and this shows us the advantage of jump search over the sequential search on ordered lists.

Further Optimization

Although all examples here deal with small lists in practice this is not always true. Sometimes the step itself can be a very large number, so once you know the interval where the desired value could be you can perform jump search again.

We saw that the best size of the step is √n, but it is not a good idea to start from the first element of the list just as we didn’t in the example above. A better option is to begin from kth item. Now we can improve the above solution.

The basic implementation of jump search can be slightly optimized!

Complexity

Obviously the complexity of the algorithm is O(√n), but once we know the interval where the value is we can improve it by applying jump search again. Indeed let’s say the list length is 1,000,000. The jump interval should be: √1000000=1000. As you can see again, you can use jump search with a new step √1000≈31. Every time we find the desired interval we can apply the jump search algorithm with a smaller step. Of course finally the step will be 1. In this case the complexity of the algorithm is no longer O(√n). Now its complexity is approaching logarithmic value. The problem is that the implementation of this approach is considered to be more difficult than the binary search, where the complexity is also O(log(n)).

Application

As almost every algorithm the jump search is very convinient for a certain kind of tasks. Yes, the binary search is easy to implement and its complexity is O(log(n)), but in case of a very large list the direct jump to the middle can be a bad idea. Then we should make a large step back if the searched value is placed at the beginning of the list.

Perhaps every one of us has performed some sort of a primitive jump search in his life without even knowing it. Do you remember cassette recorders? We used the “fast forward” key and periodically checked whether the tape was on our favorite song. Once we stopped at the middle of the song we used the “rewind” button to find exactly the beginning of the song.

This clumsy example can give us the answer of where jump search can be better than binary search. The advantage of jump search is that you need to jump back only once (in case of the basic implementation).

Jump search is very useful when jumping back is significantly slower than jumping forward!

If jumping back takes you significantly more time than jumping forward then you should use this algorithm.

Related posts:

1. Computer Algorithms: Binary Search
2. Computer Algorithms: Interpolation Search
3. Computer Algorithms: Sequential Search

The expression “linear search in sorted lists” itself sounds strange. Why should we use this algorithm for sorted lists when there are lots of other algorithms that are far more effective? As I mentioned in my previous post the sequential search is very ineffective in most of the cases and it is primary used for unordered lists. Indeed sometimes it is more useful first to sort the data and then use a faster algorithm like the binary search. On the other hand the analysis shows that for lists with less than ten items the linear search is much faster than the binary search.

Although, for instance, binary search is more effective on sorted lists, sequential search can be a better solution in some specific cases with minor changes. The problem is that when developers hear the expression “sorted list” they directly choose an algorithm different from the linear search. Perhaps the problem lays in the way we understand what an ordered list is?

What is a sorted list?

We used to think that this list (1, 1, 2, 3, 5, 8, 13) is sorted. Actually we think so because it is … sorted, but the list (3, 13, 1, 3, 3.14, 1.5, -1) is also sorted, except that we don’t know how. Thus we can think that any array is sorted, although it is not always obvious how.

There are basically two cases when sequential search can be very useful. First when the list is very short or when we know in advance that there are some values that are very frequently searched.

Let’s say we have a very large list, with hundreds of thousands of items, but actually most of the searches in that list always find the same ten values. This additional information tells us that using a binary search will be quite ineffective in this case. A possible approach, of course, is to place those values at the front of the list and to perform a sequential search.

You should choose a search algorithm by carefully examining the data you search.

Unfortunately the search will be slow when we search for some value missing from the front of the list, but then we can use another algorithm on sorted lists.

Still there is one question that should be answered. We do know that in most of the cases we search for the same values, but we do not know exactly those values. So the question is how to put the most frequently accessed values at the front of the list, since we don’t know them. Here we need some sort of auto adjustment of the list.

Self-Organization

Self-organization practically means that every time we search and find the desired value, we somehow change the list so the next search will be far more effective.

There are basically two approaches to do that.

1. To move the item one position forward to the front of the list;
2. To move the item directly at the front of the list;

Of course it depends on your case which approach you’ll choose, but it is assumed that the second option, the one that we choose to move the item directly at the front of the list, is better.

Indeed if we choose the first option and the list is (…, 24, 31) after constantly searching for those two values the array will be changing from (…, 24, 31) to (…, 31, 24) and once again to (…, 24, 31) and so on and so on.

Thus a better solution is to move the desired item directly to the front of the list. Now if we look for the value of “5″ in the list (1, 2, 4, …, 5, …, 398) it will become (5, 1, 2, …, 398) after the value is found.

By choosing this approach we can be sure that as the number of searches increases, the most frequently searched values are placed at the front of the list. Now the sequential search is quite a good solution!

Here’s an example of sequential search from my previous article. The only change is that after we find the desired value we need to move it to the front of the list.

/**
 * Performs a sequential search using sentinel
 * and changes the array after the value is found
 *
 * @param array $arr
 * @param mixed $value
 */
function sequential_search(&$arr, $value)
{
	$arr[] = $value;
	$index = 0;
 
	while ($arr[$index++] != $value);
 
	if ($index < count($arr)) {
 
		// put the item at the front of the list
		array_unshift($arr, $arr[$index-1]);
 
		// remove the value from its previous position
		unset($arr[$index]);
 
		// unset the sentinel
		unset($arr[count($arr)+1]);
 
		return true;
	}
 
	return false;
}
 
// the list
$arr = array(1, 2, 3, 3.14, 5, 4, 6, 9, 8);
 
// the value
$x = 3.14;
 
if (sequential_search($arr, $x)) {
	// now the array is changed to
	// (3.14, 1, 2, 3, 5, 4, 6, 9, 8)
	echo "The value $x is found!";
} else {
	echo "The value $x doesn't appear to be in the list!";
}

Application

Using sequential search in sorted lists can be very useful and fast, the only thing is that we need to know in advance that there are some values that are frequently searched. A typical example of this case is the contact list on your phone. Perhaps you have lots of names in there, but most of the times you search in it is to find your best friends’ and family phone numbers. That is why most of the cell phone manufacturers add to their phones the ability to predefine shortcut keys for the most frequently dialed numbers.

Here’s another use case. Let’s say that we have the same scenario as in my previous post, where username/name pairs are stored into a CSV file. We can fetch those values in a PHP array.

// list with username/name pairs
$arr = array(
	array('name' => 'James Bond', 'username' => 'jamesbond007'),
	array('name' => 'John Smith', 'username' => 'jsmith'),
	array('name' => 'John Silver', 'username' => 'hohoho'),
	array('name' => 'Yoda', 'username' => 'masteryoda900'),
	array('name' => 'Darth Vader', 'username' => 'vader'),
);

Every time a user enters the site we search for his name by his username and a welcome message is displayed. We know that some users enter the site very frequently while others do that once per month so we cannot only perform a sequential search but also we can use self-organization for the array and change the CSV file at the end.

/**
 * Performs a sequential search using a sentinel.
 * Returns the index of the item if the item was found
 * and FALSE otherwise.
 * 
 * @param array $arr
 * @param mixed $value
 */
function sequential_search(&$arr, $value)
{	
	// usign a sentinel
	$arr[] = array('username' => $value, 'name' => '');
	$index = 0;
 
	while ($arr[$index++]['username'] != $value);
 
	// if the desired element is in the array
	if ($index < count($arr)) {
 
		// push the element at the front of the list
		array_unshift($arr, $arr[$index-1]);
 
		// remove the item from its previous place
		unset($arr[$index]);
 
		// remove the sentinel
		unset($arr[count($arr)]);
 
		// return the index of the value
		return $index;
	}
 
	// the element has not been found
	return FALSE;
}
 
if (FALSE !== ($iterations = sequential_search($arr, 'vader'))) {
	echo "Hello, {$arr[0]['name']}";
	echo "Found after $iterations iterations!";
} else {
	echo "Hi, guest!";
}
 
if (FALSE !== ($iterations = sequential_search($arr, 'vader'))) {
	echo "Hello, {$arr[0]['name']}";
	echo "Found after $iterations iterations!";
} else {
	echo "Hi, guest!";
}

The result is:

Hello, Darth Vader 
Found after 5 iterations!
Hello, Darth Vader
Found after 1 iterations!

Now every time Darth Vader tries to sign in, you won’t bother him to wait a lot for sure. However I bet nobody uses CSV files to store such information, but this is only an example.

Note: all the examples in this article are written in PHP.

Related posts:

1. Computer Algorithms: Sequential Search
2. Computer Algorithms: Jump Search
3. Computer Algorithms: Binary Search