Do you know that objects in PHP5 are passed by reference, while arrays and other scalar variables are passed by value? Yes, you know it, but it’s not exactly true. Let’s see some example and let’s try to answer few questions.

// depending on the machine but both lines return
// expectedly equal values: 331208

// 331208
echo memory_get_usage();
echo memory_get_usage();

These two lines of code, expectedly return the same value (in my case 331208), which shows us that because nothing happened in between them the memory usage isn’t growing. Let’s now put some code in between them.

echo memory_get_usage(); // 331616
$a = 10;
echo memory_get_usage(); // 331696

Now because of the variable $a, we get a little more memory consuption! The same thing (with even more memory usage) happens if we have an array initialization.

echo memory_get_usage(); // 332128
$a = array(1, 2, 3, 'hello', 'world');
echo memory_get_usage(); // 332728

OK, now we see how much memory PHP is using for this very simple and small array. If we copy this array, we’d expect PHP to take twice as much memory, but that’s not the case!

echo memory_get_usage(); // 332336
$a = array(1, 2, 3, 'hello', 'world');
$b = $a;
echo memory_get_usage(); // 332984

Actually if we had $b = 10; this will consume more!!! memory than the code above.

echo memory_get_usage(); // 332336
$a = array(1, 2, 3, 'hello', 'world');
$b = 10;
echo memory_get_usage(); // 333016

This is simply because in the first case $b wasn’t a copy of $a, while in the second case we have a brand new variable on the ground, which, of course, requires memory.

Why Arrays aren’t Copied?

Actually now we see that copying by reference and by value is absolutely the same.

$a = array(1, 2, 3, 'hello', 'world');

// this line is exactly the same as ...
$b = $a;

// this line
$b = &$a;

That is because in both case the array is passed by reference. It is copied once we make changes to $b. Then in the first case $b becomes a copy of $a and it’s changed, while in the second case $b is exactly the same array as $a and every change to $b changes $a as well.

echo memory_get_usage();
$a = array(1, 2, 3, 'hello', 'world');
echo memory_get_usage();
$b = $a;
$b = array(1, 2, 3, 'goodby', 'world');
echo memory_get_usage();

Conclusion

If we take an example from Zend Framework, where often we work with arrays that are passed to the view as a “copy”:

$a = array(1, 2, 3, 'hello', 'world');
$this->view->a = $a; // this is NOT a copy

This will not consume more memory!!! The only way to make your application more memory inefficient is to change directly the $this->view->a array, so be careful!

Let’s say we have an array of data represented by some text followed by a number. Just like the movies from a movie series like “Mission Impossible” or “Pirates of the Carribean”. We know that they are often followed by the consecutive number of the episode.

Mission: Impossible 1
Mission: Impossible 2
Mission: Impossible 3
...

Since we have no more than three or four episodes we can easily sort the array if it’s not sorted initially.

$a = array('Mission: Impossible 2', 'Mission: Impossible 3', 'Mission: Impossible 1');

sort($a);

// Mission: Impossible 1
// Mission: Impossible 2
// Mission: Impossible 3
print_r($a);

However in some cases we can have more than 10 episodes. Then we can meet a problem while sorting the array above.

$a = array('Episode 1', 'Episode 2', 'Episode 11', 'Episode 112');

sort($a);

// Episode 1
// Episode 11
// Episode 112
// Episode 2
print_r($a);

Now because this is by default an alphabetical sort order we get an array that isn’t sorted to our human undestanding.

The question is how to overcome this problem?

PHP

First the thing we actually need is called “natural sort”, so PHP (with its full of handful functions library) takes care for us with natsort.

$a = array('Episode 1', 'Episode 2', 'Episode 11', 'Episode 112');

natsort($a);

// Episode 1
// Episode 2
// Episode 11
// Episode 112
print_r($a);

Now the array is sorted accordingly.

MySQL

MySQL in the other hand appears to be more hostile to natural sorting. We can just have ORDER BY with some keyword in order to sort a column using natural sort.

Given the table data:

my_table
-----------------------------------------
|	id	|	name		|
-----------------------------------------
|	1	|	Episode 2	|
|	2	|	Episode 1	|
|	3	|	Episode 112	|
|	4	|	Episode 11	|
-----------------------------------------

SELECT * FROM my_table ORDER BY name;

The query above will return the table in an alphabetical order.

my_table
-----------------------------------------
|	id	|	name		|
-----------------------------------------
|	2	|	Episode 1	|
|	4	|	Episode 11	|
|	3	|	Episode 112	|
|	1	|	Episode 2	|
-----------------------------------------

However there are some “hacks”. Here’s one of them.

SELECT * FROM my_table ORDER BY LENGTH(name), name;

Now the column is sorted correctly.

my_table
-----------------------------------------
|	id	|	name		|
-----------------------------------------
|	2	|	Episode 1	|
|	1	|	Episode 2	|
|	4	|	Episode 11	|
|	3	|	Episode 112	|
-----------------------------------------

Each natural number that is divisible only by 1 and itself is prime. Prime numbers appear to be more interesting to humans than other numbers. Why is that and why prime numbers are more important than the numbers that are divisible by 2, for instance? Perhaps the answer is that prime numbers are largely used in cryptography, although they were interesting for the ancient Egyptians and Greeks (Euclid has proved that the prime numbers are infinite circa 300 BC). The problem is that there is not a formula that can tell us which is the next prime number, although there are algorithms that check whether a given natural number is prime. It’s very important these algorithms to be very effective, especially for big numbers.

Overview

As I said each natural number that is divisible only by 1 and itself is prime. That means that 2 is the first prime number and 1 is not considered prime. It’s easy to say that 2, 3, 5 and 7 are prime numbers, but what about 983? Well, yes 983 is prime, but how do we check that? If we want to know whether n is prime the very basic approach is to check every single number between 2 and n. It’s kind of a brute force.

Implementation

The basic implementation in PHP for the very basic (brute force) approach is as follows.

Unfortunately this is one very ineffective algorithm. We don’t have to check every single number between 1 and n, it’s enough to check only the numbers between 1 and n/2-1. If we find such a divisor that will be enough to say that n isn’t prime.

Although that code above optimizes a lot our first prime checker, it’s clear that for large numbers it won’t be very effective. Indeed checking against the interval [2, n/2 -1] isn’t the optimal solution. A better approach is to check against [2, sqrt(n)]. This is correct, because if n isn’t prime it can be represented as p*q = n. Of course if p > sqrt(n), which we assume can’t be true, that will mean that q < sqrt(n).

Beside that these implementations shows how we can find prime number, they are a very good example of how an algorithm can be optimized a lot with some small changes.

Sieve of Eratosthenes

Although the sieve of Eratosthenes isn’t the exact same approach (to check whether a number is prime) it can give us a list of prime numbers quite easily. To remove numbers that aren’t prime, we start with 2 and we remove every single item from the list that is divisible by two. Then we check for the rest items of the list, as shown on the picture below.

The PHP implementation of the Eratosthenes sieve isn’t difficult.

Application

As I said prime numbers are widely used in cryptography, so they are always of a greater interest in computer science. In fact every number can be represented by the product of two prime numbers and that fact is used in cryptography as well. That’s because if we know that number, which is usually very very big, it is still very difficult to find out what are its prime multipliers. Unfortunately the algorithms in this article are very basic and can be handy only if we work with small numbers or if our machines are tremendously powerful. Fortunately in practice there are more complex algorithms for finding prime numbers. Such are the sieves of Euler, Atkin and Sundaram.

PHP by Book

Here’s how PHP developer declare a string, which is something very common in any programming language.

$my_var = 'hello world';
// or
$my_var = "hello world";

PHP Tricks

What if you do the following:

echo hello;

That appears to be correct … Well, it’s not absolutely correct. You’ll be “noticed”.

// Notice: Use of undefined constant hello
echo hello;

However if you disable error reporting, the code will be completely fine.

error_reporting(0);

// no problem now
echo hello;

Variations

What follows from the thing above is that you can use strings without quotes:

// hello
echo hello;

// hello world (concatenated)
echo hello . ' world';

// helloworld
echo hello . world;

However you can’t have spaces and most of the “special” symbols.

// syntax error
echo hello world;

// syntax error
echo hello!;

Final Words

Although you can do this in PHP, that is completely wrong. The code becomes more difficult to read and understand. In the second place you can miss a $ sign in front of a variable declaration and thus the PHP interpreter will assume this is a string. So disable error reporting isn’t so great sometimes.

Sorted data can dramatically change the speed of our program, therefore sorting algorithms are something quite special in computer science. For instance searching in a sorted list is faster than searching in an unordered list.

There are two main approaches in sorting – by comparing the elements and without comparing them. A typical algorithm from the first group is insertion sort. This algorithm is very simple and very intuitive to implement, but unfortunately it is not so effective compared to other sorting algorithms as quicksort and merge sort. Indeed insertion sort is useful for small sets of data with no more than about 20 items.

Insertion sort it is very intuitive method of sorting items and we often use it when we play card games. In this case the player often gets an unordered set of playing cards and intuitively starts to sort it. First by taking a card, making some comparisons and then putting the card on the right position.

So let’s say we have an array of data. In the first step the array is unordered, but we can say that it consists of two sub-sets: sorted and unordered, where on the first step the only item in the sorted sub-set is its first item. If the length of the array is n the algorithm is considered completed in n-1 steps. On each step our sorted subset is growing with one item. The thing is that we take the first item from the unordered sub-set and with some comparisons we put it into its place in the sorted sub-set, like on the diagram bellow.

Implementation

Here’s a quick implementation of insertion sort in PHP. The good thing is that it is easy to implement, but there are bad news too – insertion sort is slow and it is ineffective for large data sets.

$data = array(4, 2, 4, 1, 2, 6, 8, 19, 3);

function insertion_sort(&$arr)
{
	$len = count($arr);
	
	for ($i = 1; $i < $len; $i++) {
		$tmp = $arr[$i];
		$j = $i;
		
		while (($j >= 0) && ($arr[$j-1] > $tmp)) {
			$arr[$j] = $arr[$j-1];
			$j--;
		}
		$arr[$j] = $tmp;
	}
}

We can improve this code a little by using a sentinel, just like the sequential search, in order to remove one of the comparisons.

$data = array(4, 2, 4, 1, 2, 6, 8, 19, 3);

function insertion_sort_sentinel(&$arr)
{
	$len = count($arr);
	array_unshift(&$arr, -1);
	
	for ($i = 1; $i < $len+1; $i++) {
		$tmp = $arr[$i];
		$j = $i;
		
		while ($arr[$j-1] > $tmp) {
			$arr[$j] = $arr[$j-1];
			$j--;
		}
		$arr[$j] = $tmp;
	}
	array_shift(&$arr); // remove the sentinel
}

Just because we use searching the right position in an ordered array we can use binary search in order to improve even more the algorithm above. Unfortunately this doesn’t improve so much the general efficiency of this algorithm.

Complexity

As I said this algorithm is not so effective. Its complexity is O(n²) which is far worse than the O(n*log(n)) of quicksort, as you can see on the diagram bellow.

Application

This algorithm is useful for small sets of data and even if it doesn’t look like the most effective sorting algorithm, insertion sort can be useful for some reasons. First of all it is easy to implement, but it also does not require additional memory and it can be fast if the data is almost nearly sorted, which is great.

Typically a PHP programmer will write his SQL query as a string and will execute it via mysql_query.

$sql = "SELECT * FROM my_table";
$resource = mysql_query($sql);

So eventually when you want to dump this “complex” query, or whatever query there is, you can simply “echo” it and see what’s its syntax.

// this query is WRONG because of the where clause
$sql = "SELECT * FROM my_table WHERE id = ";

// dump and debug the wrong query
die($sql);

// this line won't be executed
$resource = mysql_query($sql);

So far so good, but things appear to be a bit different when you start to work with Zend Framework. Higher levels of abstraction come with slightly more difficult ways to dump (debug) your SQL queries.

OK you’ve two options. Using Zend_Db_Select or … not.

Dump Zend_Db_Select Query

Debugging queries generated with Zend_Db_Select is as easy as the following snippet. Let’s say there’s a db table “users” and typically the model is called Users.

class Users extends Zend_Db_Table
{
	protected $_name = 'users';

	public function getSomeUsers()
	{
		$select = $this->select()
				->from(this->_name)
				->where('id > ');

		$rows = $this->fetchAll($select);
	}
}

Obviously this code is wrong, because the “where” clause is wrong and we don’t pass any values to it. So now the question is how can we dump this SQL. Well, since we use Zend_Db_Select, so the simplest way to dump the SQL is by calling __toString(). The code above will become as follows.

class Users extends Zend_Db_Table
{
	protected $_name = ‘users’;

	public function getSomeUsers()
	{
		$select = $this->select()
				->from(this->_name)
				->where('id > ');

		die($select->__toString());

		$rows = $this->fetchAll($select);
	}
}

That’s great, but how can we dump update/insert queries for instance?

Dump with Zend_Db_Profiler

The way you should dump insert or update queries is a bit different. You should use Zend_Db_Profiler, which is a more complex way to debug (profile) your queries. You can not only debug insert or update queries, of course, but any queries generated by the Zend_Db abstraction. Let’s see how.

Now from a controller perspective this model can be just called as:

$users = new Users();
$users->update(array('name' => 'my name'), 'id =');

Obviously this code is wrong, again because of the where clause and now the question is how we can debug the generated SQL. The answer is: using Zend_Db_Profiler. Looking back again to the model, this should look like this.

class Users extends Zend_Db_Table
{
	protected $_name = ‘users’;

	public function updateSomeUsers()
	{
		// first enable the profiler
		$this->getProfiler()->setEnabled(true);

		// try to execute
		$this->update(array('name' => 'my name'), 'id = ');

		// dump the SQL
		Zend_Debug::dump($this->getProfiler()->getLastQueryProfile()->getQuery());

		// don't forget to disable the profiler is
		// you don't need it anymore
		$this->getProfiler()->setEnabled(false);
	}
}

The thing is that since Zend_Db is using prepared statements, the last row (the one with the Zend_Debug::…) will dump something like the following line.

UPDATE `users` SET `name` = ? WHERE (id =)

So as you can see, actually here you don’t see the name’s value, which should be “my name”. To see this you should call getQueryParams().

class Users extends Zend_Db_Table
{
	protected $_name = 'users';

	public function updateSomeUsers()
	{
		// first enable the profiler
		$this->getProfiler()->setEnabled(true);

		// try to execute
		$this->update(array('name' => 'my name'), 'id = ');

		// dump the SQL
		Zend_Debug::dump($this->getProfiler()->getLastQueryProfile()->getQuery());
		Zend_Debug::dump($this->getProfiler()->getLastQueryProfile()->getQueryParams());

		// don't forget to disable the profiler is
		// you don't need it anymore
		$this->getProfiler()->setEnabled(false);

Now the debug info will be something like this.

UPDATE `users` SET `name` = ? WHERE (id =)
array
	1 => string 'my name' (length 7)

Calling this from the controller there are some changes. Take a look at the example bellow.

$users = new Users();

// enable the profiler
$users->getAdapter()->getProfiler()->setEnabled(true);

// execute (or at least try to)
$users->update(array('name' => 'my name'), 'id =');

// debugdump the SQL
Zend_Debug::dump($users->getAdapter()->getProfiler()->getLastQueryProfile()->getQuery());
Zend_Debug::dump($users->getAdapter()->getProfiler()->getLastQueryProfile()->getQueryParams());

// don't forget to disable the profiler is
// you don't need it anymore
$users->getAdapter()->getProfiler()->setEnabled(false);

In this case you should use the longer model_name->getAdapter()

Two variants of run-length encoding are the diagram encoding and the pattern substitution algorithms. The diagram encoding is actually a very simple algorithm. Unlike run-length encoding, where the input stream must consists of many repeating elements, as “aaaaaaaa” for instance, which are very rare in a natural language, there are many so called “diagrams” in almost any natural language. In plain English there are some diagrams as “the”, “and”, “ing” (in the word “waiting” for example), “ a”, “ t”, “ e” and many doubled letters. Actually we can extend those diagrams by adding surrounding spaces. Thus we can encode not only “the”, but “ the “, which are 5 characters (2 spaces and 3 letters) with something shorter. In the other hand, as I said, in plain English there are two many doubled letters, which unfortunately aren’t something special for run-length encoding and the compression ratio will be small. Even worse the encoded text may happen to be longer than the input message. Let’s see some examples.

Let’s say we’ve to encode the message “successfully accomplished”, which consists of four doubled letters. However to compress it with run-length encoding we’ll need at least 8 characters, which doesn’t help us a lot.

// 8 chars replaced by 8 chars!?
input: 	"successfully accomplished"
output:	"su2ce2sfu2ly a2complished"

The problem is that if the input text contains numbers, “2” in particular, we’ve to chose an escape symbol (“@” for example), which we’ll use to mark where the encoded run begins. Thus if the input message is “2 successfully accomplished tasks”, it will be encoded as “2 su@2ce@2sfu@2ly a@2complished tasks”. Now the output message is longer!!! than the input string.

// the compressed message is longer!!!
input:	"2 successfully accomplished"
output:	"2 su@2ce@2sfu@2ly a@2complished tasks"

Again if the input stream contains the escape symbol, we have to find another one, and the problem is that it is often too difficult to find short escape symbol that doesn’t appear in the input text, without a full scan of the text.

That is why run-length encoding isn’t a good solution when compressing plain text, where long runs rarely appear. Well, of course, there are exceptions. For example such an exception is the lossy text compression with run-length encoding. It is intuitively clear that compressing text with loss is rarely useful, especially when you’ve to decompress exactly the same text. However there are some cases that lossy compression may be useful. Such case can be removing spaces. Indeed the text “successfully accomplished” brings us exactly the same information as “successfully accomplished”. In this case we can simply remove those spaces. Indeed we can use a marker to indicate the long run of spaces like “successfully@6 accomplished” in order to decompress the input string with absolutely no loss, but we can also throw those symbols away. This desision depends on the goal. Exactly with the same goal in mind we can remove new lines and tabs, only if we’re sure that the sense of the text is preserved. Yet again, a problem is that such long runs don’t happen to occur in random texts. That is why it’s better to use diagram encoding for plain text compression instead of run-length encoding.

Few Questions

After understanding the principles of the diagram encoding, let’s see some examples. In the example above it is better to replace doubled letters with something shorter. Let’s say # for “cc”, @ for “ss” and % for “ll”. Thus the input text will be compressed as “su#e@fu%y a#omplished”, which is shorter. But yet again what will happen if the input message contains one of the substitutions? Also we can’t say if there are many doubled letters and enough reasonable substitutions for them. A better approach is to replace patterns.

Pattern Substitution

The pattern substitution algorithm is a variant of the diagram encoding. As I said above in plain English a very commonly used pattern can be “ the “, which is five characters long. We can now replace it with something like “$%” for example. In this case the message “I send the message” will become “I send$%message”. However there are some obstacles to overcome.

The first problem is that we need to know the language and somehow to define commonly used patterns in a dictionary. What would happen with a message written in some language we don’t know nothing about. Let’s say – Latin like the example bellow.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Cras venenatis, sapien eget suscipit placerat, justo quam blandit mauris, quis tempor ante sapien sodales augue. Praesent ut mauris quam. Phasellus scelerisque, ante quis consequat tristique, metus turpis consectetur leo, vitae facilisis sapien mi eu sapien. Praesent vitae ligula elit, et faucibus augue. Sed rhoncus sodales dolor ut gravida. In quis augue ac nulla auctor mattis sed sed libero. Donec eget purus eget enim tempor porta vitae eget diam. Mauris aliquet malesuada ipsum, non pulvinar urna vestibulum ac. Donec feugiat velit vitae nunc cursus imperdiet. Donec accumsan faucibus dictum. Phasellus sed mauris sapien. Maecenas mi metus, tincidunt sed rhoncus nec, sodales non sapien.

Clearly without knowing Latin it isn’t easy to define which are those commonly used patterns. The thing is that it’s better to use pattern substitution if you know in advance the set of words and characters.

The second problem is related to decompression. It is obvious that we need to define a dictionary and this dictionary must be used when decoding the message. It will be great also if we find more patterns longer than three characters. If not, the compression ratio will be low. Unfortunately such patterns aren’t very common in any natural language.

Application

It is interesting to answer the question, how to use diagram encoding or patter substitution to compress text in natural language, especially when we don’t know the language in detail? The answer hides in the question. We wont compress natural languages, but machine language. Exactly machine (programming) languages are limited to a smaller sets of words and symbols. Isn’t it true for any programing language? Like PHP, where words like “function”, “while”, “for”, “break”, “switch”, “foreach” happen to be often in use, or HTML with its defined set of tags. Perhaps the best example is CSS, where only the values of the properties can vary. CSS files also tend to have multiple new lines, tabs and spaces, which only humans read.

The question here is why should we compress those file types. It’s clear that after the compression they will be completely useless, both for humans and machines. Yes, that is true, but what if we have to store versions of those files into a DB. Kind of a backup. Imagine you’re working for a web hosting company that has to store daily versions of the sites it’s hosting. Thus the volume of stored information even for small companies hosting only few sites can be enormous. The problem is that compressing those files with some conventional compressing tool isn’t a good idea. Thus we’ve to save a copy of the entire site every day, but as we know the difference between daily versions of a site can be small. A version control system is another solution, but then you’ve to store the plain text of the files.

Perhaps a better approach is to compress the text using pattern substitution and then saving only differences – kind of version control, which can be done with “relative encoding”.

Using the above method we can save lots of disk space and in the same time we can compress/decompress easily. Another good thing is that you can save only changes to the initial files, like version control, which can also be compressed.

Implementation

The implementation of this algorithm is again on PHP and tries only to describe the main principles of compression. In this case I tried to compress a CSS file using the compression above. Although this example is quite primitive we can see some interesting facts. First of all you only need encoding and decoding dictionaries. Practically the encoding and decoding processes are equal, so you don’t need to implement two different functions. Here in this example a native PHP function is used – str_replace, because the purpose of this algorithm is not to describe pattern substitution techniques, but pattern substitution. It assumes that today’s programming languages have string manipulation functions for the purposes of this task.

$str = file_get_contents('large_style_file.css');

$encoding_dict = array(
	"\n" 		=> '$0',
	'text' 		=> '$1',
	'color' 	=> '$2',
	'display' 	=> '$3',
	'font' 		=> '$4',
	'width' 	=> '$5',
	'height'	=> '$6',	
	' '		=> '',
);

function replace_patterns($input, $dict) 
{
	foreach ($dict as $pattern => $replace) {
		$input = str_replace($pattern, $replace, $input);
	}
	
	return $input;
}

$result = replace_patterns($str, $encoding_dict);

By only replacing few CSS properties I achieved almost 40% of compression ratio (as shows the diagram bellow). The initial file is 202 KB, while compressed it’s only 131 KB. Of course, it all depends on the CSS file, but how about replacing all property names with shorter ones. Perhaps then the compression will be even better.

In my previous post we saw how to compress data consisting of very long runs of repeating elements. This type of compression is known as “run-length encoding” and can be very handy when transferring data with no loss. The problem is that the data must follow a specific format. Thus the string “aaaaaaaabbbbbbbb” can be compressed as “a8b8”. Now a string with length 16 can be compressed as a string with length 4, which is 25% of its initial length without loosing any information. There will be a problem in case the characters (elements) were dispersed in a different way. What would happen if the characters are the same, but they don’t form long runs? What if the string was “abababababababab”? The same length, the same characters, but we cannot use run-length encoding! Indeed using this algorithm we’ll get at best the same string.

In this case, however, we can see another fact. The string consists of too many repeating elements, although not arranged one after another. We can compress this string with a bitmap. This means that we can save the positions of the occurrences of a given element with a sequence of bits, which can be easily converted into a decimal value. In the example above the string “abababababababab” can be compressed as “1010101010101010”, which is 43690 in decimals, and even better AAAA in hexadecimal. Thus the long string can be compressed. When decompressing (decoding) the message we can convert again from decimal/hexadecimal into binary and match the occurrences of the characters. Well, the example above is too simple, but let’s say only one of the characters is repeating and the rest of the string consists of different characters like this: “abacadaeafagahai”. Then we can use bitmap only for the character “a” – “1010101010101010” and compress it as “AAAA bcdefghi”. As you can see all the example strings are exactly 16 characters and that is a limitation. To use bitmaps with variable length of the data is a bit tricky and it is not always easy (if possible) to decompress it.

$data = array(
	0 	=> 1991,
	1 	=> 1992,
	2 	=> 1993,
	3 	=> 1994,
	4 	=> 1991,
	5 	=> 1992,
	6 	=> 1993,
	7 	=> 1992,
	8 	=> 1991,
	9 	=> 1991,
	10 	=> 1991,
	11 	=> 1992,
	12 	=> 1992,
	13 	=> 1991,
	14 	=> 1991,
	15 	=> 1992,	
	...
);

The JSON will encoded message will be the following (a simple but yet very large javascript array).

[1991,1992,1993,1994,1991,1992,1993,1992,1991,1991,1991,1992,1992,1991,1991,1992, ...]

However if we use bitmap compression we’ll get a “shorter” array.

$data = array(
	0 => array(1991, '1000100011100110'),
	1 => array(1992, '0100010100011001'),
	2 => array(1993, '0010001000000000'),
	3 => array(1994, '0001000000000000'),
);

Now the JSON is:

[[1991,"1000100011100110"],[1992,"0100010100011001"],[1993,"0010001000000000"],[1994,"0001000000000000"]]

It is obvious that the compression ratio is getting better and better as the uncompressed data grows. In fact, most of us know bitmap compression from images, because this algorithm is largely used for image compression. We can imagine how successful it can be when compressing black and white images (as black and white can be represented as 0 and 1s). Actually it is used for more than two colors (256 for instance) and again the level of compression is very high.

Implementation

The following implementation on PHP aims only to illustrate the bitmap compressing algorithm. As we know this algorithm can be applicable for any kind of data structures.

// too many repeating "a" characters
$msg = 'aazahalavaatalawacamaahakafaaaqaaaiauaacaaxaauaxaaaaaapaayatagaaoafaawayazavaaaazaaabararaaaaakakaaqaarazacajaazavanazaaaeanaaoajauaaaaaxalaraaapabataaavaaab';

function bitmap($message) 
{
	$i = 0;
	$bits = $rest = '';
	
	while ($v = $message[$i]) {
		if ($v == 'a') {
			$bits .= '1';
		} else {
			$bits .= '0';
			$rest .= $v;
		}
		$i++;
	}
	
	return number_format(bindec($bits), 0, '.', '') . $rest;;
}

echo bitmap($msg);

// uncompressed: 
acaaaaadaaaabalaaeaaaaganaaxakaavawamaasavajawaaaayaauaaadalanagaeaeamaarafalaazaaaiasaanaahaaazaraxaalaahaaawaaajasamahaajaakarapanaakaoakaanawalaacamauaamaal
// compressed:
152299251941730035874325065523548237677352452096zhlvtlwcmhkfqiucxuxpytgofwyzvzbrrkkqrzcjzvnzenojuxlrpbtvb

Application

This algorithm is very useful when there is an element in our data that repeats very often, so you need to investigate the nature of the data you want to compress. Actually because of this fact this algorithm is used for image compression as PNG8 or GIF.

No matter how fast today’s computers and networks are, the users will constantly need faster and faster services. To reduce the volume of the transferred data we usually use some sort of compression. That is why this computer sciences area will be always interesting to research and develop.

There are many data compression algorithms, some of them lossless, others lossy, but their main goal aways will be to spare storage space and traffic. These algorithms are very useful when talking about data transfer between two distant places. Perhaps the best example is the transfer between a web server and a browser.

In the last few years a lot of research has been done on compressing files, executed on the client side. Such files are javascript, css, htmls and images. In fact servers and clients already have some techniques to compress data, like using GZIP for instance, that can dramatically decrease the transfer. In the other hand there are lots of tools and tricks in order to decrease the size of the data.

Actually when a file is executed by the client’s virtual machine, it doesn’t matter how “beautifully” it is formatted from a programmer’s point of view. Thus the spaces, tabs and the new lines don’t bring any significant information for the environment. That is why such compressing tools like YUI Compressor, Google Closure Compiler, etc. remove those symbols. Well, they can achieve even more in order to improve the compression rate. In this post I won’t cover this, but this shows how important data compression algorithms are.

It would be great if we could just compress data with some tool. Unfortunately this is not the case and usually the compression rate depends on the data itself. It is obvious that the choice of data compression algorithm depends mainly on the data and first of all we must explore the data.

Here I’ll cover one very simple lossless data compression algorithm called “run-length encoding” that can be very useful in some cases.

Overview

This algorithm consists of replacing large sequences of repeating data with only one item of this data followed by a counter showing how many times this item is repeated. To become clearer let’s see a string example.

aaaaaaaaaabbbaxxxxyyyzyx

This string’s length is 24 and as we can see there are lots of repetitions. Using the run-length algorithm, we replace any run with shorter string followed by a counter.

a10b3a1x4y3z1y1x1

The length of this string is 17, which is approximately 70% of the initial length. Obviously this is not the optimal way to compress the given string. For instance we don’t need to use the digit “1” when the character is repeated only once. In some cases this approach can increase the length of the initial string which is exactly the opposite of what we need. In this case we’ll get the string bellow.

a10b3ax4y3zyx

Now the length of the resulting string is 13, which is 54% of the initial length! A variation of the example above is not to keep a counter of the repetitions of the character, but their position instead. Thus the initial string will be compressed as follows.

a0b10a13x14y18z21y22x23

Which of these two approaches you’ll use depends on the goal. In the second case we can achieve a good optimization of binary search.

It is clear that this algorithm is not only applicable on strings. We can achieve very good results on arrays. A typical example is the transfer of JSON from a server to a client. Then if there are large sequences of repeating data we can achieve great results.

Implementation

The implementation bellow is assuming that we’re compressing a string and it’s written on PHP. However the nature of this algorithm doesn’t restrict us to use only strings. As I said before with slight modifications we can use it with other data structures. It is important only to understand that the run-length algorithm is very useful on large sequences of repeating elements, no matter characters or array items.

$message = 'aaaaaaaaaabbbaxxxxyyyzyx';

function run_length_encode($msg)
{
	$i = $j = 0;
	$prev = '';
	$output = '';
	
	while ($msg[$i]) {
		if ($msg[$i] != $prev) {
			
			if ($i) 
				$output .= $j;
				
			$output .= $msg[$i];
				
			$prev = $msg[$i];
			
			$j = 0;
		}
		$j++;
		$i++;
	}
	
	$output .= $j;
	
	return $output;
}

// a10b3a1x4y3z1y1x1
echo run_length_encode($message);

And slightly optimized.

$message = 'aaaaaaaaaabbbaxxxxyyyzyx';

function run_length_encode($msg)
{
	$i = $j = 0;
	$prev = '';
	$output = '';
	
	while ($msg[$i]) {
		if ($msg[$i] != $prev) {
			
			if ($i && $j > 1) 
				$output .= $j;
				
			$output .= $msg[$i];
				
			$prev = $msg[$i];
			
			$j = 0;
		}
		$j++;
		$i++;
	}
	
	if ($j > 1)
		$output .= $j;
	
	return $output;
}

// a10b3ax4y3zyx
echo run_length_encode($message);

Finally a small change – now we store the position of the character.

$message = 'aaaaaaaaaabbbaxxxxyyyzyx';

function run_length_encode($msg)
{
	$i = 0;
	$prev = '';
	$output = '';
	
	while ($msg[$i]) {
		if ($msg[$i] != $prev) {
				
			$output .= $msg[$i] . $i;
				
			$prev = $msg[$i];
			
		}

		$i++;
	}
	
	return $output;
}

// a0b10a13x14y18z21y22x23
echo run_length_encode($message);

Complexity and Data Compression

We’re used to talk about complexity of an algorithm measuring time and we usually try to find the fastest implementation, like in search algorithms. Here it is not so important to compress data quickly, but to compress as much as possible so the output is as small as possible without lossing data. A great feature of run-length encoding is that this algorithm is easy to implement.

Application

We can use run-length encoding in many cases. It is commonly used to compress images and is very successful when we deal only with black and white images. Here I’ll cover another use case that I only mentioned above. Let’s say we have to transfer a very large array of data to our AJAX-powered application using JSON. Let’s say also that the data are some years, for instance the years of the premiere of a movie. There are lots of movies with a premiere in the same year, thus although the data is sorted, we actually can’t have any benefit. More important is that we have large sequences of data. Here we can use run-length encoding.

$data = array(
	0 	=> 1991,
	1 	=> 1991,
	...
	2223 	=> 1991,
	2224 	=> 1992,
	...
	19298 	=> 1995,
	19299 	=> 1996,
	...
);

As you can see to transfer the whole array can be a nightmare, especially on slow networks. It is better to compress it (i.e. with PHP’s json_encode).

// {"0":1991,"1":1991, ..., "2223":1991,"2224":1992, ..., "19298":1995,"19299":1996, ...}
echo json_encode($data);

After running run-length encoding we can receive something like the following array (note that these are only sample data and it’s up to you to decide which is the best format to store data).

$data = array(
	0 => array(1991, 2224),
	1 => array(1992, 3948),
	2 => array(1995, 2398),
	3 => array(1996, 3489),
);

And the JSON output.

// [[1991,2224],[1992,3948],[1995,2398],[1996,3489]]
echo json_encode($data);

Note that if the data is sorted we can achieve great success compressing it!!! This approach can be used for images, graphics or map coordinates.

This is only one example of how data compression can be useful in our daily work. Although the communication between the server and the client can be optimized and compressed, we can improve it. In other words we’re not always sure that the opposite side supports compression.

Well, it’s true that the client has to decompress the data, which can also be slow. Now in the first case we have only the time to transfer, as on the diagram bellow.

In the second case, we should sum the time for compression, transfer and decompression.

All this is important, but in general data compression can be handy in many cases in our daily work.

Important Note

It’s very important to say that the following results are dependant from the machine and the environment!

Source Code

Here’s the PHP source code.

function divide($n = 1) 
{
	$a = microtime(true);
	for ($i = 0; $i < $n; $i++) {
		300/2;
	}
	echo microtime(true) - $a;
}

divide(100);
//divide(1000);
//divide(10000);
//divide(100000);
//divide(1000000);
//divide(10000000);

and bitwise ...

function bitwise($n = 1)
{
	$a = microtime(true);
	for ($i = 0; $i < $n; $i++) {
		300 >> 1;
	}
	echo microtime(true) - $a;
}

bitwise(100);
//bitwise(1000);
//bitwise(10000);
//bitwise(100000);
//bitwise(1000000);
//bitwise(10000000);

Note that each method was called 6 times with the same parameter. This means that divide(100) was called 6 times and then I used the average value of these six times.

Results

I said back then in my binary search post, that in PHP the bitwise operator ">> 1" is not faster than the typical division with the "/" operator. However the results tells us that using bitwise division is slightly faster, as you can see at the diagram bellow.

n		">>"			"/"
100		0.0002334912618		0.000311803817749
1000		0.001911004384359	0.007335503896078
10000		0.013423800468445	0.039460102717081
100000		0.14417803287506	0.21413381894429
1000000		1.15839115778605	1.17152162392935
10000000	10.556711634		11.0911625623705

Conclusion

Although bitwise division is a bit faster the difference is so small that you should work with very large data in order to gain some performance.