Relative encoding is another data compression algorithm. While run-length encoding, bitmap encoding and diagram and pattern substitution were trying to reduce repeating data, with relative encoding the goal is a bit different. Indeed run-length encoding was searching for long runs of repeating elements, while pattern substitution and bitmap encoding were trying to “map” where the repetitions happen to occur.

The only problem with these algorithms is that not always the input stream of data is constructed out of repeating elements. It is clear that if the input stream contains many repeating elements there must be some way of reducing them. However that doesn’t mean that we cannot compress data if there are no repetitions. It all depends on the data. Let’s say we have the following stream to compress.

1, 2, 3, 4, 5, 6, 7

We can hardly imagine how this stream of data can be compressed. The same problem may occur when trying to compress the alphabet. Indeed the alphabet letters the very base of the words so it is the minimal part for word construction and it’s hard to compress them.

Fortunately this isn’t true always. An algorithm that tryies to deal with non repeating data is relative encoding. Let’s see the following input stream – years from a given decade (the 90’s).

1991,1991,1999,1998,1991,1993,1992,1992

Here we have 39 characters and we can reduce them. A natural approach is to remove the leading “19” as we humans often do.

91,91,99,98,91,93,92,92

Now we have a shorter string, but we can go even further with keeping only the first year. All other years will as relative to this year.

91,0,8,7,0,2,1,1

Now the volume of transferred data is reduced a lot (from 39 to 16 – more than 50%). However there are some questions we need to answer first, because the stream wont be always formatted in such pretty way. How about the next character stream?

91,94,95,95,98,100,101,102,105,110

We see that the value 100 is somehow in the middle of the interval and it is handy to use it as a base value for the relative encoding. Thus the stream above will become:

-9,-6,-5,-5,-2,100,1,2,5,10

The problem is that we can’t decide which value will be the base value so easily. What if the data was dispersed in a different way.

96,97,98,99,100,101,102,103,999,1000,1001,1002

Now the value of “100” isn’t useful, because compressing the stream will get something like this:

-4,-3,-2,-1,100,1,2,3,899,900,901,902

To group the relative values around “some” base values will be far more handy.

(-4,-3,-2,-1,100,1,2,3)(-1,1000,1,2)

However to decide which value will be the base value isn’t that easy. Also the encoding format is not so trivial. In the other hand this type of encoding can be useful in som specific cases as we can see bellow.

Implementation

The implementation of this algorithm depends on the specific task and the format of the data stream. Assuming that we’ve to transfer the stream of years in JSON from a web server to a browser, here’s a short PHP snippet.

// JSON: [1991,1991,1999,1998,1999,1998,1995,1997,1994,1993]
$years = array(1991,1991,1999,1998,1999,1998,1995,1997,1994,1993);

function relative_encoding($input)
{
	$output = array();
	$inputLength = count($input);
	
	$base = $input[0];
	
	$output[] = $base;
	
	for ($i = 1; $i < $inputLength; $i++) {
		$output[] = $input[$i] - $base;
	}
	
	return $output;
}

// JSON: [1991,0,8,7,8,7,4,6,3,2]
echo json_encode(relative_encoding($years));

Application

This algorithm may be very useful in many cases, but here’s one of them. There are plenty of map applications around the web. Some products as Google Maps, Yahoo! Maps, Bing Maps are quite famous, while there are very useful open source projects as OpenStreetMap. The web sites using these apps are thousands.

A typical use case is to transfer lots of Geo coordinates from web server to a browser using JSON. Indeed any GEO point on Earth is relative to the point (0,0), which is located near the west coast of Africa, however on large zoom levels, when there are tons of markers we can transfer the information with relative encoding.

For instance the following diagram shows San Francisco with some markers on it. Their coordinates are be relative to the point (0,0) on Earth.

Far more useful may be to encode those markers, relative to the center of the city, thus we can save some space.

However this type of compression can be tricky, for example when dragging the map and updating the marker array. In the other hand we must group markers if we have to load more than one city. That’s why we must be careful when implementing it. But in the other hand it can be very useful - for instance on initial load of the map we can reduce data and speed up the load time.

The thing is that with relative encoding we can save only changes to base value (data) - something like version control systems and thus reducing data transfer and load. Here's a graphical example. In the first case on the diagram bellow we can see that each item is stored on its own. It doesn't depend on the adjacent items and it can be completely independent of them.

However we can keep full info only for the first item and any other item will be relative to it, like on the diagram bellow.

In my previous post we saw how to compress data consisting of very long runs of repeating elements. This type of compression is known as “run-length encoding” and can be very handy when transferring data with no loss. The problem is that the data must follow a specific format. Thus the string “aaaaaaaabbbbbbbb” can be compressed as “a8b8”. Now a string with length 16 can be compressed as a string with length 4, which is 25% of its initial length without loosing any information. There will be a problem in case the characters (elements) were dispersed in a different way. What would happen if the characters are the same, but they don’t form long runs? What if the string was “abababababababab”? The same length, the same characters, but we cannot use run-length encoding! Indeed using this algorithm we’ll get at best the same string.

In this case, however, we can see another fact. The string consists of too many repeating elements, although not arranged one after another. We can compress this string with a bitmap. This means that we can save the positions of the occurrences of a given element with a sequence of bits, which can be easily converted into a decimal value. In the example above the string “abababababababab” can be compressed as “1010101010101010”, which is 43690 in decimals, and even better AAAA in hexadecimal. Thus the long string can be compressed. When decompressing (decoding) the message we can convert again from decimal/hexadecimal into binary and match the occurrences of the characters. Well, the example above is too simple, but let’s say only one of the characters is repeating and the rest of the string consists of different characters like this: “abacadaeafagahai”. Then we can use bitmap only for the character “a” – “1010101010101010” and compress it as “AAAA bcdefghi”. As you can see all the example strings are exactly 16 characters and that is a limitation. To use bitmaps with variable length of the data is a bit tricky and it is not always easy (if possible) to decompress it.

$data = array(
	0 	=> 1991,
	1 	=> 1992,
	2 	=> 1993,
	3 	=> 1994,
	4 	=> 1991,
	5 	=> 1992,
	6 	=> 1993,
	7 	=> 1992,
	8 	=> 1991,
	9 	=> 1991,
	10 	=> 1991,
	11 	=> 1992,
	12 	=> 1992,
	13 	=> 1991,
	14 	=> 1991,
	15 	=> 1992,	
	...
);

The JSON will encoded message will be the following (a simple but yet very large javascript array).

[1991,1992,1993,1994,1991,1992,1993,1992,1991,1991,1991,1992,1992,1991,1991,1992, ...]

However if we use bitmap compression we’ll get a “shorter” array.

$data = array(
	0 => array(1991, '1000100011100110'),
	1 => array(1992, '0100010100011001'),
	2 => array(1993, '0010001000000000'),
	3 => array(1994, '0001000000000000'),
);

Now the JSON is:

[[1991,"1000100011100110"],[1992,"0100010100011001"],[1993,"0010001000000000"],[1994,"0001000000000000"]]

It is obvious that the compression ratio is getting better and better as the uncompressed data grows. In fact, most of us know bitmap compression from images, because this algorithm is largely used for image compression. We can imagine how successful it can be when compressing black and white images (as black and white can be represented as 0 and 1s). Actually it is used for more than two colors (256 for instance) and again the level of compression is very high.

Implementation

The following implementation on PHP aims only to illustrate the bitmap compressing algorithm. As we know this algorithm can be applicable for any kind of data structures.

// too many repeating "a" characters
$msg = 'aazahalavaatalawacamaahakafaaaqaaaiauaacaaxaauaxaaaaaapaayatagaaoafaawayazavaaaazaaabararaaaaakakaaqaarazacajaazavanazaaaeanaaoajauaaaaaxalaraaapabataaavaaab';

function bitmap($message) 
{
	$i = 0;
	$bits = $rest = '';
	
	while ($v = $message[$i]) {
		if ($v == 'a') {
			$bits .= '1';
		} else {
			$bits .= '0';
			$rest .= $v;
		}
		$i++;
	}
	
	return number_format(bindec($bits), 0, '.', '') . $rest;;
}

echo bitmap($msg);

// uncompressed: 
acaaaaadaaaabalaaeaaaaganaaxakaavawamaasavajawaaaayaauaaadalanagaeaeamaarafalaazaaaiasaanaahaaazaraxaalaahaaawaaajasamahaajaakarapanaakaoakaanawalaacamauaamaal
// compressed:
152299251941730035874325065523548237677352452096zhlvtlwcmhkfqiucxuxpytgofwyzvzbrrkkqrzcjzvnzenojuxlrpbtvb

Application

This algorithm is very useful when there is an element in our data that repeats very often, so you need to investigate the nature of the data you want to compress. Actually because of this fact this algorithm is used for image compression as PNG8 or GIF.

No matter how fast today’s computers and networks are, the users will constantly need faster and faster services. To reduce the volume of the transferred data we usually use some sort of compression. That is why this computer sciences area will be always interesting to research and develop.

There are many data compression algorithms, some of them lossless, others lossy, but their main goal aways will be to spare storage space and traffic. These algorithms are very useful when talking about data transfer between two distant places. Perhaps the best example is the transfer between a web server and a browser.

In the last few years a lot of research has been done on compressing files, executed on the client side. Such files are javascript, css, htmls and images. In fact servers and clients already have some techniques to compress data, like using GZIP for instance, that can dramatically decrease the transfer. In the other hand there are lots of tools and tricks in order to decrease the size of the data.

Actually when a file is executed by the client’s virtual machine, it doesn’t matter how “beautifully” it is formatted from a programmer’s point of view. Thus the spaces, tabs and the new lines don’t bring any significant information for the environment. That is why such compressing tools like YUI Compressor, Google Closure Compiler, etc. remove those symbols. Well, they can achieve even more in order to improve the compression rate. In this post I won’t cover this, but this shows how important data compression algorithms are.

It would be great if we could just compress data with some tool. Unfortunately this is not the case and usually the compression rate depends on the data itself. It is obvious that the choice of data compression algorithm depends mainly on the data and first of all we must explore the data.

Here I’ll cover one very simple lossless data compression algorithm called “run-length encoding” that can be very useful in some cases.

Overview

This algorithm consists of replacing large sequences of repeating data with only one item of this data followed by a counter showing how many times this item is repeated. To become clearer let’s see a string example.

aaaaaaaaaabbbaxxxxyyyzyx

This string’s length is 24 and as we can see there are lots of repetitions. Using the run-length algorithm, we replace any run with shorter string followed by a counter.

a10b3a1x4y3z1y1x1

The length of this string is 17, which is approximately 70% of the initial length. Obviously this is not the optimal way to compress the given string. For instance we don’t need to use the digit “1” when the character is repeated only once. In some cases this approach can increase the length of the initial string which is exactly the opposite of what we need. In this case we’ll get the string bellow.

a10b3ax4y3zyx

Now the length of the resulting string is 13, which is 54% of the initial length! A variation of the example above is not to keep a counter of the repetitions of the character, but their position instead. Thus the initial string will be compressed as follows.

a0b10a13x14y18z21y22x23

Which of these two approaches you’ll use depends on the goal. In the second case we can achieve a good optimization of binary search.

It is clear that this algorithm is not only applicable on strings. We can achieve very good results on arrays. A typical example is the transfer of JSON from a server to a client. Then if there are large sequences of repeating data we can achieve great results.

Implementation

The implementation bellow is assuming that we’re compressing a string and it’s written on PHP. However the nature of this algorithm doesn’t restrict us to use only strings. As I said before with slight modifications we can use it with other data structures. It is important only to understand that the run-length algorithm is very useful on large sequences of repeating elements, no matter characters or array items.

$message = 'aaaaaaaaaabbbaxxxxyyyzyx';

function run_length_encode($msg)
{
	$i = $j = 0;
	$prev = '';
	$output = '';
	
	while ($msg[$i]) {
		if ($msg[$i] != $prev) {
			
			if ($i) 
				$output .= $j;
				
			$output .= $msg[$i];
				
			$prev = $msg[$i];
			
			$j = 0;
		}
		$j++;
		$i++;
	}
	
	$output .= $j;
	
	return $output;
}

// a10b3a1x4y3z1y1x1
echo run_length_encode($message);

And slightly optimized.

$message = 'aaaaaaaaaabbbaxxxxyyyzyx';

function run_length_encode($msg)
{
	$i = $j = 0;
	$prev = '';
	$output = '';
	
	while ($msg[$i]) {
		if ($msg[$i] != $prev) {
			
			if ($i && $j > 1) 
				$output .= $j;
				
			$output .= $msg[$i];
				
			$prev = $msg[$i];
			
			$j = 0;
		}
		$j++;
		$i++;
	}
	
	if ($j > 1)
		$output .= $j;
	
	return $output;
}

// a10b3ax4y3zyx
echo run_length_encode($message);

Finally a small change – now we store the position of the character.

$message = 'aaaaaaaaaabbbaxxxxyyyzyx';

function run_length_encode($msg)
{
	$i = 0;
	$prev = '';
	$output = '';
	
	while ($msg[$i]) {
		if ($msg[$i] != $prev) {
				
			$output .= $msg[$i] . $i;
				
			$prev = $msg[$i];
			
		}

		$i++;
	}
	
	return $output;
}

// a0b10a13x14y18z21y22x23
echo run_length_encode($message);

Complexity and Data Compression

We’re used to talk about complexity of an algorithm measuring time and we usually try to find the fastest implementation, like in search algorithms. Here it is not so important to compress data quickly, but to compress as much as possible so the output is as small as possible without lossing data. A great feature of run-length encoding is that this algorithm is easy to implement.

Application

We can use run-length encoding in many cases. It is commonly used to compress images and is very successful when we deal only with black and white images. Here I’ll cover another use case that I only mentioned above. Let’s say we have to transfer a very large array of data to our AJAX-powered application using JSON. Let’s say also that the data are some years, for instance the years of the premiere of a movie. There are lots of movies with a premiere in the same year, thus although the data is sorted, we actually can’t have any benefit. More important is that we have large sequences of data. Here we can use run-length encoding.

$data = array(
	0 	=> 1991,
	1 	=> 1991,
	...
	2223 	=> 1991,
	2224 	=> 1992,
	...
	19298 	=> 1995,
	19299 	=> 1996,
	...
);

As you can see to transfer the whole array can be a nightmare, especially on slow networks. It is better to compress it (i.e. with PHP’s json_encode).

// {"0":1991,"1":1991, ..., "2223":1991,"2224":1992, ..., "19298":1995,"19299":1996, ...}
echo json_encode($data);

After running run-length encoding we can receive something like the following array (note that these are only sample data and it’s up to you to decide which is the best format to store data).

$data = array(
	0 => array(1991, 2224),
	1 => array(1992, 3948),
	2 => array(1995, 2398),
	3 => array(1996, 3489),
);

And the JSON output.

// [[1991,2224],[1992,3948],[1995,2398],[1996,3489]]
echo json_encode($data);

Note that if the data is sorted we can achieve great success compressing it!!! This approach can be used for images, graphics or map coordinates.

This is only one example of how data compression can be useful in our daily work. Although the communication between the server and the client can be optimized and compressed, we can improve it. In other words we’re not always sure that the opposite side supports compression.

Well, it’s true that the client has to decompress the data, which can also be slow. Now in the first case we have only the time to transfer, as on the diagram bellow.

In the second case, we should sum the time for compression, transfer and decompression.

All this is important, but in general data compression can be handy in many cases in our daily work.

JavaScript, being a functional language, differs from most of the procedural/object oriented languages we know. The object oriented approach in JavaScript is rather strange. However there is much power in making objects! The syntax is really odd and there are several approaches.

Literal Notation

As many of you may know the most used notation is the JSON (JavaScript Object Notation).

{ 'key1' : 'val1'
, 'key2' : 'val2'
, 'key3' : 'val3'
}

Of course this is the very basic example. You can use as value any JavaScript object – another similar object or a function.

{ 'key1' : 'val1'
, 'key2' : { 'inner_key1' : 'inner_val1' }
, 'key3' : function() {
			return 10 + 5;
		 }
}

The two examples above are showing an anonymous object in JavaScript, but you can assign this code to some variable.

var myObject = 
	{ 'key1' : 'val1'
	, 'key2' : 'val2'
	, 'key3' : 'val3'
	}

or

var myObject =
	{ 'key1' : 'val1'
	, 'key2' : { 'inner_key1' : 'inner_val1' }
	, 'key3' : function() {
				return 10 + 5;
			 }
	}

and then you can call the properties of these objects with the ‘.’ operator:

myObject.key1;
myObject.key2.inner_key1;
myObject.key3();

So far so good – this is the literal object notation in JavaScript. However there is another “objects” in JavaScript.

Objects and Functions in JavaScript

Actually in JS the functions are also objects (classes). So you can define a function and then define another function inside or use the “this” operator.

var a = function(name)
{
	this.name = name;
	
	this.print = function() {
		alert(this.name);	
	}
}

Then you can make a new instance of this “class”.

var myObject = new a('hello world');
var anotherObject = new a('some test');

myObject.print(); // will print the string "hello world"
anotherObject.print(); // will print the string "some test"

Thus the function “a” defined within the code “var a = function(name) …” is a class in JavaScript. You can make instances (objects) of this class with the “new” operator.

You can use another notation for defining the function “a”, so both can be used and it’s up to you to decide which one is more convenient.

function a(name) 
{
	this.name = name;
	
	this.print = function() {
		alert(this.name);	
	}	
}

Prototypes

JS gives you more power. In most of the languages you’ve to define the class first and then make objects. In JS you can define a “class” then make some objects and then add some properties to the class by using the prototype.

var a = function(name) 
{
	this.name = name;
}

var myObj = new a('hello world');

a.prototype.print = function() {
	alert(this.name);	
}

myObj.print();

So you can see how you can add functions and properties to an already defined class using the prototype. Thus you can call the “print” function on the myObj object.

Inheritance

The prototype built-in property can be used to make use of one of the main OOP features – inheritance.

var SuperClass = function()
{
	this.superFunction = function() {
		return 'function in Super Class';	
	}
}

var sup = new SuperClass();
sup.superFunction(); // will return the string "function in Super Class"

Now you can add one more class and set its prototype to the SuperClass

var SubClass = function()
{
	this.subFunction = function() {
		return 'function in Sub Class';	
	}	
}

SubClass.prototype = new SuperClass();

Thus you make SubClass inherit from SuperClass. Now you can call the SuperClass properties from any SubClass object.

var sub = new SubClass();
sub.superFunction(); // will return the string "function in Super Class"
sub.subFunction(); // will return the string "function in Sub Class"

Even more! You can add properties to the super class with prototype.

SuperClass.prototype.myFunc = function() {
	return 'my new function';	
}

And then call it from the sub class:

sub.myFunc(); // will return the string "my new function"

Override a Function

What happens if you have the same function name in both classes. Actually this overrides the method in the sub class.

var SuperClass = function()
{
	this.myFunc = function() {
		return 'function in Super Class';	
	}
}

var SubClass = function()
{
	this.myFunc = function() {
		return 'function in Sub Class';	
	}	
}

SubClass.prototype = new SuperClass();

var sub = new SubClass();
sub.myFunc(); // will return the string "function in Sub Class"

while

var sup = new SuperClass();
sup.myFunc(); // will return the string "function in Super Class"

Conclusion

Now you know there are classes, inheritance and overrides in JavaScript – thus you can improve your object oriented JavaScript!

$data = array(...);
$this->_helper->json($data);

// IndexController.php
class IndexController extends Zend_Controller_Action
{
	public function indexAction()
	{
		$data = array(...);
		
		$this->view->data = Zend_Json::encode($data);	
	}	
}

The result in general is a specially formatted string. So you can simply set it up to a view member variable and pass it to the view.

// index/index.phtml
echo $this->data

In that case you’ve a .phtml file to maintain, so lets just return the string and “setNoRender” the view in our second try.

// IndexController.php
class IndexController extends Zend_Controller_Action
{
	public function indexAction()
	{
		$data = array(...);
		
		echo Zend_Json::encode($data);	
		
		$this->_helper->viewRenderer->setNoRender(true);
	}	
}

Actually this is pretty much the most clear solution, but actually you can output the JSON string and simply exit() as it’s shown in our third example.

// IndexController.php
class IndexController extends Zend_Controller_Action
{
	public function indexAction()
	{
		$data = array(...);
		
		echo Zend_Json::encode($data);	
		
		exit();
	}	
}

Which one is to be used is up to the developer’s choice, mine is the third one as it’s the minimal one.

As you may know there are different notations of objects in JavaScript, and there are slight differences. However here I’m gonna achieve the same thing with different notations. The first one is the object notation in JavaScript known mostly from JSON. There you’ve to construct a key/value pairs, divided by a colon:

var class1 = {
    a : 0,
    func1 : function() {
        console.log(class1.a);
    },
    func2 : function(a) {
        this.a = a;
    }
}
class1.func2(3);

Here we have two functions – func1 and func2 and one member variable. Func2 is setting up the member variable, it’s something like a setter in some languages like Java and PHP, and the func1 is printing the value of that variable via console.log(). Note that in the body of func1() the variable “a” is accessed with the class name: class1.a. Here the notation can be changed to this.a, which will result in the same thing.

...
console.log(this.a);
...

Notation #2

While in the notation I’ve just used is not possible to have two different instances from the same class, in the next example this is possible. In JavaScript the objects are even the functions, and you can access the function’s private variables with the dot notation.

var class2 = function() {
    this.a = 0;
    this.func1 = function() {
        console.log(this.a);
    }

    this.func2 = function(a) {
        this.a = a;
    }
}

To use this object you’ve to use the “new” operator.

var c = new class2();
c.func2(4);

This helps you create more than one instance of the same class.

var c = new class2();
var d = new class2();
c.func2(3);
d.func2(4);

(function(jQuery) {

   var supported = true;
   if (typeof localStorage == 'undefined' || typeof JSON == 'undefined')
      supported = false;
   else
      var ls = localStorage;

   this.setItem = function(key, value, lifetime) {
      if (!supported)
         return false;

      if (typeof lifetime == 'undefined')
         lifetime = 60000;

      ls.setItem(key, JSON.stringify(value));
      var time = new Date();
      ls.setItem('meta_ct_'+key, time.getTime());
      ls.setItem('meta_lt_'+key, lifetime);
   };

   this.getItem = function(key) {
      if (!supported)
         return false;

      var time = new Date();
      if (time.getTime() - ls.getItem('meta_ct_'+key) > ls.getItem('meta_lt_'+key)) {
         ls.removeItem(key);
         ls.removeItem('meta_ct_'+key);
         ls.removeItem('meta_lt_'+key);
         return false;
      }
      return JSON.parse(ls.getItem(key));
   };

   this.removeItem = function(key) {
      if (!supported)
         return false;

      ls.removeItem(key);
      ls.removeItem('meta_ct_'+key);
      ls.removeItem('meta_lt_'+key);
      return true;
   };

   jQuery.localStorage = this;

})(jQuery);

if (!$.localStorage.getItem('test')) {
   $.localStorage.setItem('test', ['stoimen'], 5000);
   console.log('dynamic');
} else {
   console.log('from cache');
}

Imagine if you have the possibility to store large JSON returned from an AJAX call on the client. That’s now reality with the localStorage object in HTML5 and it can be accessed with something like this code:

localStorage.setItem(key, value);
alert(localStorage.getItem(key));

Can we store entire objects?

Yes there is a way extending a bit the logic of the code above. The localStorage in fact stores only strings, but what about arrays and objects. You can additionally use JSON global objects and its two methods – stringify and parse:

var a = JSON.stringify({name:'obj'});
alert(JSON.parse(a));

Do browsers support all that?

Of course … not! localStorage is supported by Firefox 3.5+, Chrome 2+, Safari 4+ and MSIE 8+. The solution is not to rely on browser version, but to check for object existence, and than to combine both techniques:

localStorage.setItem('a', JSON.stringify({ name : 'obj' }));
alert(JSON.parse(localStorage.getItem('a')));