Here’s one task related to the tree data structure. Given two nodes, can you find their lowest common ancestor?

In a matter of fact this task always has a proper solution, because at least the root node is a common ancestor of all pairs of nodes. However here the task is to find the lowest one, which can be quite far from the root.

We don’t care what kind of trees we have. However the solution, as we will see, can be very different depending on the tree type. Indeed finding the lowest common ancestor can have linear complexity for binary search trees, which isn’t true for ordinary trees.

Overview

Let’s say we have a tree (not binary!) and two nodes from this tree. The task is to find their lowest common ancestor. The thing is that we don’t know much about where they appear to be in the tree.

We can think of this tree as a DOM tree of any single HTML page online. It is not binary or balanced and can’t be sure where these nodes are.

First we must find both paths from the root to each one of the target nodes. Note that this requires additional memory! Then, in linear time we can pass through these “paths” and scan them from the root down to the nodes. We expect these to arrays to be equal at least in their first element (the root). Using this scenario the lowest common ancestor is the last equal element in both arrays.

To see how this algorithm can be dramatically changed depending on the data structure, let’s see another example. Now let’s say we have a binary search tree (BST). We know that in a BST all the elements in the left sub-tree are smaller than the root and all the items on the right sub-tree are greater than the root. This is true also for the left and the right sub-trees.

Now because we’re searching the lowest common ancestor, we don’t need to collect the paths from the root to the nodes in two arrays. We just know that the greater items are on the right, while the smaller items are on the left. This can help us find the lowest ancestor starting directly from the root.

What we do is to compare the two keys of the target nodes with the root key. If one of the keys are smaller, and the other is greater than the root’s key, then obviously the root is the lowest common ancestor. This is true because one of the items will be somewhere in the left sub-tree, while the other will be in the right sub-tree.

In case both values are greater (or smaller) than the root, we can move to the right (or to the left) sub-tree and try again with the same procedure. Thus the first node which key is in between the two target values will be the lowest common ancestor.

Code

Here’s a very simple PHP implementation showing us these two algorithms.

class Tree
{
    public $node = null;
    public $id = null;
    public $parent = null;
    public $children = array();
    
    public function __construct($node, $id = null)
    {
        $this->node = $node;
        $this->id = $id;
    }
    
    public function addChild(Node &$n)
    {
        $n->parent = $this;
        $this->children[] = $n;
    }
    
    /**
     * Returns an element by its id
     * 
     * @param mixed $id
     * @return Node 
     */
    public function search($id)
    {
        if ($this->id == $id) {
            return $this;
        }

        $a = false;
        
        // search all the children starting from the left-most
        foreach ($this->children as $child) {
            $a = $child->search($id);
        }
        
        return $a;
    }
    
    /**
     * Finds a path from the root to the 
     * item and returns it as a list
     * 
     * @param mixed $id 
     * @return array
     */
    public function find_path($id, &$path)
    {
        array_push($path, $this->id);
        
        if ($this->id == $id) {
            return 1;
        }
        
        foreach ($this->children as $child)  {
            if (1 == $child->find_path($id, $path)) return 1;
            array_pop($path);
        }
    }
    
    public function __toString()
    {
        return $this->node . ' ' . $this->id . "\n";
    }
}

$dom = new Tree('DOM', 'ROOT');

$body = new Tree('BODY', 1);
$div1 = new Tree('DIV', 'div-1');
$div2 = new Tree('DIV', 'my-id');

$a = new Tree("A", 'some-link');

$dom->addChild($body);
$body->addChild($div1);
$body->addChild($div2);
$div2->addChild($a);

$path1 = $path2 = array();
$dom->find_path('div-1', $path1);
$dom->find_path('some-link', $path2);

Finding Lowest Common Ancestor in a BST

class Tree
{
    public $key;
    
    public $parent  = null;
    public $left    = null;
    public $right   = null;
    
    public function __construct($key) 
    {
        $this->key = $key;
    }
    
    public function insert(Tree $n) 
    {
        if ($this->key < $n->key) {
            if ($this->right == null) {
                // insert
                $this->right = $n;
                $n->parent = $this;
            } else {
                $this->right->insert($n);
            }
        }
        if ($this->key > $n->key) {
            if ($this->left == null) {
                // insert
                $this->left = $n;
                $n->parent = $this;
            } else {
                $this->left->insert($n);
            }
        }
    }
}

$t = new Tree(10);

$n1 = new Tree(20);
$n2 = new Tree(5);
$n3 = new Tree(7);
$n4 = new Tree(13);

$t->insert($n1);
$t->insert($n2);
$t->insert($n3);

function find_common($node1, $node2, $tree) 
{
    if ($node1->key < $tree->key && $node2->key > $tree->key) {
        return $tree;
    } else if ($node1->key < $tree->key && $node2->key < $tree->key) {
        find_common($node1, $node2, $tree->left);
    } else if ($node1->key > $tree->key && $node2->key > $tree->key) {
        find_common($node1, $node2, $tree->right);
    }
}

$node = find_common($n3, $n4, $t);

Application

A typical use-case of this algorithm is finding the lowest common ancestor of two nodes in a DOM tree. Sometimes we just need to attach an event listener to both items (even before they are attached to the DOM!). Although attaching this event to the “document” will work just fine, all the elements from the nodes up to the root will be “capturing” these events due to event bubbling. Thus attaching the event to the lowest common ancestor is a better solution.

The binary search tree is a very useful data structure, where searching can be significantly faster than searching into a linked list. However in some cases searching into a binary tree can be as slow as searching into a linked list and this mainly depends on the input sequence. Indeed in case the input is sorted the binary tree will seem much like a linked list and the search will be slow.

To overcome this we must change a bit the data structure in order to stay well balanced. It’s intuitively clear that the searching process will be better if the tree is well branched. This is when finding an item will become faster with minimal effort.

Since we know how to construct a binary search tree the only thing left is to keep it balanced. Obviously we will need to re-balance the tree on each insert and delete, which will make this data structure more difficult to maintain compared to non-balanced search trees, but searching into it will be significantly faster.

Overview

In order to balance a tree we can go for the very basic and intuitive approach. First let’s take a look of one non-balanced tree.

Compared to the balanced tree on the right from the image above with the same items we see that the root is approximately equal to its middle item. I.e. 4 is the middle item of the sequence [1,2,3,4,5,6,7]!

If we take a look of the sequence [2 3 4], clearly by building a binary tree it will look like a linked list. However if we choose the middle item for a root – we’ll easy build a balanced tree. So the only thing to do is to get the middle item out of a list.

We now see that building a balanced binary tree out of a sorted linked list isn’t that difficult. In the other hand, as I said above, on each insert we’ll have to rebalance the tree. You can think of the tree out of the values [1,2,3,4,5] and the same tree after inserting [44,45,46,47,48]. Clearly the root of the resulting tree will no longer be 3.

So we need to implement the re-balancing in three basic operations. First we need to build a linked list out of a balanced binary tree. On the second place we’ll have to find the middle item and on the third place we’ll have to build again a balanced search tree.

Hopefully the first two tasks are easy to implement, because making out a sorted list out of a binary search tree is very easy. We need just to walk through the tree from left-root-right recursively. Because smaller items are in the left sub-tree and greater items are on the right we’re sure that the resulting list will be sorted. Then finding the middle item is as easy as finding the middle index of an array know its length.

Balancing Optimization

Of course the main problem of re-balancing a tree on each insert/delete is that this operations will be slow and soon or later we’ll have problems. That can happen if we change often our data structure. That’s why we should think of some optimization.

Normally we insert and re-balance on each step, which is slow. In the other hand we can do bulk insert forgetting about the re-balancing for a while. Only after the inserts are done we can go for re-balancing the entire tree.

The same approach we can use with bulk delete. We can just set to NIL the items we want to delete, but we can keep them in memory for a while. Thus the search will stay relatively fast without rebalancing the tree. However this approach can be used carefully because we’ll keep some data in the memory without actually using it.

Implementation

Implementing balanced binary trees is more difficult than just implementing binary search trees. Here’s an example in PHP.

class Node
{
	protected   $_parent = null;
	protected   $_left = null;
	protected   $_right = null;
	protected   $_key;
    protected   $_data = null;
	
    /**
     * @param int $key
     * @param mixed $data 
     */
	public function __construct($key, $data)
	{
		$this->_key = $key;
        $this->_data = $data;
	}
    
    /**
     * Empty the node by keeping up the key, but
     * setting up the data to NULL 
     */
    public function doEmpty() 
    {
        $this->_data = null;
    }
	
    /**
     * Print the key
     * 
     * @return string
     */
	public function __toString()
	{
		return 'First name: ' . $this->_data['f_name']
                . '
'
                . 'Last name: ' . $this->_data['l_name']
                . '
' 
                . 'Birthday: ' . $this->_data['b_day'];
	}
    
    public function &getParent() { return $this->_parent; }
    public function setParent($parent) { $this->_parent = $parent; }
    
    public function &getLeft() { return $this->_left; }
    public function setLeft($left) { $this->_left = $left; }
    
    public function &getRight() { return $this->_right; }
    public function setRight($right) { $this->_right = $right; }
    
    public function &getKey() { return $this->_key; }
    public function setKey($key) { $this->_key = $key; }
    
    public function &getData() { return $this->_data; }
    public function setData($data) { $this->_data = $data; }
}

class BalancedBinaryTree
{
    /**
     * Reference to the root tree
     * 
     * @var Node 
     */
	protected $_root = null;
	
    /**
     * @param type $new
     * @param type $node
     * @return type 
     */
	protected function _insert($new, &$root)
	{
        // in case the tree is empty
        // make the new node the root of
        // the tree
		if ($root == null) {
			$root = $new;
			return;
		}
		
		if ($new->getKey() <= $root->getKey()) {
			if ($root->getLeft() == null) {
				$root->setLeft($new);
				$new->setParent($root);
			} else {
				$this->_insert($new, $root->getLeft());
			}
		} else {
			if ($root->getRight() == null) {
				$root->setRight($new);
				$new->setParent($root);
			} else {
				$this->_insert($new, $root->getRight());
			}
		}		
	}
	
    /**
     * FALSE on not found
     * 
     * @param string $firstName
     * @param BalancedBinaryTree $tree
     * @return boolean 
     */
	protected function _search($firstName, &$tree)
	{
        if ($tree == null) {
            return FALSE;
        }

        $data = $tree->getData();
		
        if ($firstName == $data['f_name']) {
			return $tree;
		}
        
        // search the left sub-tree
        return $this->_search($firstName, $tree->getLeft())
                . $this->_search($firstName, $tree->getRight());
	}
    
    /**
     *
     * @param int $key
     * @param Node $tree
     * @return FALSE or Node 
     */
    protected function _searchByKey($key, &$tree)
    {
        if ($tree == null) {
            return FALSE;
        }
        
        if ($tree->getKey() == $key) {
            return $tree;
        } else if ($tree->getKey() > $key) {
            return $this->_searchByKey($key, $tree->getLeft());
        } else {
            return $this->_searchByKey($key, $tree->getRight());
        }
    }
    
    /**
     * Returns a list out of the tree by emptying the tree. 
     * In other way the tree and the list will allocate memory
     * 
     * @param BalancedBinaryTree $tree 
     */
    protected function _leftRootRight($tree)
    {
        if ($tree == null) {
            return array();
        }
        
        return array_merge(
                $this->_leftRootRight($tree->getLeft()),
                array(array('key' => $tree->getKey(), 'data' => $tree->getData())),
                $this->_leftRootRight($tree->getRight()));
    }
    
    public function _balance($list)
    {
        if (empty($list)) {
            return;
        }
        
        // split the list
        $chunks = array_chunk($list, ceil(count($list) / 2));
        $mid = array_pop($chunks[0]);
        
        $node = new Node($mid['key'], $mid['data']);
        $this->insert($node);
        
        $this->_balance($chunks[0]);
        if (isset($chunks[1]))
            $this->_balance($chunks[1]);
    }
    
    /**
     * Balance a binary search tree 
     */
    public function balance()
    {
        $list = array();
        // make a list out of the tree
        $list = $this->_leftRootRight($this->_root);
        
        // find the medium! Because the list is ordered
        // we can find the middle element in various ways
        $chunks = array_chunk($list, ceil(count($list) / 2));
        $mid = array_pop($chunks[0]);
        
        // empty the tree
        $this->_root = null;
        
        // inser the root
        $node = new Node($mid['key'], $mid['data']);
        $this->insert($node);
        
        $this->_balance($chunks[0]);
        $this->_balance($chunks[1]);
    }
	
    /**
     * Insert a new item into the tree
     * 
     * @param type $node 
     */
	public function insert($newNode)
	{
		$this->_insert($newNode, $this->_root);
	}
	
    /**
     * Search by item key
     * 
     * @param int $key
     * @return Node or FALSE
     */
    public function searchByKey($key)
    {
        return $this->_searchByKey($key, $this->_root);
    }
    
    /**
     * @param BalancedBinary $tree
     * @return string 
     */
    protected function _print($tree)
    {
        if ($tree == null) { return ''; }
        
        return $this->_print($tree->getLeft()) . ' ' 
                . $tree->getKey() . ' ' 
                . $this->_print($tree->getRight());
    }
    
    /**
     * Print the tree from left through the root and the right 
     */
    public function __toString()
    {
        if ($this->_root == null) {
            return 'The tree is empty!';
        }

        return $this->_print($this->_root->getLeft()) . ' '
                . $this->_root->getKey() . ' '
                . $this->_print($this->_root->getRight());
    }
}

$a = new Node(90, array(
    'f_name' => 'W.A.',
    'l_name' => 'Mozart',
    'b_day' => '1756-01-27',
));

$b = new Node(100, array(
    'f_name' => 'John',
    'l_name' => 'Smith',
    'b_day' => '23.05.2039',
));

$c = new Node(80, array(
    'f_name' => 'Sarah',
    'l_name' => 'Johnnes',
    'b_day' => 'tomorrow',
));

$d = new Node(60, array(
    'f_name' => 'Ludwig Van',
    'l_name' => 'Beethoven',
    'b_day' => '1770-12-17',
));

$e = new Node(70, array(
    'f_name' => 'Barbara',
    'l_name' => 'Stefanel',
    'b_day' => 'today',
));

$t = new BalancedBinaryTree();

$t->insert($a);
$t->insert($b);
$t->insert($c);
$t->insert($d);
$t->insert($e);

echo $t;

echo $t->searchByKey(70);

$t->balance();

echo $t->searchByKey(70);

Complexity of Searching

Compared to non-balanced binary search trees we’re sure that searching into a balanced trees is quick enough. The maximum height of the tree is log(n) so the worst-case searching is O(log(n)).

Application

Searching into a balanced binary tree is fast. What is more important is that we’re sure that in the worst-case scenario the search is O(log(n)). The only problem is that keeping a tree balanced is a slow operation that consumes too much resources and must be performed carefully.

Constructing a linked list is a fairly simple task. Linked lists are a linear structure and the items are located one after another, each pointing to its predecessor and its successor. Almost every operation is easy to code in few lines and doesn’t require advanced skills. Operations like insert, delete, etc. over linked lists are performed in a linear time. Of course on small data sets this works fine, but as the data grows these operations, especially the search operation becomes too slow.

Indeed searching in a linked list has a linear complexity and in the worst case we must go through the entire list in order to find the desired element. The worst case is when the item doesn’t belong to the list and we must check every single item of the list even the last one without success. This approach seems much like the sequential search over arrays. Of course this is bad when we talk about large data sets.

In terms of arrays, we could perform binary search and go directly in the middle of the array, then jump back or forward. That is because we can access array items directly using their index. However as we saw the linked lists unlike arrays can’t benefit of a direct access and we must go item by item.

Because of this natural problem of linked lists searching is slow and obviously we can’t make it better. The only way to improve searching over dynamic data structures is to use different data structure.

The tree is a data structure where each item, except of keeping some data, keeps a reference (pointer) to its children and its parent.

Of course if the item doesn’t have children, they are NIL, then this is considered a leaf in the tree terminology. In the other hand if the item doesn’t have parent item it is considered the root.

If there is no item in the tree the tree is considered empty.

In these terms only the root has no parent, and each item can have as many children as possible. Here are some trees in form of a diagrams.

If we’re looking at the root of the tree we can assume there are two sub-trees – one left and one right. However if we isolate only one of these sub-trees we can again think of it as a tree and assume that it has one left and one right sub-trees and go recursively with this definition.

Overview

A binary tree is a tree where each item can have at most two children.

Binary trees are especially important because they can contain ordered data in a specific manner. Building a binary tree isn’t difficult at all and it’s very similar to building a linked list.
However a binary tree isn’t more successful in searching than any other tree or data structure. If the items aren’t placed in a specific order we must go through the entire tree in order to find the searched item. This isn’t a great optimization, so we must put an order in it to improve the searching process.

Binary Search Tree

The binary search tree is a specific kind of binary tree, where the each item keeps greater elements on the right, while the smaller items are on the left.

Constructing a binary search tree is easy, because we can go for inserting each item only by comparing it with the root and decide where to go (left or right) based on its value.

Implementation

The following code in PHP describes the basic principles of a binary search tree.

class Node
{
	public $parent = null;
	public $left = null;
	public $right = null;
	public $data = null;
	
	public function __construct($data)
	{
		$this->data = $data;
	}
	
	public function __toString()
	{
		return $this->data;
	}
}

class BinaryTree
{
	protected $_root = null;
	
	protected function _insert(&$new, &$node)
	{
		if ($node == null) {
			$node = $new;
			return;
		}
		
		if ($new->data <= $node->data) {
			if ($node->left == null) {
				$node->left = $new;
				$new->parent = $node;
			} else {
				$this->_insert($new, $node->left);
			}
		} else {
			if ($node->right == null) {
				$node->right = $new;
				$new->parent = $node;
			} else {
				$this->_insert($new, $node->right);
			}
		}		
	}
	
	protected function _search(&$target, &$node)
	{
		if ($target == $node) {
			return 1;
		} else if ($target->data > $node->data && isset($node->right)) {
			return $this->_search($target, $node->right);
		} else if ($target->data <= $node->data && isset($node->left)) {
			return $this->_search($target, $node->left);
		}
		
		return 0;
	}
	
	public function insert($node)
	{
		$this->_insert($node, $this->_root);
	}
	
	public function search($item) 
	{
		return $this->_search($item, $this->_root);
	}
}

$a = new Node(3);
$b = new Node(2);
$c = new Node(4);
$d = new Node(7);
$e = new Node(6);

$t = new BinaryTree();

$t->insert($a);
$t->insert($b);
$t->insert($c);
$t->insert($d);
$t->insert($e);

echo $t->search($e);

Search Complexity

Searching in binary search trees is supposed to be faster than searching into linked list. However the searching process in a BST can be very fast, but also can be as slow as on linked list. That is because depending on the input of items they can be placed only on the one side of the root.

That makes the worst-case searching as slow as on linked list which is linear O(n). However if the tree is somehow balanced we can search very quickly with O(log(n)) time.

Further Optimization

We now see how ineffective binary search trees can be, so the only thing we must care is how to keep them balanced, so the search will be faster. The answer is to maintain (during insertion) a balanced binary search tree, which is another very handy data structure.

A balanced binary search tree, or only balanced tree, is a data structure where the height of left and the right sub-trees can vary by one level at most.

Application

Binary search trees are easy to build and maintain. The great thing is that if the data is well balanced they can be very useful for searching. The only problem is that these structures can be ineffective depending on the insertion order. However if we are somehow sure that the items aren’t ordered on the input, we may expect some optimized searching compared to a linked list. Compared to balanced binary search trees, BST require much less time to build and maintain (insert, delete).

Trees are very useful when working with graphs. Actually one of the very common tasks is walking through the entire tree, which can be done in several ways. First we can go to the left sub-tree, then the root and then the right sub-tree. Or right-root-left. Or root-left-right.

However we can go in depth first often called depth-first-search or a breadth-first-search.

These two methods are designed to walk through the items in a specific order, which is very handy for some specific tasks – at least each tree is also a graph.

I wrote about binary search in my previous post, which is indeed one very fast searching algorithm, but in some cases we can achieve even faster results. Such an algorithm is the “interpolation search” – perhaps the most interesting of all searching algorithms. However we shouldn’t forget that the data must follow some limitations. In first place the array must be sorted. Also we must know the bounds of the interval.

Why is that? Well, this algorithm tries to follow the way we search a name in a phone book, or a word in the dictionary. We, humans, know in advance that in case the name we’re searching starts with a “B”, like “Bond” for instance, we should start searching near the beginning of the phone book. Thus if we’re searching the word “algorithm” in the dictionary, you know that it should be placed somewhere at the beginning. This is because we know the order of the letters, we know the interval (a-z), and somehow we intuitively know that the words are dispersed equally. These facts are enough to realize that the binary search can be a bad choice. Indeed the binary search algorithm divides the list in two equal sub-lists, which is useless if we know in advance that the searched item is somewhere in the beginning or the end of the list. Yes, we can use also jump search if the item is at the beginning, but not if it is at the end, in that case this algorithm is not so effective.

So the interpolation search is based on some simple facts. The binary search divides the interval on two equal sub-lists, as shown on the image bellow.

What will happen if we don’t use the constant ½, but another more accurate constant “C”, that can lead us closer to the searched item.

The question is how to find this value? Well, we know bounds of the interval and looking closer to the image above we can define the following formula.

C = (x-L)/(R-L)

Now we can be sure that we’re closer to the searched value.

Implementation

Here’s an implementation of interpolation search in PHP.

$list = array(201, 209, 232, 233, 332, 399, 400);
$x = 332;

function interpolation_search($list, $x)
{
	$l = 0;
	$r = count($list) - 1;

	while ($l <= $r) {
		if ($list[$l] == $list[$r]) {
			if ($list[$l] == $x) {
				return $l;
			} else {
				// not found
				return -1;
			}
		}
		
		$k = ($x - $list[$l])/($list[$r] - $list[$l]);
		
		// not found
		if ($k < 0 || $k > 1) {
			return -1;
		}
		
		$mid = round($l + $k*($r - $l));
		
		if ($x < $list[$mid]) {
			$r = $mid - 1;
		} else if ($x > $list[$mid]) {
			$l = $mid + 1;
		} else {
			// success!
			return $mid;
		}
		
		// not found
		return -1;
	}
}

echo interpolation_search($list, $x);

Complexity

The complexity of this algorithm is log₂(log₂(n)) + 1. While I wont cover its proof, I’ll say that this is very slowly growing function as you can see on the following chart.

Indeed when the values are equally dispersed into the interval this search algorithm can be extremely useful – way faster than the binary search. As you can see log₂(log₂(100 M)) ≈ 4.73 !!!

Application

As I said already this algorithm is extremely interesting and very appropriate in many use cases. Here’s an example where interpolation search can be used. Let’s say there’s an array with user data, sorted by their year of birth. We know in advance that all users are born in the 80’s. In this case sequential or even binary search can be slower than interpolation search.

$list = array(
	0 => array('year' => 1980, 'name' => 'John Smith', 'username' => 'John'),
	1 => array('year' => 1980, ...),
	...
	10394 => array('year' => 1981, 'name' => 'Tomas M.', ...),
	...
	348489 => array('year' => '1985', 'name' => 'James Bond', ...),
	...
	2808008 => array('year' => '1990', 'name' => 'W.A. Mozart', ...)
);

Now if we search for somebody born in 1981 a good approach is to use interpolation search.