Tag Archives: Technology/Internet

Diving into Node.js – Introduction & Installation

Why I Need Something Like Node.js?

First of all the use of some software is needed not because of itself, but because of the need of some specific functionality. In my case this was the need of real time news feed. Of course there is a way to make this without Node.js, as I’ll describe later in this post, but there are several disadvantages. However to begin from somewhere, let me explain what is Node.js.

Introducing Node.js

Perhaps the best way do describe what is Node.js is from its about page.

Node’s goal is to provide an easy way to build scalable network programs. In the “hello world” web server example above, many client connections can be handled concurrently. Node tells the operating system (through epoll, kqueue, /dev/poll, or select) that it should be notified when a new connection is made, and then it goes to sleep. If someone new connects, then it executes the callback. Each connection is only a small heap allocation.

In general Node is a program using the Google Chrome’s V8 JavaScript engine, which in turn is a program that can parse and execute code written in JavaScript. V8 is a very very interesting project itself. First of all from Google have developed this engine especially for one of his products – their browser Chrome. It pretends to be and by no means is one of the masterpieces of Google. It is fast and reliable engine, written in C++ and JavaScript, as Wikipedia’s page says. Actually this code is open source and can be embedded in whatever application written in C++. Thus you can have in your application a JavaScript engine. Continue reading Diving into Node.js – Introduction & Installation

How to Make MP4 Progressive with qt-faststart

A big part of the developers worldwide, who have dealt with video converting, sooner or later have been used FFMPEG. In fact I’d say that two of the main software on the video converting scene are FFMPEG and Mencoder. I began to use Mencoder back in 2005 and I used it as a DVD ripping tool, but my experience with it ends there, while now I use FFMPEG.

However FFMPEG is a converting program and many things depend on what kind of video files come as an input and what do you want to achieve as an output. Here comes the codec question, where various libraries can help. In my case, as I wrote few posts ago, I use MP4 as output encoded with h.264, both encoded with ffmpeg/libx264. So far so good, but there is one main problem that anyone used this pair knows.

Flash Player and MP4

As far as I know Flash Player 9 and beyond plays video files encoded with MP4/h.264 except the typical FLV (flash video files). This can help video sites not only to share their video content via web – within browser based flash players, but also to show them via some mobile device interface where MP4/h.264 is perhaps the only one reasonable solution. In this case the file is accessible in both medias.

By default Flash Player can play such FFMPEG encoded .mp4 files, but fist the whole file must be downloaded by the browser and only than the playback can begin. That’s because the FFMPEG puts meta info at the end of the video file, while the flash player needs it at the beginning and only than to begin playing while the file is still downloading.

qt-faststart

As the name of this software says this program helps you move the important meta info from the end to the beginning of the file. This helps the video to playback as early as possible. In an ideal circumstances everything works just fine, but there is a serious problem in fact.

qt-faststart and Server Crash

qt-faststart is responsible to move the meta info where it must be – at the beginning of the video file, but cannot finish execution when the video file is broken or the meta info doesn’t exists. Actually this is not a problem of qt-faststart, but to the video file. However this was a critical problem in my case, because the video files remains stuck for hours of execution and 99% of CPU usage.

Thankfully there is a solution and it can be found in the qt-faststart wrapper – qtfaststart.py

qtfaststart.py

written on Python this script overcomes the disadvantages of qt-faststart. As described by his author Daniel G. Taylor:

  1. Works everywhere Python can be installed
  2. Handles both 32-bit (stco) and 64-bit (co64) atoms
  3. Handles any file where the mdat atom is before the moov atom
  4. Preserves the order of other atoms
  5. Can replace the original file (if given no output file)

With its help not only qt-faststart doesn’t crash, but also the video files remain progressive as desired. The installation process is as difficult as copy/paste on the server where Python must be installed.

How to Overcome Zend_Cache_Frontend_Page’s Problem with Cookies

Zend_Cache_Frontend_Page

First of all there are several things to know about Zend Framework and caching. Whenever you work on a big web application caching is one of the mostly used mechanisms of speeding up the app and improve user performance. In general the task and the solution are pretty simple and natural.

As the application grows up the visitors become more and more impatient about what they receive. A single page is becoming slower and slower and the result is painful. First of all every time a user hits a page the application server uses the web server, a script interpreter, a database server and potentially the file system. But that’s not all. After all this output is generated on the server, as HTML in the most cases, it is sent to the client where again CSS and JavaScript engines parse and execute them.

In this scenario it’s easy to imagine how many time is spent. While there are several techniques to optimize the client side by optimizing JavaScript, CSS and the static images used for the design of the site, here I’m going to talk more about the backend.

Beside the Optimization

Let’s assume we’ve one of the very used combination between Apache (as a webserver), PHP (as server scripting language) and MySQL (as database server). Here you can choose to optimize all three of them. However beside the optimization of them one of the most simple steps you can do is to cache the output generated by these three branches of your web app.

Caching the Content

In fact you can cache only single parts of the whole process. For instance you can cache only the result returned by some slow database query. Let’s imagine a query takes about 2 seconds to execute. Now you can cache the result into a file and during the cache is active, i.e. it has not expired, the application takes it from a file stored somewhere in the file system.

In a typical Zend Framework scenario you can first setup the frontend and backend options of the cache.

$frontendOptions = array(
	'lifetime'                => 600, // in seconds - this is 10 seconds
	'automatic_serialization' => true,
);
$backendOptions = array('cache_dir' => 'cache/');
$cache = Zend_Cache::factory('Core', 'File', $frontendOptions, $backendOptions);
 
$cacheKey = md5('mykey');
 
if (!$cache->load($cacheKey)) {
	$slowQueryResult = $article->fetchAll();
	$cache->save($slowQueryResult, $cacheKey);
} else {
	$slowQueryResult = $cache->load($cacheKey);
}

You can setup different options here by setting up the cache directory, lifetime, etc.

Note that the cache directory must exists and with write permissions and Zend Framework doesn’t create it for you and will throw error.

The problem here is that now you cache only part of the generated content and in many cases this is still too slow for most of the users. However all the work (in most of the cases) of the web server, the script interpreter and the database server result in a simple HTML output. What if you have this output generated or cached for you and when the user hit the page the server will return this pre-generated code?

This is indeed very fast, because it’s similar to return a text file, as the HTML is simply formatted text.

Caching the Entire Page

Before I proceed, I’d like to say that I work with Zend Framework 1.9.x. Now in the latest versions of ZF there are new mechanisms of caching the output even Zend_Cache_Frontend_Page works fine on them.

You can simply setup the page cache within few simple lines of code:

$fo = array(
    'lifetime' => 600,
    'regexps' => array(
        '^/' => array(
	'cache' => true,
         'cache_with_cookie_variables' => true,
        ),
    )
);
 
$bo = array(
    'cache_dir' => 'cache/'
);
 
$cache = Zend_Cache::factory('Page', 'File', $fo, $bo);
$cache->start();

However my advise is to place this code as high as possible, because this will cache everything generated as output. It’s a good practice if you place this even in the bootstrap before you make the connection with the database. Actually you don’t need a database connection when you’ve to return a simple text(html) file.

This will improve your app’s performance a lot!

However there are few things to know. When you setup the cache to work even with cookie variables, you can see that hitting the page with different browsers Zend Framework will generated different cache pages. This is quite useless because than you don’t have any benefit of caching the content.

First of all let me say that THIS IS NOT A BUG! of ZF. Simply the framework will use the cookie variables to generate the cache key. It’s obvious that different browsers, even more different users, will have different cookie set and the framework will generate different cache keys for them.

Thus you’ve to change the setting to generate the cache key from cookie variable by explicitly set this option to false:

$fo = array(
    'lifetime' => 600,
    'regexps' => array(
        '^/' => array(
		 'cache' => true,
         'cache_with_cookie_variables' => true,
         'make_id_with_cookie_variables' => false,
        ),
    )
);
 
$bo = array(
    'cache_dir' => 'cache/'
);
 
$cache = Zend_Cache::factory('Page', 'File', $fo, $bo);
$cache->start();

Note that in this example we cache every single page generated by the framework explained in the regexps. This is not so good especially when the users have the possibility to login and to see customized content for them, so you can be careful what you cache.

A typical problem that this solution solves is when your application uses Google Analytics. As you may know Google Analytics sets up a cookie every time when an user hits the page, so every time the framework will generate a different cache for him and in result he won’t see any benefit and performance improvement from your site.

ffmpeg, libx264 and presets

When working with FFMPEG you can convert/encode in various formats, but in my case this should be an MP4 file, encoded with libx264. This combination is quite well known in the web community, because those videos are playable under almost any flash player and Apple products.

Now the problem is that this comes with very large list of options, which are difficult to setup, especially for newbies like me. Than everything’s difficult, once because the resulting quality is not always the same as the input file’s quality.

The Solution

Is to use presets. Thankfully ffmpeg with libx264 can be used with presets, i.e. NORMAL, HD etc. which means that you don’t have to setup a command by hand and the resulting quality is the same as the input quality.

Here’s a sample command using normal preset:

ffmpeg -i source.mp4 -acodec libfaac -ab 128k -ac 2 -vcodec libx264 -vpre normal -threads 0 -crf 22 output.mp4

here the source can be either mp4 or some other file format as FLV for instance.

Construct a Sorted PHP Linked List

This is really a draft, but I somehow decided to post it. Here’s a simple linked list in PHP, where only the add() method is slightly different. When you have an item to add it’s placed on the “right” place – the result is a sorted list.

Few notes before the code! This list is designed for integers, but I plan to “extend” it somehow and it can be improved a lot. In my future posts I’ll post something more about the complexity of the algorithm and an analysis of it.

class Node
{
    public $data;
    public $next;
    public $prev;
 
    public function __construct($data, $prev, $next)
    {
        $this->data = $data;
        $this->prev = $prev;
        $this->next = $next;
    }
}
 
class LinkedList
{
    protected $front = null;
 
    public function add($data)
    {
        if (!$this->front) {
            $node = new Node($data, null, null);
            $this->front = &$node;
        } else {
            if ($data < $this->front->data) {
                $node = new Node($data, null, $this->front);
                $this->front = $node;
                return;
            }
 
            $current = $this->front;
            while ($current) {
                if ($current->data < $data && isset($current->next) && $current->next->data > $data) {
                    $node = new Node($data, $current, $current->next);
                    $current->next = $node;
                }
                if ($current->data < $data && !isset($current->next)) {
                    $node = new Node($data, $current, $current->next);
                    $current->next = $node;
                }
                $current = $current->next;
            }
        }
    }
 
    public function printl()
    {
        $current = &$this->front;
        while($current) {
            echo $current->data, '<br />';
            $current = $current->next;
        }
    }
}
 
$list = new LinkedList();
$list->add(13);
$list->add(14);
$list->add(15);
$list->add(11);
$list->add(12);
$list->add(17);
$list->add(10);
$list->add(16);
 
$list->printl();