My blog has been moved to ariya.ofilabs.com.

Sunday, January 23, 2011

PhantomJS: minimalistic headless WebKit-based JavaScript-driven tool

PhantomJS is a headless WebKit packaged as a JavaScript-driven tool. It can be used in command-line utilities which requires web stack, or even as the basis for testing rich web application. It uses WebKit in a headless mode, so you get access to the real native and fast implementation (not a simulated environment) of various standards such as DOM, CSS selector, Canvas, SVG, and many others.

The project page contains a bunch of examples, from easy ones to some more complicated uses. Feel free to contribute more examples!

Let's look at one of the examples, the page rasterizer (yes, it's only 16 lines!):

if (phantom.state.length === 0) {
    if (phantom.args.length !== 2) {
        console.log('Usage: rasterize.js URL filename');
        phantom.exit();
    } else {
        var address = phantom.args[0];
        phantom.state = 'rasterize';
        phantom.viewportSize = { width: 600, height: 600 };
        phantom.open(address);
    }
} else {
    var output = phantom.args[1];
    phantom.sleep(200);
    phantom.render(output);
    phantom.exit();
}

If I want to have the famous PostScript tiger from its SVG source, all I have to do is to run:

phantomjs rasterize.js http://ariya.github.com/svg/tiger.svg tiger.png

But static vector graphic is boring. Replacing the above with

phantomjs rasterize.js http://raphaeljs.com/polar-clock.html clock.png

gives me Polar Clock, one notable example from RaphaelJS.

Should you need to deal with JSONP, process XML, and integrate with YQL, that's all easily done. Again, refer to the various service integration examples. Let me show one example, which is actually my favorite:

if (phantom.state.length === 0) {
    var origin, dest;
    if (phantom.args.length < 2) {
        console.log('Usage: direction.js origin destination');
        console.log('Example: direction.js "San Diego" "Palo Alto"');
        phantom.exit(1);
    }
    origin = phantom.args[0];
    dest = phantom.args[1];
    phantom.state = origin + ' to ' + dest;
    phantom.open(encodeURI('http://maps.googleapis.com/maps/api/directions/xml?origin='
        + origin +  '&destination=' + dest + 
        '&units=imperial&mode=driving&sensor=false'));
} else {
    if (phantom.loadStatus === 'fail') {
        console.log('Unable to access network');
    } else {
        var steps;
        steps = phantom.content.match(/<html_instructions>(.*)<\/html_instructions>/ig);
        if (steps == null) {
            console.log('No data available for ' + phantom.state);
        } else {
            steps.forEach(function (ins) {
                ins = ins.replace(/\&lt;/ig, '<').replace(/\&gt;/ig, '>');
                ins = ins.replace(/\<div/ig, '\n<div');
                ins = ins.replace(/<.*?>/g, '');
                console.log(ins);
            });
        }
    }
    phantom.exit();
}

If I run it like the following:

phantomjs direction.js 'Redwood City' 'Sunnyvale'

what I got is the complete driving direction:

Head east on Broadway toward El Camino Real
Take the 1st left onto El Camino Real
Turn right at Whipple Ave
Slight right to merge onto US-101 S toward San Jose
Take exit 398B to merge onto CA-85 S toward Santa Cruz/Cupertino
Take exit 22A to merge onto CA-82 S/E El Camino Real toward Sunnyvale
Destination will be on the right

Map data ©2011 Google

Make sure you check out other examples, such as getting weather forecast conditions, finding pizza in New York, looking up approximate location based on IP address, pulling the list of seasonal food, displaying tweets, and many others.

Headless execution of any web content also enables fast unit testing. Obviously, the goal is not to replace comprehensive, cross-browser framework such as Selenium or Squish for Web. Rather, it serves a quick sanity check just before you check in some changes.

Since this can happen automatically and does not need to launch any browser, even better, you can hook the test so that it executes right before a commit and actually prevents the commit if any of the test fails. It is easily done using git via its hook support. This is something I have written at Sencha blog. It demonstrated precommit hook with Jasmine, but technically it can work with any test framework.

I have been working on and off on PhantomJS for the past few years. You may be already familiar with some of its inspiration (also involving headless WebKit): SVG rasterizer, page capture, visual Google, etc. Finally I managed to overcome my laziness, cleaned up the code, and published it for your pleasure. Obviously it's not a surprise if you find out that PhantomJS uses QtWebKit.

I got a few tasks for next PhantomJS version 1.1. You are encouraged to file bugs and feature requests in the said issue tracker.

Get it while it is hot!

Saturday, January 08, 2011

command line CoffeeScript

CoffeeScript seems to be picking up some momentum these days. No doubt, it is very valuable to help writing cleaner code.

The command-line choices to run CoffeeScript compiler right now are either using Rhino (jcoffeescript) or using NodeJS. While I love NodeJS, seems that it is an overkill to require the entire NodeJS stack/infrastructure/package manager to invoke CoffeeScript compiler.

The solution is to use V8, the powerful JavaScript engine, with a little binding so that it can access file system. This is exactly filejs, something I have shown before, e.g. to invoke JSLint from command line.

Combining filejs and CoffeeScript is terribly easy. Just follow these steps.

Note: filejs does not support Windows yet. Sorry.

First of all, if you have not done it, build filejs. Go to the X2 repository, it is under the javascript/filejs folder. Open the included README.TXT and follow the instructions on how to build V8 and filejs.

After you build it, copy both filejs executable and coffee.js to somewhere in your PATH. Usually I stash that kind of stuff in ~/bin and ensure that ~/bin is in my PATH.

Now get coffee-script.js (the CoffeeScript to JavaScript compiler) and store it somewhere, e.g. ~/bin again.

Create a new file called coffee, which has the following one-line content:

filejs ~/bin/coffee.js $1

Make that file executable and then save it to ~/bin (again).

Open coffee.js and modify the value of the compiler variable to point to your coffee-script.js. Note: this must use the absolute path name.

Now you can do the following:

coffee hello.coffee

If hello.coffee is your script written in CoffeeScript, the converted JavaScript version will be dumped to the standard output.

Feel free to tweak coffee.js so that it understands and passes various CoffeeScript compiler options!

Fun, isn't it?

Saturday, January 01, 2011

X2 from Ofi Labs: wrap-up 2010

X2It got started when I needed a new home for my examples. It has even a nice logo.

sensor

accelerometer viewer for Maemo 5 (Nokia N900).

bouncing ball, where the gravity affects the movement of the ball.

box of marbles, where the gravity affects a bunch of colored marbles.

combining accelerometer and network to do inter-device marbles transfer.

motion and orientation for web applications.

web-based version of marble box.

widgets

morphing clock, where the transition between the digital and analog version is a kind of morphing effect.

qpalette viewer so you know which color is which one.

graphics

fast approximation of Gaussian blur to create a blurry drop shadow.

command-line capture tool to save maps from OpenStreetMap, MapQuest and Ovi Maps.

simple tool to list all chunks inside a PNG image.

webkit & javascript

file processing, including using jslint, in command-line using JavaScript.

play Canvas-based game as normal desktop app.

offline, command line beautifier for JavaScript code, utilizing Qt Script.

another variant of the beautifier, this time using V8.

minimalistic editing widget for JavaScript code, with custom syntax highlighting.

white background is boring? just try some color inverted web pages.

detect the closest link to ease following it on a touch device.

Canvas pixel manipulation for plasma effect.

network

simple proxy server for HTTP, in 100 lines.

tracenet: trap all network requests+replies to show them with Speed Tracer.

filterproxy: another variant of the proxy server with added URL filtering feature.