fluid animation with accelerated composition

Those who work on web-based applications on mobile platforms often recall the advice, "Use translate3d to make sure it's hardware accelerated". This advice seems magical at first, and I seldom find anyone who explains (or wants to explain) the actual machinery behind such a practical tip.

For a while, Safari (and Mobile Safari) was the only WebKit-based browser which supports hardware-accelerated CSS animation. Google Chrome caught up, QtWebKit-powered browser (like the one in Nokia N9) also finally supported it. Such a situation often gave the wrong impression that Apple kept the hardware-acceleration code for themselves.

The above two are basically the reasons for this blog post.

In case you miss it (before we dive in further), please read what I wrote before about different WebKit ports (to get the idea of implementation + back-end approach) and tiled backing store (decoupling web page complexity with smooth UX). The GraphicsContext abstraction will be specially useful in this topic. In particular, because animation is tightly related to efficient graphics.

Imagine if you have to move an image (of a unicorn, for example) from one position to another. The pseudo-code for doing it would be:

  for pos = startPosition to endPosition
    draw unicorn at pos

To ensure smooth 60fps, your inner loop has only 16 ms to draw that unicorn image. Usually this is a piece of cake because all the CPU does is sending the pixels of the unicorn image once to the GPU (in the form of texture) and then just refer the texture inside the animation loop. No heavy work is needed on the CPU and GPU sides.

If, however, what you draw is very complicated, e.g. formatted text consisting of different font typefaces and sizes, this gets hairy. The "draw" part can take more than 16 ms and the animation is not butter-smooth anymore. Because your text does not really change during the animation, only the position changes, the usual trick is to cache the text, i.e. draw it onto a buffer and just move around the buffer as needed. Again, the CPU just needs to push the buffer the GPU once:

    prepare a temporary buffer
    draw the text onto the buffer
    for pos = startPosition and endPosition
       set a new transformation matrix for the buffer

As you can imagine, that's exactly what happens when WebKit performs CSS animation. Instead of drawing your div (or whatever you animate) multiple times in different position, it prepares a layer and redirect the drawing there. After that, animation is a simple matter of manipulating the layer, e.g. moving it around. WebKit term for this (useful if you comb the source code) is accelerated composition accelerated compositing.

Side note: Mozilla has the same concept, available since Firefox 4, called Layer.

If you understand immediate vs retain mode rendering, non-composited vs composited is just like that. The idea to treat the render tree more like a scene graph, a stack of layers with different depth value.

Because composition reduces the computation burden (GPU can handle varying transformation matrix efficiently), the animation is smoother. This is not so noticeable if you have a modern machine. In the following video demo (http://youtu.be/KujWTTRkPkM), I have to use my slightly old Windows laptop to demonstrate the frames/second differences:

The excellent falling leaves animation is something you have seen before, back when WebKit support for CSS animation was announced.

Accelerated composition does not magically turn every WebKit ports capable of doing fluid animation. Analog to my previous rounded corner example, composition requires the support from the underlying platform. On Mac OS X port of WebKit, composition is mapped into CoreAnimation (part of CoreGraphics), the official API to have animated user interface. Same goes for iOS WebKit. On Chromium, it is hooked into sandboxed GPU process.

With QtWebKit, composition is achieved via Graphics View framework (read Noam's explanation for details). The previous video you have seen was created with QtWebKit, running without and with composition, i.e. QGraphicsWebView with different AcceleratedCompositingEnabled run-time setting. If you want to check out the code and try it yourself, head to the usual X2 repository and look under webkit/composition. Use spacebar (or mouse click) to switch between composited and non-composited mode. If there is no significant frame rate improvement, increase NUMBER_OF_LEAVES in leaves.js and rebuild. When compositing is active, press D to draw thin yellow border around each layer. Since it's all about Graphics View, this debugging is easy to implement. I just inject a custom BorderEffect, based on QGraphicsEffect (which I did prototype back when I was with Nokia):

Thus, there is nothing like hidden secret with respect to Safari hardware-accelerated CSS support. In fact, Safari is not different than other Mac apps. If you compile WebKit yourself and build an application with it, you would definitely get the animation with hardware acceleration support.

As the bonus, since Mac and iOS WebKit delegate the animation to CoreAnimation (CA), you can use various CA tweaks to debug it. CA_COLOR_OPAQUE=1 will emphasize each layer with red color overlay (as in the demo). While this applies to any CA-based apps (not limited to WebKit or Safari), it's still very useful nevertheless. Chromium's similar feature is --show-composited-layer-border command line option.

How does WebKit determine what to composited? Since the goal is to fully take advantage of the GPU, there are few particular operations which are suitable for such a composition. Among others are transparency (opacity < 1.0) and transformation matrix. Ideally we would just use composition for the entire web page. However, composition implies a higher memory allocation and a quite capable graphics processor. On mobile platforms, these two translate into additional critical factor: power consumption. Thus, one just needs to draw a line somewhere and stick with it. Hence, that's why currently (on iOS) translate3d and scale3d are using composition and their 2-D counterparts are not. Addendum: on the desktop WebKit, all transformed element is accelerated, regardless whether it's 2-D or 3-D.

If you make it this far, here are few final twists.

First of all, just like the tiled backing store approach I explained before, accelerated composition does not force you to use the graphics processor for everything. For efficiency, your layer (backing store) might be mapped to GPU textures. However, you are not obligated to prepare the layer, i.e. drawing onto it, using the GPU. As an example, you can use a software rasterizer to draw to a buffer which will be mapped to OpenGL texture.

In fact, a further variation of this would be not to use the GPU at all. This may come as a surprise to you but Android 2.2 (Froyo) added composition support (see the commit), albeit doing everything with in software (via its Skia graphics engine). The advantage is of course not that great (compared to using OpenGL ES entirely), however the improvement is really obvious. If you have two Android phones (of the same hardware specification), one still running the outdated 2.1 (Eclair) and the other with Froyo, just open the Falling Leaves demo and watch the frame rate difference.

With the non-GPU composition-based CSS animation in Froyo, translate3d and other similar tricks do not speed-up anything significantly. In fact, it may haunt you with bugs. For example, placing form elements in a div could wreck the touch events accuracy, mainly because the hit test procedures forget to take into account that the composited layer has moved. Things which seem to work just fine Eclair may start behaving weird under Froyo and Gingerbread. If that happens to you, check your CSS properties.

Fortunately (or unfortunately, depending on your point of view), Android madness with accelerated composition is getting better with Honeycomb and further upcoming releases. Meanwhile, just take it for granted that your magical translate3d spell has no effect on the green robots.

Last but not least, I'm pretty excited with the lightweight scene graph direction in the upcoming Qt 5. If any, this will become a better match for QtWebKit accelerated composition compared to the current Graphics View solution. This would totally destroy the myth (or misconception) that only native apps can take advantage of OpenGL (ES). Thus, if you decide to use web technologies via QtWebKit (possibly through the hybrid approach), your investment would be future-attractive!

progressive rendering via tiled backing store

Imagine you have to create a CAD-grade application, e.g. drawing the entire wireframe of a space shuttle or showing the intricacies of 9-layer printed circuit board. Basically something that involves a heavy work to display the result on the screen. On top of that, the application is still expected to perform smoothly in case the user wants to pan/scroll around and zoom in/out.

The usual known trick to achieve this is by employing a backing store, i.e. off-screen buffer that serves as the target for the drawing operations. The user interface then takes the backing store and displays it to the user. Now panning is a matter of translation and zooming is just scaling. The backing store can be updated asynchronously, thus making the user interaction decoupled from the complexity of the rendering.

Moving to a higher ninja level, the backing store can be tiled. Instead of just one giant snapshot of the rendering output, it is broken down to small tiles, say 128x128 pixels. The nice thing is because each tile can be mapped as a texture in the GPU, e.g. via glTexImage2D. Drawing each textured tile is also a (tasty) piece of cake, GL_QUAD with glBindTexture.

Another common use-case for tiling is for online maps. You probably use it every day without realizing it in Google Maps, OpenStreetMap, or other similar services. In this case, the reason is to use tiles is mainly to ease the network aspect. Instead of sending a huge image representing the area seen by the user in the viewport, actually lots of small images are transported and stitched together by the client code in the web browser.

Here is an illustration of the concept. The border of each tile is emphasized. The faded area is what you don't see (outside the viewport). Of course every time you pan and zoom, new fresh tiles are fetched so as to cover the viewport as much as possible.

When I started to use the first generation iPhone years ago, I realized that the browser (or rather, its WebKit implementation) uses the very similar trick. Instead of drawing the web page straight to the screen, it uses a tiled backing store. Zooming (via pinching) becomes really cheap, it's a matter of scaling up and down. Flicking is the same case, translating textures does not bother any mobile GPU that much.

Every iOS users know that if you manage to beat the browser and flick fast enough, it tries to catch up and fills the screen as fast as possible but every now and then you'll see some sort of checkerboard pattern. That is actually the placeholder for tiles which are not ready yet.

Since all the geeks out there likely better understand the technique with a piece of code, I'll not waste more paragraphs and present you this week's X2 example: full-featured implementation of tiled backing store in < 500 lines of Qt and C++. You can get the code from the usual X2 git repository, look under graphics/backingstore. When you compile and launch it, use mouse dragging to pan around and mouse wheel to zoom in/out. For the impatient, see the following 50-second screencast (or watch directly on YouTube):

For this particular trick, what you render actually does not matter much (it could be anything). To simplify the code, I do not use WebKit and instead just focus on SVG rendering, in particular of that famous Tiger head. The code should be pretty self-explanatory, especially for the TextureBuffer class, but here is some random note for your pleasure.

At the beginning, every tile is invalid (=0). Every time the program needs to draw a tile, it checks first if the tile is valid or not. If yes, it substitutes it with the checkerboard pattern instead (also called the default texture) and triggers an asynchronous update process. During the update, the program looks for the most important tile which needs to be updated (usually the one closes to the viewport center). What is a tile update? It's the actual rendering of the SVG, clipped exactly to the rectangular bounding box represented by the tile, into a texture.

To show the mix-n-match, I actually use Qt built-in software rasterizer to draw the SVG. That demonstrates that, even though each tile is essentially an OpenGL texture, you are not forced to use OpenGL to prepare the tile itself. This is essentially mixing rasterization done by the CPU with the texture management handled by the GPU.

As I mentioned before, panning is a matter of adjusting the translation offset. Zooming is tricky, it involves scaling up (or down) the textures appropriately. At the same time, it also triggers an asynchronous refresh. The refresh function is nothing but to reset all the tiles to invalid again, which in turns would update each one by one. This gives the following effect (illustrated in the screenshot below). If suddenly you zoom in, you would see pixelated rendering (left). After a certain refresh delay, the tile update makes the rendering crisp again (right).

Because we still need to have the outdated tiles scaled up/down (those pixelated ones), we have to keep them around for a while until the refresh process is completed. This is why there is another texture buffer called the secondary background buffer. Rest assured, when none of the tiles in the background buffer is needed anymore, the buffer is flushed.

If you really want to follow the update and refresh, try to uncomment the debug compiler define. Beside showing the individual tiles better, that flag would also intentionally slows down both update and refresh so your eyes can have more time to trace them.

BTW how would you determine the tile dimension in pixels? Unfortunately this can vary from one hardware to another. Ideally it's not too small because you'd enjoy the penalty of logical overdraw. If it's too large, you might not be progressive enough. Trial and error, that can be your enlightenment process.

Being an example, this program has a lot of simplifications. First of all, usually you want the tile update to take place in a separate thread, and probably updating few tiles at once. With a proper thread affinity, this helps improving the overall perceptive smoothness. Also, in case you know upfront that it does not impact the performance that much, using texture filtering (instead of just GL_NEAREST) for the scaling would give a better zooming illusion.

You might also see that I decided not to use the tile cache approach in the texture buffer. This is again done for simplicity. The continuous pruning of unused textures ensures that we actually don't grow the textures and kill the GPU. If you really insist on the absolutely minimal amount of overdraw and texture usage, then go for a slightly complicated cache system.

Since I'm lazy, the example is using Open GL and quad drawing. If you want to run it on a mobile platform, you have patch it so that it works with Open GL ES. In all cases, converting it to use vertex and texture arrays is likely a wise initial step. While you are there, hook the touch events so you can also do the pinch-to-zoom effect.

If you are brave enough, here is another nice final finish (as suggested by Nicolas of InfoVis and PhiloGL fame). When you zoom in, the tiles in the center are prioritized. However, when you zoom out, the tiles nearby the viewport border should get the first priority, in order to fill the viewport as fast as possible.

Progressive rendering via a tiled backing store is the easiest way to take advantage of graphics processor. It's of course just one form, probably the simplest one, of hardware acceleration.

rectangular gradient

Thorsten Zachmann, from Calligra (and previously KOffice) fame, asked me once on how to draw a different kind of gradient: a rectangular one. While Qt itself has built-in supports for linear, radial, and conical gradient types, apparently for office apps we may need more than that. In short, the goal is creating the following:

It turns out that this is not so difficult at all, about 50 lines of code. Check it out at the usual X2 repository and find it under graphics/rectgradient.

Basically it boils down to a two-step process, as illustrated below. The first one is easy, just create a linear gradient from the center going north and south. The second one is similar, but now we are going east and west and clip it to two triangles. Once we combined both, we get the rectangular gradient.

Have fun with the gradient!

command line CoffeeScript

CoffeeScript seems to be picking up some momentum these days. No doubt, it is very valuable to help writing cleaner code.

The command-line choices to run CoffeeScript compiler right now are either using Rhino (jcoffeescript) or using NodeJS. While I love NodeJS, seems that it is an overkill to require the entire NodeJS stack/infrastructure/package manager to invoke CoffeeScript compiler.

The solution is to use V8, the powerful JavaScript engine, with a little binding so that it can access file system. This is exactly filejs, something I have shown before, e.g. to invoke JSLint from command line.

Combining filejs and CoffeeScript is terribly easy. Just follow these steps.

Note: filejs does not support Windows yet. Sorry.

First of all, if you have not done it, build filejs. Go to the X2 repository, it is under the javascript/filejs folder. Open the included README.TXT and follow the instructions on how to build V8 and filejs.

After you build it, copy both filejs executable and coffee.js to somewhere in your PATH. Usually I stash that kind of stuff in ~/bin and ensure that ~/bin is in my PATH.

Now get coffee-script.js (the CoffeeScript to JavaScript compiler) and store it somewhere, e.g. ~/bin again.

Create a new file called coffee, which has the following one-line content:

filejs ~/bin/coffee.js $1

Make that file executable and then save it to ~/bin (again).

Open coffee.js and modify the value of the compiler variable to point to your coffee-script.js. Note: this must use the absolute path name.

Now you can do the following:

coffee hello.coffee

If hello.coffee is your script written in CoffeeScript, the converted JavaScript version will be dumped to the standard output.

Feel free to tweak coffee.js so that it understands and passes various CoffeeScript compiler options!

Fun, isn't it?

X2 from Ofi Labs: wrap-up 2010

It got started when I needed a new home for my examples. It has even a nice logo.

sensor

accelerometer viewer for Maemo 5 (Nokia N900).

bouncing ball, where the gravity affects the movement of the ball.

box of marbles, where the gravity affects a bunch of colored marbles.

combining accelerometer and network to do inter-device marbles transfer.

motion and orientation for web applications.

web-based version of marble box.

widgets

morphing clock, where the transition between the digital and analog version is a kind of morphing effect.

qpalette viewer so you know which color is which one.

graphics

fast approximation of Gaussian blur to create a blurry drop shadow.

command-line capture tool to save maps from OpenStreetMap, MapQuest and Ovi Maps.

simple tool to list all chunks inside a PNG image.

webkit & javascript

file processing, including using jslint, in command-line using JavaScript.

play Canvas-based game as normal desktop app.

offline, command line beautifier for JavaScript code, utilizing Qt Script.

another variant of the beautifier, this time using V8.

minimalistic editing widget for JavaScript code, with custom syntax highlighting.

white background is boring? just try some color inverted web pages.

detect the closest link to ease following it on a touch device.

Canvas pixel manipulation for plasma effect.

network

simple proxy server for HTTP, in 100 lines.

tracenet: trap all network requests+replies to show them with Speed Tracer.

filterproxy: another variant of the proxy server with added URL filtering feature.

motion vs orientation

I already showed the use of device orientation in a fun game called Box of Marbles. Based on my experience trying out this API implementation, often it is easy to get confused by device motion vs device orientation. Thus, I created two web-based applications (using on Sencha Touch): Device Motion and Device Orientation. Each basically just visualizes the values of the data in a bunch of sliders:

You can find the code in the usual X2 repository, under the subdirectory javascript/devicemotion and javascript/deviceorientation, respectively. Obviously, the rest of Sencha Touch library is needed before you can deploy and run the examples. For the lazy, just run the live demo for Device Motion and Device Orientation (they support offline mode).

While right now only iOS 4.2 on iPhone/iPad/iPod Touch implements this API, I have a strong hope that Android (post Gingerbread) and Nokia (thanks to QtWebKit and Qt Mobility) will support this cool API in the near future. Well, at least I do plan to revive my Symbian setup to custom compile QtWebKit and test it on Nokia N8.

Side note: from my limited understanding, seems that both accelerometer and gyroscope are needed in order to supply the accurate data, much like in an inertial navigation system ("Inertial Phone", anyone?).

Box of Marbles meets Device Orientation

Remember my Box of Marble demo?. It was a native Qt/C++ application running on Maemo-powered Nokia N900.

With the wave of DeviceOrientation Event Specification implementation, now it's possible to have the demo as a web app, running just inside the web browser. In fact, this is what you can see from this demo: ariya.github.com/js/marblebox.

Right now, it works with iPhone or iPad or iPod Touch running the new iOS 4.2, due to the brand new DeviceMotionEvent support. If you own a MacBook with accelerometer and you use Google Chrome, that works as well (tilt your laptop to see the gravity effect). But fear not, with the recent support to hook the acceleration data from Qt Mobilty to QtWebKit, don't be shocked if the next Nokia phone with built-in QtWebKit sports this feature as well.

While my previous demo uses Chipmunk physics engine, this web version relies on box2d-js instead, mostly because I am too lazy to port Chipmunk to JavaScript and also I want to try something else. If you are curious about the code, check the usual X2 repository under javascript/marblebox directory. Enjoy!

V8 + jslint + vim

Usually, you would want to use scripting solution, e.g. Perl/Python/whatever, to manipulate file contents. Somehow, it's also fun to use JavaScript instead. After I did the V8-based jsbeautify, I was doing V8-based jslint as well. Then I realized, let's just extend it to be generic. With API loosely modelled after CommonJS, filejs was born. Find it in the usual X2 repository under the javascript/filejs. There are two examples so far, ROT-13 and line counter. If you write more examples, feel free to pass them to me!

Of course, the twist is: use filejs to drive a command-line jslint. This is one way to do it. First build filejs (follow the instructions in the included README), then place the executable in your PATH. Create a simple shell script, which contains one line filejs /path/to/filejs/jslint.js $1 and make it executable. That's it! If you also store this shell script in your PATH, then you just need to run:

jslint source-code.js

If you use vim, jslint can be combined with Quickfix. First of all, associate *.js with the tool by putting this line in your personal .vimrc:

au FileType javascript set makeprg=jslint\ %

Now open a JavaScript file and run :make (or whatever shortcut you map this into), which will launch jslint with the current file. After a while, use :cope to open the quickfix little window, move up and down, and press Enter to bring you back to the main editor and set the cursor at the specified problem. Use standard window navigation, e.g. Ctrl+W W to switch to quickfix pane again. Check the quickfix documentation for details. All in all, you may want to map mostly used commands to some shortcuts for faster access.

Note that since this is dynamic JavaScript, rather than matching errorformat, technically I just tweaked the tool to spit something similar to what a C/C++ compiler would do. Also, if you need different jslint options, simply edit the invocation to suit your needs.

It's been a while since I blogged about vim. The last one was about the project plugin, which is surprisingly still quite popular. Hopefully this one also teaches you a trick or two, especially if you work a lot with JavaScript code.

Note 1: It does not work on Windows yet. No idea if I would have the time to do it, patch is welcomed.

Note 2: This is not a replacement for NodeJS, nor would it grow to include more functions.

Note 3: For the sake of completeness, let me mention that there are already other countless solutions for command-line jslint (using Rhino, SpiderMonkey, JSC, etc), even with vim integration.

chunks inside PNG

Here is a few minutes of hack which would hopefully help someone else:

pngchunks logo.png
File size: 5967 byte(s)

Offset    Chunk    Size
      8    IHDR      13
     33    iCCP    2627
   2672    IDAT    3271
   5955    IEND       0

Useful to quickly find out if we can further strip unnecessary chunks from the PNG image. Or even as a warming up before you fire your favorite hex viewer/editor.

It's pure, stand-alone, self-contained C code (i.e. compile using gcc -o pngchunks pngchunks.c), available at the usual X2 repository, find it under graphics/pngchunks.

yet another command-line JavaScript beautifier (based on V8)

While we're still on the topic of JavaScript, I took the afternoon break to create a command-line runner for jsbeautifier.org . In case you miss it, I did the similar thing using Qt Script module. This time however, the command-line tool uses and exploits Google V8 instead.

If you update your clone of js-beautify repository, check out the fresh v8 subdirectory. I also mirror the code in the usual X2 repository, under the javascript/jsbeautify8 subdirectory. Also, grok the included README.txt first.

Update: I added --overwrite option which (surprise!) will overwrite the original source file (and thus, use it with care). This is useful if you invoke the beautifier tool from your text editor. Nicolas also implemented various settings support so you can specify the indentation level, braces placement, etc.

This is probably the last thing the world needs now, but hey, it was a fun break.

color inversion for web pages

Something I worked on during my last few weeks with Qualcomm was color inversion for the web browser in Android. The patch was then integrated by Enrico (see the diff) because I switched job.

Later on this feature was also pulled into CyanogenMod. Thus, if you are running CyanogenMod 6 on your shiny Android device, you can try this feature already! In the web browser, just pick the Settings menu and then scroll a bit until you see a checkbox for Invert Color. You should get something like the captured screens below:

The idea behind this is to reduce the power consumption of Organic LED display, because on mostly-white web page, it hungrily grabs to 3x more power compared to LCD. This is similar to Jeff's trick of applying color filter in SurfaceFlinger.

Doing it at the user-space level gives one advantage: we can keep the embedded images non-inverted. Blindly inverting the entire screen would result in web pages look rather funky, especially those new sites with photos to accompany the articles. Of course, this hackish approach will not work 100%, (hint: double XOR?) since the web designer likely never intend the page to be color inverted. However it seems to work most of the time, at least with pages which do not have ueberartistic look-and-feel.

Faithful followers of my blog know that I already played with the color inversion ages ago, in the form of giving night mode appearance for QWebView. Thus, it happened that in one afternoon break I played with QtWebKit to do something similar to this selective color inversion.

Now, in the case of Android WebKit, the effect was rather easy to achieve. This is because Skia's SkCanvas is basically an interface which can be subclassed easily, while none of the functions in QPainter is virtual. The trick using Skia was to use a proxy canvas (a slight variation of Skia's built-in SkProxyCanvas) which I invented for this purpose but somehow also useful for another feature.

For QtWebKit however, we need to tweak it with some QPaintEngine voodoo. Check out the code at the usual X2 repository, find it under webkit/nightcapture (it still has some rooms for improvement, left as exercises for the adventurous readers). For simplicity, I made it as capturing tool so you need to pass a URL and the output filename. Unless you do something wrong, expect to get something like this:

The trick is the same as my approach for the Android patch: invert the colors of each image before drawing it, at the end invert the entire viewport. This seems like slow, but it's the best compromise I found out working with most sites. Note how we skip inverting any brush pixmap as usually brush is for the tiled background (and thus we want to keep it that way, i.e. not inverted). You can do some fancy magic with clipping and whatnot, but I doubt the end result is much better. Again, this whole stuff is a hack anyway and there will be always corner cases which will not work, no matter what approach you pick, thus there is no need to make it more complicated that it should be.

Just like they say, we code for life, i.e. the battery life.

invade & destroy

Developing games using HTML5 Canvas and JavaScript is simply the future. Something classic like Egor Balishev's RGB Invader, an entry in the 10K Apart, is usually my favorite.

If for some reasons (which I would not elaborate in this post) you need to deploy the game as a desktop or a mobile application, using Canvas and JavaScript gives another advantage: just package it with WebKit, for example QtWebKit if you do not mind using Qt.

However, since usually it is just about Canvas, often it is enough to run the game logic with a JavaScript interpreter and use some Canvas implementation for the display. This was something that I demonstrated before, i.e. running the Monster Evolution demo via JavaScriptCore and V8. Applying the same technique, and adjusting the shim so that it layers whatever RGB Invader requires, gives the following:

Basically it is using Qt Script (also in attempt to popularize it even more) along with a mix of C++/QObject and JavaScript machinery to fool the game code to think it's running in a browser environment. It was a nice afternoon fun project.

The code is in the X2 repository, find it under the javascript/invader directory. Get it while it's hot, and save our Earth!

offline, command-line beautifier for JavaScript code

There are many different ways to autoformat JavaScript code, my favorite is always jsbeautifier.org. Apparently, you can also use it locally without a web browser since it supports running it with Rhino.

It took me just few minutes to reimplement the same functionality, but using Qt Script, the fantastic ECMAScript support built into Qt.

The code is in the usual X2 repository, under the javascript/jsbeautify. Note that this is just a convenient mirror as Einar's js-beautify repository already includes this Qt Script driver.

With the rise of server-side JavaScript and desktop-esque JavaScript tools, I hope Qt Script will become more popular. It is sadly still "underrated" right now.

capture OpenStreetMap and MapQuest (and Ovi Maps)

Over a year ago, I showed how to render OpenStreetMap on your Qt application. While there were few follow-ups after that, I never found some time to clean it up. But worry no more, here is one OpenStreetMap-related example (at roughly 250 lines of code) I just made public, freshly available from X2 repository under the graphics/mapsnap. It's operated from command line, basically you pass the center latitude and longitude, zoom level (1..17, with 17 is the detailed, street-level zoom), output filename (e.g. mymap.png), and the size.

The following image is the result of running:

mapsnap 37.45108 -122.15917 12 sample.png 600 450

The tool will grab the tiles, more precisely Mapnik-rendered tile images, and stitch them for the final outcome:

Few weeks ago I wrote about MapQuest embracing OpenStreetMap. One positive impact of this awesome move is that you can show OpenStreetMap-based MapQuest tiles in your application. It's just the same map data but rendered using different styles. Running the previous example but with MapQuest flavor gives the following:

When checking out Qt Mobility after its 1.1 TP announcement, I found out that its Maps/Navigation API actually uses the tile data from Ovi Maps. If you read the source code, the tile server encoding scheme is pretty obvious. Since it is also based on Mercator projection, changing this MapSnap example to use tiles from Ovi Maps is a breeze: it's a matter of setting up the correct tileURL.

I'm curious about the terms of such Ovi Maps tile usage, though. Although it is more or less "exposed" via Qt Mobility and there is no API key whatsoever, surely it should not mean the tiles are free for everyone, should it? As a comparison, Google Static Maps API specifically allows only browser-based application. Feel free to share your investigation!

minimalistic JavaScript editing widget

In the spirit of clean-up-and-publish, here is another code example of X2: a subclass of QPlainTextEdit which acts as a nice and minimalistic JavaScript editor. Basically it just adds a sidebar for the line number and a syntax highlighter. Thanks to QPlainTextEdit and QSyntaxHighlighter, this editing widget is quite performant. At just about 500 lines, think about it as another example on how to use these two classes.

It does support specifying different colors so you can have funky color scheme if you want:

The widget is BSD licensed. Find it out in the X2 repository under the javascript/jsedit subdirectory.

It does not have fancy features such as code folding or autocomplete (not sure I would have time to add them, so patches are welcomed!), but if you want feature packed editor, use QScintilla, KDE's Kate, Qt Creator's editor, or grab something else or even write your own.

the art of blurring the shadow

In the recent weeks, sporadically I have been working on QtWebKit to fix the missing blur support for Canvas and CSS shadow (see the tracking bug 34479). This brings some good memory; before I left Nokia, I was involved in prototyping the special effect stuff, both for the low-level (non-public) QPixmapFilter and the high-level QGraphicsEffect.

Usually blurring drop shadow for a shape is a very typical: grab the alpha channel of the shape, apply the blur filter, and finally tint it with the intended shadow color. The last step is trivial using the SourceIn composition mode. The blur filter is supposed to follow the SVG specification on feGaussianBlur. The said specification mentions one possible way to approximate the perfect Gaussian blur: three successive box blurs.

Since QPixmapFilter is private anyway and QGraphicsEffect is not suitable for this task, an attempt to implement what the specification outlines became my hobby for a few evenings. This is basically what emerges as the shadow blur implementation in QtWebKit. For the sake of code reuse, I pushed the implementation to the X2 repository under the graphics/shadowblur subdirectory. The shadowBlur() function itself is BSD licensed, there is a demo program included for your pleasure:

Performance-wise, the code is as satisfactory (for such a portable implementation). Another good approach is to use stack blur, adopted among others by AGG. KHTML notably uses stack blur for its Canvas shadow blur support. Exponential blur is known to be very fast, although quality-wise it deviates farther from a true Gaussian one. The portable, raster version of QGraphicsDropShadowEffect (via QPixmapFilter) is using this algorithm.

For this particular use-case, namely blurring the shadow (as opposed to generic blur filter), I'm surprised that the Gaussian blur approximation is not necessary slower than KHTML's stack blur approach or even QGraphicsDropShadowEffect's exponential blur. I did a quick benchmark, measuring the time spent creating the shadow for two images (horizontal 149x13 and vertical 13x149) for different radii (small 4px and medium 17px), on a Core i7 machine. The outcome is shown in the following bar chart. The result was from several runs, with the overall confidence level observed to make sure it was statistically sound. Still, take it with a pinch of salt.

With a very large blur radius, e.g. 50, stack blur performance deteriorates quickly, probably because the pre-loop initial setup. Exponential blur is pretty much radius-agnostic, although I can't find cases where it wins against my shadow blur code (which is BTW only 60 lines). Larger source images would highlight the performance difference even more. Maybe I am doing something wrong, or maybe that's how it is. In any case, I insert a low-priority entry in my (already infinite) TODO list: spend some quality evenings with callgrind and examine those blur implementations.

On mobile devices however the situation is reversed. The approximated Gaussian blur is consistently slower, around 10%, compared to exponential or stack blur, when tested on 600 MHz Cortex A8-powered Nokia N900. Due to slower memory speed and smaller cache, the repetitive memory access for the successive box blurs cancels its fast processing benefit. Hopefully the 1 GHz generation CPU (like in Nokia N8) and improved memory bus will eliminate this minor slow down.

Anyway, in all cases, just like Andreas mentioned, now you can enjoy the fancy Parcycle demo in its full glory:

box of marbles redux

If you enjoy the box of marbles demo, now we extend it to include some basic network support. The premise is simple, look at this short video first (or watch on YouTube):

The code to handle the physics of the marbles remains the same, i.e. we just use Chipmunk physics engine. However, I added a simple feature to transfer marbles from one place to another. To keep it simple, it is done using UDP. The Qt network module supports UDP quite well, making the datagram code short and readable.

The intended use of this example is easy: run the desktop version and then run the mobile version (tested with Nokia N900). Each instance should find each other and start communicating. Again, we simplify the situation here and handle only 2 (two) peers. To facilitate troubleshooting, the application window title will contain the network address information, if the two peers are fully connected. To avoid complicated setup, discovery is carried out automagically through broadcast. This makes such a demo runs only under the same subnet, which is not a big deal.

It would have been much more fun doing the transfer between two smartphones (instead of a phone and my laptop). However, I own only one Nokia N900. Hint: I will not refuse your donation of Nokia N8 or (preferably) MeeGo-powered Nokia N9 (i.e. the N900's successor, whatever the real name is).

If you want to give it a try, head to the usual X2 repository and look at demo/marblenet subdirectory. Again, have the patient to follow the README file before you start compiling it.

Of course, feel free to extend this example to suit your (more wild) fantasy!

box of marbles

The next logical step after simple bouncing ball example is something which uses a real, full-featured physics engine. Box2D is usually the popular choice. Combining Box2D with Qt has been done by many people, recently demonstrated before by Andreas and Thorbjørn. I decided to pick something else, i.e. Chipmunk physics engine.

Because this is supposed to an example, I tried to make it as simple as possible (you'd be able to extend it, once you grab the basics). Basically we have a box full of colorful marbles (yes, I loved to play marbles when I was a kid, there was not any PlayStation back then). A mouse click, or a screen tap, will generated a new marble with a random color. If you run the example on Nokia N900, you can control how the marbles move and hit each other by tilting and shaking the phone.

Check the code yourself at the usual X2 repository under the demo/marblebox subdirectory. Make sure you open and follow the instructions in the included README file.

For a sneak peek, just watch this video (or enjoy on YouTube).

There is still a sequel to this marble box. And still with Chipmunk.

bouncing ball with accelerometer on N900

First of all, I apologize for my laziness in updating X2 with new code example. I have actually written quite a number of interesting examples, some of which have even been shown back in March, during my talk at Bossa Conference 10, though I did not find the time to clean up and polish them. Although I'd face new challenges in my upcoming adventure, I am quite confident I will reach the designated rate of new X2 example fortnightly.

Now let's focus on the newest example: a minor modification to the previous accelerometer code on Nokia N900 [1]. There has been confusion with my statement there: put this function is a separate thread. This is the alternative to a non-blocking D-Bus code. The main goal of course is not to be able to get faster acceleration values per second, it is only to prevent your code from being blocked by the synchronous D-Bus call.

Rather than just updating the code with the threaded version, I also added some high-school Newtonian physics. Instead of boring sliders, you'll get a ball which moves based on the acceleration [2], i.e. it follows the gravity if you keep your N900 straight.

Here is the obligatory video. Or watch directly on YouTube.

The code can be found in the X2 repository under the sensor/bouncingball subdirectory.

In the next installment, we will integrate a third-party real physics engine and make the example more alive!

[1] Another approach is to use Qt Mobility. However, QTMOBILITY-381 (which was spawn from QTMOBILITY-326) has not been solved yet (as of today).
[2] The overall math is not too scientifically correct, but hey, I always need to leave out something, for your homework :P

proxy server with filtering feature

Beside exploring San Diego, I had done some coding intermittently only. My apology if I do not update X2 with fresh new examples often enough.

Having said that, here is one network-related example: a minor tweak to the previous example of Qt-based proxy server. Basically it adds a minimalistic URL filtering support, in the form of blacklisting certain URLs which start with some predefined strings. The code is available in the usual place, X2 repository, under the directory network/filterproxy.

While major browsers support some variants of content blocking, be it via an extension like AdBlock or as a feature built-in into the browser itself, this new filterproxy should work with any browser that supports proxy. Alas, I did not bother to implement an AdBlock-compatible rule system because it would complicate the code. Again, consider this is a proof of concept only. A challenging exercise would be to fully support the most known subscription filters.

It is unheard that content filtering can dramatically improve your browsing experience. Because it cuts the bandwidth usage, it does translate to lower cost for those who are not lucky enough to get unlimited data plan. But most importantly, throwing garbage out of the web pages definitely speeds up the page loading. For this filterproxy example, I did a very unscientific benchmark and test it with Detik.com news site (now you get the answer why the included blacklist.txt contains only some basic advertisement-laden sites). The screenshots below (click to enlarge) show the unfiltered version (left) and the filtered version (right). Notice also the whopping 40% of bandwith saving!

My promise was to post two variations from that simple proxy example. This counts as one of them, and when the time allows me to clean-up to the other, you'll know it. Stay tuned and happy proxying!