2010-01-18
Layers: Cross-Platform Acceleration
So, it's been a while since I've given everyone here an update, and for that I do apologize. First of all, Direct2D support is coming well, we're working hard to land it on trunk. I've been getting swift reviews on all my code from a lot of different people, which is great, but it will still take some time since the code is complex and touches a lot of the Mozilla tree. Once it lands it will be disabled by default, but people will be able to easily enable it in their firefox nightly builds! Ofcourse you can follow the status of this work by keeping track of bug 527707. It's not perfect yet, but once again, thanks to all who've given me all the great feedback to get this to a point where it's quite usable!
Layers
So, now to the actual title of this post. Layers. Layers are another API we are designing for Mozilla, which can be used to hardware accelerate certain parts of website rendering. Normally this is where I would start a long rant on why hardware acceleration is such a good thing, but since I've already done that 2 posts ago, I'll just refer you there.
First of all it is important to point out that Layers is by no means a replacement for Direct2D. Direct2D accelerates all rendering, from fonts to paths and all such things, layers is intended to allow accelerated rendering and blending of surfaces. A layer could be accessible as many different type of cairo surfaces, possible D2D, but also any other. What this means if the layer system is designed to be easily implemented on top of different APIs, Direct3D (9 or 10) but also OpenGL. This means that we hope this to provide a performance increase for users on Mac and Linux as well.
So what are these layers?
Essentially, they're just that, layers. Normally a website will directly be rendered, as a whole, to a surface (say a window in most cases). This single surface means that unless the surface itself accelerates certain operations internally, it is hard to apply any hardware acceleration there. The layer system starts out with a LayerManager, this layermanager will be responsible for managing the layers in a certain part of screen real estate(say a window). Rather than providing a surface, the layer manager will provide layers to someone who wants to draw, and the API user (in this case our layout system) can then structure those in an ordered tree. By effectively rendering specific parts of a website (for example transformed or transparent areas) into their own layer, the layermanager can use hardware acceleration to composite those areas together effectively. There can be several types of layers, but two layers really form the core of the system: ThebesLayers and ContainerLayers.
ContainerLayers
So, container layers also live up to their name. They are a type of layer which may contain other layers, their children. These types of layers basically form the backbone of our tree. They do not themselves contain any graphical data, but they have children which are rendered into their area. It can for example be used to transform a set of children in a single transformation.
ThebesLayers
Now, these sound a bit more exotic. Thebes is our graphics library which wraps cairo. A thebes surface is a surface which can be drawn to using a thebes context. ThebesLayers are layers which are accessible as a thebes surface, and can therefor be easily used to render content to. This could for example also be a Direct2D surface. ThebesLayers form leafs of our tree, they contain content directly, and do not have any children. However, like any other layer, they may have transformations or blending effects applied to them.
Other types of layers
Why would we have other types of layers you might ask? Well, there's actually several reasons. One of the layers we will be having are video layers, the reason is that video data is generally stored not in the normal RGB color space, but rather as YUV data(luminescence and chromatic information, not necessarily at the same resolution). As with any pixel operation, our graphics hardware is especially good at converting this to the RGB type of pixels we need on the screen, therefor this layer can be used specifically for video data. Another layer we will have is a hardware layer, this type of layer contains a surface which exists, and is accessible, solely on the hardware. These would be useful for example for WebGL, where we currently have to do an expensive readback from the graphics hardware to get a frame back into software. Using the hardware layer could then blend that WebGL content directly on the graphics hardware, skipping the intermediate copy to the normal memory.
So, now that we know that, a layer tree for a website with a div in it, in which there's a video and some other content, may look like this:
Simple example(actual results may differ)
Other advantages
There is a long list of other advantages we hope to achieve with this system. One of them is that the rendering of the layer tree could occur off the main thread. This means that if we integrate animations into the system, the main thread could be busy doing something(for example processing JavaScript), but the animations will proceed smoothly. Another advantage is that it offers us a system to retain part of our rendered websites, also, for example, parts which aren't in the currently viewed area. This could bring advantages especially where the CPU speed is low (for example mobile hardware), but we still want smooth scrolling and zooming.
Conclusions
Well, that explains the very basics of the layers system we're working on. Robert O'Callahan has already been doing all the hard work of writing the very first API for layers, as well as a preliminary integration into our layout system! See bug 534425 for our progress there. Additionally I've been working on an OpenGL 2.1 and D3D10 implementation of the API so far. I had a lot of feedback of people who were disappointed we were not offering something like Direct2D on other platforms. We've not given up on bringing something like that to other platforms. Layers should however, with less risk and work, bring a significant amount of hardware acceleration to other platforms already! I hope you enjoyed reading this and I've informed you a little more about our latest work in this area, and ofcourse reassured you that we're doing everything we can to bring performance improvements to as many users as possible.
21 comments
Could every layer use a different cpu core? Would that mean that if Firefox has per plugin, tab, layer processes it could use hundreds of cpu cores if they were available? Or is that wishful thinking?
@Ferdinand: Different CPU cores could draw to each layer in theory. But a layer by itself is passive, so there would have to be changes to layout in order to do that. So for now that is wishful thinking :-).
Great job.
OpenGL 2.1 and D3D10 means a lot of laptop users may not take those advantages.
Are there any plan to support OpenGL 1.4 and D3D7/8?
@Wan Li: Well, I didn’t do most of it ;-). So, I fully intend on supporting D3D9 for laptop users. (Although D3D10_Level9 will allow laptop users to use D3D10 with Vista & Windows 7). For OpenGL I must admit my knowledge lacks a bit in that department. But atleast some level of support for OGL 1.4 should be very possible, although we have not planned any of that. OpenGL ES will be supported though.
Layers sure sounds like excessive layering to me. I really like your goals… but why not do this inside Cairo? Cairo’s API is deliberately structured to admit this sort of acceleration/caching/parallelization behind the scenes.
@Ben: So far there’s no good hardware backend for cairo(there’s some, but they’re generally incomplete and don’t deliver the desired performance). D2D performs fairly well, but that would be windows only. Cairo would also have to be made to understand YUV and many more such things.
“So far there’s no good hardware backend for cairo(there’s some, but they’re generally incomplete and don’t deliver the desired performance).”
Sure, but the same is true of Firefox. Fixing Cairo, rather than adding another software layer on top, would be a much more logical architecture.
“Cairo would also have to be made to understand YUV”
This is already happening due to the conclusions of the Gstreamer-Cairo hackfest:
http://gstreamer.freedesktop.org/wiki/VideoHackfest/Conclusions
Note also that latest cairo git master has several GLSL shaders to speed things up. Progress is happening in Cairo, so you risk duplication of effort, and ultimately having to redesign your stack yet again.
Some of this stuff could be done by adding cairo API, yes. But fundamentally, to support hardware acceleration well, and to support rendering and animation driven by a thread other than our main thread (which does all the drawing), we need a retained-mode API, i.e. an API that lets the main thread describe a “scene” data structure which includes information like animation. That doesn’t fit into cairo’s immediate-mode API.
@Ben: As Robert points out, cairo’s immediate mode nature also makes it not too suitable for hardware acceleration at this point. Yes, we could spend a lot of effort at rewriting cairo, but it would be considerably more effort I believe than building a system such as layers. For D2D the effort involved in implementing it as a backend was reasonable, so we could do that. But in D2D too we suffer from the immediate nature of cairo (i.e. the lack of support for retained paths and such things).
I certainly see the importance of multithreaded rendering. I’m no expert on the Cairo API, and having read the bugzilla tickets I have total faith in both of your expertise here.
I’m still not sure I see why Cairo’s API is necessarily immediate-mode. For example, with the SVG backend, no rendering occurs at all, so it’s hardly “immediate". It seems like what you want could be accomplished with a retained-mode Cairo backend that performs the appropriate multi-threading and caching.
Anyway, the real point is to try to write reusable software, in the hope of distributing the future maintenance burden. I can think of a lot of other projects that use Cairo, and that might like to adopt Layers if it can be cleanly separated from Gecko.
@Ben: In relation to your re-usability concern. I agree that this is a good idea. Layers is implemented as a separate component of Thebes, our graphics library. It is not integrated into the layout engine so to some extent it is reusable.
Glad to see that you made another post :).
so, when will there an updated build with layers and d2d ?
can someone give me the download link of firefox 3.7 installer with d2d most recently version
Will that work bring acceleration to svg rendering ?
yep, is there a new build that include this one and the D2D or something better than when we have for now?
Hi, are there any new build? plz, give me link on the new build.
Thank you.
hello, anybody give me link on the newwest build, please.
Tom’s Hardware has a review of Direct2D acceleration on different video cards. Apparently there is a huge problem with the latest AMD cards.
This may account for some of the performance problems you’ve been seeing.
http://www.tomshardware.com/reviews/2d-windows-gdi,2539.html
ff 3.7A2 is out, will it get into that build or what? and what about updates to the D2D thing?
Thank you for all the hard work BTW :)
Hi,
Do I understand correctly that I can now use regular nightly builds to get updates on you’re work?
This post has 1 feedback awaiting moderation...