Archive for April, 2011
Updated: added other examples from WebKit, IGEL and RIM
Yes, it may be a surprise, but that’s the beauty of Open Source – you never know where your contributions will be found. In this regard, I received a gentle mention from my friend Felipe Ortega of the Libresoft group of a nice snippet of research from Luis Canas Diaz, “Brief study of the Android community“. Luis studied the contributions to the Android code base, and splitted the contributions using the email of the originator, assigning those with “google.com” or “android.com” as internal, and classifying the others. Here is a sample of the results:
Luis added: “Having a look at the name of the domains, it is very surprising that Nokia is one of the most active contributors. This is a real paradox, the company that states that Android is its main competition helps it!. One of the effects of using libre software licenses for your work is that even your competition can use your code, currently there are Nokia commits in the following repositories:
In fact, it was Nokia participation in Maemo (and later Meego) and its funding of the dbus and bluez extensions that were later taken up by Google for Android. Intrigued by this result, I made a little experiment: I cloned the full Android gingerbread GIT repo (2.3), separated the parts that are coming from preexisting projects like the Linux kernel and the various external dependencies (many tens of project – included, to my surprise, a full Quake source code…) leaving for example Chromium but removing WebKit. I then took apart the external projects, and counted Google contributions there in an approximate way, and folded back everything. You get a rough size of 1.1GB of source code directly developed or contributed by Google, which means that around 75% of the source code of Android comes from external projects. Not bad, in terms of savings.
Update: many people commented on the strangeness of having fierce competitors working together in ways that are somehow “friendly” towards a common goal. Some of my twitter followers also found the percentage of 75% of non-Google contributions to be high, and this update is meant to be an answer for both. First of all, there is quite a long history of competitors working together in open source communities; the following sample of Eclipse contributors provide an intial demonstration of that:
But there are many other examples as well. WebKit, theweb rendering component used in basically all the mobile platforms (except Windows Mobile) and on the desktop within Chrome and Safari was originally developed by the KDE free software community, taken by Apple and more recently co-developed by Nokia, Samsung, RIM and Google:
And on WebKit page, it is possible to find the following list:
“KDE: KDE is an open source desktop environment and application development framework. The project to develop this software is an informal association. WebKit was originally created based on code from KDE’s KHTML and KJS libraries. Although the code has been extensively reworked since then, this provided the basic groundwork and seed code for the project. Many KDE contributors have also contributed to WebKit since it became an independent project, with plans that it would be used in KDE as well. This has included work on initially developing the Qt port, as well as developing the original code (KSVG2) that provides WebKit’s SVG support, and subsequent maintenance of that code.
Apple: Apple employees have contributed the majority of work on WebKit since it became an independent project. Apple uses WebKit for Safari on Mac OS X, iPhone and Windows; on the former two it is also a system framework and used by many other applications. Apple’s contribution has included extensive work on standards compliance, Web compatibility, performance, security, robustness, testing infrastructure and development of major new features.
Collabora: Collabora has worked on several improvements to the Qt and GTK+ ports since 2007, including NPAPI plugins support, and general API design and implementation. Collabora currently supports the development of the GTK+ port, its adoption by GNOME projects such as Empathy, and promotes its usage in several client projects.
Nokia: Nokia’s involvement with the WebKit project started with a port to the S60 platform for mobile devices. The S60 port exists in a branch of the public WebKit repository along with various changes to better support mobile devices. To date it has not been merged to the mainline. However, a few changes did make it in, including support for CSS queries. In 2008, Nokia acquired Trolltech. Trolltech has an extensive history of WebKit contributions, most notably the Qt port.
Google: Google employees have contributed code to WebKit as part of work on Chrome and Android, both originally secret projects. This has included work on portability, bug fixes, security improvements, and various other contributions.
Torch Mobile: Torch Mobile uses WebKit in the Iris Browser, and has contributed significantly to WebKit along the way. This has included portability work, bug fixes, and improvements to better support mobile devices. Torch Mobile has ported WebKit to Windows CE/Mobile, other undisclosed platforms, and maintains the QtWebKit git repository. Several long-time KHTML and WebKit contributors are employed by Torch Mobile.
Igalia: Igalia is a free software consultancy company employing several core developers of the GTK+ port, with contributions including bugfixing, performance, accessibility, API design and many major features. It also provides various parts of the needed infrastructure for its day to day functioning, and is involved in the spread of WebKit among its clients and in the GNOME ecosystem, for example leading the transition of the Epiphany web browser to WebKit.
Company 100: Company 100 has contributed code to WebKit as part of work on Dorothy Browser since 2009. This work includes portability, performance, bug fixes, improvements to support mobile and embedded devices. Company 100 has ported WebKit to BREW MP and other mobile platforms.
Samsung: Samsung has contributed code to WebKit EFL (Enlightenment Foundation Libraries) especially in the area of bug fixes, HTML5, EFL WebView, etc. Samsung is maintaining the official Efl build bots and the EFL early warning system.”
So, we see fierce competitors (Apple, Nokia, Google, Samsung) co-operating in a project that is clearly of interest for all of them. In a previous post I made a similar analysis for IGEL (popular developers of thin clients) and HP/Palm:
“The actual results are:
- Total published source code (without modifications) for IGEL: 1.9GB in 181 packages; total amount of patch code: 51MB in 167 files (the remaining files are not modified). Average patch size: 305KB, Patch percentage on total publisheed code: 2.68%
- Total published source code (without modifications) for Palm: 1.2GB in 106 packages; total amount of patch code: 55MB in 83 files (the remaining files are not modified). Average patch size: 664KB, Patch percentage on total published code: 4.58%
If we add the proprietary parts and the code modified we end up in the same approximate range found in the Maemo study, that is around 10% to 15% of code that is either proprietary or modified OSS directly developed by the company. IGEL reused more than 50 million lines of code, modified or developed around 1.3 million lines of code. …. Open Source allows to create a derived product (in both case of substantial complexity) reducing the cost of development to 1/20, the time to market to 1/4, the total staff necessary to more than 1/4, and in general reduce the cost of maintaining the product after delivery. I believe that it would be difficult, for anyone producing software today, to ignore this kind of results.”
This is the real end result: it would be extremely difficult for companies to compete without the added advantage of Open Source. It is simply anti-economic to try to do everything from scratch, while competing companies work together on non-differentiating elements; for this reason it should not be considered a strange fact that Nokia is an important contributor to Google Android.
I was quite intrigued by the WebP image encoding scheme created by Google, and based on the idea of a single-frame WebM movie. I performed some initial tests during initial release, and found it to be good but probably not groundbreaking. But I recently had the opportunity to read a blog post by Charles Bloom with some extensive tests, that showed that WebP was clearly on a par with a good Jpeg implementation on medium and high bitrates, but substantially better for small bitrates or constrained encodings. Another well executed test is linked there, and provide a good comparison between WebP, Jpeg and Jpeg2000, that again shows that WebP shines – really – in low bitrate condition. So, I decided to see if it was true, took some photos out of my trusted Nokia N97 and tried to convert them in a sensible way. Before flaming me about the fact that the images were not in raw format: I know it, thank you. My objective is not to perform a perfect test, but to verify Google assumptions that WebP can be used to reduce the bandwidth consumed by traditional, already encoded images while preserving most of the visual quality. This is not a quality comparison, but a “field test” to see if the technology works as described. The process I used is simple: I took some photos (I know, I am not a photographer…) selected for a mix of detail and low gradient areas; compressed them to 5% using GIMP with all Jpeg optimization enabled, took notice of size, then encoded the same source image with the WebP cwebp encoder without any parameter twiddling using the “-size” command line to match the size of the compressed Jpeg file. The WebP image was then decoded as PNG. The full set was uploaded to Flickr here, and here are some of the results:
Photo: Congress Centre, Berlin. Top: original Jpeg. middle: 5% Jpeg. Bottom: WebP at same Jpeg size.
Photo: LinuxNacht Berlin. Top: original Jpeg. middle: 5% Jpeg. Bottom: WebP at same Jpeg size.
Saltzburg castle. Top: original Jpeg. middle: 5% Jpeg. Bottom: WebP at same Jpeg size.
Venice. Top: original Jpeg. middle: 5% Jpeg. Bottom: WebP at same Jpeg size.
There is an obvious conclusion: at small file sizes, WebP handily beats Jpeg (and a good Jpeg encoder, the libjpeg-based one used by GIMP) by a large margin. Using a jpeg recompressor and repacker it is possible to even a little bit the results, but only marginally. With some test materials, like cartoons and anime, the advantage increases substantially. I can safely say that, given these results, WebP is a quite effective low-bitrate encoder, with substantial size advantages over Jpeg.
(This is an updated repost of an article originally published on OSBR)
I have followed with great interest the evolution of the Symbian open source project – from its start, through its tentative evolution, and up to its closure this month. This process of closing down is accompanied by the claim that: “the current governance structure for the Symbian platform – the foundation – is no longer appropriate.”
It seems strange. Considering the great successes of Gnome, KDE, Eclipse, and many other groups, it is curious that Symbian was not able to follow along the same path. I have always been a great believer in OSS consortia, because I think that the sharing of research and development is a main strength of the open source model, and I think that consortia are among the best ways to implement R&D sharing efficiently.
However, to work well, Consortia need to provide benefits in terms of efficiency or visibility to all the actors that participate in them, not only to the original developer group. For Nokia, we know that one of the reasons to open up Symbian was to reduce the porting effort. As Eric Raymond reports, “they did a cost analysis and concluded they couldn’t afford the engineering hours needed to port Symbian to all the hardware they needed to support. (I had this straight from a Symbian executive, face-to-face, around 2002).”
But to get other people to contribute their work, you need an advantage for them as well. What can this advantage be? For Eclipse, most of the companies developing their own integrated development environment (IDE) found it economically sensible to drop their own work and contribute to Eclipse instead. It allowed them to quickly reduce their maintenance and development costs while increasing their quality as well. The Symbian foundation should have done the same thing, but apparently missed the mark, despite having a large number of partners and members. Why?
The reason is time and focus. The Eclipse foundation had, for quite some time, basically used only IBM resources to provide support and development. In a similar way, it took WebKit (which is not quite a foundation, but follows the same basic model) more than two years before it started receiving substantial contributions, as can be found here.
And WebKit is much, much smaller than Symbian and Eclipse. For Symbian, I would estimate that it would require at least three or four years before such a project could start to receive important external contributions. That is, unless it is substantially re-engineered so that the individual parts (some of which are quite interesting and advanced, despite the claims that Symbian is a dead project) can be removed and reused by other projects as well. This is usually the starting point for long-term cooperation. Some tooling was also not in place from the beginning; the need for a separate compiler chain – one that was not open source and that in many aspect was not as advanced as open source ones – was an additional stumbling block that delayed participation.
Another problem was focus. More or less, anyone understood that for a substantial period of time, Symbian would be managed and developed mainly by Nokia. And Nokia made a total mess of differentiating what part of the platform was real, what was a stopgap for future changes, what was end-of-life, and what was the future. Who would invest, in the long term, in a platform where the only entity that could gain from it was not even that much committed to it? And before flaming me for this comment, let me say that I am a proud owner of a Nokia device, I love most Nokia products, and I think that Symbian still could have been a contender, especially through a speedier transition to Qt for the user interface. But the long list of confusing announcements and delays, changes in plans, and lack of focus on how to beat the competitors like iOS and Android clearly reduced the willingness of commercial partners to invest in the venture.
Which is a pity – Symbian still powers most phones in the world and can still enter the market with some credibility. But this later announcement sounds like a death knell. Obtain the source code through a DVD or USB key? You must be kidding. Do you really think that setting up a webpage with the code and preserving a read-only Mercurial server would be a too much of a cost? The only thing that it shows is that Nokia stopped believing in an OSS Symbian.
(Update: after the change of CEO and the extraordinary change in strategy, it is clear that the reason for ditching the original EPL code was related to its inherent patent grant, that still provides a safeguard against Nokia patents embedded in the original Symbian code. There is a new release of Symbian under a different, non-OSS license; the original code is preserved in this sourceforge project, while Tyson Key preserved the incubation projects and many ancillary documentation like wiki pages at this Google code project.)
A full copy of the original EPL