At the Philly ETE conference Michael Tiemann presented some interesting facts about open source quality, and in particular mentioned that open source software has an average defect density that is 50-150 times lower than proprietary software. As it stands, this statement is somewhat incorrect, and I would like to provide a small clarification of the context and the real values:

• First of all, the average that is mentioned by Michael is related to a small number of projects, in particular the Linux kernel, the Apache web server (and later the entire LAMP stack), and a small number of additional, “famous” projects. For all of these projects, the reality is that the defect density is substantially lower than that of comparable proprietary products. A very good article on this is Succi, Paulson, Eberlein. An Empirical Study of Open-Source and Closed-Source Software Products, IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, V.30/4, april 2004, where the study was performed. It was not the only study on the subject, but all pointed at more or less the same results.
• Other than the software engineering community, some results from companies working in the code defect identification industry also published some results, like Reasoning Inc. A Quantitative Analysis of TCP/IP Implementations in Commercial Software and in the Linux Kernel, and How Open Source and Commercial Software Compare: Database Implementations in Commercial Software and in MySQL. All results confirm the much higher quality (in terms of defect per line of code) of the academic research.
• Additional research identified a common pattern: the initial quality of the source code is roughly the same for proprietary and open source, but the defect density decreases in a much faster way with open source. So, it’s not the fact that OSS coders are on average code wonders, but that the process itself creates more opportunity for defect resolution on average. As Succi et al. pointed out: “In terms of defects, our analysis finds that the changing rate or the functions modified as a percentage of the total functions is higher in open-source projects than in closed- source projects. This supports the hypothesis that defects may be found and fixed more quickly in open-source projects than in closed-source projects and may be an added benefit for using the open-source development model.” (emphasis mine).

I have a personal opinion on why this happens, and is really related to two different phenomenons:the first aspect is related to code reuse: the general modularity and great reuse of components is in fact helping developers, because instead of recoding something (introducing new bugs) the reuse of an already debugged component reduces the overall defect density. This aspect was found in other research groups focusing on reuse; for example in a work by Mohagheghi, Conradi, Killi and Schwarz called “An Empirical Study of Software Reuse vs. Defect-Density and Stability” (available here) we can find that reuse introduces a similar degree of improvement in the bug density and the trouble report numbers of code:

As it can be observed from the graph, code originated from reuse has a significant higher quality compared to traditional code, and the gap between the two grows with the size (as expected from basic probabilistic models of defect generation and discovery).

The second aspect is that the fact that bug data is public allows a “prioritization” and a better coordination of developers on triaging and in general fixing things. This explains why this faster improvement appears not only in code that is reused, but in newly generated code as well; the sum of the two effects explains the incredible difference in quality (50-150 times), higher than any previous effort like formal methods, automated code generation and so on. And this quality differential can only grow with time, leading to a long-term push for proprietary vendor to include more and more open source code inside of their own products to reduce the growing effort of bug isolation and fixing.

I already wrote something in response to Savio Rodrigues post on the lack of differentiation in R&D expenditure between RedHat (“OSS”) and Microsoft and Tibco (“proprietary”). Savio found in the similar spending in R&D an indication that OSS companies are not implicitly more efficient, while I believe that the differences appear outside balance sheets, both in stronger offering and in competition efficiency, both not apparent in R&D spending.

A recent research from Venice University’s TEDIS seems to match at least part of my hipotesis:

“Infine, confrontando i singoli dati relativi alle sole imprese con fatturato inferiore ai 500.000 euro con la variabile relativa alle classi dimensionali delle aziende clienti (per numero di dipendenti), si può ipotizzare una correlazione tra l’utilizzo di software Open Source e la capacità di attrarre clienti di dimensioni relativamente più grandi. A parità di fatturato, insomma, le aziende “solo Open Source”, sembrano avere maggiori chances di ottenere commesse da aziende con oltre 50 dipendenti (quindi medio-grandi rispetto al nostro universo di riferimento).“

(my english translation: “Finally, comparing the individual data on firms with turnover of less than 500,000 euros with the variable on size classes of customers (by number of employees), one can hipotesize a correlation between the use of software Open Source and the ability to attract customers of relatively larger scale. At the same turnover, in other words, companies “Open Source only” seem to have more chances to obtain work orders from companies with more than 50 employees (ie medium – large compared to our universe of reference).”

This, given the relative similarity of other data (like revenue-per-employee of the cluster) provide at least an hint that OSS gives “leverage” in the kind of activities that a small company can create or propose to the market. As I wrote in my previous post: “In the smallest example (100000 lines of code, still substantial) the average staffing is reduced from more than 20 developers to slightly less than 9, bringing this project within reach even by small companies, and in my personal view it explains the exceptional take-up of OSS by new and innovative companies, that even before external sources of capital (like VCs) are capable of creating non-trivial projects with very limited resources.”

After the publication of our revised OSS business model taxonomy, we have finally finished the second survey from FLOSSMETRICS, that now covers 218 companies (again, excluding companies that have less than 30% of revenues coming from OSS, and excluding companies that sell a proprietary product based on OSS, as there are simply too many of them). Some companies are using more than one model (for example, many dual-licensing companies also sell services and support, and thus can be classified also as product specialists).

With the actual numbers:

Some important considerations: product specialists are counted only when there is a demonstrable participation of the company into the project as “main committer”; otherwise, the number of specialists would be much greater, as some projects are the center of commercial support from many companies (a good example is OpenBravo or Zope). The distribution of revenue (approximate, as most companies are not publishing revenue data) seems to match that of average IT sector, with the vast majority of companies of small size (less than 5M$), around 10% are medium sized (5 to 20M$) and very few can be classified as “large”. Another observation is the fact that platform providers, while limited in number, tend to have a much larger revenue rate than both specialists or open core companies.

Overall there seems to be no significant difference in revenuse (comparing same-class companies) between product specialists compared to open core companies, but this is based on uncertain estimates of relative revenues, and should be taken as purely speculative. What seems to be constant is the reported increase in visibility and sales leads experienced by companies that adopted a pure open source model (be it dual licensing, specialists or based on indirect revenues); as before, it is possible to check this kind of increase only through web-based metrics (that are in many cases unreliable) and by indirect measurements like user participation in forums or dedicated conferences.

The “real” value of open source in financial terms has been one of my favourite arguments, and I had the opportunity to research in this area for a long time, thanks to our work for the customers for which we provide consulting on OSS business models. I recently found a new post on this by Bruno Von Rotz, with some preliminary conclusions like “total value of USD 100 to 150 billion.. I would assume that some of it comes from enterprises and large organizations, but this is probably max 20-30% of it.”

I would like to provide first of all a validation of the first part, and a comment on the second. As for the software market, Gartner published in 2006 a very well done study on OSS in the context of the overall software market, and among the various results there is one that I found quite interesting:

Considering the fact that some parallel data points (like results from the OECD estimates on software market) more or less confirm the predictions from Gartner, we can say that OSS has a financial value of 120B$ now, and will reach 150B$ in 2010, perfectly in line with the predictions from Bruno.

What I am not convinced is the calculation of the share 0f voluntary contributions Vs. company contributions. If Gartner data is accurate (and I believe that this is the case) we can expect that companies should contribute between 40% and 50% of value to a project; and this is somewhat consistent with projects like Linux or Eclipse where there is a large ecosystem, not only of adopters but of commercial companies working on top, and where company contributions are in that range. In this sense, I believe the 20%-30% percentage mentioned by Bruno to be too restrictive; the problem is that measuring code is not the only way to measure contributions. I use frequently this as an example:

“In the year 2000, fifty outside contributors to Open Cascade provided various kinds of assistance: transferring software to other systems (IRIX 64 bits, Alpha OSF), correcting defects (memory leaks…) and translating the tutorial into Spanish, etc. Currently, there are seventy active contributors and the objective is to reach one hundred. These outside contributions are significant. Open Cascade estimates that they represent about 20 % of the value of the software.”

Do these contributions appear as source code? No, exactly as localization efforts for OpenOffice or KDE do not appear in source code metrics. My belief is that the value of OSS right now is even much larger than 120B$, and that we have simply no way to measure this “hidden” value- but it’s there.

Two days ago, Matthew Aslett was so kind to comment on our new research on OSS business models; especially our new taxonomy, based on our latest extensions of the FLOSSMETRICS methodology and the increase in the number of surveyed companies. We both share the interest in having a clear, simple and usable set of definitions to avoid confusion when referring to specific business models; and he decided to publish the (previously private) set of definitions that were adopted in the CAOS report “Open source is not a business model“. As I would like to find a converging set of definitions, I will publish here a pre-release of our next edition of the OSS guide,  that covers in more detail what Lampitt classify as “vendor licensing strategy”. My hope is to see if it is possible to find an agreed-to definition, that can be shared by researchers and experts. As already mentioned in my previous post, we introduced several changes compared to the previous edition:

• It does, for the first time, disaggregate what was originally called ITSC (Installation, training, support, consulting) because many successful companies are now specializing in a single activity. This is a significant change from 2006 (when we started collecting data on OSS models), when companies were performing in a more or less undifferentiated way all those activities. We believe that this “specialization” will continue with the enlargement of the commercial OSS market.
• We removed the “badgeware” category from the list. We found that some of the vendors that originally followed this model disappeared, and  for those remaining protection from freeriding and overall model was more or less morphed into a “open core” or “split oss/commercial”. As the visibility clause can now be included in the GPLv3, I believe that the remaining few badgeware licenses will disappear quickly.
• As for our missing model (proprietary built on open) we found that basically the majority of products use OSS inside, so we are waiting to finish our second strand of research, on how to separate those products that are entirely OSS based from those that merely “use” OSS; our current idea is related to the “substitution principle”: is it possible to market the same product when all OSS components are substituted by non-OSS ones? Does it falls in the same market, or it becomes something radically different? This is a common theme, that  tries to answer questions like “would it have been possible for Google to be based on proprietary software?”

Taking this into consideration, here is our new taxonomy:

• Dual licensing: the same software code distributed under the GPL and a commercial license. This model is mainly used by producers of developer-oriented tools and software, and works thanks to the strong coupling clause of the GPL, that requires derivative works or software directly linked to be covered under the same license. Companies not willing to release their own software under the GPL can buy a commercial license that is in a sense an exception to the binding clause; by those that value the “free as in speech” idea of free/libre software this is seen as a good compromise between helping those that abide to the GPL and receive the software for free (and make their software available as FLOSS) and benefiting through the commercial license for those that want to maintain the code proprietary. The downside of dual licensing is that external contributors must accept the same licensing regime, and this has been shown to reduce the volume of external contributions (that becomes mainly limited to bug fixes and small additions).
• Open Core (previously called split OSS/commercial): this model distinguish between a basic FLOSS software and a commercial version, based on the libre one but with the addition of proprietary plugins. Most companies adopt as license the Mozilla Public License, as it allows explicitly this form of intermixing, and allows for much greater participation from external contributions, as no acceptance of double licensing is required. The model has the intrinsic downside that the FLOSS product must be valuable to be attractive for the users, but must also be not complete enough to prevent competition with the commercial one. This balance is difficult to achieve and maintain over time; also, if the software is of large interest, developers may try to complete the missing functionality in a purely open source way, thus reducing the attractiveness of the commercial version.
• Product specialists: companies that created, or maintain a specific software project, and use a pure FLOSS license to distribute it. The main revenues are provided from services like training and consulting (the “ITSC” class) and follow the original “best code here” and “best knowledge here” of the original EUWG classification. It is based on the assumption, commonly held, that the most knowledgeable experts on a software are those that have developed it, and this way can provide services with a limited marketing effort, by leveraging the free redistribution of the code. The downside of the model is that there is a limited barrier of entry for potential competitors, as the only investment that is needed is in the acquisition of specific skills and expertise on the software itself.
• Platform providers: companies that provide selection, support, integration and services on a set of projects, collectively forming a tested and verified platform. In this sense, even Linux distributions were classified as platforms; the interesting observation is that those distributions are licensed for a significant part under pure FLOSS licenses to maximize external contributions, and leverage copyright protection to prevent outright copying but not “cloning” (the removal of copyrighted material like logos and trademark to create a new product). The main value proposition comes in the form of guaranteed quality, stability and reliability, and the certainty of support for business critical applications.
• Selection/consulting companies: companies in this class are not strictly developers, but provide consulting and selection/evaluation services on a wide range of project, in a way that is close to the analyst role. These companies tend to have very limited impact on the FLOSS communities, as the evaluation results and the evaluation process are usually a proprietary asset.
• Aggregate support providers: companies that provide a one-stop support on several separate OSS products, usually by directly employing developers or forwarding support requests to second-stage product specialists.
• Legal certification and consulting: these companies do not provide any specific code activity, but provide support in checking license compliance, sometimes also providing coverage and insurance for legal attacks; some companies employ tools for verify that code is not improperly reused across company boundaries or in an improper way.
• Training and documentation: companies that offer courses, online and physical training, additional documentation or manuals. This is usually offered as part of a support contract, but recently several large scale training center networks started offering OSS-specific courses.
• R&D cost sharing: A company or organization may need a new or improved version of a software package, and fund some consultant or software manufacturer to do the work. Later on, the resulting software is redistributed as open source to take advantage of the large pool of skilled developers who can debug and improve it. A good example is the Maemo platform, used by Nokia in its Mobile Internet Devices (like the N810); within Maemo, only 7.5% of the code is proprietary, with a reduction in costs estimated in 228M$ (and a reduction in time-to-market of one year). Another example is the Eclipse ecosystem, an integrated development environment (IDE) originally open sourced by IBM and later managed by the Eclipse Foundation. Many companies adopted Eclipse as a basis for their own product, and this way reduced the overall cost of creating a software product that provides in some way developer-oriented functionalities. There is a large number of companies, universities and individual that participate in the Eclipse ecosystem; as an example:As recently measured, IBM contributes for around 46% of the project, with individuals accounting for 25%, and a large number of companies like Oracle, Borland, Actuate and many others with percentages that go from 1 to 7%. This is similar to the results obtained from analysis of the Linux kernel, and show that when there is an healthy and large ecosystem the shared work reduces engineering cost significantly; Gosh estimates that it is possible to obtain savings in terms of software research and development of 36% through the use of FLOSS; this is, in itself, the largest actual “market” for FLOSS, as demonstrated by the fact that the majority of developers are using at least some open source software within their own code.
• Indirect revenues: A company may decide to fund open source software projects if those projects can create a significant revenue source for related products, not directly connected with source code or software. One of the most common cases is the writing of software needed to run hardware, for instance, operating system drivers for specific hardware. In fact, many hardware manufacturers are already distributing gratis software drivers. Some of them are already distributing some of their drivers (specially those for the Linux kernel) as open source software.
  The loss-leader is a traditional commercial model, common also outside of the world of software; in this model, effort is invested in an open source project to create or extend another market under different conditions. For example, hardware vendors invest in the development of software drivers for open source operating systems (like Linux) to extend the market of the hardware itself. Other ancillary models are for example those of the Mozilla foundation, that obtains a non trivial amount of money from a search engine partnership with Google (an estimated 72M$ in 2006), while SourceForge/OSTG receives the majority of revenues from ecommerce sales of the affiliate ThinkGeek site.

At the moment there is no “significant” model, with companies more or less adopting and changing model depending on the specific market or the shifting costs. For example, during the last few years a large number of companies shifted from an “open core” model to a pure “product specialist” one to leverage the external community of contributors. Many researchers are trying to identify whether there is a more “efficient” model among all those surveyed; what we found is that the most probable future outcome will be a continuous shift across model, with a long-term consolidation of development consortia (like Symbian and Eclipse) that provide strong legal infrastructure and development advantages, and product specialists that provide vertical offerings for specific markets.