[Gnu-arch-users] the way forward

[Thomas Lord]

Nice post, Cason.  [I would include Sun, Apple, Microsoft, HP, Novell/SuSe and other players in the CC list except that I don't have even a slender semblance of personal connection to any of those folk.]

Cason points out that organizations which buy from Red Hat, Novell, etc. give up freedom in exchange for support.  In this message, I'll elaborate on that a little bit, explain why it's a problem, point out ways in which Canonical increases the problem, say a word or two about "community participation", and finally wrap up with some ideas of where one might go from here.  I'll do this "whitepaper style", although this has come out a bit longer than the average whitepaper:

Bugs in the Distribution Business

and Recommendations About How to Fix Them

Thomas Lord, address@hidden

Abstract

> > > >                 The FOSS (free and open source software) industry leaders have recreated and refined the freedom-robbing nature of the proprietary software industry.   In the process, distribution vendors and proprietary ISVs win while customers and contributing engineers lose.  
> > > >                 
> > > >                 This paper explains the meaning of those claims and argues that they are true.
> > > >                 
> > > >                 This paper concludes with recommendations about how to repair the situation. 
> > > >                 
> > > >                 For better or worse, a paper which criticizes the worst practices of the leaders of our industry also doubles as a recipe-book, condensing and providing a guide to repeating and enhancing the worst offenses.   Readers in positions of relative power will have to decide for themselves how to take the information here: as corrective or as guide for further exploitation -- that's a personal, moral decision.  This aims to be a dangerous paper.  
> > > >             
> > >         
> >

Freedom vs. Popular GNU/Linux Business Models

Distribution vendors have focused on:

• freedom for 3rd party ISVs, especially ISVs selling  proprietary code
• lock-in for customers
• lock-in and for community volunteers

Taking those point by point:

1. Freedom for 3rd Party ISVs, Especially Proprietary Code Vendors

Freedom for ISVs ("independent software vendors" -- companies selling "applications") means that vendors have cooperated on standards which make third party applications easy to port between distributions, ideally by virtue of binary compatibility.   Especially in the fairly early days of these distributions, before it was clear that they would have large customer bases, vendors of major applications (e.g., Oracle) had limited incentive to port their applications to GNU/Linux yet without the availability of those apps (or free software replacements for them), the enterprise market for these distributions was limited --- a chicken and egg problem.  The prospect of a splintered GNU/Linux market made up of many maddeningly incompatible distributions, each requiring an additional support cost for each ISV, was uninviting.  Efforts such as  the Free Standards Group -- Open Standards for Open Source promised enough cross-vendor compatibility that big ISVs could perceive the credible promise of a more or less singular, cross-distribution GNU/Linux platform to support.

It's noteworthy that as the distribution business shook out a bit, the winning distribution vendors then had some incentive to claim back some "distribution-specificity" of big ISV offerings.   This led to an increased emphasis on certification programs and stronger partnering between distribution vendors and ISVs:  sure, the proprietary app XYZZY may run just fine on distributions A, B, and C but it's certified (implying an ongoing commitment to testing, etc) on A and the vendors of XYZZY and A mutually endorse one another.  In the extreme, an ISV might even refuse to sell to or support customers running distributions B and C.

There is no contradiction, from the ISV perspective, between standards and certification.   Standards assure that ISVs are not locked to any one distribution.   Certification costs ISVs little and it yields them technical, marketing, and sales help from those distribution vendors with whom they partner.  Unlike porting between incompatible distributions, certification is not an exclusive relationship nor a technically challenging one -- a classic win-win for distribution vendors and proprietary ISVs.

Finally, it's interesting to note in this context the enormous advantage enjoyed by proprietary platform vendors as compared to GNU/Linux vendors.  Proprietary platforms are, by definition, singular.   If a proprietary platform is used by a sufficiently large number of customers, ISVs have incentive to target that platform.   To facilitate and encourage that targeting, proprietary platform vendors don't have to rely on standards: they can make stronger and more useful promises.   A Sun Microsystems or Microsoft can (and do) provide key ISVs with the unique source code for their platforms and can (and do) make and keep very broad promises about how upward compatibility will be maintained between releases.   In those ways, they can maintain a relationship with ISVs that is closer in form to an intra-organizational engineering effort.   (There are arguably potential technical quality advantages to the source-permeable dividing-wall of the standards-based approach used by GNU/Linux vendors but when those are not realized in practice they have little or no effect on the business decisions in play here.)

2. Lock-in for Customers

The costs incurred by customers when they install a platform are different from those incurred by ISVs when they port to that platform.   Customers must provide system administration for their platforms, including regular patching and upgrading, backups, security, and so forth.   

The most prominent decision-driving considerations for enterprise customers selecting an operating system platform are utility and short-term cost:  Will this platform run the applications my firm depends on? and Will maintaining this platform for the next few years cost me less than maintaining that other platform?  It is worth bearing in mind that enterprise customers are historically accustomed to accepting lock-in to proprietary operating system platforms for years at a time and therefore have never (and are not) clamoring for freedom from a particular platform vendor: their driving considerations are costs and utility over small numbers of years.  When looking beyond a few years, their primary concern is not much more specific than not to be stuck with a legacy system which will be too expensive (even infinitely expensive) to migrate beyond -- though the technical prowess of the software engineering industry as a whole has given them (unrealistic) expectations that that is not a serious problem no matter what platform they choose today.

In this climate, achieving customer lock-in is fairly easy:

• make sure frequent updates are necessary and come from a single supplier
• make sure that system administration skills are distribution-specific
• control customer access to proprietary infrastructure needed for system administration
• encourage customers to rely on distribution-specific applications (raise the cost of migration-away)

The leading commercial GNU/Linux vendors embraced this circumstance and used it as a point of differentiation in as many ways as they could get away with.  Red Hat make an interesting case study:

The Red Hat Network (RHN) is a non-freely-redistributable network service providing frequent, critical upgrades to the Red Hat GNU/Linux operating system offerings.   To maintain a secure installation, enterprise users of Red Hat platforms must be subscribers to RHN network services and their attendant human support services.  In spite of the alleged openness and freedoms afforded by free software and open source licensing, RHN is very much a platform-specific, proprietary service:  the work to provide timely updates is closely guarded;  updates are provided primarily in binary form; the infrastructure which runs RHN is kept private;  RHN is not applicable to platforms distributed by other GNU/Linux distributors and the corresponding update systems of those other vendors do not apply to Red Hat's platforms.   In some enterprise configurations, a software component is installed at the customer site but customers are contractually obligated not to copy, modify, redistribute, or use for unauthorized purposes that software.  In effect, a customer who chooses a Red Hat platform for their enterprise for the next 5 years is committed to being a customer of RHN for that same duration -- customer lock-in.

Seen in this light, Red Hat is not truly a free software company: it is a proprietary software company who's proprietary product (the Red Hat Network)  happens to constitute a transmitter/receiver for free software and open source code.

The effort to lock-in customers permeates even the market for IT professionals.  For example, vendors such as Red Hat create credential-offering programs for IT professionals.   They have succeeded in substituting "Red Hat expert" for "unix expert" and "software professional" as the kind of qualification enterprise employers look for when hiring IT staff.

One net effect here is that while, to ISVs, the GNU/Linux commercial distributions are a standards-based abstract platform, to enterprise customers, each GNU/Linux commercial distribution is a de facto proprietary platform, differentiated from traditional proprietary platforms mainly by considerations such as cost.  (I say "mainly" because customer fear of historically offensive practices by Microsoft and historic reliance on expensive hardware by Sun also play a role.   Those considerations are trending downwards, though, as Microsoft becomes a more sophisticated service provider and Sun migrates to commodity hardware.)

3. Lock-in for Community Volunteers

GNU/Linux distribution vendors sit between proprietary ISVs to whom they want to grant freedom and customers whom they wish to deprive of freedom.   The vendors are in the interesting position of not employing most of the engineering talent that produces the software which constitutes a GNU/Linux platform: they depend on volunteers (and, as is much celebrated, contribute voluntary efforts of their own).

Who are the volunteers?   Studies and opinions abound.   Certainly many are employed as software engineers and many of those within the free software and open source industry.   Many are young people (under 30, let's say), often participating for a mixture of fun and inspired by the promise of career development.   Many are employed at proprietary software jobs and do free software work in their spare time.  Many have carved out limited rights from their employer to ship out some of their day-job work-product as public project participation.   Few arrive on the scene as seasoned or even well-educated engineers -- the space of volunteer participation is, as much as anything else, a de facto open university for software engineering.

Although the vendors do not employ the majority of the software engineering talent they rely on, they do collectively seek and achieve influence and control over that talent.   To understand this process, we should look first at what the vendors would want to do with influence and control over developers, and second, how they go about achieving those goals.

Volunteer Lock-in Objective: Market Dubious Standards

We observed earlier that vendors want a level of industry-wide standards conformance in order to grant certain freedoms to ISVs, especially proprietary software ISVs.   At the same time, the standards must not make GNU/Linux platform products interchangeable and should not stabilize those platforms.  This requires that the standards be developed, that work is performed to bring the relevant software components "up to snuff", and more subtly that the standards in question be generally acknowledged and approved of by the  
developer community even though the standards themselves must not challenge the goal of customer lock-in.   In short, the vendors must market poor standards.

Volunteer Lock-in Objective: Bloated Software Stacks and Intricate Dependencies

We observed earlier that, to achieve customer lock-in, vendors rely on a need for frequent updates of installed systems and benefit greatly from customer reliance on applications that are at least GNU/Linux-specific and ideally specific to the GNU/Linux platform distributed by a particular vendor.   Software which more or less works but needs frequent repair, software which thwarts migration away from the platform, and software for which maintenance work can not be broken down into independent efforts is the ideal.  In software engineering we know that the best way to achieve these aims is to produce systems comprised of many more lines of code than is actually needed, to sprinkle that code with platform-specific assumptions, to make each component depend on as many others as possible, and to make certain that the interfaces between components are poorly controlled and subject to frequent change.  The aims can be achieved by leading volunteer engineers into habits which are simply the negation of best software engineering practices.

Volunteer Lock-in Strategy: Exclude Thoughtful Engineers from Executive Management

It is well known that venture capitalists tend to eject thoughtful engineers from the companies they found, refuse to help other thoughtful engineers start new companies, and install management who are under-informed (to put it politely) about the nature of computing systems.   Such is the power and coherence of capital that it casually destroys promising engineer-led efforts, ultimately without even any good financial reason.  (See the web sites and writings of Gabriel, Graham, and Greenspun for starting points to explore this.)

With no sympathetic ears --- indeed, no ears capable of comprehending basic facts of software engineering --- in the executive corridors, there is no chance for engineers further down the hierarchy to plead their case for shifts in the practices employed in industry.   The three G's (Gabriel, Graham, and Greenspun) were at least privileged enough to have fall-back opportunities but they are exceptional that way.   In general, with this approach to corporate control, objectives such as marketing poor standards and promoting the negation of best software engineering practices face no threat of challenge from above.

Volunteer Lock-in Strategy: Dominate the Press

The often young, often under-educated volunteer community is largely informed by news and information outlets such as NewsForge, Slashdot, and O'Reilly press.   Well funded and well socially connected GNU/Linux vendors can influence these outlets in numerous ways with the aim of shaping everything from selection of editors to placement of articles and titles.   Venerable and sometimes more thoughtful news sources (e.g., ACM Queue) -- who might be (and in fact turn out to be) sometimes critical of ongoing GNU/Linux vendor practices -- can't compete: their content is addressed at readers more sophisticated than typical vendors and their business models are slow to adapt to the large free software and open source developer community.

Volunteer Lock-in Strategy: Dominate Professional Conferences

Vendors are in the economic and social position to influence (so-called) professional and trade conferences that serve the free software and open source community.   Influence can range from conventional marketing (spending a lot on booths) to more subtle forms (selection of program committees, occupation of keynote-address positions).   From these broadcast opportunities, industry executives can guide popular opinion about "what's important".

Volunteer Lock-in Strategy: Isolate and Surround Key Maintainers

GNU/Linux vendors can't employ everyone they need but they can employ and influence the thought-leaders of that community.  In most cases, a popular maintainer can be made a regular employee whose public participation is limited and who is personally "sold" on the idea that the best way to advance their project is to steer it in the directions most consonant with the needs of their employer.  In a few cases where the maintainer would otherwise have enough social capital to override their employer's intents it is helpful to preserve an appearance of independence by erecting a consortium-type non-profit around one or a few maintainers.

An innovative extension of this strategy is to invest heavily in growing new popular maintainers "in house" by spending social and press capital on promoting them.   The employer receives the benefit of much volunteer labor as the new maintainer can muster.

An innovation on top of that innovation, currently being pursued by CollabNet, is to sell training to companies which promises to teach them how they, too, can can create new thought-leader maintainers.

Volunteer Lock-in Strategy: Encourage Conflation of Vendor Prosperity with Community Success

It is helpful to promote a distorted (if not outright false) notion of a common enemy such as Microsoft.   As a vendor of proprietary software, Microsoft is no less or more offensive to the notion of software freedom than the GNU/Linux vendors themselves but they do stand out by virtue of history, size, and licensing practices.

Volunteers can be inspired to be uncritical of many GNU/Linux vendor practices by selling the message that the primary goal is to defeat Microsoft by any means necessary.

Volunteer Lock-in Strategy: Trash and Take Over Threatening Projects

At certain times, projects that arise outside of the business activities of GNU/Linux vendors may achieve a momentum and trajectory of their own which undermines the interests of those vendors (GCC under Kenner vs. Cygnus;  GNU Arch vs. Canonical).   With only moderate spending on falsely-friendly "forks", such projects can be reliably taken over and their maintainers discredited and pushed to the sidelines.

Volunteer Lock-in Strategy: Celebrate Bloated, Intertwingled Solutions

The goal of customer lock-in demands a delicate balance: one needs a GNU/Linux platform which works today but reparably breaks tomorrow, and the next day, and the day after that.  The goal is to addict customers to a service which provides an infinite stream of duct-tape patches.

Therefore, vendors must reject all efforts from their engineers to institute "major internal cleanups" to code and, to be proactive, must celebrate software systems which are sufficiently bloated and have sufficient poorly controlled interdependencies to generate a need for perpetual patching.

Simple, robust solutions can be discredited using press influence, control over thought leaders, and the like.

Volunteer Lock-in Strategy: Claim Neutrality -- "Who, me?!?"

The public in general, and the volunteer community is no exception, are generally naive about the role of social connections and biased trade arrangements at the executive and capital allocation level of industry.   Especially popular among many engineer-volunteers are sentiments of naive libertarianism and false individualism.

In that condition, it is easy and desirable to keep the efforts to influence and control the volunteer community hidden.   Messages such as "we only create opportunities to participate -- volunteers can choose or reject those as they see fit" are likely to resonate.  Truths such as the information awareness of volunteers can be controlled and the opportunities offered them can be limited can be plausibly denied.

Volunteer Lock-in Strategy: Control the Hubs of Cooperation

Volunteers are, like the rest of us, fundamentally lazy -- they are inclined to seek the most convenient means to express their good will and disinclined to be critical of the implications of those means.   Simultaneously, vendors which have the most timely access to urgent patches or new features have an advantageous position relative to their competitors.   Simultaneously, in the name of customer lock-in, there is a perpetual need to discourage volunteers from cleaning up their pessimal software engineering practices  -- one needs to sustain customer addiction to barely-tractable platforms which need constant patching and which are characterized by bloated, intertwingled software.

This is convenient for vendors: Vendors can control the portals of participation and thus install themselves as a kind of "Maxwell's Daemon" -- making room for some kinds of patching and feature addition (duct-tape and bloat-expansion work) while discouraging others (best practices;  massive clean-up; simplification work).  Fedora, OpenSuSE, and Ubuntu are the leaders in this space and their examples point the way forward for further pursuing these evil strategies.  If the Collabnet pattern repeats itself, soon we will see classes offering third parties instruction in how to deploy similar manipulative tactics against the volunteer community.

By making it easier to cooperate with a centralized hub soliciting incremental improvements to a barely functioning platform, vendors can distract volunteers from a sensible program of creating a tractable platorm.  (This could be dubbed the illusion of tractability strategy.)

The Technological Tragedy -- How Customers and Volunteer Engineers are Screwed

The result of the business and social practices analyzed above is bloated, hopelessly intertwingled software and a social arrangement of volunteers which creates and perpetuates this.  The software beyond human scale; it is out of control; and it is software on which, nevertheless, millions if not billions depend.   Proportional to the success of the GNU/Linux vendors is the degree of life-criticality of the systems they offer.   The negative properties of these offerings are sustained, encouraged, and expanded by the leadership role the vendors assume for themselves within the volunteer community.   An entire generation of socially and cooperatively inclined young engineers are led to neglect fundamental lessons of software engineering learned in the seventies (because the consequences of those lessons would be inconvenient to the vendors' business models).

Ironically, these shortcomings of the commercial free software and open source offerings are a very broad opportunity for proprietary vendors, especially Microsoft and speculatively, Google.  With a vastly more intelligently organized workforce, those firms (and other minor players such as Apple and Sun) have ample opportunity and incentive to utterly leapfrog current free software and open source offerings with technically superior offerings.

It is interesting in this context to note that these pattern of business practices have resulted in the lack of incentives to create free software projects that would seriously challenge, for example, the "Oracle hegemony".  (The curious phenomenon of the technologically anemic MySQL's ascendence combined with the struggles of the better foundation, Postgres, is interesting to observe.   It should come as no surprise, though, after witnessing the Linux kernel commercially trounce the freed-up BSD for no technically justifiable reason whatsoever.)

It is also disappointing to note the utter failure of vendors to lead their customers in the direction of having genuinely source-based installs.  In the event of any number of foreseeable crises, life and enterprise critical customers will be unable to access binary updates from their GNU/Linux vendor (who may, indeed, be in no position to produce such updates at all).

What's a Better Alternative -- Recommendations

To investors we can say this: computing systems have geopoltical significance and (or yet) your investment choices have enormous influence over their evolution and hence the environment in which you operate.   As most mammalian species know, one shouldn't defecate where one consumes and, at least metaphorically, you are no exception.   Investment and executive-level control which is, without engineering wisdom, applied by inappropriate analogy from other industries to the computing systems industry is socially irresponsible and, if it doesn't hurt you personally, it will surely hurt your children and grandchildren.  Adjust your goals and invest anyway: look to beat conservative bonds, not to find a boom-driven windfall.  Seek out and embrace opportunities for personal professional development by learning more about software engineering than you (most of you) can currently imagine there is to learn.  Stand up straight and live up to the responsibilities of your privilege.  Turn against your peers who do not adopt these principles.

A sane way to approach the problem of commercializing and profiting from the existence and interest in free software is to first understand the challenges as an engineering problem.  Engineering problems are characterized by a conjunction of financial constrains, social responsibilities, customer goals, and technical realities and degrees of freedom.   The best engineers simply refuse and indeed actively fight against deployments of capital which do not weigh these considerations carefully.   Alas, a wealth of sycophantic and irresponsible people with engineering talent are likely to be all too influential with investors.  Still, where does engineering experience lead us, in the specific case of platform development and support:

History gives us a richness of examples of platform development and support.   Reaching far back, we find ITS at MIT.  Progressing, we find Unix from Bell Labs and then BSD from the University of California.   A mere ~20 years ago, we find the Athena project at MIT and the Andrew project at CMU.   All of these have in common a modest ratio (1:a-few-or-several-hundred) of platform engineers to users.   In all cases, we have tight and often personal feedback between platform engineers and users.   In all cases we have platform engineer teams that top out at a few 10s of hackers, able to operate entirely autonomously, but also federated with similar teams elsewhere to leverage cooperation while not having to operate autonomously.  That ratio, and the human-scale platform-engineering it implies, and the federation is a formula for sanity and success.

An abstract proscription?  You bet.   If you wanna get down to specifics, give me incentive.

About the Author

The author, Tom Lord, is the creator of GNU Arch, the hackerlab C library, GNU Guile, and more.

He has enough capital left in the world to buy food for himself and his family for about 11 days.   A small amount more is promised in the form of a gift from a supporter that will extend that by a few days.   He has lacked access to professional health care for quite a few years and recently is suffering rather severely from the effects of osteoarthritis. 

Tom Lord has a reputation as a hot-head and generally bad-attitude guy but this perception is strongly contradicted by meeting him in person or examining his work product and history of dedication to helping free software succeed.

If you are an executive who has interacted personally with him you probably have the impression that he has come to hate and despise you and would do anything at all to cause you harm -- but such perceptions will not hold up if you interact in a serious-minded and helpful way with him.   You will find that he is open minded, intellectually engaging, reasonably smart about basic economics and business, and generally forgiving and forward focused.

Officially, Tom Lord has only a high school degree (though from a prestigious private school).  Unofficially, he is extremely well educated and well rounded and suffers mainly from the clique-ism, classism, and authentically bad attitudes of the FOSS industry executive leadership.


-t