OpenStack: potential for a cloud standard

Today, Rackspace open sourced its cloud platform and announced to create a collaborative effort that includes NASA, Citrix, and Dell to build an open source cloud platform, dubbed OpenStack. Finally, the world of cloud computing gets some weight behind a potential standard. Google, Microsoft, Amazon, and big SaaS players like Salesforce.com and Netsuite are getting too isolated and too powerful to be believable to drive any type of standard for interoperability in the cloud. An open source stack could really break this open.

If Dell is true to its word to distribute OpenStack with its storage and server products, and Citrix is true to its word to drive OpenStack into their customer base, this effort has some real power players behind it that could provide the counter weight needed to stop the economic lock-in of the pioneers.

This announcement is very powerful as it provides a platform to accelerate innovation particularly from the university research where long-tailed projects simply do not get the time of day from Google or Microsoft. By offering a path to get integrated in an open source cloud platform, applications and run-times for genetics and proteomics, and deep computational engineering problems like material science and molecular dynamics get an opportunity to leverage cloud computing as a collective.

Most of the innovation in these verticals is delivered through open source and university research. By building solutions into a collective, the university research groups can build momentum that they could not build with adopting solutions from commercial vendors like Amazon AWS, Google Apps, or Microsoft Azure.

It also opens up great innovation opportunities for university IT shops that have to manage clusters themselves. Grid computing has proven to be very complicated and heavy-handed for these IT teams, but hopefully an effort like OpenStack with backing from Rackspace, NASA, Dell, and Citrix can give these teams a shot in the arm. The university clusters can be run with utmost efficiency and tailored to the workload set of the university, and OpenStack gear at Rackspace or participating data centers can be used to deal with demand spikes without any modification to the cloud applications.

These types of problems will always exists and only a cloud computing standard will be able to smooth the landscape. Let's hope that OpenStack with its backers can be the first step towards that level playing field.

This is exciting news for cloud computing developers and users.

The Intercloud

Found a wonderful post by Greg Papadopoulos in which he postulates the trend towards interclouds. Greg argues that Amazon's AWS BYOS/IaaS (Bring Your Own Stack) is the perfect marriage of simplicity and functionality that it will be with us for a long time. SaaS is the new delivery norm of software, and PaaS is the needed productivity layer to hide the complexity of IaaS. The proliferation of SaaS on top of PaaS on top of IaaS is the wrath of early technology adoption when most of the functionality is still in its infancy.

As Greg writes:
"Productive and in-production are different concepts, however. And as much as AWS seems to have found the lowest common denominator on the former with IaaS, how at-scale production will actually unfold will be a watershed for the computing industry.

Getting deployed and in production raises an incredible array of concerns that the developer doesn't see. The best analogy here is to operating systems; basic sub-systems like scheduling, virtual memory and network and storage stacks are secondary concerns to most developers, but are primary to the operator/deployer who's job it is to keep the system running well at a predictable level of service.

Now layer on top of this information security, user/service identity, accounting and audit, and then do this for hundreds or thousands of applications simultaneously and you begin to see why it isn't so easy. You also begin to see why people get twitchy about the who, where, and how of their computing plant.

Make no mistake, I have no doubt that cloud (nee network, grid) computing will become the organizing principle for public and private infrastructure. The production question is what the balance will be. Which cloud approach will ultimately win? Will it be big public utility-like things, or more purpose-built private enterprise ones?

The answer: yes. There will be no more of a consolidation to a single cloud than there is to a single network. "

He goes on to say the cloud will organize much like the energy business with a handful of very large networks supported by hundreds of regional and national companies. In this comparison, Greg finds an analogy in the internetworking development. Connecting all these federated entities together has created tremendous value, and thus it is reasonable to expect that the cloud will organize as a federated system as well. But to accomplish that, the cloud community needs to develop the right standards, just like the Internet community did for internetworking so that nobody has to be afraid to become an isolated island.

Putting on my developer's hat, this fear of becoming isolated is what is holding me back to commit to Google Apps or Microsoft Azure: they feel too proprietary in this age of open source and federated clouds. One of my core requirements for cloud applications is that the application could migrate from private cloud to public cloud and vice versa. When the economic value of the application goes up or down I want to be able to allocate it on the right infrastructure to maximize profit or minimize cost. Closed environments like Salesforce.com, Google Apps, or Microsoft Azure are a concern as these environments create a fierce lock-in for my application. Encapsulating it in an OVF virtual appliance provides me much greater flexibility at the cost of not having productive elastic computing services. That capability is maturing as we speak as its value is as obvious as it is needed.

Amazon IT moves into AWS

Amazon.com attempts IT switch to cloud computing

Amazon's e-commerce site is planning to move into Amazon Web Services. Jen Boden is Amazon's e-commerce IT director.

"Boden said her organization is in the preliminary stages of moving into AWS -- she started with some simple, homegrown applications, such as a list maintained for HR, which her team moved to AWS successfully. Larger sections of IT operations will move later with the financials likely to be last, since they are the most sensitive to security and compliance needs. Planning began last year, and the whole process might take another year and a half."

From the article it is interesting to see the confirmation that Amazon's own enterprise IT needs to go through the same transformation as any other enterprise IT team that has decades of old applications and data silos to support.

This effort is a great shot in the arm for cloud computing. The past year, most enterprise class IT shops have started pilot programs to figure out how to incorporate cloud storage and cloud computing into their roadmaps. When leaders like Amazon can point to solutions that others can follow, the laggards will come on board and we can finally move to the next phase in cloud computing and that is standards. With standards, utility computing will come one step closer.

Complexity and fault-tolerance

As an engineer I frequently look towards biology to get inspired or get ideas how complex systems need to be put together to stand the test of time. In this quest, I came across a wonderful article from a Yale research team that compared the transcriptional regulatory network of a bacterium to the call graph of the Linux operating system.


“It is a commonplace metaphor that the genome is the operating system of a living organism. We wanted to see if the analogy actually holds up,” said Mark Gerstein, the Albert L. Williams Professor of Biomedical Informatics; professor of molecular biophysics and biochemistry, and computer science; and senior author of the paper.

Both E coli and the Linux networks are arranged in hierarchies, but with some notable differences in how they achieve operational efficiencies. The molecular networks in the bacteria are arranged in a pyramid, with a limited number of master regulatory genes at the top that control a broad base of specialized functions, which act independently.

In contrast, the Linux operating system is organized more like an inverted pyramid, with many different top-level routines controlling few generic functions at the bottom of the network. Gerstein said that this organization arises because software engineers tend to save money and time by building upon existing routines rather than starting systems from scratch.

“But it also means the operating system is more vulnerable to breakdowns because even simple updates to a generic routine can be very disruptive,” Gerstein said. To compensate, these generic components have to be continually fine-tuned by designers.

Operating systems are like urban streets – engineers tend to focus on areas that get a lot of traffic,” said Gerstein. “We can do this because we are designing these changes intelligently.”

As an engineer this is a very recognizable failure mode of functional design by humans. We are so focused on providing the functionality at the lowest cost possible that the system wide aspects of how the functionality should be reliably provided is lost. A biological system like the human body could lose an eye, a digit, or even a limb and we would still be able to function as a human being. But if you take one leg of a table or chair, the table or chair ceases to function as designed. It is only in high exposure or non-serviceable designs that this broader context is designed into the functionality of the system. Control systems of nuclear plants, or trading algorithms in finance are examples, as are the operating systems of deep-space vehicles.

Cloud computing today needs some innovations to address the broader system exposure of a business process executing in a remote location. For example, security and authenticity of data or a software asset are the two elements that become more nebulous in a cloud context. But solutions to these problems exist and cloud computing will adopt these as best practices. Cloud computing will become a better implementation of what private companies do with their internal IT today. Particularly at the SMB level, cloud computing is already much stronger than most internal IT processes, with well defined disaster recovery processes and geographical redundancy: two elements that are beyond the capital reach of most SMBs. Cloud computing is shaping up to be a new organization of IT capability that will enable the next generation of business process innovation.