Grid computing: Term may fade, but features will live on

Depending on who describes it, grid computing has grown from its roots in high-performance computing into an enterprise technology that provides for shared resources. Or it’s an overhyped, meaningless term that will soon disappear in the wake of advances in virtualization and utility computing.

Arguments abound on what constitutes grid computing. But at its core, grid is really an enabling technology that provides on-demand access to computing resources — including systems, storage and networking — and data, regardless of location. And because that sounds so similar to how most vendors currently define virtualization, some analysts say the term grid computing may not stick.

The concept behind the technology will likely live on, though, as customers tap into compute power available on underutilized servers, primarily through virtualization. William Fellows, an analyst at The 451 Group and author of a report titled “Grid Computing: State of the Market” (download PDF), maintains that the term will likely be both more significant and less used in 2007. “Grid computing will be more relevant as grids are used to support far more than high-performance computing tasks, but less used as vendors seek to be associated with far more activity, far higher up the stack, than grid computing.”

While the semantic arguments continue, virtualization is increasingly viewed as the best way to deliver on grid computing’s promise, by creating pools of virtual machines that can be used to set policies, provide automation and help meet service-level agreements (SLA) in companies.

Grid in the enterprise

In the meantime, many large businesses have already invested much in grid.

For example, Wachovia Corp. is spreading grid technology companywide, as Tony Bishop, senior vice president and head of architecture and engineering, explains it. The financial services giant uses grid computing to control infrastructure costs and maintain a high quality of service for clients.

Wachovia first deployed DataSynapse Inc.’s GridServer in April 2001 to support the bank’s production systems. The grid software was used as a layer between Wachovia’s financial applications and its underlying computing resources and IT infrastructure. The idea is to have a real-time application operating environment to meet critical service levels.

The software harnesses idle computing resources and generated unparalleled levels of application performance, scalability and reliability, Bishop says. Wachovia has since implemented GridServer within its credit, global risk, equity and mortgage-backed securities areas. The bank’s 10,000-processor grid is now part of an enterprise strategy to tear down application silos and replace them with an infrastructure that stretches seamlessly across the firm’s many investment products and business functions.

The resulting service-oriented enterprise platform, as Wachovia calls its ongoing service-oriented architecture (SOA) effort, is an information-processing utility for end users, who can dial up the capacity they need, when they need it. The environment consists of clients built around Microsoft Corp.’s .Net Framework and a business and data-execution services framework running Java components, including the JBoss application server.

Even given all the investment, grid is not an end unto itself. “We’ll continue to use grid computing as a demand broker that parcels out processing to servers with excess capacity,” Bishop explains, but the larger goal has become transforming Wachovia’s critical IT processing into an SOA utility. The bank’s SOA is built with IBM WebSphere appliances and middleware. Bishop says Wachovia is about halfway through its four-year migration plan. Grid technology essentially supplies the on-demand processing capacity across a range of servers.

Meanwhile, 163-year-old New York-based financial services firm Lehman Brothers Holdings Inc. is using software from Platform Computing Inc. for its enterprise-wide grid. But this is not an entirely new phenomenon; Lehman Brothers has been working with gridlike technologies and writing distributed software since 1992, according to Thanos Mitsolides, senior vice president of fixed income derivatives technology and analytics at Lehman Brothers. The firm’s derivatives, mortgage and corporate credit risk applications are all running on grid technology.

On a daily basis, there are 500 users of the grid made up of mostly IBM blade servers, running an open-source Red Hat distribution of the Linux operating system, Mitsolides says. The next major stage for this grid computing effort will be the sharing of hardware resources by the derivatives and mortgage groups.

An estimated five-year plan was put in place last year, with computational services running on Platform Computing’s Symphony software. In other words, the goal is to move from servers to services. Step 2 will be to share the hardware, using a single platform for all services. “Once everyone is comfortable on the software, it’s a small [technical] leap to share hardware, but it’s a valiant leap in terms of trust,” Mitsolides says. That’s because before the advent of Platform Symphony, users accessed each application as it ran on a specified server or server farm. Now, users will be able to run applications without worrying where the applications reside or whether there’s enough capacity available to support demand. In this new world, a range of servers will be used to supply processing power and the software will be “smart” enough to figure out where to run. Mitsolides says this notion of availability on demand has been a difficult or challenging concept for business managers to grasp, believe and trust.

Step 3 will be to move the grid implementation to other offices around the globe. So far, users notice that services are accessible via the Web, including Platform’s load-balancing, scaling, prioritizing, scheduling and monitoring capabilities. Ultimately, Mitsolides says, users will be able to access both compute and noncompute intensive services through the same application programming interface.

Grid’s expansion

Ultimately, the evolution of grid computing is all about IT’s transformation from vertically integrated silos to horizontally integrated, service-oriented systems. In fact, if Fellows predictions are accurate (see info box, below), grid computing will likely be absorbed into network fabrics.

Research firm IDC sees several different types of grids growing in importance over the next few years. IDC details this market in its report, “Worldwide Grid Computing 2006-2010 Forecast.” The first is a compute grid, which is mostly driven by hardware and uses freeware and staff-provided services. Second is a data grid, which ties together information from disparate sources. The third type, optimization grids, is how many customers define the term and includes pooled resources that are allocated based on demand.

In the future, the sweet spot for grid computing technology is expected to be in enterprise utilities. According to Ian Foster, Director of the Computation Institute at the Argonne National Laboratory and University of Chicago, “grid computing ‘fabrics’ are now poised to become the underpinning for next-generation enterprise IT architectures and be used by a much greater part of many organizations.”

In Foster’s vision, these fabrics combine virtualization, SOA, commodity hardware and open-source software.

Grid in the cloud

One example of these worlds converging is Amazon.com Inc.’s Elastic Compute Cloud (EC2), a service in beta that provides computer power on demand over the Internet. EC2 allows software and Web developers to set up virtual servers, instead of having to buy computers or hire a computer-hosting company. Developers can quickly set up a virtual computer and in minutes or hours add or subtract capacity based on their needs.

Much like Amazon’s S3 storage service, EC2 allows developers to experiment and get running with a new service almost instantly, with no capital costs. EC2 costs 10 cents per hour to run a custom server, plus data and bandwidth charges.

Although Amazon doesn’t consider its implementation a grid environment, some testers are running compute grids on EC2. Werner Vogels, vice president and chief technology officer of Amazon.com, describes the situation this way. EC2 is a “resource management layer that can run under a grid environment,” he said. “Ultimately, we provide the resources for grid computing to exist on top of our service.”

Examples of application grids that use EC2’s resources include Randall Render Rockets using EC2 as a grid to provide image rendering, enabling customers to have digital images and short movies rendered. Others have built environments on EC2 for digital file encoding, to translate MP2 to MP3 files, for example. “We’re using virtualization technology to make EC2 instances appear like full-blown servers, using dynamic allocation to assign virtual images to replicate physical servers,” Vogels explained.

Amazon currently uses EC2 internally to perform efficient resource management. This means that excess capacity on idle servers is used as required by applications and user demand throughout each day. This compares to nongrid environments, in which one application depends on one server or server farm to supply compute power, requiring copious amounts of overhead to cover any potential bursts in demand.

But it’s the business opportunity provided by the on-demand compute resources that appeals to a whole range of developers, from Web 2.0 developers to those who specialize in advanced enterprise computing development, Vogels says.

Still, despite grid’s criticality to underlying applications, Vogels says that in the past few years, the grid concept has been overhyped, and grid computing’s limitations have been exposed. Grid “still requires middleware and resource management capabilities” to make it usable for on-demand services, he explains.

In fact, the biggest challenge to the success of grid computing’s evolution is the current lack of standardization, says Foster. “Different vendors are all trying to compete, creating different methods or features that likely won’t integrate easily without standardization,” he notes. Also, would-be resource sharers must also navigate layers of technology to create a grid architecture.

But Foster, considered a father of the grid computing concept, believes the need for grid computing may create sufficient momentum to overcome the challenges. “The grid vision … of resource federation and resource-sharing across enterprises is something a lot of people are eager to see happen,” Foster says.

Barb DePompa has been writing about technology and business issues for a wide range of publishers, for more than two decades (before the dawn of the grid computing). Accurately capturing this technology during its “trough of disillusionment” phase has been a challenge. Let her know what you think, or comment below.