IBM Spectrum with Cluster Virtualization accelerates Cloud & Cognitive Computing

By Rich Ptak

(Image courtesy of IBM, Inc.) 

IBM made a big impact with announcements of pervasive encryption earlier this year. Now, with its most recent announcement, we predict they will do so again as they tackle one of the biggest frustrations and ongoing challenges to IT.

IT innovators and creators have struggled for decades trying to identify how to get the maximum utilization and optimal performance from an evolving, disparate, complex, heterogeneous IT infrastructure. Complicating the challenge is a user base demanding simple access to the latest technological innovations. The upending of market and usage models, only adds to the problem.

IBM’s latest announcement tackles the challenge head-on with an architecture and solution suite that applies cognitive computing and sophisticated software that delivers consistent, simplified access to automatically managed (and optimized) infrastructure. In their announcement, IBM provides the details of exactly how and what they deliver. We aren’t here to rehash that. We provide a selected product overview with comments on why we think this is a major advance for cloud and cognitive computing users and developers.

What’s the issue?

A perennial goal of IT operations has been to provide the best user experience possible in a cost-effective manner. Typically, operational metrics focused on the simplicity of the user interface, reliability, response times, etc. For infrastructure, the goal was optimal utilization, high performance, and reliable operations. Pursuing these goals has driven IT innovation and product development for decades. IT staff tried a variety of ways to resolve the issues, e.g. languages (Fortran, Cobol, Java), operating systems (z/OS, Unix, Linux, etc.), platforms, dedicated systems, GUIs, APIs, containers, server clusters, open systems, and clouds with limited success.

Today, despite the effort, the goal remains the same. The compute environment is more complex than ever. Users are still frustrated. IT administrators, developers, and operations staff spend too much time configuring infrastructure and juggling complex, dynamic workloads trying to meet SLAs and satisfy users. IBM Systems turned to Cognitive computing and Software Defined infrastructure to address the issues.

IBM Software Defined Infrastructure & Software

In a nutshell, IBM’s Software Defined Computing solution suite addresses the user interface, workload management, infrastructure management and solution development challenges. At the heart of the solution is Cluster Virtualization software which virtualizes access to and exploitation of servers, storage, clusters, clouds, etc., or whatever constitutes the defined available infrastructure. IBM’s innovation is to buffer the user, whether apps developer or user, from having to learn about the intricacies of the supporting computing infrastructure. It offers the prospect of automatic infrastructure configuration and management optimized to provide maximum infrastructure utilization and workload performance.

A virtualization software interface buffers the user and IT operations from the complexity of the underlying infrastructure more effectively than past attempts. Users provide app requirements and parameters. Operations staff identify performance metrics, constraints, and requirements. The solution suites for workload types (discussed below) manage and optimize infrastructure operations using cognitive computing solutions. These dynamically learn app and workload behaviors, infrastructure availability and performance, etc. measured against up to 20 different parameters to manage workloads and configurations. 

Figure 1 IBM Cluster Virtualization    (Courtesy of IBM, Inc.) Cluster Virtualization Software allows users to transparently share clusters of computing resources. Specialized software suites operating through IBM Spectrum Computing and IBM Spectrum Storage underpin the specialized suites.  Figure 1 represents how all the pieces fit together to provide end-to-end management of user activities across multiple platforms, architectures and data center environments.

Cluster Virtualization

Cluster Virtualization allows many independent applications and workloads to make use of disparate resources residing in multiple, different clusters. The workloads can be a mixture of traditional apps, such as high-performance computing or compute-intensive analytics, or next-generation workloads leveraging Hadoop, Spark, and containers, etc. A consistent interface provides users simplified access to and utilization of the total cluster infrastructure. The arrangement is highly scalable; apps and users can both run into the thousands. IBM reports the potential of running millions of jobs per day. IT operations staff benefit with cognitive computing services that automatically manage up to 20 different operational parameters to configure, provide workload management (e.g. scheduling, assignment), infrastructure scaling (up and down), etc., to optimize resource utilization and performance.

The beauty of this design lies in the extremely flexible definition of clusters. It supports a broad range of complex, mixed and heterogeneous environments encompassing from several to 1000s of systems, or VMs. Supported system types include OpenPOWER, x86, ARM, SPARC, as well as multiple operating systems and environments, including LinuxONE, Docker, containers, etc. The defined cluster can be on-premise systems or extend into public, private or hybrid clouds. Cluster virtualization can function across heterogeneous cloud environments that include IBM Cloud, IBM Cloud Private, AWS, etc.

Next Gen IBM Spectrum LSF Suites

IBM also announced significant enhancements to their IBM Spectrum LSF suites. These offer workload management options specifically targeted at the Enterprise, HPC and Workgroup segments with increasing functionality at each level. Figure 2 shows how the functionality and capability varies at each level.  Figure 2 New IBM Spectrum LSF suites  (Courtesy of IBM, Inc.)

Each level is designed to simplify user access and management with automated reconfigurations of access to resources, rapid, flexible scalability and resource utilization. All adjustments are controlled through defined policies, automatically managed and administered.

IBM has introduced new pricing terms and models which appear very attractive. Your IBM rep can provide details.

IBM Spectrum Conductor

There is much more to IBM’s announcement, including updates and additions to IBM Spectrum Conductor, such as the Deep Learning Impact module. An extensive list of enhancements was announced to speed processing, including hyper-parameter search and optimization techniques, elastic resource allocation, and Spark-specific data management. Cluster virtualization and multitenancy for deep learning are only two techniques that are included to increase resource utilization.

This module is designed to more dynamically, efficiently and rapidly extract useable business insights and value from data even as it also simplifies installation, configuration, implementation, model building and analysis with pre-built frameworks. It provides shared multi-tenant and multi-service functionality that will speed up processing and increase infrastructure utilization. IBM Services and Support is available for the entire software stack available on both IBM Power System for HPC with IBM PowerAI framework and x86 systems with Open Source frameworks. The software distribution packages contain all needed components, including all Open Source components. End-to-end workflow management operates automatically to improve operations over multiple cycles. Feedback is that the results are very effective in reaching the efficiency, acceleration and savings goals. 

Enhancements were also made to IBM Spectrum Scale to improve storage performance and operational efficiency. These include accelerated I/O performance, reduced latency between nodes, and better performance of metadata operations. From what we can tell, these all benefit from the cluster virtualization and contribute significantly to the overall performance improvements.

Conclusion

This announcement appears to provide significant evidence to justify a more detailed follow-up for any IT operation responsible for the economic support of a complex data center in a compute intensive environment. To us, the effective implementation of Cluster Virtualization with its potential to simply and economically leverage, exploit and scale heterogeneous compute clusters alone is a compelling reason for further exploration. We intend to follow developments in this area. We look forward to hearing more from users. In the meantime, we highly recommend calling your IBM rep for additional information.