Tuesday, April 26, 2016

BMC NGT + Financial Services firm = enhanced application availability!

By Rich Ptak

Enterprise transformation to become a Digital Enterprise has received a lot of attention since BMC started the discussion with its DSM initiative. Enterprises are transforming operations to digitize functions and products in response to customer demands for speedier, innovative and personalized services. To meet customer expectations, apps defining services must be rapidly developed, continuously updated and digitally deployed.

The services themselves must be intuitive, user focused, constantly available on any user device especially cloud and mobile devices. A financial services consumer may want to check an account balance, analyze market trading, examine a company’s profile and transfer funds before completing a stock trade using one or a combination of devices. A single transaction spins up multiple transactions accessing a number of diverse data sets. The service must be available 24/7, operating flawlessly at rapid speeds. Anything less risks a dissatisfied customer and loss of business. The process is repeated across thousands, even millions of customers.

More sophisticated analytics along with different data mean that traffic volumes, data manipulation and analysis are expanding with no end in sight. Data center operations are hard-pressed to keep up with this activity with existing infrastructure management tools.

With some 70% of enterprise data residing on mainframes, most of the activities will involve mainframes working across multiple subsystems with shared data residing on both the mainframe databases along with other systems. Optimal database performance requires frequent and often time and storage intensive data base reorgs and clean-ups. With increasing size, these become progressively more difficult, more disruptive, and more resource intensive.

Already stressed, data centers are being pushed to operate with minimal or even no downtime. Unfortunately, traditional data management tools were not designed for the current digital reality, including such frequent and persistent updates and reorgs of massive amounts of data. These tools require data bases to be off-line and require intensive resources that grow uncontrollably as the data volumes grow.  These issues often require IT to delay or perform partial maintenance, which only slows performance deterioration, but doesn’t often improve performance. Additionally, forced downtime incurs added costs such as lost/delayed business, loss of customers turning to other suppliers, disrupted operations, etc.

Expensive alternatives (capacity over-provisioning, manual scripts to make temporary repairs and adjustments, equipment upgrades) can maintain operations until a full reorg is run to help avoid costly penalties due to service agreement breaches.  However, these “work-arounds” have their own draw-backs and costs.

BMC’s Next Generation Technology Utilities for DB2 (NGT) are designed for the challenges of digital business. NGT intelligently automates and monitors the utility process and runs with the data base on-line and fully operational. We interviewed a senior engineer responsible for data center architecture and operations at a financial services firm on his experiences with NGT.

The Corporate Data Center

Our senior engineer is responsible for operations in a mainframe shop consisting of multiple sysplexes totaling approximately 80K MIPs processing capacity. Their database runs 60 to 70 Terabytes after compression. Operating 24 hours/7 days per week, smooth, continuous operations is critical to the success of the business.

The company faced challenges impacting many DB2 environments.  Their data bases were growing rapidly with critical ones early each week reaching sizes that seriously impaired performance. They were seeing 1000 timeouts each week during data management processing. The workload would not allow downtime for a reorg during the week so optimization was delayed to the weekends. Temporary adjustments and workarounds made manually during the week were less and less effective as the workload grew. Using available tools, the only alternative was to let performance deteriorate until weekends and then, squeeze-in reorg and maintenance. Even then, traditional management techniques took too much time, decreasing availability (risking expensive service agreement violations) and ran up huge costs.

IT staff believed automation would be too costly and difficult (if not impossible) to successfully implement as many of the 10’s of thousands of jobs run followed no DB2 object naming conventions. This meant complex, expensive, time-consuming manual customization of reorgs specifying sort lists, setting parameters, handling overrides, and tuning. They were happy to find out that with NGT, this was not true.

The Financial Service Firm’s Experience

The IT staff used existing management solutions from BMC and IBM for ‘on-line’ data management.  They needed a better solution. As mentioned, they were initially skeptical an automated management approach could work because of the complexity of their naming and workflow processes. Our engineer decided to try BMC’s NGT solution.

NGT allows reorg processes to be run in real-time without disrupting or affecting service delivery operations. Thus, it assures near peak data base performance at all times.

NGT was able to tie into the existing "grouping" system and determine which ones need to be reorged on a daily basis. Placing jobs in groups simplifies reorg management, e.g. objects are movable from group to group via a simple “drag ‘n drop”. Thousands of jobs were replaced by wildcarded or "group" reorgs that automatically and dynamically handle the whole system.

Provided basic parameters and policies, NGT schedules and automatically runs reorgs as necessary. The utilities ‘learn’ what works and what doesn’t. It ‘remembers’ what has been executed and monitors process effectiveness. It uses this data to automatically make adjustments to executing processes. This assures that reorgs are done in the most efficient, rapid and optimal manner. Required reorg jobs went from tens of thousands to between five and six thousand jobs. Now, there are many fewer exceptions requiring overrides. Where a standard job ran with thousands of overrides, they now have fewer than 100. Instead of delaying until the weekends, reorgs are now run as often as they need or wish.

NGT Utilities provides full data base availability, improved application performance and fully automated data management housekeeping. Dataset allocation automatically adjusts to fit business policies, saving time and storage space. One unforeseen benefit, built-in integrity checks revealed database issues involving thousands of corrupted indexes. The corruption had been masked because of an automatic rebuild process in their existing solutions. 

Prior to NGT, they were able to manage only about 20% of their system because of its size and complexity. NGT enables them to run approximately 6 times as many reorgs at reduced costs while automatically managing the entire system. By being able to reorg through out the week, they maintain good performance and overall save about 10%. 


Our engineer was impressed with NGT’s ease of use. He cautioned that unlike traditional tools, it doesn’t generate a lot of real-time status reporting. Experienced DB staff may have difficulty adjusting to its lack of in-process reports. Not to worry, automatically it does a lot more behind the scenes. Time and detailed post-reorg reports do much to eliminate that concern.

For example, upon completing a reorg, NGT provides a very detailed activity report and error report summary. The summary provides useful insights into what was done in the reorg which serves as a basis for further action.  Finally, it is possible to insert exit points into the utilities to customize the response to specific situations. For example, NGT can be coded to alter a table entry in response to a fault or error instead of generating a Trouble Ticket.    


The senior staff engineer found the potential for continuous, automated reorganization invaluable. NGT is an incredibly powerful solution that is able to complete in much less time without interrupting operations. He liked how it learns and improves its effectiveness with each reorg. He found it highly useful and reliable in operation.

The volume of post-reorg data can be intimidating. But, as stated before, powerful NGT utilities more than compensate for this. They do their job automatically and well by providing insights into significant hidden or previously unknown problem areas. All without ever taking the database off-line. An excellent solution to the major challenges faced by this financial services firm.