Thursday, May 18, 2017
Integrating Syncsort Ironstream® + Compuware Application Audit™ Connects Big Iron to Big Data!
By Rich Ptak
As the amount and variety of data being collected across the enterprise have skyrocketed, so has thenecessity to provide reliable, speedy, broad-based, low overhead, secure transformation, and delivery of that data. Syncsort has been a leader, offering tools and solutions to move and transform data for nearly 50 years.
Big data collection, transformation, processing and timely intelligent analysis to actionable insight continues to be one of the major problems facing enterprises today. In particular, there has been a major bottleneck due to the inability for real-time access to mainframe data.
With the announcement of the integration of Syncsort Ironstream with Compuware’s recently announced Application Audit, the situation is significantly altered. As a result, enterprises can now obtain a comprehensive, in-context view of enterprise operations, faster, more securely and more reliably than ever. Here’s our view.
Mainframe operations have always generated enormous amounts of data. This includes critical data to inform on everything from the basic operations and interaction of the infrastructure, applications, monitoring, testing, abnormal events, network operations and interactions, user activity to accounting. The list goes on and on.
Today’s business and enterprise must function in a world that is a dynamic, fast-paced, high-volume mélange of tightly integrated operations. It is a world of immediate client and user access from internal and external, remote and mobile sources and devices, some of which may very likely be the result of “black-hat” events aimed at disruption, destruction or theft of assets.
Multiple surveys, including Syncsort’s own “State of the Mainframe for 2017”, place operational security and compliance mandates at the top of enterprise executive concerns and objectives. The high cost of successful data breach attributable to non-compliance with a mandate, inadequate security or failure of a trusted asset drives an increasing focus on embedded security.
The complexity of today’s operating environments requires a comprehensive view of operations and status. That view is built using data collected from a staggering number of different topics and devices, including mainframe and distributed infrastructure.
There exist numerous SIEM (Security Incident and Event Management) and analytics engines designed to draw actionable conclusions and results from the data. But, that data even after collection must go through the ETL process before it is fit for analysis. There existed major challenges in both the speed of the transformation and access to the mainframe data that resulted in untenable delays and complications for getting actionable information from the data in a timely manner.
As we said, Syncsort has been working with customers for nearly 50 years. Their recent focus has been on delivering solutions targeted at providing Big Iron data to Big Data platforms for next-generation analytics. The emerging enhancements and extensions in machine learning applications are rapidly impacting enterprise operations. Fast, easy, reliable and timely access to data collected by the mainframe has become a critical issue. Timing and processing challenges involved in moving data on- and off-platforms is also becoming increasingly problematic.
Existing ETL and transformation efforts were too fragmented and scattered. Depending upon analytic tool vendors to develop their own interfaces not only made handling the data more complex but also delayed solutions coming to market. Syncsort’s Ironstream addresses the problem of speedy transformation and reliable delivery.
Integrating Syncsort’s Ironstream with Compuware Application Audit changes the dynamics of the ETL process. Compuware Application Audit collects the data on mainframe users, Syncsort Ironstream® makes the transformation of that (and other mainframe data) to deliver machine data in real-time to Splunk® Enterprise Security (ES) for Security Information and Event Management (SIEM) analysis.
Although currently Syncsort Ironstream works solely with Splunk, they are working with selected customers to test data transfer to data environments like Hadoop. They provide an open-ended trial using the free Ironstream Starter Edition for moving z/OS Syslog and Abend-AID data into Splunk® Enterprise.
What is unique about this technology trial is that there is no time limit on the organization’s use of the Starter Edition to move data. The Starter Edition is only limited in the range of data sources.
Syncsort has one customer moving 2-3 Terabytes of data per day, but most are currently 1TB or less. They estimate that for the 100 largest accounts, there is an average of 10TB of daily data that could be useful sent to Splunk and undoubtedly the usage will grow along with the awareness of valuable use cases.
Syncsort told us that their goal is to be the ‘go-to solution’ for moving mainframe log data from anywhere to anywhere. Their description of the problem and future trends confirm our own view of the market. Syncsort’s strategy, solution approach and product plans are well thought out, and if properly executed, will assuredly advance Syncsort toward that goal. They appear to have well-satisfied customers. We intend to follow Syncsort, perhaps even speak with some of their customers. We’ll report what we find. For now, if your mainframe security and operations are enterprise critical, we’d recommend investigating what they are offering.