Wednesday, September 25, 2013

Outages ARE Relevant!

By Bill Moran, Rich Ptak

Once again Amazon is in the public media, this time for outages in AWS, their public cloud.[1] Our earlier blog[2] commented on the GAO issued report[3] about the CIA awarding a contract for cloud services to Amazon and IBM’s subsequent protest of the award. The GAO accepted IBM’s protest but disallowed IBM’s attempt to point out the AWS’ history of significant outages as reported in the NY Times and other media during last year.

In that blog, we did not accept the government’s rejection of this part of IBM's protest. Since the CIA planned to move the vendor’s public cloud into a government datacenter, we thought that the track record of the vendor’s cloud offering to the marketplace was clearly germane.  We recommended that the government require vendors to provide data on their cloud’s marketplace performance[4]. One reason the government gave for rejecting IBM’s protest was that no information was available about Amazon SLAs (service level agreements). We suggested the government require Amazon and any other bidder to supply such information. 

Frankly, we don’t know whether or not the Amazon cloud has the reliability and security necessary to satisfy the intelligence community’s requirements. However, a failure to make a proper assessment of these issues could be very costly for the buyer. Based on the published evidence in the GAO report, it does not appear that any such assessment was made. In fact, little detail about the assessment itself has been released. 

Let’s explore this a bit further. Some years ago, we heard Scott McNealy, Sun’s CEO at the time discuss a conversation with an early purchaser of a new large Sun server. Sun had just entered the server business. The customer said they were planning to host a 911 service on the server. Scott admitted he was stunned as the customer described the significance and variety of potential problems if the system went down. Until then, Sun was a workstation company selling most of their products to engineers. It took Sun some time to adjust to the enterprise marketplace and the realities of enterprise reliability requirements. 

Whatever else one might say about IBM, one has to admit that, as a company, they understand enterprise requirements. They have produced many successful products targeted at the enterprise.
Amazon, on the other hand, has almost no track record in producing enterprise products. Of course, this does not prove that the Amazon cloud will not meet the intelligence community’s needs. However, it does indicate that the burden of proof is clearly on the CIA to require Amazon to demonstrate their cloud can do the job. It should be an Amazon responsibility to provide the necessary data on the operation of their public cloud.

Also significant is that it is no easy matter to re-architect a large and complex hardware/ software product. There are many examples of such costly occurrences that could be named, including some within the US government. At times, vendors struggle for years with these systems to meet customer requirements. In other cases, the projects have been abandoned. Generally, the problems do not really surface until after customer delivery and implementation begins. It seems reasonable to us that the CIA should take extra care to assure their cloud project does not add to this unfortunate list.

All-in-all, the final outcome of this bid is still unclear. It does appear that there are some serious weaknesses in the process that need to be addressed. We do think that one lesson to be learned is that the overall process is severely lacking in transparency. No one, not the tax-paying public, not the vendors not the government are being well-served by the secrecy that appears to be integral to the existing process. We suggest that the GAO and agencies consider a more transparent process.

[4] The project was rebid but there is an Amazon court case pending. Because of the secrecy we do not know what requirements that the CIA put on the bidders concerning the reliability of their public cloud.