Open Government and the Cloud... a Perfect Storm

It’s been just over two months since the release of the Open Government Directive (hereby referred to as “The Directive”), and the state of awareness continues to grow steadily, especially among those of us in the technology community. This isn’t unexpected; we’re the innovators and early adopters on the low end of the adoption curve (see Everett Roger’s “Diffusion of Innovations”), always a bit ahead of the masses when it comes to application of technology.

The interesting thing, however, is that Open Government really isn’t about technology. To quote The Directive, it’s about Transparency, Participation, and Collaboration. These are all values that are driven by attitude, process, communications and people – and could easily be accomplished with handwritten notes on napkins, or casual discussions over coffee somewhere – as long as people are interested and incented to do so. To be clear, Open Government is about changing the culture of Government – which starts with the human element.

That said, technology has always been the lever for human achievement, and it’s no different in the case of Open Government. The proper use of the right technical solutions will allow Open Gov solutions to scale orders of magnitude greater at fractional costs vs. nontechnical means (if there are actually any viable ones). For example, if we wanted to cultivate broad visibility and dialogue on Recovery spending, the Recovery Accountability and Transparency Board could print out all data on programs and expenditures, send it to every citizen in mail (along with regular updates), allow them to send letters back with their thoughts and comments, hire thousands of people to open and read, transcribe, and aggregate comments and feedback, etc. Or they could leverage a bit of technology to put up a site like Recovery.gov – and reach millions more people at a much lower cost. It’s probably hard to find anyone who’d credibly argue with that.

There are many technologies that we’ll all be able to employ in the pursuit of Open Government: data aggregation and publishing tools, communication protocols, business intelligence tools, visualization technologies, user experience and interface tools, and much more.

One segment of technology, in particular, has specific applicability to Open Government and can truly accelerate success: Cloud Computing.

Cloud Computing definitely has had its measure of hype – some have inferred that it’s the solution to all IT needs and ultimately marks the end of PCs, on-premise computing, etc. – just as there were many people claiming that the Internet was the end of brick-and-mortar retailers. Others have implied that Cloud Computing will never be broadly adopted, specifically in the Government space.

The truth, most likely, resides somewhere in the middle. There are many scenarios where Cloud Computing will provide clear and practical benefits – and other situations where, by itself, Cloud Computing will make little sense.

However, in my opinion, for many Open Government solutions – specifically in the area of data publishing, Cloud Computing is a slam dunk. Here’s why:

Before we move on, I’m going to assume that everyone is in general agreement on the definition of Cloud computing… and for simplicity, I’m going to use the NIST definition that can be found here: https://www.csrc.nist.gov/groups/SNS/cloud-computing/cloud-def-v15.doc

Based on that, the key characteristics for Cloud Computing are: On-demand self-service, Broad network access, Resource pooling, Rapid elasticity, Measured Service (i.e., pay-as-you-go usage).

Those qualities provide benefits for many types of solutions, but there are some specific scenarios where the benefits are extremely apparent:

• Collapsible solutions – also known as on/off solutions, or even as batch scenarios. These applications will run for a short period of time and, once completed, will then revert to a dormant state. Cloud works well here as capital is not wasted on unutilized hardware during the inactive periods – resulting in significant savings. Also, the time and cost of re-instantiating the application from a dormant state is usually significantly less on a Cloud platform (vs. having to rebuild out a datacenter).
   
• Predictable bursting – similar to collapsible solutions, but the solutions will run full time with cyclical (or otherwise predictable) periods of increased load. Again, the elasticity of Cloud platform allow for the more efficient use of hardware over time, resulting in reduced costs. In addition, the access to broad resources can help guarantee uptime and a positive user experience and the loads increase. The only way to achieve this in a traditional scenario is to build an infrastructure that can support the heaviest predicted load, with the knowledge that it will be underutilized most of the time (and wasting space, electricity, time and money vs. a Cloud platform).
   
• Agile growth – most successes start small and grow from there, the challenge is creating an infrastructure platform that can grow in line with the usage of the solution. It takes time and money to add servers to a datacenter, and often the added hardware is in excess of the current need (because you don’t want to have to keep adding servers at every percentage of increased load). Thus, you end up with blocky additions of capacity (and cost) instead of smooth, linear infrastructure growth. With a Cloud platform, adding capacity can be done incrementally and very rapidly – saving the time and IT overhead, and enabling the “right-sizing” of the infrastructure to the immediate needs (thereby saving costs). There’s also the added benefit of requiring almost no up-front capital expenditure on a Cloud platform, which minimizes the risk (and cost) of deploying new solutions, allowing organizations to be much more aggressive and innovative (and also avoid getting stuck with a big initial bill before the solution even comes online).
   
• Unknown or unpredictable load – often times, when a solution is launched to the public (and even in many internal organizations), the business managers and developers can only make a reasonable guess of the anticipated traffic (I think I’m being a bit generous here). If you’re building out your own datacenter, that means that the chance of you getting the capacity right is fairly slim – and you risk either overspending on hardware and infrastructure, or having a platform that gets crushed under the initial load (thus providing a bad experience for the users). Either way, you’re in a heap of trouble. To me, this is one of the primary benefits of Cloud platforms – particularly for solutions that are new to market: the risk of missing your traffic estimate is significantly mitigated by the capability to rapidly shrink or grow as needed. This means that you can be much more confident about meeting the load demands if the solution is wildly successful and has massive demand, and also assured that the costs you incur will be based on what you have actually used (as opposed to what you guessed you might be using). Reduced risk and reduced cost… very good things.

Those scenarios can apply to many solution areas and industries – but I’m specifically addressing Open Government here. How do they apply here?

I’m going to hit the easy one first:

Unknown/Unpredictable load
Under the Open Government Directive, many agencies are publishing a LOT of data, and I think it’s reasonable to say that just about no one has any idea what solutions are going to come of it.

At the risk of yet another movie analogy, “Open Government is like a box of chocolate… you never know what you’re gonna get.”

The fact is: we DON’T know what solutions will arise from the availability of all of this data in the future. Thus – it is absolutely impossible to predict what kind of capacity we’ll need to support these solutions. Sizing a datacenter for Open Government applications at this point in time will have significant risk, and will most likely be done wrong… which may delay progress in the Initiative, frustrate many agencies and non-government organizations that depend on the data, as not sit well with the taxpayers.

The elasticity and pay-as-you-go consumption of Cloud platforms can provide a very viable alternative here – significantly reducing costs and risk, and ensuring a positive experience for consumers of the solutions.

Agile growth
Last I checked, money isn’t falling from the sky or growing on trees, and we all understand the challenges faced by reduced tax revenues and decreasing budgets. Funding to build out datacenters is difficult to come by, and is always at risk being repurposed for other means (given the amount of snow we’ve had in the east this year, I wouldn’t doubt, or argue, the attractiveness of new plows or more salt vs. some shiny new servers).

At the same time, agencies are under pressure to comply with the Directive and/or to meet their own goals for openness and transparency. Less money to do more things… a common story for all of us.

Again, Cloud platforms can provide a viable option. No capital expenditures means that the immediate impact on budgets will be slight, and agencies can be deliberate and plan out their growth over time, while meeting the immediate requirements for the Directive (and other initiatives).

Predictable bursting/collapsible solutions One thing that we can be sure of: predictable time periods where there will be increased load on the solutions. Examples: when new legislation is proposed or is being argued, when new high-value data is published (such as job catalogs or contract proposal lists), during cyclical periods such as tax season or near elections. We may not know the baseline of how much capacity we need in the first place… but it’s a pretty fair bet to say that MORE will be needed at these times.

We could just build out datacenters that will support the expected maximum load… but the two problems with that are 1) we don’t even know what minimum load will be, and 2) explaining to the budget owners and taxpayers why scarce funds were spent on hardware that is mostly underutilized will not be a pleasing task.

Once again, Cloud platforms are specifically designed to help manage this kind of problem – providing the potential for more efficient usage of hardware, thereby reducing costs and helping to ensure that the solution will be able to meet any required bursts in usage.

 

“Wow,” you might say. “That’s a pretty compelling argument for using Cloud Computing platforms for Open Government… so what’s the hold up? Why hasn’t everyone just jumped in already?”

With any new concept, there will always be friction – and there are some very valid concerns and challenges that government organizations face:

• Cloud computing platforms are relatively new and unproven – many organizations have spent decades perfecting their systems, platforms, and processes to where they are extremely reliable and well understood. Most public Cloud computing platforms are new, and in the eyes of some, unproven – which makes them a potential risk. In my opinion, we are getting near the tail end of this argument, as the Cloud vendors get broader adoption with consistent success (for the most part), and build up their proof and collateral for reliability, security, and scalability.
   
  Frankly, I expect that most initial Open Government solutions will be non-mission critical where the risk of using a Cloud platform is minimal… there should be plenty of data and solutions that will be suitable for Cloud scenarios. Yes, there will be scenarios that will not fit, but to broadly discard the Cloud option because of its “unproven-ness” feels more like a red herring.
   
• Security and Privacy – there are both perceived and real security/privacy concerns for Cloud computing use in general… but we’re talking about OPEN GOVERNMENT data here … where the specific intent is to get the broadest visibility and most exposure . Again, I suspect that it’ll be easier to find scenarios that are suitable for Cloud platforms than not in this space… keeping in mind that you need to exercise best practices and secure programming processes (like Microsoft’s Secure Development Lifecycle - https://msdn.microsoft.com/en-us/security/cc448177.aspx ) - especially if you’re having users register with any personal data in your solutions.
   
• New processes, tools, technologies, and skills – these are the most realistic and practical problems for all of us. Cloud computing can provide real benefits for many of us – specifically in the above scenarios in Open Government solutions. The problem is: we have to get there.
   
  Most organizations don’t have skillsets around Cloud platform, publishing data to a Cloud, building Cloud-based applications, and managing Cloud solutions. In addition, they may not’ have the insight on cost to build solid cases for ROI, haven’t established processes and guidelines for their agencies and personnel, and likely have no idea where to start. Very real concerns, with very real costs and risks.

 

Yep. That last one is a tough one. Frankly, there’s no silver bullet answer to this (though you could probably pay someone a huge chunk of cash to take the problem off your plate). As with any shift into a new environment or computing paradigm, there will be transition challenges, growing pains, and some cuts and bruises along the way.

That said, there is some very good news that will hopefully keep everyone motivated and incented to consider the journey to a Cloud platform for their Open Government needs:

• It’s evolutionary, not revolutionary (may not be that hard) – You may be able to use many of your current skillsets, existing experience, tools, and solutions in the transition to the cloud. In some cases, it’ll just work – and work better. In other cases, you’ll need to get training, make modifications, and change mindsets – but it will likely be incremental. Moving to the Cloud isn’t starting over; it’s extending your current infrastructure and skills to a broader environment.
   
• There are many organizations, of all sizes, who already have expertise in Cloud computing. This area may be new to some of us… but many people have already broken this trail, and are more than happy to help you find your way. For the fear of forgetting someone who I should mention in this post… I’m going to pass and just advise you to check out Microsoft’s partners here: https://www.microsoft.com/solutionfinder/Marketplace/Home.aspx (if you’re an organization with some expertise in Cloud looking to make a business helping others, specifically Windows Azure, I recommend getting involved in the Microsoft Partner program, if you’re not already, here: https://partner.microsoft.com/ )
   
• There are some great tools that can greatly simplify the use of Cloud platforms – for example, Microsoft’s cloud platform is Windows Azure (https://www.microsoft.com/Azure)… and it is essentially a cloud computing, storage, and messaging environment that is an excellent platform for Open Government solutions. To help Government agencies (and other organizations) take advantage of the benefits of Windows Azure, my team developed the Open Government Data Initiative (https://www.microsoft.com/industry/government/opengovdata/) (OGDI), which is a set of tools and open source code that can be used to develop a solution to host Open Government data. Think of it as an OpenGov “starter kit” or “accelerator” for Windows Azure based solutions that can get you very close to a solution with very little customization. The City of Edmonton is one example of an agency that is already using Windows Azure via OGDI - you can see their implementation here: https://data.edmonton.ca/
    
• There’s a light at the end of the tunnel – the benefits that Cloud computing platforms can provide Open Government solutions are real, and it’s very possible that a strong case for rapid ROI can be made, especially for the scenarios I mention above. Net net: most likely, it will be VERY worth it.

 

The shift to Open Government, culminating with the release of the Open Government Directive, occurring at the same time as the emergence and maturity of Cloud platforms, like Windows Azure, has proven to be a very timely coincidence. In my opinion, the challenges of Open Government could not have been met successfully even a few years ago, as the enabling tools and technologies did not yet exist.

Those of us who are interested in driving the cultural, civic, and political changes that will achieve the tenets of Open Government are fortunate to have the leverage of the right technologies, such as Cloud platforms, to help us reach our goals.

-Dan

ps. If anyone needs help finding a partner to assist with their Open Government and/or Cloud initiatives – feel free to contact me through this blog… and if you’re a Partner who has skills/expertise here, I’d love to hear more about what you’re doing… please feel free to ping me.