DevOps is old hat. Deploying your own code to production and managing it is standard practice for all developers. We don’t need sysadmins any more. YAML is easy therefore CloudFormation is easy.
None of these statements are true.
This post is a summary and extension of the 10 minute talk I gave at BelTech 2020.
I've uploaded my slides from the session in case you want to see them.
10 minutes is nowhere near enough to do this subject justice, but I hope I managed to give you the gist.
I started back in the dawn of time (1976), with a reference to DeRemer and Kron's article about programming in the large versus programming in the small (pdf)deremer-kron.
The point here is that for a long time, integrating modular component-based software has been a concern. And back in 1976, the thought was that a separate language would be used to integrate software modules.
These Module Interconnection Languages do exist, and there is research available. I have not read this research (yet). However, I suggest that CloudFormation YAML could be viewed as a type of MIL, allowing software assets (for example AWS Lambda Functions) to be connected together reliably.
It's difficult to find recent (post-2007) research on MILs, suggesting these ideas may be no longer relevant. I'd love it if someone could point me at some recent thinking on this subject.
I then moved on to briefly mention the modular nature of software, referencing this 2000 article on the history of software architecturebarroca-hall.
My purpose here was simply to highlight our short memory and our never-ending quest for a better solution. This is of course an admirable quality, but one that can end with us reinventing the wheel instead of repairing a puncture.
A superb quote that comes immediately before the one in the slides is:
There had been considerable progress in describing components and their inter-connection, and in developing and managing libraries of components. All of this had been focused upon technical processes. It did not work!
So we've been struggling with tying software together for over 40 years. I suspect we will try for the next 40 too.
Let's jump forward 10 years to the rise of DevOps.
The term DevOps was first coined in 2009 by Patrick Debois when we has seeking a name for his conference. Read a brief history here.
The original website isn't available but you can see some of it on wayback machine.
Here we see that one purpose of DevOps was to bring some of the lessons learned from Agile Development to the System Administration community. The direction towards turning all infrastructure into software is inherent here too.
I have struggled with DevOps in the past because I've seen it as a management buzzword that requires developers to do the job of systems administrators, usually without giving them the resources they need to do it.
And that's the risk we take with tools like CloudFormation - we assume that we (as developers) can do it ourselves and as a result we end up making a mess of it (sometimes).
Here's a picture of my simple cloud subscription service that I created to learn the technologies and to earn some money.
This gives you a picture of the AWS Services I'm using; Cloudfront, S3, ApiGateway, DynamoDb, SES, Cognito (there's a couple that aren't in this, Route53, SecretsManager).
It's not a complicated application, and I knew exactly how I wanted it to work, but getting it to that shape was difficult, and the problem was Cloudformation.
A brief glance at my git history shows that 66% of the changes involved changes to the CloudFormation template. Most of those were my attempts to move it from one working configuration into another. I used the CodeDeploy service on AWS to manage building my stack from my git repository.
When two thirds of the changes are concerned with the infrastructure of an application that's a sign (to me) that there's a problem.
It's distinctly possible that this is a direct result of my inexperience. I'm possibly doing it wrong in some way. I am certain that my next project will be smoother. And that will probably be because I use something like the AWS CDK
Which brings me to this tweettimbray (image below)
It definitely feels to me that we are moving along the bottom axis of this cycle right now, from CloudFormation YAML to CDK Typescript. CF YAML has failed to be a successful MIL and we'll go round the cycle once again until we get there.
Might I suggest that something akin to Gherkin would be excellent for defining infrastructure? Given a choice I'd prefer a tight grammar that looks like written English over a load of verbose Javascript or even a UML diagram.
Having said that, here's a few things that would make my task creating cloud applications easier and I welcome your recommendations on how to do it.
The documentation could be better
The API docs for CloudFormation tell you all the properties you need to create a resource but not how to relate them to other resources reliably. You need to therefore keep a lot of different concepts in your mind at the same time so you can create the stack.
The tooling could be better
Anyone ever used AWS Organisation accounts? Separate accounts are probably the only way to reliably keep a separate staging or development environment in CloudFormation. You might be able to run two stacks side-by-side with different parameters (names etc.) but that's risky. That means in order to do a reasonable and safe development process alongside a deployed application you need to have multiple accounts.
Furthermore, you can't test locally. That means that you might find yourself making lots of git commits so your build process can create your stack and run the deployment for you. It is possible to deploy a stack without a commit but you still end up waiting for the stack deployment to complete before you can be sure you've written the template correctly. A way to validate and build quickly locally would reduce the cycle time.
Finally, IDE integration that gives you more than just YAML formatting errors would be excellent. This is one of the main draws of using tools like the CDK - the IDE supports the Typescript Types which gives you more confidence that you've configured it correctly.
There is lots to learn
My brief exploration of Cloud infrastructure via this application highlighted the depth of knowledge required. I didn't even mention the head-melt that is IAM. Developers need time and safe space to learn these new concepts; Instil can provide the scaffolding that can help you build your Cloud knowledge safely.
[Get in touch](/training/enquiry/?q=Cloud Architecture) with us today about Cloud Training courses.
- DeRemer F.L., Kron H.H. (1976) Programming-in-the-Large Versus Programming-in-the-Small. In: Schneider HJ., Nagl M. (eds) Programmiersprachen. Informatik-Fachberichte, vol 1. Springer, Berlin, Heidelberg↩
- Barroca L., Hall J., Hall P. (2000) An Introduction and History of Software Architectures, Components, and Reuse. In: Barroca L., Hall J., Hall P. (eds) Software Architectures. Springer, London↩
- I used the Solarized theme just about everywhere.↩
- Discovered via this tweet, the sentiment of which I am not clear.↩
