What is devops?
Devops is the new cool kid on the block and everyone wants to “do devops” but WTF does that even mean?
I won’t claim to be an expert but I will say that over a decade ago when I was called a sysadmin I used good development practices like TDD and pairing to write tooling that automated the operational tasks I was responsible for, often in a way that meant people I was working with could self serve if they wanted something like monitoring or a new server (I was using linux-vservers back then, old school). I feel like this journey at least allows me an opinion so here it is.
I often see two poor devops practices implemented out of either a lack of respect or a lack of foresight. The first is the “lets all do devops” implemented as everyone gets to build infra. The second is lets put a devops resource on every team.
Lets all do devops (as in build infra)
You know how when you want to write a web page you start with writing a TCP server to parse HTTP and HTML from a TCP stream? Or when you want to write a JSON API you start by writing your own JSON parser? No? So why do so many organisations expect developers to build their own infra? Have we traveled back in time to when people actually thought this was a good idea?
Don’t get me wrong, I’m not suggesting an old school, throw your shit over the wall style ops team, but I am describing the sensible practice of divide and conquer. Devops is collaborating on development and operations, not everyone doing the same thing over and over again.
Some people are good at building infra so figure out how those people can help developers build great apps rather then infra instead of making it a free for all, because from over a decade of personal experience the latter is actually a lot slower in terms of development. Infra isn’t simple and the person with the devops title isn’t cheap, use them for what they’re good at and save the rest of the team for what they’re good at. It’s called collaboration.
Put devops resources in each team
It doesn’t scale. If you’re only ever going to have one team then go for it but if your plan is to scale in a way that requires you to draw from one of the most contented talent pools in IT then you best have a lot of money and be prepared for tons of repeated effort and beautiful snowflakes to maintain. Seriously, why does anyone think this is a good idea?
The crux
What is it about infra that makes people loose their shit? This isn’t rocket science, re-usability, good interfaces, it’s all the stuff you already know is a good idea when you’re developing software. Or just in normal life really, you’re not building a car every time you want to drive to the shops. And you don’t ask your developers to develop all your software, no, it’s why microservices, with all their issues, actually make sense.
If the people developing your infra tooling aren’t both following good development practices themselves and making your developers lives easier then it’s not a problem with having a team dedicated to building your infra, it’s a problem with the team dedicated to building your infra. A half decent infrastructure engineer will want to push the boundaries, both with regards to reliability and usability, so the argument that it’s faster if everyone just “does devops” tells me you’ve simply got the wrong devops.
On HMRC webops with just 20 devops engineers we supported the infra on three cloud providers, one of which was notoriously unstable, constantly shifting security requirements, adhoc requests for help, and an application platform with over 700 developers contributing daily to about 300 microservices that were spread over close to 1000 containers, all the while making constant improvements to both the speed and frequency with which developers could deploy and the stability of the platform. To think we could have achieved that with either a devops in each team or asking developers to build their own infra is very shortsighted. IMO the only way to achieve this was by positioning people as close to their core skills as possible and having good inter-team collaboration.
I think maybe the issue comes down to the role of devops being a different thing to the philosophy of devops. There is no other role/philosophy combo that’s as ubiquitous or core to building software as devops (and it was like this before the phrase devops appeared). Sure you get a CI/CD philosophy and a build manager to over see it, but maybe the fact that the names are different helps to keep them separate and stop every dev from writing their own CI/CD service. Maybe it’s having the same name for the philosophy and the role that makes people think they have to do the role to live the philosophy. Maybe I should be called what I am, an infrastructure and platform engineer, that works along side other engineers like for example backed and frontend engineers. Ideally we’re all “doing devops” but we’re not all building infra.
