What the F is DevOps?

Who knows? Whenever I’ve talked to a recruiter or hiring manager, often the first minutes are dedicated to explaining and scope what the role of the DevOps team is.

Origin of DevOps

Everybody knows this right? In the traditional way of working, there were developers writing code and sysadmins taking care of servers, doing operations. Software engineers would “chuck their code over the wall” to the sysadmins and these sysadmins would deploy the code. Developers want to make changes and sysadmins want stability (“if it works don’t touch it”) thus generating a conflict of interests. “Developer Operations” or “DevOps” was born to break down that wall, or so goes the legend.

The whole idea is confusing because (as DevOps taught us?) a complex system is more stable (easier to fix when things go wrong) when frequent, small changes are applied than when doing big, infrequent changes. So going by this means the sysadmins were wrong in not wanting frequent changes [citation needed]. They were/are still very right in that they were the ones being paged at night when something went wrong and more so when they often had to guess the characterization or configuration of the application (not given or only guessed by developers who ignore how things run in production).

There’s also a “DevOpsiness” component as a modern way of doing things, primarily Infrastructure as Code (IaC) and automation, which is another way of saying “doing infrastructure with the tooling and workflow of a software engineer”. This is in contrast to the traditional sysadmin way of doing things.

And there are also some good practices typically mentioned like frequent and continous deployments, short feedback loops, (see DORA metrics for example), as well as some more vague good intentions that could be as well be part of any (software) engineering project or practice, not just DevOps, like continuous improvement or collaboration.

(By the way, the “breaking silos” is a bit of a pet peeve for me. It’s thrown around a lot but what does it mean? Of course you want collaboration among teams but a. communication grows as the square, not everybody should be aware of what everybody is doign and b. there should always be a clear owner for everything.)

Those practices are why some people say that DevOps is not a job title but a way of doing things. They are not wrong but in practicality a word’s meaning is what most people give to it at a certain point. For example “girl” in Middle English (around the 1300s) didn’t specify gender at all; it referred to a young person of either sex.

DevOps as I see It

There is no single definition of “DevOps”. I propose a practical one: “DevOps is the set of skills that companies ask for in jobs labeled as DevOps Engineer”.

Those skill areas are basically:

• CI/CD pipeline (but not always!).
• Infrastructure (cloud infrastructure for the most part).
• Production Operations (almost always):
  • Maintenance: deployments, upgrades.
  • Incidents, Monitoring & Alerting (but sometimes there’s an SRE team for this).
  • Troubleshooting (often overlaps with dev/SRE teams).

So the most common area of work for DevOps people is infrastructure, but it can be almost anything technical that is not writing application code ¯_(ツ)_/¯

From Developers to End Users (and Everything in Between)

Going left to right in an imaginary conveyor belt where at the left we have the software developers writing code and at the right we have the end users of a web application, we can find different teams with different scopes of responsibilities (ad overlaps are common too).

Beginning at the left and working directly for software engineers writing the application code, we can have Platform Engineers who write tooling for those developers (like a deployment platform). I’ve seen this specialized team only in big software companies and many if not most companies don’t have a separate Platform team; the tooling (if any) will be written by the devs themselves or by our favourite wildcard DevOps team.

Continuous Integration / Continuous Deployment (CI/CD) can be considered Platform Engineering (especially the CD part). Almost all companies have CI but I think only a minority of companies have true CD (can you automatically get a full prod-like environment from your Git development branch?).

Some big companies have dedicated teams for CI/CD (separate from a DevOps team).

Then we have the servers and network infrastructure, which is the most common (overlapped) area of work for DevOps Engineers. Some specialized servers like database servers tend to have their own specialists like DBAs but in smaller organizations like startups it is not infrequent to have the DevOps engineers take care of them as well.

Finally, we have the application(s) code running on the infrastructure and the monitoring, alerting, troubleshooting and incident management for these application(s). This also can be part of the DevOps duties or there can be a separate Site Reliability Engineering (SRE) or TechOps team for it.

Obviously there are no sharp boundaries and the scopes and teams in this left to right description is more of a gradient.

Security is an area that has its own separate team from DevOps but both these teams tend to collaborate quite a bit and there can also be some tension on how each team wants the infrastructure to be.

And to make things less standard or less agreed-upon, there are different ways of working with DevOps/SRE teams, as in for example some companies have one DevOps/SRE engineer embedded as part of the developer team. There’s a whole book that deals with all these options and more.

This is a Mess

If you haven’t worked in different companies as a “DevOps” or don’t talk to many of them frequently, you probably don’t know how messy it can get.

Just as a random sampler of personal experience in companies I’ve worked for or companies I’ve done a full interview loop.

• A software company with a DevOps team, in charge of dev tooling, CI/CD and infrastructure but only staging/testing one; there’s a separate Production team in charge of prod that acts independently. There’s a clear “chuck over the wall” with a release gating meeting included. Only prod team members can access prod servers (including logs).
• A software company where the DevOps team was in charge of the cloud infra but sharing/overlapping with other teams and no clear ownership. Also owning monitoring/incident pager but management felt they needed specialized SRE people in the team.
• A software company where the DevOps team doesn’t own the CI pipeline but owns the database administration.
• Software companies with separate platform or CI/CD teams
• A software startup with Platform, DevOps and SRE teams.
• Managers at the Director/CTO level having one DevOps role job opening with the idea for that person to be an “ambassador”, “thought leader” or “agent of change”, influencing developers and/or sysadmins so that they develop a DevOps way of doing thigs. Yes, somehow they think they can hire one person that is going to both take charge of DevOps work and is also single handedly going to change the minds and hearts of the people that have been working there for a long time.

Let’s Get Rid of “DevOps Engineer”?

If there are so many definitions or areas of work for DevOps Engineers, and there are also teams that overlap with many of those areas, why don’t we get rid of the term “DevOps Engineer” and use instead the specific areas of work (like “Platform Engineer”, “Cloud Infrastructure Engineer”, “SRE”)? The other “DevOpsiness” or good practices (collaboration, continuous improvement, short feedback loops) are that: good practices or objectives that everybody should work regardless of title, and not only for “DevOps work”.

Sure, count me in.

The problem is when a company (a startup for example but not always) wants somebody to write dev tooling, manage CI/CD, create and manage the cloud infrastructure, manage servers including database servers and monitor and own the application incidents. This is not hyperbole, this is the description of a lot of DevOps jobs. (There are many non-software companies with a small engineering force where this is something that may make sense, in the same way that the a software developing team may do everything).

I’d say, if someone advertises a DevOps position, read the fine print and assume it’s “all of the above”.