Unless you talk to a recruiter and they say something like your more of a SRE then a DevOps engineer and we really need a devops engineer. Insansity
A lot of said complexity is ultimately because we haven't embraced software as an engineering discipline but as rules lawyering. We've lost sight of actual engineering goals like cost and size, and ventured far into the world of marketing, rushing to implement every feature instead of building a sound system first, causing immeasurable economic harm in product defects, engineer burn-out, and missed opportunity.
- DevOps Engineer (Junior, Senior, Staff)
- Systems Development Engineer
- DevOps Team Manager
- Cloud Infrastructure Engineer
- Site Reliability Engineer (+Lead)
All have been meaningless except for Cloud Infrastructure Engineer. The rest described a generalist who knows how to deal with ambiguity and (usually) has the trust with things like AWS root account credentials. Most organisations have no idea what real DevOps is. If they did, they wouldn’t hire for a “DevOps Engineer”
However, a decent sized team can affort a role which focuses on the orthogonal tooling aspects which benefits all engineers who follow DevOps best practices. And that would be a DevOps Engineer.
More charitably though, I think they are not focusing on operations despite it being one of the categories - I don't see incident coordinator / manager, problem manager, etc either.
Maybe the future is all about AI agent harness runtime environment orchestration or some other uninteresting tech.
A tale which of course, flatters the planners…
DevOps is another sad story as it was a saner approach to get people skilled for both dev and ops but turns out people don't like learning new skills as much as they claim. So its meaning differed from org to org and even within the same company.
I'm asking what op believes is accidental complexity. If it was reducible we would've surely done so by now?
Actually, no one learns literally everything. "DevOps" mean that developers should prioritize learning ops over learning deeper something else (because ultimately there are only 24 hours in a day). Which sounds crazy to me; it would be better if the team had experts on different things cooperating together, rather than everyone being mediocre at everything.
That said, companies love replaceable people, or at least they love to imagine that everyone is replaceable. So when everyone becomes DevSecOpsCloudAccountantJanitor, it feels like as long at they keep one such person in the company, they got everything covered.
My feeling is more that developers want to believe that they are the superstars, and they don't want to do the "lower job" that is devops. "I'm too good to learn how to not make a mess in the CMakeLists".
> DevSecOpsCloudAccountantJanitor
I think that "DevOps" has "dev" in it for a reason. It is very close to what the devs produce. I am not saying that developers should be sysadmins or accountants or janitors. Just that they should know how to build their shit.
As someone who worked for years at a company that delivered tools in the devops space, I think you're overlooking the difference between someone who deploys apps to Kubernetes and someone who sets up and maintains kubernetes clusters on bare metal hardware that they procured.
The dev may understand a lot of things about kubernetes, but the devops person is doing a lot of things behind the scenes that the dev should never have to worry about.
- Ops for cloud services and ops for deployed robots are very different beasts, so it's really two different teams that could both reasonably be "devops".
- Ops/tools for robot developers is different since you're often dealing with simulation environments (local, cloud) and also managing access to a more constrained pool of test hardware, all of which needs to be able to quickly recover and restore itself to factory state after developers or their code have done unspeakable things.
- Deployed units are often the worst of both worlds between multiple domains. You've got the big deployment assets of a data center but without the big data pipe; nothing is virtualized so you can't just throw away a node that goes bad; you're stuck supporting hardware snowflakes (here's a text file somewhere with the list of units that have bad CPU fans and need different frequency caps); it's like IOT but with the power and disk and observability requirements of a real server.
> IOT but with the power and disk and observability requirements of a real server
And the issue with the robot is usually none of the regular observability health metrics of a real server... but when it is... "where does that dashboard live again, and is it still working?"
Titles like these always mean something different from company to company.
Even “product manager” has had different meanings at my different employers, some times with major consequences for people who got hired into those roles expecting to do one type of work and then being evaluated on different expectations.
For DevOps, there were a lot of unrealistic expectations from everyone who saw the word and assumed they knew what it meant. I saw a lot of engineering leaders come up with completely different expectations for DevOps roles, but when the chaos arrived most of them reduced the DevOps people into the responsible-for-everything people, which just burned people out.
This depends on the org, but product managers I've worked with spend a lot of time speaking directly with users, particularly in B2B shops. Between speaking with customers and communicating product plans both upwards (to GMs/VPs/exec etc) and outwards (to your own team + external teams) you really shouldn't have a lot of time to build anything as a PM, even with AI.
All of a sudden you had "product managers" inserted deep in the bowels of large systems, instead of focusing on big picture topics.
Doing ops also means entering the on-call rotations, being in the war room when a data center goes under and being on the hook when a DDOS brings down the service.
Very few people want that in their life, money be damned. Same way in an ideal world devs should also participate in customer support, but very few could really do it or handle that for years.
The issue I was facing in a previous life was the turnover. We had data scientists, DBA and many specialized roles stay pretty long, where devops moved away at about twice the rate, and work life balance was mentionned in many exit interviews.
I agree with you that it's extremely formative, and having devs involved basically from start to end of the service surface gives a lot of flexibility. The downside being that it requires a ton of different skills as well, and while people learn, many won't get comfortable with it even after months.
This makes it specially tough when people get called at 3 in the morning to deal with an emergency while they've already spent half the night sleep deprived from their kid puking all night. The worst part being that while people could understand wanting to skip turn one or two times, engineers with small kids will have enough going in their lives that most on-call instances land at a time they're busy dealing with more important matters.
When doesn't this involve devs? If you have ops separate and isolated from the development effort, a devs representation will have to get involved at some point.
So... Yes, sometimes you will.
That's exactly the issue that DevOps aims to solve, though. When you have developers who have no ownership or responsibility once the software is running on production, it becomes a "throw it over the fence" situation, and devs don't care about the operational side.
Logging? Who needs that, if there's a bug report I'll reproduce it on my local machine (and if I can't, CLOSE-WONTFIX!) Performance monitoring? That's an ops problem, tell 'em to buy better hardware. Application crashes? Works on my machine, probably a bug in the server config; ops problem. Tell them to set up a scheduled restart every hour.
You still need dedicated operations staff who deal with the overall system health and design, but yes, a developer needs to be involved when a DDOS brings down the service, because how should the system behave under heavy traffic? Can it serve up cached read-only pages? Did someone find a very CPU-intensive public endpoint that is being exploited and needs to be fixed? Only your developers can answer this question, not the ops team.
That's an org issue. For a given service dozens of roles will be involved, if devs are needed to do every single of them to care and be invested, there is a problem.
> [...] Only your developers can answer this question, not the ops team.
Yes. Those are questions that need an answer before a service is released, and the ops team need the veto power + sufficient means to escalate recurring issues in ways that will hurt the dev teams enough to make them care.
That's why they have to be the same team. Your dev team needs to: 1. understand how to write code that can be debugged and recovered in a crisis, 2. learn to write code that doesn't cause the crisis in the first place, 3. have the motivation to do so because they will be the ones paged.
Anything else produces are far worse experience for everyone involved, especially customers.
The problem I've seen with DevOps is that some fief-builder sets up a "DevOps Organization", which is literally the opposite of what DevOps means.
Having engineers understand what makes a service resilient and easy to manage, as well as giving them incentives to produce high quality code doesn't require them to do SRE. In particular, being close enough to the SRE team, getting inputs when building the feature, detailed feedback and requests on live services, and the blame when shit hits the fan, are a more scalable way of doing it IMHO.
I don't believe engineers can understand this without experiencing it. A counter argument might be "football coaches don't need to have played", but the counter to that is "players need to have played". Why would being on call make you a better player? Personally, I think being the person watching the dashboards, in realtime, scraping the logs, in real time, watching how the contagion happens, in real time, gives insights that a next-day post mortem just doesn't. There are things I've learned only from hearing something on the conference call and then asking someone about it in chat. Watching a presentation on "Why 99% of people fell of our new bike" isn't nearly as useful as riding the fucking bike.
I've been on teams (briefly) where the ops team having an absolutely shit 48 hours was simply not on the dev teams radar, because they had more important things to worry about. That of course meant that when devs needed, e.g. a new host to try something, ops just dragged their feet, sometimes for months. Perhaps I've only seen dysfunctional companies in this regard. Or perhaps teams are made of humans, and splitting dev and ops is guaranteed to fail for any organization of more than 20 people? Like, sure, a startup of ten highly motivated people - maybe that works. Never done that myself: we've always run our own shit at the startups I've run/worked. Once that startup is 50 people, I think you're destined to fail with separate dev and ops.
Been paid for this over 40 years now. Done startups and FAANGs. But yes, the rewards better be fucking worth it, and the shifts better be reasonable.
So like, I'm not certain you're wrong. I don't think you can be wrong here. It's about what you're willing to do. Personally, I've done both ways, and if I started a company, I'd only hire people willing to support their shit with a pager. I'm not sure that matters for all businesses, but I'm not interested in starting, e.g. a mom-and-pop web hosting site for local businesses. If the service matters, devs are on call. So maybe I'm self selecting by the expected $ return.
For instance I probably have a decent understanding of what our CS team deals with, the tools they use, the volume and overall content of the customer feedback and requests that are coming to us. But I don't sit at a CS desk every day for a few hours and process the tickets of the service I'm running.
At the end, its just another part of your skillset with the general aim being the best / most experienced engineer to deliver best designed software for customer/company needs. There is no real seniority without being fully-versed in roughly everything that makes software stable and maintainable and delivering it. One cannot be had without the other.
- speak to production users
- ship to production
- monitor and learn from production usage
You don't need "devopsy" skills to do that, rather organizational structures that allow for incremental change. FWIW, deployments could still be a 2005-like upload via FTP, and the benefits of DevOps could still be reaped.
We’re seeing the same thing with QA. I guess developers can QA their own deliverables.
Leaving out the key fact that the vast majority of developers have no interest in testing stuff. And thus you end up “testing in prod” like with Microsoft projects.
It’s not a coincidence Microsoft’s software quality has collapsed since they’ve eliminated dedicated QA/SDET roles.
A programmer has to be able to demonstrate that his program has the required properties. If this comes as an afterthought, it is all but certain that he won't be able to meet this obligation: only if he allows this obligation to influence his design, there is hope he can meet it. Pure a posteriori verification denies you that wholesome influence and is therefore putting the cart before the horse, but that is exactly what happens in the software houses where "programming" and "quality assurance" are done by different groups. [Needless to say, those houses deliver without warranty.]
https://www.cs.utexas.edu/~EWD/transcriptions/EWD13xx/EWD130...
Why does the ability to interact with users, being able to test code, profile/debug code, and actually manage your deployed software imply you're a master of none? Too many developers act like it's a good state of affairs that they just get locked in the basement to grind the code. And now that we have tools that can grind code for you, they're finding out why that wasn't a good state of affairs for them.
The answer is master all of them, not none of them.
I’m seeing contracts with twice as many required skills for about half of what I was making in 2019.
If someone expects a non specialized developer to create a CI/CD pipeline for a brownfield project from scratch, or implement a test strategy including runner infra etc., then that's not DevOps. It's just stupid.
Once you write software that is run in places other than your computer by people other than you, you have the right and the duty to speak to those people, to look into those places, and to push updates in small increments once you've learned something valuable from either of the two sources. "You" can be a small team of close collaborators, so not every developer has to speak to users, and not all developers need to push to prod or look into the Grafana every now or then. It's just that those responsibilities should not be gated into remote departments.
Infrastructure is still provided and maintained by Ops. That's exactly the point of my critique: giving Devs access and responsibilities is no excuse to dump the whole Ops department. Managers shouldn't give someone a DevOps-title and think they get 2 for 1.
Similarly, asking Devs to write and maintain proper test automation for their services is the bare minimum - scraping centralized QA is still a bad call, though. Somebody must build and run test infra, set and enforce standards, aggregate data, track regressions, keep the full picture etc. - but they shouldn't block incremental deployments for days.
Nothing makes you take testing your code more seriously than having to respond to an outage caused by a bug code you wrote.
Extremely narrow focus makes one visibly worse off since topics in real world are never so narrow and to deliver anything complex in any engineering one has to understand cascading consequences of every complex decision. And what is software than series of complex decisions, usually about how to handle and secure data for backend and talk to user in frontend.
Without understanding how prod is actually used, monitored, how things are fixed etc its almost impossible to design, deliver and maintain any software worth its salt in any company.
Come to interview with such attitudes and if you ever get hired it will be for lowest pay scale. Given current market for juniors it would probably not be nice for you.
The way I've always phrased it is that if a company has "DevOps" teams or titles, they're not doing DevOps by definition
The "Ops" in DevOps was intended to mean the department itself, not the tasks they performed. The idea that devs should be learning or doing Systems Administration is the same as a building architect learning/doing concrete pours. They should know about concrete and the pouring/curing process, but they don't (and shouldn't) be pouring and smoothing the concrete themselves.
But they left behind a few practices that were easy to adopt and lots of people liked (whether that makes them “the good parts of agile” or “lazy process theater” depends on who you ask). Sprints, boards, paying attention to velocity, and so on.
For those who think Software Engineering roles are extinguishing, they just need to realize roles will evolve into something new. My current role (SRE) was a nascent niche at the start of my career and I didn't even know about it for many years.
Where does this fit in on the chart?
I've never been able to take "senior" seriously since.
One of the hardest things is that the same title (and level) can mean very different things across companies, industries and over time. When collecting pay information, we take great care to merge titles that mean the same thing while distinguishing those that don't. There's other edge cases like at some companies a title will be more specific (Security Engineer), while at others they'll retain the general title (Software Engineer) but specialize in the same thing (Security).
Claude has been really helpful in wrangling this all and expanding our taxonomy :)
- Technician
- Systems Engineer
- Senior Platform Engineer
- Lead Platform Engineer
- Principal AI EngineerThe point being: I'm always amazed at the cross-purpose impulse of arguing "the title is not important" while, simultaneously, really wanting to use the title.
I don't think so.
You ask the contractor for what you want, iterate a little, and take what they give you. Sometimes you may read the results.
Yours sincerely, Software Developer with 6 years of experience. Last week for the first time in my career I've actually merged application code. It was AI generated.
I'm confused about the point he's making here with the 6 years thing.
It's a weird one, because recently I've been working on a JS based system when I don't know JS and it's definitely very very different from when it's a language you're familiar with.
Comp Sci degree is somewhat of an exception, I get that
Software "engineers" in North America don't have any actual professional board that licenses or oversees their work. Until governments actually regulate the profession through these bodies, no one can gatekeep software development job titles.
A bunch of blog posts talking about "industry standards" is not a reference point for this.
So I wonder if users of "heavy AI" should also need to have a license.
Will your model and harness be the same in enough time to develop a curriculum for it and disseminate that to enough people to give it enough of a reputation to be worth doing for students, before it's obsolete?
I'd argue no on both counts.
This work has key differences vs operating heavy machinery.
It depends on the way they use it. If you design an architecture and use an LLM to fill it in, I'd consider a more technical label. If you're just using what Claude made, and providing feedback, that's just how software was done 10 years ago except then they'd be talking to a guy instead. And it would be orders of magnitude more expensive.
From my perspective, Claude feels like 5 engineers in a box. you can make things that required small organizations to make.
If you’re building and producing valuable software, you are still a creator or software engineer full stop. We can argue over titles, but don’t demote your own creative process to passive consumption.
Software Engineering is about so many things other than code it was an ongoing debate if code and programming should even be taught at a university level course, and while contemplating my masters of S.E. I had many great conversations with faculty about the distinction. Writing code is much more akin to welding or wiring, therefore belongs at a trade school, not a university.
After all, the discussion goes, Aerospace and Electrical Engineers don’t learn the doing of welding or pulling wires through crawl spaces, why should Software Engineers concern themselves with the doing of code.
>a S.E. should be able to differentiate good code from bad
A software Engineer needs to have all phases of the SDLC in mind, and be thinking about and planning all of them. Code is a tiny piece of the puzzle.
I’m thinking about requirements gathering, designing, testing, deployment, maintenance, long term change strategy, hand off to a new owner, etc, etc.
I could always see the merits of both sides of the discussion. Then at my first few jobs it very quickly became clear that many CS grads were just there to write code and that’s it. They didn’t have the big picture ideas at all, and I immediately understood the difference of a Software Engineer.
That being said, another 2ct:
> A software Engineer needs to have all phases of the SDLC in mind, and be thinking about and planning all of them. Code is a tiny piece of the puzzle.
I would not AT ALL call it tiny. Yes, is one part of MANY. But is your final delivery. Would be like Toyota saying: “hey they don’t really have to understand what makes a good car, the cars we sell is a tiny portion in the whole development cycle”. SW engineers do need to tell apart good from bad code, as much as good from bad tests and good from bad requirements. After all, if I somehow manage to deliver good SW (working and documented) I don’t gove a crap how the requirements and teat work or don’t in the company.
Which is exactly what we’re seeing now with LLMs writing code.
Interestingly you’ll see the same building a mega bridge or skyscraper. As long as the plan has been done well, it can be assembled by relatively very unskilled labor.
Note the test plan and acceptance criteria could be one and the same, depending.
That "if" is doing an unrealistic amount of lifting, here, in practice, and that's probably why human coders have had a better run than monkeys.
Nice sentence. But it has 2 problems:
1) Define "high enough" - There is where good requirements start, make it measurable! (How much is going to cost?)
2) Related to "you’ll see the same building a mega bridge or skyscraper". I've seen enough cases where "unskilled labor" messed up building something. You will reply "that is why I said 'Acceptance Criteria'!". My take is: you are making it far more complicated than it should, taking totally unnecessary risks, and it will be invariably more expensive in the long run.
I mean, I've seen enough examples of cases where the acceptance criteria was not properly checked, because also the ones doing the test were "unskilled" and had received a "perfect plan". (Just look at any infrastructure disaster we have every year)
And even in the best case: you discover the poured cement does not meet the specifications. What now? Blow the thing and start over? Better to have skilled labor.
Again, my 2ct. experience: hire experienced people, who actually have done the work all the way down to the lowest level.
"If you think knowledge is expensive, try with ignorance"
I only particularly mention this, as it sounds like your argument has shifted from "What does software engineering entail" -- to -- "What is software engineering that CS student's don't learn"
SDLC and other “engineering” concepts are all in computer science for most universities. The line you’re drawing between CS and SE grads just doesn’t exist in practice.
Also, where did you find a 5 year long software engineering bachelor’s?
As far as I’m aware, most pipelines are a 3 year bachelor’s degree in compsci. Or a 4 year long engineering bachelor’s with honours, in software engineering.
I normally don’t post, but these sorts of comments have always irked me. They give off gatekeepy, “No True Scotsman” vibes.
Funnily enough, this kind of thinking is part of the problem the family tree shows: there are so many different titles that are meaningless, which is what other commenters have pointed out
Of course no Electrical Engineer would be able to tell you the building code requirements for how close an outlet can be to a sink, or the ground, or a window. Every electrician can.
No Aerospace Engineer will be able to weld strange metal combinations, or be great at aluminum and thick stainless and oil pipeline pipe. Every ticketed welder can.
When you want the doing done, it’s great the Engineer has a working knowledge, which no doubt makes the project better, but no way in hell you get them to do it on the real thing. You get the ticketed tradesperson.
Actually, no. It is a HR concern. HR need to put people into boxes. Most people dont actually fit but whatever.
Actual engineers, doctors and lawyers are given that title not through their universities but through legally-recognized standards bodies. These are standards bodies that certify them, review their work, can investigate deviations from standards and even delicense people if needed. These standards bodies may require university education too, but they set the standards and certify professionals, not the university.
In North America, there aren't any such bodies for software development. So trying to gatekeep this term because of a university degree is absolutely not the same thing as lawyers, doctors or mechanical engineers.
I would be fully supportive of gatekeeping this btw if such professional bodies (recognized by law, not just self-initiated) existed, but pretending that the title software engineer is the same as mechanical engineers, doctors and lawyers is ridiculous. Lawyers have to pass the bar and maintain certification with the bar, and can be decertified if found to be performing outside of those standards.
Until there are actual standards bodies overseeing this, sorry, your university degree equally means the same as anyone self-taught so long as no one actually oversees the work and certifies people.
Pretending your university degree qualifies you to this is equally emotionally sensitive ("but but but my university WAS SUPER DUPER HARD U GUYZ")
As an example, I moved from Software Engineering to IT Controls Analyst. The pay brackets are lower and there are way fewer job postings. Even this chart of software related job completely leaves out any compliance or audit function, illustrating just how little my own profession thinks of us.
sometimes I hypothesize terms and roles, like a astro physicist imagining celestial objects that cant be detected for another 100 years
and that works too sometimes, you have to assume there is a mismatch between what employers want and what talent does
my background remains the same just what I’ve been doing the last 2 years gets radically reimagined
Some data engineers do come the SQL path, which is shown , but I came from a software background and work with AWS, Azure, Spark, Python, Scala and IaC. I'm always shocked at my peers lack of basic software foundations - I had an intermediate ask me what a "Python environment" was!
Funny how an almost 100 years old title is still relevant today.
I've also seen a huge number of data scientists rebrand as "AI engineers" so I agree with that title's placement in the chart but I think that ultimately that's the wrong place for it to live. I'm convinced that AI Engineer will be a specialization of backend engineering since the work to integrate agents into production systems will be backend work not model training or other data tasks.
Team Lead, Engineering Manager, Architect, Head of Engineering...
Everyone has a different view
I'd much rather have seen how "Architect" developed and merged back; similarily how "SRE" (and Platform Engineer) following its idea "when you treat operations as a software problem" merged System Administrators with Software Engineers; or how Full Stack Engineer with AI now has gone back to Product Engineer.
Somehow linked, I find both resources complement each other.
The magic incantation I found was to use the PNG version https://ianmiell.github.io/sw-job-title-family-tree/job_titl... since tbat just lets the browser handle it without any "smarts".
The main version wouldn't let me zoom in, the libked version would only let me pan, and the svg version wouldn't let me zoom out.