DevOps'issue IMHO is developing software and dealing with the ops side are two fundamentally different propositions, and require two different life styles.
Doing ops also means entering the on-call rotations, being in the war room when a data center goes under and being on the hook when a DDOS brings down the service.
Very few people want that in their life, money be damned. Same way in an ideal world devs should also participate in customer support, but very few could really do it or handle that for years.
Last team I was on was repsonsible for supporting and maintaining their product, with help from the SRE team. It means after-hours support for issues and (maybe) releases but I think this was the right way. You became the backstop and support resource for things like data center outages and DDOS attacks that were largely the responsiblity of the (much smaller than many places) SRE team. We also put devs very closely behind front-line customer facing staff where it made sense (ex: reporting & analytics - yes, platform - probably not). It made the devs better and they might try and deny it but I think most of them actually appreciated it. #YMMV
The issue I was facing in a previous life was the turnover. We had data scientists, DBA and many specialized roles stay pretty long, where devops moved away at about twice the rate, and work life balance was mentionned in many exit interviews.
I agree with you that it's extremely formative, and having devs involved basically from start to end of the service surface gives a lot of flexibility. The downside being that it requires a ton of different skills as well, and while people learn, many won't get comfortable with it even after months.
This makes it specially tough when people get called at 3 in the morning to deal with an emergency while they've already spent half the night sleep deprived from their kid puking all night. The worst part being that while people could understand wanting to skip turn one or two times, engineers with small kids will have enough going in their lives that most on-call instances land at a time they're busy dealing with more important matters.
> being in the war room when a data center goes under
When doesn't this involve devs? If you have ops separate and isolated from the development effort, a devs representation will have to get involved at some point.
Imagine managing a gitlab instances. You're doing it for years and have a stable environment. If one of the instance gets hammered by external traffic or one of your data center dies, will you call the gitlab devs to deal with it ?
Depending on the size of the deployment. (Not sure what you mean with "one of your data center" though) If it's just you, you deal with it. If you're running a bigger deployment, you likely have GitLab devs internally that will be reachable. Even bigger - you're likely to have access to upstream devs in the ultimate tier even with self hosting.
> DevOps'issue IMHO is developing software and dealing with the ops side are two fundamentally different propositions, and require two different life styles. Doing ops also means entering the on-call rotations, being in the war room when a data center goes under and being on the hook when a DDOS brings down the service.
That's exactly the issue that DevOps aims to solve, though. When you have developers who have no ownership or responsibility once the software is running on production, it becomes a "throw it over the fence" situation, and devs don't care about the operational side.
Logging? Who needs that, if there's a bug report I'll reproduce it on my local machine (and if I can't, CLOSE-WONTFIX!) Performance monitoring? That's an ops problem, tell 'em to buy better hardware. Application crashes? Works on my machine, probably a bug in the server config; ops problem. Tell them to set up a scheduled restart every hour.
You still need dedicated operations staff who deal with the overall system health and design, but yes, a developer needs to be involved when a DDOS brings down the service, because how should the system behave under heavy traffic? Can it serve up cached read-only pages? Did someone find a very CPU-intensive public endpoint that is being exploited and needs to be fixed? Only your developers can answer this question, not the ops team.
That's an org issue. For a given service dozens of roles will be involved, if devs are needed to do every single of them to care and be invested, there is a problem.
> [...] Only your developers can answer this question, not the ops team.
Yes. Those are questions that need an answer before a service is released, and the ops team need the veto power + sufficient means to escalate recurring issues in ways that will hurt the dev teams enough to make them care.
[−]lowbloodsugar · 2026-11-14 Sat 21:16 UTC ·
link
>DevOps'issue IMHO is developing software and dealing with the ops side are two fundamentally different propositions
That's why they have to be the same team. Your dev team needs to: 1. understand how to write code that can be debugged and recovered in a crisis, 2. learn to write code that doesn't cause the crisis in the first place, 3. have the motivation to do so because they will be the ones paged.
Anything else produces are far worse experience for everyone involved, especially customers.
The problem I've seen with DevOps is that some fief-builder sets up a "DevOps Organization", which is literally the opposite of what DevOps means.
These are all good point, that I think aren't really related to having devs do operations.
Having engineers understand what makes a service resilient and easy to manage, as well as giving them incentives to produce high quality code doesn't require them to do SRE. In particular, being close enough to the SRE team, getting inputs when building the feature, detailed feedback and requests on live services, and the blame when shit hits the fan, are a more scalable way of doing it IMHO.
[−]lowbloodsugar · 2026-11-15 Sun 00:33 UTC ·
link
>Having engineers understand what makes a service resilient and easy to manage,
I don't believe engineers can understand this without experiencing it. A counter argument might be "football coaches don't need to have played", but the counter to that is "players need to have played". Why would being on call make you a better player? Personally, I think being the person watching the dashboards, in realtime, scraping the logs, in real time, watching how the contagion happens, in real time, gives insights that a next-day post mortem just doesn't. There are things I've learned only from hearing something on the conference call and then asking someone about it in chat. Watching a presentation on "Why 99% of people fell of our new bike" isn't nearly as useful as riding the fucking bike.
I've been on teams (briefly) where the ops team having an absolutely shit 48 hours was simply not on the dev teams radar, because they had more important things to worry about. That of course meant that when devs needed, e.g. a new host to try something, ops just dragged their feet, sometimes for months. Perhaps I've only seen dysfunctional companies in this regard. Or perhaps teams are made of humans, and splitting dev and ops is guaranteed to fail for any organization of more than 20 people? Like, sure, a startup of ten highly motivated people - maybe that works. Never done that myself: we've always run our own shit at the startups I've run/worked. Once that startup is 50 people, I think you're destined to fail with separate dev and ops.
Been paid for this over 40 years now. Done startups and FAANGs. But yes, the rewards better be fucking worth it, and the shifts better be reasonable.
So like, I'm not certain you're wrong. I don't think you can be wrong here. It's about what you're willing to do. Personally, I've done both ways, and if I started a company, I'd only hire people willing to support their shit with a pager. I'm not sure that matters for all businesses, but I'm not interested in starting, e.g. a mom-and-pop web hosting site for local businesses. If the service matters, devs are on call. So maybe I'm self selecting by the expected $ return.
Any developer who doesn't at least *understand* the ops side of the fence and has never spent time actually *doing* it, will make bad decisions along the way from an ops perspective. These are bad developers.
I see a stark difference between understanding how it works and doing it in a professional manner.
For instance I probably have a decent understanding of what our CS team deals with, the tools they use, the volume and overall content of the customer feedback and requests that are coming to us. But I don't sit at a CS desk every day for a few hours and process the tickets of the service I'm running.
You should at least be able to step in and do that job, however, if necessary. That's different than that being the most efficient use of your time, but it's something every good developer should be *capable* of doing.
Understanding anything is not binary, and evolves over time. Feeling like you understand something and then having at least some continuous hands-on experience are incomparable levels.
At the end, its just another part of your skillset with the general aim being the best / most experienced engineer to deliver best designed software for customer/company needs. There is no real seniority without being fully-versed in roughly everything that makes software stable and maintainable and delivering it. One cannot be had without the other.
Doing ops also means entering the on-call rotations, being in the war room when a data center goes under and being on the hook when a DDOS brings down the service.
Very few people want that in their life, money be damned. Same way in an ideal world devs should also participate in customer support, but very few could really do it or handle that for years.
The issue I was facing in a previous life was the turnover. We had data scientists, DBA and many specialized roles stay pretty long, where devops moved away at about twice the rate, and work life balance was mentionned in many exit interviews.
I agree with you that it's extremely formative, and having devs involved basically from start to end of the service surface gives a lot of flexibility. The downside being that it requires a ton of different skills as well, and while people learn, many won't get comfortable with it even after months.
This makes it specially tough when people get called at 3 in the morning to deal with an emergency while they've already spent half the night sleep deprived from their kid puking all night. The worst part being that while people could understand wanting to skip turn one or two times, engineers with small kids will have enough going in their lives that most on-call instances land at a time they're busy dealing with more important matters.
When doesn't this involve devs? If you have ops separate and isolated from the development effort, a devs representation will have to get involved at some point.
So... Yes, sometimes you will.
That's exactly the issue that DevOps aims to solve, though. When you have developers who have no ownership or responsibility once the software is running on production, it becomes a "throw it over the fence" situation, and devs don't care about the operational side.
Logging? Who needs that, if there's a bug report I'll reproduce it on my local machine (and if I can't, CLOSE-WONTFIX!) Performance monitoring? That's an ops problem, tell 'em to buy better hardware. Application crashes? Works on my machine, probably a bug in the server config; ops problem. Tell them to set up a scheduled restart every hour.
You still need dedicated operations staff who deal with the overall system health and design, but yes, a developer needs to be involved when a DDOS brings down the service, because how should the system behave under heavy traffic? Can it serve up cached read-only pages? Did someone find a very CPU-intensive public endpoint that is being exploited and needs to be fixed? Only your developers can answer this question, not the ops team.
That's an org issue. For a given service dozens of roles will be involved, if devs are needed to do every single of them to care and be invested, there is a problem.
> [...] Only your developers can answer this question, not the ops team.
Yes. Those are questions that need an answer before a service is released, and the ops team need the veto power + sufficient means to escalate recurring issues in ways that will hurt the dev teams enough to make them care.
That's why they have to be the same team. Your dev team needs to: 1. understand how to write code that can be debugged and recovered in a crisis, 2. learn to write code that doesn't cause the crisis in the first place, 3. have the motivation to do so because they will be the ones paged.
Anything else produces are far worse experience for everyone involved, especially customers.
The problem I've seen with DevOps is that some fief-builder sets up a "DevOps Organization", which is literally the opposite of what DevOps means.
Having engineers understand what makes a service resilient and easy to manage, as well as giving them incentives to produce high quality code doesn't require them to do SRE. In particular, being close enough to the SRE team, getting inputs when building the feature, detailed feedback and requests on live services, and the blame when shit hits the fan, are a more scalable way of doing it IMHO.
I don't believe engineers can understand this without experiencing it. A counter argument might be "football coaches don't need to have played", but the counter to that is "players need to have played". Why would being on call make you a better player? Personally, I think being the person watching the dashboards, in realtime, scraping the logs, in real time, watching how the contagion happens, in real time, gives insights that a next-day post mortem just doesn't. There are things I've learned only from hearing something on the conference call and then asking someone about it in chat. Watching a presentation on "Why 99% of people fell of our new bike" isn't nearly as useful as riding the fucking bike.
I've been on teams (briefly) where the ops team having an absolutely shit 48 hours was simply not on the dev teams radar, because they had more important things to worry about. That of course meant that when devs needed, e.g. a new host to try something, ops just dragged their feet, sometimes for months. Perhaps I've only seen dysfunctional companies in this regard. Or perhaps teams are made of humans, and splitting dev and ops is guaranteed to fail for any organization of more than 20 people? Like, sure, a startup of ten highly motivated people - maybe that works. Never done that myself: we've always run our own shit at the startups I've run/worked. Once that startup is 50 people, I think you're destined to fail with separate dev and ops.
Been paid for this over 40 years now. Done startups and FAANGs. But yes, the rewards better be fucking worth it, and the shifts better be reasonable.
So like, I'm not certain you're wrong. I don't think you can be wrong here. It's about what you're willing to do. Personally, I've done both ways, and if I started a company, I'd only hire people willing to support their shit with a pager. I'm not sure that matters for all businesses, but I'm not interested in starting, e.g. a mom-and-pop web hosting site for local businesses. If the service matters, devs are on call. So maybe I'm self selecting by the expected $ return.
For instance I probably have a decent understanding of what our CS team deals with, the tools they use, the volume and overall content of the customer feedback and requests that are coming to us. But I don't sit at a CS desk every day for a few hours and process the tickets of the service I'm running.
At the end, its just another part of your skillset with the general aim being the best / most experienced engineer to deliver best designed software for customer/company needs. There is no real seniority without being fully-versed in roughly everything that makes software stable and maintainable and delivering it. One cannot be had without the other.