As someone who has been in the "robot devops" space for like a decade now, I think with robot companies it becomes particularly acute because:
- Ops for cloud services and ops for deployed robots are very different beasts, so it's really two different teams that could both reasonably be "devops".
- Ops/tools for robot developers is different since you're often dealing with simulation environments (local, cloud) and also managing access to a more constrained pool of test hardware, all of which needs to be able to quickly recover and restore itself to factory state after developers or their code have done unspeakable things.
- Deployed units are often the worst of both worlds between multiple domains. You've got the big deployment assets of a data center but without the big data pipe; nothing is virtualized so you can't just throw away a node that goes bad; you're stuck supporting hardware snowflakes (here's a text file somewhere with the list of units that have bad CPU fans and need different frequency caps); it's like IOT but with the power and disk and observability requirements of a real server.
Plus there's the mess of "different audiences need different views of info about the robots in order to triage" and those lines are murky, and someone's always trying a new tool, and "ugh, the new thing is great for group A, but now I can't efficiently see Y, and the old tool was fine (for me)"
> IOT but with the power and disk and observability requirements of a real server
And the issue with the robot is usually none of the regular observability health metrics of a real server... but when it is... "where does that dashboard live again, and is it still working?"
- Ops for cloud services and ops for deployed robots are very different beasts, so it's really two different teams that could both reasonably be "devops".
- Ops/tools for robot developers is different since you're often dealing with simulation environments (local, cloud) and also managing access to a more constrained pool of test hardware, all of which needs to be able to quickly recover and restore itself to factory state after developers or their code have done unspeakable things.
- Deployed units are often the worst of both worlds between multiple domains. You've got the big deployment assets of a data center but without the big data pipe; nothing is virtualized so you can't just throw away a node that goes bad; you're stuck supporting hardware snowflakes (here's a text file somewhere with the list of units that have bad CPU fans and need different frequency caps); it's like IOT but with the power and disk and observability requirements of a real server.
> IOT but with the power and disk and observability requirements of a real server
And the issue with the robot is usually none of the regular observability health metrics of a real server... but when it is... "where does that dashboard live again, and is it still working?"