Having done all three over my career (local machines as student, University's HPC cluster as grad/postgrad and eventually cloud in the private sector), I'd 100% rent out compute. Buying a custom rig for local ML training makes zero sense in 2026. With renting you'll be able to train bigger models/datasets faster on much better hardware while paying far less in total. Especially if you're doing project-based work.
If you use a custom disk image too, you can have it boot, apply security updates, reboot, then ssh in within minutes.
Work for however long, shut down. I do this with linode and their GPU VMs. If you shutdown nightly and on weekends, the costs drop dramatically on a monthly spend.
I only need it a few hours per day, so very cheap for custom compute.
Oh, you're just talking about deleting it when you're not using it in general, rather than during peak times. Makes sense.
[−]newComputerWhoT · 2026-11-15 Sun 19:12 UTC ·
link
The usage pattern i've seen on flexible compute is more aligned with 'I need this process to execute, run it when it's cheap' not 'I'm going to kick off this process, stop it if it get's expensive.' for example AWS Spot Instances
I mean, there are lots of reasons why you'd want something local, but I agree that the cost makes it unreasonable right now.
[−]probably_wrong · 2026-11-15 Sun 12:13 UTC ·
link
Friends of mine wanted to buy a GPU server last year for their new project. Management told them no, that there was a deal with cloud providers and that investing the budget in cloud compute was better.
The project ran out of budget roughly three months in. From what I heard, status meetings consist of everyone getting together once a month, asking "have we gotten any more budget yet? no?", and then wait until the project runs out.
Cloud providers need to make a profit too, and they've gotten very good at ensuring that their clients forget this point. The whole advantage of cloud compute is that you don't pay for the time you're not using it, but once you reach a certain point you're the one covering the overhead for everybody else.
As a scientist with a big workstation and access to several HPCs I can say that there is nothing better than having a powerful workstation. Yes, doing some sort of long runs is better on a cluster, but being able to do things locally without queue delays and various limitations imposed by HPCs is extremely valuable for initial development/experiments.
HPC ops teams are either really good or they're a dumping ground for grumpy DBAs/Operations people who have no skin in the game of getting anything done and it's a complete nightmare working with them.
As former HPC ops albeit in public service, we also tend to have funding for everything but ops personnel. This means you also often cannot afford the personnel you might be wishing for as they tend to go more towards better paying fields like devops etc.
It might (I say might) be worth your looking at xc [1] as a way to get more performance out of local/LAN hardware - assuming your program is data-parallel anyway.
[−]SubiculumCode · 2026-11-15 Sun 17:50 UTC ·
link
Thanks!!
[−]SubiculumCode · 2026-11-15 Sun 17:49 UTC ·
link
Exactly. The number of times you go, "oops, it would be better if I changed X first" matters less if you can restart right away without waiting to restarts (and also using up your credits on the aborted run).
Work for however long, shut down. I do this with linode and their GPU VMs. If you shutdown nightly and on weekends, the costs drop dramatically on a monthly spend.
I only need it a few hours per day, so very cheap for custom compute.
The custom volume keeps your persistent install, configs, work in progress, etc.
You can programmatically kick off a linode, attach the volume, then boot, as well as delete, all via curl or python.
So not variable rates, but no rates when the beast is deleted.
The project ran out of budget roughly three months in. From what I heard, status meetings consist of everyone getting together once a month, asking "have we gotten any more budget yet? no?", and then wait until the project runs out.
Cloud providers need to make a profit too, and they've gotten very good at ensuring that their clients forget this point. The whole advantage of cloud compute is that you don't pay for the time you're not using it, but once you reach a certain point you're the one covering the overhead for everybody else.
1: https://github.com/ThrudTheBarbarian/xc