Okay. So the AI bubble bursts.
Uncategorized
19
Posts
5
Posters
2
Views
-
Okay. So the AI bubble bursts. It’s no longer financially viable to provide LLMs as a service. Okay. But the actual tech still exists. What do I, as a coder, need to own in that world to run something like ChatGPT or Claude locally?
Like… what does it cost for a model to run for just one person, self hosted?
-
Like… what does it cost for a model to run for just one person, self hosted?
@mos_8502 There's a lot of it depends, but most of the cost is AI is in training the models rather than the queries themselves. If you have a machine with a newish GPU and you download a model like granite or phi, the cost is your time and a nominal amount of electricity.
-
@mos_8502 There's a lot of it depends, but most of the cost is AI is in training the models rather than the queries themselves. If you have a machine with a newish GPU and you download a model like granite or phi, the cost is your time and a nominal amount of electricity.
@vwbusguy I have an old GPU, but 64GB of system RAM and 24 logical cores in my Xeon workstation?
-
@mos_8502 There's a lot of it depends, but most of the cost is AI is in training the models rather than the queries themselves. If you have a machine with a newish GPU and you download a model like granite or phi, the cost is your time and a nominal amount of electricity.
@mos_8502 Hank Green gave a good breakdown of the it.
-
Okay. So the AI bubble bursts. It’s no longer financially viable to provide LLMs as a service. Okay. But the actual tech still exists. What do I, as a coder, need to own in that world to run something like ChatGPT or Claude locally?
@mos_8502 LM Studio or oLlama and a nice open model
-
Like… what does it cost for a model to run for just one person, self hosted?
-
@swelljoe Fucking hell.
-
@mos_8502 This is not entirely accurate, but this is the cheapest way I've found to run something close to a reasonable coding model (quantized, so still not comparable to the frontier models, but comparable to frontier models from a couple years ago). Someone on YouTube coupled two of them to run a ~200GB model that gets close to modern standards. So, $4200.
A Mac Pro with 512GB of unified memory gets you in the ballpark of the frontier. (Though there aren't open models at quite that level.)
-
@swelljoe Fucking hell.
@mos_8502 that's a quarter what it cost a year ago. 🤷
-
@mos_8502 that's a quarter what it cost a year ago. 🤷
@mos_8502 cheapest option a year or two ago was probably 4 3090s on a server motherboard.
-
@mos_8502 This is not entirely accurate, but this is the cheapest way I've found to run something close to a reasonable coding model (quantized, so still not comparable to the frontier models, but comparable to frontier models from a couple years ago). Someone on YouTube coupled two of them to run a ~200GB model that gets close to modern standards. So, $4200.
A Mac Pro with 512GB of unified memory gets you in the ballpark of the frontier. (Though there aren't open models at quite that level.)
@mos_8502 GLM 4.7 can run in 205GB at the 2-bit quantization. Some of that can be system memory, so if you had a system with a couple of large GPUs and a ton of system RAM, you could run a very good open model...among the best, comparable to Sonnet 4.5. Still not Opus/Codex 5.2 level, but it'll write working code. https://unsloth.ai/docs/models/glm-4.7
-
@vwbusguy I have an old GPU, but 64GB of system RAM and 24 logical cores in my Xeon workstation?
@mos_8502 pytorch can use cpu. You'll just have to be a little more patient.
-
@mos_8502 GLM 4.7 can run in 205GB at the 2-bit quantization. Some of that can be system memory, so if you had a system with a couple of large GPUs and a ton of system RAM, you could run a very good open model...among the best, comparable to Sonnet 4.5. Still not Opus/Codex 5.2 level, but it'll write working code. https://unsloth.ai/docs/models/glm-4.7
@mos_8502 so, it continues to be cheapest to buy the investor-subsidized compute and GPU that OpenAI, Anthropic, Google, Microsoft, etc. want to provide. But, it does feel bad to me to trust in that or become dependent on that, especially since the models are all proprietary and I don't know what they're doing exactly or what they're doing with my data or that they'll do in the future.
-
@vwbusguy I have an old GPU, but 64GB of system RAM and 24 logical cores in my Xeon workstation?
-
-
-
@mos_8502 so, it continues to be cheapest to buy the investor-subsidized compute and GPU that OpenAI, Anthropic, Google, Microsoft, etc. want to provide. But, it does feel bad to me to trust in that or become dependent on that, especially since the models are all proprietary and I don't know what they're doing exactly or what they're doing with my data or that they'll do in the future.
@mos_8502 nvidia has a $3999 computer, the DGX Spark, that is roughly the same speed and specs as the AMD option, in benchmarks, but the nvidia ecosystem is more mature. And, ASUS makes a mini PC with the same nvidia chipset and specs for $2999. So, if you need the nvidia level of compatibility, you have to spend more for 128GB. But, that's become less important in the past year or so as AMD has invested in AI tooling. So, for now, AMD is a bargain, relative to Apple or nvidia.
Feed RSS
Gli ultimi otto messaggi ricevuti dalla Federazione
-
@terminaltilt and if you really have to go cloud, don't go to them, don't surrender and become their captive
-
@alsivx non ho ancora capito con quale logica hanno progettato il tutto. I loro prodotti sono sempre stati venduti come quelli "facili".
-
@stefano stay safe and all the best for today!
-
«Sorry, Baby», violenta perdita dell’innocenza | il manifesto
https://ilmanifesto.it/sorry-baby-violenta-perdita-dellinnocenza
> (Visioni) Agnes è una mente brillante, la studente migliore, quella che con le sue tesi riesce a catturare l'attenzione dei docenti di una piccola università del New England. In particolare del professore Preston Decker, autore di un romanzo attraverso il quale si è guadagnato la fiducia e la stima incondizionata della laureanda. Lei è benvoluta da
-
@terminaltilt this has been my thought for some time. The jacking up of prices being not just a market effect byproduct, but an intentional strategy to kill personal computing.
-
> long lasting
[citation needed]
Jokes aside, I agree with the principle, but I have myself doubts that it would prevent them from buying out politicians and have that being dismantled over time. 8-/
-
@benzinazero a dire il vero non è da ottobre, ma proprio da inizio 2025 perché la vidi a Firenze e ci feci pure un post che non so se è andato cancellato
-
@cancel I appreciate that the introduction is honest that both classical and jazz theory are limited in their usefulness if one's goal is to learn how to make contemporary music. unfortunately that is kinda my goal. it does explain why my prior attempts at a theory driven approach failed
