Okay. So the AI bubble bursts.

[mos_8502@studio8502.ca]

@swelljoe Fucking hell.

[swelljoe@mas.to]

@mos_8502 This is not entirely accurate, but this is the cheapest way I've found to run something close to a reasonable coding model (quantized, so still not comparable to the frontier models, but comparable to frontier models from a couple years ago). Someone on YouTube coupled two of them to run a ~200GB model that gets close to modern standards. So, $4200.

A Mac Pro with 512GB of unified memory gets you in the ballpark of the frontier. (Though there aren't open models at quite that level.)

[swelljoe@mas.to]

@mos_8502 that's a quarter what it cost a year ago. 🤷

[swelljoe@mas.to]

@mos_8502 cheapest option a year or two ago was probably 4 3090s on a server motherboard.

[swelljoe@mas.to]

@mos_8502 GLM 4.7 can run in 205GB at the 2-bit quantization. Some of that can be system memory, so if you had a system with a couple of large GPUs and a ton of system RAM, you could run a very good open model...among the best, comparable to Sonnet 4.5. Still not Opus/Codex 5.2 level, but it'll write working code. https://unsloth.ai/docs/models/glm-4.7

[vwbusguy@mastodon.online]

@mos_8502 pytorch can use cpu. You'll just have to be a little more patient.

[swelljoe@mas.to]

@mos_8502 so, it continues to be cheapest to buy the investor-subsidized compute and GPU that OpenAI, Anthropic, Google, Microsoft, etc. want to provide. But, it does feel bad to me to trust in that or become dependent on that, especially since the models are all proprietary and I don't know what they're doing exactly or what they're doing with my data or that they'll do in the future.

[ids1024@mathstodon.xyz]

@mos_8502 @vwbusguy As I understand, for large AI models you want a lot of VRAM (ideally more than even high end modern gaming GPUs). So that old GPU and workstation hardware wouldn't be especially helpful. (If you want it to be reasonably fast.)

[vwbusguy@mastodon.online]

@ids1024 @mos_8502 The worst thing is offloading GPU compute to system memory for a large model. It can be swapping on a spinner bad. The good news is it's unlikely an individual would really need to use the larger model versions.

[mos_8502@studio8502.ca]

@vwbusguy @ids1024 My thinking is, longer term, for whatever use can be made of it, I would prefer a little box I can put on a shelf that sits on my local network and provides roughly the same interface as Claude or ChatGPT. Something that doesn’t suck down too much power.

[swelljoe@mas.to]

@mos_8502 nvidia has a $3999 computer, the DGX Spark, that is roughly the same speed and specs as the AMD option, in benchmarks, but the nvidia ecosystem is more mature. And, ASUS makes a mini PC with the same nvidia chipset and specs for $2999. So, if you need the nvidia level of compatibility, you have to spend more for 128GB. But, that's become less important in the past year or so as AMD has invested in AI tooling. So, for now, AMD is a bargain, relative to Apple or nvidia.