Setting up the new servers for my new Mastodon instance.
Uncategorized
1
Posts
1
Posters
3
Views
-
Setting up the new servers for my new Mastodon instance. (It's just a toy deployment.)
-
undefined aeva@mastodon.gamedev.place shared this topic
Gli ultimi otto messaggi ricevuti dalla Federazione
-
@mos_8502 nvidia has a $3999 computer, the DGX Spark, that is roughly the same speed and specs as the AMD option, in benchmarks, but the nvidia ecosystem is more mature. And, ASUS makes a mini PC with the same nvidia chipset and specs for $2999. So, if you need the nvidia level of compatibility, you have to spend more for 128GB. But, that's become less important in the past year or so as AMD has invested in AI tooling. So, for now, AMD is a bargain, relative to Apple or nvidia.
-
@vwbusguy @ids1024 My thinking is, longer term, for whatever use can be made of it, I would prefer a little box I can put on a shelf that sits on my local network and provides roughly the same interface as Claude or ChatGPT. Something that doesn’t suck down too much power.
-
@ids1024 @mos_8502 The worst thing is offloading GPU compute to system memory for a large model. It can be swapping on a spinner bad. The good news is it's unlikely an individual would really need to use the larger model versions.
-
@mos_8502 @vwbusguy As I understand, for large AI models you want a lot of VRAM (ideally more than even high end modern gaming GPUs). So that old GPU and workstation hardware wouldn't be especially helpful. (If you want it to be reasonably fast.)
-
@mos_8502 so, it continues to be cheapest to buy the investor-subsidized compute and GPU that OpenAI, Anthropic, Google, Microsoft, etc. want to provide. But, it does feel bad to me to trust in that or become dependent on that, especially since the models are all proprietary and I don't know what they're doing exactly or what they're doing with my data or that they'll do in the future.
-
@mos_8502 pytorch can use cpu. You'll just have to be a little more patient.
-
@mos_8502 GLM 4.7 can run in 205GB at the 2-bit quantization. Some of that can be system memory, so if you had a system with a couple of large GPUs and a ton of system RAM, you could run a very good open model...among the best, comparable to Sonnet 4.5. Still not Opus/Codex 5.2 level, but it'll write working code. https://unsloth.ai/docs/models/glm-4.7
-
@mos_8502 cheapest option a year or two ago was probably 4 3090s on a server motherboard.
