braid
Uncategorized
1
Posts
1
Posters
8
Views
-
undefined oblomov@sociale.network shared this topic on
Feed RSS
Gli ultimi otto messaggi ricevuti dalla Federazione
-
-
@mos_8502 LM Studio or oLlama and a nice open model
-
emptiness levels are holding at 100%
(100%) ■■■■■■■■■■
-
@mos_8502 Hank Green gave a good breakdown of the it.
-
@vwbusguy I have an old GPU, but 64GB of system RAM and 24 logical cores in my Xeon workstation?
-
@mos_8502 There's a lot of it depends, but most of the cost is AI is in training the models rather than the queries themselves. If you have a machine with a newish GPU and you download a model like granite or phi, the cost is your time and a nominal amount of electricity.
-
Like… what does it cost for a model to run for just one person, self hosted?
-
@mos_8502 Mainly get a capable model (like granite) locally on your box and call it with pytorch. I have a python script on my laptop for calling llm queries completely locally. Having a newish GPU will help.
