Piero Bosio Social Web Site Personale

@ids1024 @mos_8502 The worst thing is offloading GPU compute to system memory for a large model. It can be swapping on a spinner bad. The good news is it's unlikely an individual would really need to use the larger model versions.

@mos_8502 pytorch can use cpu. You'll just have to be a little more patient.

@mos_8502 Hank Green gave a good breakdown of the it.

https://youtu.be/H_c6MWk7PQc

@mos_8502 There's a lot of it depends, but most of the cost is AI is in training the models rather than the queries themselves. If you have a machine with a newish GPU and you download a model like granite or phi, the cost is your time and a nominal amount of electricity.

@mos_8502 Mainly get a capable model (like granite) locally on your box and call it with pytorch. I have a python script on my laptop for calling llm queries completely locally. Having a newish GPU will help.