has anyone started using a dedicated server for using open source ai stuff? is there something like umbrel, but for ai stuff? umbrel has llamagpt, but it takes up a lot of resources, so prob not a good idea to run that on the same box as a bitcoin/lightning node.
Most advanced AI stuff needs GPUs. In theory huggingface but run locally.
i just wanna run open source llms for privacy and to not have to pay companies to use their apis
For llms use ollama.
my macbook is too shitty. that's why i want to get a dedicated machine. would prefer to just buy a machine with all the necessary hardware rather than building it piece by piece.
Your shitty MacBook is better than the one Laszlo mined thousands of bitcoins with in 2010.
sir, that's cause it was easy to mine bitcoin back then. you have not seen my macbook. barely any disk space left. friggen thing is about to die. 😂
I don't think there is anything similar like umbrel for ai stuff. But anyone can install ollama on ther desktop and try things out before deciding between upgrading their GPU, building dedicated server with multiple GPUs, or just pay for AI services that support API call.
my macbook is old and can barely run the primal web app. so i doubt it would be good enough to run anything. 😂
Depending on how old it is, the best thing to try now is to install ollama, try a small-ish model such as phi3-mini, and see if you like it
i want to use llama 3.1 with 405 billion params though 😂
actually, maybe the 70 billion param one would be fine 😂
Build a 3x 3090 machine. Keep us updated 😁
Check out @Start9 / StartOS with FreeGPT. Also Venice AI is worth looking at from privacy preservation perspective.
doesn't it take up a lot of resources to run? doesn't seem like a good idea to run it on a start9 along with a bitcoin/lightning node.
The newer 7B models like Llama3 or Mistrel work great on well-resourced devices like the ones we sell.
No real harm in running it alongside Bitcoin and Lightning, but you also don't have to. It's your server, only install what you want.
So it runs smoothly on the Server One with both a bitcoin node and an AI chat app hooked up to Llama 3.1 8B? Is OpenUI available on Start9?
https://github.com/wandb/openui
Yeah that model runs just fine. Nothing compared to ChatGPT speeds but it's affordable consumer hardware. No, nobody has packaged OpenUI for StartOS yet, looks cool though.
Buy an expensive GPU, stick it in a box, install Pinokio.computer & holesail.io and you have a remote window into all the ai tools you want
Oddbean new post about login | logout