Oddbean new post about | logout
▲ ▼
ynniv | 2 months ago (raw) | root | parent | reply | flag 
 If you really don't have options, you could see how fast Phi-3 mini is for you. That one's small enough to run on many phones.
▲ ▼
S!ayer | 2 months ago (raw) | root | parent | reply | flag 
 oh yeah? 

Nothing crazy, I just want to mess around locally/offline use cases for LLMs , I've been looking around but never really deployed locally yet
▲ ▼
ynniv | 2 months ago (raw) | root | parent | reply | flag 
 If you don't have a GPU or a Mac, try LLM Farm (iOS) or mllm (Android)
▲ ▼
S!ayer | 2 months ago (raw) | root | parent | reply | flag +2 
 I have a PC - 6GB card and 16GB RAM. 

I mean,  for the life of me I can't seem to find the right model to use. Llama seems great, in fact, imo it runs better than ChatGPT but that's because they have more nlp from Facebook/Meta platforms... 

Is there a way to run lllama on local without the need for AWS or Azure or a 4090 GTX
▲ ▼
ynniv | 2 months ago (raw) | root | parent | reply | flag +1 
 You should be able to run https://ollama.com with Microsoft's `phi3:mini-128k` or `llama3.1:8b-instruct-q3_K_M`. You can see the pre-configured models and their rough sizes at https://ollama.com/library.
▲ ▼
S!ayer | 2 months ago (raw) | root | parent | reply | flag +3 
 I'll just use Eris