GPU is pretty much all that matters, more specifically, the amount of video ram on the card.  I incidentally got an older 16GB nvidia 3060 before I learned about LLMs and it can do quite a lot for $300.  I'd say you want 16GB minimum.  24GB is best you can do on normal cards.