After like 4 days I got tabby api to be a (partially implemented) drop in replacement for ollama so you can use open webui with xl2 models
Almost quit, couldn't tell if I was being regarded or not
like if the venture was even worth it (obvi was regarded)
If that were true, would you expect approx equal pay via patreon vs substack?
Like of patron was open and just voluntary donations and substack was gated
vLLM is nutty
Before I was doing naive queuing with text-gen-webui for parsing documents and getting 1 page/sec thinking it was good (way better than human speed)
An 8b on a 3090 handles 60 reqs/second with vLLM
The key behind vLLM is stuffing the GPU as full of prompts as possible. So there's one model and many prompts. As prompts get finished they're refilled with more prompts
Old way: 1 page/sec
vLLM 1x3090: 6 page/sec
Realized it wasn't using both cards, now its at 12 pages/sec
Gonna un-voltage-limit the cards next and find diff quant, idk if its even quanted
nostr:nprofile1qqsgydql3q4ka27d9wnlrmus4tvkrnc8ftc4h8h5fgyln54gl0a7dgspzemhxue69uhhyetvv9ujuurjd9kkzmpwdejhgqg5waehxw309aex2mrp0yhxgctdw4eju6t0qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyfehcpn's language-posting got me downloading Bob esponja and los Simpsons
There's a perspective where it makes sense to keep throwing money at it.
Even 'borrowing' from the future would be the right thing to do.
If the end result is AGI that changes everything, its analogous to inflating away debt. It would make everything so cheap and abundant that the initial cost doesn't matter, and getting there as fast as possible is the right thing to do.
That said, its not where my moneys going
My ai lead-gen web scraper is working nicely.
Minor tweaks like context limiting (currently naive tag-based depth) and a better way to avoid off-site links
This is cool and all, but even if it isconstitionallhy protected, its still kind of weak.
All constitution law analysis boils down to "if the gov really wants something, its going to get it"
There's different levels of scrutiny but they basically restate the above.
At the highest, strict scrutiny, they need "a compelling government interest" + the rules/means are "narrowly tailored" to that interest.
The control over money is the life blood of the government, I don't see anything more compelling from the gov's perspective
I think it was less true. Actually, no I'd say equally true
maybe its a cumulative thing, like the longer we have access to cheap delicious food the fatter we'll get
V interesting.
Seems like a big vector is cospends
I wonder if anyone's building async siloed transactions
Like keep coins separate and send gradually to a variety of destination addresses
Oddbean new post about login | logout
Notes by John | export