Oddbean new post about login | logout Ah thats a bummer to learn. I thought Apple would have exposed the API they use in their Translate app but I’d never looked into it. Maybe when they put LLMs on device later this year we can use those?
You usually need to download a large model to make it work. We have neural cores, would love to put them to use.
@tyiu looked a while back, and we were disappointed not to find apple translation API available to devs
MLC LLM has an iOS implementation of an offline capable chat app using LLMs. However, it’s several gigabytes large and performance isn’t great on my iPhone (faster on my MacBook). Hoping for better performance and more direct translation APIs as a library as folks develop in this space more. https://apps.apple.com/us/app/mlc-chat/id6448482937 https://github.com/mlc-ai/mlc-llm
I was assuming they will just bundle a decent small LLM with iOS 18. But on second thought they will probably chicken out, because if it’s on device how do you update it every week to patch the “exploits” of people making it say racist things or describing how to make a bomb.