Oddbean new post about login | logout I’ve spent a non-trivial amount of time investigating how to bring offline machine translations to iOS. Because I want that for Damus and other Nostr iOS apps. It’s wild that no one has done this yet, including Apple. Except maybe Google, but their ML Kit library is horribly maintained and unusable for the modern iOS development environment. All I’ve been able to find are low-level C++ tools and models that you have to figure out how to piece together. Going to keep searching.
It’s not a problem that Google or Apple would be incentivized to solve as their model is to gather all possible information on us so that they can provide the best possible product (us) to their customers (governments, advertisers).
When it comes to language translation, it seems that the trend is the biggest ML models do the best. Meta's AI lab that has arguably the best translation models found that training a giant NN to translate from any 1k languages as input to any 1k as output, performs much better than a NN model specific to any single pair of languages (like only English <-> Spanish). Anyway, this is all to say that maybe language translation quality vs on-device compute has been a blocker?
Ah thats a bummer to learn. I thought Apple would have exposed the API they use in their Translate app but I’d never looked into it. Maybe when they put LLMs on device later this year we can use those?
You usually need to download a large model to make it work. We have neural cores, would love to put them to use.
@tyiu looked a while back, and we were disappointed not to find apple translation API available to devs
MLC LLM has an iOS implementation of an offline capable chat app using LLMs. However, it’s several gigabytes large and performance isn’t great on my iPhone (faster on my MacBook). Hoping for better performance and more direct translation APIs as a library as folks develop in this space more. https://apps.apple.com/us/app/mlc-chat/id6448482937 https://github.com/mlc-ai/mlc-llm