← All Posts

Your Phone Can Now Run a 400B AI Model — No Cloud Required

An iPhone 17 Pro just ran a 400 billion parameter language model locally. No API keys. No monthly subscriptions. No data leaving your device.

This happened yesterday. A team called ANEMLL demonstrated running a massive LLM directly on consumer hardware — and the Hacker News community lost it (638 points, 281 comments).

For indie hackers, this isn't just a cool demo. It's a paradigm shift.

What Actually Happened

ANEMLL (Artificial Neural Engine Machine Learning Library) is an open-source project that ports large language models to Apple's Neural Engine. Their latest demo showed a 400B parameter model running inference on an iPhone 17 Pro.

This isn't vaporware. It's real hardware running real models. And the software stack is open source.

Why This Matters for Builders

Let me break down what this means if you're building products:

Think about that. Every AI-powered feature you ship today has a per-use cost. Every API call is money leaving your pocket. On-device inference eliminates that entirely.

But cost is just the beginning:

• Privacy by default. User data never leaves their phone. No GDPR headaches, no data breach risk.
• Works offline. Airplane mode? Rural area? Dead zone? Your app still works.
• No rate limits. No throttling, no "you've exceeded your quota" errors.
• Zero latency after load. No network round trips. The model is right there.

The Open Source Stack

ANEMLL provides a complete pipeline:

• Model Conversion: Convert any Hugging Face LLaMA model (including DeepSeek distilled variants) to CoreML format
• ANE Optimization: Intelligent model splitting for iOS (1GB limit) and macOS (2GB limit)
• Inference Tools: Ready-to-use chat interfaces with conversation history and performance monitoring

They're currently on alpha release 0.1.1. Early, but functional.

What You Can Build Right Now

Here's where my mind goes as someone who ships products:

1. Privacy-first journaling app. AI-powered mood analysis and insights that never sync to a server. Parents would pay for this for their kids.

2. Offline translation. Travel apps that work in the subway, the plane, the middle of nowhere. The market for this is massive and underserved.

3. Local code assistant. A coding companion that works in restricted environments — government, healthcare, finance — where cloud AI is banned.

4. Smart home controller. Process voice commands locally. No always-on microphone sending data to the cloud.

None of these require a PhD in ML. The ANEMLL stack handles the hard parts. You build the product layer.

The Honest Take

Let's be real about where we are:

Running a 400B model on a phone is impressive but constrained. Context windows are limited. Generation speed isn't going to beat GPT-4 for complex reasoning tasks. The alpha software will have bugs.

But here's what matters: the trajectory is clear. Hardware keeps getting faster. Model architectures keep getting more efficient. Quantization techniques keep squeezing more into less.

A year ago, running a 7B model on a phone was news. Today it's 400B. That's a 57x improvement in one year.

Where do you think we'll be in another year?

What Smart Builders Do

The indie hackers who win aren't the ones waiting for perfect technology. They're the ones building with what's available today and positioning for what's coming tomorrow.

Start experimenting with on-device inference now. Understand the constraints. Build your mental model of what works and what doesn't. When the hardware catches up to your ambitions (and it will), you'll already know exactly what to build.

The cloud AI era isn't ending tomorrow. But the on-device AI era has officially started. And the window for building in this space while it's still early? That window is open right now.

Source: HN #9 today — 638 points, 281 comments