Bridging the Walled Gardens

Think about the last time you needed to find something important. Not a file. Not a web page. Something personal. That sushi restaurant your friend recommended in a WhatsApp message a few weeks ago. The time you agreed to meet someone, buried in an iMessage thread you've long since scrolled past. The address someone texted you — or was it an email? Or a voice note?

You know this information exists somewhere in your digital life. You just can't reach it.

This bugged me enough that I decided to do something about it. Using Claude Code, I built Cortex — a personal knowledge bot that runs locally on a Mac desktop and can answer questions about my life by searching across iMessage, WhatsApp, Gmail, Google Calendar, Google Drive, and Apple Photos metadata. Something that would have taken me weeks or months to build on my own, I was able to get working in a matter of hours. And it's genuinely changed how I find things.

This post is the story of that journey — what I built, how Claude Code made it possible, and what I learned along the way.

The gap that exists today

There's been an explosion of AI integrations lately. Plug your email into this. Connect your calendar to that. Give this app access to your documents. AI assistants are increasingly connecting to data sources — but predominantly the ones with clean APIs.

Think about where your most important personal information actually lives though. It's your conversations. The group chat where plans get made. The WhatsApp thread where someone sent you an address. The iMessage exchange where you confirmed a price. The voice note with the name you keep forgetting. Messaging is the connective tissue of your life — and it's a data source that's largely untouched by AI integrations today.

iMessage and WhatsApp don't have consumer APIs. That's understandable — there are good reasons platforms keep messaging data tightly controlled. But it does mean that nobody's building the cross-platform personal AI that can search across all of your data sources at once.

So I built one myself.

The data is already on your computer

Your iMessages are stored in a SQLite database on your Mac. Right now. At ~/Library/Messages/chat.db. Every text, every group chat, every reaction — queryable with SQL if you know where to look. Your WhatsApp data is on disk too. Your emails are a few API calls away. Your calendar, your contacts, your documents — all either local or accessible through authentication you've already granted.

I wasn't building an enormous data pipeline from scratch. I was unlocking what was already there.

How MCP ties it all together

If you're reading this, you probably already know about the Model Context Protocol — MCP. For the uninitiated: it's an open protocol, originally from Anthropic and now governed by the Linux Foundation, that lets you turn any data source into a tool an LLM can use. You write a small server that exposes your data through a standardised interface, and any MCP-compatible AI can discover and use it.

What you might not have considered is what happens when you point MCP at your own personal data. I built six servers — one each for iMessage, WhatsApp, Gmail, Google Drive, Calendar, and Apple Photos. Here's roughly how it fits together:

graph LR
    User["Prompt"] --> Planner["Planner"]
    Planner --> iMessage["iMessage"]
    Planner --> WhatsApp["WhatsApp"]
    Planner --> Gmail["Gmail"]
    Planner --> Drive["Google Drive"]
    Planner --> Calendar["Calendar"]
    Planner --> Photos["Apple Photos"]
    iMessage --> Response["Response"]
    WhatsApp --> Response
    Gmail --> Response
    Drive --> Response
    Calendar --> Response
    Photos --> Response
    Response -->|"loop if needed"| Planner
    style User fill:#800020,color:#fff,stroke:#800020
    style Planner fill:#2d6a4f,color:#fff,stroke:#2d6a4f
    style Response fill:#2d6a4f,color:#fff,stroke:#2d6a4f

When I ask Cortex a question, the planner figures out which servers to query, calls the right tools, and synthesises the results. "Who was I supposed to meet this week?" hits Calendar and iMessage. "What did someone send me about the project?" searches WhatsApp and Gmail simultaneously, because it knows that one person maps to different identifiers on different platforms — a phone number here, an email address there.

That last part — cross-platform identity resolution — turned out to be one of the most fascinating problems in the whole project. More on that in a later post.

Building it with Claude Code

Yes, this is another blog post where someone who can't code properly tells you about the thing they built with AI. I'm a biomedical engineer by training. I can hack things together in Python. I am not, by any reasonable definition, a software engineer.

Cortex was my first proper project with Claude Code. And I started naively. Very naively. I didn't understand what plan mode was. I'd dump vague requests into the terminal and wonder why the output was a mess.

But session by session, I started picking up the things you're meant to pick up as a power user. I discovered CLAUDE.md and how it lets you give the agent persistent context about your project. I found Sequential Thinking for breaking down complex problems. I started using Context7 for pulling in up-to-date documentation. I wired up MCP servers for my GitHub repo, my Notion backlog, and my build system. I wrote up bugs and feature ideas in Notion — properly scoped, with context and acceptance criteria — then pointed Claude Code at them one by one.

Each session got more efficient. Each output got better. Not because the AI suddenly improved — but because I was getting better at working with it. For someone who thinks in systems and architecture but doesn't dream in Python — that's transformative.

Why I didn't just use OpenClaw

This whole thing would have been a lot simpler if I'd just used OpenClaw. Wire it up, point it at my data, done.

But I was terrified. Genuinely. This system would have access to my iMessages, my emails, my photos — the most private data I have. Handing all of that to a black box I didn't fully understand wasn't something I could talk myself into, no matter how good the documentation was. I didn't fancy waking up one morning to find my Google account banned because some dependency was doing something unexpected with my credentials.

And honestly? I wanted the challenge of building something from scratch. I wanted to design my own planner — the bit that decides which MCP servers to call, whether to run them in parallel, when to do a second round of tool calls. I wanted to design the retrieval architecture for each data source individually, because they're fundamentally different. Emails and calendar events are structured — you can query them precisely. Messaging data is messy, conversational, full of context you only understand if you read the surrounding messages.

For iMessage and WhatsApp, I ended up building a hybrid retrieval system that combines embedding-based semantic search with keyword matching, improved ranking with relevancy boosting, and played around with concepts I'd never heard of before — like HyDE query expansion, where the system generates a hypothetical answer to your question first and then uses that to search. Turns out it dramatically improves recall for vague queries like "what was that restaurant someone mentioned last month?"

Could I have got something working faster with OpenClaw? Almost certainly. But I would have missed the chance to experiment with what actually moves the needle on accuracy — and I wouldn't have understood where my data was going at every step.

What it's actually like to use

The thing I didn't expect is how quickly you stop thinking about where information lives. Whether it was an email, an iMessage, a WhatsApp — you just ask the question and Cortex figures out where to look.

I asked it recently when I was last in the UK. It found emails with flight booking confirmations. It also checked my Apple Photos metadata and identified when I last took a photo with UK location data. Two completely different sources, same answer, cross-referenced.

But the one that really sold me was more mundane. I got a letter from an insurance company I didn't recognise, asking for a renewal. I tried searching my email. Nothing obvious. Tried thinking back to when I might have signed up. Couldn't place it. So I asked Cortex. It found a confirmation email from over a year ago buried in a thread I'd completely forgotten about. Turned out it wasn't spam — it was a legitimate policy I'd actually used. That's the kind of thing that's hard to demo in a flashy way, but it's genuinely useful in a way that most AI features aren't.

All of this works well with frontier models. But one area I'm keen to explore is whether you can get comparable results with local inference — running everything on-device without any API calls. Models like Qwen 3.5 and the recently released Gemma 4 are getting remarkably capable, and the idea of a truly private personal AI that never phones home is compelling. That might end up being its own post.

The whole thing — from idea to working prototype — took hours, not weeks. That's the Claude Code part of the story. The models are ready. The protocol exists. The data is sitting right there on your machine. You just need to connect it all up.