Hermes: My AI
Chief of Staff
Most of what a chief of staff does is remember things and connect them. So I built one. It runs on a second brain: a single memory that reads my meetings, my email, and my voice memos, then wires them into one connected graph while I sleep.
I talk to it through Howard, a bot in my pocket. He talks back every morning with a brief and a short list of things that need me. The result is an agent that remembers more about me than I do. Here is how the whole thing is wired.
Meet Howard
Meet My AI Chief of Staff: A Live Demo
The full hour, recorded live: the second brain, Howard answering audience questions, and the 12 hours a week it gives back.
Build it with me, in Tahoe.
This system is one of the two builds at the Tahoe Build Retreat, a small hands-on weekend I host on the lake. Eight people, two build days, a hike up Mt. Tallac and a private boat on Emerald Bay. You leave with your own AI chief of staff wired to your own life, a wealth tracker app, and the Mac Mini it all runs on.
The Shape Of It
Two halves of one system.
A chief of staff is memory plus an interface. Mine splits cleanly into the same two parts.
The second brain (the memory)
A single markdown vault that holds everything: people, companies, deals, and the ideas that connect them. Plain text files, so it outlives any app I use to read them. This is the durable memory every agent reads from and writes back to.
Howard (the interface)
A bot in Telegram. I text or voice-memo it all day, and it texts back: a morning brief, the short list of things that need me, and follow-up nudges. It is the front door to the memory, the part I actually touch.
Sources In
I don't type my life into a wiki.
I tried that. It lasts about 4 days. Instead, the things I already do flow in on their own and get distilled into linked notes, every day. The raw source stays where it is; the vault keeps the extract plus a link. And it isn't just what happens from here forward — it has ingested the back catalog too.
Meetings
Every call transcript gets ingested. The people and companies in it become entity files, and the things I said worth keeping get distilled into concepts, so any agent in the system can understand how I think.
Email, live
Howard reads all of my email. He decides which threads are worth keeping in the vault and which ones need a follow-up, and surfaces both. I don't triage an inbox anymore.
Voice memos
I talk to Howard on the go. The thought lands verbatim and becomes a filed, linked note overnight, no app, no tagging.
Email, back to 1999
It has read every email I've sent since 1999. Out of that archive came timelines, decisions, pivots, and ways of thinking — twenty-seven years of me, distilled into artifacts.
Google Drive
Same treatment for everything I've ever written or presented: decks, docs, essays. The raw file stays in Drive; the vault keeps the distilled extract and a link back.
My CRM
Years of contact notes from Dex, my personal CRM, ingested into markdown. It's a living CRM now, wired into the same graph as the meetings and the email.
One rule I care about most: distill everyone else, capture me verbatim.
Everyone else in a meeting gets summarized. When I say something worth keeping, in a memo, a meeting, or an email, it gets saved word for word as a “musing.” Those nuggets get distilled into artifacts that become how Howard thinks. Over time he sounds less like a tool and more like me. Compression would kill that signal, so on my own words it is forbidden.
One Brain, Many Agents
The vault doesn't belong to Howard.
The second brain is the universal memory. Any agent can pick it up cold, read the same files, and show full comprehension of my life and my businesses. Howard was just the first hire.
Howard
Chief of staffRuns my day: the morning brief, the follow-ups, the errands, the memory. He's the front door to everything on this page, and the agent I actually talk to.
Bob
CEO, software businessRuns my software company. He and his employees live in one wing of the second brain — their deep work, decisions, and daily ops all read and write the same files.
Riker
CMOMines my recent call transcripts for concepts worth publishing and turns them into blog and social drafts. I say something once in a meeting; he does the rest.
In six months or a year, someone will probably turn this into a product. For now it feels like living in the future.
More Than Memory
Howard doesn't just remember. He runs things.
The memory is the foundation, but the point was always to act on it. So I kept handing Howard more of my life, and he kept earning the next piece.
Access to nearly everything
Howard sees my wealth, my relationships, my work, virtually every part of my life. That breadth is the point: a chief of staff who only knows half of you can only help with half of you.
Meals and groceries
Howard plans all my meals and orders the groceries through Instacart every Monday and Wednesday. The family has started to love the new recipes, which was not the part I expected.
Becoming more like me
The verbatim nuggets from my memos, meetings, and email get distilled into artifacts that shape how Howard thinks. The longer this runs, the more his judgment sounds like mine.
The result: he remembers more about me than I do.
Last week I was on a call and blanked on a name from a conversation five months earlier. One text to Howard and I had the answer before the moment passed. That's the payoff of the graph: wikilinks connect the notes, and a semantic retrieval layer called QMD lets any agent search by meaning, not just keywords.
The Second Brain, Working While I Sleep
Detail gets compressed upward.
The trick to long-horizon memory is that no layer reads the raw firehose. A chain of scheduled routines compresses detail up, one level at a time, so a year from now I can still find the thread without drowning in transcripts.
Daily consolidation
Every nightFolds the day's loose breadcrumbs into one clean note, links the people and ideas it touched, and refreshes the 500-word hot cache every agent reads first.
Weekly review
MondaysReads the week's daily notes, not the raw chatter, and pulls recurring patterns up into the durable layer.
Monthly review
1st of the monthReads the four weekly reviews and writes the month. Each layer reads the one below it, never the firehose underneath.
Connections pass
SundaysReads the recent graph, wires high-confidence links between notes that should connect, and promotes themes that keep recurring into their own concept nodes.
The goal was never a tidy archive. It is a second brain whose thoughts are connected and navigable like neurons firing: sources arrive, get distilled, and the weekly connection pass keeps wiring the graph so old ideas resurface against new ones.
The Laundry List
What we actually use it for.
The pattern: anything repeatable goes on a schedule, and anything I do twice becomes a skill he keeps. Ask for a trip and he already knows how I book flights. Together, it's the job description of a chief of staff — organized by the parts of life he touches.
Productivity
Relationships
Wealth
Health
Household
Travel
Content
The Record
Go Deeper
Want the longer story?
I wrote up how a folder of text files and a bot named Howard became the memory I never had time to keep.