Building a Smarter Home
Organizing our home's information and making it accessible
It started with a lock or a light switch in 2016, and within a few months, I could turn off the lights, lock the doors, change the colors of the lights, operate the speakers all from my phone, and I could even talk to the speakers. The home, like the phone, was now smart. I still couldn’t find my keys half the time, but if I forgot to turn off the lights before heading out, that problem was solved.
Fast forward ten years, and at the start of 2026, our home had stopped feeling smart. The problems decomposed broadly into two buckets. First, performance felt like it had degraded, where asking for our lights to turn off or on could result in delays of up to 30 seconds to a minute. Part of this can be explained by the fact that there are a variety of network delays associated with those requests, ultimately introducing a latency that defeats the purpose of just using the physical switch.
Second, my expectations for what a smart home should be had increased. In a world in which AI is taking off, a smart home to me is one that can answer arbitrary questions about itself, or better yet, one in which a robot could come in and help out. Do I need to pick up dishwasher detergent on the way home? Where did I leave that book by Michael Pollan? Or where did I leave the keys? That would require distilling and understanding properties about our home, as well as their spatial relationships with each other for a robot helper.
Part of the reason my expectations have changed is because of the pace of technology. Even with a good understanding of deep learning and computer vision, building something like this in 2016 would have involved staffing up a medium-sized team to build a prototype, and hoping that interim results might buy time to launch and iterate towards a longer term roadmap. In 2026, with AI coding agents and significantly more powerful models, I felt confident that I could take this on on my own.
Table Stakes - Smart, Faster Lighting
The first task was to get to parity with what I already had with our existing home automation app, but making it more responsive. This was surprisingly simple with Claude Code. The first version of it was ready in 30 minutes, with millisecond latency from the app within an hour.
While next steps here could involve incorporating locks, the garage door, the dishwasher, and the washer/dryer, I felt confident enough that we could get there that I switch to the next task: creating an object-focused map of our home
An object-focused map of our home
In a 1945 article in The Atlantic, Vannevar Bush’s “As We May Think” imagined a memory extension system for humans, and while most people looking back on that have described it this as foreseeing the world wide web, the world wide web can’t tell me where I left our copy of Michael Pollan’s book.
However, I was able to design a system that could do that for our home, along with answering other relevant questions.
What about the robots?
Currently, we have a Roomba on each floor of our home that sometimes crash into walls and furniture, so I’ve started thinking about what might happen if a more sophisticated robot were to present itself in our home. It could query the system above, but how would we provide it directions? To address this, I’ve started working through building a smarter map of our home. This is still a work in progress, and currently the maps are just from a handful of photos.
Because of our DGX Sparks, I’ve been able to run many models locally, paying for modest electricity costs, and without requiring special cooling systems.
Closing thoughts
A lot of this really comes down to questions about how to design and organize the information, to make sense of the physical world to simplify our lives. Part of this could come from being better organized in the physical world, but in the absence of that well-organized system, everything is in its place if you have a system for finding it.
On that note, as I was writing this post, Andrey Karpathy posted about an LLM Wiki system for building a personal knowledge base, and if I squint, it’s touching on related ideas of how we organize knowledge and information for AI.
Have we reduced many of the problems in AI to building better search engines?