Newsletter 003 || Enter the Rabbithole

Hello from a hot and humid New York! This has been the summer of sports, and NYC is alive as ever. We don’t know about y’all, but our work rhythm has been upended by watch parties, heart stopping basketball, and World Cup nailbiters. And yet, we’re back on the 5th of the month to show you what we’ve been noodling on: a new demo! A call for performances! Photos from our last event! Highlights from the 5×5 community!

Before you keep reading, think of a podcast you like, or an informative video on Youtube.

Got it?

Ok keep reading.

Last month we contemplated the transcript as a more dynamic interface, especially in a world mediated by AI agents. One of our former Coursera colleagues, friends, and readers (!!), Jenny Wolochow, replied to our post and sent over a follow-up idea: wouldn’t it be cool to have a companion to the live transcript that identified references that you, the listener, didn’t already know? And wouldn’t it be even cooler if it then automagically researched them in near-real time? Today’s demo pulls on this thread.

We took Jenny’s prompt alongside our previous explorations into the dynamic transcript and began with the technical approach to getting live transcription data into a shape that could be enhanced in many different ways:

Core idea 1: As text gets transcribed, instead of waiting for the entire transcript to come through before making sense of it, what if we chunk it up and make those chunks available for further enrichment?

Core idea 2: Enrichments can happen in parallel, and each one can have a different expectation of latency. In other words, some enrichments need to happen very quickly; others can happen more slowly. For example, people, places, and things could be identified quickly; but sometimes their meaning changes over the course of the conversation. Our architecture needs to be able to handle this.

We put these ideas together with Jenny’s in our demo for the week, Rabbithole.

Just try it before we explain how it works. Hopefully it’s self-explanatory enough. It works best on desktop using Chrome.

OK — you’ve now listened to one of our favorite podcasts or searched for your own pod or video. What’s going on in the demo?

As you listen, if you gave it enough time to get past sponsorships and other intro preamble stuff, you might have noticed people, places, things, ideas, techniques, and other identifiable concepts starting to stream in as cards. If you clicked on the cards, you might have noticed some research, with the option to dive deeper or link out to web sources.

Listening to Cooking Issues, Dave Arnold’s podcast about food science and the restaurant/cocktail industry, what stood out is how well the “entity extraction” picked up idiosyncratic people and concepts. It knew to identify Quinn, one of the contributors to the podcast who isn’t really famous otherwise, and generated an accurate blurb about him.

If you notice the extraction missed something you wanted to learn more about, you can highlight the transcript to create a card. You can also create a card by clicking the “+” button under the card feed, dismiss cards that aren’t useful, or edit them if they got something wrong in the ID.

Finally, when you pause media, you can generate a summary up to the point where you stopped. If you resume and pause later, you can update your summary and note what changed in between.

We’ll save the deeper, more technical, dive for a future Substack. Instead we’ll leave you with a few next explorations:

Riff 1: What would this look like if operating on a Zoom/Google Meets video call? If the AI was connected to a work knowledge base (think CRM + docs + a map of what matters to your org), would it be able to surface important people, open questions, useful conversation starters based on work context?

Riff 2: How do we modify this existing demo for mobile? Many of us listen to podcasts on-the-go, and won’t be looking at the screen as we listen. How do we modify this design, and feature set, to make real-time features useful when not persistently looking at a screen?

Riff 3: What would a reading-first version of this look like? If you’re reading an academic paper or a long essay, what would it look like to have this as a sort of note-taker companion?

As always, be like Jenny and send us your thoughts, feedback, feature requests, or projects you want to jam on with us at [email protected] and [email protected].

We’re bringing back SHIFT! If you’ve been following Nikhil’s art projects, you might remember that in 2024 he hosted a series of experimental performance shows at Black Brick Project in Brooklyn. They were on hiatus last year, but by popular demand (aka 3 people asking to perform 😛 ) we’re bringing it back!

These are meant to be 10-15 minute experimental sets - something that pushes your own artistry, and challenges audiences. They do not need to be musical, though most sets to-date have been; they do not need to follow any particular media format, in fact.

Send Nikhil a few sentences on what you’d like to perform, or nominate someone he should reach out to at [email protected], subject line: SHIFT.

We hosted Onye Ahanotu, founder and head winemaker at Ikenga Wines, at Fontainhas Dumbo in June. Onye is based in Berkeley, CA, so we were excited to show New York what he’s been brewing.

For many, this was the first time getting a chance to try palm wine, let alone Onye’s unique take. Onye is a biochemist by background, and used his unique skillset to make palm wine through a series of fermentations of a proprietary set of ingredients, rather than by importing palm from West Africa — a process that is both quite environmentally taxing, and difficult to pull off due to palm sap’s tendency to go bad quickly.

Donna Wang took photos of the night, including the one below and the others at this link!

We are dabbling with platforms for our writing. This monthly newsletter is being sent via Beehiiv, for now. Our documentation and free-ranging essays are on Substack and will come at different intervals. We’ll keep an archive of everything on Substack for ease of reference. Maybe we’ll move the newsletter there too.

If you receive an email from Substack when we post something and you don’t want it, no hard feelings if you stop following us there! On the flipside, if you like what we’re doing, please share + encourage your friends to follow along too 🙂

It’s serviceberry season in New York. Perhaps you’ve never heard of this tree or its fruit; and if you have, you likely know it from foraging, not from a grocery store.

Robin Wall Kimmerer, known for her book Braiding Sweetgrass , wrote an essay that asks what the study of economics looks like if you start from abundance instead of scarcity. She uses the serviceberry tree to begin an exploration of ecological economics and gift economies centered on circulation over accumulation and reciprocity over extraction with enough rigor that it reads as a parallel system worth building rather than a full replacement or unachievable utopia.

Till next time,

Nikhil & Jon