A Note from Nikhil and Jon:
Hello! We appreciate the engagement and positive vibes after our first newsletter last month, and now we’re back on the 5th of this month for another one.
We’ll kick it off with a demo of an idea that evolved when we were in Nikhil’s studio whiteboarding something while transcribing our conversation. We began imagining how much richer transcription might become. It made us wonder…
An Evolving Conversation
Doesn’t it feel like more gets lost in translation these days? Like — despite the flurry of messages we’re all constantly sending and receiving, the chance of being misread, misunderstood, or misinterpreted seems to be higher than ever.
At the same time, more of our conversations are being mediated by AI via text. Our meetings produce transcripts; those transcripts get interpreted by AIs and people alike. The emotions we embed into our voice and bodies get lost in that transcription process. The evolution of our ideas throughout the flow of conversations are hidden in plain sight; not always picked up in conversation summaries, but possible to glean with closer inspection of who said what, when, and why.
We had fun last newsletter building a little demo based on a thought circling our heads, and decided we wanted to build another demo. This time it’s an exploration of how we might add more context into our interactions with AI and each other.
First, just try the demo. Hopefully the experience is clear enough as-is. The end is where the magic is, and where you may want to come back here for a little explainer. Bear with us if a couple of the interpretation steps are a little slow — it’ll work.
OK— go!
Let’s Go Knicks
What you should have experienced is something like this:
You draw a character
You click guess and the AI guesses what you’re drawing, to varying levels of success
You draw some more, build out a scene
The AI guesses again
Click “narrate”, and the AI tells you how the drawing session developed, not just a summary
“Redraw from transcript” will reconstruct what you drew — we’ll explain how below
Handoff to agent allows you to explore the transcript yourself with your own agent outside of this demo
As you draw, every stroke on the canvas is being logged as if this were a conversation being transcribed. At the end, you’ve created a drawing transcript, with geometry data, a send of sequencing, and other descriptive data associated with your canvas. You can check those at any time by unfurling the “show events” button on the side panel.
We’re interested in what happens when you consider events over time rather than just snapshots. How do ideas develop? How do ideas change with one stroke of the pen, or one pivot in a conversation?
What we found most interesting in this demo is that we could replay the whole drawing, just from data. Sometimes the replays are quite accurate. Sometimes they’re pretty bad. And, depending on the model and prompt, sometimes the model re-interprets the data itself and does things like add color:
Now, imagine applying the same kind of processing to a meeting. Or even a meeting with a whiteboarding session. Not only would you have the normal record of what happened, you might start to get a map of how the conversation unfolded.
Of course, you’ll also have some variability of outputs introduced by a stochastic-feeling agent. If you’re building products, the harnesses that mediate between human context and LLMs is the most interesting space for constraining variability. If you’re an artist, the variability itself and the behavior it induces in us and other agents is a fascinating place to explore.
We’ve begun prototyping a couple other ideas along these lines:
When might a (speech-based) conversation reach a critical point where the discussion flows one way, instead of another. Could we model ways that the conversation might have gone? Can we create artifacts that are branching and non-linear instead of the flattened transcripts we get now?
How could we pack more information into the literal English characters in a transcript to convey emotional information? Ťħę șčřįpt míğhť löòk liķe thiš and each accent and underscore might map to emotional inflections inferred from the real voice or body language someone exhibits.
Stay tuned for some demos and documentation of these experiments. Sign up for our Substack, where we’ll share thoughts and demos on a different cadence than we have with this monthly newsletter.
Oh also! Send us your agent handoff files — we want to see if we can reconstruct your drawings just from data. Email us at: nikhil@5×5.studio and jon@5×5.studio
Come try palm wine! 6/11 @ Fontainhas Dumbo
Have you had palm wine before?
If you’re lucky enough to have sampled some in West Africa, South India, or one the many parts of the world that produce this refreshing drink from the sap of palm trees, you’ll be glad to hear it’s starting to be might right here in the US of A.
Ikenga Wines founder, Onye Ahanotu, will be visiting NYC on June 11th to share how he makes his palm wine out in Berkeley, CA. Onye has a unique background in biochemical engineering, and brings his intimate knowledge of lab-level fermentation to produce palm wines inspired by the West African palm wines he first saw on his occasional family trips to Nigeria.
We’ll be hosting the event at another 5×5 favorite — Fontainhas in Dumbo, BK. Fontainhas is named after a famous town quarter in Goa, India — which, not coincidentally, has a long tradition of brewing its own palm wines. RSVP here, and bring your friends who loving trying new food and drinks!
Cool Stuff From the 5×5 Community
Artist Sally Kong made a short film shot on Super 8 film about a robot she built. She already showed the film at MoMa — and now you can watch it / read about it too.
Biodesigner Will Eliot is featured at New Inc’s DEMO Showcase at the New Museum. Go check out his work centered around multi-species collaboration; think meal worms meet furniture design. After you see Will’s work, stay for other mind-expanding pieces in the realm of creative science, social architecture, and more.
Photographer Mateo Wilches’ photography is on display at Salmagundi NY in their Bold Strokes show. Today, June 5th, is the last day to view.
Storyteller Lindsey Lerner is dropping a new Field Note today. If you haven’t read or listened to any of her field notes yet, you’re in for a treat — it’s like Humans of NY meets spoken word, about creative processes. You’re finding out who this Field Note is about at the same time that we are 👀
If you liked Lindsey Lerner’s Field Note above, check out the workshop she’s giving on her approach to storytelling https://luma.com/szxw3fnr
The annual Biodesign Challenge Summit is being held next week on June 11-12. If you want to see cutting-edge, inspiring ideas that imagine biology as a force for moving beyond industrial processes, this is for you.
A long-form piece for the road: We consider this talk to be a classic. “How I built a toaster — from scratch” walks through designer Thomas Thwaites’ sisyphean feat of sourcing and building every single component of a toaster from scratch — including mining the metals and building the electrical circuits. It’s a fascinating and funny look at just how interdependent we are on each other. Give it a watch when you have 10 minutes.
Till next time,
Nikhil & Jon