Prompting Realities is a low-threshold framework for building LLM-powered tangible interactions.
Describe your object to a language model in plain words. Talk to it. The model answers you in natural language and, in the same breath, emits structured JSON that drives your motors, lights and servos. No training, no fine-tuning, no custom parser — you write a description and a schema, and the thing starts behaving.
You write a plain-language description of your artifact — what it is, what it can do, what its variables mean physically. That description is the whole “programming” step. The model learns the correspondence between a number in a JSON field and what it does in the room.
You talk to the artifact through a chat interface — a Telegram bot, the Control Hub, a voice loop. The conversation thread doubles as memory: rules, personalities and configurations you set earlier stay in play without being re-sent.
Every reply carries two outputs: language for you, and a JSON payload for the hardware. The payload travels over MQTT to a microcontroller, which executes it. That loop — prompt, structured response, actuation — is the whole framework.
What you need: an OpenAI-compatible API key, an MQTT broker (local or hosted), a microcontroller running CircuitPython or Arduino, and a Control Hub account. What you don't need: machine-learning experience, a fine-tuned model, or any code that parses natural language.
Mahan Mehrvarz
Three motorized windmills on a topographic model, driven from a Telegram chatbot. Conversational commands become motor speeds and directions, sent as JSON over MQTT. This is the reference implementation — the one documented in the GitHub repo.
Three prototypes on identical Pimoroni Pan-Tilt HAT hardware. Same two servos every time; everything that differs is in the description and the schema.
From the thesis Animistic interactions in conversation with LLM-powered robots — K. Hoefnagel, MSc Industrial Design Engineering, TU Delft, 2025 →
Kenwyn Hoefnagel
Adds a PiCamera with OpenCV object detection. The servos track what they see in a tight Python loop while the model narrates what it thinks the thing is doing.
Kenwyn Hoefnagel
Extends the schema with dance_pattern arrays, making the model a choreographer: give it a genre or a mood and it composes the movement.
Kenwyn Hoefnagel
Accepts photos over Telegram and uses a vision model to recognise itself in them, then composes a response from enumerated libraries of movement styles, patterns and combinations.
Two narrative prototypes that push on what counts as output.
From the thesis Prototyping AI-enabled tangible storytelling — T. Pan, MSc Design for Interaction, TU Delft, 2025 →
Tianrui Pan
Three coloured cubes enact a noir narrative. You move two of them; the system moves the red one on a repurposed 3D-printer pulley, driven by what image recognition makes of the scene you just built.
Tianrui Pan
The consolidated system: a wooden table with an embedded monitor, ArUco-marked game pieces, an overhead webcam and voice in/out. Two AI threads — a Story AI and a Supervisor AI — run in parallel, tracking four separate memory registers.
The Control Hub is where your things live. Register an artifact, write and revise its description, choose a model, define the JSON schema it should emit, and watch the conversation and the resulting payloads side by side as it runs. Sign-up is open — bring your own API key and a board, and you can have something moving in an afternoon.
Create an account →Start-to-finish guides for building an LLM thing — from a bare board to an object you can hold a conversation with. The first one takes you through a lamp: wire the LED, write its description and schema, and ask it for a colour.
Wire a microcontroller kit to a language model, so a physical object can be given a character and answer for itself.
Start the tutorial →Prompting Realities has been run as a hands-on workshop with design students and researchers: participants arrive with an everyday object and leave with a version of it they can talk to. If you'd like to run one with your group, get in touch.
Request a workshop →