Your first LLM thing

About half a dayNo experience needed

Wire a microcontroller kit to a language model, so a physical object can be given a character and answer for itself.

Requirements

  • Connected Interaction Kit
  • OpenAI API key

This workshop session is based on a research-through-design project called Prompting Realities. Run by Mahan Mehrvarz for the Speculative Design Studio course, MSc Design for Interaction, TU Delft Industrial Design Engineering, 2025.

Before you start

You need your own MQTT broker before you start

MQTT is a small publish/subscribe protocol for machine-to-machine messages: a device publishes a short message to a named topic, a broker passes it on, and anything subscribed to that topic receives it. Everything here goes through one broker.

The broker used on the day belonged to the faculty. Its dashboard is still visible in the screenshots, but its address and credentials are not published and you will not be able to connect to it. Set up your own: sign up at shiftr.io and create a broker — its page gives you the host, a username and a password. Those are the three values written below as <your-namespace>.cloud.shiftr.io, <your broker username> and <your broker password>. Anything else written as <like this> is likewise yours to fill in, including the Wi-Fi the device joins.

Any MQTT broker will do — shiftr.io is used here because its live dashboard draws the traffic as it happens, which makes debugging a silent device much easier. Its documentation covers the rest.

The Control Hub dashboard, with an LLM thing selected and its MQTT routing showing the broker host and topic
A

Connecting the boards to the internet

You can find a lot of useful code and information about your kit on their website. We are following the preparation tutorial from the original Connected Interaction Kit.

First let's prepare the setup

Connect your board via the USB cable to your computer.

The Connected Interaction Kit board connected to a laptop over USB

Installing the proper interface to write and edit code on your board

Before you write your first program, you need to complete one last step:

  1. Visit codewith.mu and click the green Download button.
  2. Select the correct link for your operating system (Windows or Mac OS).
  3. Once the download is complete, launch the installer and follow the instructions on the screen.
  4. On its first run, Mu will ask you to select a mode. Choose CircuitPython.
The Mu editor mode selection dialog with CircuitPython chosen

After you are done, you should see the Mu editor up and running on your laptop.

The Mu editor open and ready on the desktop
B

Quick test

1

Connect the chainable NeoPixel LED to the pin (D13) of your board.

The chainable NeoPixel LED wired to pin D13 with a grove cable

Make sure you connect the grove cable to the in side.

For more information about the RGB LED, see this link.

2

Open the code.py file and replace its content with the code below. Don't forget to save.

code.py
# --- Imports
import time
import json
import board
import neopixel
from MQTT import Create_MQTT
from settings import settings

# --- Variables
# Configure the pin here (any NeoPixel-capable pin)
LED_PIN = board.D13
NUM_LEDS = 1

led = neopixel.NeoPixel(
    LED_PIN,
    NUM_LEDS,
    auto_write=False,
    pixel_order=neopixel.GRBW,
)

# Current LED state: "[R, G, B, Brightness]"
led_state = [255, 255, 255, 200]

# MQTT setup
client_id = settings["mqtt_clientid"]
mqtt_topic = settings.get("mqtt_topic")
mqtt_client = Create_MQTT(client_id)


# --- Functions
def apply_single_led(led_object, values):
    """Apply [R, G, B, Brightness] to a single NeoPixel object."""
    if not isinstance(values, list) or len(values) != 4:
        return

    r, g, b, brightness = values

    # Brightness 0-255 -> 0.0-1.0
    led_object.brightness = max(0.0, min(1.0, brightness / 255.0))

    # With pixel_order=neopixel.GRBW, we still pass (R, G, B, W)
    led_object.fill((r, g, b, 0))
    led_object.show()


def apply_led():
    """Apply current state to the single LED."""
    apply_single_led(led, led_state)


def on_message(client, topic, message):
    """Handle incoming MQTT messages to update the LED color."""
    global led_state
    try:
        data = json.loads(message)

        # Expect messages like: {"led": [R, G, B, Brightness]}
        if "led" in data and isinstance(data["led"], list) and len(data["led"]) == 4:
            led_state = data["led"]
            apply_led()
            print("Updated led:", led_state)
        else:
            print("Received invalid LED payload:", data)
    except Exception as e:
        print("Error parsing MQTT message:", e)


# --- Setup
# Clear LED first
led.fill((0, 0, 0, 0))
led.show()

# Apply initial state
apply_led()

# Configure MQTT callbacks and subscription
mqtt_client.on_message = on_message
mqtt_client.subscribe(mqtt_topic)
print("Subscribed to topic:", mqtt_topic)


# --- Main loop
while True:
    mqtt_client.loop(timeout=0.2)
    time.sleep(0.1)
3

Connect your board to the laptop and load the settings.py file. Replace <Your device name> with your actual group name and make sure the rest looks like below. You don't need the < > signs there!

settings.py
settings ={
"ssid": "<your Wi-Fi network>",
"password": "<your Wi-Fi password>",
"mqtt_clientid": "<Your device name>",
"broker": "<your-namespace>.cloud.shiftr.io",
"mqtt_user": "<your broker username>",
"mqtt_password": "<your broker password>",
"mqtt_port": 1883,
"mqtt_topic": "RGBLight"
}
4

You should see this in the serial monitor of the Mu editor:

The serial monitor output confirming the board subscribed to its MQTT topic

Click here:

The serial button in the Mu editor toolbar
This will show up at the bottom of the Mu editor
5

Follow up with the instructions in the class.

C

Getting up and running with promptingrealities.com

The Prompting Realities sign-up screen
1

You need to sign up and make an account. A magic link is sent to your email address and by clicking on the link (also check your spam folder) you'll be logged in.

The left side panel of the Control Hub with the add-new-LLM-thing control
2

From the left side panel, add a new LLM thing.

3

A new LLM thing is added and now you can configure it based on the first example. The first step is to give it a name. In this example we call it Neopixel.

The configuration studio with the new LLM thing named Neopixel

The main part of the interface is the configuration studio. In order to create an LLM thing you need to add information to the four tabs:

  1. Prompt Instruction
  2. JSON Schema
  3. MQTT
  4. OpenAI API Key

A quick way would be to use one of the examples from the Prompting Realities GitHub repository. Let's do one single RGB LED.

D

Creating your first LLM thing

1

Use the text below in your Prompt Instruction text field.

Prompt Instruction
You are sending values for a RGB LED to change its color based on the command or prompt from the user.
You send three values with each response.
R: for red
G: for green
B: for blue
X: the last value is the brightness
-Here is an example of the values inside the values object
{
  "led": [255, 100, 50, 200]
}

Ds:
-You cannot really do beyond this color change. If the user asked for other things be transparent.
-If user says something not about the LED. You can answer but don't change the color.

Do Not Do:
-in the response object do not include any technical term about the RGB values. Just refer to colors you created and engage the users in conversation.
-Don't be wordy keep it very short.
2

Use the JSON Schema below.

JSON Schema
{
  "name": "led",
  "schema": {
    "type": "object",
    "required": [
      "answer",
      "MQTT_value"
    ],
    "properties": {
      "answer": {
        "type": "string"
      },
      "MQTT_value": {
        "type": "object",
        "required": [
          "led"
        ],
        "properties": {
          "led": {
            "type": "array",
            "items": {
              "type": "number"
            },
            "maxItems": 4,
            "minItems": 4
          }
        },
        "additionalProperties": false
      }
    },
    "additionalProperties": false
  },
  "strict": true
}

The JSON schema defines exactly one acceptable shape of data for something called led. Its job is to:

  1. say what fields must exist
  2. say what type each field must be
  3. forbid anything extra
  4. enforce exact sizes and structures

This schema enforces the following:

  1. a human-readable message (answer)
  2. a machine-readable LED command (MQTT_value.led)
  3. the LED command is always four numbers
  4. nothing else is tolerated
3

In the MQTT tab you have the following fields to fill in: Broker Host address, Username, port, password, Topic.

The MQTT tab of the configuration studio with broker host, username, port, password and topic fields

Broker, Username, Port and Password are often the same as in your settings.py file (check Part B, step 3).

MQTT tab
broker: <your-namespace>.cloud.shiftr.io
username: <your broker username>
port: 1883
password: <your broker password>

But what about the topic? The topic is something shared between your LLM thing and your hardware (CircuitPython board). Both should subscribe to the same topic. This prevents your messages from being sent to other boards and is key for communication between the LLM thing and your board. Just pick something simple — myled, mahan-led, light1, coloredlight.

4

The last step is to add a valid OpenAI API key. You will be given one if you are part of a workshop, but in general any OpenAI API key works.

5

Press save!

Now, as you see, your Run LLM thing button in the top right corner is green and you can go ahead and run it.

The configuration studio with the Run LLM thing button lit green

Below the configuration panel you'll see the client access panel. You can open the chat interface, copy the link and paste it somewhere, or scan the QR code to go straight to the chat interface.

The client access panel showing the chat link and a QR code

You can start to chat with your RGB LED now. Enjoy!

The NeoPixel glowing blue next to the chat, answering a question about the colour of the sky
E

Exploring other actuators in the Kit

You can explore other examples from the GitHub folder. You can chat with a servo, a buzzer or a vibration motor from your kit. The preferred order is: two LEDs, vibration motor, servo motor, buzzer.

Resources