editor/packages/mcp/examples/renovate-from-photos.md

Renovate an existing flat from photos

This example shows how an agent can combine the renovation_from_photos prompt with the analyze_floorplan_image and analyze_room_photo vision tools to propose a renovation plan grounded in real photos.

Note: the vision tools use MCP sampling (createMessage), which Claude Desktop supports today. Hosts without sampling support will get a structured sampling_unavailable error; fall back to the text-only from_brief prompt in that case.

The brief

The user drops four photos into the chat:

1. A floorplan PDF page (exported as PNG).
2. A photo of the current living room.
3. A photo of the current kitchen.
4. An inspirational photo from a magazine — a minimal Scandinavian loft.

And types:

User: Claude, help me plan a renovation. Here's the current plan and two room photos. I want something like this Scandinavian reference — open-plan, neutral tones, keep the footprint.

What the agent does

The host loads the renovation_from_photos prompt:

currentPhotos: ["data:image/png;base64,...", "data:image/jpeg;base64,..."]
referencePhotos: ["data:image/jpeg;base64,..."]
goals: "Open-plan living/kitchen, neutral tones, keep the footprint."

The prompt tells the agent to (1) analyze the floorplan, (2) analyze each room photo, (3) seed a scene from the floorplan, (4) compare against the reference, and (5) propose patches.

1. Extract the floorplan

// tool: analyze_floorplan_image
{
  "name": "analyze_floorplan_image",
  "arguments": {
    "image": "data:image/png;base64,iVBORw0KGgoAAAANS...",
    "scaleHint": "1 m grid, total footprint ~9.5 m × 7 m"
  }
}

Under the hood, the tool issues an MCP sampling request to the host with the image and a structured prompt asking for walls, rooms, and approximate dimensions. The response is validated against the tool's output schema:

{
  "walls": [
    { "start": [0, 0], "end": [9.5, 0], "thickness": 0.25 },
    { "start": [9.5, 0], "end": [9.5, 7], "thickness": 0.25 },
    { "start": [9.5, 7], "end": [0, 7], "thickness": 0.25 },
    { "start": [0, 7], "end": [0, 0], "thickness": 0.25 },
    { "start": [4.5, 0], "end": [4.5, 7], "thickness": 0.15 },
    { "start": [4.5, 3.5], "end": [9.5, 3.5], "thickness": 0.15 }
  ],
  "rooms": [
    { "label": "Living",  "polygon": [[0, 0], [4.5, 0], [4.5, 7], [0, 7]] },
    { "label": "Kitchen", "polygon": [[4.5, 0], [9.5, 0], [9.5, 3.5], [4.5, 3.5]] },
    { "label": "Bedroom", "polygon": [[4.5, 3.5], [9.5, 3.5], [9.5, 7], [4.5, 7]] }
  ],
  "approximateDimensions": { "widthMeters": 9.5, "depthMeters": 7, "areaSqMeters": 66.5 },
  "confidence": 0.82
}

2. Analyze the room photos

// tool: analyze_room_photo
{
  "name": "analyze_room_photo",
  "arguments": { "image": "data:image/jpeg;base64,/9j/4AAQ..." }
}

Response:

{
  "approximateDimensions": { "widthMeters": 4.4, "depthMeters": 5.8, "heightMeters": 2.5 },
  "identifiedFixtures": [
    { "kind": "sofa",         "approximatePosition": [2.2, 3.5] },
    { "kind": "coffee-table", "approximatePosition": [2.2, 2.4] },
    { "kind": "tv-unit",      "approximatePosition": [0.3, 2.0] }
  ],
  "identifiedWindows": [
    { "wallHint": "south", "approximateWidth": 1.4, "approximateHeight": 1.5 }
  ]
}

The kitchen photo is analyzed the same way.

3. Seed the scene

The agent reads get_scene, confirms the default empty Site → Building → Level is present, and then batch-creates walls matching the floorplan:

// tool: apply_patch
{
  "name": "apply_patch",
  "arguments": {
    "patches": [
      { "op": "create", "parentId": "level-1",
        "node": { "type": "wall", "start": [0, 0], "end": [9.5, 0],
                  "thickness": 0.25, "height": 2.5 } },
      /* ...remaining perimeter + partition walls from the vision result... */
    ]
  }
}

The agent then calls set_zone three times to seed the Living / Kitchen / Bedroom polygons from the floorplan rooms.

4. Cut the identified openings

For each window the vision tool reported, the agent calls cut_opening against the corresponding perimeter wall:

{
  "name": "cut_opening",
  "arguments": {
    "wallId": "wall-south",
    "type": "window",
    "position": 0.5,
    "width": 1.4,
    "height": 1.5
  }
}

5. Propose the renovation

Guided by the reference photo's analysis (bright neutrals, open plan, minimal furnishing), the agent proposes a single logical patch:

• Remove the partition wall between Living and Kitchen.
• Relocate the kitchen island further west.
• Delete the bulky TV unit item; leave the sofa and coffee table.
• Re-label the merged zone "Open-Plan Living / Kitchen".

All of that goes into one apply_patch:

{
  "name": "apply_patch",
  "arguments": {
    "patches": [
      { "op": "delete", "id": "wall-partition-living-kitchen", "cascade": false },
      { "op": "update", "id": "zone-living", "data": { "label": "Open-Plan Living / Kitchen",
                                                         "polygon": [[0, 0], [9.5, 0],
                                                                     [9.5, 3.5], [0, 3.5]] } },
      { "op": "delete", "id": "zone-kitchen", "cascade": false }
      /* + item moves / deletes for the TV unit etc. */
    ]
  }
}

The user can walk back with undo; redo returns them to the proposal.

6. Sanity-check

// tool: validate_scene
{ "name": "validate_scene", "arguments": {} }
// → { "valid": true, "errors": [] }

// tool: check_collisions
{ "name": "check_collisions", "arguments": { "levelId": "level-1" } }
// → { "collisions": [] }

The agent reports a summary of the changes plus the approximate new usable area (from the summary resource), and the user opens the scene in @pascal-app/viewer to see the renovated 3D layout.

Takeaways

• The vision tools only return data. They don't mutate the scene — the agent is explicit about every structural change via apply_patch.
• Photos supply priors (approximate dimensions, fixture types) that a brief-only workflow can't. Combine them with from_brief-style prompts when the user has both a reference and concrete text goals.
• All renovation steps are a single temporal step per patch, so the user can compare before/after with undo / redo.