From 3406dad8f55388c9a97489749fc7ebd966a0224d Mon Sep 17 00:00:00 2001 From: Adrian Perez Date: Sat, 18 Apr 2026 17:51:37 +0200 Subject: [PATCH] docs(mcp): add README, examples, and changelog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README.md: install/quick start; configs for Claude Desktop, Claude Code, and Cursor; programmatic usage; tables covering all 21 tools, 4 resources, and 3 prompts; limitations; development commands. - CHANGELOG.md: 0.1.0 entry in Keep a Changelog format. - examples/generate-apartment.md: prose transcript using from_brief to build an 80 m² 2-bed apartment, showing apply_patch, set_zone, cut_opening, validate_scene. - examples/renovate-from-photos.md: prose transcript using the vision tools + renovation_from_photos prompt. - examples/embed-in-agent.ts: compilable TypeScript showing programmatic usage via InMemoryTransport. Co-Authored-By: Claude Opus 4.7 (1M context) --- packages/mcp/CHANGELOG.md | 29 +++ packages/mcp/README.md | 205 +++++++++++++++ packages/mcp/examples/embed-in-agent.ts | 82 ++++++ packages/mcp/examples/generate-apartment.md | 242 ++++++++++++++++++ packages/mcp/examples/renovate-from-photos.md | 202 +++++++++++++++ 5 files changed, 760 insertions(+) create mode 100644 packages/mcp/CHANGELOG.md create mode 100644 packages/mcp/README.md create mode 100644 packages/mcp/examples/embed-in-agent.ts create mode 100644 packages/mcp/examples/generate-apartment.md create mode 100644 packages/mcp/examples/renovate-from-photos.md diff --git a/packages/mcp/CHANGELOG.md b/packages/mcp/CHANGELOG.md new file mode 100644 index 00000000..f536a4bb --- /dev/null +++ b/packages/mcp/CHANGELOG.md @@ -0,0 +1,29 @@ +# Changelog + +All notable changes to `@pascal-app/mcp` will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [0.1.0] - 2026-04-18 + +### Added + +- Initial release. +- `SceneBridge` headless adapter for `@pascal-app/core` with RAF polyfill so + the Zustand store and Zundo temporal middleware run cleanly in Node. +- 19 MCP tools covering scene querying (`get_scene`, `get_node`, + `describe_node`, `find_nodes`, `measure`), mutation (`apply_patch`, + `create_level`, `create_wall`, `place_item`, `cut_opening`, `set_zone`, + `duplicate_level`, `delete_node`), undo/redo (`undo`, `redo`), export + (`export_json`, `export_glb`), validation (`validate_scene`, + `check_collisions`), plus 2 vision tools (`analyze_floorplan_image`, + `analyze_room_photo`) backed by MCP sampling. +- 4 MCP resources: `pascal://scene/current`, + `pascal://scene/current/summary`, `pascal://catalog/items`, and + `pascal://constraints/{levelId}`. +- 3 MCP prompts: `from_brief`, `iterate_on_feedback`, and + `renovation_from_photos`. +- stdio and Streamable HTTP transports. +- `pascal-mcp` CLI binary with `--stdio`, `--http --port`, and `--scene` + flags. diff --git a/packages/mcp/README.md b/packages/mcp/README.md new file mode 100644 index 00000000..450c347d --- /dev/null +++ b/packages/mcp/README.md @@ -0,0 +1,205 @@ +# @pascal-app/mcp + +Model Context Protocol server for the Pascal 3D editor. Drives the +`@pascal-app/core` scene graph from any MCP-compatible AI host. + +The server runs headlessly in Node — no browser, no WebGPU, no React — and +exposes the same scene mutations used by the editor UI (create walls, place +items, cut openings, undo, etc.) as MCP tools, resources, and prompts. + +## Install + +```bash +bun add @pascal-app/mcp # or: npm i @pascal-app/mcp +``` + +`@pascal-app/core` is a peer dependency; Bun workspaces resolve it automatically. + +## Quick start + +Launch the server over stdio in one line: + +```bash +bunx pascal-mcp # or: npx pascal-mcp +``` + +Load an initial scene from disk: + +```bash +pascal-mcp --stdio --scene ./my-scene.json +``` + +Expose it as HTTP for remote hosts: + +```bash +pascal-mcp --http --port 8787 +``` + +## Claude Desktop config + +Edit `~/Library/Application Support/Claude/claude_desktop_config.json` +(macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows): + +```json +{ + "mcpServers": { + "pascal": { + "command": "bunx", + "args": ["pascal-mcp"] + } + } +} +``` + +If `bunx` isn't on your PATH, substitute `npx` or point `command` at the +absolute path of the `pascal-mcp` binary inside your project. + +## Claude Code config + +Via the CLI: + +```bash +claude mcp add pascal bunx pascal-mcp +``` + +Or add to `.mcp.json` at the repo root: + +```json +{ + "mcpServers": { + "pascal": { + "command": "bunx", + "args": ["pascal-mcp"] + } + } +} +``` + +## Cursor config + +In Cursor settings (`settings.json`): + +```json +{ + "mcp.servers": { + "pascal": { + "command": "bunx", + "args": ["pascal-mcp"] + } + } +} +``` + +## Programmatic use + +Embed the server in your own Node process using the in-memory transport. The +example below runs a full client/server pair inside a single script — useful +for agent frameworks and tests. + +```ts +import { createPascalMcpServer, SceneBridge } from '@pascal-app/mcp' +import { Client } from '@modelcontextprotocol/sdk/client/index.js' +import { InMemoryTransport } from '@modelcontextprotocol/sdk/inMemory.js' + +const bridge = new SceneBridge() +bridge.loadDefault() +const server = createPascalMcpServer({ bridge }) + +const [srvT, cliT] = InMemoryTransport.createLinkedPair() +const client = new Client({ name: 'my-agent', version: '0.1.0' }) +await Promise.all([server.connect(srvT), client.connect(cliT)]) + +const tools = await client.listTools() +console.log('available tools:', tools.tools.map((t) => t.name)) + +const scene = await client.callTool({ name: 'get_scene', arguments: {} }) +console.log(scene) +``` + +See [`examples/embed-in-agent.ts`](./examples/embed-in-agent.ts) for a +compilable version. + +## Tools + +All tools validate their inputs and outputs with Zod. Mutation tools are +captured by Zundo's temporal middleware as a single undoable step. + +| Name | Purpose | Key input | Output | +| --- | --- | --- | --- | +| `get_scene` | Return the full scene graph. | — | `{ nodes, rootNodeIds, collections }` | +| `get_node` | Fetch a node by id. | `{ id }` | the node, or `InvalidParams` if not found | +| `describe_node` | Node summary with ancestry, children count and properties. | `{ id }` | `{ id, type, parentId, ancestry[], childrenCount, properties, description }` | +| `find_nodes` | Filter nodes by type / parent / zone / level. | `{ type?, parentId?, zoneId?, levelId? }` | `{ nodes: AnyNode[] }` | +| `measure` | Distance between two nodes; area when applicable. | `{ fromId, toId }` | `{ distanceMeters, areaSqMeters?, units: 'meters' }` | +| `apply_patch` | Batched create/update/delete/move, validated and dry-run before commit. | `{ patches: Patch[] }` | `{ applied: number }` | +| `create_level` | Add a new level to a building. | `{ buildingId, elevation, height, label? }` | `{ levelId }` | +| `create_wall` | Add a wall to a level. | `{ levelId, start, end, thickness?, height? }` | `{ wallId }` | +| `place_item` | Place a catalog item on a slab, ceiling, or wall with placement validation. | `{ catalogItemId, targetNodeId, position, rotation? }` | `{ itemId }` or `{ error: 'invalid_placement', reason }` | +| `cut_opening` | Cut a door or window opening into a wall. | `{ wallId, type: 'door' \| 'window', position, width, height }` | `{ openingId }` | +| `set_zone` | Create a zone/room polygon on a level. | `{ levelId, polygon, label, properties? }` | `{ zoneId }` | +| `duplicate_level` | Clone a level and all of its descendants. | `{ levelId }` | `{ newLevelId, newNodeIds[] }` | +| `delete_node` | Delete a node; cascades when `cascade: true`. | `{ id, cascade? }` | `{ deletedIds: [] }` | +| `undo` | Step back through temporal history. | `{ steps? }` | `{ undone: number }` | +| `redo` | Step forward through temporal history. | `{ steps? }` | `{ redone: number }` | +| `export_json` | Serialize the scene graph as JSON. | `{ pretty? }` | `{ json: string }` | +| `export_glb` | Stubbed: GLB export requires the browser renderer. | — | throws `not_implemented` | +| `validate_scene` | Zod-validate every node and parent-child integrity. | — | `{ valid, errors: { nodeId, path, message }[] }` | +| `check_collisions` | Find overlapping items and out-of-bounds placements. | `{ levelId? }` | `{ collisions: { aId, bId, kind }[] }` | +| `analyze_floorplan_image` | Vision tool: extract walls, rooms, and approximate dimensions from a floorplan image. | `{ image, scaleHint? }` | `{ walls, rooms, approximateDimensions, confidence }` | +| `analyze_room_photo` | Vision tool: extract approximate dimensions and fixtures from a room photo. | `{ image }` | `{ approximateDimensions, identifiedFixtures, identifiedWindows }` | + +The vision tools require the MCP host to support the sampling capability +(`createMessage`). Hosts that don't will see a structured +`sampling_unavailable` error. + +## Resources + +| URI | MIME | Purpose | +| --- | --- | --- | +| `pascal://scene/current` | `application/json` | Full `{ nodes, rootNodeIds, collections }` snapshot. | +| `pascal://scene/current/summary` | `text/markdown` | Human-readable summary with node counts, bounding box, and level areas. | +| `pascal://catalog/items` | `application/json` | Item catalog; returns `{ status: 'catalog_unavailable', items: [] }` in headless mode if no catalog is provided. | +| `pascal://constraints/{levelId}` | `application/json` | Slab footprints and wall polygons for the given level — useful as planner context. | + +## Prompts + +| Name | Args | Purpose | +| --- | --- | --- | +| `from_brief` | `{ brief: string, constraints?: string }` | Guided workflow for turning a prose brief (e.g. "2-bed apartment in 80 m²") into an incremental sequence of `apply_patch` calls starting from an empty site. | +| `iterate_on_feedback` | `{ feedback: string }` | Minimal-diff instructions: examine the current scene, then propose the smallest patch set that satisfies the feedback. | +| `renovation_from_photos` | `{ currentPhotos: string[], referencePhotos: string[], goals: string }` | Chains the vision tools with the scene mutation tools to produce a renovation plan grounded in photos. | + +## Limitations + +- `export_glb` returns `not_implemented`. GLB export depends on the Three.js + renderer and isn't reachable headlessly without a large additional effort. +- Vision tools require MCP host sampling support. Claude Desktop supports + this; some MCP clients don't. +- Systems (wall mitering, slab triangulation, CSG cutouts, roof / stair + generation) run inside React hooks in the editor. Headless mode doesn't + regenerate derived geometry — but all node data remains fully manipulable. + Consumers that need rendered geometry run `@pascal-app/viewer` in a browser + host. +- Core's `loadAssetUrl` / `saveAsset` are browser-only; items that reference + `asset://` URLs aren't resolvable in Node. Supply absolute URLs or + `data:` URLs for item assets if you need them usable outside the browser. +- `dirtyNodes` accumulates in headless mode because no renderer consumes it. + Call `bridge.flushDirty()` if observability matters to your consumer. + +## Development + +```bash +bun install +bun run --cwd packages/mcp build +bun test +``` + +Smoke-test the stdio binary end-to-end: + +```bash +bun run --cwd packages/mcp smoke +``` + +## License + +MIT diff --git a/packages/mcp/examples/embed-in-agent.ts b/packages/mcp/examples/embed-in-agent.ts new file mode 100644 index 00000000..52a96e64 --- /dev/null +++ b/packages/mcp/examples/embed-in-agent.ts @@ -0,0 +1,82 @@ +/** + * Programmatic `@pascal-app/mcp` usage. + * + * Runs a full MCP client/server pair over the in-memory transport inside a + * single Node process. Useful for agent frameworks and tests that want to + * drive Pascal without spawning a subprocess. + * + * Compile with the package's `tsc --build`, or run directly with Bun: + * + * bun run packages/mcp/examples/embed-in-agent.ts + */ + +import { Client } from '@modelcontextprotocol/sdk/client/index.js' +import { InMemoryTransport } from '@modelcontextprotocol/sdk/inMemory.js' +import { createPascalMcpServer, SceneBridge } from '@pascal-app/mcp' + +async function main(): Promise { + // 1. Spin up the headless bridge. `loadDefault()` seeds a Site → Building → + // Level stack so the client has something to query immediately. + const bridge = new SceneBridge() + bridge.loadDefault() + const server = createPascalMcpServer({ bridge }) + + // 2. Link the server to an in-memory client. Exactly the same API surface + // as the stdio / HTTP transports, but without any process boundary. + const [srvT, cliT] = InMemoryTransport.createLinkedPair() + const client = new Client({ name: 'my-agent', version: '0.1.0' }) + await Promise.all([server.connect(srvT), client.connect(cliT)]) + + // 3. Discover available capabilities. + const tools = await client.listTools() + console.log( + 'available tools:', + tools.tools.map((t) => t.name), + ) + + // 4. Inspect the current scene. + const scene = await client.callTool({ name: 'get_scene', arguments: {} }) + console.log('scene snapshot:', JSON.stringify(scene, null, 2)) + + // 5. Find the default level, create a 5 m wall, and undo it. + const levels = await client.callTool({ + name: 'find_nodes', + arguments: { type: 'level' }, + }) + const levelId = (levels.structuredContent as { nodes: Array<{ id: string }> }).nodes[0]?.id + + if (levelId) { + const created = await client.callTool({ + name: 'create_wall', + arguments: { + levelId, + start: [0, 0], + end: [5, 0], + thickness: 0.2, + height: 2.5, + }, + }) + console.log('created wall:', created.structuredContent) + + const undone = await client.callTool({ name: 'undo', arguments: { steps: 1 } }) + console.log('undone:', undone.structuredContent) + } + + // 6. Validate and export. + const validation = await client.callTool({ name: 'validate_scene', arguments: {} }) + console.log('validation:', validation.structuredContent) + + const exported = await client.callTool({ + name: 'export_json', + arguments: { pretty: true }, + }) + console.log('export size:', (exported.structuredContent as { json: string }).json.length) + + await client.close() + await server.close() +} + +main().catch((err) => { + console.error(err) + process.exit(1) +}) diff --git a/packages/mcp/examples/generate-apartment.md b/packages/mcp/examples/generate-apartment.md new file mode 100644 index 00000000..6ce2d20f --- /dev/null +++ b/packages/mcp/examples/generate-apartment.md @@ -0,0 +1,242 @@ +# Generate a 2-bed apartment from a brief + +This example walks through a realistic session with an MCP host (Claude +Desktop, Claude Code, or Cursor) that has `pascal-mcp` configured. The agent +uses the `from_brief` prompt to turn a short brief into a concrete scene. + +## The brief + +> **User:** Claude, create a 2-bedroom 1-bath apartment in 80 m² in Spain. + +The host UI lets the user select the **`from_brief`** prompt and fills in: + +```text +brief: "2-bedroom 1-bath apartment in 80 m² in Spain, open-plan living / + kitchen, bathroom on the interior wall" +constraints: "Spanish building regulations; ceiling height 2.5 m" +``` + +## What the agent does + +The prompt returns a system message instructing the agent to start from an +empty site, read the current scene, and emit incremental `apply_patch` calls. +The agent proceeds roughly like this: + +### 1. Inspect the current scene + +```jsonc +// tool: get_scene +{ "name": "get_scene", "arguments": {} } +``` + +Response (trimmed): + +```jsonc +{ + "nodes": { + "site-1": { "type": "site", "id": "site-1", "children": [/* ... */] }, + "building-1": { "type": "building", "id": "building-1", "parentId": "site-1" }, + "level-1": { "type": "level", "id": "level-1", "parentId": "building-1", + "elevation": 0, "height": 2.5 } + }, + "rootNodeIds": ["site-1"] +} +``` + +The default scene is a Site → Building → Level stack with no walls. The +agent decides to work on `level-1` and targets a 10 m × 8 m = 80 m² outline. + +### 2. Create the perimeter walls + +The agent chooses a rectangular outline with its origin at (0, 0): + +```jsonc +// tool: apply_patch +{ + "name": "apply_patch", + "arguments": { + "patches": [ + { "op": "create", "parentId": "level-1", + "node": { "type": "wall", "start": [0, 0], "end": [10, 0], + "thickness": 0.2, "height": 2.5 } }, + { "op": "create", "parentId": "level-1", + "node": { "type": "wall", "start": [10, 0], "end": [10, 8], + "thickness": 0.2, "height": 2.5 } }, + { "op": "create", "parentId": "level-1", + "node": { "type": "wall", "start": [10, 8], "end": [0, 8], + "thickness": 0.2, "height": 2.5 } }, + { "op": "create", "parentId": "level-1", + "node": { "type": "wall", "start": [0, 8], "end": [0, 0], + "thickness": 0.2, "height": 2.5 } } + ] + } +} +``` + +Response: + +```jsonc +{ "applied": 4 } +``` + +### 3. Create interior partitions + +Two bedrooms on the east side, bathroom on the interior wall, open-plan +living / kitchen on the west. + +```jsonc +// tool: apply_patch +{ + "name": "apply_patch", + "arguments": { + "patches": [ + { "op": "create", "parentId": "level-1", + "node": { "type": "wall", "start": [5.5, 0], "end": [5.5, 8], + "thickness": 0.15, "height": 2.5 } }, + { "op": "create", "parentId": "level-1", + "node": { "type": "wall", "start": [5.5, 4], "end": [10, 4], + "thickness": 0.15, "height": 2.5 } }, + { "op": "create", "parentId": "level-1", + "node": { "type": "wall", "start": [5.5, 5.5], "end": [8, 5.5], + "thickness": 0.15, "height": 2.5 } }, + { "op": "create", "parentId": "level-1", + "node": { "type": "wall", "start": [8, 4], "end": [8, 5.5], + "thickness": 0.15, "height": 2.5 } } + ] + } +} +``` + +### 4. Define zones + +The agent declares the rooms so later queries and item placement can target +them by name: + +```jsonc +// tool: set_zone (called once per zone) +{ + "name": "set_zone", + "arguments": { + "levelId": "level-1", + "label": "Living / Kitchen", + "polygon": [[0, 0], [5.5, 0], [5.5, 8], [0, 8]] + } +} +// → { "zoneId": "zone-living" } + +{ + "name": "set_zone", + "arguments": { + "levelId": "level-1", + "label": "Bedroom 1", + "polygon": [[5.5, 0], [10, 0], [10, 4], [5.5, 4]] + } +} +// → { "zoneId": "zone-bed1" } + +{ + "name": "set_zone", + "arguments": { + "levelId": "level-1", + "label": "Bedroom 2", + "polygon": [[5.5, 5.5], [10, 5.5], [10, 8], [5.5, 8]] + } +} +// → { "zoneId": "zone-bed2" } + +{ + "name": "set_zone", + "arguments": { + "levelId": "level-1", + "label": "Bathroom", + "polygon": [[5.5, 4], [8, 4], [8, 5.5], [5.5, 5.5]] + } +} +// → { "zoneId": "zone-bath" } +``` + +### 5. Cut doors and windows + +The agent uses `cut_opening` to add entry doors on each interior partition +and windows on the south and east façades: + +```jsonc +// tool: cut_opening (called once per opening) +{ + "name": "cut_opening", + "arguments": { + "wallId": "wall-south", // perimeter wall [0,0] → [10,0] + "type": "window", + "position": 0.25, // 25% along centerline + "width": 1.2, + "height": 1.2 + } +} +// → { "openingId": "window-south-1" } +``` + +```jsonc +{ + "name": "cut_opening", + "arguments": { + "wallId": "wall-bed1", // partition wall to Bedroom 1 + "type": "door", + "position": 0.4, + "width": 0.9, + "height": 2.1 + } +} +// → { "openingId": "door-bed1" } +``` + +The agent repeats this for Bedroom 2's door, the bathroom door, and two +more windows on the east façade. + +### 6. Validate and report + +```jsonc +// tool: validate_scene +{ "name": "validate_scene", "arguments": {} } +``` + +Response: + +```jsonc +{ "valid": true, "errors": [] } +``` + +The agent then reads the scene summary for its response to the user: + +```jsonc +// resource: pascal://scene/current/summary +{ "uri": "pascal://scene/current/summary" } +``` + +The host displays the returned Markdown: 1 site, 1 building, 1 level, 8 +walls, 4 zones, 3 doors, 3 windows; usable area ~78 m²; perimeter ~36 m. + +### 7. Iterate + +The user follows up: + +> **User:** Swap the bathroom and bedroom 2 — I want the bathroom near the +> entrance. + +The agent loads the `iterate_on_feedback` prompt and issues a single +`apply_patch` that updates the polygon of `zone-bath` and `zone-bed2` and +moves the corresponding partition walls. Because mutation goes through the +Zustand store, the user can `undo` the change if they dislike it: + +```jsonc +{ "name": "undo", "arguments": { "steps": 1 } } +// → { "undone": 1 } +``` + +## Takeaways + +- Mutations batch inside a single `apply_patch` so that `undo` rolls back + the whole logical change. +- Zones are not walls — they're polygon annotations that make later queries + (`find_nodes({ zoneId })`) and planning steps much easier for the agent. +- The agent never needs to speak to `@pascal-app/viewer`: everything the + host sees flows through tools + resources + prompts. diff --git a/packages/mcp/examples/renovate-from-photos.md b/packages/mcp/examples/renovate-from-photos.md new file mode 100644 index 00000000..09fc4a0f --- /dev/null +++ b/packages/mcp/examples/renovate-from-photos.md @@ -0,0 +1,202 @@ +# Renovate an existing flat from photos + +This example shows how an agent can combine the `renovation_from_photos` +prompt with the `analyze_floorplan_image` and `analyze_room_photo` vision +tools to propose a renovation plan grounded in real photos. + +> **Note:** the vision tools use MCP sampling (`createMessage`), which +> Claude Desktop supports today. Hosts without sampling support will get a +> structured `sampling_unavailable` error; fall back to the text-only +> `from_brief` prompt in that case. + +## The brief + +The user drops four photos into the chat: + +1. A floorplan PDF page (exported as PNG). +2. A photo of the current living room. +3. A photo of the current kitchen. +4. An inspirational photo from a magazine — a minimal Scandinavian loft. + +And types: + +> **User:** Claude, help me plan a renovation. Here's the current plan and +> two room photos. I want something like this Scandinavian reference — +> open-plan, neutral tones, keep the footprint. + +## What the agent does + +The host loads the **`renovation_from_photos`** prompt: + +```text +currentPhotos: ["data:image/png;base64,...", "data:image/jpeg;base64,..."] +referencePhotos: ["data:image/jpeg;base64,..."] +goals: "Open-plan living/kitchen, neutral tones, keep the footprint." +``` + +The prompt tells the agent to (1) analyze the floorplan, (2) analyze each +room photo, (3) seed a scene from the floorplan, (4) compare against the +reference, and (5) propose patches. + +### 1. Extract the floorplan + +```jsonc +// tool: analyze_floorplan_image +{ + "name": "analyze_floorplan_image", + "arguments": { + "image": "data:image/png;base64,iVBORw0KGgoAAAANS...", + "scaleHint": "1 m grid, total footprint ~9.5 m × 7 m" + } +} +``` + +Under the hood, the tool issues an MCP sampling request to the host with +the image and a structured prompt asking for walls, rooms, and +approximate dimensions. The response is validated against the tool's +output schema: + +```jsonc +{ + "walls": [ + { "start": [0, 0], "end": [9.5, 0], "thickness": 0.25 }, + { "start": [9.5, 0], "end": [9.5, 7], "thickness": 0.25 }, + { "start": [9.5, 7], "end": [0, 7], "thickness": 0.25 }, + { "start": [0, 7], "end": [0, 0], "thickness": 0.25 }, + { "start": [4.5, 0], "end": [4.5, 7], "thickness": 0.15 }, + { "start": [4.5, 3.5], "end": [9.5, 3.5], "thickness": 0.15 } + ], + "rooms": [ + { "label": "Living", "polygon": [[0, 0], [4.5, 0], [4.5, 7], [0, 7]] }, + { "label": "Kitchen", "polygon": [[4.5, 0], [9.5, 0], [9.5, 3.5], [4.5, 3.5]] }, + { "label": "Bedroom", "polygon": [[4.5, 3.5], [9.5, 3.5], [9.5, 7], [4.5, 7]] } + ], + "approximateDimensions": { "widthMeters": 9.5, "depthMeters": 7, "areaSqMeters": 66.5 }, + "confidence": 0.82 +} +``` + +### 2. Analyze the room photos + +```jsonc +// tool: analyze_room_photo +{ + "name": "analyze_room_photo", + "arguments": { "image": "data:image/jpeg;base64,/9j/4AAQ..." } +} +``` + +Response: + +```jsonc +{ + "approximateDimensions": { "widthMeters": 4.4, "depthMeters": 5.8, "heightMeters": 2.5 }, + "identifiedFixtures": [ + { "kind": "sofa", "approximatePosition": [2.2, 3.5] }, + { "kind": "coffee-table", "approximatePosition": [2.2, 2.4] }, + { "kind": "tv-unit", "approximatePosition": [0.3, 2.0] } + ], + "identifiedWindows": [ + { "wallHint": "south", "approximateWidth": 1.4, "approximateHeight": 1.5 } + ] +} +``` + +The kitchen photo is analyzed the same way. + +### 3. Seed the scene + +The agent reads `get_scene`, confirms the default empty Site → Building → +Level is present, and then batch-creates walls matching the floorplan: + +```jsonc +// tool: apply_patch +{ + "name": "apply_patch", + "arguments": { + "patches": [ + { "op": "create", "parentId": "level-1", + "node": { "type": "wall", "start": [0, 0], "end": [9.5, 0], + "thickness": 0.25, "height": 2.5 } }, + /* ...remaining perimeter + partition walls from the vision result... */ + ] + } +} +``` + +The agent then calls `set_zone` three times to seed the Living / Kitchen / +Bedroom polygons from the floorplan rooms. + +### 4. Cut the identified openings + +For each window the vision tool reported, the agent calls `cut_opening` +against the corresponding perimeter wall: + +```jsonc +{ + "name": "cut_opening", + "arguments": { + "wallId": "wall-south", + "type": "window", + "position": 0.5, + "width": 1.4, + "height": 1.5 + } +} +``` + +### 5. Propose the renovation + +Guided by the reference photo's analysis (bright neutrals, open plan, +minimal furnishing), the agent proposes a single logical patch: + +- Remove the partition wall between Living and Kitchen. +- Relocate the kitchen island further west. +- Delete the bulky TV unit item; leave the sofa and coffee table. +- Re-label the merged zone `"Open-Plan Living / Kitchen"`. + +All of that goes into one `apply_patch`: + +```jsonc +{ + "name": "apply_patch", + "arguments": { + "patches": [ + { "op": "delete", "id": "wall-partition-living-kitchen", "cascade": false }, + { "op": "update", "id": "zone-living", "data": { "label": "Open-Plan Living / Kitchen", + "polygon": [[0, 0], [9.5, 0], + [9.5, 3.5], [0, 3.5]] } }, + { "op": "delete", "id": "zone-kitchen", "cascade": false } + /* + item moves / deletes for the TV unit etc. */ + ] + } +} +``` + +The user can walk back with `undo`; `redo` returns them to the proposal. + +### 6. Sanity-check + +```jsonc +// tool: validate_scene +{ "name": "validate_scene", "arguments": {} } +// → { "valid": true, "errors": [] } + +// tool: check_collisions +{ "name": "check_collisions", "arguments": { "levelId": "level-1" } } +// → { "collisions": [] } +``` + +The agent reports a summary of the changes plus the approximate new +usable area (from the summary resource), and the user opens the scene in +`@pascal-app/viewer` to see the renovated 3D layout. + +## Takeaways + +- The vision tools only return **data**. They don't mutate the scene — + the agent is explicit about every structural change via `apply_patch`. +- Photos supply priors (approximate dimensions, fixture types) that a + brief-only workflow can't. Combine them with `from_brief`-style + prompts when the user has both a reference and concrete text goals. +- All renovation steps are a single temporal step per patch, so the user + can compare before/after with `undo` / `redo`.