feat(tools): mcp-server added monitor-device tool

Add a `monitor_device` MCP tool that lets an AI agent run a scripted,
non-interactive `esp-idf-monitor` session against a flashed device and
get back a short status plus a log file path, instead of raw serial
output inline.

Under the hood:

- The agent supplies a plain-text command body (expect/send/sleep/reset/
  exit/comments). `assemble_monitor_script_from_agent_commands()` frames
  it into a script the monitor's non-interactive command mode can
  consume via stdin: it appends `exit` if the agent didn't already end
  with one, and rewrites every bare `expect <regex>` into `expect
  --timeout <timeout_sec> <regex>` via `_monitor_normalize_expect_line()`
  (an already-bounded `expect --timeout ...` line is left untouched so
  the monitor itself reports a bad value). A leading `reset` is not
  prepended - the monitor already resets the chip when it opens the
  port - and any `reset` the agent wrote is left in place.
  `_monitor_parse_sleep_duration()` extracts each `sleep <n>` duration.
  The effective timeout is the sum of every bounded expect duration
  plus every sleep duration. Scripts whose sum exceeds
  `MONITOR_MAX_SCRIPT_SEC` are rejected. If the script has neither
  expect nor sleep (for example only `send`), `timeout_sec` is used so
  the process still has a kill bound.

- `monitor_device()` runs `python -m esp_idf_monitor` via
  `subprocess.run(..., input=script, timeout=2 * effective_timeout)`.
  `no_reset` is forwarded as `--no-reset` so the connection reset can be
  skipped; an explicit `-p` is forwarded when a port is given. Extra
  arguments match `idf.py monitor` where a build exists: baud (`baud`
  tool arg, else `monitor_baud` from `project_description.json`),
  toolchain prefix, `--target`/`--revision`, coredump/panic decode, and
  ELF files with the app ELF first. The 2x hard timeout is a safety net
  independent of the script's own `expect --timeout`/`exit` logic; on
  `TimeoutExpired` the process is killed but any output already captured
  is preserved and logged. `decode_stream()` normalizes that captured
  output, which can be `bytes` on the timeout path even though the
  process otherwise runs in text mode.

- The monitor's exit code drives the reported status via
  `_monitor_status()`, using `EXIT_EXPECT_TIMEOUT` and
  `EXIT_SCRIPT_ERROR` from `esp_idf_monitor.base.constants`: 0 is
  success, 110 means an `expect` pattern never showed up before its
  `--timeout` elapsed, 2 means the monitor rejected the script (bad
  syntax/timeout/regex), anything else is reported generically.

- Serial output and the monitor's own messages share one pipe
  (`stderr=STDOUT`) so decoded panic backtraces stay next to the lines
  that triggered them. `_save_monitor_output()` writes the full merge to
  `<tempdir>/esp_idf_mcp_log/action_monitor/monitor_<timestamp>.log` and
  reports a dedicated `Log file:` line. On non-zero exit or process
  kill, a short tail of that same merge is also returned inline so the
  agent has some failure context without a second file read. If the log
  file can't be written, it falls back to inlining a truncated tail.

Closes https://github.com/espressif/esp-idf/issues/18757
Closes https://github.com/espressif/esp-idf/pull/18385

Co-authored-by: Cursor <cursoragent@cursor.com>

tools/idf_py_actions/mcp_ext.py

@@ -2,16 +2,23 @@
 # SPDX-License-Identifier: Apache-2.0
 
 import json
+import math
 import os
 import subprocess
 import sys
+import tempfile
+import time
 from pathlib import Path
+from typing import Annotated
 from typing import Any
 
+from esp_idf_monitor.base.constants import EXIT_EXPECT_TIMEOUT
+from esp_idf_monitor.base.constants import EXIT_SCRIPT_ERROR
 from rich_click import Context
 
 from idf_py_actions.errors import FatalError
 from idf_py_actions.tools import PropertyDict
+from idf_py_actions.tools import get_sdkconfig_value
 from idf_py_actions.tools import get_target
 from idf_py_actions.tools import idf_version
 
@@ -39,6 +46,37 @@ except ImportError:
         MCP_AVAILABLE = False
 
 
+try:
+    # Field() feeds per-argument descriptions into a tool's inputSchema (see the
+    # Annotated[...] argument hints on monitor_device below).
+    # Annotated evaluates Field(...) at import time, and this whole module is imported
+    # unconditionally by idf.py on every run, regardless of the command invoked - so without
+    # this fallback, idf.py would break entirely on installs that lack the optional
+    # "mcp" feature. pydantic ships as a dependency of mcp, so this import only
+    # fails when mcp itself is not installed either (see MCP_AVAILABLE above).
+    from pydantic import Field
+except ImportError:
+
+    def Field(**kwargs: Any) -> Any:  # type: ignore[misc] # noqa: N802
+        return None
+
+
+# Root folder (inside the OS temp directory) that ESP-IDF MCP tool logs are
+# written under, one subfolder per tool.
+MCP_LOG_ROOT_DIR_NAME = 'esp_idf_mcp_log'
+
+# Bounded wait injected into 'expect' lines that arrive without --timeout.
+DEFAULT_MONITOR_TIMEOUT_SEC = 20.0
+# Whole-script cap: sum of expect and sleep durations.
+MONITOR_MAX_SCRIPT_SEC = 600.0
+# Commands the MCP monitor script may contain. Everything else (flash, log, ...)
+# is rejected so this tool cannot trigger a rebuild or toggle monitor UI state.
+MONITOR_SCRIPT_COMMANDS = frozenset({'expect', 'send', 'sleep', 'reset', 'bootloader', 'exit'})
+# Size limit for text embedded directly in a tool result, in characters. Serial
+# output belongs in the log file, not in the agent's context.
+MONITOR_TAIL_CHARS = 1500
+
+
 def _is_valid_project_dir(directory: str) -> bool:
     """
     Determine if the given directory is a valid ESP-IDF project directory.
@@ -71,6 +109,160 @@ def _is_valid_project_dir(directory: str) -> bool:
     return False
 
 
+def _derive_build_dir(effective_dir: str, launch_dir: str, args: PropertyDict) -> str:
+    """Use idf.py's build_dir when it belongs to this project; otherwise ``<project>/build``."""
+    if launch_dir == effective_dir:
+        build_dir = args.get('build_dir', '')
+        if build_dir:
+            return str(build_dir)
+    return os.path.join(effective_dir, 'build')
+
+
+def _load_project_description(
+    launch_dir: str, args: PropertyDict
+) -> tuple[str | None, str | None, dict[str, Any] | None]:
+    """Resolve project dir, build dir, and ``project_description.json`` without failing.
+
+    Returns ``(project_dir, build_dir, description)``. Any piece that cannot be
+    resolved is ``None``. Missing project, build directory, or description is not
+    an error.
+    """
+    project_dir = resolve_default_project_dir(launch_dir)
+    if project_dir is None:
+        return None, None, None
+    build_dir = _derive_build_dir(project_dir, launch_dir, args)
+    if not os.path.exists(build_dir):
+        return project_dir, None, None
+    desc_path = os.path.join(build_dir, 'project_description.json')
+    try:
+        with open(desc_path, encoding='utf-8') as f:
+            description = json.load(f)
+    except (OSError, ValueError):
+        return project_dir, build_dir, None
+    if not isinstance(description, dict):
+        return project_dir, build_dir, None
+    return project_dir, build_dir, description
+
+
+def _monitor_normalize_expect_line(line: str, default_expect_duration: float) -> tuple[str, float | None]:
+    """Inject '--timeout default_expect_duration' into a bare 'expect' line.
+
+    Returns (line, expect_duration): expect_duration is the bound this line
+    ends up with, or None if the line already carries a --timeout the monitor
+    will reject - left untouched so the monitor reports the script error
+    itself. Timeout syntax is parsed the same way CommandReader parses it.
+    """
+    command, _, argument = line.partition(' ')
+    argument = argument.strip()
+    if argument.startswith('--timeout'):
+        parts = argument.split(None, 2)
+        try:
+            expect_duration = float(parts[1])
+        except (IndexError, ValueError):
+            return line, None
+        if not math.isfinite(expect_duration) or expect_duration <= 0:
+            return line, None
+        return line, expect_duration
+    return (
+        ' '.join(filter(None, [command, '--timeout', f'{default_expect_duration:g}', argument])),
+        default_expect_duration,
+    )
+
+
+def _monitor_parse_sleep_duration(line: str) -> float | None:
+    """Return the duration of a 'sleep' line, or None if the monitor would
+    reject it (such a line is skipped, so it costs no time)."""
+    argument = line.partition(' ')[2].strip()
+    try:
+        sleep_duration = float(argument)
+    except ValueError:
+        return None
+    if not math.isfinite(sleep_duration) or sleep_duration <= 0:
+        return None
+    return sleep_duration
+
+
+def _save_monitor_output(output: str) -> str:
+    """Save the monitor's output to a log file and describe where it went, or
+    inline a tail on failure - keeps the full serial dump out of the agent's context.
+
+    *output* is the monitor's merged stdout and stderr: the chip prints serial
+    data on stdout while the monitor's own '--- ' lines (decoded panic
+    backtraces among them) go to stderr, so both streams are captured through
+    one pipe to keep a decoded backtrace next to the raw one that triggered it.
+    """
+    log_dir = Path(tempfile.gettempdir()) / MCP_LOG_ROOT_DIR_NAME / 'action_monitor'
+    log_path = log_dir / f'monitor_{time.strftime("%Y%m%d_%H%M%S")}.log'
+    try:
+        log_dir.mkdir(parents=True, exist_ok=True)
+        log_path.write_text(output, encoding='utf-8', errors='replace')
+        # Path on its own line so an agent cannot paraphrase the character
+        # count and drop the location.
+        return f'Log file: {log_path}\nMonitor output: {len(output)} characters. Read or grep that file.'
+    except OSError as e:
+        return f'Log file could not be written ({e}), so here is its tail instead:\n{tail(output, MONITOR_TAIL_CHARS)}'
+
+
+def _monitor_status(returncode: int) -> str:
+    """Explain a monitor exit code, so the agent does not have to scrape stderr."""
+    if returncode == 0:
+        return 'Monitor script completed successfully (exit code 0).'
+    if returncode == EXIT_EXPECT_TIMEOUT:
+        return (
+            f'An expect pattern was not seen before its --timeout elapsed (exit code {returncode}). '
+            'The monitor aborted the rest of the script.'
+        )
+    if returncode == EXIT_SCRIPT_ERROR:
+        return (
+            f'The monitor rejected the script (exit code {returncode}): invalid expect syntax, '
+            'timeout value or regular expression.'
+        )
+    return f'The monitor exited with code {returncode}.'
+
+
+def _monitor_project_args(baud: str | int | None, launch_dir: str, args: PropertyDict) -> list[str]:
+    """Baud, optional decode metadata, and ELF files derived from the project build."""
+    extra: list[str] = []
+    _, build_dir, description = _load_project_description(launch_dir, args)
+
+    # baud: explicit argument, else monitor_baud from the description
+    if baud is not None:
+        extra += ['-b', str(baud)]
+    elif description is not None and description.get('monitor_baud') is not None:
+        extra += ['-b', str(description['monitor_baud'])]
+
+    if description is None or not build_dir:
+        return extra
+
+    # ELF files (app ELF first, same order as idf.py monitor)
+    elf_file = os.path.join(build_dir, description.get('app_elf', '') or '')
+    elf_list = [str(elf) for elf in Path(build_dir).rglob('*.elf')]
+    if elf_file and elf_file in elf_list:
+        elf_list.insert(0, elf_list.pop(elf_list.index(elf_file)))
+    # without ELF files, metadata has no additional value to monitor
+    if not elf_list:
+        return extra
+
+    # toolchain / target / decode flags (only useful with ELFs)
+    toolchain_prefix = description.get('monitor_toolprefix')
+    if toolchain_prefix:
+        extra += ['--toolchain-prefix', str(toolchain_prefix)]
+    config_file = str(description.get('config_file') or '')
+    coredump_decode = get_sdkconfig_value(config_file, 'CONFIG_ESP_COREDUMP_DECODE')
+    if coredump_decode is not None:
+        extra += ['--decode-coredumps', coredump_decode]
+    target = description.get('target')
+    if target:
+        extra += ['--target', str(target)]
+    revision = description.get('min_rev')
+    if revision:
+        extra += ['--revision', str(revision)]
+    if get_sdkconfig_value(config_file, 'CONFIG_IDF_TARGET_ARCH_RISCV'):
+        extra += ['--decode-panic', 'backtrace']
+    extra += elf_list
+    return extra
+
+
 def resolve_default_project_dir(launch_dir: str) -> str | None:
     """
     Returns the first valid ESP-IDF project directory from the server's launch
@@ -110,6 +302,88 @@ def resolve_tool_project_dir(explicit_dir: str | None, launch_dir: str) -> tuple
     )
 
 
+def assemble_monitor_script_from_agent_commands(
+    commands: str,
+    timeout_sec: float = DEFAULT_MONITOR_TIMEOUT_SEC,
+) -> tuple[str, float]:
+    """Frame the agent's command body into a runnable monitor script.
+
+    Bounds every 'expect' with --timeout (default *timeout_sec*) and appends
+    'exit' if missing, so the monitor always terminates on its own. A leading
+    'reset' is not added: the monitor already resets the chip when it opens the
+    port. Agent-supplied 'reset' lines are left in place.
+
+    Returns (script, effective_timeout). Raises ValueError if *commands* has no
+    command at all, if a line is not one of MONITOR_SCRIPT_COMMANDS, or if the
+    script would run longer than MONITOR_MAX_SCRIPT_SEC.
+
+    effective_timeout is the sum of every bounded expect duration plus every
+    sleep duration. If the script has neither (for example only 'send'),
+    *timeout_sec* is used instead so the monitor process still has a kill bound.
+    """
+    lines: list[str] = []
+    effective_timeout = 0.0
+    has_command = False
+    allowed = ', '.join(sorted(MONITOR_SCRIPT_COMMANDS))
+
+    for raw_line in commands.splitlines():
+        line = raw_line.strip()
+        if not line:
+            continue
+        if line.startswith('#'):
+            lines.append(line)
+            continue
+        command = line.partition(' ')[0].lower()
+        if command not in MONITOR_SCRIPT_COMMANDS:
+            raise ValueError(f'Unsupported monitor command {command!r}. Allowed commands: {allowed}.')
+        has_command = True
+        if command == 'expect':
+            line, expect_duration = _monitor_normalize_expect_line(line, timeout_sec)
+            if expect_duration is not None:
+                effective_timeout += expect_duration
+        elif command == 'sleep':
+            sleep_duration = _monitor_parse_sleep_duration(line)
+            if sleep_duration is not None:
+                effective_timeout += sleep_duration
+        lines.append(line)
+
+    if not has_command:
+        raise ValueError(
+            'No monitor commands given. Pass a commands body with at least one command, '
+            'for example: expect --timeout 20 Hello world!'
+        )
+
+    if not lines or lines[-1].lower() != 'exit':
+        lines.append('exit')
+
+    if effective_timeout <= 0:
+        effective_timeout = timeout_sec
+    if effective_timeout > MONITOR_MAX_SCRIPT_SEC:
+        raise ValueError(f'This script would run for {effective_timeout:g}s. The limit is {MONITOR_MAX_SCRIPT_SEC:g}s.')
+    return '\n'.join(lines) + '\n', effective_timeout
+
+
+def decode_stream(stream: Any) -> str:
+    """Decode a subprocess stream that may be bytes even in text mode.
+
+    TimeoutExpired carries the output collected so far as bytes on POSIX, while
+    a completed run in text mode yields str.
+    """
+    if stream is None:
+        return ''
+    if isinstance(stream, bytes):
+        return stream.decode('utf-8', errors='replace')
+    return str(stream)
+
+
+def tail(text: str, limit: int) -> str:
+    """Return at most the last *limit* characters of *text*, marking a cut."""
+    text = text.strip()
+    if len(text) <= limit:
+        return text
+    return f'[...truncated...]\n{text[-limit:]}'
+
+
 def action_extensions(base_actions: dict, project_path: str) -> dict:
     """ESP-IDF MCP Server Extension"""
 
@@ -274,6 +548,126 @@ def action_extensions(base_actions: dict, project_path: str) -> dict:
                 print(f'ERROR: Flash failed: {str(e)}', file=sys.stderr)
                 return f'Error flashing: {str(e)}'
 
+        @mcp.tool(
+            description=(
+                'Run a serial monitor session against a flashed device, driven by a script '
+                '(runs `python -m esp_idf_monitor` in non-interactive command mode). '
+                'Use this to observe what a device prints and to wait for specific output. '
+                'Write the commands body from the project context or from the exact instructions '
+                'of the user; allowed commands are expect, send, sleep, reset, bootloader and exit. '
+                'The monitor already resets the chip when it opens the port, so do not start the '
+                'script with reset unless you need an extra reset later. Set no_reset to skip that '
+                'connection reset. ALWAYS end the script with an explicit exit as its last line. '
+                'Every expect should be bounded: "expect --timeout <seconds> <regex>". '
+                f'The whole script may take at most {MONITOR_MAX_SCRIPT_SEC:g} seconds. '
+                'Exit code 0 means the script finished, 110 means an expect pattern never appeared, '
+                '2 means the script itself was invalid. '
+                'Returns a short status plus a Log file: line with the absolute path of the monitor '
+                'log, which holds the serial output together with the monitor messages. '
+                'Always quote that path when answering the user; read or '
+                'grep the file instead of expecting the whole log inline. '
+                'The port is optional and is autodetected when omitted; see project://devices '
+                'when several devices are connected.'
+            )
+        )
+        def monitor_device(
+            commands: Annotated[
+                str,
+                Field(
+                    description=(
+                        'Monitor script body, one command per line (expect, send, sleep, reset, bootloader, exit). '
+                        'Bound every wait as "expect --timeout <seconds> <regex>" (default 20 seconds). '
+                        f'The whole script may take at most {MONITOR_MAX_SCRIPT_SEC:g} seconds. '
+                        'A reset at the start is unnecessary: the monitor resets on connect unless no_reset. '
+                        'Leave any later reset in the script. End with exit; the server appends one if missing. '
+                        'Lines starting with # are comments.'
+                    )
+                ),
+            ],
+            timeout_sec: Annotated[
+                float,
+                Field(
+                    description=(
+                        'Wait, in seconds, injected into expect lines that omit --timeout. '
+                        f'The whole script may take at most {MONITOR_MAX_SCRIPT_SEC:g} seconds, '
+                        'the sum of expect and sleep durations. The monitor process is killed after '
+                        'twice that time. If the script has no expect or sleep, twice timeout_sec is used.'
+                    )
+                ),
+            ] = DEFAULT_MONITOR_TIMEOUT_SEC,
+            port: str | None = None,
+            baud: str | int | None = None,
+            no_reset: bool = False,
+        ) -> str:
+            """Run a scripted esp-idf-monitor session and return a status plus a log path.
+
+            Args:
+                port: Optional serial port such as /dev/ttyUSB0 or COM3. Leave as
+                    None to let the monitor autodetect it.
+                baud: Optional monitor baud rate. When omitted, ``monitor_baud``
+                    from ``project_description.json`` is used if that file exists.
+                no_reset: Pass --no-reset so the monitor does not reset the chip
+                    when it opens the port. Agent-supplied reset commands in the
+                    script are left unchanged.
+            """
+            if not math.isfinite(timeout_sec) or timeout_sec <= 0:
+                return f'timeout_sec must be a finite number greater than 0, got {timeout_sec}.'
+
+            try:
+                script, effective_timeout = assemble_monitor_script_from_agent_commands(
+                    commands, timeout_sec=timeout_sec
+                )
+            except ValueError as e:
+                return str(e)
+            hard_timeout = 2 * effective_timeout
+
+            cmd = [sys.executable, '-m', 'esp_idf_monitor']
+            if port:
+                cmd.extend(['-p', port])
+            cmd.extend(_monitor_project_args(baud, project_path, args))
+            if no_reset:
+                cmd.append('--no-reset')
+
+            print(f'INFO: Running monitor: {" ".join(cmd)} (hard timeout {hard_timeout:g}s)', file=sys.stderr)
+            print(f'INFO: Monitor script:\n{script}', file=sys.stderr)
+
+            try:
+                result = subprocess.run(
+                    cmd,
+                    input=script,
+                    stdout=subprocess.PIPE,
+                    # One pipe for both streams keeps the monitor's decoded backtraces
+                    # in order with the serial lines they belong to.
+                    stderr=subprocess.STDOUT,
+                    text=True,
+                    timeout=hard_timeout,
+                )
+            except subprocess.TimeoutExpired as e:
+                print(f'ERROR: Monitor killed after {hard_timeout:g}s', file=sys.stderr)
+                output = decode_stream(e.output)
+                parts = [
+                    f'The monitor did not exit on its own and was killed after {hard_timeout:g} seconds '
+                    '(twice the time the script should have taken). '
+                    'The serial output captured until then was kept.',
+                    _save_monitor_output(output),
+                ]
+                output_tail = tail(output, MONITOR_TAIL_CHARS)
+                if output_tail:
+                    parts.append(f'Last output before the kill:\n{output_tail}')
+                return '\n'.join(parts)
+            except Exception as e:
+                print(f'ERROR: Monitor failed to run: {e}', file=sys.stderr)
+                return f'Failed to run the monitor: {e}'
+
+            print(f'INFO: Monitor exited with code {result.returncode}', file=sys.stderr)
+            output = result.stdout or ''
+            parts = [_monitor_status(result.returncode), _save_monitor_output(output)]
+            if result.returncode != 0:
+                output_tail = tail(output, MONITOR_TAIL_CHARS)
+                if output_tail:
+                    parts.append(f'Last output before the exit:\n{output_tail}')
+            return '\n'.join(parts)
+
         @mcp.tool(
             description=(
                 'Create a new ESP-IDF project from the sample template (runs `idf.py create-project`). '
@@ -354,7 +748,7 @@ def action_extensions(base_actions: dict, project_path: str) -> dict:
         @mcp.resource('project://config')
         def get_project_config() -> str:
             """Get current project configuration"""
-            effective_dir = resolve_default_project_dir(project_path)
+            effective_dir, build_dir, description = _load_project_description(project_path, args)
             config: dict[str, Any] = {}
 
             if effective_dir is None:
@@ -365,24 +759,15 @@ def action_extensions(base_actions: dict, project_path: str) -> dict:
                 return json.dumps(config, indent=2)
             config['project_path'] = effective_dir
 
-            # Use the build_dir from idf.py args when the project matches; otherwise derive it.
-            build_dir = (
-                args.get('build_dir', '') if project_path == effective_dir else os.path.join(effective_dir, 'build')
-            )
-
-            if not os.path.exists(build_dir):
+            if build_dir is None:
                 config['build_dir_exists'] = False
                 return json.dumps(config, indent=2)
 
             config['build_dir'] = build_dir
-            proj_desc_fn = os.path.join(build_dir, 'project_description.json')
-            config['project_description'] = 'Project description does not exist'
-
-            try:
-                with open(proj_desc_fn, encoding='utf-8') as f:
-                    config['project_description'] = json.load(f)
-            except (OSError, ValueError):
-                pass
+            if description is not None:
+                config['project_description'] = description
+            else:
+                config['project_description'] = 'Project description does not exist'
 
             return json.dumps(config, indent=2)
 
@@ -405,10 +790,7 @@ def action_extensions(base_actions: dict, project_path: str) -> dict:
                 status['target'] = get_target(effective_dir)
                 status['idf_version'] = idf_version()
 
-                # Use the build_dir from idf.py args when the project matches; otherwise derive it.
-                build_dir = (
-                    args.get('build_dir', '') if project_path == effective_dir else os.path.join(effective_dir, 'build')
-                )
+                build_dir = _derive_build_dir(effective_dir, project_path, args)
                 if os.path.exists(build_dir):
                     status['build_dir'] = build_dir
                     artifacts = ['bootloader', 'partition_table', 'app-flash', 'flash_args']