Compare commits

..

4 Commits

Author SHA1 Message Date
dependabot[bot]
3d739c795f chore(deps): bump the github-actions group with 4 updates
Bumps the github-actions group with 4 updates: [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action), [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action), [docker/login-action](https://github.com/docker/login-action) and [docker/build-push-action](https://github.com/docker/build-push-action).


Updates `docker/setup-qemu-action` from 4.1.0 to 4.2.0
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v4.1.0...v4.2.0)

Updates `docker/setup-buildx-action` from 4.1.0 to 4.2.0
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v4.1.0...v4.2.0)

Updates `docker/login-action` from 4.2.0 to 4.4.0
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v4.2.0...v4.4.0)

Updates `docker/build-push-action` from 7.2.0 to 7.3.0
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v7.2.0...v7.3.0)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
- dependency-name: docker/setup-buildx-action
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
- dependency-name: docker/login-action
  dependency-version: 4.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
- dependency-name: docker/build-push-action
  dependency-version: 7.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-06 16:14:33 +00:00
Weilong Liao
2a7c02af8e fix: remove qq official reply id on proactive fallback so oversized passive replies can succeed via proactive sending (#9169) 2026-07-06 23:48:30 +08:00
lingyun14
56f2533951 fix(mcp): attempt to fix epoll busy-wait caused by leaked SSE exit stack (#8307)
* fix(mcp): prevent epoll busy-wait caused by leaked SSE exit stack

* fix(mcp): address review feedback on owner task connection lifecycle

* Update mcp_client.py
2026-07-06 23:06:48 +08:00
killer-qert
afb0079116 fix: exit MCP anyio contexts in the lifecycle task on disable (#9132)
* fix: exit MCP anyio contexts in the lifecycle task on disable

#9070 moved MCP shutdown cleanup into the per-server lifecycle task so
that the anyio cancel scopes entered in connect_to_server() are exited
from the task that entered them. Running the cleanup through
asyncio.shield() defeats that: shield wraps the coroutine in a new
task, so disabling a server still fails with "Attempted to exit cancel
scope in a different task than it was entered in" and leaves the scope
state corrupted -- the failure mode behind the CPU spin reported in
#9068.

Await _terminate_mcp_client() directly instead. The graceful disable
path has no cancellation in flight, and on forced shutdown the
lifecycle task still runs its finally block in-task, so the shield only
served to break task affinity.

Closes #9068

* fix: absorb late cancellations during MCP shutdown cleanup

A cancellation delivered while the lifecycle task is already running its
finally-block cleanup would abort _terminate_mcp_client() halfway,
stranding the runtime entry and the MCP transport. Retry the cleanup in
the same task and uncancel() the absorbed request instead, so a forced
shutdown can no longer skip it.

* fix: guard Task.uncancel() for Python 3.10 runtimes

Task.uncancel() only exists on Python 3.11+. On 3.10 absorbing the
CancelledError is sufficient, so skip the call when unavailable.
2026-07-06 23:04:41 +08:00
8 changed files with 227 additions and 399 deletions

View File

@@ -71,20 +71,20 @@ jobs:
echo "build_date=$build_date" >> $GITHUB_OUTPUT
- name: Set QEMU
uses: docker/setup-qemu-action@v4.1.0
uses: docker/setup-qemu-action@v4.2.0
- name: Set Docker Buildx
uses: docker/setup-buildx-action@v4.1.0
uses: docker/setup-buildx-action@v4.2.0
- name: Log in to DockerHub
uses: docker/login-action@v4.2.0
uses: docker/login-action@v4.4.0
with:
username: ${{ secrets.DOCKER_HUB_USERNAME }}
password: ${{ secrets.DOCKER_HUB_PASSWORD }}
- name: Login to GitHub Container Registry
if: env.HAS_GHCR_TOKEN == 'true'
uses: docker/login-action@v4.2.0
uses: docker/login-action@v4.4.0
with:
registry: ghcr.io
username: ${{ env.GHCR_OWNER }}
@@ -105,7 +105,7 @@ jobs:
echo "EOF" >> $GITHUB_OUTPUT
- name: Build and Push Nightly Image
uses: docker/build-push-action@v7.2.0
uses: docker/build-push-action@v7.3.0
with:
context: .
platforms: linux/amd64,linux/arm64
@@ -171,27 +171,27 @@ jobs:
cp -r dashboard/dist astrbot/dashboard/dist
- name: Set QEMU
uses: docker/setup-qemu-action@v4.1.0
uses: docker/setup-qemu-action@v4.2.0
- name: Set Docker Buildx
uses: docker/setup-buildx-action@v4.1.0
uses: docker/setup-buildx-action@v4.2.0
- name: Log in to DockerHub
uses: docker/login-action@v4.2.0
uses: docker/login-action@v4.4.0
with:
username: ${{ secrets.DOCKER_HUB_USERNAME }}
password: ${{ secrets.DOCKER_HUB_PASSWORD }}
- name: Login to GitHub Container Registry
if: env.HAS_GHCR_TOKEN == 'true'
uses: docker/login-action@v4.2.0
uses: docker/login-action@v4.4.0
with:
registry: ghcr.io
username: ${{ env.GHCR_OWNER }}
password: ${{ secrets.GHCR_GITHUB_TOKEN }}
- name: Build and Push Release Image
uses: docker/build-push-action@v7.2.0
uses: docker/build-push-action@v7.3.0
with:
context: .
platforms: linux/amd64,linux/arm64

View File

@@ -392,10 +392,16 @@ def _normalize_mcp_input_schema(schema: dict[str, Any]) -> dict[str, Any]:
class MCPClient:
def __init__(self) -> None:
# Initialize session and client objects
self.session: mcp.ClientSession | None = None
self.exit_stack = AsyncExitStack()
self._old_exit_stacks: list[AsyncExitStack] = [] # Track old stacks for cleanup
# Each connection runs in its own task so that anyio cancel scopes
# are always exited from the task that entered them, preventing
# RuntimeError: Attempted to exit cancel scope in a different task
self._connection_task: asyncio.Task | None = None
self._old_connection_tasks: list[asyncio.Task] = []
# Internal; managed exclusively by _run_connection.
self.exit_stack: AsyncExitStack | None = None
self.name: str | None = None
self.active: bool = True
@@ -403,14 +409,67 @@ class MCPClient:
self.server_errlogs: list[str] = []
self.running_event = asyncio.Event()
# Store connection config for reconnection
self._mcp_server_config: dict | None = None
self._server_name: str | None = None
self._reconnect_lock = asyncio.Lock() # Lock for thread-safe reconnection
self._reconnecting: bool = False # For logging and debugging
self._reconnecting: bool = False
async def _run_connection(
self,
mcp_server_config: dict,
name: str,
ready: asyncio.Future,
) -> None:
"""Own the full lifetime of one MCP connection.
This coroutine is always run inside a dedicated asyncio.Task
(_connection_task). Because *this task* is the one that enters every
anyio cancel scope (via sse_client / streamablehttp_client), anyio's
_host_task check is always satisfied when the stack is later closed —
either in the task's own finally block (normal path) or when the task
is cancelled from outside (cleanup / reconnect path).
This avoids the
RuntimeError: Attempted to exit cancel scope in a different task
that previously occurred when aclose() was called from a different task
or from the asyncio async-generator GC finalizer.
"""
# Capture the stack in a local variable so that if self.exit_stack is
# overwritten by a concurrent _run_connection (during reconnect), this
# task's finally block still closes only the resources it opened.
stack = self.exit_stack = AsyncExitStack()
try:
try:
await self._do_connect(mcp_server_config, name)
except Exception as exc:
if not ready.done():
ready.set_exception(exc)
raise
else:
if not ready.done():
ready.set_result(None)
# Hold the connection open until cancelled.
await asyncio.Event().wait()
finally:
try:
await stack.aclose()
except Exception as e:
logger.debug(f"Error closing exit stack for {name}: {e}")
# Clear the instance reference only if it still points to this task's
# stack; a concurrent reconnect may have already replaced it.
if self.exit_stack is stack:
self.exit_stack = None
# Guard against the task exiting before ready was resolved.
if not ready.done():
ready.set_exception(RuntimeError("Connection task exited early"))
async def connect_to_server(self, mcp_server_config: dict, name: str) -> None:
"""Connect to MCP server
"""Connect to MCP server by spawning a dedicated owner task.
The owner task (_connection_task) holds the AsyncExitStack and all
anyio cancel scopes for the lifetime of this connection. To disconnect,
cancel _connection_task — the finally block in _run_connection will call
aclose() from within the correct task context.
If `url` parameter exists:
1. When transport is specified as `streamable_http`, use Streamable HTTP connection.
@@ -421,10 +480,47 @@ class MCPClient:
mcp_server_config (dict): Configuration for the MCP server. See https://modelcontextprotocol.io/quickstart/server
"""
# Store config for reconnection
self._mcp_server_config = mcp_server_config
self._server_name = name
ready: asyncio.Future = asyncio.get_running_loop().create_future()
# Defensively cancel any existing connection task that was not cleaned
# up before this call (e.g. if connect_to_server is called twice).
if self._connection_task and not self._connection_task.done():
self._cancel_connection_task(self._connection_task)
self._connection_task = None
self._connection_task = asyncio.create_task(
self._run_connection(mcp_server_config, name, ready),
name=f"mcp-conn:{name}",
)
try:
await ready
except asyncio.CancelledError:
# Caller was cancelled while waiting — tear down the connection task.
# cancel() is asynchronous; the task will not finish until the next
# event-loop iteration, so we track it in _old_connection_tasks so
# that cleanup() can await it later.
if self._connection_task and not self._connection_task.done():
self._cancel_connection_task(self._connection_task)
self._connection_task = None
raise
except Exception:
# _do_connect raised; the connection task's finally block may still
# be running (e.g. awaiting stack.aclose()). Track it so that
# cleanup() can await it, but do NOT cancel it — we want the
# finally block to finish cleaning up resources naturally.
if self._connection_task and not self._connection_task.done():
self._old_connection_tasks.append(self._connection_task)
self._connection_task = None
raise
async def _do_connect(self, mcp_server_config: dict, name: str) -> None:
"""Internal: perform the actual connection inside _run_connection's task."""
# exit_stack is always set by _run_connection before _do_connect is called.
assert self.exit_stack is not None
cfg = _prepare_config(mcp_server_config.copy())
def logging_callback(
@@ -563,16 +659,32 @@ class MCPClient:
self.tools = response.tools
return response
def _cancel_connection_task(self, task: asyncio.Task) -> None:
"""Cancel a connection owner task and track it until it finishes."""
# Prune already-finished tasks to avoid accumulating references over
# many reconnections in a long-running process.
self._old_connection_tasks = [
t for t in self._old_connection_tasks if not t.done()
]
if task.done():
return
task.cancel()
self._old_connection_tasks.append(task)
async def _reconnect(self) -> None:
"""Reconnect to the MCP server using the stored configuration.
Cancels the current _connection_task (which owns the exit_stack and all
anyio cancel scopes) and starts a fresh one. Because each connection
task enters and exits its own anyio cancel scope, there is no
cross-task cancel-scope violation and no GC finalizer surprise.
Uses asyncio.Lock to ensure thread-safe reconnection in concurrent environments.
Raises:
Exception: raised when reconnection fails
"""
async with self._reconnect_lock:
# Check if already reconnecting (useful for logging)
if self._reconnecting:
logger.debug(
f"MCP Client {self._server_name} is already reconnecting, skipping"
@@ -588,17 +700,16 @@ class MCPClient:
f"Attempting to reconnect to MCP server {self._server_name}..."
)
# Save old exit_stack for later cleanup (don't close it now to avoid cancel scope issues)
if self.exit_stack:
self._old_exit_stacks.append(self.exit_stack)
# Mark old session as invalid
# Cancel the old connection task. Its finally block will call
# exit_stack.aclose() from within the correct task context, so
# anyio cancel scopes are exited cleanly without triggering the
# GC-finalizer busy-spin bug.
if self._connection_task and not self._connection_task.done():
self._cancel_connection_task(self._connection_task)
self._connection_task = None
self.session = None
# Create new exit stack for new connection
self.exit_stack = AsyncExitStack()
# Reconnect using stored config
# Reconnect — this creates a new _connection_task.
await self.connect_to_server(self._mcp_server_config, self._server_name)
await self.list_tools_and_save()
@@ -663,19 +774,20 @@ class MCPClient:
return await _call_with_retry()
async def cleanup(self) -> None:
"""Clean up resources including old exit stacks from reconnections"""
# Close current exit stack
try:
await self.exit_stack.aclose()
except Exception as e:
logger.debug(f"Error closing current exit stack: {e}")
"""Clean up resources by cancelling the connection owner task."""
# Cancel current and any old connection tasks via the shared helper so
# all cancellation + tracking behaviour goes through one code path.
if self._connection_task:
self._cancel_connection_task(self._connection_task)
self._connection_task = None
# Don't close old exit stacks as they may be in different task contexts
# They will be garbage collected naturally
# Just clear the list to release references
self._old_exit_stacks.clear()
if self._old_connection_tasks:
pending = [t for t in self._old_connection_tasks if not t.done()]
if pending:
await asyncio.gather(*pending, return_exceptions=True)
self._old_connection_tasks.clear()
# Set running_event first to unblock any waiting tasks
# Set running_event to unblock any waiting tasks
self.running_event.set()

View File

@@ -35,9 +35,6 @@ from astrbot.core.star.star_manager import PluginManager
from astrbot.core.subagent_orchestrator import SubAgentOrchestrator
from astrbot.core.umop_config_router import UmopConfigRouter
from astrbot.core.updator import AstrBotUpdator
from astrbot.core.utils.event_loop_diagnostics import (
create_event_loop_diagnostic_tasks,
)
from astrbot.core.utils.llm_metadata import update_llm_metadata
from astrbot.core.utils.migra_helper import migra
from astrbot.core.utils.temp_dir_cleaner import TempDirCleaner
@@ -299,18 +296,13 @@ class AstrBotCoreLifecycle:
self.temp_dir_cleaner.run(),
name="temp_dir_cleaner",
)
diagnostic_tasks = create_event_loop_diagnostic_tasks()
# 把插件中注册的所有协程函数注册到事件总线中并执行
extra_tasks = []
for task in self.star_context._register_tasks:
extra_tasks.append(asyncio.create_task(task, name=task.__name__)) # type: ignore
tasks_ = [
event_bus_task,
*diagnostic_tasks,
*(extra_tasks if extra_tasks else []),
]
tasks_ = [event_bus_task, *(extra_tasks if extra_tasks else [])]
if cron_task:
tasks_.append(cron_task)
if temp_dir_cleaner_task:

View File

@@ -502,6 +502,7 @@ class QQOfficialMessageEvent(AstrMessageEvent):
logger.info("[QQOfficial] 回复消息失败: %s, 尝试使用主动发送接口。", err)
if payload.get("msg_id"):
fallback_payload = payload.copy()
fallback_payload.pop("msg_id", None)
try:
ret = await send_func(fallback_payload)
logger.info("[QQOfficial] 使用主动发送接口发送成功。")

View File

@@ -723,8 +723,21 @@ class FunctionToolManager:
logger.debug(f"MCP client {name} task was cancelled")
raise
finally:
# Cleanup in the same task that entered the anyio contexts
await asyncio.shield(self._terminate_mcp_client(name))
# Cleanup in the same task that entered the anyio contexts:
# asyncio.shield() would schedule the coroutine as a separate
# Task, and anyio cancel scopes cannot exit across tasks (#9068).
# Absorb late cancellations so a forced shutdown cannot abort
# the cleanup halfway.
task = asyncio.current_task()
while True:
try:
await self._terminate_mcp_client(name)
break
except asyncio.CancelledError:
# Task.uncancel() is 3.11+; on 3.10 absorbing the
# cancellation is sufficient.
if task is not None and hasattr(task, "uncancel"):
task.uncancel()
lifecycle_task = asyncio.create_task(
connect_and_lifecycle(), name=f"mcp-client:{name}"

View File

@@ -1,216 +0,0 @@
import asyncio
import faulthandler
from dataclasses import dataclass
from pathlib import Path
from typing import TextIO
from astrbot import logger
from astrbot.core.utils.astrbot_path import get_astrbot_data_path
DEFAULT_LAG_MONITOR_ENABLED = True
DEFAULT_LAG_MONITOR_INTERVAL = 5.0
DEFAULT_LAG_MONITOR_THRESHOLD = 15.0
DEFAULT_WATCHDOG_ENABLED = True
DEFAULT_WATCHDOG_INTERVAL = 5.0
DEFAULT_WATCHDOG_TIMEOUT = 30.0
DEFAULT_WATCHDOG_LOG_RELATIVE_PATH = Path("logs") / "event_loop_watchdog.log"
DEFAULT_WATCHDOG_LOG_MAX_BYTES = 1024 * 1024
@dataclass(frozen=True)
class EventLoopDiagnosticSettings:
"""Settings for event loop lag and blockage diagnostics.
Args:
lag_monitor_enabled: Whether to log event loop scheduling lag.
lag_monitor_interval: Seconds between lag monitor wakeups.
lag_monitor_threshold: Minimum lag seconds before logging a warning.
watchdog_enabled: Whether to arm the faulthandler watchdog.
watchdog_interval: Seconds between faulthandler watchdog refreshes.
watchdog_timeout: Seconds without event loop progress before dumping stacks.
watchdog_log_path: File that receives faulthandler watchdog output.
watchdog_log_max_bytes: Maximum watchdog log bytes before rotation.
"""
lag_monitor_enabled: bool
lag_monitor_interval: float
lag_monitor_threshold: float
watchdog_enabled: bool
watchdog_interval: float
watchdog_timeout: float
watchdog_log_path: Path
watchdog_log_max_bytes: int
def _watchdog_log_path() -> Path:
"""Resolve the watchdog stack dump log path.
Returns:
Absolute path for watchdog stack dump output.
"""
return Path(get_astrbot_data_path()) / DEFAULT_WATCHDOG_LOG_RELATIVE_PATH
def load_event_loop_diagnostic_settings() -> EventLoopDiagnosticSettings:
"""Load fixed event loop diagnostic settings.
Returns:
Event loop diagnostic settings.
"""
return EventLoopDiagnosticSettings(
lag_monitor_enabled=DEFAULT_LAG_MONITOR_ENABLED,
lag_monitor_interval=DEFAULT_LAG_MONITOR_INTERVAL,
lag_monitor_threshold=DEFAULT_LAG_MONITOR_THRESHOLD,
watchdog_enabled=DEFAULT_WATCHDOG_ENABLED,
watchdog_interval=DEFAULT_WATCHDOG_INTERVAL,
watchdog_timeout=DEFAULT_WATCHDOG_TIMEOUT,
watchdog_log_path=_watchdog_log_path(),
watchdog_log_max_bytes=DEFAULT_WATCHDOG_LOG_MAX_BYTES,
)
async def monitor_event_loop_lag(
*,
interval: float = DEFAULT_LAG_MONITOR_INTERVAL,
warn_after: float = DEFAULT_LAG_MONITOR_THRESHOLD,
) -> None:
"""Log a warning when the event loop wakes significantly later than expected.
Args:
interval: Seconds between monitor wakeups.
warn_after: Minimum lag seconds before logging a warning.
"""
loop = asyncio.get_running_loop()
expected = loop.time() + interval
while True:
await asyncio.sleep(interval)
now = loop.time()
lag = now - expected
if lag > warn_after:
logger.warning(
"Event loop lag detected: %.3fs (threshold %.3fs).",
lag,
warn_after,
)
expected = now + interval
def _rotate_watchdog_log_file(log_path: Path, max_bytes: int) -> None:
"""Rotate the watchdog log when it reaches the configured size limit.
Args:
log_path: Current watchdog log path.
max_bytes: Maximum current log size before rotation.
"""
try:
if not log_path.exists() or log_path.stat().st_size < max_bytes:
return
rotated_path = log_path.with_name(f"{log_path.name}.1")
if rotated_path.exists():
rotated_path.unlink()
log_path.replace(rotated_path)
except OSError as e:
logger.warning("Failed to rotate event loop watchdog log %s: %s", log_path, e)
def _open_watchdog_log_file(log_path: Path, max_bytes: int) -> TextIO:
"""Open the watchdog log file after applying size-based rotation.
Args:
log_path: Current watchdog log path.
max_bytes: Maximum current log size before rotation.
Returns:
Writable text file object for faulthandler output.
"""
log_path.parent.mkdir(parents=True, exist_ok=True)
_rotate_watchdog_log_file(log_path, max_bytes)
return log_path.open("a", encoding="utf-8")
async def faulthandler_event_loop_watchdog(
*,
timeout: float = DEFAULT_WATCHDOG_TIMEOUT,
interval: float = DEFAULT_WATCHDOG_INTERVAL,
dump_file: TextIO | None = None,
dump_path: Path | None = None,
max_bytes: int = DEFAULT_WATCHDOG_LOG_MAX_BYTES,
) -> None:
"""Dump all thread stacks if the event loop is blocked for too long.
Args:
timeout: Seconds without watchdog refresh before faulthandler dumps stacks.
interval: Seconds between watchdog refreshes while the event loop is healthy.
dump_file: File object that receives faulthandler output.
dump_path: Path that receives faulthandler output when dump_file is unset.
max_bytes: Maximum current log size before rotation.
"""
log_path = dump_path or _watchdog_log_path()
try:
while True:
faulthandler.cancel_dump_traceback_later()
output: TextIO | None = None
should_close = False
try:
output = dump_file or _open_watchdog_log_file(log_path, max_bytes)
should_close = dump_file is None
faulthandler.dump_traceback_later(
timeout,
repeat=False,
file=output,
)
await asyncio.sleep(interval)
except Exception as e:
logger.warning("Event loop faulthandler watchdog failed: %s", e)
await asyncio.sleep(interval)
finally:
faulthandler.cancel_dump_traceback_later()
if should_close and output is not None:
output.close()
finally:
faulthandler.cancel_dump_traceback_later()
def create_event_loop_diagnostic_tasks() -> list[asyncio.Task]:
"""Create enabled event loop diagnostic tasks for the current loop.
Returns:
A list of created asyncio tasks.
"""
settings = load_event_loop_diagnostic_settings()
tasks: list[asyncio.Task] = []
if settings.lag_monitor_enabled:
tasks.append(
asyncio.create_task(
monitor_event_loop_lag(
interval=settings.lag_monitor_interval,
warn_after=settings.lag_monitor_threshold,
),
name="event_loop_lag_monitor",
)
)
if settings.watchdog_enabled:
logger.info(
"Event loop faulthandler watchdog enabled: timeout=%.3fs interval=%.3fs. "
"If the loop is blocked, Python thread stacks will be written to %s "
"(rotates at %d bytes).",
settings.watchdog_timeout,
settings.watchdog_interval,
settings.watchdog_log_path,
settings.watchdog_log_max_bytes,
)
tasks.append(
asyncio.create_task(
faulthandler_event_loop_watchdog(
timeout=settings.watchdog_timeout,
interval=settings.watchdog_interval,
dump_path=settings.watchdog_log_path,
max_bytes=settings.watchdog_log_max_bytes,
),
name="event_loop_faulthandler_watchdog",
)
)
return tasks

View File

@@ -1,134 +0,0 @@
import asyncio
import pytest
from astrbot.core.utils import event_loop_diagnostics as diagnostics
def test_load_event_loop_diagnostic_settings_defaults():
"""Default settings enable lag monitoring and stack dump watchdog."""
settings = diagnostics.load_event_loop_diagnostic_settings()
assert settings.lag_monitor_enabled is True
assert settings.lag_monitor_interval == diagnostics.DEFAULT_LAG_MONITOR_INTERVAL
assert settings.lag_monitor_threshold == diagnostics.DEFAULT_LAG_MONITOR_THRESHOLD
assert settings.watchdog_enabled is True
assert settings.watchdog_interval == diagnostics.DEFAULT_WATCHDOG_INTERVAL
assert settings.watchdog_timeout == diagnostics.DEFAULT_WATCHDOG_TIMEOUT
assert settings.watchdog_log_max_bytes == diagnostics.DEFAULT_WATCHDOG_LOG_MAX_BYTES
@pytest.mark.asyncio
async def test_create_event_loop_diagnostic_tasks_defaults():
"""Default diagnostics should create both event loop diagnostic tasks."""
tasks = diagnostics.create_event_loop_diagnostic_tasks()
try:
assert [task.get_name() for task in tasks] == [
"event_loop_lag_monitor",
"event_loop_faulthandler_watchdog",
]
finally:
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
@pytest.mark.asyncio
async def test_faulthandler_watchdog_cancels_pending_dump(monkeypatch):
"""The faulthandler watchdog should cancel its pending dump on shutdown."""
calls = []
class FakeFaultHandler:
def cancel_dump_traceback_later(self):
calls.append("cancel")
def dump_traceback_later(self, timeout, repeat, file):
calls.append(("dump", timeout, repeat, file))
fake_faulthandler = FakeFaultHandler()
monkeypatch.setattr(diagnostics, "faulthandler", fake_faulthandler)
task = asyncio.create_task(
diagnostics.faulthandler_event_loop_watchdog(timeout=10, interval=1)
)
await asyncio.sleep(0)
task.cancel()
await asyncio.gather(task, return_exceptions=True)
assert any(isinstance(call, tuple) and call[0] == "dump" for call in calls)
assert calls[-1] == "cancel"
@pytest.mark.asyncio
async def test_faulthandler_watchdog_writes_rotating_log(tmp_path, monkeypatch):
"""The faulthandler watchdog should write to and rotate its log file."""
log_path = tmp_path / "logs" / "event_loop_watchdog.log"
log_path.parent.mkdir()
log_path.write_text("x" * 8, encoding="utf-8")
calls = []
class FakeFaultHandler:
def cancel_dump_traceback_later(self):
calls.append("cancel")
def dump_traceback_later(self, timeout, repeat, file):
calls.append(("dump", timeout, repeat, file.name))
file.write("watchdog dump\n")
file.flush()
fake_faulthandler = FakeFaultHandler()
monkeypatch.setattr(diagnostics, "faulthandler", fake_faulthandler)
task = asyncio.create_task(
diagnostics.faulthandler_event_loop_watchdog(
timeout=10,
interval=1,
dump_path=log_path,
max_bytes=4,
)
)
await asyncio.sleep(0)
task.cancel()
await asyncio.gather(task, return_exceptions=True)
assert log_path.read_text(encoding="utf-8") == "watchdog dump\n"
assert log_path.with_name("event_loop_watchdog.log.1").read_text(
encoding="utf-8"
) == "x" * 8
assert any(isinstance(call, tuple) and call[0] == "dump" for call in calls)
@pytest.mark.asyncio
async def test_faulthandler_watchdog_survives_dump_failure(tmp_path, monkeypatch):
"""The watchdog should keep running after faulthandler arm failures."""
log_path = tmp_path / "event_loop_watchdog.log"
armed_again = asyncio.Event()
calls = []
class FakeFaultHandler:
def cancel_dump_traceback_later(self):
calls.append("cancel")
def dump_traceback_later(self, timeout, repeat, file):
calls.append(("dump", timeout, repeat, file.name))
if len([call for call in calls if isinstance(call, tuple)]) == 1:
raise RuntimeError("boom")
armed_again.set()
fake_faulthandler = FakeFaultHandler()
monkeypatch.setattr(diagnostics, "faulthandler", fake_faulthandler)
task = asyncio.create_task(
diagnostics.faulthandler_event_loop_watchdog(
timeout=10,
interval=0.01,
dump_path=log_path,
)
)
await asyncio.wait_for(armed_again.wait(), timeout=1)
task.cancel()
await asyncio.gather(task, return_exceptions=True)
dump_calls = [call for call in calls if isinstance(call, tuple)]
assert len(dump_calls) >= 2

View File

@@ -1,3 +1,4 @@
import asyncio
import json
import pytest
@@ -348,6 +349,65 @@ def test_firecrawl_tools_are_registered_as_builtin_tools():
assert manager.is_builtin_tool("firecrawl_extract_web_page") is True
@pytest.mark.asyncio
async def test_mcp_shutdown_cleanup_runs_in_lifecycle_task(monkeypatch):
"""Disabling an MCP server must clean up in the task that connected.
anyio cancel scopes entered in connect_to_server() can only be exited
from the same task, otherwise the scope state is corrupted and its
cancellation loop spins at 100% CPU (#9068).
"""
manager = FunctionToolManager()
seen = {}
async def fake_connect(self, config, name):
seen["connect_task"] = asyncio.current_task()
async def fake_list_tools(self):
self.tools = []
async def fake_cleanup(self):
seen["cleanup_task"] = asyncio.current_task()
monkeypatch.setattr(ftm.MCPClient, "connect_to_server", fake_connect)
monkeypatch.setattr(ftm.MCPClient, "list_tools_and_save", fake_list_tools)
monkeypatch.setattr(ftm.MCPClient, "cleanup", fake_cleanup)
await manager.enable_mcp_server("dummy", {"command": "python"}, timeout=5)
await manager.disable_mcp_server("dummy", timeout=5)
assert seen["cleanup_task"] is seen["connect_task"]
assert "dummy" not in manager.mcp_client_dict
@pytest.mark.asyncio
async def test_mcp_shutdown_cleanup_survives_late_cancellation(monkeypatch):
"""A cancellation arriving mid-cleanup must not abort the cleanup."""
manager = FunctionToolManager()
cleanup_calls = []
async def fake_connect(self, config, name):
pass
async def fake_list_tools(self):
self.tools = []
async def fake_cleanup(self):
cleanup_calls.append(asyncio.current_task())
if len(cleanup_calls) == 1:
raise asyncio.CancelledError()
monkeypatch.setattr(ftm.MCPClient, "connect_to_server", fake_connect)
monkeypatch.setattr(ftm.MCPClient, "list_tools_and_save", fake_list_tools)
monkeypatch.setattr(ftm.MCPClient, "cleanup", fake_cleanup)
await manager.enable_mcp_server("dummy", {"command": "python"}, timeout=5)
await manager.disable_mcp_server("dummy", timeout=5)
assert len(cleanup_calls) == 2
assert "dummy" not in manager.mcp_client_dict
@pytest.mark.asyncio
async def test_modelscope_sync_enables_only_synced_servers(monkeypatch):
class FakeResponse: