From 1124dfe2953b1fdd79eea370f449c5244bfca31f Mon Sep 17 00:00:00 2001 From: Soulter <905617992@qq.com> Date: Tue, 7 Jul 2026 00:26:03 +0800 Subject: [PATCH] docs: broaden diagnostics guide --- docs/en/others/diagnostics.md | 90 ++++++++++++++++++++-------------- docs/zh/others/diagnostics.md | 92 +++++++++++++++++++++-------------- 2 files changed, 109 insertions(+), 73 deletions(-) diff --git a/docs/en/others/diagnostics.md b/docs/en/others/diagnostics.md index f922e5e18..c4b34f463 100644 --- a/docs/en/others/diagnostics.md +++ b/docs/en/others/diagnostics.md @@ -1,18 +1,59 @@ # Diagnostics -When AstrBot appears to slow down or get stuck without an obvious error, start with the event loop diagnostics. The event loop schedules messages, plugins, scheduled jobs, model requests, and tool calls. If it is blocked by synchronous code or cannot resume for a long time, many features can look delayed at once. +This page provides a general checklist for diagnosing AstrBot issues. When something goes wrong, first identify which stage is affected, then collect the relevant logs. This makes issue reports easier to reproduce and investigate. -## Common Symptoms +## Common Issue Types + +- Startup failures: the process exits after startup, WebUI is unavailable, or database/config loading fails. +- Platform connection issues: QQ, OneBot, Telegram, WeCom, or other platforms cannot connect, receive messages, or send messages. +- Model request issues: replies take too long, requests time out, 429/5xx errors appear, or the proxy/API base is unavailable. +- Plugin or MCP issues: enabling fails, tool calls fail, dependencies fail to install, or an MCP service does not respond. +- Slow tasks or abnormal CPU usage: messages, proactive tasks, or scheduled tasks start but continue much later; or the process keeps unusually high CPU usage. + +## Logs to Check First + +Start with the main AstrBot log: + +```text +data/logs/astrbot.log +``` + +For Docker deployments, also check container logs: + +```bash +docker logs +``` + +If the issue involves slow tasks, abnormal CPU usage, or multiple sessions becoming slow at the same time, also check the event loop diagnostic logs: + +```text +data/logs/event_loop_watchdog.log +data/logs/event_loop_watchdog.log.1 +``` + +`event_loop_watchdog.log` rotates to `.1` after it exceeds 1 MB. + +## General Checklist + +1. Identify the time of the incident, then collect logs from 1 to 3 minutes before and after it. +2. Check the scope: all platforms, one platform, one group, one user, or one plugin. +3. For slow or missing model replies, check the model provider status, API key, API base, proxy, network, request timeout, and retry logs. +4. For plugin or MCP issues, disable recently installed or updated plugins first, then check plugin dependencies and MCP service logs. +5. For platform messaging issues, check whether the adapter is connected, platform-side settings are correct, and callback/WebSocket URLs are reachable. +6. For slow tasks or abnormal CPU usage, see "Event Loop Lag Diagnostics" below. + +## Event Loop Lag Diagnostics + +The event loop schedules messages, plugins, scheduled jobs, model requests, and tool calls. If it is blocked by synchronous code, many features can look delayed at once. + +Common symptoms: -- A group or private message reaches AstrBot, but processing continues tens of seconds or minutes later. - Logs stop after `ready to request llm provider`, `acquired session lock for llm request`, or a tool result, then continue much later. - A proactive or scheduled task starts, but one step in the middle takes a long time to continue. -- Multiple platforms or sessions become slow at the same time, instead of only one user's request being slow. -- CPU usage stays unusually high, or CPU usage is low but requests wait a long time for an external service. +- Multiple platforms or sessions become slow at the same time. +- CPU usage stays at 100%, or CPU usage is low but requests keep waiting for an external service. -## Diagnostic Logs - -AstrBot logs event loop lag. If the lag exceeds the threshold, the main log contains an entry like: +If the main log contains the following entry, the event loop experienced visible scheduling delay: ```text Event loop lag detected: 18.432s (threshold 15.000s). @@ -24,31 +65,7 @@ If the event loop does not resume for a long time, AstrBot writes Python thread data/logs/event_loop_watchdog.log ``` -When the file exceeds 1 MB, it rotates to: - -```text -data/logs/event_loop_watchdog.log.1 -``` - -Also check the main log: - -```text -data/logs/astrbot.log -``` - -For Docker deployments, you can also run: - -```bash -docker logs -``` - -## How to Investigate - -1. First check whether `Event loop lag detected` appears in the logs. If it does, AstrBot's main event loop experienced visible scheduling delay. -2. Open `data/logs/event_loop_watchdog.log` and inspect the top frames. Useful clues often include plugin functions, platform adapters, MCP tools, synchronous network requests, `time.sleep()`, `subprocess.run()`, or CPU-heavy loops. -3. If there is no event loop lag log but one conversation still waits for a long time, check the model provider, proxy, network, tool timeout, or MCP server response time first. -4. If only one session is stuck, a previous request in that session may still be running. If all sessions are stuck, the event loop is more likely blocked. -5. If CPU stays at 100%, focus on the watchdog stack and MCP/plugin calls. If CPU is low, the process is more likely waiting for an external network or model service. +When reading this file, focus on the top frames. Useful clues often include plugin functions, platform adapters, MCP tools, synchronous network requests, `time.sleep()`, `subprocess.run()`, or CPU-heavy loops. ## What to Include in an Issue @@ -56,10 +73,11 @@ When filing an issue, include as much of the following as possible: - Approximate time of the incident and timezone. - AstrBot version, deployment method (Docker, manual deployment, desktop client, etc.), and operating system. -- Trigger path: normal chat, group chat, scheduled task, MCP tool, plugin feature, etc. +- Trigger path: startup, normal chat, group chat, platform callback, scheduled task, MCP tool, plugin feature, etc. +- Scope: all sessions, one platform, one group, one user, or one plugin. - Logs from `data/logs/astrbot.log` for 1 to 3 minutes around the incident. -- `data/logs/event_loop_watchdog.log` and `data/logs/event_loop_watchdog.log.1` if they exist. -- For Docker deployments, the matching `docker logs` output. +- If the issue involves lag or abnormal CPU usage, include `data/logs/event_loop_watchdog.log` and `data/logs/event_loop_watchdog.log.1`. +- For Docker deployments, include the matching `docker logs` output. - Installed plugin list, and whether the issue still happens after disabling third-party plugins. Before sharing logs, redact API keys, tokens, cookies, private chat content, and other sensitive information. diff --git a/docs/zh/others/diagnostics.md b/docs/zh/others/diagnostics.md index 28bc4ee71..ab9a2c65b 100644 --- a/docs/zh/others/diagnostics.md +++ b/docs/zh/others/diagnostics.md @@ -1,18 +1,59 @@ # 异常诊断 -当 AstrBot 出现“看起来没有报错，但明显变慢或卡住”的情况时，可以先从事件循环诊断日志入手。事件循环是 AstrBot 调度消息、插件、定时任务、模型请求和工具调用的核心；如果它被同步阻塞或长期无法恢复，很多功能都会表现为延迟。 +本文用于整理 AstrBot 出现异常时的通用排查方法。遇到问题时，先确定问题发生在哪个阶段，再收集对应日志；这样提交 Issue 时更容易复现和定位。 -## 常见现象 +## 常见问题类型 -- 群聊或私聊消息已经进入 AstrBot，但过了几十秒甚至几分钟才继续处理。 -- 日志停在 `ready to request llm provider`、`acquired session lock for llm request`、工具调用结果之后，很久才出现下一条 Agent 日志。 -- 主动任务、定时任务已经触发，但中间某一步迟迟不继续。 -- 多个平台或多个会话同时变慢，而不是只有单个用户的请求慢。 -- 进程 CPU 异常升高，或 CPU 不高但请求长时间等待外部服务返回。 +- 启动失败：进程启动后退出、WebUI 打不开、数据库或配置加载失败。 +- 平台连接异常：QQ、OneBot、Telegram、企业微信等平台无法连接、收不到消息或无法发送消息。 +- 模型请求异常：长时间无回复、频繁超时、报 429/5xx、代理或 API Base 不可用。 +- 插件或 MCP 异常：启用失败、工具调用失败、依赖安装失败、MCP 服务无响应。 +- 任务卡顿或 CPU 异常：消息、主动任务、定时任务已经触发，但中间步骤很久才继续；或进程 CPU 长时间异常升高。 -## 查看诊断日志 +## 先看哪些日志 -AstrBot 会记录事件循环延迟。如果延迟超过阈值，主日志中会出现类似日志： +优先查看 AstrBot 主日志： + +```text +data/logs/astrbot.log +``` + +如果使用 Docker 部署，也可以查看容器日志： + +```bash +docker logs +``` + +如果问题和任务卡顿、CPU 异常、多个会话同时变慢有关，还要查看事件循环诊断日志： + +```text +data/logs/event_loop_watchdog.log +data/logs/event_loop_watchdog.log.1 +``` + +`event_loop_watchdog.log` 超过 1MB 后会轮转为 `.1`。 + +## 通用排查步骤 + +1. 确认问题发生的时间点，并截取该时间前后 1 到 3 分钟的日志。 +2. 确认问题范围：是所有平台都异常，还是只有某个平台、某个群、某个用户、某个插件异常。 +3. 如果是模型无回复或很慢，检查模型服务商状态、API Key、API Base、代理、网络、请求超时和重试日志。 +4. 如果是插件或 MCP 问题，先禁用最近安装或更新的插件，观察问题是否消失；同时检查插件依赖和 MCP 服务日志。 +5. 如果是平台收发消息异常，检查平台适配器是否已连接、平台后台配置是否正确、回调地址或 WebSocket 地址是否可访问。 +6. 如果是卡顿或 CPU 异常，参考下方“事件循环卡顿诊断”。 + +## 事件循环卡顿诊断 + +事件循环负责调度消息、插件、定时任务、模型请求和工具调用。如果它被同步代码阻塞，很多功能都会表现为延迟。 + +常见现象： + +- 日志停在 `ready to request llm provider`、`acquired session lock for llm request`、工具调用结果之后，很久才继续。 +- 主动任务或定时任务已经触发，但中间某一步迟迟不继续。 +- 多个平台或多个会话同时变慢。 +- CPU 长时间 100%，或 CPU 不高但请求一直等待外部服务返回。 + +如果主日志出现以下内容，说明事件循环经历了明显延迟： ```text Event loop lag detected: 18.432s (threshold 15.000s). @@ -24,41 +65,18 @@ Event loop lag detected: 18.432s (threshold 15.000s). data/logs/event_loop_watchdog.log ``` -该文件超过 1MB 后会轮转为： - -```text -data/logs/event_loop_watchdog.log.1 -``` - -你也可以同时查看主日志： - -```text -data/logs/astrbot.log -``` - -如果使用 Docker 部署，也可以用： - -```bash -docker logs -``` - -## 排查思路 - -1. 先确认是否有 `Event loop lag detected` 日志。如果有，说明 AstrBot 主事件循环确实经历了明显延迟。 -2. 查看 `data/logs/event_loop_watchdog.log`，关注栈顶附近正在执行的代码。常见线索包括插件函数、平台适配器、MCP 工具、同步网络请求、`time.sleep()`、`subprocess.run()`、CPU 密集循环等。 -3. 如果没有事件循环延迟日志，但某次对话仍然卡很久，优先检查模型服务商、代理、网络、工具调用超时、MCP 服务响应时间。 -4. 如果只有某个会话卡住，可能是该会话内前一个请求还没有结束；如果所有会话都卡住，更可能是事件循环被阻塞。 -5. 如果 CPU 长时间 100%，优先关注 watchdog 栈和 MCP/插件相关调用；如果 CPU 很低，更常见的是等待外部网络或模型服务返回。 +查看该文件时，重点关注栈顶附近正在执行的代码。常见线索包括插件函数、平台适配器、MCP 工具、同步网络请求、`time.sleep()`、`subprocess.run()`、CPU 密集循环等。 ## 提交 Issue 时请附带 -提交问题时，请尽量提供以下信息，方便定位： +提交问题时，请尽量提供以下信息： - 问题发生的大致时间点和时区。 - AstrBot 版本、部署方式（Docker、手动部署、桌面客户端等）、操作系统。 -- 触发方式：普通聊天、群聊、定时任务、MCP 工具、插件功能等。 +- 触发方式：启动、普通聊天、群聊、平台回调、定时任务、MCP 工具、插件功能等。 +- 影响范围：所有会话、某个平台、某个群、某个用户，还是某个插件。 - `data/logs/astrbot.log` 中问题发生前后 1 到 3 分钟的日志。 -- `data/logs/event_loop_watchdog.log` 和 `data/logs/event_loop_watchdog.log.1`（如果存在）。 +- 如果存在卡顿或 CPU 异常，请附带 `data/logs/event_loop_watchdog.log` 和 `data/logs/event_loop_watchdog.log.1`。 - 如果使用 Docker，请附带对应时间段的 `docker logs`。 - 已安装插件列表，以及问题是否在禁用第三方插件后仍然出现。