docs: add reliability monitoring implementation plan

.gitignore

@@ -26,3 +26,4 @@ backend/.env
 .DS_Store
 Thumbs.db
 .claude/
+.worktrees/

docs/superpowers/plans/2026-08-03-reliable-ping-alerting.md

@@ -0,0 +1,465 @@
+# PingWatch 可靠连通性监测与企业微信告警 Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.
+
+**Goal:** 将 PingWatch 改造成可区分离线与间歇性丢包故障、可靠发送企业微信应用通知，并能在 Ubuntu/OpenResty 环境安全运行的监测服务。
+
+**Architecture:** fping 每轮为每个设备取得多包汇总，独立状态机依据连续轮次和滑动窗口计算 online/degraded/offline。状态转换在同一数据库事务内创建事件与通知出箱；独立投递器重试企业微信消息。OpenResty 在宿主机提供 Vue 构建文件并反向代理仅绑定回环地址的 FastAPI 服务。
+
+**Tech Stack:** Python 3.12、FastAPI、SQLAlchemy asyncio、PostgreSQL 16、fping、httpx、pytest、Vue 3、Element Plus、OpenResty、Docker Compose。
+
+## Global Constraints
+
+- 目标能力等级暂按 C1；所有需求使用 PINGWATCH_(可靠性监测)_001 追溯。
+- 默认检测周期为 30 秒，每轮 3 包；离线为连续 2 轮 100% 丢包，故障为最近 5 轮至少 20% 丢包，恢复为连续 3 轮零丢包。
+- 只监测显式录入且 is_enabled=true 的 IPv4/IPv6 地址；不得增加网段发现或端口扫描。
+- 凭据仅通过运行时环境变量注入；Git、镜像、日志、API 响应和文档不得包含 secret、token 或口令。
+- 企业微信默认向应用可见范围发送；不提供系统页面维护接收人。可选 WECOM_TO_PARTY 只允许由受控环境变量设置。
+- 后端仅以 127.0.0.1:8001 暴露给 Ubuntu 宿主机 OpenResty；PostgreSQL 不映射宿主机端口。
+- 不可信输入必须经 Pydantic/显式校验，fping 只通过参数数组启动，禁止 shell=True。
+- 除非命令明确指定其他目录，所有 Python pytest 命令均从 backend/ 目录运行。
+
+---
+
+## Planned File Structure
+
+| 文件 | 责任 |
+| --- | --- |
+| backend/app/models/{device,ping_record,alert_event,notification_outbox}.py | 设备策略、探测汇总、领域事件及通知出箱持久化。 |
+| backend/app/services/{fping_runner,state_machine,pinger,notification_dispatcher,alerter}.py | 命令输出解析、状态计算、轮次编排、可靠投递和事件创建。 |
+| backend/app/api/{devices,alerts,health}.py 与 backend/app/schemas/* | 受保护设备/事件接口和不含敏感信息的健康探针。 |
+| backend/tests/* | 状态、解析、投递、API、配置和迁移验证。 |
+| frontend/src/views/{Devices,Alerts,Dashboard}.vue | 健康度、探测质量、事件和投递结果展示。 |
+| deploy/openresty、deploy/systemd、docs/operations | OpenResty、受控启动、部署、回退、备份运行材料。 |
+
+## Task 1: 建立测试与运行配置契约
+
+**Files:**
+- Modify: backend/requirements.txt, backend/app/config.py, .gitignore
+- Create: backend/.env.example, backend/pytest.ini, backend/app/services/settings_validation.py
+- Create: backend/tests/conftest.py, backend/tests/test_settings_validation.py
+
+**Interfaces:**
+- Produces: validate_runtime_settings(settings: Settings) -> None.
+- Produces: monitoring defaults and bounded retry/WeCom configuration fields.
+
+- [ ] **Step 1: Write the failing configuration tests**
+
+~~~python
+import pytest
+from pydantic import ValidationError
+from app.config import Settings
+from app.services.settings_validation import validate_runtime_settings
+
+def test_production_rejects_legacy_jwt_secret():
+    settings = Settings(environment="production", secret_key="change-me-to-a-long-random-string")
+    with pytest.raises(ValueError, match="SECRET_KEY"):
+        validate_runtime_settings(settings)
+
+def test_probe_packet_count_must_be_positive():
+    with pytest.raises(ValidationError):
+        Settings(probe_packets_per_round=0)
+~~~
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: python -m pytest backend/tests/test_settings_validation.py -v  
+Expected: FAIL because settings fields and validation module are absent.
+
+- [ ] **Step 3: Write minimal configuration implementation**
+
+Add pinned test dependencies pytest, pytest-asyncio and aiosqlite. Add Settings fields: environment, probe_packets_per_round=3 (1..10), offline_consecutive_rounds=2 (1..10), degraded_window_rounds=5 (2..60), degraded_loss_percent=20.0 (1..100), recovery_consecutive_clean_rounds=3 (1..20), wecom_notification_max_attempts, wecom_retry_base_seconds and WECOM_TO_PARTY. validate_runtime_settings rejects production default JWT keys, production TLS-disabled Casdoor, and enabled WeCom delivery without all three credentials.
+
+~~~python
+def validate_runtime_settings(settings: Settings) -> None:
+    if settings.environment == "production" and settings.secret_key == LEGACY_DEFAULT_SECRET:
+        raise ValueError("SECRET_KEY must be provided by the runtime environment")
+~~~
+
+Create backend/.env.example with empty secret values only; keep real .env ignored.
+
+- [ ] **Step 4: Run focused verification**
+
+Run: python -m pytest backend/tests/test_settings_validation.py -v  
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+~~~bash
+git add backend/requirements.txt backend/pytest.ini backend/.env.example .gitignore backend/app/config.py backend/app/services/settings_validation.py backend/tests
+git commit -m "test: establish monitoring configuration contract"
+~~~
+
+## Task 2: 持久化探测质量、设备策略、事件与通知出箱
+
+**Files:**
+- Modify: backend/app/models/device.py, backend/app/models/ping_record.py, backend/app/models/alert_event.py, backend/app/models/__init__.py, backend/app/core/deps.py
+- Create: backend/app/models/notification_outbox.py, backend/migrations/versions/20260803_reliability_monitoring.py
+- Create: backend/tests/test_models_and_migration.py
+
+**Interfaces:**
+- Produces: DeviceMonitoringPolicy.from_device(device, settings) -> DeviceMonitoringPolicy.
+- Produces: PingRecord.sent_count, received_count, packet_loss_percent, average_rtt_ms, is_valid, failure_reason.
+- Produces: NotificationOutbox with status pending, sending, sent, failed.
+
+- [ ] **Step 1: Write failing persistence tests**
+
+~~~python
+async def test_probe_record_stores_packet_summary(db_session):
+    record = PingRecord(device_id=1, sent_count=3, received_count=2,
+                        packet_loss_percent=33.33, average_rtt_ms=12.5, is_valid=True)
+    db_session.add(record)
+    await db_session.commit()
+    assert record.packet_loss_percent == 33.33
+
+async def test_outbox_defaults_to_pending(db_session):
+    item = NotificationOutbox(alert_event_id=1, message_content="masked summary")
+    db_session.add(item)
+    await db_session.commit()
+    assert item.status == NotificationStatus.pending
+~~~
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: python -m pytest backend/tests/test_models_and_migration.py -v  
+Expected: FAIL because fields and NotificationOutbox do not exist.
+
+- [ ] **Step 3: Implement models and migration**
+
+Extend Device with nullable per-device policy overrides and allow degraded status. Add an index on probe (device_id, created_at), and a status/next_attempt_at index on outbox. Add AlertTypeEnum.degraded and fields previous_status, current_status, last_notification_error, notification_attempts. Implement one idempotent migration runner called before create_all; it records revision 20260803_reliability_monitoring in schema_migrations and never deletes existing data.
+
+- [ ] **Step 4: Run focused verification**
+
+Run: python -m pytest backend/tests/test_models_and_migration.py -v  
+Expected: PASS, including two sequential migration executions against the same database.
+
+- [ ] **Step 5: Commit**
+
+~~~bash
+git add backend/app/models backend/app/core/deps.py backend/migrations backend/tests/test_models_and_migration.py
+git commit -m "feat: persist probe quality and notification outbox"
+~~~
+
+## Task 3: 实现多包 fping 解析和纯状态机
+
+**Files:**
+- Create: backend/app/services/fping_runner.py, backend/app/services/state_machine.py
+- Modify: backend/app/services/pinger.py
+- Create: backend/tests/test_fping_runner.py, backend/tests/test_state_machine.py
+
+**Interfaces:**
+- Produces: ProbeResult(ip, sent_count, received_count, average_rtt_ms, is_valid, failure_reason).
+- Produces: parse_fping_count_output(output, expected_ips, packets_per_round) -> dict[str, ProbeResult].
+- Produces: evaluate_health(previous_status, recent, policy) -> StateDecision.
+
+- [ ] **Step 1: Write failing parser and transition tests**
+
+~~~python
+def test_parses_loss_and_average_rtt():
+    output = "10.0.0.8 : 12.4 - 12.6\n10.0.0.9 : - - -"
+    result = parse_fping_count_output(output, {"10.0.0.8", "10.0.0.9"}, 3)
+    assert result["10.0.0.8"].received_count == 2
+    assert result["10.0.0.9"].packet_loss_percent == 100.0
+
+def test_two_full_loss_rounds_take_device_offline(policy):
+    decision = evaluate_health("online", [loss(3), loss(3)], policy)
+    assert (decision.next_status, decision.event_type) == ("offline", "offline")
+
+def test_five_round_window_with_loss_is_degraded(policy):
+    decision = evaluate_health("online", [partial(3, 2)] * 5, policy)
+    assert decision.next_status == "degraded"
+
+def test_three_clean_rounds_recovers(policy):
+    decision = evaluate_health("offline", [clean(3)] * 3, policy)
+    assert (decision.next_status, decision.event_type) == ("online", "recovered")
+~~~
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: python -m pytest backend/tests/test_fping_runner.py backend/tests/test_state_machine.py -v  
+Expected: FAIL because runner and state machine modules are absent.
+
+- [ ] **Step 3: Implement minimal parser and state machine**
+
+Invoke fping with the argument sequence fping -C <count> -q -t <milliseconds> <ip...>; capture stdout and stderr because quiet count output is emitted to stderr. Numeric reply tokens count as success and dash tokens as loss. Missing/malformed expected IP output creates is_valid=False and cannot trigger a device failure.
+
+State priority: invalid records retain state; configured consecutive full loss gives offline; configured aggregate window loss gives degraded; only offline/degraded plus configured clean records gives online; otherwise retain state. StateDecision contains next_status, event_type, should_notify, reason.
+
+- [ ] **Step 4: Refactor Pinger to persist exactly one summary per enabled device**
+
+Pinger.run_one_round saves every ProbeResult, reads the required recent valid records, computes StateDecision, updates Device, and returns DeviceStateChange values within one transaction. It must use asyncio.create_subprocess_exec(*command) and never a command string.
+
+- [ ] **Step 5: Run focused verification**
+
+Run: python -m pytest backend/tests/test_fping_runner.py backend/tests/test_state_machine.py -v  
+Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+~~~bash
+git add backend/app/services/fping_runner.py backend/app/services/state_machine.py backend/app/services/pinger.py backend/tests/test_fping_runner.py backend/tests/test_state_machine.py
+git commit -m "feat: classify offline and degraded connectivity"
+~~~
+
+## Task 4: 用事务出箱可靠投递企业微信事件
+
+**Files:**
+- Create: backend/app/services/notification_dispatcher.py
+- Modify: backend/app/services/alerter.py, backend/app/services/scheduler.py, backend/app/services/pinger.py
+- Create: backend/tests/test_notification_dispatcher.py, backend/tests/test_alert_workflow.py
+
+**Interfaces:**
+- Consumes: list[DeviceStateChange] from Pinger.run_one_round().
+- Produces: Alerter.record_transition(change, db) -> AlertEvent.
+- Produces: NotificationDispatcher.dispatch_due(db, now) -> DispatchSummary.
+- Produces: WeComClient.send_text(content) -> DeliveryResult.
+
+- [ ] **Step 1: Write failing outbox tests**
+
+~~~python
+async def test_transition_creates_event_and_pending_outbox_in_one_commit(db_session, change):
+    event = await Alerter().record_transition(change, db_session)
+    outbox = await db_session.scalar(select(NotificationOutbox).where(
+        NotificationOutbox.alert_event_id == event.id))
+    assert outbox.status == NotificationStatus.pending
+
+async def test_retryable_failure_reschedules(db_session, pending_message, frozen_time):
+    client = FakeWeComClient(DeliveryResult(False, "timeout", True))
+    await NotificationDispatcher(client).dispatch_due(db_session, frozen_time)
+    assert pending_message.status == NotificationStatus.pending
+    assert pending_message.attempt_count == 1
+~~~
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: python -m pytest backend/tests/test_notification_dispatcher.py backend/tests/test_alert_workflow.py -v  
+Expected: FAIL because dispatcher interfaces are absent.
+
+- [ ] **Step 3: Implement atomic event/outbox recording**
+
+Replace in-memory pending alerts. record_transition inserts AlertEvent and NotificationOutbox before caller commit. Offline escalation annotates/resolves the prior open degraded event. Recovery closes one unresolved event, stores duration, and creates a recovered notification. Render notifications with event type, device, IP, location, project, loss summary, time and duration; split batches at a fixed safe count.
+
+- [ ] **Step 4: Implement WeCom client and retry dispatcher**
+
+Use httpx.AsyncClient with explicit connect/read/write timeout. Cache access_token until server expiry minus safety margin. Default payload uses touser="@all"; only use toparty when WECOM_TO_PARTY is non-empty. Never log token-bearing URLs, request JSON, response body, secret, or message content. Retry transport/timeouts, 429 and 5xx with next_attempt_at = now + base_seconds * 2 ** (attempt_count - 1); mark failed at configured maximum.
+
+- [ ] **Step 5: Connect dispatcher to scheduler**
+
+Run dispatch_due after a successful probe transaction and at a small bounded interval. The scheduler has one task and a lock; cancellation waits for the active probe/dispatch cycle to close its session.
+
+- [ ] **Step 6: Run verification and commit**
+
+Run: python -m pytest backend/tests/test_notification_dispatcher.py backend/tests/test_alert_workflow.py -v  
+Expected: PASS.
+
+~~~bash
+git add backend/app/services/notification_dispatcher.py backend/app/services/alerter.py backend/app/services/scheduler.py backend/app/services/pinger.py backend/tests/test_notification_dispatcher.py backend/tests/test_alert_workflow.py
+git commit -m "feat: deliver alert events through persistent outbox"
+~~~
+
+## Task 5: 校验设备输入并提供健康/审计 API
+
+**Files:**
+- Modify: backend/app/schemas/device.py, backend/app/schemas/alert.py, backend/app/api/devices.py, backend/app/api/alerts.py, backend/app/main.py
+- Create: backend/app/api/health.py
+- Create: backend/tests/test_devices_api.py, backend/tests/test_alerts_api.py, backend/tests/test_health_api.py
+
+**Interfaces:**
+- Produces: DeviceCreate and DeviceUpdate using IPvAnyAddress and bounded policy fields.
+- Produces: GET /api/health -> {"status": "ok", "database": "ready"} after SELECT 1.
+- Produces: alert DTO transition and notification-status fields.
+
+- [ ] **Step 1: Write failing API tests**
+
+~~~python
+async def test_create_rejects_command_like_ip(authenticated_client):
+    response = await authenticated_client.post("/api/devices",
+        json={"name": "bad", "ip": "127.0.0.1;id"})
+    assert response.status_code == 422
+
+async def test_health_requires_database_readiness(client):
+    response = await client.get("/api/health")
+    assert response.json() == {"status": "ok", "database": "ready"}
+~~~
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: python -m pytest backend/tests/test_devices_api.py backend/tests/test_alerts_api.py backend/tests/test_health_api.py -v  
+Expected: FAIL on missing validation/health behavior.
+
+- [ ] **Step 3: Implement schema and import validation**
+
+Use IPvAnyAddress, Field bounds and field_validator to map blank overrides to None. Before decoding CSV enforce CSV_MAX_BYTES; reject invalid UTF-8 as 400; limit returned line errors to 100; de-duplicate both database IPs and earlier accepted CSV rows. Device and alert responses expose only aggregate notification status, never message content or external error body.
+
+- [ ] **Step 4: Implement health and API projections**
+
+Health router runs SELECT 1. The lifespan calls validate_runtime_settings before scheduling. CORS accepts only exact configured origins. Device lists return latest valid summary and policy fields; alerts return degraded/recovered filters, transition and retry fields.
+
+- [ ] **Step 5: Run verification and commit**
+
+Run: python -m pytest backend/tests/test_devices_api.py backend/tests/test_alerts_api.py backend/tests/test_health_api.py -v  
+Expected: PASS.
+
+~~~bash
+git add backend/app/schemas backend/app/api backend/app/main.py backend/tests/test_devices_api.py backend/tests/test_alerts_api.py backend/tests/test_health_api.py
+git commit -m "feat: expose validated monitoring health and event APIs"
+~~~
+
+## Task 6: 更新前端运维视图
+
+**Files:**
+- Modify: frontend/package.json, frontend/package-lock.json, frontend/src/api/index.js
+- Modify: frontend/src/views/Devices.vue, frontend/src/views/Alerts.vue, frontend/src/views/Dashboard.vue
+- Create: frontend/src/utils/status.js, frontend/src/utils/status.test.js
+
+**Interfaces:**
+- Produces: statusLabel(status) and statusTagType(status) supporting online, degraded, offline, unknown.
+- Consumes: enhanced Task 5 device and alert DTOs.
+
+- [ ] **Step 1: Write a failing presentation test**
+
+~~~javascript
+import { describe, expect, it } from "vitest"
+import { statusLabel, statusTagType } from "./status"
+
+describe("status presentation", () => {
+  it("marks degraded connectivity as a warning", () => {
+    expect(statusLabel("degraded")).toBe("故障")
+    expect(statusTagType("degraded")).toBe("warning")
+  })
+})
+~~~
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: npm --prefix frontend run test -- --run  
+Expected: FAIL because the test script is absent.
+
+- [ ] **Step 3: Implement minimal UI contract**
+
+Add vitest development dependency and test script. Add status utility with explicit mappings. Devices view adds state, packet loss, RTT and last probe columns plus bounded policy form fields. Alerts view adds degraded filter, transition, delivery status and retry count. Dashboard adds degraded count. Use Vue interpolation exclusively; never use v-html for device data.
+
+- [ ] **Step 4: Run verification and commit**
+
+Run: npm --prefix frontend run test -- --run  
+Expected: PASS.
+
+Run: npm --prefix frontend run build  
+Expected: PASS.
+
+~~~bash
+git add frontend/package.json frontend/package-lock.json frontend/src/api/index.js frontend/src/utils frontend/src/views/Devices.vue frontend/src/views/Alerts.vue frontend/src/views/Dashboard.vue
+git commit -m "feat: display degraded connectivity and delivery state"
+~~~
+
+## Task 7: 加固认证、容器暴露与 Ubuntu/OpenResty 交付
+
+**Files:**
+- Modify: backend/app/core/auth.py, backend/Dockerfile, docker-compose.yml
+- Create: deploy/openresty/pingwatch.conf, deploy/systemd/pingwatch.service
+- Create: docs/operations/ubuntu-openresty-deployment.md, docs/operations/rollback-and-backup.md
+- Create: backend/tests/test_auth_security.py, deploy/tests/test_compose_exposure.ps1
+
+**Interfaces:**
+- Produces: exchange_code_for_user(code) that never accepts unverified JWT claims in production.
+- Produces: OpenResty routes for /, /api/, /ws and /api/health.
+
+- [ ] **Step 1: Write failing security/deployment assertions**
+
+~~~python
+async def test_production_rejects_unverified_casdoor_token(monkeypatch):
+    monkeypatch.setattr(settings, "environment", "production")
+    monkeypatch.setattr(settings, "casdoor_certificate", "")
+    assert await exchange_code_for_user("code") is None
+~~~
+
+~~~powershell
+$compose = Get-Content -Raw .\docker-compose.yml
+if ($compose -match 'POSTGRES_PASSWORD=') { throw 'tracked compose contains password' }
+if ($compose -match 'NET_ADMIN') { throw 'container has excess capability' }
+if ($compose -notmatch '127.0.0.1:8001:8000') { throw 'backend is not loopback-bound' }
+~~~
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: python -m pytest backend/tests/test_auth_security.py -v  
+Expected: FAIL because OAuth disables TLS verification or accepts unverified claims.
+
+Run: pwsh -File deploy/tests/test_compose_exposure.ps1  
+Expected: FAIL because Compose contains an embedded password and NET_ADMIN.
+
+- [ ] **Step 3: Implement minimal hardening**
+
+Require certificate/TLS verification before accepting production Casdoor id_token; remove unverified-claims fallback. Compose reads database credentials only from server-only environment file, exposes backend as 127.0.0.1:8001:8000, does not run frontend container in production, does not expose database, uses cap_drop ALL plus cap_add NET_RAW and no-new-privileges. Backend image has non-secret health check.
+
+- [ ] **Step 4: Add OpenResty and operating material**
+
+OpenResty listens on the agreed IP port, serves /opt/pingwatch/frontend, uses SPA try_files, proxies /api/ to 127.0.0.1:8001 and upgrades /ws, limits request body to 15m, and has separate access/error logs. Deployment guide gives preflight, server-only env permissions, build/copy, Compose lifecycle, OpenResty test/reload, health/API/WebSocket smoke tests and approved rollback. Backup guide defines PostgreSQL volume backup before upgrades and restore of prior frontend build/image.
+
+- [ ] **Step 5: Run verification and commit**
+
+Run: python -m pytest backend/tests/test_auth_security.py -v  
+Expected: PASS.
+
+Run: pwsh -File deploy/tests/test_compose_exposure.ps1  
+Expected: PASS.
+
+Run: docker compose config  
+Expected: PASS with no database port and no plaintext secret in tracked Compose file.
+
+~~~bash
+git add backend/app/core/auth.py backend/Dockerfile docker-compose.yml deploy docs/operations backend/tests/test_auth_security.py
+git commit -m "feat: secure OpenResty deployment package"
+~~~
+
+## Task 8: 全量验证、追溯证据与发布准备
+
+**Files:**
+- Create: docs/traceability/2026-08-03-reliability-monitoring.md
+- Create: docs/test-reports/2026-08-03-reliability-monitoring.md
+- Create: README.md
+
+**Interfaces:**
+- Produces: each of five requirement IDs mapped to code, test and Ubuntu/OpenResty validation evidence.
+
+- [ ] **Step 1: Execute full automated verification**
+
+Run: python -m pytest backend/tests -v from backend/  
+Expected: PASS.
+
+Run: npm --prefix frontend run test -- --run  
+Expected: PASS.
+
+Run: npm --prefix frontend run build  
+Expected: PASS.
+
+Run: docker compose config  
+Expected: PASS.
+
+- [ ] **Step 2: Execute local Compose smoke test**
+
+Run: docker compose up --build -d, then curl -fsS http://127.0.0.1:8001/api/health, then docker compose down.  
+Expected: health contains status=ok and database=ready; no test data or secret file is committed.
+
+- [ ] **Step 3: Write traceability/test reports**
+
+Traceability maps requirements _01 through _05 to implementation file, exact automated test name and Ubuntu/OpenResty validation command. Test report records command, timestamp, result, limitations, and states that real network validation needs authorized test IPs and configured enterprise WeChat credentials. It excludes credentials, private IP inventories and full notification contents.
+
+- [ ] **Step 4: Scan final changes and commit evidence**
+
+Run: git diff main...HEAD --check  
+Expected: PASS.
+
+Run: rg -n '(SECRET|PASSWORD|TOKEN|PRIVATE KEY)\s*=\s*[^" ]+' -g '!*.example' -g '!docs/superpowers/**' .  
+Expected: no hardcoded secret assignment.
+
+~~~bash
+git add README.md docs/traceability docs/test-reports
+git commit -m "docs: add reliability monitoring verification evidence"
+~~~
+
+## Self-Review
+
+- Spec coverage: Tasks 1-4 implement multi-packet probing, state transitions, durable notification, recovery and batch/node safety; Tasks 5-6 implement API/UI; Task 7 implements OpenResty, secrets, authentication and container controls; Task 8 provides C1 deployment evidence and traceability.
+- Placeholder scan: the plan contains executable commands, named files, test behavior and data contracts for each task.
+- Type consistency: ProbeResult, DeviceMonitoringPolicy, StateDecision, DeviceStateChange, NotificationOutbox and DeliveryResult are defined before their consuming tasks.

docs/superpowers/specs/2026-08-03-pingwatch-reliability-design.md

@@ -0,0 +1,119 @@
+# PingWatch 连通性监测与企业微信告警改造设计
+
+**日期：** 2026-08-03  
+**状态：** 已确认，待实现  
+**需求编号：** `PINGWATCH_(可靠性监测)_001`  
+**目标能力等级：** 暂按 C1 基线，待项目负责人确认
+
+## 1. 目标、范围与验收
+
+### 1.1 目标
+
+将现有的批量 ICMP 连通性检测改造成可区分离线与业务故障的监测系统，并向企业微信应用可见部门发送及时、可追溯的通知。
+
+| 编号 | 可验证需求 | 验收条件 |
+| --- | --- | --- |
+| `PINGWATCH_(可靠性监测)_001_01` | 及时发现离线设备 | 每 30 秒检测一轮，每轮 3 个 ICMP 包；连续 2 轮 100% 丢包进入 `offline` 并创建/通知事件。 |
+| `PINGWATCH_(可靠性监测)_001_02` | 识别影响业务的间歇性丢包 | 最近 5 轮（15 包）丢包率不低于 20%、但未离线时进入 `degraded` 并通知。 |
+| `PINGWATCH_(可靠性监测)_001_03` | 发现设备恢复 | 已告警设备连续 3 轮零丢包后进入 `online`，关闭关联事件并通知恢复。 |
+| `PINGWATCH_(可靠性监测)_001_04` | 通知可靠可追溯 | 企业微信调用失败时事件保留，按有限退避重试；每次投递结果可查询。 |
+| `PINGWATCH_(可靠性监测)_001_05` | Ubuntu/OpenResty 运行 | 以 `http://10.10.10.14:<port>` 提供访问；OpenResty 承载静态站点并反代 API/WebSocket。 |
+
+### 1.2 非目标
+
+- 不增加未授权网段扫描、端口扫描或资产发现。
+- 不在首期实现按历史基线自动学习阈值。
+- 不自建企业微信成员/部门管理，应用可见范围及接收部门由企业微信管理端控制。
+
+### 1.3 数据分类与风险
+
+- 设备名称、IP、位置、项目和告警记录按内部运维数据处理；企业微信密钥、JWT 密钥、数据库口令和 OAuth 凭据为敏感配置。
+- 主要风险：网络抖动误告警、监控节点自身断网、企业微信不可用、存储膨胀、特权过大、凭据泄露。
+
+## 2. 方案比较与决策
+
+| 方案 | 优点 | 缺点 | 决策 |
+| --- | --- | --- | --- |
+| 单包连续失败阈值 | 简单 | 无法量化间歇性丢包 | 不采用 |
+| 多包探测 + 滑动窗口状态机 | 规则可解释、及时且抗抖动 | 需要保存汇总数据 | **采用** |
+| 自适应历史基线 | 对不同链路更精细 | 学习期、复杂度和解释成本高 | 后续评估 |
+
+## 3. 架构与数据流
+
+```text
+受管 IP → fping 批量探测 → 轮次汇总/状态机 → 事件出箱 → 企业微信应用（应用可见范围）
+                              │                   │
+                              ├→ PostgreSQL ←──────┤
+                              └→ API/WebSocket → OpenResty → 浏览器（IP:端口）
+```
+
+1. 调度器不允许检测轮次重叠；每 30 秒加载启用设备并批量执行 `fping`。
+2. 一轮记录每台设备的发包数、收包数、丢包率、平均 RTT 与结果有效性。命令或解析异常产生系统事件，不能被当作设备丢包。
+3. 状态机基于最新有效检测记录计算状态；状态改变时原子地写入领域事件和待投递记录。
+4. 投递 worker 从数据库读取待发送记录，向企业微信应用发送应用消息；由企业微信应用的可见范围限定接收部门。成功、失败、下次重试时间均持久化。
+5. 前端通过 API/WebSocket 展示设备当前健康度、最近丢包率、时延与事件投递结果。
+
+## 4. 状态机与告警规则
+
+| 现状态 | 条件 | 新状态 | 动作 |
+| --- | --- | --- | --- |
+| `unknown` / `online` | 连续 2 轮均 100% 丢包 | `offline` | 创建离线事件，进入待通知队列。 |
+| `unknown` / `online` | 最近 5 轮丢包率 ≥20%，且不满足离线 | `degraded` | 创建业务故障事件，进入待通知队列。 |
+| `degraded` | 满足离线规则 | `offline` | 升级现有故障事件，通知离线。 |
+| `offline` / `degraded` | 连续 3 轮零丢包 | `online` | 关闭未恢复事件，发送恢复通知。 |
+| 任意 | 无有效检测结果 | 保持原状态 | 创建受限频率的系统事件；不发送设备故障通知。 |
+
+- 同一事件只在首次状态变更时通知；未恢复事件默认每 4 小时提醒一次，间隔可配置。
+- 每个设备可覆盖全局规则：检测间隔、单轮发包数、离线轮数、故障窗口、故障丢包率、恢复轮数和提醒间隔。
+- 上游心跳失败或单轮大面积离线时，创建“监控节点异常/批量故障”系统事件并通知；不将设备事件静默丢弃。
+
+## 5. 数据、接口与页面
+
+### 5.1 数据模型
+
+- 扩展 `ping_records`：`sent_count`、`received_count`、`packet_loss_percent`、`average_rtt_ms`、`is_valid`、`failure_reason`。
+- 扩展 `devices`：监测策略覆盖字段与 `current_status`（加入 `degraded`）。
+- 扩展 `alert_events`：加入 `degraded`、状态变更前后值、关联恢复事件、通知尝试数、最后通知错误和下一次通知时间。
+- 新增 `notification_outbox`：为每个待发送通知保存内容摘要、投递范围摘要、状态、尝试次数、锁定时间和投递结果。
+
+数据库变更应使用可重复执行、可回退的迁移；保留现有数据。
+
+### 5.2 API 与 UI
+
+- 设备 API 返回当前状态、最新有效结果和设备级策略；创建、编辑、批量导入均校验 IPv4/IPv6、数值范围、单文件大小、字符编码及重复 IP。
+- 告警 API 支持 `offline`、`degraded`、`recovered`、`system` 筛选，并返回持续时长、恢复关联和通知状态。
+- 设备列表展示“在线/故障/离线/未知”、最近丢包率、时延和最近检测时间；告警页展示事件、持续时间和投递状态。
+
+## 6. 企业微信与错误处理
+
+- 应用使用环境变量提供的 Corp ID、Agent ID、Secret；默认向应用可见范围内的成员发送，接收部门仅在企业微信应用管理端配置。若后续需要缩小范围，可增加受控的 `WECOM_TO_PARTY` 环境变量，不在系统页面维护接收人。
+- access token 仅在进程内带有效期缓存；请求使用明确连接/读取超时。失败事件按有限次数的指数退避重试，达到上限后标记失败，保留人工处理证据。
+- 日志只记录错误码、事件 ID、设备 ID、轮次和重试次数；不得记录 token、secret、口令或完整敏感响应。
+- 认证/OAuth 必须校验证书与 JWT 签名；不得以 `verify=False` 或未验签方式绕过校验。
+
+## 7. Ubuntu + OpenResty 部署
+
+1. 宿主机 OpenResty 监听指定 IP 端口，直接提供前端构建目录，反向代理 `/api/` 和 `/ws/` 至 `127.0.0.1` 后端端口。
+2. Docker Compose 运行后端和 PostgreSQL；PostgreSQL 不映射宿主机端口，后端只绑定回环地址。
+3. OpenResty 配置 WebSocket Upgrade、超时、请求体大小限制、访问日志与健康检查路由。没有域名时首期在受控内网使用 HTTP；需要 TLS 时由企业内部 CA 或含 IP SAN 的证书在 OpenResty 终止。
+4. 后端容器按最小权限运行，移除 `NET_ADMIN`，仅保留探测所需的 `NET_RAW`；所有运行密钥由受控环境文件注入，不进入 Git 或镜像。
+5. 交付运行手册，包括配置清单、备份/恢复、启动/停止、OpenResty 重载、健康检查、升级和回退。发布前执行备份与冒烟检查。
+
+## 8. 测试和首批验收
+
+- 单元测试：状态转移、阈值、窗口计算、防抖、事件去重、过期提醒、企业微信重试、IP/CSV/配置校验。
+- 集成测试：异步数据库迁移、API 权限、出箱持久化、推送失败恢复、调度器不重叠。
+- 部署测试：Compose 启动、OpenResty 反向代理、WebSocket、健康检查、PostgreSQL 不可从宿主机访问、容器能力最小化。
+- 运行验证：使用明确授权的测试 IP 验证离线、间歇丢包、恢复、监控节点异常和部门通知。
+
+## 9. 合规控制与可追溯性
+
+| 分类 | 控制 | 依据/说明 |
+| --- | --- | --- |
+| 规范要求（C1） | 部署前完成测试并提交报告；提供版本、制品/代码关联、部署计划、操作步骤、回退及备份方案；部署后按用例验证。 | `13-deployment.md` §2.2.1、§2.2.2、§2.6。 |
+| 规范要求 | 防止将不可信输入用于拼接 SQL、命令执行或不安全文件上传。 | `12-security.md` §3.6.1.1、§3.6.1.3、§3.6.2.2。 |
+| 规范要求 | 生产变更前开启主机安全管控与防火墙，并纳入运行状态监控。 | `12-security.md` §5.1（强制）。 |
+| 工程建议 | 使用事务性事件出箱、状态机防抖、最小容器能力、依赖/密钥扫描与版本固定。 | 可靠性和安全工程实践，不宣称为上述规范的特定强制目录或工具。 |
+| 待确认 | 项目最终 C 级别、企业微信部门 ID、对外端口、内网 TLS 要求、备份责任人和部署审批人。 | 由项目负责人/运行方确认。 |
+
+需求、设计、实现、测试、制品和部署记录应通过上述需求编号建立双向关联。实现完成后补充追踪矩阵和实际证据。