# PingWatch 可靠连通性监测与企业微信告警 Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

**Goal:** 将 PingWatch 改造成可区分离线与间歇性丢包故障、可靠发送企业微信应用通知，并能在 Ubuntu/OpenResty 环境安全运行的监测服务。

**Architecture:** fping 每轮为每个设备取得多包汇总，独立状态机依据连续轮次和滑动窗口计算 online/degraded/offline。状态转换在同一数据库事务内创建事件与通知出箱；独立投递器重试企业微信消息。OpenResty 在宿主机提供 Vue 构建文件并反向代理仅绑定回环地址的 FastAPI 服务。

**Tech Stack:** Python 3.12、FastAPI、SQLAlchemy asyncio、PostgreSQL 16、fping、httpx、pytest、Vue 3、Element Plus、OpenResty、Docker Compose。

## Global Constraints

- 目标能力等级暂按 C1；所有需求使用 PINGWATCH_(可靠性监测)_001 追溯。
- 默认检测周期为 30 秒，每轮 3 包；离线为连续 2 轮 100% 丢包，故障为最近 5 轮至少 20% 丢包，恢复为连续 3 轮零丢包。
- 只监测显式录入且 is_enabled=true 的 IPv4/IPv6 地址；不得增加网段发现或端口扫描。
- 凭据仅通过运行时环境变量注入；Git、镜像、日志、API 响应和文档不得包含 secret、token 或口令。
- 企业微信默认向应用可见范围发送；不提供系统页面维护接收人。可选 WECOM_TO_PARTY 只允许由受控环境变量设置。
- 后端仅以 127.0.0.1:8001 暴露给 Ubuntu 宿主机 OpenResty；PostgreSQL 不映射宿主机端口。
- 不可信输入必须经 Pydantic/显式校验，fping 只通过参数数组启动，禁止 shell=True。
- 除非命令明确指定其他目录，所有 Python pytest 命令均从 backend/ 目录运行。

---

## Planned File Structure

| 文件 | 责任 |
| --- | --- |
| backend/app/models/{device,ping_record,alert_event,notification_outbox}.py | 设备策略、探测汇总、领域事件及通知出箱持久化。 |
| backend/app/services/{fping_runner,state_machine,pinger,notification_dispatcher,alerter}.py | 命令输出解析、状态计算、轮次编排、可靠投递和事件创建。 |
| backend/app/api/{devices,alerts,health}.py 与 backend/app/schemas/* | 受保护设备/事件接口和不含敏感信息的健康探针。 |
| backend/tests/* | 状态、解析、投递、API、配置和迁移验证。 |
| frontend/src/views/{Devices,Alerts,Dashboard}.vue | 健康度、探测质量、事件和投递结果展示。 |
| deploy/openresty、deploy/systemd、docs/operations | OpenResty、受控启动、部署、回退、备份运行材料。 |

## Task 1: 建立测试与运行配置契约

**Files:**
- Modify: backend/requirements.txt, backend/app/config.py, .gitignore
- Create: backend/.env.example, backend/pytest.ini, backend/app/services/settings_validation.py
- Create: backend/tests/conftest.py, backend/tests/test_settings_validation.py

**Interfaces:**
- Produces: validate_runtime_settings(settings: Settings) -> None.
- Produces: monitoring defaults and bounded retry/WeCom configuration fields.

- [ ] **Step 1: Write the failing configuration tests**

~~~python
import pytest
from pydantic import ValidationError
from app.config import Settings
from app.services.settings_validation import validate_runtime_settings

def test_production_rejects_legacy_jwt_secret():
    settings = Settings(environment="production", secret_key="change-me-to-a-long-random-string")
    with pytest.raises(ValueError, match="SECRET_KEY"):
        validate_runtime_settings(settings)

def test_probe_packet_count_must_be_positive():
    with pytest.raises(ValidationError):
        Settings(probe_packets_per_round=0)
~~~

- [ ] **Step 2: Run test to verify it fails**

Run: python -m pytest backend/tests/test_settings_validation.py -v  
Expected: FAIL because settings fields and validation module are absent.

- [ ] **Step 3: Write minimal configuration implementation**

Add pinned test dependencies pytest, pytest-asyncio and aiosqlite. Add Settings fields: environment, probe_packets_per_round=3 (1..10), offline_consecutive_rounds=2 (1..10), degraded_window_rounds=5 (2..60), degraded_loss_percent=20.0 (1..100), recovery_consecutive_clean_rounds=3 (1..20), wecom_notification_max_attempts, wecom_retry_base_seconds and WECOM_TO_PARTY. validate_runtime_settings rejects production default JWT keys, production TLS-disabled Casdoor, and enabled WeCom delivery without all three credentials.

~~~python
def validate_runtime_settings(settings: Settings) -> None:
    if settings.environment == "production" and settings.secret_key == LEGACY_DEFAULT_SECRET:
        raise ValueError("SECRET_KEY must be provided by the runtime environment")
~~~

Create backend/.env.example with empty secret values only; keep real .env ignored.

- [ ] **Step 4: Run focused verification**

Run: python -m pytest backend/tests/test_settings_validation.py -v  
Expected: PASS.

- [ ] **Step 5: Commit**

~~~bash
git add backend/requirements.txt backend/pytest.ini backend/.env.example .gitignore backend/app/config.py backend/app/services/settings_validation.py backend/tests
git commit -m "test: establish monitoring configuration contract"
~~~

## Task 2: 持久化探测质量、设备策略、事件与通知出箱

**Files:**
- Modify: backend/app/models/device.py, backend/app/models/ping_record.py, backend/app/models/alert_event.py, backend/app/models/__init__.py, backend/app/core/deps.py
- Create: backend/app/models/notification_outbox.py, backend/migrations/versions/20260803_reliability_monitoring.py
- Create: backend/tests/test_models_and_migration.py

**Interfaces:**
- Produces: DeviceMonitoringPolicy.from_device(device, settings) -> DeviceMonitoringPolicy.
- Produces: PingRecord.sent_count, received_count, packet_loss_percent, average_rtt_ms, is_valid, failure_reason.
- Produces: NotificationOutbox with status pending, sending, sent, failed.

- [ ] **Step 1: Write failing persistence tests**

~~~python
async def test_probe_record_stores_packet_summary(db_session):
    record = PingRecord(device_id=1, sent_count=3, received_count=2,
                        packet_loss_percent=33.33, average_rtt_ms=12.5, is_valid=True)
    db_session.add(record)
    await db_session.commit()
    assert record.packet_loss_percent == 33.33

async def test_outbox_defaults_to_pending(db_session):
    item = NotificationOutbox(alert_event_id=1, message_content="masked summary")
    db_session.add(item)
    await db_session.commit()
    assert item.status == NotificationStatus.pending
~~~

- [ ] **Step 2: Run test to verify it fails**

Run: python -m pytest backend/tests/test_models_and_migration.py -v  
Expected: FAIL because fields and NotificationOutbox do not exist.

- [ ] **Step 3: Implement models and migration**

Extend Device with nullable per-device policy overrides and allow degraded status. Add an index on probe (device_id, created_at), and a status/next_attempt_at index on outbox. Add AlertTypeEnum.degraded and fields previous_status, current_status, last_notification_error, notification_attempts. Implement one idempotent migration runner called before create_all; it records revision 20260803_reliability_monitoring in schema_migrations and never deletes existing data.

- [ ] **Step 4: Run focused verification**

Run: python -m pytest backend/tests/test_models_and_migration.py -v  
Expected: PASS, including two sequential migration executions against the same database.

- [ ] **Step 5: Commit**

~~~bash
git add backend/app/models backend/app/core/deps.py backend/migrations backend/tests/test_models_and_migration.py
git commit -m "feat: persist probe quality and notification outbox"
~~~

## Task 3: 实现多包 fping 解析和纯状态机

**Files:**
- Create: backend/app/services/fping_runner.py, backend/app/services/state_machine.py
- Modify: backend/app/services/pinger.py
- Create: backend/tests/test_fping_runner.py, backend/tests/test_state_machine.py

**Interfaces:**
- Produces: ProbeResult(ip, sent_count, received_count, average_rtt_ms, is_valid, failure_reason).
- Produces: parse_fping_count_output(output, expected_ips, packets_per_round) -> dict[str, ProbeResult].
- Produces: evaluate_health(previous_status, recent, policy) -> StateDecision.

- [ ] **Step 1: Write failing parser and transition tests**

~~~python
def test_parses_loss_and_average_rtt():
    output = "10.0.0.8 : 12.4 - 12.6\n10.0.0.9 : - - -"
    result = parse_fping_count_output(output, {"10.0.0.8", "10.0.0.9"}, 3)
    assert result["10.0.0.8"].received_count == 2
    assert result["10.0.0.9"].packet_loss_percent == 100.0

def test_two_full_loss_rounds_take_device_offline(policy):
    decision = evaluate_health("online", [loss(3), loss(3)], policy)
    assert (decision.next_status, decision.event_type) == ("offline", "offline")

def test_five_round_window_with_loss_is_degraded(policy):
    decision = evaluate_health("online", [partial(3, 2)] * 5, policy)
    assert decision.next_status == "degraded"

def test_three_clean_rounds_recovers(policy):
    decision = evaluate_health("offline", [clean(3)] * 3, policy)
    assert (decision.next_status, decision.event_type) == ("online", "recovered")
~~~

- [ ] **Step 2: Run test to verify it fails**

Run: python -m pytest backend/tests/test_fping_runner.py backend/tests/test_state_machine.py -v  
Expected: FAIL because runner and state machine modules are absent.

- [ ] **Step 3: Implement minimal parser and state machine**

Invoke fping with the argument sequence fping -C <count> -q -t <milliseconds> <ip...>; capture stdout and stderr because quiet count output is emitted to stderr. Numeric reply tokens count as success and dash tokens as loss. Missing/malformed expected IP output creates is_valid=False and cannot trigger a device failure.

State priority: invalid records retain state; configured consecutive full loss gives offline; configured aggregate window loss gives degraded; only offline/degraded plus configured clean records gives online; otherwise retain state. StateDecision contains next_status, event_type, should_notify, reason.

- [ ] **Step 4: Refactor Pinger to persist exactly one summary per enabled device**

Pinger.run_one_round saves every ProbeResult, reads the required recent valid records, computes StateDecision, updates Device, and returns DeviceStateChange values within one transaction. It must use asyncio.create_subprocess_exec(*command) and never a command string.

- [ ] **Step 5: Run focused verification**

Run: python -m pytest backend/tests/test_fping_runner.py backend/tests/test_state_machine.py -v  
Expected: PASS.

- [ ] **Step 6: Commit**

~~~bash
git add backend/app/services/fping_runner.py backend/app/services/state_machine.py backend/app/services/pinger.py backend/tests/test_fping_runner.py backend/tests/test_state_machine.py
git commit -m "feat: classify offline and degraded connectivity"
~~~

## Task 4: 用事务出箱可靠投递企业微信事件

**Files:**
- Create: backend/app/services/notification_dispatcher.py
- Modify: backend/app/services/alerter.py, backend/app/services/scheduler.py, backend/app/services/pinger.py
- Create: backend/tests/test_notification_dispatcher.py, backend/tests/test_alert_workflow.py

**Interfaces:**
- Consumes: list[DeviceStateChange] from Pinger.run_one_round().
- Produces: Alerter.record_transition(change, db) -> AlertEvent.
- Produces: NotificationDispatcher.dispatch_due(db, now) -> DispatchSummary.
- Produces: WeComClient.send_text(content) -> DeliveryResult.

- [ ] **Step 1: Write failing outbox tests**

~~~python
async def test_transition_creates_event_and_pending_outbox_in_one_commit(db_session, change):
    event = await Alerter().record_transition(change, db_session)
    outbox = await db_session.scalar(select(NotificationOutbox).where(
        NotificationOutbox.alert_event_id == event.id))
    assert outbox.status == NotificationStatus.pending

async def test_retryable_failure_reschedules(db_session, pending_message, frozen_time):
    client = FakeWeComClient(DeliveryResult(False, "timeout", True))
    await NotificationDispatcher(client).dispatch_due(db_session, frozen_time)
    assert pending_message.status == NotificationStatus.pending
    assert pending_message.attempt_count == 1
~~~

- [ ] **Step 2: Run test to verify it fails**

Run: python -m pytest backend/tests/test_notification_dispatcher.py backend/tests/test_alert_workflow.py -v  
Expected: FAIL because dispatcher interfaces are absent.

- [ ] **Step 3: Implement atomic event/outbox recording**

Replace in-memory pending alerts. record_transition inserts AlertEvent and NotificationOutbox before caller commit. Offline escalation annotates/resolves the prior open degraded event. Recovery closes one unresolved event, stores duration, and creates a recovered notification. Render notifications with event type, device, IP, location, project, loss summary, time and duration; split batches at a fixed safe count.

- [ ] **Step 4: Implement WeCom client and retry dispatcher**

Use httpx.AsyncClient with explicit connect/read/write timeout. Cache access_token until server expiry minus safety margin. Default payload uses touser="@all"; only use toparty when WECOM_TO_PARTY is non-empty. Never log token-bearing URLs, request JSON, response body, secret, or message content. Retry transport/timeouts, 429 and 5xx with next_attempt_at = now + base_seconds * 2 ** (attempt_count - 1); mark failed at configured maximum.

- [ ] **Step 5: Connect dispatcher to scheduler**

Run dispatch_due after a successful probe transaction and at a small bounded interval. The scheduler has one task and a lock; cancellation waits for the active probe/dispatch cycle to close its session.

- [ ] **Step 6: Run verification and commit**

Run: python -m pytest backend/tests/test_notification_dispatcher.py backend/tests/test_alert_workflow.py -v  
Expected: PASS.

~~~bash
git add backend/app/services/notification_dispatcher.py backend/app/services/alerter.py backend/app/services/scheduler.py backend/app/services/pinger.py backend/tests/test_notification_dispatcher.py backend/tests/test_alert_workflow.py
git commit -m "feat: deliver alert events through persistent outbox"
~~~

## Task 5: 校验设备输入并提供健康/审计 API

**Files:**
- Modify: backend/app/schemas/device.py, backend/app/schemas/alert.py, backend/app/api/devices.py, backend/app/api/alerts.py, backend/app/main.py
- Create: backend/app/api/health.py
- Create: backend/tests/test_devices_api.py, backend/tests/test_alerts_api.py, backend/tests/test_health_api.py

**Interfaces:**
- Produces: DeviceCreate and DeviceUpdate using IPvAnyAddress and bounded policy fields.
- Produces: GET /api/health -> {"status": "ok", "database": "ready"} after SELECT 1.
- Produces: alert DTO transition and notification-status fields.

- [ ] **Step 1: Write failing API tests**

~~~python
async def test_create_rejects_command_like_ip(authenticated_client):
    response = await authenticated_client.post("/api/devices",
        json={"name": "bad", "ip": "127.0.0.1;id"})
    assert response.status_code == 422

async def test_health_requires_database_readiness(client):
    response = await client.get("/api/health")
    assert response.json() == {"status": "ok", "database": "ready"}
~~~

- [ ] **Step 2: Run test to verify it fails**

Run: python -m pytest backend/tests/test_devices_api.py backend/tests/test_alerts_api.py backend/tests/test_health_api.py -v  
Expected: FAIL on missing validation/health behavior.

- [ ] **Step 3: Implement schema and import validation**

Use IPvAnyAddress, Field bounds and field_validator to map blank overrides to None. Before decoding CSV enforce CSV_MAX_BYTES; reject invalid UTF-8 as 400; limit returned line errors to 100; de-duplicate both database IPs and earlier accepted CSV rows. Device and alert responses expose only aggregate notification status, never message content or external error body.

- [ ] **Step 4: Implement health and API projections**

Health router runs SELECT 1. The lifespan calls validate_runtime_settings before scheduling. CORS accepts only exact configured origins. Device lists return latest valid summary and policy fields; alerts return degraded/recovered filters, transition and retry fields.

- [ ] **Step 5: Run verification and commit**

Run: python -m pytest backend/tests/test_devices_api.py backend/tests/test_alerts_api.py backend/tests/test_health_api.py -v  
Expected: PASS.

~~~bash
git add backend/app/schemas backend/app/api backend/app/main.py backend/tests/test_devices_api.py backend/tests/test_alerts_api.py backend/tests/test_health_api.py
git commit -m "feat: expose validated monitoring health and event APIs"
~~~

## Task 6: 更新前端运维视图

**Files:**
- Modify: frontend/package.json, frontend/package-lock.json, frontend/src/api/index.js
- Modify: frontend/src/views/Devices.vue, frontend/src/views/Alerts.vue, frontend/src/views/Dashboard.vue
- Create: frontend/src/utils/status.js, frontend/src/utils/status.test.js

**Interfaces:**
- Produces: statusLabel(status) and statusTagType(status) supporting online, degraded, offline, unknown.
- Consumes: enhanced Task 5 device and alert DTOs.

- [ ] **Step 1: Write a failing presentation test**

~~~javascript
import { describe, expect, it } from "vitest"
import { statusLabel, statusTagType } from "./status"

describe("status presentation", () => {
  it("marks degraded connectivity as a warning", () => {
    expect(statusLabel("degraded")).toBe("故障")
    expect(statusTagType("degraded")).toBe("warning")
  })
})
~~~

- [ ] **Step 2: Run test to verify it fails**

Run: npm --prefix frontend run test -- --run  
Expected: FAIL because the test script is absent.

- [ ] **Step 3: Implement minimal UI contract**

Add vitest development dependency and test script. Add status utility with explicit mappings. Devices view adds state, packet loss, RTT and last probe columns plus bounded policy form fields. Alerts view adds degraded filter, transition, delivery status and retry count. Dashboard adds degraded count. Use Vue interpolation exclusively; never use v-html for device data.

- [ ] **Step 4: Run verification and commit**

Run: npm --prefix frontend run test -- --run  
Expected: PASS.

Run: npm --prefix frontend run build  
Expected: PASS.

~~~bash
git add frontend/package.json frontend/package-lock.json frontend/src/api/index.js frontend/src/utils frontend/src/views/Devices.vue frontend/src/views/Alerts.vue frontend/src/views/Dashboard.vue
git commit -m "feat: display degraded connectivity and delivery state"
~~~

## Task 7: 加固认证、容器暴露与 Ubuntu/OpenResty 交付

**Files:**
- Modify: backend/app/core/auth.py, backend/Dockerfile, docker-compose.yml
- Create: deploy/openresty/pingwatch.conf, deploy/systemd/pingwatch.service
- Create: docs/operations/ubuntu-openresty-deployment.md, docs/operations/rollback-and-backup.md
- Create: backend/tests/test_auth_security.py, deploy/tests/test_compose_exposure.ps1

**Interfaces:**
- Produces: exchange_code_for_user(code) that never accepts unverified JWT claims in production.
- Produces: OpenResty routes for /, /api/, /ws and /api/health.

- [ ] **Step 1: Write failing security/deployment assertions**

~~~python
async def test_production_rejects_unverified_casdoor_token(monkeypatch):
    monkeypatch.setattr(settings, "environment", "production")
    monkeypatch.setattr(settings, "casdoor_certificate", "")
    assert await exchange_code_for_user("code") is None
~~~

~~~powershell
$compose = Get-Content -Raw .\docker-compose.yml
if ($compose -match 'POSTGRES_PASSWORD=') { throw 'tracked compose contains password' }
if ($compose -match 'NET_ADMIN') { throw 'container has excess capability' }
if ($compose -notmatch '127.0.0.1:8001:8000') { throw 'backend is not loopback-bound' }
~~~

- [ ] **Step 2: Run test to verify it fails**

Run: python -m pytest backend/tests/test_auth_security.py -v  
Expected: FAIL because OAuth disables TLS verification or accepts unverified claims.

Run: pwsh -File deploy/tests/test_compose_exposure.ps1  
Expected: FAIL because Compose contains an embedded password and NET_ADMIN.

- [ ] **Step 3: Implement minimal hardening**

Require certificate/TLS verification before accepting production Casdoor id_token; remove unverified-claims fallback. Compose reads database credentials only from server-only environment file, exposes backend as 127.0.0.1:8001:8000, does not run frontend container in production, does not expose database, uses cap_drop ALL plus cap_add NET_RAW and no-new-privileges. Backend image has non-secret health check.

- [ ] **Step 4: Add OpenResty and operating material**

OpenResty listens on the agreed IP port, serves /opt/pingwatch/frontend, uses SPA try_files, proxies /api/ to 127.0.0.1:8001 and upgrades /ws, limits request body to 15m, and has separate access/error logs. Deployment guide gives preflight, server-only env permissions, build/copy, Compose lifecycle, OpenResty test/reload, health/API/WebSocket smoke tests and approved rollback. Backup guide defines PostgreSQL volume backup before upgrades and restore of prior frontend build/image.

- [ ] **Step 5: Run verification and commit**

Run: python -m pytest backend/tests/test_auth_security.py -v  
Expected: PASS.

Run: pwsh -File deploy/tests/test_compose_exposure.ps1  
Expected: PASS.

Run: docker compose config  
Expected: PASS with no database port and no plaintext secret in tracked Compose file.

~~~bash
git add backend/app/core/auth.py backend/Dockerfile docker-compose.yml deploy docs/operations backend/tests/test_auth_security.py
git commit -m "feat: secure OpenResty deployment package"
~~~

## Task 8: 全量验证、追溯证据与发布准备

**Files:**
- Create: docs/traceability/2026-08-03-reliability-monitoring.md
- Create: docs/test-reports/2026-08-03-reliability-monitoring.md
- Create: README.md

**Interfaces:**
- Produces: each of five requirement IDs mapped to code, test and Ubuntu/OpenResty validation evidence.

- [ ] **Step 1: Execute full automated verification**

Run: python -m pytest backend/tests -v from backend/  
Expected: PASS.

Run: npm --prefix frontend run test -- --run  
Expected: PASS.

Run: npm --prefix frontend run build  
Expected: PASS.

Run: docker compose config  
Expected: PASS.

- [ ] **Step 2: Execute local Compose smoke test**

Run: docker compose up --build -d, then curl -fsS http://127.0.0.1:8001/api/health, then docker compose down.  
Expected: health contains status=ok and database=ready; no test data or secret file is committed.

- [ ] **Step 3: Write traceability/test reports**

Traceability maps requirements _01 through _05 to implementation file, exact automated test name and Ubuntu/OpenResty validation command. Test report records command, timestamp, result, limitations, and states that real network validation needs authorized test IPs and configured enterprise WeChat credentials. It excludes credentials, private IP inventories and full notification contents.

- [ ] **Step 4: Scan final changes and commit evidence**

Run: git diff main...HEAD --check  
Expected: PASS.

Run: rg -n '(SECRET|PASSWORD|TOKEN|PRIVATE KEY)\s*=\s*[^" ]+' -g '!*.example' -g '!docs/superpowers/**' .  
Expected: no hardcoded secret assignment.

~~~bash
git add README.md docs/traceability docs/test-reports
git commit -m "docs: add reliability monitoring verification evidence"
~~~

## Self-Review

- Spec coverage: Tasks 1-4 implement multi-packet probing, state transitions, durable notification, recovery and batch/node safety; Tasks 5-6 implement API/UI; Task 7 implements OpenResty, secrets, authentication and container controls; Task 8 provides C1 deployment evidence and traceability.
- Placeholder scan: the plan contains executable commands, named files, test behavior and data contracts for each task.
- Type consistency: ProbeResult, DeviceMonitoringPolicy, StateDecision, DeviceStateChange, NotificationOutbox and DeliveryResult are defined before their consuming tasks.