fix: 企微 API 通过代理绕过 IP 白名单限制，补充错误日志

- 新增 WECHAT_API_BASE_URL 配置项，支持企微 API 代理转发
- _send_message 增加 API 错误日志（errcode + errmsg）
- _get_access_token 增加失败日志
- handle_text 不再为空，返回菜单引导提示
- 防重入触发时通知用户等待时间
- menu.py API 调用统一使用配置化的 base URL

Co-Authored-By: Claude <noreply@anthropic.com>

.env.example

@@ -0,0 +1,47 @@
+# 应用
+DEBUG=false
+LOG_LEVEL=INFO
+
+# 数据库
+DATABASE_URL=postgresql+asyncpg://gx-gp-notify:password@10.10.10.14:5432/gx-gp-notify
+
+# 爬虫
+CRAWLER_BASE_URL=https://zfcg.gxzf.gov.cn
+CRAWLER_KEYWORDS=["大化"]
+CRAWLER_MAX_PAGES=10
+CRAWLER_TIMEOUT=30
+
+# 企业微信
+WECHAT_ENABLED=true
+WECHAT_CORP_ID=ww69e8e44636f47780
+WECHAT_AGENT_ID=1000007
+WECHAT_SECRET=
+WECHAT_TOKEN=
+WECHAT_ENCODING_AES_KEY=
+WECHAT_PORT=18001
+WECHAT_HOST=0.0.0.0
+# 企微 API 代理（可选），用于绕过 IP 白名单限制
+# 留空则直连 https://qyapi.weixin.qq.com
+WECHAT_API_BASE_URL=https://qyapi.weixin.qq.com
+
+# 定时任务
+SCHEDULER_ENABLED=true
+SCHEDULER_CRON=0 8-21 * * *
+
+# LogHive 日志系统
+LOGHIVE_ENDPOINT=http://10.10.10.14:8000
+LOGHIVE_PROJECT=gx-gp-notify
+LOGHIVE_API_KEY=
+
+# AI 分析 (DeepSeek)
+AI_ENABLED=false
+# AI 管理白名单（企微用户ID，逗号分隔），留空表示所有人可操作
+AI_WHITELIST=
+AI_API_KEY=sk-your-deepseek-api-key
+AI_BASE_URL=https://api.deepseek.com/v1
+AI_MODEL=deepseek-chat
+AI_TIMEOUT=30
+AI_ANALYSIS_TITLE=🔔 中国电信可承接项目
+# 自定义分析提示词（可选），不设置则用代码内置默认值
+# 使用 \n 表示换行，支持 {title} {purchase_name} {announcement_type} {content} 四个占位符
+# AI_PROMPT_TEMPLATE=你是一个政府采购项目分析师...\n\n--- 公告信息 ---\n标题：{title}\n...

.gitignore

@@ -0,0 +1,18 @@
+.claude/
+.env
+logs/
+*.log
+__pycache__/
+*.pyc
+.venv/
+.ruff_cache/
+.pytest_cache/
+*.egg-info/
+dist/
+config.yaml
+gx_gp_monitor/config/config.yaml
+onu.md
+vendor/build/
+vendor/*.egg-info/
+# Added by code-review-graph
+.code-review-graph/

README.md

@@ -0,0 +1,108 @@
+# GX-GP-Notify
+
+广西政府采购公告监控系统，自动爬取广西政府采购网和大化县政府网的公告，通过企业微信推送关键词匹配的公告。
+
+## 功能
+
+- **定时爬取**：按 cron 规则自动爬取，工作日 8:00-21:00 每小时执行
+- **关键词过滤**：广西政采网公告按关键词过滤（默认：`大化`），大化县政府网全量爬取
+- **企微推送**：新公告实时推送到企业微信应用
+- **去重**：基于内容 hash 去重，避免重复入库和推送
+- **节假日感知**：非工作日自动跳过爬取
+
+## 数据来源
+
+| 来源 | 说明 | 过滤 |
+|------|------|------|
+| 广西政府采购网 | 采购公告、结果公告、合同公告等 13 个分类 | 关键词过滤 |
+| 大化县政府网 | 大化县门户网站采购公告 | 全量爬取 |
+
+## 企微菜单功能
+
+| 菜单 | 说明 |
+|------|------|
+| 最新公告 | 触发一次爬取，推送两个来源合并后最新 6 条（时间升序） |
+| 手动爬取 | 立即触发全量爬取并推送新公告 |
+| 今日统计 | 查看今日新增和累计公告数 |
+| 系统状态 | 查看待推送数量、定时任务状态 |
+| 监控配置 | 查看关键词、爬取页数、定时规则 |
+| 工作日状态 | 查看今天是否为工作日 |
+| 同步节假日 | 同步当年节假日数据 |
+| 暂停/恢复任务 | 控制定时爬取任务 |
+
+## 快速部署
+
+### 1. 配置环境变量
+
+复制 `.env.example` 为 `.env` 并填写：
+
+```env
+# 数据库
+DATABASE_URL=postgresql+asyncpg://user:password@host:5432/gx-gp-notify
+
+# 企业微信
+WECHAT_CORP_ID=your_corp_id
+WECHAT_AGENT_ID=your_agent_id
+WECHAT_SECRET=your_secret
+WECHAT_TOKEN=your_token
+WECHAT_ENCODING_AES_KEY=your_aes_key
+
+# 爬虫
+CRAWLER_KEYWORDS=["大化"]
+CRAWLER_MAX_PAGES=10
+SCHEDULER_CRON=0 8-21 * * 1-5
+```
+
+### 2. 启动服务
+
+```bash
+docker compose -f docker/docker-compose.yml up -d
+```
+
+### 3. 数据库迁移
+
+```bash
+docker compose -f docker/docker-compose.yml exec app alembic upgrade head
+```
+
+### 4. 配置企微回调
+
+在企业微信后台将回调 URL 设置为：
+
+```
+http://your-server:18001/wechat/callback
+```
+
+## 开发
+
+```bash
+# 安装依赖
+pip install -e ".[dev]"
+
+# 运行测试
+pytest
+
+# 代码检查
+ruff check .
+```
+
+## 技术栈
+
+- **框架**：FastAPI + uvicorn
+- **数据库**：PostgreSQL + SQLAlchemy (async) + Alembic
+- **爬虫**：httpx + BeautifulSoup4
+- **定时任务**：APScheduler
+- **推送**：企业微信应用消息 API
+- **部署**：Docker
+
+## 项目结构
+
+```
+app/
+├── api/          # HTTP 接口（公告查询、手动触发爬取等）
+├── crawler/      # 爬虫（广西政采网、大化县政府网）
+├── models/       # 数据库模型
+├── scheduler/    # 定时任务
+├── services/     # 业务逻辑（pipeline、过滤、通知）
+└── wechat/       # 企业微信（消息处理、推送客户端）
+```

README_企业微信.md

@@ -1,231 +0,0 @@
-# 广西政府采购网公告监控系统 - 企业微信集成
-
-## 概述
-
-本系统已集成企业微信功能，支持：
-- 企业微信应用消息推送
-- 回调服务器处理用户交互
-- 应用菜单管理
-- 实时爬取和自定义搜索
-
-## 配置要求
-
-### 1. 企业微信应用配置
-
-在 `config/config.yaml` 中配置企业微信信息：
-
-```yaml
-# 企业微信应用配置
-wechat_app:
-  enabled: true                    # 是否启用企业微信通知
-  corp_id: "ww69e8e44636f47780"   # 企业ID
-  agent_id: "1000007"             # 应用ID
-  secret: "SmelCwKFoL0E9ATWFzr-w7gsfXBTN72lT1UqnNd0HpI"  # 应用Secret
-  token: "DmvL98cAF6x9CFtQZwqD2emGL8S7HxA"               # Token
-  encoding_aes_key: "yAc4OoSCP92YTefHXYfw27WeG9oF11W9d6nw6QYlU3D"  # 消息加密Key
-  port: 18001                     # 服务端口
-```
-
-### 2. 企业微信管理后台设置
-
-1. **设置回调URL**
-   - 在企业微信管理后台的应用设置中
-   - 设置回调URL为：`http://your-server-ip:18001/api/v1/wechat/callback`
-   - Token和EncodingAESKey需要与配置文件一致
-
-2. **设置应用可见范围**
-   - 配置应用对哪些部门或成员可见
-   - 确保有基础接口权限
-
-## 功能说明
-
-### 1. 消息推送
-
-系统会在以下情况下自动推送消息到企业微信：
-
-- **爬取完成通知**：每次执行爬取后，推送筛选结果
-- **关键词匹配通知**：发现匹配关键词的公告时推送
-- **系统状态通知**：系统启动、错误等重要事件
-
-消息格式参考 `企业微信消息示例.md`
-
-### 2. 应用菜单
-
-系统提供以下菜单功能：
-
-#### 操作菜单
-- **立即爬取**：执行一次完整的公告爬取
-- **今日总结**：查看今日公告统计（开发中）
-- **自定义爬取**：输入关键词进行搜索
-
-#### 帮助菜单
-- **使用说明**：显示详细使用指南
-- **联系我们**：联系信息
-
-### 3. 文本交互
-
-支持以下文本命令：
-
-```
-# 关键词搜索
-直接发送关键词，如：大化、信息化
-
-# 手动爬取
-爬取 [关键词]    # 如：爬取 大化
-
-# 获取帮助
-帮助            # 显示使用说明
-help
-
-# 查看总结
-总结            # 查看今日统计
-```
-
-## 部署和运行
-
-### 1. 安装依赖
-
-```bash
-pip install -r gx_gp_monitor/requirements.txt
-```
-
-### 2. 测试功能
-
-```bash
-# 测试企业微信模块
-python test_wechat_server.py
-```
-
-### 3. 启动回调服务器
-
-```bash
-# 方法1：使用专用启动脚本
-python start_wechat_server.py
-
-# 方法2：使用主程序
-python gx_gp_monitor/main.py wechat-server
-```
-
-### 4. 管理菜单
-
-```bash
-# 创建菜单
-python gx_gp_monitor/main.py wechat-menu --action create
-
-# 删除菜单
-python gx_gp_monitor/main.py wechat-menu --action delete
-
-# 获取菜单信息
-python gx_gp_monitor/main.py wechat-menu --action get
-
-# 测试菜单操作
-python gx_gp_monitor/main.py wechat-menu --action test
-```
-
-### 5. 正常爬取
-
-```bash
-# 执行爬取（会自动推送消息）
-python gx_gp_monitor/main.py crawl
-```
-
-## 技术实现
-
-### 1. 回调服务器
-
-- **框架**：Flask
-- **端口**：18001
-- **路径**：`/api/v1/wechat/callback`
-- **功能**：处理企业微信的回调消息和URL验证
-
-### 2. 消息加解密
-
-- **库来源**：企业微信官方Python示例
-- **位置**：`gx_gp_monitor/wechat/WXBizMsgCrypt.py`
-- **功能**：实现消息加密、解密和签名验证
-
-### 3. 消息处理
-
-- **事件处理**：关注、取消关注、菜单点击
-- **文本处理**：关键词搜索、手动命令
-- **响应格式**：XML格式的企业微信消息
-
-### 4. 菜单管理
-
-- **创建菜单**：自动创建预定义菜单
-- **删除菜单**：清理现有菜单
-- **获取菜单**：查看当前菜单配置
-
-## 故障排除
-
-### 1. 回调URL验证失败
-
-- 检查Token和EncodingAESKey是否正确配置
-- 确保服务器可从公网访问
-- 检查端口18001是否开放
-
-### 2. 消息推送失败
-
-- 检查企业微信应用配置（corp_id, agent_id, secret）
-- 确认应用有消息发送权限
-- 查看应用可见范围设置
-
-### 3. 菜单创建失败
-
-- 检查应用是否有菜单管理权限
-- 确认Token有效且有管理员权限
-
-### 4. 导入错误
-
-```bash
-# 确保依赖已安装
-pip install flask pycryptodome
-
-# 测试导入
-python -c "from gx_gp_monitor.wechat.callback_server import WeChatCallbackServer; print('OK')"
-```
-
-## 日志查看
-
-系统会记录详细的企业微信操作日志：
-
-```bash
-# 查看日志
-tail -f logs/gx_gp_monitor.log | grep wechat
-```
-
-## 安全注意事项
-
-1. **保护配置文件**：不要将包含密钥的配置文件提交到版本控制
-2. **网络安全**：确保回调服务器只接受来自企业微信的请求
-3. **权限控制**：合理设置应用可见范围和权限
-4. **定期更新**：定期更新企业微信应用Secret
-
-## 开发说明
-
-### 项目结构
-
-```
-gx_gp_monitor/wechat/
-├── __init__.py              # 模块初始化
-├── callback_server.py      # Flask回调服务器
-├── message_handler.py       # 消息处理逻辑
-├── menu_manager.py          # 菜单管理
-├── WXBizMsgCrypt.py         # 企业微信加密库
-└── ierror.py               # 错误码定义
-```
-
-### 扩展功能
-
-如需添加新功能，可以：
-
-1. 在 `message_handler.py` 中添加新的消息处理逻辑
-2. 在 `menu_manager.py` 中修改菜单配置
-3. 在 `callback_server.py` 中添加新的路由
-
-## 联系支持
-
-如有问题，请检查：
-1. 系统日志
-2. 企业微信管理后台错误信息
-3. 网络连接和配置

alembic.ini

@@ -0,0 +1,149 @@
+# A generic, single database configuration.
+
+[alembic]
+# path to migration scripts.
+# this is typically a path given in POSIX (e.g. forward slashes)
+# format, relative to the token %(here)s which refers to the location of this
+# ini file
+script_location = %(here)s/alembic
+
+# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
+# Uncomment the line below if you want the files to be prepended with date and time
+# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
+# for all available tokens
+# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
+# Or organize into date-based subdirectories (requires recursive_version_locations = true)
+# file_template = %%(year)d/%%(month).2d/%%(day).2d_%%(hour).2d%%(minute).2d_%%(second).2d_%%(rev)s_%%(slug)s
+
+# sys.path path, will be prepended to sys.path if present.
+# defaults to the current working directory.  for multiple paths, the path separator
+# is defined by "path_separator" below.
+prepend_sys_path = .
+
+
+# timezone to use when rendering the date within the migration file
+# as well as the filename.
+# If specified, requires the tzdata library which can be installed by adding
+# `alembic[tz]` to the pip requirements.
+# string value is passed to ZoneInfo()
+# leave blank for localtime
+# timezone =
+
+# max length of characters to apply to the "slug" field
+# truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+# set to 'true' to allow .pyc and .pyo files without
+# a source .py file to be detected as revisions in the
+# versions/ directory
+# sourceless = false
+
+# version location specification; This defaults
+# to <script_location>/versions.  When using multiple version
+# directories, initial revisions must be specified with --version-path.
+# The path separator used here should be the separator specified by "path_separator"
+# below.
+# version_locations = %(here)s/bar:%(here)s/bat:%(here)s/alembic/versions
+
+# path_separator; This indicates what character is used to split lists of file
+# paths, including version_locations and prepend_sys_path within configparser
+# files such as alembic.ini.
+# The default rendered in new alembic.ini files is "os", which uses os.pathsep
+# to provide os-dependent path splitting.
+#
+# Note that in order to support legacy alembic.ini files, this default does NOT
+# take place if path_separator is not present in alembic.ini.  If this
+# option is omitted entirely, fallback logic is as follows:
+#
+# 1. Parsing of the version_locations option falls back to using the legacy
+#    "version_path_separator" key, which if absent then falls back to the legacy
+#    behavior of splitting on spaces and/or commas.
+# 2. Parsing of the prepend_sys_path option falls back to the legacy
+#    behavior of splitting on spaces, commas, or colons.
+#
+# Valid values for path_separator are:
+#
+# path_separator = :
+# path_separator = ;
+# path_separator = space
+# path_separator = newline
+#
+# Use os.pathsep. Default configuration used for new projects.
+path_separator = os
+
+# set to 'true' to search source files recursively
+# in each "version_locations" directory
+# new in Alembic version 1.10
+# recursive_version_locations = false
+
+# the output encoding used when revision files
+# are written from script.py.mako
+# output_encoding = utf-8
+
+# database URL.  This is consumed by the user-maintained env.py script only.
+# other means of configuring database URLs may be customized within the env.py
+# file.
+sqlalchemy.url = driver://user:pass@localhost/dbname
+
+
+[post_write_hooks]
+# post_write_hooks defines scripts or Python functions that are run
+# on newly generated revision scripts.  See the documentation for further
+# detail and examples
+
+# format using "black" - use the console_scripts runner, against the "black" entrypoint
+# hooks = black
+# black.type = console_scripts
+# black.entrypoint = black
+# black.options = -l 79 REVISION_SCRIPT_FILENAME
+
+# lint with attempts to fix using "ruff" - use the module runner, against the "ruff" module
+# hooks = ruff
+# ruff.type = module
+# ruff.module = ruff
+# ruff.options = check --fix REVISION_SCRIPT_FILENAME
+
+# Alternatively, use the exec runner to execute a binary found on your PATH
+# hooks = ruff
+# ruff.type = exec
+# ruff.executable = ruff
+# ruff.options = check --fix REVISION_SCRIPT_FILENAME
+
+# Logging configuration.  This is also consumed by the user-maintained
+# env.py script only.
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARNING
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARNING
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

alembic/env.py

@@ -0,0 +1,34 @@
+import asyncio
+from alembic import context
+from sqlalchemy.ext.asyncio import create_async_engine
+from app.models.announcement import Base
+from app.models.holiday import ChineseHoliday  # noqa: F401
+from app.config import settings
+
+target_metadata = Base.metadata
+
+
+def run_migrations_offline():
+    context.configure(url=settings.database_url, target_metadata=target_metadata,
+                      literal_binds=True, dialect_opts={"paramstyle": "named"})
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def do_run_migrations(connection):
+    context.configure(connection=connection, target_metadata=target_metadata)
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+async def run_migrations_online():
+    connectable = create_async_engine(settings.database_url, echo=True)
+    async with connectable.connect() as connection:
+        await connection.run_sync(do_run_migrations)
+    await connectable.dispose()
+
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    asyncio.run(run_migrations_online())

alembic/script.py.mako

@@ -0,0 +1,28 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision: str = ${repr(up_revision)}
+down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)}
+branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
+depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    ${downgrades if downgrades else "pass"}

alembic/versions/1f59799a5083_add_chinese_holidays_table.py

@@ -0,0 +1,37 @@
+"""add_chinese_holidays_table
+
+Revision ID: 1f59799a5083
+Revises: eed20ee8cc26
+Create Date: 2026-05-09 18:14:31.144256
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '1f59799a5083'
+down_revision: Union[str, Sequence[str], None] = 'eed20ee8cc26'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    op.create_table(
+        'chinese_holidays',
+        sa.Column('date', sa.Date(), nullable=False),
+        sa.Column('is_workday', sa.Boolean(), nullable=False, server_default='true'),
+        sa.Column('year', sa.Integer(), nullable=False),
+        sa.Column('description', sa.String(length=100), nullable=False, server_default=''),
+        sa.PrimaryKeyConstraint('date'),
+    )
+    op.create_index('idx_chinese_holidays_year', 'chinese_holidays', ['year'])
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    op.drop_index('idx_chinese_holidays_year', table_name='chinese_holidays')
+    op.drop_table('chinese_holidays')

alembic/versions/567cd83d63c2_create_announcements_table.py

@@ -0,0 +1,194 @@
+"""create_announcements_table
+
+Revision ID: 567cd83d63c2
+Revises: 
+Create Date: 2026-05-09 13:33:26.593143
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+# revision identifiers, used by Alembic.
+revision: str = '567cd83d63c2'
+down_revision: Union[str, Sequence[str], None] = None
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table('auto_announcements')
+    op.drop_table('announcement_sources')
+    op.drop_index(op.f('idx_dahuagov_content_hash'), table_name='dahuagov_announcements')
+    op.drop_index(op.f('idx_dahuagov_created_at'), table_name='dahuagov_announcements')
+    op.drop_index(op.f('idx_dahuagov_publish_date'), table_name='dahuagov_announcements')
+    op.drop_table('dahuagov_announcements')
+    op.drop_index(op.f('idx_crawl_results_crawled_at'), table_name='crawl_results')
+    op.drop_table('crawl_results')
+    op.drop_table('manual_announcements')
+    op.add_column('announcements', sa.Column('is_sent', sa.Boolean(), nullable=False))
+    op.alter_column('announcements', 'purchase_name',
+               existing_type=sa.VARCHAR(length=200),
+               nullable=False)
+    op.alter_column('announcements', 'content_url',
+               existing_type=sa.TEXT(),
+               nullable=False)
+    op.alter_column('announcements', 'content_hash',
+               existing_type=sa.VARCHAR(length=32),
+               type_=sa.String(length=64),
+               nullable=False)
+    op.alter_column('announcements', 'crawl_mode',
+               existing_type=sa.VARCHAR(length=20),
+               nullable=False,
+               existing_server_default=sa.text("'auto'::character varying"))
+    op.alter_column('announcements', 'is_new',
+               existing_type=sa.BOOLEAN(),
+               nullable=False,
+               existing_server_default=sa.text('true'))
+    op.alter_column('announcements', 'keyword_matched',
+               existing_type=sa.BOOLEAN(),
+               nullable=False,
+               existing_server_default=sa.text('false'))
+    op.alter_column('announcements', 'created_at',
+               existing_type=postgresql.TIMESTAMP(),
+               nullable=False,
+               existing_server_default=sa.text('CURRENT_TIMESTAMP'))
+    op.alter_column('announcements', 'updated_at',
+               existing_type=postgresql.TIMESTAMP(),
+               nullable=False,
+               existing_server_default=sa.text('CURRENT_TIMESTAMP'))
+    op.drop_index(op.f('idx_announcements_content_hash'), table_name='announcements')
+    op.drop_index(op.f('idx_announcements_created_at'), table_name='announcements')
+    op.drop_index(op.f('idx_announcements_publish_date'), table_name='announcements')
+    op.drop_index(op.f('idx_announcements_source_code'), table_name='announcements')
+    op.drop_column('announcements', 'date_filtered')
+    op.drop_column('announcements', 'is_today')
+    op.drop_column('announcements', 'crawled_at')
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('announcements', sa.Column('crawled_at', postgresql.TIMESTAMP(), autoincrement=False, nullable=True))
+    op.add_column('announcements', sa.Column('is_today', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True))
+    op.add_column('announcements', sa.Column('date_filtered', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True))
+    op.create_index(op.f('idx_announcements_source_code'), 'announcements', ['source_code'], unique=False)
+    op.create_index(op.f('idx_announcements_publish_date'), 'announcements', [sa.literal_column('publish_date DESC')], unique=False)
+    op.create_index(op.f('idx_announcements_created_at'), 'announcements', [sa.literal_column('created_at DESC')], unique=False)
+    op.create_index(op.f('idx_announcements_content_hash'), 'announcements', ['content_hash'], unique=False)
+    op.alter_column('announcements', 'updated_at',
+               existing_type=postgresql.TIMESTAMP(),
+               nullable=True,
+               existing_server_default=sa.text('CURRENT_TIMESTAMP'))
+    op.alter_column('announcements', 'created_at',
+               existing_type=postgresql.TIMESTAMP(),
+               nullable=True,
+               existing_server_default=sa.text('CURRENT_TIMESTAMP'))
+    op.alter_column('announcements', 'keyword_matched',
+               existing_type=sa.BOOLEAN(),
+               nullable=True,
+               existing_server_default=sa.text('false'))
+    op.alter_column('announcements', 'is_new',
+               existing_type=sa.BOOLEAN(),
+               nullable=True,
+               existing_server_default=sa.text('true'))
+    op.alter_column('announcements', 'crawl_mode',
+               existing_type=sa.VARCHAR(length=20),
+               nullable=True,
+               existing_server_default=sa.text("'auto'::character varying"))
+    op.alter_column('announcements', 'content_hash',
+               existing_type=sa.String(length=64),
+               type_=sa.VARCHAR(length=32),
+               nullable=True)
+    op.alter_column('announcements', 'content_url',
+               existing_type=sa.TEXT(),
+               nullable=True)
+    op.alter_column('announcements', 'purchase_name',
+               existing_type=sa.VARCHAR(length=200),
+               nullable=True)
+    op.drop_column('announcements', 'is_sent')
+    op.create_table('manual_announcements',
+    sa.Column('id', sa.INTEGER(), autoincrement=True, nullable=False),
+    sa.Column('title', sa.VARCHAR(length=500), autoincrement=False, nullable=False),
+    sa.Column('publish_date', postgresql.TIMESTAMP(), autoincrement=False, nullable=False),
+    sa.Column('purchase_name', sa.VARCHAR(length=200), autoincrement=False, nullable=True),
+    sa.Column('content_url', sa.TEXT(), autoincrement=False, nullable=True),
+    sa.Column('source_code', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('source_name', sa.VARCHAR(length=100), autoincrement=False, nullable=False),
+    sa.Column('announcement_type', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('crawled_at', postgresql.TIMESTAMP(), autoincrement=False, nullable=True),
+    sa.Column('created_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('updated_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('content_hash', sa.VARCHAR(length=32), autoincrement=False, nullable=True),
+    sa.Column('keyword_matched', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True),
+    sa.Column('date_filtered', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True),
+    sa.Column('is_new', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True),
+    sa.Column('is_today', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True),
+    sa.PrimaryKeyConstraint('id', name=op.f('manual_announcements_pkey'))
+    )
+    op.create_table('crawl_results',
+    sa.Column('id', sa.INTEGER(), autoincrement=True, nullable=False),
+    sa.Column('source_code', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('status', sa.VARCHAR(length=20), autoincrement=False, nullable=False),
+    sa.Column('total_count', sa.INTEGER(), server_default=sa.text('0'), autoincrement=False, nullable=True),
+    sa.Column('new_count', sa.INTEGER(), server_default=sa.text('0'), autoincrement=False, nullable=True),
+    sa.Column('error_message', sa.TEXT(), autoincrement=False, nullable=True),
+    sa.Column('crawled_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('duration', sa.DOUBLE_PRECISION(precision=53), server_default=sa.text('0.0'), autoincrement=False, nullable=True),
+    sa.ForeignKeyConstraint(['source_code'], ['announcement_sources.code'], name=op.f('crawl_results_source_code_fkey')),
+    sa.PrimaryKeyConstraint('id', name=op.f('crawl_results_pkey'))
+    )
+    op.create_index(op.f('idx_crawl_results_crawled_at'), 'crawl_results', [sa.literal_column('crawled_at DESC')], unique=False)
+    op.create_table('dahuagov_announcements',
+    sa.Column('id', sa.INTEGER(), autoincrement=True, nullable=False),
+    sa.Column('title', sa.VARCHAR(length=500), autoincrement=False, nullable=False),
+    sa.Column('publish_date', postgresql.TIMESTAMP(), autoincrement=False, nullable=False),
+    sa.Column('purchase_name', sa.VARCHAR(length=200), autoincrement=False, nullable=True),
+    sa.Column('content_url', sa.TEXT(), autoincrement=False, nullable=True),
+    sa.Column('source_code', sa.VARCHAR(length=50), server_default=sa.text("'dahuagov'::character varying"), autoincrement=False, nullable=False),
+    sa.Column('source_name', sa.VARCHAR(length=100), server_default=sa.text("'大化县政府网采购公告'::character varying"), autoincrement=False, nullable=False),
+    sa.Column('announcement_type', sa.VARCHAR(length=50), server_default=sa.text("'purchase'::character varying"), autoincrement=False, nullable=False),
+    sa.Column('crawled_at', postgresql.TIMESTAMP(), autoincrement=False, nullable=True),
+    sa.Column('created_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('updated_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('content_hash', sa.VARCHAR(length=32), autoincrement=False, nullable=True),
+    sa.Column('is_new', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True),
+    sa.PrimaryKeyConstraint('id', name=op.f('dahuagov_announcements_pkey')),
+    sa.UniqueConstraint('content_hash', name=op.f('dahuagov_announcements_content_hash_key'), postgresql_include=[], postgresql_nulls_not_distinct=False)
+    )
+    op.create_index(op.f('idx_dahuagov_publish_date'), 'dahuagov_announcements', [sa.literal_column('publish_date DESC')], unique=False)
+    op.create_index(op.f('idx_dahuagov_created_at'), 'dahuagov_announcements', [sa.literal_column('created_at DESC')], unique=False)
+    op.create_index(op.f('idx_dahuagov_content_hash'), 'dahuagov_announcements', ['content_hash'], unique=False)
+    op.create_table('announcement_sources',
+    sa.Column('code', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('category_id', sa.INTEGER(), autoincrement=False, nullable=False),
+    sa.Column('name', sa.VARCHAR(length=100), autoincrement=False, nullable=False),
+    sa.Column('type', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.PrimaryKeyConstraint('code', name=op.f('announcement_sources_pkey'))
+    )
+    op.create_table('auto_announcements',
+    sa.Column('id', sa.INTEGER(), autoincrement=True, nullable=False),
+    sa.Column('title', sa.VARCHAR(length=500), autoincrement=False, nullable=False),
+    sa.Column('publish_date', postgresql.TIMESTAMP(), autoincrement=False, nullable=False),
+    sa.Column('purchase_name', sa.VARCHAR(length=200), autoincrement=False, nullable=True),
+    sa.Column('content_url', sa.TEXT(), autoincrement=False, nullable=True),
+    sa.Column('source_code', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('source_name', sa.VARCHAR(length=100), autoincrement=False, nullable=False),
+    sa.Column('announcement_type', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('crawled_at', postgresql.TIMESTAMP(), autoincrement=False, nullable=True),
+    sa.Column('created_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('updated_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('content_hash', sa.VARCHAR(length=32), autoincrement=False, nullable=True),
+    sa.Column('keyword_matched', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True),
+    sa.Column('date_filtered', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True),
+    sa.Column('is_new', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True),
+    sa.Column('is_today', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True),
+    sa.PrimaryKeyConstraint('id', name=op.f('auto_announcements_pkey')),
+    sa.UniqueConstraint('content_hash', name=op.f('auto_announcements_content_hash_key'), postgresql_include=[], postgresql_nulls_not_distinct=False)
+    )
+    # ### end Alembic commands ###

alembic/versions/6e8f4c2d1b0a_add_ai_analysis_columns.py

@@ -0,0 +1,30 @@
+"""add_ai_analysis_columns
+
+Revision ID: 6e8f4c2d1b0a
+Revises: 1f59799a5083
+Create Date: 2026-05-21 10:00:00.000000
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '6e8f4c2d1b0a'
+down_revision: Union[str, Sequence[str], None] = '1f59799a5083'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """添加 AI 分析相关字段"""
+    op.add_column('announcements', sa.Column('ai_relevant', sa.Boolean(), nullable=True))
+    op.add_column('announcements', sa.Column('ai_analysis', sa.Text(), nullable=True))
+
+
+def downgrade() -> None:
+    """回滚"""
+    op.drop_column('announcements', 'ai_analysis')
+    op.drop_column('announcements', 'ai_relevant')

alembic/versions/eed20ee8cc26_add_is_sent_column.py

@@ -0,0 +1,194 @@
+"""add_is_sent_column
+
+Revision ID: eed20ee8cc26
+Revises: 567cd83d63c2
+Create Date: 2026-05-09 15:28:04.189854
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+# revision identifiers, used by Alembic.
+revision: str = 'eed20ee8cc26'
+down_revision: Union[str, Sequence[str], None] = '567cd83d63c2'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_index(op.f('idx_crawl_results_crawled_at'), table_name='crawl_results')
+    op.drop_table('crawl_results')
+    op.drop_table('auto_announcements')
+    op.drop_index(op.f('idx_dahuagov_content_hash'), table_name='dahuagov_announcements')
+    op.drop_index(op.f('idx_dahuagov_created_at'), table_name='dahuagov_announcements')
+    op.drop_index(op.f('idx_dahuagov_publish_date'), table_name='dahuagov_announcements')
+    op.drop_table('dahuagov_announcements')
+    op.drop_table('manual_announcements')
+    op.drop_table('announcement_sources')
+    op.add_column('announcements', sa.Column('is_sent', sa.Boolean(), nullable=False, server_default=sa.false()))
+    op.alter_column('announcements', 'purchase_name',
+               existing_type=sa.VARCHAR(length=200),
+               nullable=False)
+    op.alter_column('announcements', 'content_url',
+               existing_type=sa.TEXT(),
+               nullable=False)
+    op.alter_column('announcements', 'content_hash',
+               existing_type=sa.VARCHAR(length=32),
+               type_=sa.String(length=64),
+               nullable=False)
+    op.alter_column('announcements', 'crawl_mode',
+               existing_type=sa.VARCHAR(length=20),
+               nullable=False,
+               existing_server_default=sa.text("'auto'::character varying"))
+    op.alter_column('announcements', 'is_new',
+               existing_type=sa.BOOLEAN(),
+               nullable=False,
+               existing_server_default=sa.text('true'))
+    op.alter_column('announcements', 'keyword_matched',
+               existing_type=sa.BOOLEAN(),
+               nullable=False,
+               existing_server_default=sa.text('false'))
+    op.alter_column('announcements', 'created_at',
+               existing_type=postgresql.TIMESTAMP(),
+               nullable=False,
+               existing_server_default=sa.text('CURRENT_TIMESTAMP'))
+    op.alter_column('announcements', 'updated_at',
+               existing_type=postgresql.TIMESTAMP(),
+               nullable=False,
+               existing_server_default=sa.text('CURRENT_TIMESTAMP'))
+    op.drop_index(op.f('idx_announcements_content_hash'), table_name='announcements')
+    op.drop_index(op.f('idx_announcements_created_at'), table_name='announcements')
+    op.drop_index(op.f('idx_announcements_publish_date'), table_name='announcements')
+    op.drop_index(op.f('idx_announcements_source_code'), table_name='announcements')
+    op.drop_column('announcements', 'date_filtered')
+    op.drop_column('announcements', 'crawled_at')
+    op.drop_column('announcements', 'is_today')
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('announcements', sa.Column('is_today', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True))
+    op.add_column('announcements', sa.Column('crawled_at', postgresql.TIMESTAMP(), autoincrement=False, nullable=True))
+    op.add_column('announcements', sa.Column('date_filtered', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True))
+    op.create_index(op.f('idx_announcements_source_code'), 'announcements', ['source_code'], unique=False)
+    op.create_index(op.f('idx_announcements_publish_date'), 'announcements', [sa.literal_column('publish_date DESC')], unique=False)
+    op.create_index(op.f('idx_announcements_created_at'), 'announcements', [sa.literal_column('created_at DESC')], unique=False)
+    op.create_index(op.f('idx_announcements_content_hash'), 'announcements', ['content_hash'], unique=False)
+    op.alter_column('announcements', 'updated_at',
+               existing_type=postgresql.TIMESTAMP(),
+               nullable=True,
+               existing_server_default=sa.text('CURRENT_TIMESTAMP'))
+    op.alter_column('announcements', 'created_at',
+               existing_type=postgresql.TIMESTAMP(),
+               nullable=True,
+               existing_server_default=sa.text('CURRENT_TIMESTAMP'))
+    op.alter_column('announcements', 'keyword_matched',
+               existing_type=sa.BOOLEAN(),
+               nullable=True,
+               existing_server_default=sa.text('false'))
+    op.alter_column('announcements', 'is_new',
+               existing_type=sa.BOOLEAN(),
+               nullable=True,
+               existing_server_default=sa.text('true'))
+    op.alter_column('announcements', 'crawl_mode',
+               existing_type=sa.VARCHAR(length=20),
+               nullable=True,
+               existing_server_default=sa.text("'auto'::character varying"))
+    op.alter_column('announcements', 'content_hash',
+               existing_type=sa.String(length=64),
+               type_=sa.VARCHAR(length=32),
+               nullable=True)
+    op.alter_column('announcements', 'content_url',
+               existing_type=sa.TEXT(),
+               nullable=True)
+    op.alter_column('announcements', 'purchase_name',
+               existing_type=sa.VARCHAR(length=200),
+               nullable=True)
+    op.drop_column('announcements', 'is_sent')
+    op.create_table('announcement_sources',
+    sa.Column('code', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('category_id', sa.INTEGER(), autoincrement=False, nullable=False),
+    sa.Column('name', sa.VARCHAR(length=100), autoincrement=False, nullable=False),
+    sa.Column('type', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.PrimaryKeyConstraint('code', name=op.f('announcement_sources_pkey'))
+    )
+    op.create_table('manual_announcements',
+    sa.Column('id', sa.INTEGER(), autoincrement=True, nullable=False),
+    sa.Column('title', sa.VARCHAR(length=500), autoincrement=False, nullable=False),
+    sa.Column('publish_date', postgresql.TIMESTAMP(), autoincrement=False, nullable=False),
+    sa.Column('purchase_name', sa.VARCHAR(length=200), autoincrement=False, nullable=True),
+    sa.Column('content_url', sa.TEXT(), autoincrement=False, nullable=True),
+    sa.Column('source_code', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('source_name', sa.VARCHAR(length=100), autoincrement=False, nullable=False),
+    sa.Column('announcement_type', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('crawled_at', postgresql.TIMESTAMP(), autoincrement=False, nullable=True),
+    sa.Column('created_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('updated_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('content_hash', sa.VARCHAR(length=32), autoincrement=False, nullable=True),
+    sa.Column('keyword_matched', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True),
+    sa.Column('date_filtered', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True),
+    sa.Column('is_new', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True),
+    sa.Column('is_today', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True),
+    sa.PrimaryKeyConstraint('id', name=op.f('manual_announcements_pkey'))
+    )
+    op.create_table('dahuagov_announcements',
+    sa.Column('id', sa.INTEGER(), autoincrement=True, nullable=False),
+    sa.Column('title', sa.VARCHAR(length=500), autoincrement=False, nullable=False),
+    sa.Column('publish_date', postgresql.TIMESTAMP(), autoincrement=False, nullable=False),
+    sa.Column('purchase_name', sa.VARCHAR(length=200), autoincrement=False, nullable=True),
+    sa.Column('content_url', sa.TEXT(), autoincrement=False, nullable=True),
+    sa.Column('source_code', sa.VARCHAR(length=50), server_default=sa.text("'dahuagov'::character varying"), autoincrement=False, nullable=False),
+    sa.Column('source_name', sa.VARCHAR(length=100), server_default=sa.text("'大化县政府网采购公告'::character varying"), autoincrement=False, nullable=False),
+    sa.Column('announcement_type', sa.VARCHAR(length=50), server_default=sa.text("'purchase'::character varying"), autoincrement=False, nullable=False),
+    sa.Column('crawled_at', postgresql.TIMESTAMP(), autoincrement=False, nullable=True),
+    sa.Column('created_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('updated_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('content_hash', sa.VARCHAR(length=32), autoincrement=False, nullable=True),
+    sa.Column('is_new', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True),
+    sa.PrimaryKeyConstraint('id', name=op.f('dahuagov_announcements_pkey')),
+    sa.UniqueConstraint('content_hash', name=op.f('dahuagov_announcements_content_hash_key'), postgresql_include=[], postgresql_nulls_not_distinct=False)
+    )
+    op.create_index(op.f('idx_dahuagov_publish_date'), 'dahuagov_announcements', [sa.literal_column('publish_date DESC')], unique=False)
+    op.create_index(op.f('idx_dahuagov_created_at'), 'dahuagov_announcements', [sa.literal_column('created_at DESC')], unique=False)
+    op.create_index(op.f('idx_dahuagov_content_hash'), 'dahuagov_announcements', ['content_hash'], unique=False)
+    op.create_table('auto_announcements',
+    sa.Column('id', sa.INTEGER(), autoincrement=True, nullable=False),
+    sa.Column('title', sa.VARCHAR(length=500), autoincrement=False, nullable=False),
+    sa.Column('publish_date', postgresql.TIMESTAMP(), autoincrement=False, nullable=False),
+    sa.Column('purchase_name', sa.VARCHAR(length=200), autoincrement=False, nullable=True),
+    sa.Column('content_url', sa.TEXT(), autoincrement=False, nullable=True),
+    sa.Column('source_code', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('source_name', sa.VARCHAR(length=100), autoincrement=False, nullable=False),
+    sa.Column('announcement_type', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('crawled_at', postgresql.TIMESTAMP(), autoincrement=False, nullable=True),
+    sa.Column('created_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('updated_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('content_hash', sa.VARCHAR(length=32), autoincrement=False, nullable=True),
+    sa.Column('keyword_matched', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True),
+    sa.Column('date_filtered', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True),
+    sa.Column('is_new', sa.BOOLEAN(), server_default=sa.text('true'), autoincrement=False, nullable=True),
+    sa.Column('is_today', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True),
+    sa.PrimaryKeyConstraint('id', name=op.f('auto_announcements_pkey')),
+    sa.UniqueConstraint('content_hash', name=op.f('auto_announcements_content_hash_key'), postgresql_include=[], postgresql_nulls_not_distinct=False)
+    )
+    op.create_table('crawl_results',
+    sa.Column('id', sa.INTEGER(), autoincrement=True, nullable=False),
+    sa.Column('source_code', sa.VARCHAR(length=50), autoincrement=False, nullable=False),
+    sa.Column('status', sa.VARCHAR(length=20), autoincrement=False, nullable=False),
+    sa.Column('total_count', sa.INTEGER(), server_default=sa.text('0'), autoincrement=False, nullable=True),
+    sa.Column('new_count', sa.INTEGER(), server_default=sa.text('0'), autoincrement=False, nullable=True),
+    sa.Column('error_message', sa.TEXT(), autoincrement=False, nullable=True),
+    sa.Column('crawled_at', postgresql.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP'), autoincrement=False, nullable=True),
+    sa.Column('duration', sa.DOUBLE_PRECISION(precision=53), server_default=sa.text('0.0'), autoincrement=False, nullable=True),
+    sa.ForeignKeyConstraint(['source_code'], ['announcement_sources.code'], name=op.f('crawl_results_source_code_fkey')),
+    sa.PrimaryKeyConstraint('id', name=op.f('crawl_results_pkey'))
+    )
+    op.create_index(op.f('idx_crawl_results_crawled_at'), 'crawl_results', [sa.literal_column('crawled_at DESC')], unique=False)
+    # ### end Alembic commands ###

app/api/announcements.py

@@ -0,0 +1,116 @@
+from datetime import date, datetime
+
+from fastapi import APIRouter, Depends, HTTPException, Query
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.deps import get_db
+from app.models.announcement import Announcement
+from app.models.schemas import AnnouncementListResponse, AnnouncementResponse
+
+router = APIRouter()
+
+
+def _parse_date(value: str | None) -> date | None:
+    if not value:
+        return None
+    return datetime.strptime(value, "%Y-%m-%d").date()
+
+
+@router.get("/announcements", response_model=AnnouncementListResponse)
+async def list_announcements(
+    page: int = Query(1, ge=1),
+    page_size: int = Query(20, ge=1, le=100),
+    source_code: str | None = None,
+    keyword: str | None = None,
+    start_date: str | None = None,
+    end_date: str | None = None,
+    crawl_mode: str | None = None,
+    db: AsyncSession = Depends(get_db),
+):
+    conditions = []
+    if source_code:
+        conditions.append(Announcement.source_code == source_code)
+    if crawl_mode:
+        conditions.append(Announcement.crawl_mode == crawl_mode)
+    start = _parse_date(start_date)
+    end = _parse_date(end_date)
+    if start:
+        conditions.append(Announcement.publish_date >= start)
+    if end:
+        conditions.append(Announcement.publish_date <= end)
+    if keyword:
+        conditions.append(
+            Announcement.title.ilike(f"%{keyword}%")
+        )
+
+    base_query = select(Announcement)
+    if conditions:
+        base_query = base_query.where(*conditions)
+
+    count_query = select(func.count()).select_from(base_query.subquery())
+    total_result = await db.execute(count_query)
+    total = total_result.scalar() or 0
+
+    items_query = base_query.order_by(Announcement.publish_date.desc()) \
+        .offset((page - 1) * page_size).limit(page_size)
+    items_result = await db.execute(items_query)
+    items = items_result.scalars().all()
+
+    return AnnouncementListResponse(
+        total=total,
+        page=page,
+        page_size=page_size,
+        items=[AnnouncementResponse.model_validate(item) for item in items],
+    )
+
+
+@router.get("/announcements/today", response_model=AnnouncementListResponse)
+async def get_today_announcements(db: AsyncSession = Depends(get_db)):
+    today = datetime.now().date()
+    result = await db.execute(
+        select(Announcement).where(
+            func.date(Announcement.publish_date) == today
+        ).order_by(Announcement.publish_date.desc())
+    )
+    items = result.scalars().all()
+    return AnnouncementListResponse(
+        total=len(items), page=1, page_size=len(items),
+        items=[AnnouncementResponse.model_validate(item) for item in items],
+    )
+
+
+@router.get("/announcements/stats")
+async def get_stats(db: AsyncSession = Depends(get_db)):
+    total = await db.execute(select(func.count()).select_from(Announcement))
+    today_count = await db.execute(
+        select(func.count()).where(
+            func.date(Announcement.publish_date) == func.current_date()
+        ).select_from(Announcement)
+    )
+    new_count = await db.execute(
+        select(func.count()).where(Announcement.is_new == True)  # noqa: E712
+        .select_from(Announcement)
+    )
+    unsent = await db.execute(
+        select(func.count()).where(
+            Announcement.is_sent == False, Announcement.is_new == True  # noqa: E712
+        ).select_from(Announcement)
+    )
+    return {
+        "total": total.scalar() or 0,
+        "today": today_count.scalar() or 0,
+        "new": new_count.scalar() or 0,
+        "unsent": unsent.scalar() or 0,
+    }
+
+
+@router.get("/announcements/{announcement_id}", response_model=AnnouncementResponse)
+async def get_announcement(announcement_id: int, db: AsyncSession = Depends(get_db)):
+    result = await db.execute(
+        select(Announcement).where(Announcement.id == announcement_id)
+    )
+    item = result.scalar_one_or_none()
+    if item is None:
+        raise HTTPException(status_code=404, detail="公告不存在")
+    return AnnouncementResponse.model_validate(item)

app/api/crawl.py

@@ -0,0 +1,68 @@
+from fastapi import APIRouter
+
+from app.api.deps import get_crawl_service
+from app.models.schemas import CrawlTriggerRequest
+
+router = APIRouter()
+
+
+@router.post("/crawl/trigger")
+async def trigger_crawl(request: CrawlTriggerRequest):
+    service = get_crawl_service()
+    if request.spider_name:
+        names = [request.spider_name]
+    else:
+        names = service.get_spider_names()
+
+    all_results = []
+    total_stored = 0
+    total_notified = 0
+    for name in names:
+        results = await service.run_spider(name)
+        all_results.extend(results)
+        for r in results:
+            if r.pipeline_result:
+                total_stored += r.pipeline_result.stored
+                total_notified += r.pipeline_result.notified
+
+    return {
+        "spiders_run": names,
+        "total_announcements": sum(r.total_count for r in all_results),
+        "total_stored": total_stored,
+        "total_notified": total_notified,
+        "errors": [r.error_message for r in all_results if not r.success],
+    }
+
+
+@router.get("/crawl/status")
+async def crawl_status():
+    service = get_crawl_service()
+    return {
+        "spiders": service.get_spider_names(),
+        "running": False,
+    }
+
+
+@router.get("/crawl/sources")
+async def crawl_sources():
+    import json
+
+    from app.config import settings
+    sources = json.loads(settings.announcement_sources)
+    result = [
+        {"code": code, "name": info["name"], "type": info["type"]}
+        for code, info in sources.items()
+    ]
+    # 加入独立爬虫来源（非 GXGP 子来源的独立 Spider）
+    service = get_crawl_service()
+    for name in service.get_spider_names():
+        if name == "gxgp":
+            continue  # gxgp 的子来源已在上面列出
+        spider = service.spiders.get(name)
+        if spider:
+            result.append({
+                "code": spider.source_code,
+                "name": spider.source_name,
+                "type": "independent",
+            })
+    return {"sources": result}

app/api/deps.py

@@ -0,0 +1,28 @@
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.crawler.dahuagov_spider import DahuagovSpider
+from app.crawler.gxgp_spider import GXGPSpider
+from app.services.crawl_service import CrawlService
+from app.services.notification_service import NotificationService
+
+
+async def get_db() -> AsyncSession:
+    from app.main import async_session
+    async with async_session() as session:
+        yield session
+
+
+_crawl_service: CrawlService | None = None
+
+
+def get_crawl_service() -> CrawlService:
+    global _crawl_service
+    if _crawl_service is None:
+        from app.main import async_session
+        _crawl_service = CrawlService(
+            db_session_factory=async_session,
+            notification_service=NotificationService(),
+        )
+        _crawl_service.register(GXGPSpider())
+        _crawl_service.register(DahuagovSpider())
+    return _crawl_service

app/api/holidays.py

@@ -0,0 +1,30 @@
+from fastapi import APIRouter, Depends, Query
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.deps import get_db
+from app.services.holiday_service import now_in_china, sync_holidays
+
+router = APIRouter()
+
+
+@router.post("/holidays/sync")
+async def sync_holidays_endpoint(
+    year: int | None = Query(None, description="同步年份，默认当前年份"),
+    db: AsyncSession = Depends(get_db),
+):
+    if year is None:
+        year = now_in_china().year
+    count = await sync_holidays(db, year)
+    return {"status": "ok", "year": year, "synced": count}
+
+
+@router.get("/holidays/today")
+async def get_today_status(db: AsyncSession = Depends(get_db)):
+    from app.services.holiday_service import is_workday
+    today = now_in_china()
+    workday = await is_workday(db, today)
+    return {
+        "date": today.isoformat(),
+        "is_workday": workday,
+        "message": "工作日，正常爬取" if workday else "非工作日，跳过爬取",
+    }

app/api/router.py

@@ -0,0 +1,11 @@
+from fastapi import APIRouter
+
+from app.api import announcements, crawl, holidays, wechat
+from app.api import scheduler as scheduler_module
+
+api_router = APIRouter(prefix="/api/v1")
+api_router.include_router(announcements.router, tags=["announcements"])
+api_router.include_router(crawl.router, tags=["crawl"])
+api_router.include_router(holidays.router, tags=["holidays"])
+api_router.include_router(wechat.router, tags=["wechat"])
+api_router.include_router(scheduler_module.router)

app/api/scheduler.py

@@ -0,0 +1,30 @@
+from fastapi import APIRouter
+
+from app.models.schemas import JobResponse
+from app.scheduler.jobs import scheduler
+
+router = APIRouter(prefix="/scheduler", tags=["scheduler"])
+
+
+@router.get("/jobs")
+async def list_jobs():
+    jobs = []
+    for job in scheduler.get_jobs():
+        jobs.append(JobResponse(
+            id=job.id,
+            name=job.name,
+            next_run_time=str(job.next_run_time) if job.next_run_time else None,
+        ))
+    return {"jobs": jobs}
+
+
+@router.post("/pause/{job_id}")
+async def pause_job(job_id: str):
+    scheduler.pause_job(job_id)
+    return {"status": "paused", "job_id": job_id}
+
+
+@router.post("/resume/{job_id}")
+async def resume_job(job_id: str):
+    scheduler.resume_job(job_id)
+    return {"status": "resumed", "job_id": job_id}

app/api/wechat.py

@@ -0,0 +1,68 @@
+from fastapi import APIRouter, Request, Response
+
+from app.wechat.handler import WeChatMessageHandler
+
+router = APIRouter()
+
+_handler: WeChatMessageHandler | None = None
+
+
+def get_handler() -> WeChatMessageHandler:
+    global _handler
+    if _handler is None:
+        _handler = WeChatMessageHandler()
+    return _handler
+
+
+@router.get("/wechat/callback")
+async def wechat_verify(request: Request):
+    handler = get_handler()
+    params = request.query_params
+    echostr = handler.verify_url(
+        params.get("msg_signature", ""),
+        params.get("timestamp", ""),
+        params.get("nonce", ""),
+        params.get("echostr", ""),
+    )
+    if echostr:
+        return Response(content=echostr, media_type="text/plain")
+    return Response(content="verification failed", status_code=403)
+
+
+@router.post("/wechat/callback")
+async def wechat_callback(request: Request):
+    handler = get_handler()
+    params = request.query_params
+    post_data = await request.body()
+    post_text = post_data.decode("utf-8")
+
+    xml_tree = handler.decrypt_message(
+        post_text,
+        params.get("msg_signature", ""),
+        params.get("timestamp", ""),
+        params.get("nonce", ""),
+    )
+    if xml_tree is None:
+        return Response(content="decrypt failed", status_code=403)
+
+    msg_type = xml_tree.find("MsgType")
+    msg_type = msg_type.text if msg_type is not None else "unknown"
+
+    if msg_type == "event":
+        event = xml_tree.find("Event")
+        event_key = xml_tree.find("EventKey")
+        from_user = xml_tree.find("FromUserName")
+        await handler.handle_event(
+            event.text if event is not None else "",
+            event_key.text if event_key is not None else None,
+            from_user.text if from_user is not None else "",
+        )
+    elif msg_type == "text":
+        content = xml_tree.find("Content")
+        from_user = xml_tree.find("FromUserName")
+        await handler.handle_text(
+            content.text if content is not None else "",
+            from_user.text if from_user is not None else "",
+        )
+
+    return Response(content="success")

app/config.py

@@ -0,0 +1,83 @@
+
+from pydantic_settings import BaseSettings, SettingsConfigDict
+from pydantic import field_validator
+
+
+class Settings(BaseSettings):
+    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")
+
+    # 应用
+    debug: bool = False
+    log_level: str = "INFO"
+
+    # 数据库
+    database_url: str = "postgresql+asyncpg://gx-gp-notify:password@localhost:5432/gx-gp-notify"
+
+    # 爬虫
+    crawler_base_url: str = "https://zfcg.gxzf.gov.cn"
+    crawler_keywords: list[str] = ["大化"]
+    crawler_max_pages: int = 10
+    crawler_timeout: int = 30
+    crawler_page_size: int = 100
+
+    # 企业微信
+    wechat_enabled: bool = True
+    wechat_corp_id: str = ""
+    wechat_agent_id: str = ""
+    wechat_secret: str = ""
+    wechat_token: str = ""
+    wechat_encoding_aes_key: str = ""
+    wechat_port: int = 18001
+    wechat_host: str = "0.0.0.0"
+    wechat_api_base_url: str = "https://qyapi.weixin.qq.com"
+
+    # 定时任务
+    scheduler_enabled: bool = True
+    scheduler_cron: str = "0 8-21 * * *"
+
+    # LogHive 日志系统
+    loghive_endpoint: str = "http://10.10.10.14:8000"
+    loghive_project: str = "gx-gp-notify"
+    loghive_api_key: str = ""
+
+    # AI 分析 (DeepSeek)
+    ai_enabled: bool = False
+    ai_whitelist: str = ""
+    ai_api_key: str = ""
+    ai_base_url: str = "https://api.deepseek.com/v1"
+    ai_model: str = "deepseek-chat"
+    ai_timeout: int = 30
+    ai_analysis_title: str = "🔔 中国电信可承接项目"
+    ai_prompt_template: str = (
+        "你是一个政府采购项目分析师，专门帮助中国电信识别可以承接的项目。\n\n"
+        "可承接范围包括但不限于：\n"
+        "- 通信工程、光缆建设、基站建设\n"
+        "- 信息化系统建设、系统集成\n"
+        "- 云计算、大数据、政务云\n"
+        "- 物联网、智慧城市、智慧园区\n"
+        "- 安防监控、视频会议、应急通信\n"
+        "- 网络运维、网络优化、IDC 服务\n"
+        "- 5G 应用、专线服务\n\n"
+        "--- 公告信息 ---\n"
+        "标题：{title}\n"
+        "采购人：{purchase_name}\n"
+        "公告类型：{announcement_type}\n\n"
+        "--- 公告正文 ---\n"
+        "{content}\n\n"
+        "请用 JSON 格式回答：\n"
+        '{{"is_relevant": true/false, "reason": "简要判断理由", "business_type": "业务分类"}}'
+    )
+
+    @field_validator("ai_prompt_template", mode="before")
+    @classmethod
+    def convert_newlines(cls, v: str) -> str:
+        """将 .env 中字面 \\n 转换为真实换行"""
+        if isinstance(v, str) and "\\n" in v:
+            return v.replace("\\n", "\n")
+        return v
+
+    # 公告来源（JSON 字符串，从环境变量读取）
+    announcement_sources: str = '{"ZcyAnnouncement1":{"category_id":66485,"name":"采购公告","type":"purchase"},"ZcyAnnouncement2":{"category_id":66485,"name":"结果公告","type":"result"},"ZcyAnnouncement3":{"category_id":66485,"name":"合同公告","type":"contract"},"ZcyAnnouncement4":{"category_id":66485,"name":"更正公告","type":"correction"},"ZcyAnnouncement5":{"category_id":66485,"name":"招标文件预公示","type":"pre_announcement"},"ZcyAnnouncement6":{"category_id":66485,"name":"单一来源公示","type":"single_source"},"ZcyAnnouncement7":{"category_id":66485,"name":"电子卖场公示","type":"electronic_market"},"ZcyAnnouncement10":{"category_id":66485,"name":"履约验收公示","type":"acceptance"},"ZcyAnnouncement11":{"category_id":66485,"name":"工程类公告","type":"engineering"},"ZcyAnnouncement20":{"category_id":66485,"name":"框架协议征集公告","type":"framework_agreement"},"ZcyAnnouncement21":{"category_id":66485,"name":"框架协议入围结果公告","type":"framework_result"},"ZcyAnnouncement23":{"category_id":66485,"name":"框架协议成交结果汇总公告","type":"framework_summary"},"61-266648":{"category_id":66485,"name":"采购意向公开","type":"intention"}}'  # noqa: E501
+
+
+settings = Settings()

app/crawler/base.py

@@ -0,0 +1,56 @@
+import hashlib
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+from datetime import datetime
+
+
+@dataclass
+class PipelineResult:
+    stored: int = 0
+    filtered: int = 0
+    notified: int = 0
+
+
+@dataclass
+class CrawlResult:
+    source_code: str
+    source_name: str
+    total_count: int = 0
+    new_count: int = 0
+    announcements: list = field(default_factory=list)
+    error_message: str | None = None
+    crawled_at: datetime = field(default_factory=datetime.now)
+    duration: float = 0.0
+    pipeline_result: PipelineResult | None = None
+
+    @property
+    def success(self) -> bool:
+        return self.error_message is None
+
+
+@dataclass
+class PipelineConfig:
+    filter_enabled: bool = True
+    keywords: list[str] = field(default_factory=list)
+    dedup_enabled: bool = True
+    notify_mode: str = "filtered"
+    mark_sent: bool = False
+
+
+class BaseSpider(ABC):
+    name: str
+    source_code: str
+    source_name: str
+
+    @abstractmethod
+    async def crawl(self) -> CrawlResult:
+        ...
+
+    def get_pipeline_config(self) -> PipelineConfig:
+        return PipelineConfig()
+
+    @staticmethod
+    def generate_content_hash(title: str, publish_date: str, purchase_name: str,
+                              content_url: str, source_code: str) -> str:
+        content = f"{title}|{publish_date}|{purchase_name}|{content_url}|{source_code}"
+        return hashlib.sha256(content.encode("utf-8")).hexdigest()

app/crawler/dahuagov_spider.py

@@ -0,0 +1,101 @@
+import asyncio
+import random
+from datetime import datetime
+
+import httpx
+
+from app.config import settings
+from app.crawler.base import BaseSpider, CrawlResult, PipelineConfig
+from app.crawler.parsers import parse_dahuagov_html, parse_dahuagov_detail_pubdate
+
+
+class DahuagovSpider(BaseSpider):
+    name = "dahuagov"
+    source_code = "dahuagov"
+    source_name = "大化县政府网采购公告"
+
+    BASE_URL = "http://www.gxdh.gov.cn"
+    ANNOUNCEMENT_PATH = "/xxgk/zdlyxxgk/ggzypzly/zfcgly/cggg/"
+
+    def get_pipeline_config(self) -> PipelineConfig:
+        return PipelineConfig(
+            filter_enabled=False,
+            keywords=[],
+            dedup_enabled=True,
+            notify_mode="all",
+            mark_sent=True,
+        )
+
+    async def crawl(self) -> CrawlResult:
+        start_time = datetime.now()
+        url = self.BASE_URL + self.ANNOUNCEMENT_PATH
+
+        async with httpx.AsyncClient(timeout=settings.crawler_timeout) as client:
+            await self._delay()
+            try:
+                headers = {
+                    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
+                    "Accept": "text/html,application/xhtml+xml",
+                    "Accept-Language": "zh-CN,zh;q=0.9",
+                    "Referer": self.BASE_URL,
+                }
+                response = await client.get(url, headers=headers)
+                if response.status_code != 200:
+                    return CrawlResult(
+                        source_code=self.source_code,
+                        source_name=self.source_name,
+                        error_message=f"HTTP {response.status_code}",
+                        crawled_at=start_time,
+                    )
+                html = response.text
+            except Exception as e:
+                return CrawlResult(
+                    source_code=self.source_code,
+                    source_name=self.source_name,
+                    error_message=str(e),
+                    crawled_at=start_time,
+                )
+
+            announcements = parse_dahuagov_html(html, start_time)
+            await self._fetch_detail_dates(client, announcements, headers)
+
+        duration = (datetime.now() - start_time).total_seconds()
+
+        return CrawlResult(
+            source_code=self.source_code,
+            source_name=self.source_name,
+            total_count=len(announcements),
+            new_count=len(announcements),
+            announcements=announcements,
+            crawled_at=start_time,
+            duration=duration,
+        )
+
+    async def _fetch_detail_dates(
+        self,
+        client: httpx.AsyncClient,
+        announcements: list[dict],
+        headers: dict,
+    ) -> None:
+        """并发抓取详情页，用 PubDate meta 更新精确发布时间"""
+        sem = asyncio.Semaphore(5)
+
+        async def fetch_one(ann: dict) -> None:
+            async with sem:
+                await asyncio.sleep(random.uniform(0.3, 0.8))
+                try:
+                    resp = await client.get(ann["content_url"], headers=headers)
+                    if resp.status_code == 200:
+                        pub = parse_dahuagov_detail_pubdate(resp.text)
+                        if pub:
+                            ann["publish_date"] = pub
+                            ann["is_today"] = pub.date() == datetime.now().date()
+                except Exception:
+                    pass  # 保留列表页日期作为 fallback
+
+        await asyncio.gather(*[fetch_one(ann) for ann in announcements])
+
+    async def _delay(self):
+        delay = random.uniform(1.0, 3.0)
+        await asyncio.sleep(delay)
+

app/crawler/gxgp_spider.py

@@ -0,0 +1,118 @@
+import json
+import random
+import time
+from datetime import datetime
+
+import httpx
+
+from app.config import settings
+from app.crawler.base import BaseSpider, CrawlResult, PipelineConfig
+from app.crawler.parsers import extract_pagination, parse_gxgp_api_response
+
+
+class GXGPSpider(BaseSpider):
+    name = "gxgp"
+    source_code = "gxgp"
+    source_name = "广西政府采购网"
+
+    def __init__(self):
+        self.base_url = settings.crawler_base_url
+        self.announcement_api = f"{self.base_url}/portal/category"
+
+    def get_pipeline_config(self) -> PipelineConfig:
+        return PipelineConfig(
+            filter_enabled=True,
+            keywords=list(settings.crawler_keywords),
+            dedup_enabled=True,
+            notify_mode="filtered",
+            mark_sent=True,
+        )
+
+    async def crawl(self, sources: list[str] | None = None,
+                    max_pages: int | None = None) -> CrawlResult:
+        if max_pages is None:
+            max_pages = settings.crawler_max_pages
+        if sources is None:
+            source_map = json.loads(settings.announcement_sources)
+            sources = list(source_map.keys())
+
+        start_time = datetime.now()
+        all_announcements = []
+        error_messages = []
+
+        async with httpx.AsyncClient(timeout=settings.crawler_timeout) as client:
+            for source_code in sources:
+                source_info = json.loads(settings.announcement_sources).get(source_code)
+                if not source_info:
+                    continue
+
+                category_id = source_info["category_id"]
+                source_name = source_info["name"]
+
+                for page_no in range(1, max_pages + 1):
+                    if page_no > 1:
+                        await self._delay()
+
+                    try:
+                        data = await self._fetch_page(
+                            client, source_code, category_id, page_no
+                        )
+                        if data is None:
+                            break
+
+                        records = parse_gxgp_api_response(
+                            data, source_code, source_name,
+                            start_time, category_id
+                        )
+                        if not records:
+                            break
+
+                        all_announcements.extend(records)
+
+                        pagination = extract_pagination(data)
+                        if not pagination["has_next"]:
+                            break
+                    except Exception as e:
+                        error_messages.append(f"{source_code} page {page_no}: {e}")
+                        break
+
+        duration = (datetime.now() - start_time).total_seconds()
+        return CrawlResult(
+            source_code=self.source_code,
+            source_name=self.source_name,
+            total_count=len(all_announcements),
+            new_count=len(all_announcements),
+            announcements=all_announcements,
+            error_message="; ".join(error_messages) if error_messages else None,
+            crawled_at=start_time,
+            duration=duration,
+        )
+
+    async def _fetch_page(self, client: httpx.AsyncClient, source_code: str,
+                          category_id: int, page_no: int) -> dict | None:
+        payload = {
+            "keyword": "",
+            "publishDateBegin": "",
+            "publishDateEnd": "",
+            "pageNo": page_no,
+            "pageSize": settings.crawler_page_size,
+            "categoryCode": source_code,
+            "_t": int(time.time() * 1000),
+        }
+        headers = {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
+            "Content-Type": "application/json;charset=UTF-8",
+            "Origin": self.base_url,
+            "Referer": f"{self.base_url}/site/category?parentId={category_id}&childrenCode={source_code}",  # noqa: E501
+        }
+        response = await client.post(
+            self.announcement_api, json=payload, headers=headers
+        )
+        if response.status_code != 200:
+            return None
+        return response.json()
+
+    async def _delay(self):
+        import asyncio
+        delay = random.uniform(1.0, 3.0)
+        await asyncio.sleep(delay)

app/crawler/parsers.py

@@ -0,0 +1,324 @@
+import hashlib
+from datetime import datetime
+from typing import Any
+from urllib.parse import urljoin
+from zoneinfo import ZoneInfo
+
+_TZ = ZoneInfo("Asia/Shanghai")
+
+from bs4 import BeautifulSoup
+
+
+def parse_gxgp_api_response(
+    response_data: dict[str, Any],
+    source_code: str,
+    source_name: str,
+    crawled_at: datetime,
+    category_id: int,
+) -> list[dict[str, Any]]:
+    if not response_data.get("success"):
+        return []
+    data = response_data.get("result", {}).get("data", {})
+    records = data.get("data", [])
+    if not records:
+        return []
+
+    results = []
+    for record in records:
+        title = str(record.get("title", "")).strip()
+        if not title:
+            continue
+
+        timestamp = record.get("publishDate")
+        if not timestamp:
+            continue
+        try:
+            publish_date = datetime.fromtimestamp(int(timestamp) / 1000, tz=_TZ).replace(tzinfo=None)
+        except (ValueError, TypeError):
+            continue
+
+        purchase_name = str(record.get("purchaseName", "")).strip()
+        article_id = record.get("articleId")
+        if not article_id:
+            continue
+
+        content_url = (
+            f"https://zfcg.gxzf.gov.cn/site/detail?"
+            f"parentId={category_id}&articleId={article_id}"
+        )
+
+        announce = {
+            "title": title,
+            "publish_date": publish_date,
+            "purchase_name": purchase_name,
+            "content_url": content_url,
+            "source_code": source_code,
+            "source_name": source_name,
+            "announcement_type": "purchase",
+            "crawl_mode": "auto",
+            "is_new": True,
+            "is_today": publish_date.date() == datetime.now().date(),
+        }
+        announce["content_hash"] = _generate_hash(announce)
+        results.append(announce)
+
+    return results
+
+
+def extract_pagination(response_data: dict[str, Any]) -> dict[str, Any]:
+    data = response_data.get("result", {}).get("data", {})
+    return {
+        "total": data.get("total", 0),
+        "page_no": data.get("pageNo", 1),
+        "page_size": data.get("pageSize", 100),
+        "pages": data.get("pages", 0),
+        "empty": data.get("empty", True),
+        "has_next": data.get("hasNext", False),
+        "has_previous": data.get("hasPrevious", False),
+    }
+
+
+def parse_dahuagov_html(html: str, crawled_at: datetime) -> list[dict[str, Any]]:
+    soup = BeautifulSoup(html, "html.parser")
+    lists = soup.find_all("ul", class_="more-list")
+    if not lists:
+        return []
+
+    results = []
+    base_url = "http://www.gxdh.gov.cn"
+    base_path = "/xxgk/zdlyxxgk/ggzypzly/zfcgly/cggg/"
+
+    for ul in lists:
+        for li in ul.find_all("li"):
+            date_span = li.find("span")
+            if not date_span:
+                continue
+            date_text = date_span.get_text(strip=True)
+            try:
+                publish_date = datetime.strptime(date_text, "%Y-%m-%d")
+            except ValueError:
+                continue
+
+            link_tag = li.find("a")
+            if not link_tag:
+                continue
+            title = link_tag.get("title", "") or link_tag.get_text(strip=True)
+            href = link_tag.get("href", "")
+            if not title or not href:
+                continue
+
+            if href.startswith("./") or href.startswith("../"):
+                content_url = urljoin(base_url + base_path, href)
+            elif href.startswith("/"):
+                content_url = base_url + href
+            elif href.startswith("http"):
+                content_url = href
+            else:
+                content_url = urljoin(base_url + base_path, href)
+
+            announce = {
+                "title": title,
+                "publish_date": publish_date,
+                "purchase_name": "大化瑶族自治县",
+                "content_url": content_url,
+                "source_code": "dahuagov",
+                "source_name": "大化县政府网采购公告",
+                "announcement_type": "purchase",
+                "crawl_mode": "auto",
+                "is_new": True,
+                "is_today": publish_date.date() == datetime.now().date(),
+            }
+            announce["content_hash"] = _generate_hash(announce)
+            results.append(announce)
+
+    return results
+
+
+def parse_dahuagov_detail_pubdate(html: str) -> datetime | None:
+    """从详情页 <meta name="PubDate"> 解析精确发布时间"""
+    soup = BeautifulSoup(html, "html.parser")
+    meta = soup.find("meta", attrs={"name": "PubDate"})
+    if not meta:
+        return None
+    content = meta.get("content", "").strip()
+    for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M", "%Y-%m-%d"):
+        try:
+            return datetime.strptime(content, fmt)
+        except ValueError:
+            continue
+    return None
+
+
+async def extract_page_content(url: str, timeout: int = 30) -> str | None:
+    """抓取详情页并用 BeautifulSoup 提取正文纯文本"""
+    import urllib.parse
+
+    import httpx
+
+    # ---- 站点专用处理 ----
+
+    # ① 政采云 SPA (HTTP API, 无需渲染)
+    if "zfcg.gxzf.gov.cn" in url and "articleId=" in url:
+        return await _extract_zcy_content(url, timeout)
+
+    # ② 大化县政府网 (静态HTML)
+    if "www.gxdh.gov.cn" in url or "gxdh.gov.cn" in url:
+        return await _extract_dahuagov_content(url, timeout)
+
+    # ---- 通用 HTML 提取 ----
+    try:
+        headers = {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
+            "Accept": "text/html,application/xhtml+xml",
+            "Accept-Language": "zh-CN,zh;q=0.9",
+        }
+        async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
+            resp = await client.get(url, headers=headers)
+            if resp.status_code != 200:
+                return None
+
+            soup = BeautifulSoup(resp.text, "html.parser")
+
+            # 尝试多种常见正文容器选择器
+            selectors = [
+                "div.article-content", "div.content", "div.TRS_Editor",
+                "div.Custom_UnionStyle", "div.pages_content", "div#content",
+                "div.main-content", "article", ".article", ".detail-content",
+                ".text-content", ".news-content", ".detail-article",
+                "div.article-con", ".trs_editor_view", ".TRS_UEDITOR",
+                ".trs_paper_default", ".article-content",
+            ]
+            for selector in selectors:
+                container = soup.select_one(selector)
+                if container:
+                    for tag in container.find_all(["script", "style"]):
+                        tag.decompose()
+                    text = container.get_text(separator="\n", strip=True)
+                    if len(text) > 50:
+                        return text
+
+            # 兜底：取 body 内所有文本
+            body = soup.find("body")
+            if body:
+                for tag in body.find_all(["script", "style", "nav", "footer", "header"]):
+                    tag.decompose()
+                text = body.get_text(separator="\n", strip=True)
+                lines = [l.strip() for l in text.split("\n") if l.strip()]
+                text = "\n".join(lines[:200])
+                if len(text) > 50:
+                    return text
+
+            return None
+    except Exception:
+        return None
+
+
+async def _extract_zcy_content(url: str, timeout: int = 30) -> str | None:
+    """从政采云 SPA 隐藏 API 提取公告正文"""
+    import urllib.parse
+
+    import httpx
+    from bs4 import BeautifulSoup
+
+    params = urllib.parse.parse_qs(urllib.parse.urlparse(url).query)
+    article_id = params.get("articleId", [None])[0]
+    parent_id = params.get("parentId", [None])[0]
+    if not article_id:
+        return None
+
+    api_url = "https://zfcg.gxzf.gov.cn/portal/detail"
+    if parent_id:
+        api_url += f"?articleId={article_id}&parentId={parent_id}"
+    else:
+        api_url += f"?articleId={article_id}"
+
+    headers = {
+        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
+        "Accept": "application/json, text/plain, */*",
+        "Referer": url,
+    }
+
+    try:
+        async with httpx.AsyncClient(timeout=timeout) as client:
+            resp = await client.get(api_url, headers=headers)
+            if resp.status_code != 200:
+                return None
+
+            data = resp.json()
+            if not data.get("success"):
+                return None
+
+            content_html = data.get("result", {}).get("data", {}).get("content", "")
+            if not content_html:
+                return None
+
+            soup = BeautifulSoup(content_html, "html.parser")
+            for tag in soup.find_all(["script", "style"]):
+                tag.decompose()
+            text = soup.get_text(separator="\n", strip=True)
+
+            # 清理过短行和多余空白
+            lines = [l.strip() for l in text.split("\n") if len(l.strip()) > 5]
+            text = "\n".join(lines[:300])
+            return text if len(text) > 50 else None
+    except Exception:
+        return None
+
+
+async def _extract_dahuagov_content(url: str, timeout: int = 30) -> str | None:
+    """从大化县政府网详情页提取正文"""
+    import httpx
+    from bs4 import BeautifulSoup
+
+    headers = {
+        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
+        "Accept": "text/html,application/xhtml+xml",
+        "Accept-Language": "zh-CN,zh;q=0.9",
+    }
+
+    try:
+        async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
+            resp = await client.get(url, headers=headers)
+            if resp.status_code != 200:
+                return None
+
+            soup = BeautifulSoup(resp.text, "html.parser")
+
+            # 大化县政府网正文容器
+            selectors = [
+                "div.article-con", ".trs_editor_view", ".TRS_UEDITOR",
+                ".trs_paper_default", "div.content", "div.TRS_Editor",
+                "div.article-content", "div.Custom_UnionStyle",
+                "div#content", "div.main-content", "article", ".detail-content",
+            ]
+            for selector in selectors:
+                container = soup.select_one(selector)
+                if container:
+                    for tag in container.find_all(["script", "style"]):
+                        tag.decompose()
+                    text = container.get_text(separator="\n", strip=True)
+                    if len(text) > 50:
+                        lines = [l.strip() for l in text.split("\n") if l.strip()]
+                        return "\n".join(lines[:300])
+
+            # 兜底
+            body = soup.find("body")
+            if body:
+                for tag in body.find_all(["script", "style", "nav", "footer", "header"]):
+                    tag.decompose()
+                text = body.get_text(separator="\n", strip=True)
+                lines = [l.strip() for l in text.split("\n") if l.strip()]
+                text = "\n".join(lines[:200])
+                if len(text) > 50:
+                    return text
+            return None
+    except Exception:
+        return None
+
+
+def _generate_hash(ann: dict[str, Any]) -> str:
+    content = (
+        f"{ann['title']}|{ann['publish_date'].strftime('%Y-%m-%d')}"
+        f"|{ann['purchase_name']}|{ann['content_url']}|{ann['source_code']}"
+    )
+    return hashlib.sha256(content.encode("utf-8")).hexdigest()

app/main.py

@@ -0,0 +1,56 @@
+import logging
+from contextlib import asynccontextmanager
+
+from fastapi import FastAPI
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+
+from app.api.router import api_router
+from app.config import settings
+
+engine = create_async_engine(settings.database_url, echo=settings.debug)
+async_session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
+
+
+async def get_db() -> AsyncSession:
+    async with async_session() as session:
+        yield session
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    logging.basicConfig(
+        level=getattr(logging, settings.log_level),
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+    )
+    # LogHive 远程日志（仅当配置了 API Key 时启用）
+    if settings.loghive_api_key:
+        from loghive_client import LogHiveHandler
+        handler = LogHiveHandler(
+            project=settings.loghive_project,
+            api_key=settings.loghive_api_key,
+            endpoint=settings.loghive_endpoint,
+            level=logging.INFO,
+        )
+        logging.getLogger().addHandler(handler)
+        logging.info("LogHive 日志系统已连接")
+    from app.scheduler.jobs import shutdown_scheduler, start_scheduler
+    start_scheduler()
+    yield
+    shutdown_scheduler()
+    await engine.dispose()
+
+
+app = FastAPI(
+    title="广西政府采购网公告监控系统",
+    version="2.0.0",
+    lifespan=lifespan,
+    docs_url="/docs" if settings.debug else None,
+    redoc_url=None,
+)
+
+app.include_router(api_router)
+
+
+@app.get("/health")
+async def health():
+    return {"status": "ok"}

app/models/announcement.py

@@ -0,0 +1,44 @@
+import hashlib
+from datetime import datetime
+
+from sqlalchemy import Boolean, DateTime, Integer, String, Text, func
+from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
+
+
+class Base(DeclarativeBase):
+    pass
+
+
+class Announcement(Base):
+    __tablename__ = "announcements"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
+    title: Mapped[str] = mapped_column(String(500), nullable=False)
+    publish_date: Mapped[datetime] = mapped_column(DateTime, nullable=False)
+    purchase_name: Mapped[str] = mapped_column(String(200), default="")
+    content_url: Mapped[str] = mapped_column(Text, default="")
+    source_code: Mapped[str] = mapped_column(String(50), nullable=False)
+    source_name: Mapped[str] = mapped_column(String(100), nullable=False)
+    announcement_type: Mapped[str] = mapped_column(String(50), default="purchase")
+    content_hash: Mapped[str] = mapped_column(String(64), unique=True, nullable=False)
+    crawl_mode: Mapped[str] = mapped_column(String(20), default="auto")
+    is_new: Mapped[bool] = mapped_column(Boolean, default=True)
+    is_sent: Mapped[bool] = mapped_column(Boolean, default=False)
+    keyword_matched: Mapped[bool] = mapped_column(Boolean, default=False)
+    ai_relevant: Mapped[bool | None] = mapped_column(Boolean, nullable=True, default=None)
+    ai_analysis: Mapped[str | None] = mapped_column(Text, nullable=True, default=None)
+    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=func.now())
+    updated_at: Mapped[datetime] = mapped_column(DateTime, server_default=func.now(), onupdate=func.now())  # noqa: E501
+
+    @staticmethod
+    def generate_hash(title: str, publish_date: str, purchase_name: str,
+                      content_url: str, source_code: str) -> str:
+        content = f"{title}|{publish_date}|{purchase_name}|{content_url}|{source_code}"
+        return hashlib.sha256(content.encode("utf-8")).hexdigest()
+
+    @staticmethod
+    def source_map() -> dict:
+        import json
+
+        from app.config import settings
+        return json.loads(settings.announcement_sources)

app/models/holiday.py

@@ -0,0 +1,15 @@
+from datetime import date
+
+from sqlalchemy import Boolean, Date, Integer, String
+from sqlalchemy.orm import Mapped, mapped_column
+
+from app.models.announcement import Base
+
+
+class ChineseHoliday(Base):
+    __tablename__ = "chinese_holidays"
+
+    date: Mapped[date] = mapped_column(Date, primary_key=True)
+    is_workday: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
+    year: Mapped[int] = mapped_column(Integer, nullable=False, index=True)
+    description: Mapped[str] = mapped_column(String(100), nullable=False, default="")

app/models/schemas.py

@@ -0,0 +1,57 @@
+from datetime import datetime
+
+from pydantic import BaseModel
+
+
+class AnnouncementResponse(BaseModel):
+    id: int
+    title: str
+    publish_date: datetime
+    purchase_name: str
+    content_url: str
+    source_code: str
+    source_name: str
+    announcement_type: str
+    crawl_mode: str
+    is_new: bool
+    is_sent: bool
+    keyword_matched: bool
+    created_at: datetime
+
+    model_config = {"from_attributes": True}
+
+
+class AnnouncementListResponse(BaseModel):
+    total: int
+    page: int
+    page_size: int
+    items: list[AnnouncementResponse]
+
+
+class CrawlTriggerRequest(BaseModel):
+    spider_name: str | None = None
+    keywords: list[str] | None = None
+    sources: list[str] | None = None
+    manual: bool = False
+
+
+class CrawlStatusResponse(BaseModel):
+    running: bool
+    last_crawl_time: datetime | None = None
+    total_sources: int
+
+
+class SourceInfo(BaseModel):
+    code: str
+    name: str
+    type: str
+
+
+class SourcesResponse(BaseModel):
+    sources: list[SourceInfo]
+
+
+class JobResponse(BaseModel):
+    id: str
+    name: str
+    next_run_time: str | None = None

app/scheduler/jobs.py

@@ -0,0 +1,81 @@
+import logging
+from datetime import datetime
+from zoneinfo import ZoneInfo
+
+from apscheduler.schedulers.asyncio import AsyncIOScheduler
+from apscheduler.triggers.cron import CronTrigger
+
+from app.api.deps import get_db, get_crawl_service
+from app.config import settings
+
+logger = logging.getLogger(__name__)
+scheduler = AsyncIOScheduler()
+
+NIGHT_START = 22  # 22:00
+NIGHT_END = 8     # 08:00
+TZ = ZoneInfo("Asia/Shanghai")
+
+
+def _is_night_time() -> bool:
+    """22:00 ~ 次日 08:00 夜间时段"""
+    current = datetime.now(TZ).hour
+    return current >= NIGHT_START or current < NIGHT_END
+
+
+async def _should_skip() -> bool:
+    """检查是否应该跳过爬取"""
+    if _is_night_time():
+        logger.info("夜间时段 (22:00-08:00)，跳过爬取")
+        return True
+
+    from app.services.holiday_service import is_workday
+    async for db in get_db():
+        if not await is_workday(db):
+            logger.info("非工作日，跳过爬取")
+            return True
+    return False
+
+
+async def scheduled_crawl():
+    if await _should_skip():
+        return
+
+    logger.info("开始定时爬取任务")
+    service = get_crawl_service()
+    for name in service.get_spider_names():
+        try:
+            results = await service.run_spider(name)
+            for r in results:
+                if not r.success:
+                    logger.error(f"Spider {name} 失败: {r.error_message}")
+                elif r.pipeline_result:
+                    logger.info(
+                        f"Spider {name}: 抓取{r.total_count}条, "
+                        f"新增{r.pipeline_result.stored}条, "
+                        f"通知{r.pipeline_result.notified}条"
+                    )
+                else:
+                    logger.info(f"Spider {name}: 抓取{r.total_count}条 (无新增)")
+        except Exception as e:
+            logger.error(f"Spider {name} 异常: {e}")
+    logger.info("定时爬取任务完成")
+
+
+def start_scheduler():
+    if not settings.scheduler_enabled:
+        return
+    trigger = CronTrigger.from_crontab(settings.scheduler_cron, timezone=TZ)
+    scheduler.add_job(
+        scheduled_crawl,
+        trigger=trigger,
+        id="scheduled_crawl",
+        name="定时爬取",
+    )
+    scheduler.start()
+    logger.info(f"APScheduler 已启动 (cron: {settings.scheduler_cron})")
+
+
+def shutdown_scheduler():
+    if scheduler.running:
+        scheduler.shutdown(wait=False)
+        logger.info("APScheduler 已停止")

app/services/ai_analyzer.py

@@ -0,0 +1,147 @@
+import json
+import logging
+from dataclasses import dataclass
+from typing import Any
+
+import httpx
+
+from app.config import settings
+from app.crawler.parsers import extract_page_content
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class AiResult:
+    """DeepSeek 分析结果"""
+    is_relevant: bool = False
+    reason: str = ""
+    business_type: str = ""
+    error: str | None = None
+    content_snippet: str | None = None  # 提取到的正文前 200 字，供入库参考
+
+
+class AiAnalyzer:
+    """AI 分析器 — 调用 DeepSeek 判断公告是否为中国电信可承接项目"""
+
+    def __init__(self):
+        self.api_key = settings.ai_api_key
+        self.base_url = settings.ai_base_url.rstrip("/")
+        self.model = settings.ai_model
+        self.timeout = settings.ai_timeout
+        self.prompt_template = settings.ai_prompt_template
+
+    async def analyze(self, announcement: dict[str, Any]) -> AiResult:
+        """分析单条公告"""
+        if not self.api_key:
+            return AiResult(error="AI_API_KEY 未配置")
+
+        # 1. 获取公告正文
+        content_url = announcement.get("content_url", "")
+        content = None
+        content_snippet = None
+        if content_url:
+            content = await extract_page_content(content_url, self.timeout)
+
+        if content:
+            content_snippet = content[:200]
+        else:
+            logger.warning("无法获取公告正文: %s", content_url)
+
+        # 2. 构建 prompt
+        prompt = self.prompt_template.format(
+            title=announcement.get("title", ""),
+            purchase_name=announcement.get("purchase_name", ""),
+            announcement_type=announcement.get("announcement_type", ""),
+            content=content or "（无法获取正文，请仅根据标题和采购人信息判断）",
+        )
+
+        # 3. 调用 DeepSeek API
+        try:
+            result = await self._call_deepseek(prompt)
+            if result.error:
+                return AiResult(error=result.error, content_snippet=content_snippet)
+            return AiResult(
+                is_relevant=result.is_relevant,
+                reason=result.reason,
+                business_type=result.business_type,
+                content_snippet=content_snippet,
+            )
+        except Exception as e:
+            logger.exception("AI 分析异常")
+            return AiResult(error=str(e), content_snippet=content_snippet)
+
+    async def analyze_batch(
+        self, announcements: list[dict[str, Any]], max_concurrent: int = 3
+    ) -> list[AiResult]:
+        """批量分析，控制并发数"""
+        import asyncio
+
+        sem = asyncio.Semaphore(max_concurrent)
+
+        async def _limited(ann: dict[str, Any]) -> AiResult:
+            async with sem:
+                return await self.analyze(ann)
+
+        tasks = [_limited(ann) for ann in announcements]
+        return await asyncio.gather(*tasks)
+
+    async def _call_deepseek(self, prompt: str) -> AiResult:
+        """调用 DeepSeek Chat API"""
+        url = f"{self.base_url}/chat/completions"
+        headers = {
+            "Authorization": f"Bearer {self.api_key}",
+            "Content-Type": "application/json",
+        }
+        payload = {
+            "model": self.model,
+            "messages": [
+                {
+                    "role": "system",
+                    "content": "你是一个专业的政府采购项目分析师。请根据公告信息判断是否为中国电信可以承接的项目，并用 JSON 格式回答。",
+                },
+                {"role": "user", "content": prompt},
+            ],
+            "temperature": 0.3,  # 低温度，提高判断一致性
+            "max_tokens": 512,
+        }
+
+        async with httpx.AsyncClient(timeout=self.timeout) as client:
+            response = await client.post(url, headers=headers, json=payload)
+            if response.status_code != 200:
+                return AiResult(
+                    error=f"API 请求失败 (HTTP {response.status_code}): {response.text[:200]}"
+                )
+
+            data = response.json()
+            choices = data.get("choices", [])
+            if not choices:
+                return AiResult(error="API 返回空 choices")
+
+            content = choices[0].get("message", {}).get("content", "")
+            return self._parse_response(content)
+
+    @staticmethod
+    def _parse_response(content: str) -> AiResult:
+        """从 LLM 回复中提取 JSON 结果"""
+        # 清理可能的 markdown 代码块标记
+        content = content.strip()
+        if content.startswith("```"):
+            # 移除 ```json 或 ``` 包裹
+            lines = content.split("\n")
+            if lines[0].strip().startswith("```"):
+                lines = lines[1:]
+            if lines and lines[-1].strip() == "```":
+                lines = lines[:-1]
+            content = "\n".join(lines).strip()
+
+        try:
+            result = json.loads(content)
+            return AiResult(
+                is_relevant=bool(result.get("is_relevant", False)),
+                reason=str(result.get("reason", "")),
+                business_type=str(result.get("business_type", "")),
+            )
+        except (json.JSONDecodeError, ValueError) as e:
+            logger.warning("JSON 解析失败: %s\n原始内容: %s", e, content[:200])
+            return AiResult(error=f"JSON 解析失败: {e}")

app/services/ai_state.py

@@ -0,0 +1,51 @@
+"""AI 运行时状态 — 支持企微菜单动态开关"""
+
+import logging
+
+from app.config import settings
+
+logger = logging.getLogger(__name__)
+
+# 运行时覆盖值，None 表示使用 settings.ai_enabled
+_runtime_override: bool | None = None
+
+
+def is_ai_enabled() -> bool:
+    """获取 AI 分析当前是否启用（考虑运行时覆盖）"""
+    if _runtime_override is not None:
+        return _runtime_override
+    return settings.ai_enabled
+
+
+def set_ai_enabled(enabled: bool) -> bool:
+    """设置 AI 分析运行时开关，返回是否真的发生了变化"""
+    global _runtime_override
+    current = is_ai_enabled()
+    if enabled == current:
+        return False
+    _runtime_override = enabled
+    status = "启用" if enabled else "禁用"
+    logger.info("AI 分析已通过企微菜单%s", status)
+    return True
+
+
+def get_ai_status_text() -> str:
+    """获取 AI 状态文本"""
+    return "已启用" if is_ai_enabled() else "已禁用"
+
+
+def get_whitelist() -> list[str]:
+    """获取 AI 白名单用户列表"""
+    raw = settings.ai_whitelist
+    if not raw:
+        return []
+    return [u.strip() for u in raw.split(",") if u.strip()]
+
+
+def is_whitelisted(user_id: str) -> bool:
+    """检查用户是否在 AI 白名单中（不区分大小写）"""
+    whitelist = get_whitelist()
+    if not whitelist:
+        # 白名单为空则所有人都可以操作
+        return True
+    return user_id.lower() in [u.lower() for u in whitelist]

app/services/crawl_service.py

@@ -0,0 +1,74 @@
+import logging
+
+from sqlalchemy.ext.asyncio import async_sessionmaker
+
+from app.crawler.base import BaseSpider, CrawlResult
+from app.services.notification_service import NotificationService
+from app.services.pipeline import PostCrawlPipeline
+
+logger = logging.getLogger(__name__)
+
+
+class CrawlService:
+    def __init__(self, db_session_factory: async_sessionmaker,
+                 notification_service: NotificationService):
+        self.db_session_factory = db_session_factory
+        self.notification_service = notification_service
+        self.spiders: dict[str, BaseSpider] = {}
+
+    def register(self, spider: BaseSpider):
+        self.spiders[spider.name] = spider
+
+    async def run_all(self) -> list[CrawlResult]:
+        results = []
+        for name in self.spiders:
+            spider_results = await self.run_spider(name)
+            results.extend(spider_results)
+        return results
+
+    async def run_spider(self, name: str) -> list[CrawlResult]:
+        spider = self.spiders.get(name)
+        if spider is None:
+            return [CrawlResult(
+                source_code=name, source_name=name,
+                error_message=f"Spider not found: {name}",
+            )]
+
+        # 1. Crawl
+        result = await spider.crawl()
+
+        # 2. Pipeline: store → filter → notify
+        if result.success and result.announcements:
+            try:
+                config = spider.get_pipeline_config()
+                async with self.db_session_factory() as db:
+                    pipeline = PostCrawlPipeline(
+                        db_session=db,
+                        notification_service=self.notification_service,
+                    )
+                    pipe_result = await pipeline.process(
+                        result.announcements, config,
+                    )
+                    result.pipeline_result = pipe_result
+                    result.new_count = pipe_result.stored
+                    logger.info(
+                        f"Spider {name}: total={result.total_count}, "
+                        f"stored={pipe_result.stored}, "
+                        f"filtered={pipe_result.filtered}, "
+                        f"notified={pipe_result.notified}"
+                    )
+            except Exception as e:
+                logger.error(f"Spider {name} pipeline 异常: {e}")
+                if not result.error_message:
+                    result.error_message = str(e)
+
+        return [result]
+
+    def get_spider_names(self) -> list[str]:
+        return list(self.spiders.keys())
+
+    def get_pipeline_config(self, name: str):
+        spider = self.spiders.get(name)
+        if spider:
+            return spider.get_pipeline_config()
+        return None

app/services/filter_service.py

@@ -0,0 +1,53 @@
+from datetime import date
+from typing import Any
+
+
+def filter_by_keywords(announcements: list[dict[str, Any]],
+                       keywords: list[str]) -> list[dict[str, Any]]:
+    if not keywords:
+        return announcements
+
+    filtered = []
+    for ann in announcements:
+        search_text = f"{ann.get('title', '')} {ann.get('purchase_name', '')}"
+        if any(kw in search_text for kw in keywords):
+            ann["keyword_matched"] = True
+            filtered.append(ann)
+
+    return filtered
+
+
+def filter_by_date(announcements: list[dict[str, Any]],
+                   start_date: date | None = None,
+                   end_date: date | None = None) -> list[dict[str, Any]]:
+    if not start_date and not end_date:
+        return announcements
+
+    filtered = []
+    for ann in announcements:
+        pub_date = ann.get("publish_date")
+        if not pub_date:
+            continue
+        if not isinstance(pub_date, date):
+            pub_date = pub_date.date() if hasattr(pub_date, "date") else None
+        if pub_date is None:
+            continue
+
+        if start_date and pub_date < start_date:
+            continue
+        if end_date and pub_date > end_date:
+            continue
+        filtered.append(ann)
+
+    return filtered
+
+
+def dedup_by_hash(announcements: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    seen = set()
+    result = []
+    for ann in announcements:
+        h = ann.get("content_hash")
+        if h and h not in seen:
+            seen.add(h)
+            result.append(ann)
+    return result

app/services/holiday_service.py

@@ -0,0 +1,101 @@
+import logging
+from datetime import date
+
+import httpx
+from sqlalchemy.dialects.postgresql import insert as pg_insert
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.holiday import ChineseHoliday
+
+logger = logging.getLogger(__name__)
+
+HOLIDAY_API = "http://timor.tech/api/holiday/year"
+
+
+def now_in_china() -> date:
+    from datetime import datetime
+    from zoneinfo import ZoneInfo
+    return datetime.now(ZoneInfo("Asia/Shanghai")).date()
+
+
+async def sync_holidays(db: AsyncSession, year: int) -> int:
+    """同步中国节假日数据，返回更新的记录数"""
+    url = f"{HOLIDAY_API}/{year}"
+    async with httpx.AsyncClient(timeout=15) as client:
+        response = await client.get(url)
+        data = response.json()
+
+    if data.get("code") != 0:
+        logger.error(f"节假日 API 返回错误: {data}")
+        return 0
+
+    holidays = data.get("holiday", {})
+    if not holidays:
+        return 0
+
+    # 也标记周末（周六日但非调休工作日）
+    from datetime import timedelta
+    current = date(year, 1, 1)
+    end = date(year, 12, 31)
+
+    records: dict[date, dict] = {}
+    while current <= end:
+        dow = current.weekday()  # 0=Mon, 6=Sun
+        # 默认：周一~五为工作日，周六日为非工作日
+        default_workday = dow < 5
+        records[current] = {
+            "date": current,
+            "is_workday": default_workday,
+            "year": year,
+            "description": "",
+        }
+        current += timedelta(days=1)
+
+    # 覆盖节假日数据
+    # holiday=true  → 休息日（无论 wage 值）
+    # holiday=false → 调休补班日（周末也要上班）
+    for date_str, info_str in holidays.items():
+        d = date.fromisoformat(f"{year}-{date_str}")
+        if d.year != year:
+            continue
+        info = info_str if isinstance(info_str, dict) else {}
+        holiday = info.get("holiday", False)
+        name = info.get("name", "")
+
+        records[d]["description"] = name
+        records[d]["is_workday"] = not holiday  # holiday=false → 调休工作日
+
+    # Upsert
+    values = list(records.values())
+    stmt = pg_insert(ChineseHoliday).values(values)
+    stmt = stmt.on_conflict_do_update(
+        index_elements=["date"],
+        set_={"is_workday": stmt.excluded.is_workday,
+              "description": stmt.excluded.description},
+    )
+    result = await db.execute(stmt)
+    await db.commit()
+    logger.info(f"已同步 {year} 年节假日，{len(values)} 天")
+    return result.rowcount
+
+
+async def is_workday(db: AsyncSession, day: date | None = None) -> bool:
+    """判断某天是否为工作日"""
+    if day is None:
+        day = now_in_china()
+    from sqlalchemy import select
+    result = await db.execute(
+        select(ChineseHoliday.is_workday).where(ChineseHoliday.date == day)
+    )
+    row = result.fetchone()
+    if row is None:
+        # 无数据时，按周判断
+        return day.weekday() < 5
+    return row[0]
+
+
+async def was_yesterday_workday(db: AsyncSession) -> bool:
+    """昨天是工作日吗"""
+    from datetime import timedelta
+    yesterday = now_in_china() - timedelta(days=1)
+    return await is_workday(db, yesterday)

app/services/notification_service.py

@@ -0,0 +1,78 @@
+from typing import Any
+
+from app.config import settings
+from app.wechat.client import WeChatClient
+
+
+class NotificationService:
+    def __init__(self):
+        self.client = WeChatClient()
+
+    async def send(self, announcements: list[dict[str, Any]]) -> int:
+        if not settings.wechat_enabled:
+            return 0
+        if not announcements:
+            return 0
+
+        sent = 0
+        for ann in announcements:
+            try:
+                # 发送普通 textcard
+                title = ann.get("title", "")
+                if len(title) > 128:
+                    title = title[:125] + "..."
+
+                purchase_name = ann.get("purchase_name", "")
+                if len(purchase_name) > 25:
+                    purchase_name = purchase_name[:22] + "..."
+
+                pub_date = ann.get("publish_date")
+                time_str = pub_date.strftime("%Y-%m-%d %H:%M") if pub_date else "时间未知"
+
+                source_name = ann.get("source_name", "")
+
+                description = f"{source_name} | {purchase_name} | {time_str}"
+
+                url = ann.get("content_url", "")
+
+                if await self.client.send_textcard(title, description, url):
+                    sent += 1
+
+                # AI 标记为可承接的，额外发送着重通知
+                ai_result = ann.get("ai_result")
+                if ai_result and ai_result.get("is_relevant"):
+                    await self._send_ai_emphasis(ann, ai_result)
+
+            except Exception:
+                continue
+
+        return sent
+
+    async def _send_ai_emphasis(
+        self, ann: dict[str, Any], ai_result: dict[str, Any]
+    ) -> bool:
+        """发送 AI 分析的着重通知（markdown 格式）"""
+        title = ann.get("title", "")
+        purchase_name = ann.get("purchase_name", "")
+        pub_date = ann.get("publish_date")
+        time_str = pub_date.strftime("%Y-%m-%d %H:%M") if pub_date else "时间未知"
+        url = ann.get("content_url", "")
+
+        reason = ai_result.get("reason", "")
+        business_type = ai_result.get("business_type", "")
+
+        # 企业微信 markdown 格式
+        md = (
+            f"{settings.ai_analysis_title}\n"
+            f"---\n"
+            f"**标题：** [{title}]({url})\n"
+            f"> 采购人：{purchase_name}\n"
+            f"> 发布时间：{time_str}\n\n"
+            f"**🤖 AI 分析：**\n"
+            f"> {reason}\n\n"
+            f"**🏷 业务分类：** {business_type}\n"
+            f"---\n"
+            f"[📄 查看公告原文]({url})"
+        )
+
+        return await self.client.send_markdown(md)

app/services/pipeline.py

@@ -0,0 +1,189 @@
+import logging
+from typing import Any
+
+from app.config import settings
+from app.crawler.base import PipelineConfig, PipelineResult
+from app.services.ai_state import is_ai_enabled
+from app.services.filter_service import dedup_by_hash
+
+logger = logging.getLogger(__name__)
+
+
+class PostCrawlPipeline:
+    def __init__(self, db_session, notification_service):
+        self.db = db_session
+        self.notify = notification_service
+
+    async def process(self, announcements: list[dict[str, Any]],
+                      config: PipelineConfig) -> PipelineResult:
+        result = PipelineResult()
+
+        if not announcements:
+            return result
+
+        # 1. Dedup
+        if config.dedup_enabled:
+            announcements = dedup_by_hash(announcements)
+
+        # 2. 先标记 keyword_matched，再存库
+        if config.filter_enabled and config.keywords:
+            for a in announcements:
+                a["keyword_matched"] = self._match_keywords(a, config.keywords)
+
+        # 3. Store to database
+        stored = await self._save_to_db(announcements)
+        result.stored = stored
+
+        to_notify = announcements
+
+        # 4. Filter
+        if config.filter_enabled and config.keywords:
+            before = len(to_notify)
+            to_notify = [a for a in to_notify if a.get("keyword_matched")]
+            result.filtered = before - len(to_notify)
+
+        # 5. Skip already-notified
+        if to_notify:
+            to_notify = await self._exclude_sent(to_notify)
+
+        # 5.5 AI 分析（可选，支持运行时开关）
+        if is_ai_enabled() and to_notify:
+            await self._ai_analyze(to_notify)
+
+        # 6. Notify
+        if config.notify_mode == "all":
+            result.notified = await self._send_notifications(to_notify)
+        elif config.notify_mode == "filtered":
+            if config.filter_enabled and config.keywords:
+                result.notified = await self._send_notifications(to_notify)
+            elif not config.filter_enabled:
+                result.notified = await self._send_notifications(to_notify)
+
+        # 7. Mark sent
+        if config.mark_sent and result.notified > 0:
+            await self._mark_sent(to_notify)
+
+        return result
+
+    async def _save_to_db(self, announcements: list[dict[str, Any]]) -> int:
+        from sqlalchemy.dialects.postgresql import insert
+
+        from app.models.announcement import Announcement
+
+        if not announcements:
+            return 0
+
+        values = [{
+            "title": a["title"],
+            "publish_date": a["publish_date"],
+            "purchase_name": a.get("purchase_name", ""),
+            "content_url": a.get("content_url", ""),
+            "source_code": a["source_code"],
+            "source_name": a["source_name"],
+            "announcement_type": a.get("announcement_type", "purchase"),
+            "content_hash": a["content_hash"],
+            "crawl_mode": a.get("crawl_mode", "auto"),
+            "is_new": a.get("is_new", True),
+            "is_sent": False,
+            "keyword_matched": a.get("keyword_matched", False),
+        } for a in announcements]
+
+        stmt = insert(Announcement).values(values)
+        stmt = stmt.on_conflict_do_nothing(index_elements=["content_hash"])
+
+        result_proxy = await self.db.execute(stmt)
+        await self.db.commit()
+        return result_proxy.rowcount if result_proxy.rowcount >= 0 else len(values)
+
+    async def _send_notifications(self, announcements: list[dict[str, Any]]) -> int:
+        return await self.notify.send(announcements)
+
+    async def _mark_sent(self, announcements: list[dict[str, Any]]) -> int:
+        from sqlalchemy import update
+
+        from app.models.announcement import Announcement
+
+        hashes = [a["content_hash"] for a in announcements if a.get("content_hash")]
+        if not hashes:
+            return 0
+
+        stmt = (
+            update(Announcement)
+            .where(Announcement.content_hash.in_(hashes))
+            .values(is_sent=True)
+        )
+        result = await self.db.execute(stmt)
+        await self.db.commit()
+        return result.rowcount
+
+    async def _exclude_sent(self, announcements: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        from sqlalchemy import select
+
+        from app.models.announcement import Announcement
+
+        hashes = [a["content_hash"] for a in announcements if a.get("content_hash")]
+        if not hashes:
+            return announcements
+
+        stmt = select(Announcement.content_hash).where(
+            Announcement.content_hash.in_(hashes),
+            Announcement.is_sent == True,  # noqa: E712
+        )
+        result = await self.db.execute(stmt)
+        sent_hashes = {row[0] for row in result.fetchall()}
+
+        return [a for a in announcements if a.get("content_hash") not in sent_hashes]
+
+    @staticmethod
+    def _match_keywords(announcement: dict[str, Any], keywords: list[str]) -> bool:
+        text = f"{announcement.get('title', '')} {announcement.get('purchase_name', '')}"
+        return any(kw in text for kw in keywords)
+
+    async def _ai_analyze(self, announcements: list[dict[str, Any]]) -> None:
+        """对公告列表执行 AI 分析，将结果附加到每条公告的 ai_result 字段，并更新数据库"""
+        from app.services.ai_analyzer import AiAnalyzer
+
+        analyzer = AiAnalyzer()
+        logger.info("AI 分析开始：共 %d 条公告", len(announcements))
+
+        results = await analyzer.analyze_batch(announcements)
+
+        ai_updates = []
+        for ann, ai_result in zip(announcements, results):
+            ann["ai_result"] = {
+                "is_relevant": ai_result.is_relevant,
+                "reason": ai_result.reason,
+                "business_type": ai_result.business_type,
+            }
+            if ai_result.error:
+                logger.warning("AI 分析失败 [%s]: %s", ann.get("title", "")[:30], ai_result.error)
+            else:
+                ai_updates.append({
+                    "content_hash": ann["content_hash"],
+                    "ai_relevant": ai_result.is_relevant,
+                    "ai_analysis": ai_result.reason,
+                })
+                if ai_result.is_relevant:
+                    logger.info("AI 标记可承接项目: %s (%s)", ann.get("title", "")[:40], ai_result.business_type)
+
+        # 批量更新数据库中的 AI 分析结果
+        if ai_updates:
+            await self._update_ai_results(ai_updates)
+
+    async def _update_ai_results(self, updates: list[dict[str, Any]]) -> None:
+        """批量更新公告的 AI 分析结果到数据库"""
+        from sqlalchemy import update
+
+        from app.models.announcement import Announcement
+
+        for u in updates:
+            stmt = (
+                update(Announcement)
+                .where(Announcement.content_hash == u["content_hash"])
+                .values(
+                    ai_relevant=u["ai_relevant"],
+                    ai_analysis=u["ai_analysis"],
+                )
+            )
+            await self.db.execute(stmt)
+        await self.db.commit()

app/wechat/client.py

@@ -0,0 +1,88 @@
+import logging
+import time
+
+import httpx
+
+from app.config import settings
+
+logger = logging.getLogger(__name__)
+
+
+class WeChatClient:
+    def __init__(self):
+        self._access_token: str | None = None
+        self._token_expires_at: float = 0
+
+    async def _get_access_token(self) -> str | None:
+        now = time.time()
+        if self._access_token and now < self._token_expires_at:
+            return self._access_token
+
+        url = f"{settings.wechat_api_base_url}/cgi-bin/gettoken"
+        params = {
+            "corpid": settings.wechat_corp_id,
+            "corpsecret": settings.wechat_secret,
+        }
+        async with httpx.AsyncClient(timeout=30) as client:
+            response = await client.get(url, params=params)
+            data = response.json()
+            if data.get("errcode") == 0:
+                self._access_token = data["access_token"]
+                self._token_expires_at = now + data.get("expires_in", 7200) - 300
+                return self._access_token
+            logger.error(f"获取 access_token 失败: {data}")
+        return None
+
+    async def send_text(self, content: str, to_user: str = "@all") -> bool:
+        return await self._send_message("text", {"content": content}, to_user)
+
+    async def send_markdown(self, content: str, to_user: str = "@all") -> bool:
+        return await self._send_message("markdown", {"content": content}, to_user)
+
+    async def send_textcard(
+        self,
+        title: str,
+        description: str,
+        url: str,
+        to_user: str = "@all",
+        btn_txt: str = "查看详情",
+    ) -> bool:
+        return await self._send_message(
+            "textcard",
+            {
+                "title": title,
+                "description": description,
+                "url": url,
+                "btntxt": btn_txt,
+            },
+            to_user,
+        )
+
+    async def _send_message(
+        self, msgtype: str, msg_data: dict, to_user: str = "@all"
+    ) -> bool:
+        token = await self._get_access_token()
+        if not token:
+            logger.error("无法获取 access_token，跳过消息发送")
+            return False
+
+        url = f"{settings.wechat_api_base_url}/cgi-bin/message/send"
+        params = {"access_token": token}
+        body = {
+            "touser": to_user,
+            "msgtype": msgtype,
+            "agentid": int(settings.wechat_agent_id),
+            msgtype: msg_data,
+        }
+
+        async with httpx.AsyncClient(timeout=30) as client:
+            response = await client.post(url, params=params, json=body)
+            data = response.json()
+            errcode = data.get("errcode")
+            if errcode == 0:
+                return True
+            logger.error(
+                f"企业微信消息发送失败: errcode={errcode} errmsg={data.get('errmsg')} "
+                f"msgtype={msgtype} touser={to_user}"
+            )
+            return False

app/wechat/crypto.py

@@ -1,20 +1,20 @@
 #!/usr/bin/env python
-# -*- encoding:utf-8 -*-
 
 """ 对企业微信发送给企业后台的消息加解密示例代码.
 @copyright: Copyright (c) 1998-2014 Tencent Inc.
 
 """
 # ------------------------------------------------------------------------
-import logging
 import base64
-import random
 import hashlib
-import time
+import logging
-import struct
+import random
-from Crypto.Cipher import AES
-import xml.etree.cElementTree as ET
 import socket
+import struct
+import time
+import xml.etree.ElementTree as ET
+
+from Crypto.Cipher import AES
 
 try:
     import ierror
@@ -29,11 +29,11 @@ except ImportError:
 """
 
 
-class FormatException(Exception):
+class FormatError(Exception):
     pass
 
 
-def throw_exception(message, exception_class=FormatException):
+def throw_exception(message, exception_class=FormatError):
     """my define raise exception function"""
     raise exception_class(message)
 
@@ -104,7 +104,7 @@ class XMLParse:
         return resp_xml
 
 
-class PKCS7Encoder():
+class PKCS7Encoder:
     """提供基于PKCS7算法的加解密接口"""
 
     block_size = 32
@@ -134,7 +134,7 @@ class PKCS7Encoder():
         return decrypted[:-pad]
 
 
-class Prpcrypt(object):
+class Prpcrypt:
     """提供接收和推送给企业微信消息的加解密接口"""
 
     def __init__(self, key):
@@ -151,7 +151,7 @@ class Prpcrypt(object):
         """
         # 16位随机字符串添加到明文开头
         text = text.encode()
-        text = self.get_random_str() + struct.pack("I", socket.htonl(len(text))) + text + receiveid.encode()
+        text = self.get_random_str() + struct.pack("I", socket.htonl(len(text))) + text + receiveid.encode()  # noqa: E501
 
         # 使用自定义的填充方式对明文进行补位填充
         pkcs7 = PKCS7Encoder()
@@ -206,14 +206,14 @@ class Prpcrypt(object):
         return str(random.randint(1000000000000000, 9999999999999999)).encode()
 
 
-class WXBizMsgCrypt(object):
+class WXBizMsgCrypt:
     # 构造函数
     def __init__(self, sToken, sEncodingAESKey, sReceiveId):
         try:
             self.key = base64.b64decode(sEncodingAESKey + "=")
             assert len(self.key) == 32
-        except:
+        except Exception:
-            throw_exception("[error]: EncodingAESKey unvalid !", FormatException)
+            throw_exception("[error]: EncodingAESKey unvalid !", FormatError)
             # return ierror.WXBizMsgCrypt_IllegalAesKey,None
         self.m_sToken = sToken
         self.m_sReceiveId = sReceiveId
@@ -240,9 +240,9 @@ class WXBizMsgCrypt(object):
     def EncryptMsg(self, sReplyMsg, sNonce, timestamp=None):
         # 将企业回复用户的消息加密打包
         # @param sReplyMsg: 企业号待回复用户的消息，xml格式的字符串
-        # @param sTimeStamp: 时间戳，可以自己生成，也可以用URL参数的timestamp,如为None则自动用当前时间
+        # @param sTimeStamp: 时间戳，可以自己生成，也可以用URL参数的timestamp,如为None则自动用当前时间  # noqa: E501
-        # @param sNonce: 随机串，可以自己生成，也可以用URL参数的nonce
+        # @param sNonce: 随机串，可以自己生成，也可以用URL参数的nonce  # noqa: E501
-        # sEncryptMsg: 加密后的可以直接回复用户的密文，包括msg_signature, timestamp, nonce, encrypt的xml格式的字符串,
+        # sEncryptMsg: 加密后的可以直接回复用户的密文，包括msg_signature, timestamp, nonce, encrypt的xml格式的字符串,  # noqa: E501
         # return：成功0，sEncryptMsg,失败返回对应的错误码None
         pc = Prpcrypt(self.key)
         ret, encrypt = pc.encrypt(sReplyMsg, self.m_sReceiveId)

app/wechat/handler.py

@@ -0,0 +1,387 @@
+import asyncio
+import logging
+import time
+import xml.etree.ElementTree as ET
+
+from app.config import settings
+from app.wechat.crypto import WXBizMsgCrypt
+from app.wechat.client import WeChatClient
+
+logger = logging.getLogger(__name__)
+
+# 防重入：记录最近一次触发爬取的时间戳，60秒内不重复执行
+_last_crawl_time: float = 0.0
+_crawl_lock = asyncio.Lock()
+
+# 防重入：最新公告爬取，60秒内不重复执行
+_last_latest_time: float = 0.0
+_latest_lock = asyncio.Lock()
+
+
+class WeChatMessageHandler:
+    def __init__(self):
+        self.wxcpt = WXBizMsgCrypt(
+            sToken=settings.wechat_token,
+            sEncodingAESKey=settings.wechat_encoding_aes_key,
+            sReceiveId=settings.wechat_corp_id,
+        )
+        self.client = WeChatClient()
+
+    def verify_url(
+        self, msg_signature: str, timestamp: str, nonce: str, echostr: str
+    ) -> str | None:
+        ret, sEchoStr = self.wxcpt.VerifyURL(
+            msg_signature, timestamp, nonce, echostr
+        )
+        if ret == 0:
+            return (
+                sEchoStr.decode("utf-8")
+                if isinstance(sEchoStr, bytes)
+                else sEchoStr
+            )
+        return None
+
+    def decrypt_message(
+        self,
+        post_data: str,
+        msg_signature: str,
+        timestamp: str,
+        nonce: str,
+    ) -> ET.Element | None:
+        ret, xml_content = self.wxcpt.DecryptMsg(
+            post_data, msg_signature, timestamp, nonce
+        )
+        if ret != 0:
+            return None
+        return ET.fromstring(xml_content)
+
+    def encrypt_response(
+        self, response_xml: str, nonce: str, timestamp: str
+    ) -> str | None:
+        ret, encrypted = self.wxcpt.EncryptMsg(response_xml, nonce, timestamp)
+        if ret == 0:
+            return encrypted
+        return None
+
+    async def handle_event(
+        self, event: str, event_key: str | None, from_user: str
+    ) -> str | None:
+        if event != "click" or not event_key:
+            return None
+
+        if event_key == "today_stats":
+            return await self._handle_today_stats(from_user)
+        elif event_key == "trigger_crawl":
+            return await self._handle_trigger_crawl(from_user)
+        elif event_key == "sync_holidays":
+            return await self._handle_sync_holidays(from_user)
+        elif event_key == "monitor_config":
+            return await self._handle_monitor_config(from_user)
+        elif event_key == "system_status":
+            return await self._handle_system_status(from_user)
+        elif event_key == "workday_status":
+            return await self._handle_workday_status(from_user)
+        elif event_key == "latest_announcements":
+            return await self._handle_latest_announcements(from_user)
+        elif event_key == "pause_scheduler":
+            return await self._handle_pause_scheduler(from_user)
+        elif event_key == "resume_scheduler":
+            return await self._handle_resume_scheduler(from_user)
+        elif event_key == "toggle_ai":
+            return await self._handle_toggle_ai(from_user)
+        return None
+
+    async def handle_text(self, content: str, from_user: str) -> str | None:
+        """文本消息：返回帮助提示"""
+        help_text = (
+            "请使用菜单操作：\n"
+            "---\n"
+            "📋 最新公告 - 获取最新公告\n"
+            "📊 查询 → 监控配置/系统状态/今日工作日\n"
+            "⚙️ 系统管理 → 立即爬取/同步节假日/AI 分析"
+        )
+        ok = await self.client.send_text(help_text, from_user)
+        if not ok:
+            logger.warning(f"发送帮助消息失败, touser={from_user}")
+        return None
+
+    async def _handle_today_stats(self, from_user: str) -> str | None:
+        from app.api.deps import get_db
+
+        try:
+            async for db in get_db():
+                from sqlalchemy import func, select
+                from app.models.announcement import Announcement
+
+                total_result = await db.execute(
+                    select(func.count()).select_from(Announcement)
+                )
+                total = total_result.scalar() or 0
+
+                today_result = await db.execute(
+                    select(func.count()).where(
+                        func.date(Announcement.publish_date) == func.current_date()
+                    ).select_from(Announcement)
+                )
+                today = today_result.scalar() or 0
+
+                text = f"今日新增: {today} 条\n累计公告: {total} 条"
+                await self.client.send_text(text, from_user)
+        except Exception as e:
+            logger.error(f"查询统计失败: {e}")
+            await self.client.send_text("查询失败，请稍后再试", from_user)
+
+    async def _handle_trigger_crawl(self, from_user: str) -> str | None:
+        global _last_crawl_time
+        from app.api.deps import get_crawl_service
+
+        # 防重入：企业微信会对同一事件重试多次，60秒内只执行一次
+        async with _crawl_lock:
+            now = time.monotonic()
+            if now - _last_crawl_time < 60:
+                remaining = int(60 - (now - _last_crawl_time))
+                logger.info(f"爬取请求被忽略（防重入），距上次 {now - _last_crawl_time:.1f}s")
+                await self.client.send_text(
+                    f"爬取任务进行中，请 {remaining} 秒后再试", from_user
+                )
+                return None
+            _last_crawl_time = now
+
+        await self.client.send_text("开始爬取，请稍候...", from_user)
+
+        try:
+            service = get_crawl_service()
+            results = await service.run_all()
+            total = sum(r.total_count for r in results)
+            stored = sum(
+                r.pipeline_result.stored for r in results
+                if r.pipeline_result
+            )
+            notified = sum(
+                r.pipeline_result.notified for r in results
+                if r.pipeline_result
+            )
+            errors = [r.error_message for r in results if not r.success]
+            msg = f"爬取完成\n抓取: {total} 条\n新增: {stored} 条\n推送: {notified} 条"
+            if errors:
+                msg += f"\n异常: {errors[0][:50]}"
+            await self.client.send_text(msg, from_user)
+        except Exception as e:
+            logger.error(f"手动爬取失败: {e}")
+            await self.client.send_text(f"爬取失败: {e}", from_user)
+
+    async def _handle_sync_holidays(self, from_user: str) -> str | None:
+        from app.services.holiday_service import now_in_china, sync_holidays
+        from app.api.deps import get_db
+
+        try:
+            async for db in get_db():
+                year = now_in_china().year
+                count = await sync_holidays(db, year)
+                await self.client.send_text(
+                    f"已同步 {year} 年节假日\n共 {count} 条记录", from_user,
+                )
+        except Exception as e:
+            logger.error(f"同步节假日失败: {e}")
+            await self.client.send_text(f"同步失败: {e}", from_user)
+
+    async def _handle_monitor_config(self, from_user: str) -> str | None:
+        import json
+        from app.services.ai_state import get_ai_status_text
+        try:
+            keywords = settings.crawler_keywords
+            sources = json.loads(settings.announcement_sources)
+            source_names = "、".join(v["name"] for v in sources.values())
+            text = (
+                f"📋 监控配置\n"
+                f"---\n"
+                f"监控关键词: {', '.join(keywords)}\n"
+                f"爬取页数: {settings.crawler_max_pages} 页\n"
+                f"定时规则: {settings.scheduler_cron}\n"
+                f"公告来源: {source_names}\n"
+                f"---\n"
+                f"🤖 AI 分析: {get_ai_status_text()}\n"
+                f"AI 模型: {settings.ai_model}\n"
+                f"重点标记: {settings.ai_analysis_title}"
+            )
+            await self.client.send_text(text, from_user)
+        except Exception as e:
+            logger.error(f"查询监控配置失败: {e}")
+            await self.client.send_text("查询失败，请稍后再试", from_user)
+
+    async def _handle_system_status(self, from_user: str) -> str | None:
+        from app.api.deps import get_db
+        from sqlalchemy import func, select
+        from app.models.announcement import Announcement
+        from app.services.ai_state import get_ai_status_text
+
+        try:
+            async for db in get_db():
+                total_result = await db.execute(
+                    select(func.count()).select_from(Announcement)
+                )
+                total = total_result.scalar() or 0
+
+                today_result = await db.execute(
+                    select(func.count()).where(
+                        func.date(Announcement.publish_date) == func.current_date()
+                    ).select_from(Announcement)
+                )
+                today = today_result.scalar() or 0
+
+                unsent_result = await db.execute(
+                    select(func.count()).where(
+                        Announcement.is_sent == False,  # noqa: E712
+                        Announcement.keyword_matched == True,  # noqa: E712
+                    ).select_from(Announcement)
+                )
+                unsent = unsent_result.scalar() or 0
+
+                # AI 标记统计
+                ai_relevant_result = await db.execute(
+                    select(func.count()).where(
+                        Announcement.ai_relevant == True,  # noqa: E712
+                    ).select_from(Announcement)
+                )
+                ai_relevant = ai_relevant_result.scalar() or 0
+
+                scheduler_status = "已启用" if settings.scheduler_enabled else "已禁用"
+                text = (
+                    f"📊 系统状态\n"
+                    f"---\n"
+                    f"累计公告: {total} 条\n"
+                    f"今日新增: {today} 条\n"
+                    f"待推送: {unsent} 条\n"
+                    f"---\n"
+                    f"定时任务: {scheduler_status}\n"
+                    f"定时规则: {settings.scheduler_cron}\n"
+                    f"---\n"
+                    f"🤖 AI 分析: {get_ai_status_text()}\n"
+                    f"AI 标记项目: {ai_relevant} 条"
+                )
+                await self.client.send_text(text, from_user)
+        except Exception as e:
+            logger.error(f"查询系统状态失败: {e}")
+            await self.client.send_text("查询失败，请稍后再试", from_user)
+
+    async def _handle_workday_status(self, from_user: str) -> str | None:
+        from app.services.holiday_service import now_in_china, is_workday
+        from app.api.deps import get_db
+
+        try:
+            async for db in get_db():
+                today = now_in_china()
+                workday = await is_workday(db, today)
+                status = "工作日，正常爬取" if workday else "非工作日，跳过爬取"
+                text = f"今天 {today.strftime('%Y-%m-%d %A')}\n{status}"
+                await self.client.send_text(text, from_user)
+        except Exception as e:
+            logger.error(f"查询工作日状态失败: {e}")
+            await self.client.send_text("查询失败，请稍后再试", from_user)
+
+    async def _handle_latest_announcements(self, from_user: str) -> str | None:
+        global _last_latest_time
+        from app.api.deps import get_db, get_crawl_service
+        from sqlalchemy import select, asc, or_
+        from app.models.announcement import Announcement
+
+        # 防重入：60秒内整个函数只执行一次（含推送）
+        async with _latest_lock:
+            now = time.monotonic()
+            if (now - _last_latest_time) < 60:
+                logger.info("最新公告请求被忽略（防重入）")
+                return None
+            _last_latest_time = now
+
+        try:
+            await self.client.send_text("正在获取最新公告，请稍候...", from_user)
+            service = get_crawl_service()
+            await service.run_all()
+
+            # 查：广西政采网关键词匹配 + 大化县政府网全部，取最新6条后按时间升序推送
+            async for db in get_db():
+                from sqlalchemy import select, desc, or_
+                result = await db.execute(
+                    select(Announcement)
+                    .where(or_(
+                        Announcement.keyword_matched == True,  # noqa: E712
+                        Announcement.source_code == "dahuagov",
+                    ))
+                    .order_by(desc(Announcement.publish_date))
+                    .limit(6)
+                )
+                items = result.scalars().all()
+
+                if not items:
+                    await self.client.send_text("暂无公告", from_user)
+                    return None
+
+                # 按时间升序推送，企微里向上滑即为时间正序
+                for ann in sorted(items, key=lambda a: a.publish_date):
+                    title = ann.title if len(ann.title) <= 128 else ann.title[:125] + "..."
+                    purchase_name = ann.purchase_name or ""
+                    if len(purchase_name) > 25:
+                        purchase_name = purchase_name[:22] + "..."
+                    time_str = ann.publish_date.strftime("%Y-%m-%d %H:%M") if ann.publish_date else "时间未知"
+                    source_name = ann.source_name or ann.source_code or ""
+                    description = f"{source_name} | {purchase_name} | {time_str}"
+                    await self.client.send_textcard(title, description, ann.content_url or "", from_user)
+                    await asyncio.sleep(1)
+        except Exception as e:
+            logger.error(f"查询最新公告失败: {e}")
+            await self.client.send_text("查询失败，请稍后再试", from_user)
+
+    async def _handle_pause_scheduler(self, from_user: str) -> str | None:
+        try:
+            from app.scheduler.jobs import scheduler
+            if scheduler.running:
+                scheduler.pause()
+                await self.client.send_text("定时任务已暂停", from_user)
+            else:
+                await self.client.send_text("定时任务未在运行", from_user)
+        except Exception as e:
+            logger.error(f"暂停定时任务失败: {e}")
+            await self.client.send_text(f"操作失败: {e}", from_user)
+
+    async def _handle_toggle_ai(self, from_user: str) -> str | None:
+        from app.services.ai_state import (
+            get_ai_status_text,
+            get_whitelist,
+            is_ai_enabled,
+            is_whitelisted,
+            set_ai_enabled,
+        )
+
+        # 白名单校验
+        whitelist = get_whitelist()
+        if whitelist and not is_whitelisted(from_user):
+            await self.client.send_text(
+                f"⚠️ 你没有权限操作 AI 分析开关\n"
+                f"当前仅以下用户可操作：\n{', '.join(whitelist)}",
+                from_user,
+            )
+            return None
+
+        current = is_ai_enabled()
+        changed = set_ai_enabled(not current)
+        if not changed:
+            await self.client.send_text(
+                f"AI 分析当前已是「{get_ai_status_text()}」状态，无需切换",
+                from_user,
+            )
+            return None
+
+        await self.client.send_text(
+            f"✅ AI 分析已切换为「{get_ai_status_text()}」\n"
+            f"下次爬取触发时生效",
+            from_user,
+        )
+
+    async def _handle_resume_scheduler(self, from_user: str) -> str | None:
+        try:
+            from app.scheduler.jobs import scheduler
+            scheduler.resume()
+            await self.client.send_text("定时任务已恢复", from_user)
+        except Exception as e:
+            logger.error(f"恢复定时任务失败: {e}")
+            await self.client.send_text(f"操作失败: {e}", from_user)

app/wechat/ierror.py

@@ -1,10 +1,9 @@
 #!/usr/bin/env python
-# -*- coding: utf-8 -*-
 #########################################################################
 # Author: jonyqin
 # Created Time: Thu 11 Sep 2014 01:53:58 PM CST
 # File Name: ierror.py
-# Description:定义错误码含义 
+# Description:定义错误码含义
 #########################################################################
 WXBizMsgCrypt_OK = 0
 WXBizMsgCrypt_ValidateSignature_Error = -40001

app/wechat/menu.py

@@ -0,0 +1,112 @@
+import logging
+
+import httpx
+
+from app.config import settings
+
+logger = logging.getLogger(__name__)
+
+MENU = {
+    "button": [
+        {
+            "name": "最新公告",
+            "type": "click",
+            "key": "latest_announcements",
+        },
+        {
+            "name": "查询",
+            "sub_button": [
+                {
+                    "name": "监控配置",
+                    "type": "click",
+                    "key": "monitor_config",
+                },
+                {
+                    "name": "系统状态",
+                    "type": "click",
+                    "key": "system_status",
+                },
+                {
+                    "name": "今日工作日",
+                    "type": "click",
+                    "key": "workday_status",
+                },
+            ],
+        },
+        {
+            "name": "系统管理",
+            "sub_button": [
+                {
+                    "name": "立即爬取",
+                    "type": "click",
+                    "key": "trigger_crawl",
+                },
+                {
+                    "name": "同步节假日",
+                    "type": "click",
+                    "key": "sync_holidays",
+                },
+                {
+                    "name": "AI 分析",
+                    "type": "click",
+                    "key": "toggle_ai",
+                },
+            ],
+        },
+    ]
+}
+
+
+class MenuManager:
+    def __init__(self, client=None):
+        self.client = client
+
+    async def _get_token(self) -> str | None:
+        from app.wechat.client import WeChatClient
+        c = self.client or WeChatClient()
+        return await c._get_access_token()
+
+    async def create(self) -> bool:
+        token = await self._get_token()
+        if not token:
+            return False
+
+        url = f"{settings.wechat_api_base_url}/cgi-bin/menu/create"
+        params = {"access_token": token, "agentid": int(settings.wechat_agent_id)}
+
+        async with httpx.AsyncClient(timeout=15) as client:
+            response = await client.post(url, params=params, json=MENU)
+            data = response.json()
+            if data.get("errcode") == 0:
+                logger.info("企微菜单创建成功")
+                return True
+            logger.error(f"企微菜单创建失败: {data}")
+        return False
+
+    async def delete(self) -> bool:
+        token = await self._get_token()
+        if not token:
+            return False
+
+        url = f"{settings.wechat_api_base_url}/cgi-bin/menu/delete"
+        params = {"access_token": token, "agentid": int(settings.wechat_agent_id)}
+
+        async with httpx.AsyncClient(timeout=15) as client:
+            response = await client.get(url, params=params)
+            data = response.json()
+            return data.get("errcode") == 0
+
+    async def get(self) -> dict | None:
+        token = await self._get_token()
+        if not token:
+            return None
+
+        url = f"{settings.wechat_api_base_url}/cgi-bin/menu/get"
+        params = {"access_token": token, "agentid": int(settings.wechat_agent_id)}
+
+        async with httpx.AsyncClient(timeout=15) as client:
+            response = await client.get(url, params=params)
+            data = response.json()
+            if data.get("errcode") == 0:
+                return data
+        return None

docker/Dockerfile

@@ -0,0 +1,21 @@
+FROM python:3.12-slim
+
+LABEL maintainer="GX-gp-notify Dev Team" \
+      version="2.0.0" \
+      description="广西政府采购网公告监控系统"
+
+WORKDIR /app
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libpq-dev && \
+    rm -rf /var/lib/apt/lists/*
+
+COPY pyproject.toml .
+COPY vendor/ vendor/
+RUN pip install --no-cache-dir -e ".[dev]" && pip install --no-cache-dir ./vendor/
+
+COPY . .
+
+EXPOSE 8000
+
+CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

docker/docker-compose.yml

@@ -0,0 +1,17 @@
+services:
+  app:
+    build:
+      context: ..
+      dockerfile: docker/Dockerfile
+      tags:
+        - gx-gp-notify:2.0.0
+        - gx-gp-notify:latest
+    image: gx-gp-notify:2.0.0
+    container_name: gx-gp-notify
+    ports:
+      - "18001:8000"
+    env_file:
+      - ../.env
+    volumes:
+      - ../logs:/app/logs
+    restart: unless-stopped

docs/superpowers/specs/2026-05-09-fastapi-docker-migration-design.md

@@ -0,0 +1,516 @@
+# FastAPI + Docker 迁移设计文档
+
+**日期**: 2026-05-09
+**版本**: 1.0
+**状态**: 设计阶段
+
+---
+
+## 1. 背景与目标
+
+将现有的广西政府采购网公告监控系统从 Flask CLI 架构迁移到 FastAPI + Docker 部署方案，同时提升代码质量、安全性和可维护性。
+
+### 核心目标
+
+- FastAPI 替代 Flask 回调服务器 + argparse CLI，统一为一个 Web 应用
+- Docker 单容器部署，连接现有 PostgreSQL
+- SQLAlchemy ORM 替代原始 psycopg2 操作
+- APScheduler 内置定时任务
+- 环境变量管理配置，敏感信息不提交 git
+- 异步改造（httpx + asyncpg）
+
+---
+
+## 2. 新项目结构
+
+```
+gx-gp-notify/
+├── app/
+│   ├── __init__.py
+│   ├── main.py                  # FastAPI 应用入口 + lifespan
+│   ├── config.py                # Pydantic Settings (环境变量驱动)
+│   ├── api/
+│   │   ├── __init__.py
+│   │   ├── router.py            # 统一路由注册
+│   │   ├── deps.py              # 依赖注入 (get_db, get_scheduler)
+│   │   ├── announcements.py     # 公告 CRUD 端点
+│   │   ├── crawl.py             # 爬取触发端点
+│   │   └── wechat.py            # 企业微信回调端点
+│   ├── models/
+│   │   ├── __init__.py
+│   │   ├── announcement.py      # SQLAlchemy 模型
+│   │   └── schemas.py           # Pydantic 请求/响应模型
+│   ├── crawler/
+│   │   ├── __init__.py
+│   │   ├── base.py              # Spider 抽象基类 + Pipeline 配置
+│   │   ├── gxgp_spider.py       # 广西政府采购网爬虫 (API 模式)
+│   │   ├── dahuagov_spider.py   # 大化县政府网爬虫 (HTML 模式)
+│   │   └── parsers.py           # 数据解析器
+│   ├── services/
+│   │   ├── __init__.py
+│   │   ├── crawl_service.py     # 爬取编排器 (注册/调用所有 Spider)
+│   │   ├── pipeline.py          # 统一后处理管道 (存储→筛选→推送)
+│   │   ├── filter_service.py    # 筛选逻辑
+│   │   └── notification_service.py  # 通知服务
+│   ├── wechat/
+│   │   ├── __init__.py
+│   │   ├── handler.py           # 消息处理器
+│   │   ├── menu.py              # 菜单管理
+│   │   ├── crypto.py            # WXBizMsgCrypt 加解密
+│   │   └── client.py            # 企业微信 API 客户端
+│   └── scheduler/
+│       ├── __init__.py
+│       └── jobs.py              # APScheduler 定时任务
+├── tests/
+│   ├── __init__.py
+│   ├── conftest.py              # pytest fixtures
+│   ├── test_crawler/            # 爬虫解析器测试
+│   ├── test_api/                # API 端点测试
+│   └── test_services/           # 业务逻辑测试
+├── alembic/                     # 数据库迁移
+│   └── versions/
+├── docker/
+│   ├── Dockerfile
+│   └── docker-compose.yml
+├── pyproject.toml               # 项目元数据 + 依赖 + 工具配置
+├── .env.example                 # 环境变量模板
+├── .gitignore
+└── logs/                        # 日志目录 (gitignore)
+```
+
+---
+
+## 3. 爬虫模块化设计
+
+### 3.1 设计动机
+
+系统爬取两个不同来源的公告：广西政府采购网（API 模式，13 种公告类型）和大化县政府网（HTML 解析，单一来源）。两个爬虫的后处理流程不同：
+
+| 对比 | 广西政府采购网 | 大化县政府网 |
+|------|---------------|-------------|
+| 爬取方式 | POST JSON API | GET HTML + BeautifulSoup |
+| 反爬机制 | 敏感词检查 | 无 |
+| 分页 | 多页轮询 | 单页 |
+| 筛选策略 | 关键词 + 日期 + 去重 | 不过滤 |
+| 推送策略 | 只推送关键词匹配的 | 全部推送 |
+| 已发送标记 | is_new 标记 | is_new → mark_sent |
+
+为避免两个爬虫各自维护一套存储/筛选/推送逻辑，采用 **Spider 接口 + Pipeline 策略模式** 分离关注点。
+
+### 3.2 架构
+
+```
+┌─────────────────────────────────────────────────┐
+│              CrawlOrchestrator                   │
+│    (注册所有 Spider，统一调度爬取)                 │
+└─────────────────────┬───────────────────────────┘
+                      │
+        ┌─────────────┼─────────────┐
+        │             │             │
+┌───────▼──────┐ ┌────▼──────┐ ┌────▼──────────┐
+│ GXGPSpider   │ │ Dahuagov  │ │ (FutureSpider) │
+│ (API 模式)    │ │ Spider    │ │                │
+│ 13 sources   │ │ (HTML模式) │ │                │
+└───────┬──────┘ └────┬──────┘ └────┬───────────┘
+        │             │             │
+        └─────────────┼─────────────┘
+                      │  List[Announcement]
+                      │
+┌─────────────────────▼───────────────────────────┐
+│              PostCrawlPipeline                   │
+│                                                  │
+│  每个 Spider 声明 PipelineConfig:                 │
+│   - filter_enabled: 是否启用关键词筛选            │
+│   - keywords: 筛选关键词列表                      │
+│   - dedup_enabled: 是否去重                      │
+│   - notify_mode: "filtered" | "all"             │
+│   - mark_sent: 推送后是否标记已发送               │
+│                                                  │
+│  统一处理流程:                                    │
+│  存储 → 去重 → 筛选(可选) → 推送(可选) → 标记(可选) │
+└──────────────────────────────────────────────────┘
+```
+
+### 3.3 Spider 基类
+
+```python
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from typing import List
+
+@dataclass
+class PipelineConfig:
+    """Spider 后处理策略配置"""
+    filter_enabled: bool = True
+    keywords: List[str] = None
+    dedup_enabled: bool = True
+    notify_mode: str = "filtered"  # "filtered" | "all"
+    mark_sent: bool = False
+
+
+class BaseSpider(ABC):
+    """爬虫基类 — 只负责爬取+解析，不管后续如何处理"""
+
+    name: str                            # spider 名称
+    source_code: str                     # 来源代码
+    source_name: str                     # 来源名称
+
+    @abstractmethod
+    async def crawl(self) -> CrawlResult:
+        """执行爬取，返回包含 Announcement 列表的 CrawlResult"""
+        ...
+
+    def get_pipeline_config(self) -> PipelineConfig:
+        """返回后处理策略（子类可覆盖）"""
+        return PipelineConfig()
+```
+
+### 3.4 两个 Spider 的 PipelineConfig
+
+```python
+# GXGPSpider — 关键词筛选模式
+PipelineConfig(
+    filter_enabled=True,
+    keywords=["大化"],          # 从配置读取
+    dedup_enabled=True,
+    notify_mode="filtered",     # 只推送匹配关键词的
+    mark_sent=False,            # 用 is_new 标记，不单独 mark_sent
+)
+
+# DahuagovSpider — 全量推送模式
+PipelineConfig(
+    filter_enabled=False,
+    keywords=[],
+    dedup_enabled=True,
+    notify_mode="all",          # 全部推送
+    mark_sent=True,             # 推送后标记已发送
+)
+```
+
+### 3.5 统一后处理管道
+
+```python
+class PostCrawlPipeline:
+    """统一后处理管道 — 所有 Spider 共用"""
+
+    async def process(self, announcements: List[Announcement],
+                     config: PipelineConfig) -> PipelineResult:
+        # 1. 统一存入 announcements 表 (按 source_code 区分)
+        # 2. 去重 (基于 content_hash)
+        # 3. 根据 config.filter_enabled 决定是否筛选
+        # 4. 根据 config.notify_mode 决定推送策略
+        #    - "all": 推送全部新公告
+        #    - "filtered": 只推送匹配关键词的
+        # 5. 根据 config.mark_sent 标记已发送
+        # 6. 生成 Markdown (可选)
+        ...
+```
+
+### 3.6 优势
+
+- **新增爬虫源**只需实现 `BaseSpider.crawl()` + 声明 `PipelineConfig`，推送/存储逻辑零改动
+- **推送逻辑单一入口**，修改一处对所有源生效
+- 每个 Spider 文件只关注"怎么爬"和"怎么解析"，不再包含存储/筛选/推送代码
+- `dahuagov_announcements` 独立表可合并到统一的 `announcements` 表，用 `source_code='dahuagov'` 区分
+
+---
+
+## 4. API 端点设计
+
+### 3.1 公告查询
+
+| 方法 | 路径 | 说明 |
+|------|------|------|
+| `GET` | `/api/v1/announcements` | 查询公告列表（分页、来源筛选、日期筛选、关键词搜索） |
+| `GET` | `/api/v1/announcements/{id}` | 获取单条公告详情 |
+| `GET` | `/api/v1/announcements/today` | 获取今日公告 |
+| `GET` | `/api/v1/announcements/stats` | 获取公告统计信息 |
+
+### 3.2 爬取控制
+
+| 方法 | 路径 | 说明 |
+|------|------|------|
+| `POST` | `/api/v1/crawl/trigger` | 手动触发一次爬取（支持关键词/来源参数） |
+| `GET` | `/api/v1/crawl/status` | 查看最近爬取状态 |
+| `GET` | `/api/v1/crawl/sources` | 获取已配置的公告来源列表 |
+
+### 3.3 定时任务
+
+| 方法 | 路径 | 说明 |
+|------|------|------|
+| `GET` | `/api/v1/scheduler/jobs` | 查看活跃的定时任务 |
+| `POST` | `/api/v1/scheduler/pause/{job_id}` | 暂停某个定时任务 |
+| `POST` | `/api/v1/scheduler/resume/{job_id}` | 恢复某个定时任务 |
+
+### 3.4 企业微信
+
+| 方法 | 路径 | 说明 |
+|------|------|------|
+| `GET` `POST` | `/api/v1/wechat/callback` | 企业微信回调入口（URL验证 + 消息接收） |
+
+### 3.5 系统
+
+| 方法 | 路径 | 说明 |
+|------|------|------|
+| `GET` | `/health` | 健康检查（包含数据库连通性） |
+
+### 3.6 通用特性
+
+- FastAPI 自动生成 Swagger UI (`/docs`) 和 ReDoc (`/redoc`)
+- 统一分页格式：`{ total, page, page_size, items }`
+- 统一错误响应：`{ detail: "error message" }`
+- 依赖注入：数据库 session、配置、调度器均通过 FastAPI `Depends()` 注入
+- CORS 默认关闭，通过 `CORS_ORIGINS` 环境变量可选开启
+
+---
+
+## 5. 数据模型
+
+### 5.1 公告表 (announcements)
+
+合并现有的 `announcements`、`auto_announcements`、`manual_announcements`、`dahuagov_announcements` 四张表为一张，通过 `source_code` 字段区分来源，通过 `PipelineConfig` 控制每个来源的推送策略。
+
+| 字段 | 类型 | 说明 |
+|------|------|------|
+| id | Integer | 主键 |
+| title | String(500) | 公告标题 |
+| publish_date | DateTime | 发布时间 |
+| purchase_name | String(200) | 发布单位 |
+| content_url | Text | 内容链接 |
+| source_code | String(50) | 来源代码（ZcyAnnouncement1 / dahuagov 等） |
+| source_name | String(100) | 来源名称 |
+| announcement_type | String(50) | 公告类型枚举值 |
+| content_hash | String(64) | SHA256 内容哈希，UNIQUE INDEX |
+| crawl_mode | String(20) | "auto" / "manual" |
+| is_new | Boolean | 是否新公告 |
+| is_sent | Boolean | 是否已推送 |
+| keyword_matched | Boolean | 是否匹配关键词 |
+| created_at | DateTime | 创建时间 |
+| updated_at | DateTime | 更新时间 |
+
+### 5.2 设计决策
+
+- **四表合一**：不再为不同来源或抓取模式创建独立表。`source_code` 区分来源（`ZcyAnnouncement1` ~ `dahuagov`），`crawl_mode` 区分自动/手动
+- **is_sent 字段**：替代原来大化县的 `is_new → mark_sent` 模式，同时为广西采购网的公告提供统一的已推送标记
+- **keyword_matched 字段**：在筛选阶段标记，持久化到数据库方便后续查询
+
+### 5.3 变更说明
+
+- `content_hash` 从 MD5 (32位) 改为 SHA256 (64位)
+- 去掉 `announcement_sources` 表，来源信息从环境变量配置动态读取
+- 去掉 `crawl_results` 表，爬取结果通过日志记录
+- 使用 Alembic 管理数据库迁移
+- 旧数据迁移：通过 Alembic migration 脚本将现有四张表的数据合并到新表
+
+---
+
+## 5. 配置管理
+
+### 5.1 pydantic-settings
+
+所有配置通过 `pydantic-settings` 的 `BaseSettings` 加载，来源优先级：环境变量 > `.env` 文件 > 默认值。
+
+### 5.2 环境变量
+
+```bash
+# 应用
+DEBUG=false
+LOG_LEVEL=INFO
+
+# 数据库（连接现有 PostgreSQL）
+DATABASE_URL=postgresql+asyncpg://gx-gp-notify:password@10.10.10.14:5432/gx-gp-notify
+
+# 爬虫
+CRAWLER_BASE_URL=https://zfcg.gxzf.gov.cn
+CRAWLER_KEYWORDS=["大化"]
+CRAWLER_MAX_PAGES=10
+CRAWLER_TIMEOUT=30
+
+# 企业微信
+WECHAT_ENABLED=true
+WECHAT_CORP_ID=ww69e8e44636f47780
+WECHAT_AGENT_ID=1000007
+WECHAT_SECRET=<secret>
+WECHAT_TOKEN=<token>
+WECHAT_ENCODING_AES_KEY=<aes_key>
+
+# 定时任务
+SCHEDULER_ENABLED=true
+SCHEDULER_CRON=0 8,14,18 * * *
+
+# Markdown 输出
+MARKDOWN_OUTPUT_FILE=onu.md
+```
+
+### 5.3 安全措施
+
+- `.env` 文件加入 `.gitignore`
+- `config.yaml` 中的敏感信息不再提交到仓库
+- 使用 SHA256 替代 MD5 做内容哈希
+- 建议清理 git 历史中的敏感信息
+
+---
+
+## 6. Docker 部署
+
+### 6.1 架构
+
+```
+docker-compose.yml
+┌─────────────────────────────┐
+│  app (FastAPI)              │
+│  - uvicorn 服务器           │
+│  - APScheduler (内置)       │
+│  - 端口 8000                │
+│  - 挂载: ./logs             │
+└──────────┬──────────────────┘
+           │ TCP 连接
+┌──────────▼──────────────────┐
+│  外部 PostgreSQL (现有)      │
+│  10.10.10.14:5432           │
+└─────────────────────────────┘
+```
+
+### 6.2 Dockerfile
+
+```dockerfile
+FROM python:3.12-slim
+WORKDIR /app
+COPY pyproject.toml .
+RUN pip install --no-cache-dir .
+COPY . .
+CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
+```
+
+### 6.3 docker-compose.yml
+
+```yaml
+services:
+  app:
+    build: .
+    ports:
+      - "8000:8000"
+    env_file:
+      - .env
+    volumes:
+      - ./logs:/app/logs
+    restart: unless-stopped
+```
+
+### 6.4 启动步骤
+
+```bash
+cp .env.example .env        # 编辑 .env 填入真实配置
+docker compose up -d        # 启动服务
+docker compose logs -f app  # 查看日志
+```
+
+---
+
+## 7. 技术栈
+
+| 组件 | 当前 | 迁移后 |
+|------|------|--------|
+| Web 框架 | Flask (仅回调用) | FastAPI |
+| CLI | argparse | 不再需要，API 替代 |
+| 数据库驱动 | psycopg2 (同步) | asyncpg (异步) + SQLAlchemy 2.0 |
+| HTTP 客户端 | requests (同步) | httpx (异步) |
+| 定时任务 | schedule 库 | APScheduler |
+| 配置 | PyYAML + config.yaml | pydantic-settings + .env |
+| 加密 | pycryptodome + cryptography | 保留 |
+| 部署 | 裸机 + uWSGI | Docker + uvicorn |
+| 代码检查 | 无 | ruff |
+| 测试 | 无 | pytest + httpx |
+| 迁移 | 手动 CREATE TABLE | Alembic |
+
+---
+
+## 8. 额外改进
+
+### 8.1 异步改造
+
+- 爬虫网络请求：`httpx.AsyncClient` 替代 `requests`
+- 数据库：`asyncpg` + SQLAlchemy async engine
+- 通知发送：`httpx.AsyncClient`
+- API 端点使用 `async def`
+
+### 8.2 自动化测试
+
+- 爬虫解析器单元测试（Mock 网页响应，验证解析逻辑）
+- 筛选逻辑单元测试
+- API 端点集成测试（用 `httpx.AsyncClient` + 测试数据库）
+- pytest + pytest-asyncio + pytest-cov
+
+### 8.3 日志改进
+
+- 结构化日志（JSON 格式）
+- Docker 环境下输出到 stdout
+- 保留文件日志供宿主机查看（挂载 `./logs` 目录）
+
+### 8.4 错误处理
+
+- 统一异常处理中间件
+- 健康检查 `/health` 含数据库连通性验证
+- 关键操作结构化日志记录
+
+### 8.5 代码质量
+
+- `ruff` 替代 flake8 + black
+- `pyproject.toml` 统一管理依赖和工具配置
+- 全面 Type Hints
+
+### 8.6 .gitignore
+
+```gitignore
+.env
+logs/
+*.log
+__pycache__/
+*.pyc
+.venv/
+.ruff_cache/
+.pytest_cache/
+*.egg-info/
+```
+
+---
+
+## 9. 迁移策略
+
+### 分步执行
+
+1. 创建新项目结构骨架（FastAPI app + config）
+2. 数据库模型 + Alembic 迁移
+3. 迁移爬虫模块（异步改造）
+4. 迁移服务层（筛选、通知、Markdown）
+5. 迁移 WeChat 回调（Flask → FastAPI route）
+6. 添加 APScheduler 定时任务
+7. 编写测试
+8. Docker 化
+9. 清理旧文件
+
+### 向后兼容
+
+- 保留 `dahuagov_announcements` 表结构不变
+- 企业微信回调 URL 路径保持不变 (`/api/v1/wechat/callback`)
+- 数据库迁移使用 Alembic，不丢失现有数据
+
+---
+
+## 10. 未包含的内容
+
+- 不做多用户认证/授权（个人使用）
+- 不做 Celery 分布式任务（个人使用，APScheduler 足够）
+- 不做 Redis 缓存
+- 不做前端 UI
+
+---
+
+## 11. 风险与缓解
+
+| 风险 | 缓解 |
+|------|------|
+| 数据库迁移丢失数据 | Alembic 自动生成迁移脚本，先在测试环境验证 |
+| 异步爬虫被目标站限流 | 保留现有延迟/重试机制，httpx 支持同样的超时配置 |
+| 企业微信回调兼容性 | 回调路径不变，加解密逻辑完全复用现有代码 |
+| Docker 网络访问 10.10.10.14 | 确认 Docker 宿主机可访问该 IP，必要时用 `host.docker.internal` |

gx_gp_monitor/README.md

@@ -1,281 +0,0 @@
-# 广西政府采购网公告监控系统
-
-广西政府采购网公告爬取和监控的智能系统，支持多种公告类型的自动爬取、智能筛选、数据存储和通知推送。
-
-## 功能特性
-
-### 🕷️ 智能爬虫监控
-- 自动爬取广西政府采购网(zfcg.gxzf.gov.cn)的各类采购公告
-- 支持10+种公告类型：采购公告、结果公告、合同公告、更正公告、招标文件预公示、单一来源公示、电子卖场公示、履约验收公示、工程类公告、框架协议征集公告等
-- 智能反爬虫机制：随机User-Agent、代理轮换、请求延迟控制
-
-### 🎯 多维度智能筛选
-- **关键词过滤**：支持多关键词精确匹配和模糊匹配
-- **日期范围筛选**：支持开始日期和结束日期过滤
-- **自动去重**：基于内容哈希的智能去重机制
-- **来源筛选**：按公告类型进行筛选
-
-### 💾 数据持久化存储
-- **PostgreSQL数据库**：高效的数据存储和查询
-- **自动清理**：默认90天数据保留，可配置
-- **性能优化**：连接池、索引优化、批量操作
-
-### 🔔 高可用性保障
-- **重试机制**：指数退避重试策略
-- **超时控制**：连接超时和读取超时设置
-- **幂等操作**：防止重复处理
-- **异常处理**：完善的异常捕获和恢复
-- **结构化日志**：详细的运行日志记录
-
-### 📱 通知系统
-- **企业微信通知**：支持文本和Markdown格式消息
-- **实时推送**：新公告发现即时通知
-- **系统告警**：异常情况自动告警
-
-### 📊 Markdown输出
-- **自动生成**：将公告输出为Markdown格式文件（onu.md）
-- **结构化展示**：按来源分组、时间排序
-- **今日高亮**：突出显示今日发布的公告
-
-## 安装和使用
-
-### 环境要求
-- Python 3.8+
-- PostgreSQL 12+
-
-### 安装步骤
-
-1. **克隆项目**
-```bash
-cd /path/to/your/workspace
-# 项目已创建在 gx_gp_monitor 目录下
-```
-
-2. **创建虚拟环境**
-```bash
-python3 -m venv venv
-source venv/bin/activate  # Linux/Mac
-# 或
-venv\Scripts\activate     # Windows
-```
-
-3. **安装依赖**
-```bash
-pip install -r requirements.txt
-```
-
-4. **配置数据库**
-```bash
-# 创建PostgreSQL数据库
-createdb gx-gp-notify
-
-# 修改配置文件 config/config.yaml 中的数据库连接信息
-```
-
-5. **修改配置**
-```bash
-# 编辑 config/config.yaml 文件
-# 设置数据库连接、企业微信配置、关键词等
-```
-
-### 使用方法
-
-#### 命令行使用
-
-```bash
-# 查看帮助
-python main.py --help
-
-# 执行一次爬取
-python main.py crawl
-
-# 带参数爬取
-python main.py crawl --keywords "大化" "信息化" --max-pages 5
-
-# 启动调度器（定时任务）
-python main.py scheduler
-
-# 数据清理
-python main.py cleanup --days 30
-
-# 查看系统状态
-python main.py status
-```
-
-#### 编程接口使用
-
-```python
-from gx_gp_monitor.crawler.spider import crawl_announcements
-from gx_gp_monitor.filters.filters import filter_from_config
-from gx_gp_monitor.storage.postgresql import save_announcements_to_storage
-from gx_gp_monitor.storage.md_generator import generate_onu_md
-from gx_gp_monitor.notification.wechat import send_announcements_notification
-
-# 执行爬取
-results = crawl_announcements()
-
-# 筛选公告
-filter_obj = filter_from_config()
-filtered_announcements, stats = filter_obj.filter(all_announcements)
-
-# 保存到数据库
-saved_count = save_announcements_to_storage(filtered_announcements)
-
-# 生成Markdown文件
-generate_onu_md(filtered_announcements)
-
-# 发送通知
-send_announcements_notification(filtered_announcements)
-```
-
-## 配置说明
-
-### 主要配置文件：`config/config.yaml`
-
-```yaml
-# 调试模式
-debug: false
-
-# 日志配置
-log_level: INFO
-log_file: logs/gx_gp_monitor.log
-
-# 爬虫配置
-crawler:
-  base_url: "https://zfcg.gxzf.gov.cn"
-  timeout: 30
-  max_retries: 3
-  keyword: ["大化", "信息化"]  # 关键词筛选
-  max_pages: 10
-
-# 数据库配置
-database:
-  enabled: true
-  host: "localhost"
-  port: 5432
-  name: "gx-gp-notify"
-  user: "your_user"
-  password: "your_password"
-  data_retention_days: 90
-
-# 企业微信配置
-wechat_app:
-  enabled: true
-  corp_id: "your_corp_id"
-  agent_id: "your_agent_id"
-  secret: "your_secret"
-  token: "your_token"
-  encoding_aes_key: "your_aes_key"
-
-# 调度配置
-scheduler:
-  enabled: true
-  jobs:
-    - name: "daily_crawl"
-      cron: "0 8,14,18 * * *"
-      enabled: true
-```
-
-### 公告来源配置
-
-系统支持以下公告类型：
-
-- `ZcyAnnouncement1`: 采购公告
-- `ZcyAnnouncement2`: 结果公告
-- `ZcyAnnouncement3`: 合同公告
-- `ZcyAnnouncement4`: 更正公告
-- `ZcyAnnouncement5`: 招标文件预公示
-- `ZcyAnnouncement6`: 单一来源公示
-- `ZcyAnnouncement7`: 电子卖场公示
-- `ZcyAnnouncement10`: 履约验收公示
-- `ZcyAnnouncement11`: 工程类公告
-- `ZcyAnnouncement20`: 框架协议征集公告
-- `ZcyAnnouncement21`: 框架协议入围结果公告
-- `ZcyAnnouncement23`: 框架协议成交结果汇总公告
-- `61-266648`: 采购意向公开
-
-## 架构设计
-
-```
-gx_gp_monitor/
-├── config/           # 配置管理
-├── core/            # 核心模块
-│   ├── models.py           # 数据模型
-│   ├── config_manager.py   # 配置管理
-│   ├── database.py         # 数据库操作
-│   ├── logger.py           # 日志管理
-│   └── reliability.py      # 高可用性
-├── crawler/         # 爬虫模块
-│   ├── spider.py           # 爬虫核心
-│   └── parsers.py          # 数据解析
-├── filters/         # 筛选模块
-├── storage/         # 存储模块
-│   ├── postgresql.py       # PostgreSQL存储
-│   └── md_generator.py     # Markdown生成
-├── notification/    # 通知模块
-├── scheduler/       # 调度模块
-└── main.py          # 主程序入口
-```
-
-## 日志和监控
-
-### 日志文件
-- 默认日志文件：`logs/gx_gp_monitor.log`
-- 日志轮转：10MB大小限制，保留5个备份文件
-- 日志级别：DEBUG、INFO、WARNING、ERROR、CRITICAL
-
-### 监控指标
-- 爬取统计：成功/失败次数、响应时间
-- 数据统计：公告数量、新增数量、去重统计
-- 系统状态：内存使用、磁盘空间、健康检查
-
-## 故障排除
-
-### 常见问题
-
-1. **数据库连接失败**
-   - 检查PostgreSQL服务是否运行
-   - 验证数据库连接配置
-   - 确认用户权限
-
-2. **爬虫请求失败**
-   - 检查网络连接
-   - 验证目标网站是否可访问
-   - 调整请求间隔和重试策略
-
-3. **企业微信通知失败**
-   - 检查企业微信配置
-   - 验证应用ID和密钥
-   - 确认网络能访问企业微信API
-
-4. **权限问题**
-   - 确保日志目录和输出文件目录有写入权限
-   - 检查数据库用户权限
-
-### 调试模式
-
-启用调试模式获取更详细的日志：
-
-```yaml
-debug: true
-log_level: DEBUG
-```
-
-## 许可证
-
-本项目采用 MIT 许可证。
-
-## 贡献
-
-欢迎提交Issue和Pull Request来改进这个项目。
-
-## 版本历史
-
-- **v1.0.0** (2024-01-XX)
-  - 初始版本发布
-  - 支持10+种公告类型爬取
-  - 实现智能筛选和去重
-  - PostgreSQL数据存储
-  - 企业微信通知
-  - Markdown文件输出
-  - 定时任务调度

gx_gp_monitor/__init__.py

@@ -1,7 +0,0 @@
-"""
-广西政府采购网公告监控系统
-广西政府采购网公告爬取和监控的智能系统
-"""
-
-__version__ = "1.0.0"
-__author__ = "GX GP Monitor Team"

gx_gp_monitor/__main__.py

@@ -1,11 +0,0 @@
-"""
-包的main入口，使项目可以直接通过 python -m gx_gp_monitor 运行
-"""
-
-try:
-    from .main import main
-except ImportError:
-    from main import main
-
-if __name__ == "__main__":
-    main()

gx_gp_monitor/config/__init__.py

@@ -1 +0,0 @@
-"""配置管理模块"""

gx_gp_monitor/config/config.yaml

@@ -1,135 +0,0 @@
-# 广西政府采购网公告监控系统配置文件
-# 复制此文件为 config.yaml 并修改相应配置
-
-# 调试模式
-debug: false
-
-# 日志配置
-log_level: INFO
-log_file: logs/gx_gp_monitor.log
-log_max_size: 10485760  # 10MB
-log_backup_count: 5
-
-# 爬虫配置
-crawler:
-  base_url: "https://zfcg.gxzf.gov.cn"
-  timeout: 30                     # 请求超时时间（秒）
-  max_retries: 3                  # 最大重试次数
-  retry_delay: 1.0               # 重试初始延迟
-  max_retry_delay: 60.0          # 重试最大延迟
-  backoff_factor: 2.0            # 退避因子
-  user_agents:                    # User-Agent列表
-    - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
-    - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15"
-    - "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
-  proxies: []                     # 代理列表
-  request_delay: 1.0             # 请求间延迟
-  request_delay_max: 3.0         # 请求间最大延迟
-  keyword: ["大化", "信息化"]    # 关键词筛选（支持多个关键词）
-  start_date: ""                 # 开始日期 (YYYY-MM-DD)
-  end_date: ""                   # 结束日期 (YYYY-MM-DD)
-  max_pages: 10                  # 最大页数
-  page_size: 100                 # 每页大小
-
-# 数据库配置
-database:
-  enabled: true                   # 是否启用数据库存储
-  type: postgresql               # 数据库类型
-  host: "10.10.10.14"           # 数据库主机
-  port: 5432                     # 数据库端口
-  name: "gx-gp-notify"          # 数据库名称
-  user: "gx-gp-notify"          # 数据库用户名
-  password: "MA6RBX4F6Bd5DGmw"   # 数据库密码
-  pool_size: 5                   # 连接池大小
-  max_overflow: 10               # 最大连接数
-  pool_timeout: 30               # 连接超时时间
-  pool_recycle: 3600            # 连接回收时间（秒）
-  data_retention_days: 90        # 数据保留天数
-  auto_cleanup: true             # 是否自动清理过期数据
-
-# 企业微信通知配置
-wechat_app:
-  enabled: true                    # 是否启用企业微信通知
-  corp_id: "ww69e8e44636f47780"                      # 企业ID
-  agent_id: "1000007"                     # 应用ID
-  secret: "SmelCwKFoL0E9ATWFzr-w7gsfXBTN72lT1UqnNd0HpI"                       # 应用Secret
-  token: "DmvL98cAF6x9CFtQZwqD2emGL8S7HxA"                        # Token
-  encoding_aes_key: "yAc4OoSCP92YTefHXYfw27WeG9oF11W9d6nw6QYlU3D"             # 消息加密Key
-  port: 18001                      # 服务端口
-  host: "0.0.0.0"                 # 服务主机
-  debug: false                     # 调试模式
-  use_proxy: false                 # 是否使用代理API
-  proxy_api_url: "https://api.v6ole.top"  # 代理API地址
-
-# Markdown输出配置
-markdown:
-  enabled: true                    # 是否启用Markdown输出
-  output_file: "onu.md"           # 输出文件路径
-  max_entries: 1000               # 最大条目数
-  include_today_highlight: true   # 是否突出显示今日公告
-  template_file: "templates/announcement.md"  # 模板文件
-
-# 公告来源配置
-sources:
-  ZcyAnnouncement1:
-    category_id: 66485
-    name: "采购公告"
-    type: "purchase"
-  ZcyAnnouncement2:
-    category_id: 66485
-    name: "结果公告"
-    type: "result"
-  ZcyAnnouncement3:
-    category_id: 66485
-    name: "合同公告"
-    type: "contract"
-  ZcyAnnouncement4:
-    category_id: 66485
-    name: "更正公告"
-    type: "correction"
-  ZcyAnnouncement5:
-    category_id: 66485
-    name: "招标文件预公示"
-    type: "pre_announcement"
-  ZcyAnnouncement6:
-    category_id: 66485
-    name: "单一来源公示"
-    type: "single_source"
-  ZcyAnnouncement7:
-    category_id: 66485
-    name: "电子卖场公示"
-    type: "electronic_market"
-  ZcyAnnouncement10:
-    category_id: 66485
-    name: "履约验收公示"
-    type: "acceptance"
-  ZcyAnnouncement11:
-    category_id: 66485
-    name: "工程类公告"
-    type: "engineering"
-  ZcyAnnouncement20:
-    category_id: 66485
-    name: "框架协议征集公告"
-    type: "framework_agreement"
-  ZcyAnnouncement21:
-    category_id: 66485
-    name: "框架协议入围结果公告"
-    type: "framework_result"
-  ZcyAnnouncement23:
-    category_id: 66485
-    name: "框架协议成交结果汇总公告"
-    type: "framework_summary"
-  "61-266648":
-    category_id: 66485
-    name: "采购意向公开"
-    type: "intention"
-
-# 调度配置已移除 - 如需定时任务功能，请重新添加
-
-# 监控配置
-monitoring:
-  enabled: true                   # 是否启用监控
-  health_check_interval: 300      # 健康检查间隔（秒）
-  alert_on_failure: true          # 失败时是否告警
-  max_consecutive_failures: 3     # 最大连续失败次数
-  metrics_enabled: true           # 是否启用指标收集

gx_gp_monitor/core/__init__.py

@@ -1 +0,0 @@
-"""核心模块"""

gx_gp_monitor/core/config_manager.py

@@ -1,394 +0,0 @@
-"""
-配置管理模块
-负责加载、验证和管理系统配置
-"""
-
-import os
-import yaml
-from typing import Dict, Any, Optional
-from pathlib import Path
-from dataclasses import dataclass
-from enum import Enum
-
-
-class LogLevel(Enum):
-    """日志级别枚举"""
-    DEBUG = "DEBUG"
-    INFO = "INFO"
-    WARNING = "WARNING"
-    ERROR = "ERROR"
-    CRITICAL = "CRITICAL"
-
-
-@dataclass
-class CrawlerConfig:
-    """爬虫配置"""
-    base_url: str
-    timeout: int
-    max_retries: int
-    retry_delay: float
-    max_retry_delay: float
-    backoff_factor: float
-    user_agents: list
-    proxies: list
-    request_delay: float
-    request_delay_max: float
-    keyword: list
-    start_date: str
-    end_date: str
-    max_pages: int
-    page_size: int
-
-
-@dataclass
-class DatabaseConfig:
-    """数据库配置"""
-    enabled: bool
-    type: str
-    host: str
-    port: int
-    name: str
-    user: str
-    password: str
-    pool_size: int
-    max_overflow: int
-    pool_timeout: int
-    pool_recycle: int
-    data_retention_days: int
-    auto_cleanup: bool
-
-
-@dataclass
-class WeChatConfig:
-    """企业微信配置"""
-    enabled: bool
-    corp_id: str
-    agent_id: str
-    secret: str
-    token: str
-    encoding_aes_key: str
-    port: int
-    host: str
-    debug: bool
-    use_proxy: bool
-    proxy_api_url: str
-
-
-@dataclass
-class MarkdownConfig:
-    """Markdown配置"""
-    enabled: bool
-    output_file: str
-    max_entries: int
-    include_today_highlight: bool
-    template_file: str
-
-
-@dataclass
-class SchedulerConfig:
-    """调度配置"""
-    enabled: bool
-    timezone: str
-    jobs: list
-
-
-@dataclass
-class MonitoringConfig:
-    """监控配置"""
-    enabled: bool
-    health_check_interval: int
-    alert_on_failure: bool
-    max_consecutive_failures: int
-    metrics_enabled: bool
-
-
-@dataclass
-class SystemConfig:
-    """系统配置"""
-    debug: bool
-    log_level: LogLevel
-    log_file: str
-    log_max_size: int
-    log_backup_count: int
-    crawler: CrawlerConfig
-    database: DatabaseConfig
-    wechat_app: WeChatConfig
-    markdown: MarkdownConfig
-    scheduler: Optional[SchedulerConfig]
-    monitoring: MonitoringConfig
-    sources: Dict[str, Dict[str, Any]]
-
-
-class ConfigManager:
-    """配置管理器"""
-
-    def __init__(self, config_file: Optional[str] = None):
-        """
-        初始化配置管理器
-
-        Args:
-            config_file: 配置文件路径，如果为None则使用默认路径
-        """
-        if config_file is None:
-            # 默认配置文件路径
-            current_dir = Path(__file__).parent.parent
-            self.config_file = current_dir / "config" / "config.yaml"
-        else:
-            self.config_file = Path(config_file)
-
-        self._config_data = {}
-        self._config = None
-
-    def load_config(self) -> SystemConfig:
-        """
-        加载配置文件
-
-        Returns:
-            SystemConfig: 系统配置对象
-
-        Raises:
-            FileNotFoundError: 配置文件不存在
-            yaml.YAMLError: 配置文件格式错误
-            ValueError: 配置验证失败
-        """
-        if not self.config_file.exists():
-            raise FileNotFoundError(f"配置文件不存在: {self.config_file}")
-
-        try:
-            with open(self.config_file, 'r', encoding='utf-8') as f:
-                self._config_data = yaml.safe_load(f)
-        except yaml.YAMLError as e:
-            raise yaml.YAMLError(f"配置文件格式错误: {e}")
-
-        # 验证配置
-        self._validate_config()
-
-        # 解析配置
-        self._config = self._parse_config()
-        return self._config
-
-    def _validate_config(self):
-        """验证配置完整性"""
-        required_keys = [
-            'debug', 'log_level', 'log_file', 'log_max_size', 'log_backup_count',
-            'crawler', 'database', 'wechat_app', 'markdown',
-            'monitoring', 'sources'
-        ]
-
-        for key in required_keys:
-            if key not in self._config_data:
-                raise ValueError(f"配置文件缺少必需的配置项: {key}")
-
-        # 验证爬虫配置
-        crawler_required = [
-            'base_url', 'timeout', 'max_retries', 'retry_delay', 'max_retry_delay',
-            'backoff_factor', 'user_agents', 'proxies', 'request_delay',
-            'request_delay_max', 'keyword', 'max_pages', 'page_size'
-        ]
-
-        for key in crawler_required:
-            if key not in self._config_data['crawler']:
-                raise ValueError(f"爬虫配置缺少必需项: {key}")
-
-        # 验证数据库配置
-        if self._config_data.get('database', {}).get('enabled', False):
-            db_required = ['type', 'host', 'port', 'name', 'user', 'password']
-            for key in db_required:
-                if key not in self._config_data['database']:
-                    raise ValueError(f"数据库配置缺少必需项: {key}")
-
-        # 验证企业微信配置
-        if self._config_data.get('wechat_app', {}).get('enabled', False):
-            wechat_required = ['corp_id', 'agent_id', 'secret', 'token', 'encoding_aes_key']
-            for key in wechat_required:
-                if key not in self._config_data['wechat_app']:
-                    raise ValueError(f"企业微信配置缺少必需项: {key}")
-
-    def _parse_config(self) -> SystemConfig:
-        """解析配置数据"""
-        crawler_data = self._config_data['crawler']
-        crawler = CrawlerConfig(
-            base_url=crawler_data['base_url'],
-            timeout=crawler_data['timeout'],
-            max_retries=crawler_data['max_retries'],
-            retry_delay=crawler_data['retry_delay'],
-            max_retry_delay=crawler_data['max_retry_delay'],
-            backoff_factor=crawler_data['backoff_factor'],
-            user_agents=crawler_data['user_agents'],
-            proxies=crawler_data['proxies'],
-            request_delay=crawler_data['request_delay'],
-            request_delay_max=crawler_data['request_delay_max'],
-            keyword=crawler_data['keyword'],
-            start_date=crawler_data.get('start_date', ''),
-            end_date=crawler_data.get('end_date', ''),
-            max_pages=crawler_data['max_pages'],
-            page_size=crawler_data['page_size']
-        )
-
-        db_data = self._config_data['database']
-        database = DatabaseConfig(
-            enabled=db_data.get('enabled', False),
-            type=db_data.get('type', 'postgresql'),
-            host=db_data.get('host', 'localhost'),
-            port=db_data.get('port', 5432),
-            name=db_data.get('name', ''),
-            user=db_data.get('user', ''),
-            password=db_data.get('password', ''),
-            pool_size=db_data.get('pool_size', 5),
-            max_overflow=db_data.get('max_overflow', 10),
-            pool_timeout=db_data.get('pool_timeout', 30),
-            pool_recycle=db_data.get('pool_recycle', 3600),
-            data_retention_days=db_data.get('data_retention_days', 90),
-            auto_cleanup=db_data.get('auto_cleanup', True)
-        )
-
-        wechat_data = self._config_data['wechat_app']
-        wechat_app = WeChatConfig(
-            enabled=wechat_data.get('enabled', False),
-            corp_id=wechat_data.get('corp_id', ''),
-            agent_id=wechat_data.get('agent_id', ''),
-            secret=wechat_data.get('secret', ''),
-            token=wechat_data.get('token', ''),
-            encoding_aes_key=wechat_data.get('encoding_aes_key', ''),
-            port=wechat_data.get('port', 18001),
-            host=wechat_data.get('host', '0.0.0.0'),
-            debug=wechat_data.get('debug', False),
-            use_proxy=wechat_data.get('use_proxy', False),
-            proxy_api_url=wechat_data.get('proxy_api_url', 'https://api.v6ole.top')
-        )
-
-        md_data = self._config_data['markdown']
-        markdown = MarkdownConfig(
-            enabled=md_data.get('enabled', True),
-            output_file=md_data.get('output_file', 'onu.md'),
-            max_entries=md_data.get('max_entries', 1000),
-            include_today_highlight=md_data.get('include_today_highlight', True),
-            template_file=md_data.get('template_file', 'templates/announcement.md')
-        )
-
-        # scheduler配置为可选
-        scheduler = None
-        if 'scheduler' in self._config_data:
-            scheduler_data = self._config_data['scheduler']
-            scheduler = SchedulerConfig(
-                enabled=scheduler_data.get('enabled', False),
-                timezone=scheduler_data.get('timezone', 'Asia/Shanghai'),
-                jobs=scheduler_data.get('jobs', [])
-            )
-
-        monitoring_data = self._config_data['monitoring']
-        monitoring = MonitoringConfig(
-            enabled=monitoring_data.get('enabled', True),
-            health_check_interval=monitoring_data.get('health_check_interval', 300),
-            alert_on_failure=monitoring_data.get('alert_on_failure', True),
-            max_consecutive_failures=monitoring_data.get('max_consecutive_failures', 3),
-            metrics_enabled=monitoring_data.get('metrics_enabled', True)
-        )
-
-        return SystemConfig(
-            debug=self._config_data['debug'],
-            log_level=LogLevel(self._config_data['log_level']),
-            log_file=self._config_data['log_file'],
-            log_max_size=self._config_data['log_max_size'],
-            log_backup_count=self._config_data['log_backup_count'],
-            crawler=crawler,
-            database=database,
-            wechat_app=wechat_app,
-            markdown=markdown,
-            scheduler=scheduler,
-            monitoring=monitoring,
-            sources=self._config_data['sources']
-        )
-
-    def get_config(self) -> SystemConfig:
-        """
-        获取配置对象
-
-        Returns:
-            SystemConfig: 系统配置对象
-
-        Raises:
-            RuntimeError: 配置未加载
-        """
-        if self._config is None:
-            raise RuntimeError("配置未加载，请先调用 load_config()")
-        return self._config
-
-    def reload_config(self) -> SystemConfig:
-        """
-        重新加载配置
-
-        Returns:
-            SystemConfig: 重新加载的系统配置对象
-        """
-        self._config = None
-        return self.load_config()
-
-    def get_value(self, key_path: str, default=None) -> Any:
-        """
-        通过路径获取配置值
-
-        Args:
-            key_path: 配置路径，如 'crawler.timeout' 或 'database.host'
-            default: 默认值
-
-        Returns:
-            配置值或默认值
-        """
-        keys = key_path.split('.')
-        value = self._config_data
-
-        try:
-            for key in keys:
-                value = value[key]
-            return value
-        except (KeyError, TypeError):
-            return default
-
-
-# 全局配置管理器实例
-_config_manager = None
-
-
-def get_config_manager(config_file: Optional[str] = None) -> ConfigManager:
-    """
-    获取全局配置管理器实例
-
-    Args:
-        config_file: 配置文件路径
-
-    Returns:
-        ConfigManager: 配置管理器实例
-    """
-    global _config_manager
-    if _config_manager is None:
-        _config_manager = ConfigManager(config_file)
-    return _config_manager
-
-
-def load_config(config_file: Optional[str] = None) -> SystemConfig:
-    """
-    加载系统配置
-
-    Args:
-        config_file: 配置文件路径
-
-    Returns:
-        SystemConfig: 系统配置对象
-    """
-    manager = get_config_manager(config_file)
-    return manager.load_config()
-
-
-def get_config() -> SystemConfig:
-    """
-    获取当前加载的配置
-
-    Returns:
-        SystemConfig: 系统配置对象
-
-    Raises:
-        RuntimeError: 配置未加载
-    """
-    manager = get_config_manager()
-    return manager.get_config()

gx_gp_monitor/core/database.py

@@ -1,612 +0,0 @@
-"""
-PostgreSQL数据库连接和操作模块
-提供数据库连接池、CRUD操作、数据清理等功能
-"""
-
-import psycopg2
-from psycopg2 import pool, extras
-from psycopg2.extras import RealDictCursor
-from contextlib import contextmanager
-from typing import List, Dict, Any, Optional, Generator
-from datetime import datetime, timedelta
-import threading
-from dataclasses import asdict
-
-from .models import Announcement, AnnouncementSource, AnnouncementType, CrawlResult, CrawlStatus
-from .config_manager import get_config
-from .logger import get_logger
-from .reliability import retry_on_exception, RetryConfig
-
-
-logger = get_logger(__name__)
-
-
-class DatabaseConnectionPool:
-    """数据库连接池管理器"""
-
-    _instance = None
-    _pool = None
-    _lock = threading.Lock()
-
-    def __new__(cls):
-        if cls._instance is None:
-            with cls._lock:
-                if cls._instance is None:
-                    cls._instance = super().__new__(cls)
-        return cls._instance
-
-    def __init__(self):
-        if self._pool is None:
-            self._pool = None
-            self._config = None
-
-    def init_pool(self, config):
-        """
-        初始化连接池
-
-        Args:
-            config: 数据库配置
-        """
-        if self._pool is not None:
-            return
-
-        try:
-            self._config = config
-            self._pool = psycopg2.pool.SimpleConnectionPool(
-                minconn=config.pool_size,
-                maxconn=config.pool_size + config.max_overflow,
-                host=config.host,
-                port=config.port,
-                database=config.name,
-                user=config.user,
-                password=config.password,
-                connect_timeout=config.pool_timeout
-            )
-            logger.info("数据库连接池初始化成功")
-        except Exception as e:
-            logger.error(f"数据库连接池初始化失败: {str(e)}")
-            raise
-
-    def get_connection(self):
-        """
-        获取数据库连接
-
-        Returns:
-            数据库连接对象
-
-        Raises:
-            Exception: 获取连接失败
-        """
-        if self._pool is None:
-            raise Exception("数据库连接池未初始化")
-
-        try:
-            conn = self._pool.getconn()
-            # 设置自动提交为False，需要手动提交
-            conn.autocommit = False
-            return conn
-        except Exception as e:
-            logger.error(f"获取数据库连接失败: {str(e)}")
-            raise
-
-    def return_connection(self, conn):
-        """
-        返回数据库连接到连接池
-
-        Args:
-            conn: 数据库连接对象
-        """
-        if self._pool and conn:
-            try:
-                self._pool.putconn(conn)
-            except Exception as e:
-                logger.warning(f"返回数据库连接失败: {str(e)}")
-
-    def close_all(self):
-        """关闭所有连接"""
-        if self._pool:
-            try:
-                self._pool.closeall()
-                logger.info("数据库连接池已关闭")
-            except Exception as e:
-                logger.error(f"关闭数据库连接池失败: {str(e)}")
-
-
-# 全局连接池实例
-_connection_pool = DatabaseConnectionPool()
-
-
-@contextmanager
-def get_db_connection():
-    """
-    获取数据库连接的上下文管理器
-
-    Yields:
-        数据库连接对象
-    """
-    conn = None
-    try:
-        conn = _connection_pool.get_connection()
-        yield conn
-    except Exception as e:
-        logger.error(f"数据库连接错误: {str(e)}")
-        raise
-    finally:
-        if conn:
-            _connection_pool.return_connection(conn)
-
-
-@contextmanager
-def get_db_cursor(commit: bool = True):
-    """
-    获取数据库游标的上下文管理器
-
-    Args:
-        commit: 是否自动提交事务
-
-    Yields:
-        数据库游标对象
-    """
-    with get_db_connection() as conn:
-        cursor = None
-        try:
-            cursor = conn.cursor(cursor_factory=RealDictCursor)
-            yield cursor
-            if commit:
-                conn.commit()
-        except Exception as e:
-            conn.rollback()
-            logger.error(f"数据库操作错误: {str(e)}")
-            raise
-        finally:
-            if cursor:
-                cursor.close()
-
-
-class DatabaseManager:
-    """数据库管理器"""
-
-    def __init__(self):
-        self.config = get_config()
-        if self.config.database.enabled:
-            _connection_pool.init_pool(self.config.database)
-        else:
-            logger.warning("数据库功能已禁用")
-
-    def init_database(self):
-        """初始化数据库表结构"""
-        if not self.config.database.enabled:
-            return
-
-        logger.info("开始初始化数据库表结构")
-
-        # 创建表的SQL语句
-        create_tables_sql = """
-        -- 公告表
-        CREATE TABLE IF NOT EXISTS announcements (
-            id SERIAL PRIMARY KEY,
-            title VARCHAR(500) NOT NULL,
-            publish_date TIMESTAMP NOT NULL,
-            purchase_name VARCHAR(200),
-            content_url TEXT,
-            source_code VARCHAR(50) NOT NULL,
-            source_name VARCHAR(100) NOT NULL,
-            announcement_type VARCHAR(50) NOT NULL,
-            crawled_at TIMESTAMP,
-            created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-            updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-            content_hash VARCHAR(32) UNIQUE,
-            keyword_matched BOOLEAN DEFAULT FALSE,
-            date_filtered BOOLEAN DEFAULT TRUE,
-            is_new BOOLEAN DEFAULT TRUE,
-            is_today BOOLEAN DEFAULT FALSE
-        );
-
-        -- 公告来源表
-        CREATE TABLE IF NOT EXISTS announcement_sources (
-            code VARCHAR(50) PRIMARY KEY,
-            category_id INTEGER NOT NULL,
-            name VARCHAR(100) NOT NULL,
-            type VARCHAR(50) NOT NULL
-        );
-
-        -- 爬取结果表
-        CREATE TABLE IF NOT EXISTS crawl_results (
-            id SERIAL PRIMARY KEY,
-            source_code VARCHAR(50) NOT NULL,
-            status VARCHAR(20) NOT NULL,
-            total_count INTEGER DEFAULT 0,
-            new_count INTEGER DEFAULT 0,
-            error_message TEXT,
-            crawled_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-            duration FLOAT DEFAULT 0.0,
-            FOREIGN KEY (source_code) REFERENCES announcement_sources(code)
-        );
-
-        -- 创建索引
-        CREATE INDEX IF NOT EXISTS idx_announcements_publish_date ON announcements(publish_date DESC);
-        CREATE INDEX IF NOT EXISTS idx_announcements_source_code ON announcements(source_code);
-        CREATE INDEX IF NOT EXISTS idx_announcements_content_hash ON announcements(content_hash);
-        CREATE INDEX IF NOT EXISTS idx_announcements_created_at ON announcements(created_at DESC);
-        CREATE INDEX IF NOT EXISTS idx_crawl_results_crawled_at ON crawl_results(crawled_at DESC);
-
-        -- 创建更新时间触发器
-        CREATE OR REPLACE FUNCTION update_updated_at_column()
-        RETURNS TRIGGER AS $$
-        BEGIN
-            NEW.updated_at = CURRENT_TIMESTAMP;
-            RETURN NEW;
-        END;
-        $$ language 'plpgsql';
-
-        DROP TRIGGER IF EXISTS update_announcements_updated_at ON announcements;
-        CREATE TRIGGER update_announcements_updated_at
-            BEFORE UPDATE ON announcements
-            FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
-        """
-
-        with get_db_cursor() as cursor:
-            cursor.execute(create_tables_sql)
-            logger.info("数据库表结构初始化完成")
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def save_announcement(self, announcement: Announcement) -> bool:
-        """
-        保存公告到数据库
-
-        Args:
-            announcement: 公告对象
-
-        Returns:
-            bool: 保存是否成功
-        """
-        if not self.config.database.enabled:
-            return False
-
-        # 生成内容哈希（如果还没有）
-        if not announcement.content_hash:
-            announcement.generate_content_hash()
-
-        sql = """
-        INSERT INTO announcements (
-            title, publish_date, purchase_name, content_url, source_code, source_name,
-            announcement_type, crawled_at, content_hash, keyword_matched,
-            date_filtered, is_new, is_today
-        ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
-        ON CONFLICT (content_hash) DO NOTHING
-        """
-
-        values = (
-            announcement.title,
-            announcement.publish_date,
-            announcement.purchase_name,
-            announcement.content_url,
-            announcement.source_code,
-            announcement.source_name,
-            announcement.announcement_type.value,
-            announcement.crawled_at,
-            announcement.content_hash,
-            announcement.keyword_matched,
-            announcement.date_filtered,
-            announcement.is_new,
-            announcement.is_today
-        )
-
-        try:
-            with get_db_cursor() as cursor:
-                cursor.execute(sql, values)
-                affected_rows = cursor.rowcount
-                if affected_rows > 0:
-                    logger.debug(f"成功保存公告: {announcement.title[:50]}...")
-                    return True
-                else:
-                    logger.debug(f"公告已存在，跳过保存: {announcement.title[:50]}...")
-                    return False
-        except Exception as e:
-            logger.error(f"保存公告失败: {str(e)}")
-            return False
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def save_announcements_batch(self, announcements: List[Announcement]) -> int:
-        """
-        批量保存公告
-
-        Args:
-            announcements: 公告列表
-
-        Returns:
-            int: 成功保存的数量
-        """
-        if not self.config.database.enabled:
-            return 0
-
-        if not announcements:
-            return 0
-
-        # 为没有哈希的公告生成哈希
-        for announcement in announcements:
-            if not announcement.content_hash:
-                announcement.generate_content_hash()
-
-        sql = """
-        INSERT INTO announcements (
-            title, publish_date, purchase_name, content_url, source_code, source_name,
-            announcement_type, crawled_at, content_hash, keyword_matched,
-            date_filtered, is_new, is_today
-        ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
-        ON CONFLICT (content_hash) DO NOTHING
-        """
-
-        values = []
-        for announcement in announcements:
-            values.append((
-                announcement.title,
-                announcement.publish_date,
-                announcement.purchase_name,
-                announcement.content_url,
-                announcement.source_code,
-                announcement.source_name,
-                announcement.announcement_type.value,
-                announcement.crawled_at,
-                announcement.content_hash,
-                announcement.keyword_matched,
-                announcement.date_filtered,
-                announcement.is_new,
-                announcement.is_today
-            ))
-
-        try:
-            with get_db_cursor() as cursor:
-                extras.execute_batch(cursor, sql, values)
-                affected_rows = cursor.rowcount
-                logger.info(f"批量保存公告完成，成功保存 {affected_rows} 条")
-                return affected_rows
-        except Exception as e:
-            logger.error(f"批量保存公告失败: {str(e)}")
-            return 0
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def get_announcements(self,
-                         source_code: Optional[str] = None,
-                         start_date: Optional[datetime] = None,
-                         end_date: Optional[datetime] = None,
-                         limit: int = 100,
-                         offset: int = 0) -> List[Announcement]:
-        """
-        查询公告
-
-        Args:
-            source_code: 来源代码过滤
-            start_date: 开始日期
-            end_date: 结束日期
-            limit: 限制数量
-            offset: 偏移量
-
-        Returns:
-            List[Announcement]: 公告列表
-        """
-        if not self.config.database.enabled:
-            return []
-
-        sql = """
-        SELECT * FROM announcements
-        WHERE 1=1
-        """
-        params = []
-
-        if source_code:
-            sql += " AND source_code = %s"
-            params.append(source_code)
-
-        if start_date:
-            sql += " AND publish_date >= %s"
-            params.append(start_date)
-
-        if end_date:
-            sql += " AND publish_date <= %s"
-            params.append(end_date)
-
-        sql += " ORDER BY publish_date DESC LIMIT %s OFFSET %s"
-        params.extend([limit, offset])
-
-        try:
-            with get_db_cursor() as cursor:
-                cursor.execute(sql, params)
-                rows = cursor.fetchall()
-
-                announcements = []
-                for row in rows:
-                    # 转换数据类型
-                    row_dict = dict(row)
-                    row_dict['announcement_type'] = AnnouncementType(row_dict['announcement_type'])
-                    announcements.append(Announcement.from_dict(row_dict))
-
-                return announcements
-        except Exception as e:
-            logger.error(f"查询公告失败: {str(e)}")
-            return []
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def save_crawl_result(self, result: CrawlResult) -> bool:
-        """
-        保存爬取结果
-
-        Args:
-            result: 爬取结果对象
-
-        Returns:
-            bool: 保存是否成功
-        """
-        if not self.config.database.enabled:
-            return False
-
-        sql = """
-        INSERT INTO crawl_results (
-            source_code, status, total_count, new_count, error_message,
-            crawled_at, duration
-        ) VALUES (%s, %s, %s, %s, %s, %s, %s)
-        """
-
-        values = (
-            result.source.code,
-            result.status.value,
-            result.total_count,
-            result.new_count,
-            result.error_message,
-            result.crawled_at,
-            result.duration
-        )
-
-        try:
-            with get_db_cursor() as cursor:
-                cursor.execute(sql, values)
-                logger.debug(f"保存爬取结果: {result.source.name}")
-                return True
-        except Exception as e:
-            logger.error(f"保存爬取结果失败: {str(e)}")
-            return False
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def cleanup_expired_data(self, days: int = 90) -> int:
-        """
-        清理过期数据
-
-        Args:
-            days: 保留天数
-
-        Returns:
-            int: 清理的记录数
-        """
-        if not self.config.database.enabled:
-            return 0
-
-        cutoff_date = datetime.now() - timedelta(days=days)
-
-        sql = "DELETE FROM announcements WHERE created_at < %s"
-        try:
-            with get_db_cursor() as cursor:
-                cursor.execute(sql, (cutoff_date,))
-                deleted_count = cursor.rowcount
-                logger.info(f"清理过期数据完成，删除 {deleted_count} 条记录")
-                return deleted_count
-        except Exception as e:
-            logger.error(f"清理过期数据失败: {str(e)}")
-            return 0
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def get_statistics(self) -> Dict[str, Any]:
-        """
-        获取统计信息
-
-        Returns:
-            Dict[str, Any]: 统计数据
-        """
-        if not self.config.database.enabled:
-            return {}
-
-        sql = """
-        SELECT
-            COUNT(*) as total_announcements,
-            COUNT(CASE WHEN is_today THEN 1 END) as today_announcements,
-            COUNT(CASE WHEN is_new THEN 1 END) as new_announcements,
-            COUNT(DISTINCT source_code) as sources_count,
-            MAX(crawled_at) as last_crawl_time
-        FROM announcements
-        """
-
-        try:
-            with get_db_cursor() as cursor:
-                cursor.execute(sql)
-                result = cursor.fetchone()
-                return dict(result) if result else {}
-        except Exception as e:
-            logger.error(f"获取统计信息失败: {str(e)}")
-            return {}
-
-    def is_announcement_exists(self, content_hash: str) -> bool:
-        """
-        检查公告是否已存在
-
-        Args:
-            content_hash: 内容哈希
-
-        Returns:
-            bool: 是否存在
-        """
-        if not self.config.database.enabled:
-            return False
-
-        sql = "SELECT 1 FROM announcements WHERE content_hash = %s LIMIT 1"
-
-        try:
-            with get_db_cursor() as cursor:
-                cursor.execute(sql, (content_hash,))
-                return cursor.fetchone() is not None
-        except Exception as e:
-            logger.error(f"检查公告存在性失败: {str(e)}")
-            return False
-
-    def get_recent_announcements(self, hours: int = 24) -> List[Announcement]:
-        """
-        获取最近的公告
-
-        Args:
-            hours: 最近小时数
-
-        Returns:
-            List[Announcement]: 公告列表
-        """
-        if not self.config.database.enabled:
-            return []
-
-        cutoff_time = datetime.now() - timedelta(hours=hours)
-
-        sql = """
-        SELECT * FROM announcements
-        WHERE crawled_at >= %s
-        ORDER BY crawled_at DESC
-        """
-
-        try:
-            with get_db_cursor() as cursor:
-                cursor.execute(sql, (cutoff_time,))
-                rows = cursor.fetchall()
-
-                announcements = []
-                for row in rows:
-                    row_dict = dict(row)
-                    row_dict['announcement_type'] = AnnouncementType(row_dict['announcement_type'])
-                    announcements.append(Announcement.from_dict(row_dict))
-
-                return announcements
-        except Exception as e:
-            logger.error(f"获取最近公告失败: {str(e)}")
-            return []
-
-
-# 全局数据库管理器实例
-_db_manager = None
-
-
-def get_database_manager() -> DatabaseManager:
-    """
-    获取数据库管理器实例
-
-    Returns:
-        DatabaseManager: 数据库管理器实例
-    """
-    global _db_manager
-    if _db_manager is None:
-        _db_manager = DatabaseManager()
-    return _db_manager
-
-
-def init_database():
-    """初始化数据库"""
-    manager = get_database_manager()
-    manager.init_database()
-
-
-def cleanup_database():
-    """清理数据库连接"""
-    _connection_pool.close_all()

gx_gp_monitor/core/logger.py

@@ -1,354 +0,0 @@
-"""
-统一日志管理模块
-提供结构化日志记录功能，支持控制台和文件输出
-"""
-
-import os
-import sys
-import logging
-import logging.handlers
-from pathlib import Path
-from typing import Optional, Dict, Any
-from datetime import datetime
-
-from .config_manager import get_config
-
-
-class ColoredFormatter(logging.Formatter):
-    """带颜色的日志格式化器"""
-
-    # ANSI颜色代码
-    COLORS = {
-        'DEBUG': '\033[36m',      # 青色
-        'INFO': '\033[32m',       # 绿色
-        'WARNING': '\033[33m',    # 黄色
-        'ERROR': '\033[31m',      # 红色
-        'CRITICAL': '\033[35m',   # 紫色
-    }
-    RESET = '\033[0m'  # 重置颜色
-
-    def format(self, record):
-        # 检查是否已经包含ANSI颜色代码
-        if '\033[' in record.levelname:
-            # 如果已经着色，直接返回原始格式
-            return super().format(record)
-
-        # 添加颜色
-        if record.levelname in self.COLORS:
-            # 为levelname添加颜色
-            colored_levelname = f"{self.COLORS[record.levelname]}{record.levelname}{self.RESET}"
-            # 使用原始levelname来确定消息的颜色
-            record.msg = f"{self.COLORS[record.levelname]}{record.msg}{self.RESET}"
-            record.levelname = colored_levelname
-
-        return super().format(record)
-
-
-class Logger:
-    """统一日志管理器"""
-
-    _instance = None
-    _initialized = False
-
-    def __new__(cls):
-        if cls._instance is None:
-            cls._instance = super().__new__(cls)
-        return cls._instance
-
-    def __init__(self):
-        if not self._initialized:
-            self._loggers = {}
-            self._config = None
-            self._initialized = True
-
-    def init_logger(self, name: str = "gx_gp_monitor", config=None) -> logging.Logger:
-        """
-        初始化日志器
-
-        Args:
-            name: 日志器名称
-            config: 配置对象，如果为None则从全局配置加载
-
-        Returns:
-            logging.Logger: 配置好的日志器实例
-        """
-        if name in self._loggers:
-            return self._loggers[name]
-
-        # 获取配置
-        if config is None:
-            try:
-                self._config = get_config()
-            except RuntimeError:
-                # 配置未加载，使用默认配置
-                self._config = self._get_default_config()
-        else:
-            self._config = config
-
-        # 创建日志器
-        logger = logging.getLogger(name)
-        logger.setLevel(getattr(logging, self._config.log_level.value))
-
-        # 避免重复添加处理器
-        if logger.handlers:
-            return logger
-
-        # 创建格式化器
-        formatter = logging.Formatter(
-            '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
-            datefmt='%Y-%m-%d %H:%M:%S'
-        )
-
-        # 控制台处理器
-        console_handler = logging.StreamHandler(sys.stdout)
-        console_handler.setLevel(getattr(logging, self._config.log_level.value))
-
-        # 使用彩色格式化器（如果支持）
-        if sys.platform != 'win32' and 'TERM' in os.environ:
-            colored_formatter = ColoredFormatter(
-                '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
-                datefmt='%Y-%m-%d %H:%M:%S'
-            )
-            console_handler.setFormatter(colored_formatter)
-        else:
-            console_handler.setFormatter(formatter)
-
-        logger.addHandler(console_handler)
-
-        # 文件处理器（如果配置了日志文件）
-        if self._config.log_file:
-            log_dir = Path(self._config.log_file).parent
-            log_dir.mkdir(parents=True, exist_ok=True)
-
-            file_handler = logging.handlers.RotatingFileHandler(
-                self._config.log_file,
-                maxBytes=self._config.log_max_size,
-                backupCount=self._config.log_backup_count,
-                encoding='utf-8'
-            )
-            file_handler.setLevel(getattr(logging, self._config.log_level.value))
-            file_handler.setFormatter(formatter)
-            logger.addHandler(file_handler)
-
-        self._loggers[name] = logger
-        return logger
-
-    def _get_default_config(self):
-        """获取默认配置"""
-        from .config_manager import LogLevel
-
-        class DefaultConfig:
-            def __init__(self):
-                self.log_level = LogLevel.INFO
-                self.log_file = "logs/gx_gp_monitor.log"
-                self.log_max_size = 10485760  # 10MB
-                self.log_backup_count = 5
-
-        return DefaultConfig()
-
-    def get_logger(self, name: str = "gx_gp_monitor") -> logging.Logger:
-        """
-        获取日志器
-
-        Args:
-            name: 日志器名称
-
-        Returns:
-            logging.Logger: 日志器实例
-        """
-        if name not in self._loggers:
-            return self.init_logger(name)
-        return self._loggers[name]
-
-    def log_crawl_start(self, source_name: str, logger: Optional[logging.Logger] = None):
-        """记录爬取开始"""
-        if logger is None:
-            logger = self.get_logger()
-        logger.info(f"开始爬取 {source_name}")
-
-    def log_crawl_success(self, source_name: str, count: int, duration: float,
-                         logger: Optional[logging.Logger] = None):
-        """记录爬取成功"""
-        if logger is None:
-            logger = self.get_logger()
-        logger.info(f"{source_name} 爬取完成，共获取 {count} 条公告，耗时 {duration:.2f}秒")
-
-    def log_crawl_error(self, source_name: str, error: str,
-                       logger: Optional[logging.Logger] = None):
-        """记录爬取错误"""
-        if logger is None:
-            logger = self.get_logger()
-        logger.error(f"{source_name} 爬取失败: {error}")
-
-    def log_announcement_filtered(self, reason: str, count: int,
-                                 logger: Optional[logging.Logger] = None):
-        """记录公告筛选信息"""
-        if logger is None:
-            logger = self.get_logger()
-        logger.info(f"公告筛选 - {reason}: {count} 条")
-
-    def log_database_operation(self, operation: str, table: str, count: int = 0,
-                              logger: Optional[logging.Logger] = None):
-        """记录数据库操作"""
-        if logger is None:
-            logger = self.get_logger()
-        if count > 0:
-            logger.info(f"数据库操作 - {operation} {table}: {count} 条记录")
-        else:
-            logger.info(f"数据库操作 - {operation} {table}")
-
-    def log_notification_sent(self, channel: str, recipient_count: int,
-                             logger: Optional[logging.Logger] = None):
-        """记录通知发送"""
-        if logger is None:
-            logger = self.get_logger()
-        logger.info(f"通知发送 - {channel}: 向 {recipient_count} 个接收者发送")
-
-    def log_system_metrics(self, metrics: Dict[str, Any],
-                          logger: Optional[logging.Logger] = None):
-        """记录系统指标"""
-        if logger is None:
-            logger = self.get_logger()
-        metrics_str = ", ".join([f"{k}={v}" for k, v in metrics.items()])
-        logger.info(f"系统指标: {metrics_str}")
-
-    def log_performance_warning(self, operation: str, duration: float, threshold: float,
-                               logger: Optional[logging.Logger] = None):
-        """记录性能警告"""
-        if logger is None:
-            logger = self.get_logger()
-        logger.warning(f"性能警告 - {operation} 耗时 {duration:.2f}秒，超过阈值 {threshold:.2f}秒")
-
-
-# 全局日志管理器实例
-_logger_manager = Logger()
-
-
-def get_logger(name: str = "gx_gp_monitor") -> logging.Logger:
-    """
-    获取日志器
-
-    Args:
-        name: 日志器名称
-
-    Returns:
-        logging.Logger: 日志器实例
-    """
-    return _logger_manager.get_logger(name)
-
-
-def init_logger(name: str = "gx_gp_monitor", config=None) -> logging.Logger:
-    """
-    初始化并获取日志器
-
-    Args:
-        name: 日志器名称
-        config: 配置对象
-
-    Returns:
-        logging.Logger: 日志器实例
-    """
-    return _logger_manager.init_logger(name, config)
-
-
-def log_function_call(func_name: str, args: Optional[Dict[str, Any]] = None,
-                     logger: Optional[logging.Logger] = None):
-    """
-    装饰器：记录函数调用
-
-    Args:
-        func_name: 函数名称
-        args: 函数参数
-        logger: 日志器实例
-    """
-    def decorator(func):
-        def wrapper(*args, **kwargs):
-            nonlocal logger
-            if logger is None:
-                logger = get_logger()
-
-            start_time = datetime.now()
-            logger.debug(f"调用函数: {func_name}")
-
-            try:
-                result = func(*args, **kwargs)
-                duration = (datetime.now() - start_time).total_seconds()
-                logger.debug(f"函数 {func_name} 执行完成，耗时 {duration:.3f}秒")
-                return result
-            except Exception as e:
-                duration = (datetime.now() - start_time).total_seconds()
-                logger.error(f"函数 {func_name} 执行失败，耗时 {duration:.3f}秒: {str(e)}")
-                raise
-
-        return wrapper
-    return decorator
-
-
-# 便捷函数
-def log_info(message: str, logger: Optional[logging.Logger] = None):
-    """记录信息日志"""
-    if logger is None:
-        logger = get_logger()
-    logger.info(message)
-
-
-def log_warning(message: str, logger: Optional[logging.Logger] = None):
-    """记录警告日志"""
-    if logger is None:
-        logger = get_logger()
-    logger.warning(message)
-
-
-def log_error(message: str, logger: Optional[logging.Logger] = None):
-    """记录错误日志"""
-    if logger is None:
-        logger = get_logger()
-    logger.error(message)
-
-
-def log_debug(message: str, logger: Optional[logging.Logger] = None):
-    """记录调试日志"""
-    if logger is None:
-        logger = get_logger()
-    logger.debug(message)
-
-
-# 便捷的爬取日志记录函数
-def log_crawl_start(source_name: str):
-    """记录爬取开始"""
-    _logger_manager.log_crawl_start(source_name)
-
-
-def log_crawl_success(source_name: str, count: int, duration: float):
-    """记录爬取成功"""
-    _logger_manager.log_crawl_success(source_name, count, duration)
-
-
-def log_crawl_error(source_name: str, error: str):
-    """记录爬取错误"""
-    _logger_manager.log_crawl_error(source_name, error)
-
-
-def log_announcement_filtered(reason: str, count: int):
-    """记录公告筛选信息"""
-    _logger_manager.log_announcement_filtered(reason, count)
-
-
-def log_database_operation(operation: str, table: str, count: int = 0):
-    """记录数据库操作"""
-    _logger_manager.log_database_operation(operation, table, count)
-
-
-def log_notification_sent(channel: str, recipient_count: int):
-    """记录通知发送"""
-    _logger_manager.log_notification_sent(channel, recipient_count)
-
-
-def log_system_metrics(metrics: Dict[str, Any]):
-    """记录系统指标"""
-    _logger_manager.log_system_metrics(metrics)
-
-
-def log_performance_warning(operation: str, duration: float, threshold: float):
-    """记录性能警告"""
-    _logger_manager.log_performance_warning(operation, duration, threshold)

gx_gp_monitor/core/models.py

@@ -1,255 +0,0 @@
-"""
-数据模型定义
-定义系统使用的数据结构和模型
-"""
-
-from dataclasses import dataclass, field
-from datetime import datetime
-from typing import Optional, List, Dict, Any
-from enum import Enum
-
-
-class AnnouncementType(Enum):
-    """公告类型枚举"""
-    PURCHASE = "purchase"                    # 采购公告
-    RESULT = "result"                        # 结果公告
-    CONTRACT = "contract"                    # 合同公告
-    CORRECTION = "correction"                # 更正公告
-    PRE_ANNOUNCEMENT = "pre_announcement"    # 招标文件预公示
-    SINGLE_SOURCE = "single_source"          # 单一来源公示
-    ELECTRONIC_MARKET = "electronic_market"  # 电子卖场公示
-    ACCEPTANCE = "acceptance"                # 履约验收公示
-    ENGINEERING = "engineering"              # 工程类公告
-    FRAMEWORK_AGREEMENT = "framework_agreement"  # 框架协议征集公告
-    FRAMEWORK_RESULT = "framework_result"    # 框架协议入围结果公告
-    FRAMEWORK_SUMMARY = "framework_summary"  # 框架协议成交结果汇总公告
-    INTENTION = "intention"                  # 采购意向公开
-
-
-class CrawlStatus(Enum):
-    """爬取状态枚举"""
-    PENDING = "pending"      # 待爬取
-    RUNNING = "running"      # 爬取中
-    SUCCESS = "success"      # 成功
-    FAILED = "failed"        # 失败
-    PARTIAL = "partial"      # 部分成功
-
-
-@dataclass
-class AnnouncementSource:
-    """公告来源"""
-    code: str                    # 来源代码，如 "ZcyAnnouncement1"
-    category_id: int            # 分类ID
-    name: str                   # 显示名称，如 "采购公告"
-    type: AnnouncementType      # 公告类型
-
-
-@dataclass
-class Announcement:
-    """公告数据模型"""
-    id: Optional[int] = None                          # 数据库ID
-    title: str = ""                                   # 公告标题
-    publish_date: datetime = field(default_factory=datetime.now)  # 发布时间
-    purchase_name: str = ""                           # 发布单位
-    content_url: str = ""                            # 内容链接
-    source_code: str = ""                            # 来源代码
-    source_name: str = ""                            # 来源名称
-    announcement_type: AnnouncementType = AnnouncementType.PURCHASE  # 公告类型
-
-    # 爬取相关字段
-    crawled_at: Optional[datetime] = None            # 爬取时间
-    created_at: Optional[datetime] = None            # 创建时间
-    updated_at: Optional[datetime] = None            # 更新时间
-
-    # 去重字段
-    content_hash: Optional[str] = None               # 内容哈希，用于去重
-
-    # 筛选相关
-    keyword_matched: bool = False                    # 是否匹配关键词
-    date_filtered: bool = True                       # 是否在日期范围内
-
-    # 业务字段
-    is_new: bool = True                              # 是否为新公告
-    is_today: bool = False                           # 是否为今日公告
-
-    def __post_init__(self):
-        """后初始化处理"""
-        if isinstance(self.announcement_type, str):
-            self.announcement_type = AnnouncementType(self.announcement_type)
-
-        if self.publish_date and isinstance(self.publish_date, str):
-            try:
-                self.publish_date = datetime.fromisoformat(self.publish_date.replace('Z', '+00:00'))
-            except ValueError:
-                # 如果解析失败，使用当前时间
-                self.publish_date = datetime.now()
-
-        # 判断是否为今日公告
-        today = datetime.now().date()
-        if self.publish_date:
-            self.is_today = self.publish_date.date() == today
-
-    @property
-    def publish_date_str(self) -> str:
-        """获取发布日期字符串"""
-        return self.publish_date.strftime("%Y-%m-%d") if self.publish_date else ""
-
-    @property
-    def crawled_at_str(self) -> str:
-        """获取爬取时间字符串"""
-        return self.crawled_at.strftime("%Y-%m-%d %H:%M:%S") if self.crawled_at else ""
-
-    def to_dict(self) -> Dict[str, Any]:
-        """转换为字典"""
-        return {
-            "id": self.id,
-            "title": self.title,
-            "publish_date": self.publish_date.isoformat() if self.publish_date else None,
-            "purchase_name": self.purchase_name,
-            "content_url": self.content_url,
-            "source_code": self.source_code,
-            "source_name": self.source_name,
-            "announcement_type": self.announcement_type.value,
-            "crawled_at": self.crawled_at.isoformat() if self.crawled_at else None,
-            "created_at": self.created_at.isoformat() if self.created_at else None,
-            "updated_at": self.updated_at.isoformat() if self.updated_at else None,
-            "content_hash": self.content_hash,
-            "keyword_matched": self.keyword_matched,
-            "date_filtered": self.date_filtered,
-            "is_new": self.is_new,
-            "is_today": self.is_today
-        }
-
-    @classmethod
-    def from_dict(cls, data: Dict[str, Any]) -> 'Announcement':
-        """从字典创建实例"""
-        # 处理枚举类型
-        if 'announcement_type' in data and isinstance(data['announcement_type'], str):
-            data['announcement_type'] = AnnouncementType(data['announcement_type'])
-
-        # 处理日期时间
-        for date_field in ['publish_date', 'crawled_at', 'created_at', 'updated_at']:
-            if date_field in data and data[date_field] and isinstance(data[date_field], str):
-                try:
-                    data[date_field] = datetime.fromisoformat(data[date_field].replace('Z', '+00:00'))
-                except ValueError:
-                    data[date_field] = None
-
-        return cls(**data)
-
-    def generate_content_hash(self) -> str:
-        """生成内容哈希用于去重"""
-        import hashlib
-        content = f"{self.title}|{self.publish_date_str}|{self.purchase_name}|{self.content_url}|{self.source_code}"
-        self.content_hash = hashlib.md5(content.encode('utf-8')).hexdigest()
-        return self.content_hash
-
-    def matches_keywords(self, keywords: List[str]) -> bool:
-        """检查是否匹配关键词"""
-        if not keywords:
-            return True
-
-        search_text = f"{self.title} {self.purchase_name}".lower()
-        for keyword in keywords:
-            if keyword.lower() in search_text:
-                self.keyword_matched = True
-                return True
-
-        self.keyword_matched = False
-        return False
-
-    def in_date_range(self, start_date: Optional[str], end_date: Optional[str]) -> bool:
-        """检查是否在日期范围内"""
-        if not self.publish_date:
-            self.date_filtered = False
-            return False
-
-        publish_date = self.publish_date.date()
-
-        try:
-            if start_date:
-                start = datetime.fromisoformat(start_date).date()
-                if publish_date < start:
-                    self.date_filtered = False
-                    return False
-
-            if end_date:
-                end = datetime.fromisoformat(end_date).date()
-                if publish_date > end:
-                    self.date_filtered = False
-                    return False
-
-            self.date_filtered = True
-            return True
-        except ValueError:
-            # 日期格式错误时，默认通过
-            self.date_filtered = True
-            return True
-
-
-@dataclass
-class CrawlResult:
-    """爬取结果"""
-    source: AnnouncementSource               # 公告来源
-    status: CrawlStatus                      # 爬取状态
-    total_count: int = 0                     # 总公告数
-    new_count: int = 0                       # 新增公告数
-    error_message: Optional[str] = None      # 错误信息
-    announcements: List[Announcement] = field(default_factory=list)  # 公告列表
-    crawled_at: datetime = field(default_factory=datetime.now)       # 爬取时间
-    duration: float = 0.0                    # 爬取耗时（秒）
-
-
-@dataclass
-class CrawlSession:
-    """爬取会话"""
-    session_id: str                          # 会话ID
-    start_time: datetime                     # 开始时间
-    end_time: Optional[datetime] = None      # 结束时间
-    status: CrawlStatus = CrawlStatus.PENDING  # 会话状态
-    total_sources: int = 0                   # 总来源数
-    completed_sources: int = 0               # 已完成来源数
-    total_announcements: int = 0             # 总公告数
-    new_announcements: int = 0               # 新增公告数
-    results: List[CrawlResult] = field(default_factory=list)  # 各来源结果
-
-    @property
-    def duration(self) -> float:
-        """获取会话持续时间"""
-        if self.end_time and self.start_time:
-            return (self.end_time - self.start_time).total_seconds()
-        elif self.start_time:
-            return (datetime.now() - self.start_time).total_seconds()
-        return 0.0
-
-    @property
-    def progress(self) -> float:
-        """获取完成进度（0-1）"""
-        if self.total_sources == 0:
-            return 0.0
-        return self.completed_sources / self.total_sources
-
-
-@dataclass
-class NotificationMessage:
-    """通知消息"""
-    title: str                               # 消息标题
-    content: str                             # 消息内容
-    message_type: str = "text"               # 消息类型：text, markdown, card
-    recipients: List[str] = field(default_factory=lambda: ["@all"])  # 接收者列表
-    attachments: Optional[Dict[str, Any]] = None  # 附件信息
-    created_at: datetime = field(default_factory=datetime.now)  # 创建时间
-
-
-@dataclass
-class SystemMetrics:
-    """系统指标"""
-    timestamp: datetime = field(default_factory=datetime.now)  # 时间戳
-    total_announcements: int = 0               # 总公告数
-    today_announcements: int = 0               # 今日公告数
-    new_announcements_today: int = 0           # 今日新增公告数
-    crawl_sessions_today: int = 0              # 今日爬取会话数
-    last_crawl_duration: float = 0.0           # 最后一次爬取耗时
-    database_size: int = 0                     # 数据库大小（字节）
-    memory_usage: float = 0.0                  # 内存使用率
-    disk_usage: float = 0.0                    # 磁盘使用率

gx_gp_monitor/core/reliability.py

@@ -1,489 +0,0 @@
-"""
-高可用性模块
-提供重试机制、超时控制、幂等操作、异常处理和恢复功能
-"""
-
-import time
-import random
-import hashlib
-from contextlib import contextmanager
-from functools import wraps
-from typing import Callable, Any, Optional, Type, Union, List
-from datetime import datetime, timedelta
-import threading
-import requests
-from requests.adapters import HTTPAdapter
-from urllib3.util.retry import Retry
-
-from .logger import get_logger
-from .config_manager import get_config
-
-
-logger = get_logger(__name__)
-
-
-class RetryConfig:
-    """重试配置"""
-
-    def __init__(self,
-                 max_retries: int = 3,
-                 initial_delay: float = 1.0,
-                 max_delay: float = 60.0,
-                 backoff_factor: float = 2.0,
-                 jitter: bool = True):
-        """
-        初始化重试配置
-
-        Args:
-            max_retries: 最大重试次数
-            initial_delay: 初始延迟时间（秒）
-            max_delay: 最大延迟时间（秒）
-            backoff_factor: 退避因子
-            jitter: 是否添加随机抖动
-        """
-        self.max_retries = max_retries
-        self.initial_delay = initial_delay
-        self.max_delay = max_delay
-        self.backoff_factor = backoff_factor
-        self.jitter = jitter
-
-
-class TimeoutConfig:
-    """超时配置"""
-
-    def __init__(self,
-                 connect_timeout: float = 10.0,
-                 read_timeout: float = 30.0,
-                 total_timeout: Optional[float] = None):
-        """
-        初始化超时配置
-
-        Args:
-            connect_timeout: 连接超时时间（秒）
-            read_timeout: 读取超时时间（秒）
-            total_timeout: 总超时时间（秒）
-        """
-        self.connect_timeout = connect_timeout
-        self.read_timeout = read_timeout
-        self.total_timeout = total_timeout or (connect_timeout + read_timeout)
-
-
-class CircuitBreakerState:
-    """熔断器状态"""
-    CLOSED = "closed"       # 关闭状态，正常工作
-    OPEN = "open"           # 打开状态，快速失败
-    HALF_OPEN = "half_open" # 半开状态，测试恢复
-
-
-class CircuitBreaker:
-    """熔断器实现"""
-
-    def __init__(self,
-                 failure_threshold: int = 5,
-                 recovery_timeout: int = 60,
-                 expected_exception: Type[Exception] = Exception):
-        """
-        初始化熔断器
-
-        Args:
-            failure_threshold: 失败阈值
-            recovery_timeout: 恢复超时时间（秒）
-            expected_exception: 期望的异常类型
-        """
-        self.failure_threshold = failure_threshold
-        self.recovery_timeout = recovery_timeout
-        self.expected_exception = expected_exception
-
-        self.state = CircuitBreakerState.CLOSED
-        self.failure_count = 0
-        self.last_failure_time = None
-        self._lock = threading.Lock()
-
-    def __call__(self, func: Callable) -> Callable:
-        """装饰器实现"""
-        @wraps(func)
-        def wrapper(*args, **kwargs):
-            return self._execute_with_circuit_breaker(func, *args, **kwargs)
-        return wrapper
-
-    def _execute_with_circuit_breaker(self, func: Callable, *args, **kwargs) -> Any:
-        """使用熔断器执行函数"""
-        if self.state == CircuitBreakerState.OPEN:
-            if self._should_attempt_reset():
-                self.state = CircuitBreakerState.HALF_OPEN
-                logger.info("熔断器半开，尝试恢复")
-            else:
-                raise CircuitBreakerOpenException("熔断器已打开")
-
-        try:
-            result = func(*args, **kwargs)
-            self._on_success()
-            return result
-        except self.expected_exception as e:
-            self._on_failure()
-            raise
-
-    def _should_attempt_reset(self) -> bool:
-        """检查是否应该尝试重置"""
-        if self.last_failure_time is None:
-            return True
-        return (datetime.now() - self.last_failure_time).total_seconds() >= self.recovery_timeout
-
-    def _on_success(self):
-        """成功时的处理"""
-        with self._lock:
-            if self.state == CircuitBreakerState.HALF_OPEN:
-                self.state = CircuitBreakerState.CLOSED
-                self.failure_count = 0
-                logger.info("熔断器关闭，服务恢复正常")
-
-    def _on_failure(self):
-        """失败时的处理"""
-        with self._lock:
-            self.failure_count += 1
-            self.last_failure_time = datetime.now()
-
-            if self.failure_count >= self.failure_threshold:
-                self.state = CircuitBreakerState.OPEN
-                logger.warning(f"熔断器打开，失败次数达到阈值: {self.failure_count}")
-
-
-class CircuitBreakerOpenException(Exception):
-    """熔断器打开异常"""
-    pass
-
-
-class IdempotencyKey:
-    """幂等性键生成器"""
-
-    @staticmethod
-    def generate(*args, **kwargs) -> str:
-        """
-        生成幂等性键
-
-        Args:
-            *args: 位置参数
-            **kwargs: 关键字参数
-
-        Returns:
-            str: 幂等性键
-        """
-        # 将参数转换为字符串并排序
-        key_parts = []
-
-        # 处理位置参数
-        for i, arg in enumerate(args):
-            key_parts.append(f"arg_{i}:{str(arg)}")
-
-        # 处理关键字参数（排序以保证一致性）
-        for key in sorted(kwargs.keys()):
-            key_parts.append(f"{key}:{str(kwargs[key])}")
-
-        # 生成哈希
-        key_string = "|".join(key_parts)
-        return hashlib.md5(key_string.encode('utf-8')).hexdigest()
-
-
-class IdempotencyManager:
-    """幂等性管理器"""
-
-    def __init__(self):
-        self._executed_keys = set()
-        self._lock = threading.Lock()
-
-    def is_executed(self, key: str) -> bool:
-        """
-        检查操作是否已执行
-
-        Args:
-            key: 幂等性键
-
-        Returns:
-            bool: 是否已执行
-        """
-        with self._lock:
-            return key in self._executed_keys
-
-    def mark_executed(self, key: str):
-        """
-        标记操作已执行
-
-        Args:
-            key: 幂等性键
-        """
-        with self._lock:
-            self._executed_keys.add(key)
-
-    def clear_expired_keys(self, max_age_seconds: int = 3600):
-        """
-        清理过期的键（简化实现，实际应该使用时间戳）
-
-        Args:
-            max_age_seconds: 最大年龄（秒）
-        """
-        # 这里简化实现，实际项目中应该记录时间戳
-        pass
-
-
-def retry_on_exception(retry_config: Optional[RetryConfig] = None,
-                      exceptions: tuple = (Exception,),
-                      logger: Optional[Any] = None) -> Callable:
-    """
-    重试装饰器
-
-    Args:
-        retry_config: 重试配置
-        exceptions: 需要重试的异常类型
-        logger: 日志器
-
-    Returns:
-        Callable: 装饰器函数
-    """
-    if retry_config is None:
-        retry_config = RetryConfig()
-
-    if logger is None:
-        logger = get_logger()
-
-    def decorator(func: Callable) -> Callable:
-        @wraps(func)
-        def wrapper(*args, **kwargs):
-            last_exception = None
-
-            for attempt in range(retry_config.max_retries + 1):
-                try:
-                    return func(*args, **kwargs)
-                except exceptions as e:
-                    last_exception = e
-
-                    if attempt < retry_config.max_retries:
-                        # 计算延迟时间
-                        delay = min(
-                            retry_config.initial_delay * (retry_config.backoff_factor ** attempt),
-                            retry_config.max_delay
-                        )
-
-                        # 添加随机抖动
-                        if retry_config.jitter:
-                            delay = delay * (0.5 + random.random() * 0.5)
-
-                        logger.warning(
-                            f"函数 {func.__name__} 执行失败 (尝试 {attempt + 1}/{retry_config.max_retries + 1}): {str(e)}，"
-                            f"等待 {delay:.2f} 秒后重试"
-                        )
-                        time.sleep(delay)
-                    else:
-                        logger.error(
-                            f"函数 {func.__name__} 在 {retry_config.max_retries + 1} 次尝试后仍然失败: {str(e)}"
-                        )
-
-            raise last_exception
-
-        return wrapper
-    return decorator
-
-
-def timeout_wrapper(timeout_config: Optional[TimeoutConfig] = None) -> Callable:
-    """
-    超时装饰器
-
-    Args:
-        timeout_config: 超时配置
-
-    Returns:
-        Callable: 装饰器函数
-    """
-    if timeout_config is None:
-        timeout_config = TimeoutConfig()
-
-    def decorator(func: Callable) -> Callable:
-        @wraps(func)
-        def wrapper(*args, **kwargs):
-            import signal
-
-            def timeout_handler(signum, frame):
-                raise TimeoutError(f"函数 {func.__name__} 执行超时")
-
-            # 设置信号处理器
-            old_handler = signal.signal(signal.SIGALRM, timeout_handler)
-            signal.alarm(int(timeout_config.total_timeout))
-
-            try:
-                result = func(*args, **kwargs)
-                signal.alarm(0)  # 取消闹钟
-                return result
-            finally:
-                signal.signal(signal.SIGALRM, old_handler)
-
-        return wrapper
-    return decorator
-
-
-@contextmanager
-def session_with_retry(timeout_config: Optional[TimeoutConfig] = None,
-                      retry_config: Optional[RetryConfig] = None):
-    """
-    创建带有重试机制的HTTP会话
-
-    Args:
-        timeout_config: 超时配置
-        retry_config: 重试配置
-
-    Yields:
-        requests.Session: 配置好的会话对象
-    """
-    if timeout_config is None:
-        timeout_config = TimeoutConfig()
-
-    if retry_config is None:
-        retry_config = RetryConfig()
-
-    session = requests.Session()
-
-    # 配置重试策略
-    retry_strategy = Retry(
-        total=retry_config.max_retries,
-        backoff_factor=retry_config.backoff_factor,
-        status_forcelist=[429, 500, 502, 503, 504],
-    )
-
-    adapter = HTTPAdapter(max_retries=retry_strategy)
-    session.mount("http://", adapter)
-    session.mount("https://", adapter)
-
-    # 设置默认超时
-    session.timeout = (timeout_config.connect_timeout, timeout_config.read_timeout)
-
-    try:
-        yield session
-    finally:
-        session.close()
-
-
-def safe_execute(func: Callable,
-                fallback: Optional[Callable] = None,
-                exceptions: tuple = (Exception,),
-                logger: Optional[Any] = None) -> Any:
-    """
-    安全执行函数，提供降级处理
-
-    Args:
-        func: 要执行的函数
-        fallback: 降级函数
-        exceptions: 需要捕获的异常类型
-        logger: 日志器
-
-    Returns:
-        Any: 函数执行结果或降级结果
-    """
-    if logger is None:
-        logger = get_logger()
-
-    try:
-        return func()
-    except exceptions as e:
-        logger.error(f"函数执行失败: {str(e)}")
-        if fallback:
-            try:
-                logger.info("执行降级函数")
-                return fallback()
-            except Exception as fallback_e:
-                logger.error(f"降级函数也执行失败: {str(fallback_e)}")
-        return None
-
-
-class HealthChecker:
-    """健康检查器"""
-
-    def __init__(self, check_interval: int = 300):
-        """
-        初始化健康检查器
-
-        Args:
-            check_interval: 检查间隔（秒）
-        """
-        self.check_interval = check_interval
-        self.last_check = None
-        self.is_healthy = True
-        self.consecutive_failures = 0
-        self.max_consecutive_failures = 3
-
-    def check_health(self) -> bool:
-        """
-        执行健康检查
-
-        Returns:
-            bool: 健康状态
-        """
-        current_time = datetime.now()
-
-        # 检查是否需要执行检查
-        if (self.last_check and
-            (current_time - self.last_check).total_seconds() < self.check_interval):
-            return self.is_healthy
-
-        self.last_check = current_time
-
-        try:
-            # 执行健康检查逻辑
-            self._perform_health_check()
-            self.is_healthy = True
-            self.consecutive_failures = 0
-            logger.info("健康检查通过")
-            return True
-        except Exception as e:
-            self.consecutive_failures += 1
-            logger.warning(f"健康检查失败 ({self.consecutive_failures}/{self.max_consecutive_failures}): {str(e)}")
-
-            if self.consecutive_failures >= self.max_consecutive_failures:
-                self.is_healthy = False
-                logger.error("连续健康检查失败，系统标记为不健康")
-
-            return False
-
-    def _perform_health_check(self):
-        """执行具体的健康检查逻辑"""
-        # 这里可以添加数据库连接检查、外部服务检查等
-        config = get_config()
-
-        # 检查数据库连接（如果启用）
-        if config.database.enabled:
-            # 这里应该检查数据库连接
-            pass
-
-        # 检查网络连接
-        try:
-            requests.get("https://www.baidu.com", timeout=5)
-        except:
-            raise Exception("网络连接检查失败")
-
-
-# 全局实例
-_circuit_breaker = CircuitBreaker()
-_idempotency_manager = IdempotencyManager()
-_health_checker = HealthChecker()
-
-
-def get_circuit_breaker() -> CircuitBreaker:
-    """获取全局熔断器实例"""
-    return _circuit_breaker
-
-
-def get_idempotency_manager() -> IdempotencyManager:
-    """获取全局幂等性管理器实例"""
-    return _idempotency_manager
-
-
-def get_health_checker() -> HealthChecker:
-    """获取全局健康检查器实例"""
-    return _health_checker
-
-
-def check_system_health() -> bool:
-    """
-    检查系统健康状态
-
-    Returns:
-        bool: 系统是否健康
-    """
-    return _health_checker.check_health()

gx_gp_monitor/crawler/__init__.py

@@ -1 +0,0 @@
-"""爬虫模块"""

gx_gp_monitor/crawler/parsers.py

@@ -1,301 +0,0 @@
-"""
-数据解析器模块
-负责解析广西政府采购网的API响应数据
-"""
-
-import json
-from typing import List, Dict, Any, Optional
-from datetime import datetime
-
-try:
-    from ..core.models import Announcement, AnnouncementSource, AnnouncementType
-    from ..core.logger import get_logger
-except ImportError:
-    from core.models import Announcement, AnnouncementSource, AnnouncementType
-    from core.logger import get_logger
-
-
-logger = get_logger(__name__)
-
-
-class AnnouncementParser:
-    """公告数据解析器"""
-
-    @staticmethod
-    def parse_api_response(response_data: Dict[str, Any],
-                          source: AnnouncementSource,
-                          crawled_at: datetime) -> List[Announcement]:
-        """
-        解析API响应数据
-
-        Args:
-            response_data: API响应数据
-            source: 公告来源
-            crawled_at: 爬取时间
-
-        Returns:
-            List[Announcement]: 解析后的公告列表
-        """
-        if not response_data or not isinstance(response_data, dict):
-            logger.warning("API响应数据无效")
-            return []
-
-        try:
-            # 检查响应状态
-            if not response_data.get("success", False):
-                logger.warning(f"API响应失败: {response_data.get('message', '未知错误')}")
-                return []
-
-            # 获取数据部分
-            result = response_data.get("result", {})
-            data = result.get("data", {})
-            records = data.get("data", [])
-
-            if not records:
-                logger.info(f"来源 {source.name} 没有新数据")
-                return []
-
-            announcements = []
-            for record in records:
-                try:
-                    announcement = AnnouncementParser._parse_single_record(
-                        record, source, crawled_at)
-                    if announcement:
-                        announcements.append(announcement)
-                except Exception as e:
-                    logger.warning(f"解析公告记录失败: {str(e)}, 记录: {record}")
-                    continue
-
-            logger.info(f"成功解析 {len(announcements)}/{len(records)} 条公告记录")
-            return announcements
-
-        except Exception as e:
-            logger.error(f"解析API响应数据失败: {str(e)}")
-            return []
-
-    @staticmethod
-    def _parse_single_record(record: Dict[str, Any],
-                           source: AnnouncementSource,
-                           crawled_at: datetime) -> Optional[Announcement]:
-        """
-        解析单个公告记录
-
-        Args:
-            record: 公告记录数据
-            source: 公告来源
-            crawled_at: 爬取时间
-
-        Returns:
-            Optional[Announcement]: 解析后的公告对象
-        """
-        try:
-            # 提取基本字段
-            title_raw = record.get("title", "")
-            title = str(title_raw).strip() if title_raw is not None else ""
-            if not title:
-                return None
-
-            # 解析发布时间
-            publish_timestamp = record.get("publishDate")
-            if not publish_timestamp:
-                logger.warning(f"公告缺少发布时间: {title[:50]}...")
-                return None
-
-            try:
-                # 时间戳转换为datetime
-                publish_date = datetime.fromtimestamp(int(publish_timestamp) / 1000)
-            except (ValueError, TypeError) as e:
-                logger.warning(f"发布时间格式错误: {publish_timestamp}, 错误: {str(e)}")
-                return None
-
-            # 提取其他字段
-            purchase_name_raw = record.get("purchaseName", "")
-            purchase_name = str(purchase_name_raw).strip() if purchase_name_raw is not None else ""
-            article_id = record.get("articleId")
-
-            if not article_id:
-                logger.warning(f"公告缺少文章ID: {title[:50]}...")
-                return None
-
-            # 构建内容链接
-            content_url = AnnouncementParser._build_content_url(
-                source.category_id, source.code, article_id)
-
-            # 创建公告对象
-            announcement = Announcement(
-                title=title,
-                publish_date=publish_date,
-                purchase_name=purchase_name,
-                content_url=content_url,
-                source_code=source.code,
-                source_name=source.name,
-                announcement_type=source.type,
-                crawled_at=crawled_at,
-                is_new=True  # 默认标记为新公告
-            )
-
-            # 生成内容哈希用于去重
-            announcement.generate_content_hash()
-
-            return announcement
-
-        except Exception as e:
-            logger.error(f"解析单个公告记录失败: {str(e)}")
-            return None
-
-    @staticmethod
-    def _build_content_url(category_id: int, source_code: str, article_id: int) -> str:
-        """
-        构建公告内容链接
-
-        Args:
-            category_id: 分类ID
-            source_code: 来源代码
-            article_id: 文章ID
-
-        Returns:
-            str: 内容链接
-        """
-        return f"https://zfcg.gxzf.gov.cn/site/detail?parentId={category_id}&articleId={article_id}"
-
-    @staticmethod
-    def validate_response_structure(response_data: Dict[str, Any]) -> bool:
-        """
-        验证API响应数据结构
-
-        Args:
-            response_data: API响应数据
-
-        Returns:
-            bool: 结构是否有效
-        """
-        try:
-            if not isinstance(response_data, dict):
-                return False
-
-            # 检查必需的字段
-            if "success" not in response_data:
-                return False
-
-            if not response_data.get("success", False):
-                return False
-
-            result = response_data.get("result", {})
-            if not isinstance(result, dict):
-                return False
-
-            data = result.get("data", {})
-            if not isinstance(data, dict):
-                return False
-
-            records = data.get("data", [])
-            if not isinstance(records, list):
-                return False
-
-            return True
-
-        except Exception as e:
-            logger.warning(f"验证响应结构失败: {str(e)}")
-            return False
-
-    @staticmethod
-    def extract_pagination_info(response_data: Dict[str, Any]) -> Dict[str, Any]:
-        """
-        提取分页信息
-
-        Args:
-            response_data: API响应数据
-
-        Returns:
-            Dict[str, Any]: 分页信息
-        """
-        try:
-            result = response_data.get("result", {})
-            data = result.get("data", {})
-
-            return {
-                "total": data.get("total", 0),
-                "page_no": data.get("pageNo", 1),
-                "page_size": data.get("pageSize", 15),
-                "pages": data.get("pages", 0),
-                "empty": data.get("empty", True),
-                "has_next": data.get("hasNext", False),
-                "has_previous": data.get("hasPrevious", False)
-            }
-        except Exception as e:
-            logger.warning(f"提取分页信息失败: {str(e)}")
-            return {
-                "total": 0,
-                "page_no": 1,
-                "page_size": 15,
-                "pages": 0,
-                "empty": True,
-                "has_next": False,
-                "has_previous": False
-            }
-
-
-class SensitiveWordChecker:
-    """敏感词检查器"""
-
-    @staticmethod
-    def parse_check_response(response_data: Dict[str, Any]) -> bool:
-        """
-        解析敏感词检查响应
-
-        Args:
-            response_data: 检查响应数据
-
-        Returns:
-            bool: 检查是否通过
-        """
-        try:
-            if not isinstance(response_data, dict):
-                logger.warning("敏感词检查响应格式无效")
-                return False
-
-            success = response_data.get("success", False)
-            if not success:
-                message = response_data.get("message", "未知错误")
-                logger.warning(f"敏感词检查失败: {message}")
-                return False
-
-            return True
-
-        except Exception as e:
-            logger.error(f"解析敏感词检查响应失败: {str(e)}")
-            return False
-
-
-class ErrorResponseParser:
-    """错误响应解析器"""
-
-    @staticmethod
-    def parse_error(response_data: Dict[str, Any]) -> str:
-        """
-        解析错误响应
-
-        Args:
-            response_data: 错误响应数据
-
-        Returns:
-            str: 错误信息
-        """
-        try:
-            if not isinstance(response_data, dict):
-                return "响应格式无效"
-
-            # 尝试提取错误信息
-            error_msg = response_data.get("message") or response_data.get("msg")
-            if error_msg:
-                return str(error_msg)
-
-            # 检查状态码
-            errcode = response_data.get("errcode")
-            if errcode:
-                return f"错误码: {errcode}"
-
-            return "未知错误"
-
-        except Exception as e:
-            return f"解析错误响应失败: {str(e)}"

gx_gp_monitor/crawler/spider.py

@@ -1,458 +0,0 @@
-"""
-爬虫核心模块
-实现广西政府采购网公告的智能爬取功能
-"""
-
-import time
-import random
-import json
-from typing import List, Dict, Any, Optional, Tuple
-from datetime import datetime
-from urllib.parse import urljoin
-import requests
-from fake_useragent import UserAgent
-
-try:
-    # 尝试相对导入
-    from ..core.models import Announcement, AnnouncementSource, AnnouncementType, CrawlResult, CrawlStatus
-    from ..core.config_manager import get_config
-    from ..core.logger import get_logger, log_crawl_start, log_crawl_success, log_crawl_error
-    from ..core.reliability import (
-        retry_on_exception, RetryConfig, session_with_retry,
-        TimeoutConfig, safe_execute, check_system_health
-    )
-    from .parsers import AnnouncementParser, SensitiveWordChecker, ErrorResponseParser
-except ImportError:
-    # 尝试绝对导入
-    from core.models import Announcement, AnnouncementSource, AnnouncementType, CrawlResult, CrawlStatus
-    from core.config_manager import get_config
-    from core.logger import get_logger, log_crawl_start, log_crawl_success, log_crawl_error
-    from core.reliability import (
-        retry_on_exception, RetryConfig, session_with_retry,
-        TimeoutConfig, safe_execute, check_system_health
-    )
-    from crawler.parsers import AnnouncementParser, SensitiveWordChecker, ErrorResponseParser
-
-
-logger = get_logger(__name__)
-
-
-class GXGPSpider:
-    """广西政府采购网爬虫"""
-
-    def __init__(self):
-        self.config = get_config()
-        self.ua = UserAgent()
-
-        # API端点
-        self.base_url = self.config.crawler.base_url
-        self.announcement_api = urljoin(self.base_url, "/portal/category")
-        self.sensitive_check_api = urljoin(self.base_url, "/portal/sensitiveWords/check")
-
-        # 会话管理
-        self.session = None
-
-        # 统计信息
-        self.request_count = 0
-        self.error_count = 0
-
-    def init_session(self):
-        """初始化会话"""
-        if self.session is None:
-            timeout_config = TimeoutConfig(
-                connect_timeout=self.config.crawler.timeout,
-                read_timeout=self.config.crawler.timeout
-            )
-
-            retry_config = RetryConfig(
-                max_retries=self.config.crawler.max_retries,
-                initial_delay=self.config.crawler.retry_delay,
-                max_delay=self.config.crawler.max_retry_delay,
-                backoff_factor=self.config.crawler.backoff_factor
-            )
-
-            self.session = requests.Session()
-
-            # 配置重试和超时
-            adapter = requests.adapters.HTTPAdapter(
-                pool_connections=10,
-                pool_maxsize=20,
-                max_retries=0  # 我们使用自己的重试逻辑
-            )
-            self.session.mount('http://', adapter)
-            self.session.mount('https://', adapter)
-
-            # 设置默认超时
-            self.session.timeout = (timeout_config.connect_timeout, timeout_config.read_timeout)
-
-        return self.session
-
-    def close_session(self):
-        """关闭会话"""
-        if self.session:
-            self.session.close()
-            self.session = None
-
-    def get_random_user_agent(self) -> str:
-        """获取随机User-Agent"""
-        try:
-            return self.ua.random
-        except:
-            # fallback到配置的user agents
-            return random.choice(self.config.crawler.user_agents)
-
-    def get_random_proxy(self) -> Optional[Dict[str, str]]:
-        """获取随机代理"""
-        if not self.config.crawler.proxies:
-            return None
-
-        proxy = random.choice(self.config.crawler.proxies)
-        return {
-            "http": proxy,
-            "https": proxy
-        }
-
-    def check_sensitive_words(self, payload: Dict[str, Any],
-                            category_code: str, childrencode: str) -> bool:
-        """
-        执行敏感词检查
-
-        Args:
-            payload: 请求参数
-            category_code: 分类代码
-            childrencode: 子分类代码
-
-        Returns:
-            bool: 检查是否通过
-        """
-        try:
-            session = self.init_session()
-            user_agent = self.get_random_user_agent()
-
-            headers = {
-                "User-Agent": user_agent,
-                "Content-Type": "application/json;charset=UTF-8",
-                "Origin": self.base_url,
-                "Referer": f"{self.base_url}/site/category?parentId={category_code}&childrenCode={childrencode}",
-                "Cookie": "_zcy_log_client_uuid=71e283e0-23d2-11f0-844a-eb67dfa7ab64"
-            }
-
-            proxies = self.get_random_proxy()
-
-            logger.debug(f"执行敏感词检查: {category_code}/{childrencode}")
-
-            response = session.post(
-                self.sensitive_check_api,
-                json=payload,
-                headers=headers,
-                proxies=proxies,
-                timeout=self.session.timeout
-            )
-
-            self.request_count += 1
-
-            if response.status_code == 200:
-                result = response.json()
-                return SensitiveWordChecker.parse_check_response(result)
-            else:
-                logger.warning(f"敏感词检查请求失败, 状态码: {response.status_code}")
-                return False
-
-        except Exception as e:
-            logger.error(f"敏感词检查异常: {str(e)}")
-            self.error_count += 1
-            return False
-
-    def fetch_announcements_page(self, source: AnnouncementSource,
-                               page_no: int = 1) -> Tuple[Optional[Dict[str, Any]], Optional[str]]:
-        """
-        获取公告列表页数据
-
-        Args:
-            source: 公告来源
-            page_no: 页码
-
-        Returns:
-            Tuple[Optional[Dict[str, Any]], Optional[str]]: (响应数据, 错误信息)
-        """
-        try:
-            session = self.init_session()
-
-            # 构建请求参数
-            payload = {
-                "keyword": "",  # 关键词筛选，我们在筛选模块处理
-                "publishDateBegin": self.config.crawler.start_date or "",
-                "publishDateEnd": self.config.crawler.end_date or "",
-                "pageNo": page_no,
-                "pageSize": self.config.crawler.page_size,
-                "categoryCode": source.code,
-                "_t": int(time.time() * 1000)
-            }
-
-            # 先执行敏感词检查
-            if not self.check_sensitive_words(payload, str(source.category_id), source.code):
-                return None, "敏感词检查失败"
-
-            # 执行公告数据请求
-            user_agent = self.get_random_user_agent()
-            headers = {
-                "User-Agent": user_agent,
-                "Content-Type": "application/json;charset=UTF-8",
-                "Origin": self.base_url,
-                "Referer": f"{self.base_url}/site/category?parentId={source.category_id}&childrenCode={source.code}",
-                "Cookie": "_zcy_log_client_uuid=71e283e0-23d2-11f0-844a-eb67dfa7ab64"
-            }
-
-            proxies = self.get_random_proxy()
-
-            logger.debug(f"请求公告数据: {source.name} 第{page_no}页")
-
-            # 添加请求间延迟
-            if page_no > 1:
-                delay = random.uniform(
-                    self.config.crawler.request_delay,
-                    self.config.crawler.request_delay_max
-                )
-                time.sleep(delay)
-
-            response = session.post(
-                self.announcement_api,
-                json=payload,
-                headers=headers,
-                proxies=proxies,
-                timeout=self.session.timeout
-            )
-
-            self.request_count += 1
-
-            if response.status_code == 200:
-                data = response.json()
-                if data.get("success", False):
-                    return data, None
-                else:
-                    error_msg = ErrorResponseParser.parse_error(data)
-                    return None, f"API返回失败: {error_msg}"
-            else:
-                return None, f"请求失败, 状态码: {response.status_code}"
-
-        except requests.exceptions.Timeout as e:
-            self.error_count += 1
-            return None, f"请求超时: {str(e)}"
-        except requests.exceptions.ProxyError as e:
-            self.error_count += 1
-            return None, f"代理错误: {str(e)}"
-        except Exception as e:
-            self.error_count += 1
-            logger.error(f"获取公告数据异常: {str(e)}")
-            return None, f"请求异常: {str(e)}"
-
-    def crawl_source(self, source: AnnouncementSource,
-                    max_pages: Optional[int] = None) -> CrawlResult:
-        """
-        爬取单个来源的公告
-
-        Args:
-            source: 公告来源
-            max_pages: 最大页数限制
-
-        Returns:
-            CrawlResult: 爬取结果
-        """
-        if max_pages is None:
-            max_pages = self.config.crawler.max_pages
-
-        log_crawl_start(source.name)
-
-        start_time = datetime.now()
-        result = CrawlResult(
-            source=source,
-            status=CrawlStatus.RUNNING,
-            crawled_at=start_time
-        )
-
-        try:
-            page = 1
-            all_announcements = []
-
-            while page <= max_pages:
-                # 获取页面数据
-                response_data, error_msg = self.fetch_announcements_page(source, page)
-
-                if error_msg:
-                    logger.warning(f"{source.name} 第{page}页获取失败: {error_msg}")
-                    result.status = CrawlStatus.FAILED
-                    result.error_message = error_msg
-                    break
-
-                if not response_data:
-                    logger.info(f"{source.name} 第{page}页无数据")
-                    break
-
-                # 解析分页信息
-                pagination = AnnouncementParser.extract_pagination_info(response_data)
-                result.total_count = pagination["total"]
-
-                # 解析公告数据
-                announcements = AnnouncementParser.parse_api_response(
-                    response_data, source, start_time)
-
-                if not announcements:
-                    logger.info(f"{source.name} 第{page}页解析到0条公告")
-                    break
-
-                all_announcements.extend(announcements)
-
-                # 检查是否还有下一页
-                if not pagination["has_next"] or pagination["empty"]:
-                    break
-
-                page += 1
-
-            # 更新结果
-            result.announcements = all_announcements
-            result.new_count = len(all_announcements)  # 这里的新增数需要在筛选后确定
-
-            if result.status != CrawlStatus.FAILED:
-                result.status = CrawlStatus.SUCCESS
-
-            duration = (datetime.now() - start_time).total_seconds()
-            result.duration = duration
-
-            log_crawl_success(source.name, len(all_announcements), duration)
-
-        except Exception as e:
-            duration = (datetime.now() - start_time).total_seconds()
-            result.duration = duration
-            result.status = CrawlStatus.FAILED
-            result.error_message = str(e)
-
-            log_crawl_error(source.name, str(e))
-
-        return result
-
-    @retry_on_exception(RetryConfig(max_retries=2))
-    def crawl_all_sources(self, sources: Optional[List[AnnouncementSource]] = None) -> List[CrawlResult]:
-        """
-        爬取所有来源的公告
-
-        Args:
-            sources: 指定的来源列表，如果为None则使用配置中的所有来源
-
-        Returns:
-            List[CrawlResult]: 所有来源的爬取结果
-        """
-        # 系统健康检查
-        if not check_system_health():
-            logger.error("系统健康检查失败，跳过爬取")
-            return []
-
-        if sources is None:
-            sources = self._load_sources_from_config()
-
-        logger.info(f"开始爬取 {len(sources)} 个公告来源")
-
-        results = []
-
-        for source in sources:
-            try:
-                result = self.crawl_source(source)
-                results.append(result)
-
-                # 检查是否需要暂停
-                if result.status == CrawlStatus.FAILED:
-                    logger.warning(f"来源 {source.name} 爬取失败，继续下一个来源")
-                    continue
-
-            except Exception as e:
-                logger.error(f"爬取来源 {source.name} 时发生未预期错误: {str(e)}")
-                # 创建失败结果
-                failed_result = CrawlResult(
-                    source=source,
-                    status=CrawlStatus.FAILED,
-                    error_message=str(e),
-                    crawled_at=datetime.now()
-                )
-                results.append(failed_result)
-
-        # 统计总结果
-        total_announcements = sum(len(r.announcements) for r in results if r.status == CrawlStatus.SUCCESS)
-        success_count = sum(1 for r in results if r.status == CrawlStatus.SUCCESS)
-        failed_count = len(results) - success_count
-
-        logger.info(
-            f"爬取完成: 共处理 {len(results)} 个来源，"
-            f"成功 {success_count} 个，失败 {failed_count} 个，"
-            f"获取 {total_announcements} 条公告"
-        )
-
-        return results
-
-    def _load_sources_from_config(self) -> List[AnnouncementSource]:
-        """从配置加载公告来源"""
-        sources = []
-
-        for code, source_config in self.config.sources.items():
-            try:
-                source = AnnouncementSource(
-                    code=code,
-                    category_id=source_config["category_id"],
-                    name=source_config["name"],
-                    type=AnnouncementType(source_config["type"])
-                )
-                sources.append(source)
-            except Exception as e:
-                logger.warning(f"加载来源配置失败 {code}: {str(e)}")
-                continue
-
-        return sources
-
-    def get_stats(self) -> Dict[str, Any]:
-        """获取爬虫统计信息"""
-        return {
-            "request_count": self.request_count,
-            "error_count": self.error_count,
-            "error_rate": self.error_count / max(self.request_count, 1),
-            "session_active": self.session is not None
-        }
-
-    def reset_stats(self):
-        """重置统计信息"""
-        self.request_count = 0
-        self.error_count = 0
-
-
-def create_spider() -> GXGPSpider:
-    """
-    创建爬虫实例
-
-    Returns:
-        GXGPSpider: 爬虫实例
-    """
-    return GXGPSpider()
-
-
-def crawl_announcements(keywords: Optional[List[str]] = None,
-                       sources: Optional[List[str]] = None) -> List[CrawlResult]:
-    """
-    便捷函数：爬取公告
-
-    Args:
-        keywords: 关键词过滤（暂时未使用，在筛选模块处理）
-        sources: 来源代码列表
-
-    Returns:
-        List[CrawlResult]: 爬取结果
-    """
-    spider = create_spider()
-
-    try:
-        # 过滤来源
-        if sources:
-            all_sources = spider._load_sources_from_config()
-            filtered_sources = [s for s in all_sources if s.code in sources]
-        else:
-            filtered_sources = None
-
-        return spider.crawl_all_sources(filtered_sources)
-    finally:
-        spider.close_session()

gx_gp_monitor/filters/__init__.py

@@ -1 +0,0 @@
-"""筛选模块"""

gx_gp_monitor/filters/filters.py

@@ -1,471 +0,0 @@
-"""
-智能筛选模块
-提供关键词过滤、日期范围筛选、自动去重等功能
-"""
-
-from typing import List, Optional, Dict, Any, Set
-from datetime import datetime, date
-from dataclasses import dataclass
-import re
-
-try:
-    from ..core.models import Announcement
-    from ..core.logger import get_logger
-    from ..core.database import get_database_manager
-except ImportError:
-    from core.models import Announcement
-    from core.logger import get_logger
-    from core.database import get_database_manager
-
-
-logger = get_logger(__name__)
-
-
-@dataclass
-class FilterCriteria:
-    """筛选条件"""
-    keywords: List[str] = None              # 关键词列表
-    start_date: Optional[date] = None       # 开始日期
-    end_date: Optional[date] = None         # 结束日期
-    sources: List[str] = None               # 来源代码列表
-    exclude_today: bool = False             # 排除今日公告
-    case_sensitive: bool = False            # 关键词匹配是否区分大小写
-    fuzzy_match: bool = False               # 是否启用模糊匹配
-
-    def __post_init__(self):
-        if self.keywords is None:
-            self.keywords = []
-        if self.sources is None:
-            self.sources = []
-
-
-@dataclass
-class FilterResult:
-    """筛选结果"""
-    total_count: int = 0                    # 总数
-    filtered_count: int = 0                 # 筛选后数量
-    keyword_filtered: int = 0               # 关键词筛选数量
-    date_filtered: int = 0                  # 日期筛选数量
-    duplicate_filtered: int = 0             # 去重筛选数量
-    source_filtered: int = 0                # 来源筛选数量
-    today_excluded: int = 0                 # 排除今日数量
-
-
-class KeywordFilter:
-    """关键词筛选器"""
-
-    def __init__(self, case_sensitive: bool = False, fuzzy_match: bool = False):
-        """
-        初始化关键词筛选器
-
-        Args:
-            case_sensitive: 是否区分大小写
-            fuzzy_match: 是否启用模糊匹配
-        """
-        self.case_sensitive = case_sensitive
-        self.fuzzy_match = fuzzy_match
-        self._compiled_patterns = {}
-
-    def _compile_patterns(self, keywords: List[str]):
-        """编译关键词模式"""
-        flags = 0 if self.case_sensitive else re.IGNORECASE
-
-        for keyword in keywords:
-            if keyword not in self._compiled_patterns:
-                if self.fuzzy_match:
-                    # 简单的模糊匹配：将关键词中的空格替换为灵活匹配
-                    pattern = r'.*'.join(re.escape(word) for word in keyword.split())
-                else:
-                    pattern = re.escape(keyword)
-
-                self._compiled_patterns[keyword] = re.compile(pattern, flags)
-
-    def matches(self, text: str, keywords: List[str]) -> bool:
-        """
-        检查文本是否匹配关键词
-
-        Args:
-            text: 要检查的文本
-            keywords: 关键词列表
-
-        Returns:
-            bool: 是否匹配
-        """
-        if not keywords or not text:
-            return True  # 没有关键词或文本时，认为匹配
-
-        if not self.fuzzy_match:
-            # 精确匹配
-            search_text = text if self.case_sensitive else text.lower()
-            search_keywords = keywords if self.case_sensitive else [k.lower() for k in keywords]
-
-            for keyword in search_keywords:
-                if keyword in search_text:
-                    return True
-            return False
-        else:
-            # 正则匹配
-            self._compile_patterns(keywords)
-            for keyword in keywords:
-                pattern = self._compiled_patterns[keyword]
-                if pattern.search(text):
-                    return True
-            return False
-
-    def filter_announcements(self, announcements: List[Announcement],
-                           keywords: List[str]) -> List[Announcement]:
-        """
-        筛选匹配关键词的公告
-
-        Args:
-            announcements: 公告列表
-            keywords: 关键词列表
-
-        Returns:
-            List[Announcement]: 筛选后的公告列表
-        """
-        if not keywords:
-            # 没有关键词时，所有公告都匹配
-            for announcement in announcements:
-                announcement.keyword_matched = True
-            return announcements
-
-        filtered = []
-        for announcement in announcements:
-            # 构建搜索文本
-            search_text = f"{announcement.title} {announcement.purchase_name}"
-
-            if self.matches(search_text, keywords):
-                announcement.keyword_matched = True
-                filtered.append(announcement)
-            else:
-                announcement.keyword_matched = False
-
-        logger.info(f"关键词筛选: {len(announcements)} -> {len(filtered)} 条公告")
-        return filtered
-
-
-class DateFilter:
-    """日期筛选器"""
-
-    def filter_announcements(self, announcements: List[Announcement],
-                           start_date: Optional[date] = None,
-                           end_date: Optional[date] = None,
-                           exclude_today: bool = False) -> List[Announcement]:
-        """
-        筛选日期范围内的公告
-
-        Args:
-            announcements: 公告列表
-            start_date: 开始日期
-            end_date: 结束日期
-            exclude_today: 是否排除今日公告
-
-        Returns:
-            List[Announcement]: 筛选后的公告列表
-        """
-        if not start_date and not end_date and not exclude_today:
-            # 没有日期限制时，所有公告都通过
-            for announcement in announcements:
-                announcement.date_filtered = True
-            return announcements
-
-        filtered = []
-        today = date.today()
-
-        for announcement in announcements:
-            if not announcement.publish_date:
-                # 没有发布日期的公告不通过筛选
-                announcement.date_filtered = False
-                continue
-
-            publish_date = announcement.publish_date.date()
-
-            # 检查日期范围
-            date_in_range = True
-
-            if start_date and publish_date < start_date:
-                date_in_range = False
-
-            if end_date and publish_date > end_date:
-                date_in_range = False
-
-            # 检查是否排除今日
-            if exclude_today and publish_date == today:
-                date_in_range = False
-
-            announcement.date_filtered = date_in_range
-
-            if date_in_range:
-                filtered.append(announcement)
-
-        logger.info(f"日期筛选: {len(announcements)} -> {len(filtered)} 条公告")
-        return filtered
-
-
-class DuplicateFilter:
-    """去重筛选器"""
-
-    def __init__(self, use_database: bool = True):
-        """
-        初始化去重筛选器
-
-        Args:
-            use_database: 是否使用数据库检查重复
-        """
-        self.use_database = use_database
-        self.db_manager = get_database_manager() if use_database else None
-
-    def filter_announcements(self, announcements: List[Announcement]) -> List[Announcement]:
-        """
-        去除重复的公告
-
-        Args:
-            announcements: 公告列表
-
-        Returns:
-            List[Announcement]: 去重后的公告列表
-        """
-        if not announcements:
-            return announcements
-
-        seen_hashes = set()
-        filtered = []
-
-        for announcement in announcements:
-            # 生成内容哈希
-            if not announcement.content_hash:
-                announcement.generate_content_hash()
-
-            content_hash = announcement.content_hash
-
-            # 检查是否已存在
-            is_duplicate = False
-
-            if self.use_database and self.db_manager:
-                # 数据库检查
-                is_duplicate = self.db_manager.is_announcement_exists(content_hash)
-            else:
-                # 内存检查
-                if content_hash in seen_hashes:
-                    is_duplicate = True
-                seen_hashes.add(content_hash)
-
-            if not is_duplicate:
-                filtered.append(announcement)
-            # 注意：这里不设置is_new标志，因为这是在爬取后的筛选阶段
-
-        duplicate_count = len(announcements) - len(filtered)
-        if duplicate_count > 0:
-            logger.info(f"去重筛选: 移除了 {duplicate_count} 条重复公告")
-
-        return filtered
-
-
-class SourceFilter:
-    """来源筛选器"""
-
-    def filter_announcements(self, announcements: List[Announcement],
-                           allowed_sources: List[str]) -> List[Announcement]:
-        """
-        筛选指定来源的公告
-
-        Args:
-            announcements: 公告列表
-            allowed_sources: 允许的来源代码列表
-
-        Returns:
-            List[Announcement]: 筛选后的公告列表
-        """
-        if not allowed_sources:
-            return announcements
-
-        filtered = []
-        for announcement in announcements:
-            if announcement.source_code in allowed_sources:
-                filtered.append(announcement)
-
-        logger.info(f"来源筛选: {len(announcements)} -> {len(filtered)} 条公告")
-        return filtered
-
-
-class AnnouncementFilter:
-    """公告智能筛选器"""
-
-    def __init__(self, criteria: FilterCriteria = None):
-        """
-        初始化公告筛选器
-
-        Args:
-            criteria: 筛选条件
-        """
-        self.criteria = criteria or FilterCriteria()
-
-        # 初始化各个筛选器
-        self.keyword_filter = KeywordFilter(
-            case_sensitive=self.criteria.case_sensitive,
-            fuzzy_match=self.criteria.fuzzy_match
-        )
-        self.date_filter = DateFilter()
-        self.duplicate_filter = DuplicateFilter()
-        self.source_filter = SourceFilter()
-
-    def filter(self, announcements: List[Announcement]) -> tuple[List[Announcement], FilterResult]:
-        """
-        执行完整的筛选流程
-
-        Args:
-            announcements: 原始公告列表
-
-        Returns:
-            tuple: (筛选后的公告列表, 筛选结果统计)
-        """
-        result = FilterResult()
-        result.total_count = len(announcements)
-
-        # 1. 来源筛选
-        if self.criteria.sources:
-            announcements = self.source_filter.filter_announcements(
-                announcements, self.criteria.sources)
-            result.source_filtered = result.total_count - len(announcements)
-
-        # 2. 去重筛选
-        announcements = self.duplicate_filter.filter_announcements(announcements)
-        result.duplicate_filtered = result.total_count - len(announcements) - result.source_filtered
-
-        # 3. 日期筛选
-        announcements = self.date_filter.filter_announcements(
-            announcements,
-            self.criteria.start_date,
-            self.criteria.end_date,
-            self.criteria.exclude_today
-        )
-        result.date_filtered = result.total_count - len(announcements) - result.source_filtered - result.duplicate_filtered
-
-        # 4. 关键词筛选
-        if self.criteria.keywords:
-            announcements = self.keyword_filter.filter_announcements(
-                announcements, self.criteria.keywords)
-            result.keyword_filtered = result.total_count - len(announcements) - result.source_filtered - result.duplicate_filtered - result.date_filtered
-
-        result.filtered_count = len(announcements)
-
-        logger.info(
-            f"筛选完成: 总数 {result.total_count} -> 筛选后 {result.filtered_count} "
-            f"(关键词: {result.keyword_filtered}, 日期: {result.date_filtered}, "
-            f"去重: {result.duplicate_filtered}, 来源: {result.source_filtered})"
-        )
-
-        return announcements, result
-
-    def quick_filter(self, announcement: Announcement) -> bool:
-        """
-        快速筛选单个公告（用于实时筛选）
-
-        Args:
-            announcement: 公告对象
-
-        Returns:
-            bool: 是否通过筛选
-        """
-        # 检查来源
-        if self.criteria.sources and announcement.source_code not in self.criteria.sources:
-            return False
-
-        # 检查日期
-        if not announcement.in_date_range(
-            self.criteria.start_date.isoformat() if self.criteria.start_date else None,
-            self.criteria.end_date.isoformat() if self.criteria.end_date else None
-        ):
-            return False
-
-        # 检查关键词
-        if self.criteria.keywords:
-            search_text = f"{announcement.title} {announcement.purchase_name}"
-            if not self.keyword_filter.matches(search_text, self.criteria.keywords):
-                return False
-
-        return True
-
-    def update_criteria(self, criteria: FilterCriteria):
-        """
-        更新筛选条件
-
-        Args:
-            criteria: 新的筛选条件
-        """
-        self.criteria = criteria
-
-        # 重新初始化筛选器
-        self.keyword_filter = KeywordFilter(
-            case_sensitive=self.criteria.case_sensitive,
-            fuzzy_match=self.criteria.fuzzy_match
-        )
-
-
-def create_default_filter(keywords: List[str] = None,
-                         start_date: str = None,
-                         end_date: str = None) -> AnnouncementFilter:
-    """
-    创建默认筛选器
-
-    Args:
-        keywords: 关键词列表
-        start_date: 开始日期字符串 (YYYY-MM-DD)
-        end_date: 结束日期字符串 (YYYY-MM-DD)
-
-    Returns:
-        AnnouncementFilter: 筛选器实例
-    """
-    criteria = FilterCriteria()
-
-    if keywords:
-        criteria.keywords = keywords
-
-    if start_date:
-        try:
-            criteria.start_date = datetime.fromisoformat(start_date).date()
-        except ValueError:
-            logger.warning(f"无效的开始日期格式: {start_date}")
-
-    if end_date:
-        try:
-            criteria.end_date = datetime.fromisoformat(end_date).date()
-        except ValueError:
-            logger.warning(f"无效的结束日期格式: {end_date}")
-
-    return AnnouncementFilter(criteria)
-
-
-def filter_from_config() -> AnnouncementFilter:
-    """
-    从配置文件创建筛选器
-
-    Returns:
-        AnnouncementFilter: 配置化的筛选器
-    """
-    try:
-        from ..core.config_manager import get_config
-    except ImportError:
-        from core.config_manager import get_config
-
-    config = get_config()
-
-    criteria = FilterCriteria()
-    criteria.keywords = config.crawler.keyword
-    criteria.sources = list(config.sources.keys())  # 默认包含所有来源
-
-    # 日期范围
-    if config.crawler.start_date:
-        try:
-            criteria.start_date = datetime.fromisoformat(config.crawler.start_date).date()
-        except ValueError:
-            pass
-
-    if config.crawler.end_date:
-        try:
-            criteria.end_date = datetime.fromisoformat(config.crawler.end_date).date()
-        except ValueError:
-            pass
-
-    return AnnouncementFilter(criteria)

gx_gp_monitor/logs/gx_gp_monitor.log

@@ -1,858 +0,0 @@
-2026-01-07 17:16:51 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:16:51 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:16:51 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:16:52 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:16:52 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:16:52 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:16:52 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:16:52 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:16:52 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:16:52 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:16:52 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:20:31 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:20:31 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:20:31 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:20:31 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:20:31 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:20:31 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:20:31 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:20:31 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:20:31 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:20:31 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:20:31 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:21:02 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:21:02 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:21:02 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:21:02 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:21:02 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:21:02 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:21:02 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:21:02 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:21:02 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:21:02 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:21:02 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:22:10 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:22:10 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:22:10 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:22:10 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:22:10 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:22:10 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:22:10 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:22:10 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:22:10 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:22:10 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:22:10 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:22:44 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:22:44 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:22:44 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:22:44 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:22:44 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:22:44 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:22:44 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:22:44 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:22:44 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:22:44 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:22:44 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:22:46 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:22:46 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:22:46 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:22:47 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:22:47 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:22:47 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:22:47 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:22:47 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:22:47 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:22:47 - __main__ - INFO - 开始执行爬取任务
-2026-01-07 17:22:47 - crawler.spider - INFO - 开始爬取 13 个公告来源
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 采购公告 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 结果公告 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 合同公告 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 更正公告 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 招标文件预公示 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 单一来源公示 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 电子卖场公示 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 履约验收公示 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 工程类公告 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 框架协议征集公告 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 框架协议入围结果公告 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 框架协议成交结果汇总公告 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - ERROR - 爬取来源 采购意向公开 时发生未预期错误: 'Logger' object has no attribute 'log_crawl_start'
-2026-01-07 17:22:47 - crawler.spider - INFO - 爬取完成: 共处理 13 个来源，成功 0 个，失败 13 个，获取 0 条公告
-2026-01-07 17:22:47 - __main__ - INFO - 爬取到 0 条原始公告
-2026-01-07 17:22:47 - __main__ - ERROR - 爬取任务执行失败: attempted relative import beyond top-level package
-2026-01-07 17:22:47 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:22:47 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:23:10 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:23:10 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:23:10 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:23:10 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:23:10 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:23:10 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:23:10 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:23:10 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:23:10 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:23:10 - __main__ - INFO - 开始执行爬取任务
-2026-01-07 17:23:10 - crawler.spider - INFO - 开始爬取 13 个公告来源
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 采购公告 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 结果公告 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 合同公告 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 更正公告 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 招标文件预公示 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 单一来源公示 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 电子卖场公示 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 履约验收公示 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 工程类公告 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 框架协议征集公告 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 框架协议入围结果公告 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 框架协议成交结果汇总公告 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - ERROR - 爬取来源 采购意向公开 时发生未预期错误: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - crawler.spider - INFO - 爬取完成: 共处理 13 个来源，成功 0 个，失败 13 个，获取 0 条公告
-2026-01-07 17:23:10 - __main__ - INFO - 爬取到 0 条原始公告
-2026-01-07 17:23:10 - __main__ - ERROR - 爬取任务执行失败: attempted relative import beyond top-level package
-2026-01-07 17:23:10 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:23:10 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:23:34 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:23:34 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:23:34 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:23:35 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:23:35 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:23:35 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:23:35 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:23:35 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:23:35 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:23:35 - __main__ - INFO - 开始执行爬取任务
-2026-01-07 17:23:35 - crawler.spider - INFO - 开始爬取 13 个公告来源
-2026-01-07 17:23:35 - gx_gp_monitor - INFO - 开始爬取 采购公告
-2026-01-07 17:24:05 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:05 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:05 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:05 - crawler.parsers - INFO - 成功解析 97/100 条公告记录
-2026-01-07 17:24:05 - gx_gp_monitor - INFO - 采购公告 爬取完成，共获取 97 条公告，耗时 30.46秒
-2026-01-07 17:24:05 - gx_gp_monitor - INFO - 开始爬取 结果公告
-2026-01-07 17:24:05 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:24:05 - gx_gp_monitor - INFO - 结果公告 爬取完成，共获取 100 条公告，耗时 0.18秒
-2026-01-07 17:24:05 - gx_gp_monitor - INFO - 开始爬取 合同公告
-2026-01-07 17:24:06 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:24:06 - gx_gp_monitor - INFO - 合同公告 爬取完成，共获取 100 条公告，耗时 0.17秒
-2026-01-07 17:24:06 - gx_gp_monitor - INFO - 开始爬取 更正公告
-2026-01-07 17:24:06 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:24:06 - gx_gp_monitor - INFO - 更正公告 爬取完成，共获取 100 条公告，耗时 0.23秒
-2026-01-07 17:24:06 - gx_gp_monitor - INFO - 开始爬取 招标文件预公示
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - INFO - 成功解析 78/100 条公告记录
-2026-01-07 17:24:06 - gx_gp_monitor - INFO - 招标文件预公示 爬取完成，共获取 78 条公告，耗时 0.17秒
-2026-01-07 17:24:06 - gx_gp_monitor - INFO - 开始爬取 单一来源公示
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:06 - crawler.parsers - INFO - 成功解析 0/100 条公告记录
-2026-01-07 17:24:06 - crawler.spider - INFO - 单一来源公示 第1页解析到0条公告
-2026-01-07 17:24:06 - gx_gp_monitor - INFO - 单一来源公示 爬取完成，共获取 0 条公告，耗时 0.18秒
-2026-01-07 17:24:06 - gx_gp_monitor - INFO - 开始爬取 电子卖场公示
-2026-01-07 17:24:06 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:24:06 - gx_gp_monitor - INFO - 电子卖场公示 爬取完成，共获取 100 条公告，耗时 0.31秒
-2026-01-07 17:24:06 - gx_gp_monitor - INFO - 开始爬取 履约验收公示
-2026-01-07 17:24:07 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:24:07 - gx_gp_monitor - INFO - 履约验收公示 爬取完成，共获取 100 条公告，耗时 0.16秒
-2026-01-07 17:24:07 - gx_gp_monitor - INFO - 开始爬取 工程类公告
-2026-01-07 17:24:07 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:24:07 - gx_gp_monitor - INFO - 工程类公告 爬取完成，共获取 100 条公告，耗时 0.27秒
-2026-01-07 17:24:07 - gx_gp_monitor - INFO - 开始爬取 框架协议征集公告
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - INFO - 成功解析 62/100 条公告记录
-2026-01-07 17:24:07 - gx_gp_monitor - INFO - 框架协议征集公告 爬取完成，共获取 62 条公告，耗时 0.20秒
-2026-01-07 17:24:07 - gx_gp_monitor - INFO - 开始爬取 框架协议入围结果公告
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:07 - crawler.parsers - INFO - 成功解析 13/100 条公告记录
-2026-01-07 17:24:07 - gx_gp_monitor - INFO - 框架协议入围结果公告 爬取完成，共获取 13 条公告，耗时 0.17秒
-2026-01-07 17:24:07 - gx_gp_monitor - INFO - 开始爬取 框架协议成交结果汇总公告
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - ERROR - 解析单个公告记录失败: 'NoneType' object has no attribute 'strip'
-2026-01-07 17:24:08 - crawler.parsers - INFO - 成功解析 0/100 条公告记录
-2026-01-07 17:24:08 - crawler.spider - INFO - 框架协议成交结果汇总公告 第1页解析到0条公告
-2026-01-07 17:24:08 - gx_gp_monitor - INFO - 框架协议成交结果汇总公告 爬取完成，共获取 0 条公告，耗时 0.29秒
-2026-01-07 17:24:08 - gx_gp_monitor - INFO - 开始爬取 采购意向公开
-2026-01-07 17:24:08 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:24:08 - gx_gp_monitor - INFO - 采购意向公开 爬取完成，共获取 100 条公告，耗时 0.17秒
-2026-01-07 17:24:08 - crawler.spider - INFO - 爬取完成: 共处理 13 个来源，成功 13 个，失败 0 个，获取 950 条公告
-2026-01-07 17:24:08 - __main__ - INFO - 爬取到 950 条原始公告
-2026-01-07 17:24:08 - __main__ - ERROR - 爬取任务执行失败: attempted relative import beyond top-level package
-2026-01-07 17:24:08 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:24:08 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:24:17 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:24:17 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:24:17 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:24:17 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:24:17 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:24:17 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:24:17 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:24:17 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:24:17 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:24:17 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:24:17 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:26:34 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:26:34 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:26:34 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:26:34 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:26:34 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:26:34 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:26:34 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:26:34 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:26:34 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:26:34 - __main__ - INFO - 开始执行爬取任务
-2026-01-07 17:26:34 - crawler.spider - INFO - 开始爬取 13 个公告来源
-2026-01-07 17:26:34 - gx_gp_monitor - INFO - 开始爬取 采购公告
-2026-01-07 17:26:34 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:34 - gx_gp_monitor - INFO - 采购公告 爬取完成，共获取 100 条公告，耗时 0.48秒
-2026-01-07 17:26:34 - gx_gp_monitor - INFO - 开始爬取 结果公告
-2026-01-07 17:26:35 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:35 - gx_gp_monitor - INFO - 结果公告 爬取完成，共获取 100 条公告，耗时 0.21秒
-2026-01-07 17:26:35 - gx_gp_monitor - INFO - 开始爬取 合同公告
-2026-01-07 17:26:35 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:35 - gx_gp_monitor - INFO - 合同公告 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-07 17:26:35 - gx_gp_monitor - INFO - 开始爬取 更正公告
-2026-01-07 17:26:35 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:35 - gx_gp_monitor - INFO - 更正公告 爬取完成，共获取 100 条公告，耗时 0.23秒
-2026-01-07 17:26:35 - gx_gp_monitor - INFO - 开始爬取 招标文件预公示
-2026-01-07 17:26:35 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:35 - gx_gp_monitor - INFO - 招标文件预公示 爬取完成，共获取 100 条公告，耗时 0.20秒
-2026-01-07 17:26:35 - gx_gp_monitor - INFO - 开始爬取 单一来源公示
-2026-01-07 17:26:35 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:35 - gx_gp_monitor - INFO - 单一来源公示 爬取完成，共获取 100 条公告，耗时 0.17秒
-2026-01-07 17:26:35 - gx_gp_monitor - INFO - 开始爬取 电子卖场公示
-2026-01-07 17:26:36 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:36 - gx_gp_monitor - INFO - 电子卖场公示 爬取完成，共获取 100 条公告，耗时 0.40秒
-2026-01-07 17:26:36 - gx_gp_monitor - INFO - 开始爬取 履约验收公示
-2026-01-07 17:26:36 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:36 - gx_gp_monitor - INFO - 履约验收公示 爬取完成，共获取 100 条公告，耗时 0.22秒
-2026-01-07 17:26:36 - gx_gp_monitor - INFO - 开始爬取 工程类公告
-2026-01-07 17:26:36 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:36 - gx_gp_monitor - INFO - 工程类公告 爬取完成，共获取 100 条公告，耗时 0.27秒
-2026-01-07 17:26:36 - gx_gp_monitor - INFO - 开始爬取 框架协议征集公告
-2026-01-07 17:26:36 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:36 - gx_gp_monitor - INFO - 框架协议征集公告 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-07 17:26:36 - gx_gp_monitor - INFO - 开始爬取 框架协议入围结果公告
-2026-01-07 17:26:37 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:37 - gx_gp_monitor - INFO - 框架协议入围结果公告 爬取完成，共获取 100 条公告，耗时 0.25秒
-2026-01-07 17:26:37 - gx_gp_monitor - INFO - 开始爬取 框架协议成交结果汇总公告
-2026-01-07 17:26:37 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:37 - gx_gp_monitor - INFO - 框架协议成交结果汇总公告 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-07 17:26:37 - gx_gp_monitor - INFO - 开始爬取 采购意向公开
-2026-01-07 17:26:37 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:37 - gx_gp_monitor - INFO - 采购意向公开 爬取完成，共获取 100 条公告，耗时 0.17秒
-2026-01-07 17:26:37 - crawler.spider - INFO - 爬取完成: 共处理 13 个来源，成功 13 个，失败 0 个，获取 1300 条公告
-2026-01-07 17:26:37 - __main__ - INFO - 爬取到 1300 条原始公告
-2026-01-07 17:26:37 - __main__ - ERROR - 爬取任务执行失败: attempted relative import beyond top-level package
-2026-01-07 17:26:37 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:26:37 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:26:57 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:26:57 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:26:57 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:26:58 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:26:58 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:26:58 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:26:58 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:26:58 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:26:58 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:26:58 - __main__ - INFO - 开始执行爬取任务
-2026-01-07 17:26:58 - crawler.spider - INFO - 开始爬取 13 个公告来源
-2026-01-07 17:26:58 - gx_gp_monitor - INFO - 开始爬取 采购公告
-2026-01-07 17:26:58 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:58 - gx_gp_monitor - INFO - 采购公告 爬取完成，共获取 100 条公告，耗时 0.37秒
-2026-01-07 17:26:58 - gx_gp_monitor - INFO - 开始爬取 结果公告
-2026-01-07 17:26:58 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:58 - gx_gp_monitor - INFO - 结果公告 爬取完成，共获取 100 条公告，耗时 0.23秒
-2026-01-07 17:26:58 - gx_gp_monitor - INFO - 开始爬取 合同公告
-2026-01-07 17:26:58 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:58 - gx_gp_monitor - INFO - 合同公告 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-07 17:26:58 - gx_gp_monitor - INFO - 开始爬取 更正公告
-2026-01-07 17:26:59 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:59 - gx_gp_monitor - INFO - 更正公告 爬取完成，共获取 100 条公告，耗时 0.20秒
-2026-01-07 17:26:59 - gx_gp_monitor - INFO - 开始爬取 招标文件预公示
-2026-01-07 17:26:59 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:59 - gx_gp_monitor - INFO - 招标文件预公示 爬取完成，共获取 100 条公告，耗时 0.21秒
-2026-01-07 17:26:59 - gx_gp_monitor - INFO - 开始爬取 单一来源公示
-2026-01-07 17:26:59 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:59 - gx_gp_monitor - INFO - 单一来源公示 爬取完成，共获取 100 条公告，耗时 0.16秒
-2026-01-07 17:26:59 - gx_gp_monitor - INFO - 开始爬取 电子卖场公示
-2026-01-07 17:26:59 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:59 - gx_gp_monitor - INFO - 电子卖场公示 爬取完成，共获取 100 条公告，耗时 0.38秒
-2026-01-07 17:26:59 - gx_gp_monitor - INFO - 开始爬取 履约验收公示
-2026-01-07 17:26:59 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:26:59 - gx_gp_monitor - INFO - 履约验收公示 爬取完成，共获取 100 条公告，耗时 0.18秒
-2026-01-07 17:26:59 - gx_gp_monitor - INFO - 开始爬取 工程类公告
-2026-01-07 17:27:00 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:27:00 - gx_gp_monitor - INFO - 工程类公告 爬取完成，共获取 100 条公告，耗时 0.25秒
-2026-01-07 17:27:00 - gx_gp_monitor - INFO - 开始爬取 框架协议征集公告
-2026-01-07 17:27:00 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:27:00 - gx_gp_monitor - INFO - 框架协议征集公告 爬取完成，共获取 100 条公告，耗时 0.21秒
-2026-01-07 17:27:00 - gx_gp_monitor - INFO - 开始爬取 框架协议入围结果公告
-2026-01-07 17:27:00 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:27:00 - gx_gp_monitor - INFO - 框架协议入围结果公告 爬取完成，共获取 100 条公告，耗时 0.17秒
-2026-01-07 17:27:00 - gx_gp_monitor - INFO - 开始爬取 框架协议成交结果汇总公告
-2026-01-07 17:27:00 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:27:00 - gx_gp_monitor - INFO - 框架协议成交结果汇总公告 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-07 17:27:00 - gx_gp_monitor - INFO - 开始爬取 采购意向公开
-2026-01-07 17:27:01 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:27:01 - gx_gp_monitor - INFO - 采购意向公开 爬取完成，共获取 100 条公告，耗时 0.25秒
-2026-01-07 17:27:01 - crawler.spider - INFO - 爬取完成: 共处理 13 个来源，成功 13 个，失败 0 个，获取 1300 条公告
-2026-01-07 17:27:01 - __main__ - INFO - 爬取到 1300 条原始公告
-2026-01-07 17:27:01 - filters.filters - INFO - 来源筛选: 1300 -> 1300 条公告
-2026-01-07 17:27:01 - filters.filters - INFO - 关键词筛选: 1300 -> 18 条公告
-2026-01-07 17:27:01 - filters.filters - INFO - 筛选完成: 总数 1300 -> 筛选后 18 (关键词: 1282, 日期: 0, 去重: 0, 来源: 0)
-2026-01-07 17:27:01 - __main__ - INFO - 筛选后剩余 18 条公告
-2026-01-07 17:27:01 - storage.postgresql - INFO - 开始保存 18 条公告到数据库
-2026-01-07 17:27:01 - core.database - INFO - 批量保存公告完成，成功保存 1 条
-2026-01-07 17:27:01 - storage.postgresql - INFO - 成功保存 1 条公告到数据库
-2026-01-07 17:27:01 - storage.md_generator - INFO - Markdown文件已保存到: onu.md (共 18 条公告)
-2026-01-07 17:27:01 - notification.wechat - INFO - 企业微信服务初始化完成
-2026-01-07 17:27:01 - notification.wechat - INFO - 成功获取企业微信访问令牌
-2026-01-07 17:27:01 - notification.wechat - INFO - Markdown消息发送成功
-2026-01-07 17:27:01 - __main__ - INFO - 爬取任务完成: {'success': True, 'total_crawled': 1300, 'filtered': 18, 'saved': 1, 'markdown_generated': True, 'notification_sent': True, 'filter_stats': {'keyword_filtered': 1282, 'date_filtered': 0, 'duplicate_filtered': 0, 'source_filtered': 0}}
-2026-01-07 17:27:01 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:27:01 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:29:05 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:29:05 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:29:05 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:29:06 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:29:06 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:29:06 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:29:06 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:29:06 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:29:06 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:29:06 - scheduler.scheduler - INFO - 添加定时任务: daily_crawl (0 8,14,18 * * *)
-2026-01-07 17:29:06 - scheduler.scheduler - INFO - 添加定时任务: data_cleanup (0 2 * * *)
-2026-01-07 17:31:58 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:31:58 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:31:58 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:31:58 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:31:58 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:31:58 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:31:58 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:31:58 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:31:58 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:34:32 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:34:32 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:34:32 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:34:32 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:34:32 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:34:32 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:34:32 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:34:32 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:34:32 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:34:35 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:34:35 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:34:35 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:34:35 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:34:35 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:34:35 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:34:35 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:34:35 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:34:35 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:34:35 - __main__ - INFO - 开始执行爬取任务
-2026-01-07 17:34:35 - crawler.spider - INFO - 开始爬取 13 个公告来源
-2026-01-07 17:34:35 - gx_gp_monitor - INFO - 开始爬取 采购公告
-2026-01-07 17:35:05 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:05 - gx_gp_monitor - INFO - 采购公告 爬取完成，共获取 100 条公告，耗时 30.44秒
-2026-01-07 17:35:05 - gx_gp_monitor - INFO - 开始爬取 结果公告
-2026-01-07 17:35:06 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:06 - gx_gp_monitor - INFO - 结果公告 爬取完成，共获取 100 条公告，耗时 0.26秒
-2026-01-07 17:35:06 - gx_gp_monitor - INFO - 开始爬取 合同公告
-2026-01-07 17:35:06 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:06 - gx_gp_monitor - INFO - 合同公告 爬取完成，共获取 100 条公告，耗时 0.20秒
-2026-01-07 17:35:06 - gx_gp_monitor - INFO - 开始爬取 更正公告
-2026-01-07 17:35:06 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:06 - gx_gp_monitor - INFO - 更正公告 爬取完成，共获取 100 条公告，耗时 0.23秒
-2026-01-07 17:35:06 - gx_gp_monitor - INFO - 开始爬取 招标文件预公示
-2026-01-07 17:35:06 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:06 - gx_gp_monitor - INFO - 招标文件预公示 爬取完成，共获取 100 条公告，耗时 0.18秒
-2026-01-07 17:35:06 - gx_gp_monitor - INFO - 开始爬取 单一来源公示
-2026-01-07 17:35:07 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:07 - gx_gp_monitor - INFO - 单一来源公示 爬取完成，共获取 100 条公告，耗时 0.18秒
-2026-01-07 17:35:07 - gx_gp_monitor - INFO - 开始爬取 电子卖场公示
-2026-01-07 17:35:07 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:07 - gx_gp_monitor - INFO - 电子卖场公示 爬取完成，共获取 100 条公告，耗时 0.34秒
-2026-01-07 17:35:07 - gx_gp_monitor - INFO - 开始爬取 履约验收公示
-2026-01-07 17:35:07 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:07 - gx_gp_monitor - INFO - 履约验收公示 爬取完成，共获取 100 条公告，耗时 0.21秒
-2026-01-07 17:35:07 - gx_gp_monitor - INFO - 开始爬取 工程类公告
-2026-01-07 17:35:07 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:07 - gx_gp_monitor - INFO - 工程类公告 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-07 17:35:07 - gx_gp_monitor - INFO - 开始爬取 框架协议征集公告
-2026-01-07 17:35:07 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:07 - gx_gp_monitor - INFO - 框架协议征集公告 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-07 17:35:07 - gx_gp_monitor - INFO - 开始爬取 框架协议入围结果公告
-2026-01-07 17:35:08 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:08 - gx_gp_monitor - INFO - 框架协议入围结果公告 爬取完成，共获取 100 条公告，耗时 0.18秒
-2026-01-07 17:35:08 - gx_gp_monitor - INFO - 开始爬取 框架协议成交结果汇总公告
-2026-01-07 17:35:08 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:08 - gx_gp_monitor - INFO - 框架协议成交结果汇总公告 爬取完成，共获取 100 条公告，耗时 0.25秒
-2026-01-07 17:35:08 - gx_gp_monitor - INFO - 开始爬取 采购意向公开
-2026-01-07 17:35:08 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:35:08 - gx_gp_monitor - INFO - 采购意向公开 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-07 17:35:08 - crawler.spider - INFO - 爬取完成: 共处理 13 个来源，成功 13 个，失败 0 个，获取 1300 条公告
-2026-01-07 17:35:08 - __main__ - INFO - 爬取到 1300 条原始公告
-2026-01-07 17:35:08 - filters.filters - INFO - 来源筛选: 1300 -> 1300 条公告
-2026-01-07 17:35:08 - filters.filters - INFO - 去重筛选: 移除了 18 条重复公告
-2026-01-07 17:35:08 - filters.filters - INFO - 关键词筛选: 1282 -> 0 条公告
-2026-01-07 17:35:08 - filters.filters - INFO - 筛选完成: 总数 1300 -> 筛选后 0 (关键词: 1282, 日期: 0, 去重: 18, 来源: 0)
-2026-01-07 17:35:08 - __main__ - INFO - 筛选后剩余 0 条公告
-2026-01-07 17:35:08 - storage.md_generator - INFO - Markdown文件已保存到: onu.md (共 0 条公告)
-2026-01-07 17:35:08 - __main__ - INFO - 爬取任务完成: {'success': True, 'total_crawled': 1300, 'filtered': 0, 'saved': 0, 'markdown_generated': True, 'notification_sent': False, 'filter_stats': {'keyword_filtered': 1282, 'date_filtered': 0, 'duplicate_filtered': 18, 'source_filtered': 0}}
-2026-01-07 17:35:38 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-07 17:35:38 - __main__ - INFO - 版本: 1.0.0
-2026-01-07 17:35:38 - __main__ - INFO - 配置文件: 默认配置
-2026-01-07 17:35:38 - core.reliability - INFO - 健康检查通过
-2026-01-07 17:35:38 - core.database - INFO - 数据库连接池初始化成功
-2026-01-07 17:35:38 - core.database - INFO - 开始初始化数据库表结构
-2026-01-07 17:35:38 - core.database - INFO - 数据库表结构初始化完成
-2026-01-07 17:35:38 - storage.postgresql - INFO - 存储初始化完成
-2026-01-07 17:35:38 - __main__ - INFO - 应用初始化完成
-2026-01-07 17:35:38 - __main__ - INFO - 开始执行爬取任务
-2026-01-07 17:35:38 - crawler.spider - INFO - 开始爬取 13 个公告来源
-2026-01-07 17:35:38 - gx_gp_monitor - INFO - 开始爬取 采购公告
-2026-01-07 17:36:09 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:09 - gx_gp_monitor - INFO - 采购公告 爬取完成，共获取 100 条公告，耗时 30.49秒
-2026-01-07 17:36:09 - gx_gp_monitor - INFO - 开始爬取 结果公告
-2026-01-07 17:36:09 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:09 - gx_gp_monitor - INFO - 结果公告 爬取完成，共获取 100 条公告，耗时 0.20秒
-2026-01-07 17:36:09 - gx_gp_monitor - INFO - 开始爬取 合同公告
-2026-01-07 17:36:09 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:09 - gx_gp_monitor - INFO - 合同公告 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-07 17:36:09 - gx_gp_monitor - INFO - 开始爬取 更正公告
-2026-01-07 17:36:09 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:09 - gx_gp_monitor - INFO - 更正公告 爬取完成，共获取 100 条公告，耗时 0.27秒
-2026-01-07 17:36:09 - gx_gp_monitor - INFO - 开始爬取 招标文件预公示
-2026-01-07 17:36:10 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:10 - gx_gp_monitor - INFO - 招标文件预公示 爬取完成，共获取 100 条公告，耗时 0.28秒
-2026-01-07 17:36:10 - gx_gp_monitor - INFO - 开始爬取 单一来源公示
-2026-01-07 17:36:10 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:10 - gx_gp_monitor - INFO - 单一来源公示 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-07 17:36:10 - gx_gp_monitor - INFO - 开始爬取 电子卖场公示
-2026-01-07 17:36:10 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:10 - gx_gp_monitor - INFO - 电子卖场公示 爬取完成，共获取 100 条公告，耗时 0.36秒
-2026-01-07 17:36:10 - gx_gp_monitor - INFO - 开始爬取 履约验收公示
-2026-01-07 17:36:11 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 履约验收公示 爬取完成，共获取 100 条公告，耗时 0.27秒
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 开始爬取 工程类公告
-2026-01-07 17:36:11 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 工程类公告 爬取完成，共获取 100 条公告，耗时 0.21秒
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 开始爬取 框架协议征集公告
-2026-01-07 17:36:11 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 框架协议征集公告 爬取完成，共获取 100 条公告，耗时 0.15秒
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 开始爬取 框架协议入围结果公告
-2026-01-07 17:36:11 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 框架协议入围结果公告 爬取完成，共获取 100 条公告，耗时 0.24秒
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 开始爬取 框架协议成交结果汇总公告
-2026-01-07 17:36:11 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 框架协议成交结果汇总公告 爬取完成，共获取 100 条公告，耗时 0.16秒
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 开始爬取 采购意向公开
-2026-01-07 17:36:11 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-07 17:36:11 - gx_gp_monitor - INFO - 采购意向公开 爬取完成，共获取 100 条公告，耗时 0.18秒
-2026-01-07 17:36:11 - crawler.spider - INFO - 爬取完成: 共处理 13 个来源，成功 13 个，失败 0 个，获取 1300 条公告
-2026-01-07 17:36:11 - __main__ - INFO - 爬取到 1300 条原始公告
-2026-01-07 17:36:11 - filters.filters - INFO - 来源筛选: 1300 -> 1300 条公告
-2026-01-07 17:36:12 - filters.filters - INFO - 去重筛选: 移除了 18 条重复公告
-2026-01-07 17:36:12 - filters.filters - INFO - 关键词筛选: 1282 -> 0 条公告
-2026-01-07 17:36:12 - filters.filters - INFO - 筛选完成: 总数 1300 -> 筛选后 0 (关键词: 1282, 日期: 0, 去重: 18, 来源: 0)
-2026-01-07 17:36:12 - __main__ - INFO - 筛选后剩余 0 条公告
-2026-01-07 17:36:12 - storage.md_generator - INFO - Markdown文件已保存到: onu.md (共 0 条公告)
-2026-01-07 17:36:12 - __main__ - INFO - 爬取任务完成: {'success': True, 'total_crawled': 1300, 'filtered': 0, 'saved': 0, 'markdown_generated': True, 'notification_sent': False, 'filter_stats': {'keyword_filtered': 1282, 'date_filtered': 0, 'duplicate_filtered': 18, 'source_filtered': 0}}
-2026-01-08 08:17:30 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
-2026-01-08 08:17:30 - __main__ - INFO - 版本: 1.0.0
-2026-01-08 08:17:30 - __main__ - INFO - 配置文件: 默认配置
-2026-01-08 08:17:30 - core.reliability - INFO - 健康检查通过
-2026-01-08 08:17:30 - core.database - INFO - 数据库连接池初始化成功
-2026-01-08 08:17:30 - core.database - INFO - 开始初始化数据库表结构
-2026-01-08 08:17:30 - core.database - INFO - 数据库表结构初始化完成
-2026-01-08 08:17:30 - storage.postgresql - INFO - 存储初始化完成
-2026-01-08 08:17:30 - __main__ - INFO - 应用初始化完成
-2026-01-08 08:17:30 - __main__ - INFO - 开始执行爬取任务
-2026-01-08 08:17:30 - crawler.spider - INFO - 开始爬取 13 个公告来源
-2026-01-08 08:17:30 - gx_gp_monitor - INFO - 开始爬取 采购公告
-2026-01-08 08:17:31 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:31 - gx_gp_monitor - INFO - 采购公告 爬取完成，共获取 100 条公告，耗时 0.53秒
-2026-01-08 08:17:31 - gx_gp_monitor - INFO - 开始爬取 结果公告
-2026-01-08 08:17:31 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:31 - gx_gp_monitor - INFO - 结果公告 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-08 08:17:31 - gx_gp_monitor - INFO - 开始爬取 合同公告
-2026-01-08 08:17:31 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:31 - gx_gp_monitor - INFO - 合同公告 爬取完成，共获取 100 条公告，耗时 0.20秒
-2026-01-08 08:17:31 - gx_gp_monitor - INFO - 开始爬取 更正公告
-2026-01-08 08:17:31 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:31 - gx_gp_monitor - INFO - 更正公告 爬取完成，共获取 100 条公告，耗时 0.28秒
-2026-01-08 08:17:31 - gx_gp_monitor - INFO - 开始爬取 招标文件预公示
-2026-01-08 08:17:31 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:31 - gx_gp_monitor - INFO - 招标文件预公示 爬取完成，共获取 100 条公告，耗时 0.18秒
-2026-01-08 08:17:31 - gx_gp_monitor - INFO - 开始爬取 单一来源公示
-2026-01-08 08:17:32 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:32 - gx_gp_monitor - INFO - 单一来源公示 爬取完成，共获取 100 条公告，耗时 0.16秒
-2026-01-08 08:17:32 - gx_gp_monitor - INFO - 开始爬取 电子卖场公示
-2026-01-08 08:17:32 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:32 - gx_gp_monitor - INFO - 电子卖场公示 爬取完成，共获取 100 条公告，耗时 0.34秒
-2026-01-08 08:17:32 - gx_gp_monitor - INFO - 开始爬取 履约验收公示
-2026-01-08 08:17:32 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:32 - gx_gp_monitor - INFO - 履约验收公示 爬取完成，共获取 100 条公告，耗时 0.48秒
-2026-01-08 08:17:32 - gx_gp_monitor - INFO - 开始爬取 工程类公告
-2026-01-08 08:17:33 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:33 - gx_gp_monitor - INFO - 工程类公告 爬取完成，共获取 100 条公告，耗时 0.23秒
-2026-01-08 08:17:33 - gx_gp_monitor - INFO - 开始爬取 框架协议征集公告
-2026-01-08 08:17:33 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:33 - gx_gp_monitor - INFO - 框架协议征集公告 爬取完成，共获取 100 条公告，耗时 0.18秒
-2026-01-08 08:17:33 - gx_gp_monitor - INFO - 开始爬取 框架协议入围结果公告
-2026-01-08 08:17:33 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:33 - gx_gp_monitor - INFO - 框架协议入围结果公告 爬取完成，共获取 100 条公告，耗时 0.23秒
-2026-01-08 08:17:33 - gx_gp_monitor - INFO - 开始爬取 框架协议成交结果汇总公告
-2026-01-08 08:17:33 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:33 - gx_gp_monitor - INFO - 框架协议成交结果汇总公告 爬取完成，共获取 100 条公告，耗时 0.16秒
-2026-01-08 08:17:33 - gx_gp_monitor - INFO - 开始爬取 采购意向公开
-2026-01-08 08:17:33 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
-2026-01-08 08:17:33 - gx_gp_monitor - INFO - 采购意向公开 爬取完成，共获取 100 条公告，耗时 0.19秒
-2026-01-08 08:17:33 - crawler.spider - INFO - 爬取完成: 共处理 13 个来源，成功 13 个，失败 0 个，获取 1300 条公告
-2026-01-08 08:17:33 - __main__ - INFO - 爬取到 1300 条原始公告
-2026-01-08 08:17:33 - filters.filters - INFO - 来源筛选: 1300 -> 1300 条公告
-2026-01-08 08:17:34 - filters.filters - INFO - 去重筛选: 移除了 17 条重复公告
-2026-01-08 08:17:34 - filters.filters - INFO - 关键词筛选: 1283 -> 0 条公告
-2026-01-08 08:17:34 - filters.filters - INFO - 筛选完成: 总数 1300 -> 筛选后 0 (关键词: 1283, 日期: 0, 去重: 17, 来源: 0)
-2026-01-08 08:17:34 - __main__ - INFO - 筛选后剩余 0 条公告
-2026-01-08 08:17:34 - storage.md_generator - INFO - Markdown文件已保存到: onu.md (共 0 条公告)
-2026-01-08 08:17:34 - __main__ - INFO - 爬取任务完成: {'success': True, 'total_crawled': 1300, 'filtered': 0, 'saved': 0, 'markdown_generated': True, 'notification_sent': False, 'filter_stats': {'keyword_filtered': 1283, 'date_filtered': 0, 'duplicate_filtered': 17, 'source_filtered': 0}}

gx_gp_monitor/main.py

@@ -1,433 +0,0 @@
-#!/usr/bin/env python3
-"""
-广西政府采购网公告监控系统主程序
-广西政府采购网公告爬取和监控的智能系统
-"""
-
-import sys
-import argparse
-import signal
-from pathlib import Path
-
-# 添加项目根目录到Python路径
-project_root = Path(__file__).parent
-sys.path.insert(0, str(project_root))
-
-try:
-    # 尝试相对导入
-    from .core.config_manager import load_config, get_config
-    from .core.logger import init_logger, get_logger
-    from .core.reliability import check_system_health
-    from .crawler.spider import crawl_announcements
-    from .filters.filters import filter_from_config
-    from .storage.postgresql import init_storage, save_announcements_to_storage, save_all_announcements_by_source_to_storage, cleanup_storage
-    from .storage.md_generator import generate_onu_md
-    from .notification.wechat import send_announcements_notification, send_system_notification
-    from .wechat.callback_server import get_callback_server
-    from .wechat.menu_manager import WeChatMenuManager
-except ImportError:
-    try:
-        # 尝试绝对导入（直接运行脚本时）
-        from core.config_manager import load_config, get_config
-        from core.logger import init_logger, get_logger
-        from core.reliability import check_system_health
-        from crawler.spider import crawl_announcements
-        from filters.filters import filter_from_config
-        from storage.postgresql import init_storage, save_announcements_to_storage, save_all_announcements_by_source_to_storage, cleanup_storage
-        from storage.md_generator import generate_onu_md
-        from notification.wechat import send_announcements_notification, send_system_notification
-        # 企业微信模块动态导入，避免循环导入问题
-        wechat_available = True
-        try:
-            from wechat.callback_server import get_callback_server
-            from wechat.menu_manager import WeChatMenuManager
-        except ImportError:
-            wechat_available = False
-            print("企业微信模块不可用", file=sys.stderr)
-    except ImportError as e:
-        print(f"导入错误: {e}", file=sys.stderr)
-        print("请确保依赖已正确安装: pip install -r requirements.txt", file=sys.stderr)
-        sys.exit(1)
-
-
-logger = get_logger(__name__)
-
-
-class GXGPMonitorApp:
-    """广西政府采购网监控系统应用"""
-
-    def __init__(self):
-        self.config = None
-        self.logger = None
-        self.running = False
-
-    def initialize(self, config_file: str = None):
-        """初始化应用"""
-        try:
-            # 加载配置
-            self.config = load_config(config_file)
-
-            # 初始化日志
-            self.logger = init_logger(config=self.config)
-
-            logger.info("=== 广西政府采购网公告监控系统启动 ===")
-            logger.info(f"版本: 1.0.0")
-            logger.info(f"配置文件: {config_file or '默认配置'}")
-
-            # 系统健康检查
-            if not check_system_health():
-                logger.warning("系统健康检查失败，但继续运行")
-
-            # 初始化存储
-            init_storage()
-
-            logger.info("应用初始化完成")
-            return True
-
-        except Exception as e:
-            print(f"应用初始化失败: {str(e)}", file=sys.stderr)
-            return False
-
-    def run_crawl(self, keywords: list = None, sources: list = None, max_pages: int = None):
-        """执行爬取任务"""
-        try:
-            logger.info("开始执行爬取任务")
-
-            # 执行爬取
-            crawl_results = crawl_announcements(keywords, sources)
-
-            if not crawl_results:
-                logger.info("爬取完成：无数据")
-                return {"success": True, "results": []}
-
-            # 收集所有公告
-            all_announcements = []
-            for result in crawl_results:
-                if result.announcements:
-                    all_announcements.extend(result.announcements)
-
-            logger.info(f"爬取到 {len(all_announcements)} 条原始公告")
-
-            # 先保存所有公告（按来源分组，每源保留最新100条）
-            all_saved_stats = save_all_announcements_by_source_to_storage(all_announcements, max_per_source=100)
-            all_saved_count = sum(all_saved_stats.values())
-            logger.info(f"保存所有公告完成：共保存 {all_saved_count} 条，按来源统计: {all_saved_stats}")
-
-            # 筛选公告
-            filter_obj = filter_from_config()
-            filtered_announcements, filter_stats = filter_obj.filter(all_announcements)
-
-            logger.info(f"筛选后剩余 {len(filtered_announcements)} 条公告")
-
-            # 保存筛选后的公告（用于标记关键词匹配等）
-            saved_count = save_announcements_to_storage(filtered_announcements)
-
-            # 生成Markdown文件
-            md_success = generate_onu_md(filtered_announcements)
-
-            # 发送通知
-            notify_success = False
-            if filtered_announcements and self.config.wechat_app.enabled:
-                notify_success = send_announcements_notification(filtered_announcements)
-
-            result = {
-                "success": True,
-                "total_crawled": len(all_announcements),
-                "filtered": len(filtered_announcements),
-                "saved": saved_count,
-                "markdown_generated": md_success,
-                "notification_sent": notify_success,
-                "filter_stats": {
-                    "keyword_filtered": filter_stats.keyword_filtered,
-                    "date_filtered": filter_stats.date_filtered,
-                    "duplicate_filtered": filter_stats.duplicate_filtered,
-                    "source_filtered": filter_stats.source_filtered
-                }
-            }
-
-            logger.info(f"爬取任务完成: {result}")
-            return result
-
-        except Exception as e:
-            logger.error(f"爬取任务执行失败: {str(e)}")
-            return {"success": False, "error": str(e)}
-
-    def run_cleanup(self, days: int = None):
-        """执行数据清理任务"""
-        try:
-            logger.info("开始执行数据清理任务")
-
-            deleted_count = cleanup_storage(days)
-
-            logger.info(f"数据清理完成：删除 {deleted_count} 条过期数据")
-
-            # 发送通知
-            if deleted_count > 0 and self.config.wechat_app.enabled:
-                send_system_notification(
-                    "数据清理完成",
-                    f"已清理 {deleted_count} 条过期数据"
-                )
-
-            return {"success": True, "deleted": deleted_count}
-
-        except Exception as e:
-            logger.error(f"数据清理任务执行失败: {str(e)}")
-            return {"success": False, "error": str(e)}
-
-    def run_wechat_server(self, host: str = '0.0.0.0', port: int = 18001):
-        """启动企业微信回调服务器"""
-        if not wechat_available:
-            return {"success": False, "error": "企业微信模块不可用"}
-
-        try:
-            logger.info("启动企业微信回调服务器")
-
-            # 获取回调服务器
-            callback_server = get_callback_server()
-
-            # 启动服务器
-            callback_server.run(host=host, port=port, debug=self.config.debug)
-
-            return {"success": True, "host": host, "port": port}
-
-        except Exception as e:
-            logger.error(f"启动企业微信回调服务器失败: {str(e)}")
-            return {"success": False, "error": str(e)}
-
-    def manage_wechat_menu(self, action: str) -> Dict[str, Any]:
-        """管理企业微信菜单"""
-        if not wechat_available:
-            return {"success": False, "error": "企业微信模块不可用"}
-
-        try:
-            logger.info(f"执行企业微信菜单操作: {action}")
-
-            menu_manager = WeChatMenuManager()
-
-            if action == 'create':
-                success = menu_manager.create_menu()
-                result = {"success": success, "action": "create"}
-            elif action == 'delete':
-                success = menu_manager.delete_menu()
-                result = {"success": success, "action": "delete"}
-            elif action == 'get':
-                menu_info = menu_manager.get_menu()
-                result = {"success": menu_info is not None, "action": "get", "menu": menu_info}
-            elif action == 'test':
-                test_results = menu_manager.test_menu_operations()
-                result = {"success": True, "action": "test", "results": test_results}
-            else:
-                result = {"success": False, "error": f"未知操作: {action}"}
-
-            if result["success"]:
-                logger.info(f"企业微信菜单操作成功: {action}")
-            else:
-                logger.error(f"企业微信菜单操作失败: {action}")
-
-            return result
-
-        except Exception as e:
-            logger.error(f"企业微信菜单管理异常: {str(e)}")
-            return {"success": False, "error": str(e)}
-
-    def show_status(self):
-        """显示系统状态"""
-        try:
-            status = {
-                "system": {
-                    "version": "1.0.0",
-                    "healthy": check_system_health()
-                },
-                "config": {
-                    "debug": self.config.debug,
-                    "log_level": self.config.log_level.value
-                },
-                "database": {
-                    "enabled": self.config.database.enabled,
-                    "type": self.config.database.type
-                },
-                "wechat": {
-                    "enabled": self.config.wechat_app.enabled
-                }
-            }
-
-            # 格式化输出
-            print("\n=== 系统状态 ===")
-            print(f"系统健康: {'正常' if status['system']['healthy'] else '异常'}")
-            print(f"调试模式: {'开启' if status['config']['debug'] else '关闭'}")
-            print(f"日志级别: {status['config']['log_level']}")
-            print(f"数据库: {'启用' if status['database']['enabled'] else '禁用'} ({status['database']['type']})")
-            print(f"企业微信: {'启用' if status['wechat']['enabled'] else '禁用'}")
-
-            return status
-
-        except Exception as e:
-            logger.error(f"获取系统状态失败: {str(e)}")
-            return {"error": str(e)}
-
-
-def create_argument_parser():
-    """创建命令行参数解析器"""
-    parser = argparse.ArgumentParser(
-        description="广西政府采购网公告监控系统",
-        formatter_class=argparse.RawDescriptionHelpFormatter,
-        epilog="""
-使用示例:
-  python main.py crawl                    # 执行一次爬取
-  python main.py crawl --keywords "大化"   # 爬取指定关键词
-  python main.py cleanup                  # 执行数据清理
-  python main.py status                   # 查看系统状态
-  python main.py wechat-server            # 启动企业微信回调服务器
-  python main.py wechat-menu --action create  # 创建企业微信菜单
-        """
-    )
-
-    parser.add_argument(
-        'command',
-        choices=['crawl', 'cleanup', 'status', 'wechat-server', 'wechat-menu'],
-        help='要执行的命令'
-    )
-
-    parser.add_argument(
-        '--config', '-c',
-        help='配置文件路径'
-    )
-
-    # crawl命令的参数
-    parser.add_argument(
-        '--keywords', '-k',
-        nargs='+',
-        help='关键词过滤（多个关键词用空格分隔）'
-    )
-
-    parser.add_argument(
-        '--sources', '-s',
-        nargs='+',
-        help='来源代码过滤（多个来源用空格分隔）'
-    )
-
-    parser.add_argument(
-        '--max-pages',
-        type=int,
-        help='最大爬取页数'
-    )
-
-    # cleanup命令的参数
-    parser.add_argument(
-        '--days', '-d',
-        type=int,
-        help='清理多少天前的过期数据'
-    )
-
-    # wechat-server命令的参数
-    parser.add_argument(
-        '--host', '-H',
-        default='0.0.0.0',
-        help='服务器监听主机地址 (默认: 0.0.0.0)'
-    )
-
-    parser.add_argument(
-        '--port', '-P',
-        type=int,
-        default=18001,
-        help='服务器监听端口 (默认: 18001)'
-    )
-
-    # wechat-menu命令的参数
-    parser.add_argument(
-        '--action', '-a',
-        choices=['create', 'delete', 'get', 'test'],
-        default='create',
-        help='菜单操作类型 (默认: create)'
-    )
-
-    return parser
-
-
-def main():
-    """主函数"""
-    parser = create_argument_parser()
-    args = parser.parse_args()
-
-    # 创建应用实例
-    app = GXGPMonitorApp()
-
-    # 初始化应用
-    if not app.initialize(args.config):
-        sys.exit(1)
-
-    try:
-        if args.command == 'crawl':
-            # 执行爬取
-            result = app.run_crawl(
-                keywords=args.keywords,
-                sources=args.sources,
-                max_pages=args.max_pages
-            )
-
-            if result["success"]:
-                print("✅ 爬取任务执行成功")
-                print(f"   爬取公告: {result['total_crawled']}")
-                print(f"   筛选后: {result['filtered']}")
-                print(f"   保存数量: {result['saved']}")
-                if result.get("markdown_generated"):
-                    print("   Markdown文件: 已生成")
-                if result.get("notification_sent"):
-                    print("   通知发送: 已发送")
-            else:
-                print(f"❌ 爬取任务执行失败: {result.get('error', '未知错误')}")
-                sys.exit(1)
-
-        elif args.command == 'cleanup':
-            # 执行清理
-            result = app.run_cleanup(days=args.days)
-            if result["success"]:
-                print(f"✅ 数据清理完成，删除 {result['deleted']} 条记录")
-            else:
-                print(f"❌ 数据清理失败: {result.get('error', '未知错误')}")
-                sys.exit(1)
-
-        elif args.command == 'status':
-            # 显示状态
-            app.show_status()
-
-        elif args.command == 'wechat-server':
-            # 启动企业微信回调服务器
-            result = app.run_wechat_server(host=args.host, port=args.port)
-            if result["success"]:
-                print(f"✅ 企业微信回调服务器已启动: {result['host']}:{result['port']}")
-                print("   回调地址: /api/v1/wechat/callback")
-            else:
-                print(f"❌ 企业微信回调服务器启动失败: {result.get('error', '未知错误')}")
-                sys.exit(1)
-
-        elif args.command == 'wechat-menu':
-            # 企业微信菜单管理
-            result = app.manage_wechat_menu(action=args.action)
-            if result["success"]:
-                if args.action == 'create':
-                    print("✅ 企业微信菜单创建成功")
-                elif args.action == 'delete':
-                    print("✅ 企业微信菜单删除成功")
-                elif args.action == 'get':
-                    print("✅ 企业微信菜单获取成功")
-                    if result.get("menu"):
-                        print("菜单信息:", json.dumps(result["menu"], indent=2, ensure_ascii=False))
-                elif args.action == 'test':
-                    print("✅ 企业微信菜单测试完成")
-                    print("测试结果:", result.get("results"))
-            else:
-                error = result.get("error", "未知错误")
-                print(f"❌ 企业微信菜单操作失败: {error}")
-                sys.exit(1)
-
-    except KeyboardInterrupt:
-        logger.info("收到中断信号，正在退出...")
-    except Exception as e:
-        logger.error(f"程序执行异常: {str(e)}")
-        print(f"❌ 程序执行异常: {str(e)}", file=sys.stderr)
-        sys.exit(1)
-
-
-if __name__ == "__main__":
-    main()

gx_gp_monitor/notification/__init__.py

@@ -1 +0,0 @@
-"""通知模块"""

gx_gp_monitor/notification/wechat.py

@@ -1,868 +0,0 @@
-"""
-企业微信通知模块
-提供企业微信消息发送功能，支持文本和Markdown格式
-"""
-
-import requests
-import json
-import time
-import hashlib
-from typing import Optional, Dict, Any, List
-from datetime import datetime
-
-try:
-    from ..core.config_manager import get_config
-    from ..core.logger import get_logger
-    from ..core.models import Announcement
-    from ..core.reliability import retry_on_exception, RetryConfig, safe_execute
-    from ..storage.md_generator import AnnouncementMarkdownFormatter
-except ImportError:
-    from core.config_manager import get_config
-    from core.logger import get_logger
-    from core.models import Announcement
-    from core.reliability import retry_on_exception, RetryConfig, safe_execute
-    from storage.md_generator import AnnouncementMarkdownFormatter
-
-
-logger = get_logger(__name__)
-
-
-class WeChatService:
-    """企业微信服务"""
-
-    def __init__(self):
-        self.config = get_config().wechat_app
-        self._access_token = None
-        self._token_expires_at = 0
-
-        logger.info("企业微信服务初始化完成")
-
-    def _get_access_token(self) -> Optional[str]:
-        """
-        获取访问令牌
-
-        Returns:
-            Optional[str]: 访问令牌
-        """
-        current_time = time.time()
-
-        # 检查令牌是否仍然有效
-        if self._access_token and current_time < self._token_expires_at:
-            return self._access_token
-
-        try:
-            # 构建请求URL
-            if self.config.use_proxy and hasattr(self.config, 'proxy_api_url'):
-                url = f"{self.config.proxy_api_url}/cgi-bin/gettoken"
-            else:
-                url = "https://qyapi.weixin.qq.com/cgi-bin/gettoken"
-
-            params = {
-                "corpid": self.config.corp_id,
-                "corpsecret": self.config.secret
-            }
-
-            logger.debug("正在获取企业微信访问令牌")
-
-            response = requests.get(url, params=params, timeout=30)
-            result = response.json()
-
-            if result.get("errcode") == 0:
-                self._access_token = result.get("access_token")
-                # 提前5分钟过期
-                expires_in = result.get("expires_in", 7200) - 300
-                self._token_expires_at = current_time + expires_in
-
-                logger.info("成功获取企业微信访问令牌")
-                return self._access_token
-            else:
-                logger.error(f"获取访问令牌失败: {result}")
-                return None
-
-        except Exception as e:
-            logger.error(f"获取访问令牌异常: {str(e)}")
-            return None
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def send_text_message(self, content: str,
-                         to_user: str = "@all",
-                         to_party: str = "",
-                         to_tag: str = "") -> bool:
-        """
-        发送文本消息
-
-        Args:
-            content: 消息内容
-            to_user: 接收者用户ID，多个用|分隔，@all表示全体
-            to_party: 接收者部门ID，多个用|分隔
-            to_tag: 接收者标签ID，多个用|分隔
-
-        Returns:
-            bool: 发送是否成功
-        """
-        try:
-            access_token = self._get_access_token()
-            if not access_token:
-                logger.error("无法获取访问令牌，发送失败")
-                return False
-
-            # 构建请求URL
-            if self.config.use_proxy and hasattr(self.config, 'proxy_api_url'):
-                url = f"{self.config.proxy_api_url}/cgi-bin/message/send"
-            else:
-                url = "https://qyapi.weixin.qq.com/cgi-bin/message/send"
-
-            params = {"access_token": access_token}
-
-            data = {
-                "touser": to_user,
-                "toparty": to_party,
-                "totag": to_tag,
-                "msgtype": "text",
-                "agentid": self.config.agent_id,
-                "text": {
-                    "content": content
-                }
-            }
-
-            logger.debug(f"发送文本消息: {content[:100]}...")
-
-            response = requests.post(url, params=params, json=data, timeout=30)
-            result = response.json()
-
-            if result.get("errcode") == 0:
-                logger.info("文本消息发送成功")
-                return True
-            else:
-                logger.error(f"文本消息发送失败: {result}")
-                return False
-
-        except Exception as e:
-            logger.error(f"发送文本消息异常: {str(e)}")
-            return False
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def send_markdown_message(self, content: str,
-                             to_user: str = "@all",
-                             to_party: str = "",
-                             to_tag: str = "") -> bool:
-        """
-        发送Markdown消息
-
-        Args:
-            content: Markdown格式的消息内容
-            to_user: 接收者用户ID
-            to_party: 接收者部门ID
-            to_tag: 接收者标签ID
-
-        Returns:
-            bool: 发送是否成功
-        """
-        try:
-            access_token = self._get_access_token()
-            if not access_token:
-                logger.error("无法获取访问令牌，发送失败")
-                return False
-
-            # 构建请求URL
-            if self.config.use_proxy and hasattr(self.config, 'proxy_api_url'):
-                url = f"{self.config.proxy_api_url}/cgi-bin/message/send"
-            else:
-                url = "https://qyapi.weixin.qq.com/cgi-bin/message/send"
-
-            params = {"access_token": access_token}
-
-            data = {
-                "touser": to_user,
-                "toparty": to_party,
-                "totag": to_tag,
-                "msgtype": "markdown",
-                "agentid": self.config.agent_id,
-                "markdown": {
-                    "content": content
-                }
-            }
-
-            logger.debug("发送Markdown消息")
-
-            response = requests.post(url, params=params, json=data, timeout=30)
-            result = response.json()
-
-            if result.get("errcode") == 0:
-                logger.info("Markdown消息发送成功")
-                return True
-            else:
-                logger.error(f"Markdown消息发送失败: {result}")
-                return False
-
-        except Exception as e:
-            logger.error(f"发送Markdown消息异常: {str(e)}")
-            return False
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def send_textcard_message(self, title: str, description: str, url: str,
-                            to_user: str = "@all", to_party: str = "", to_tag: str = "",
-                            btn_txt: str = "查看详情") -> bool:
-        """
-        发送文本卡片消息
-
-        Args:
-            title: 标题
-            description: 描述内容（支持HTML）
-            url: 点击跳转的链接
-            to_user: 接收者用户ID
-            to_party: 接收者部门ID
-            to_tag: 接收者标签ID
-            btn_txt: 按钮文字
-
-        Returns:
-            bool: 发送是否成功
-        """
-        try:
-            access_token = self._get_access_token()
-            if not access_token:
-                logger.error("无法获取访问令牌，发送失败")
-                return False
-
-            # 构建请求URL
-            if self.config.use_proxy and hasattr(self.config, 'proxy_api_url'):
-                url_endpoint = f"{self.config.proxy_api_url}/cgi-bin/message/send"
-            else:
-                url_endpoint = "https://qyapi.weixin.qq.com/cgi-bin/message/send"
-
-            params = {"access_token": access_token}
-
-            data = {
-                "touser": to_user,
-                "toparty": to_party,
-                "totag": to_tag,
-                "msgtype": "textcard",
-                "agentid": self.config.agent_id,
-                "textcard": {
-                    "title": title,
-                    "description": description,
-                    "url": url,
-                    "btntxt": btn_txt
-                },
-                "enable_id_trans": 0,
-                "enable_duplicate_check": 0,
-                "duplicate_check_interval": 1800
-            }
-
-            logger.debug(f"发送文本卡片消息: {title}")
-
-            response = requests.post(url_endpoint, params=params, json=data, timeout=30)
-            result = response.json()
-
-            if result.get("errcode") == 0:
-                logger.info("文本卡片消息发送成功")
-                return True
-            else:
-                logger.error(f"文本卡片消息发送失败: {result}")
-                return False
-
-        except Exception as e:
-            logger.error(f"发送文本卡片消息异常: {str(e)}")
-            return False
-
-    def send_announcement_notification(self, announcements: List[Announcement],
-                                      max_count: int = 20) -> bool:
-        """
-        发送公告通知（每条公告发送一条单独的文本卡片消息）
-
-        Args:
-            announcements: 公告列表
-            max_count: 最大显示数量
-
-        Returns:
-            bool: 是否至少有一条消息发送成功
-        """
-        if not announcements:
-            logger.info("没有新公告，跳过通知")
-            return True
-
-        success_count = 0
-        total_count = len(announcements)
-
-        logger.info(f"开始发送 {total_count} 条公告通知，每条单独发送")
-
-        for i, announcement in enumerate(announcements[:max_count], 1):
-            try:
-                logger.debug(f"发送第 {i}/{min(total_count, max_count)} 条公告: {announcement.title[:30]}...")
-
-                # 为每条公告生成单独的文本卡片
-                if self.send_single_announcement_notification(announcement):
-                    success_count += 1
-                    logger.debug(f"第 {i} 条公告发送成功")
-                else:
-                    logger.warning(f"第 {i} 条公告发送失败: {announcement.title[:30]}...")
-
-                # 添加短暂延迟，避免发送过快
-                if i < len(announcements[:max_count]):
-                    import time
-                    time.sleep(0.5)
-
-            except Exception as e:
-                logger.error(f"发送第 {i} 条公告时发生异常: {str(e)}")
-                continue
-
-        logger.info(f"公告通知发送完成: {success_count}/{min(total_count, max_count)} 条成功")
-
-        if total_count > max_count:
-            logger.info(f"还有 {total_count - max_count} 条公告未发送（超过最大数量限制）")
-
-        return success_count > 0
-
-    def send_single_announcement_notification(self, announcement: Announcement) -> bool:
-        """
-        发送单条公告的通知（文本卡片消息）
-
-        Args:
-            announcement: 单条公告
-
-        Returns:
-            bool: 发送是否成功
-        """
-        try:
-            # 生成单条公告的文本卡片内容
-            title, description, url = self._generate_single_textcard_notification(announcement)
-
-            # 发送文本卡片消息
-            return self.send_textcard_message(title, description, url, btn_txt="查看详情")
-
-        except Exception as e:
-            logger.error(f"发送单条公告通知失败: {str(e)}")
-            return False
-
-    def _generate_single_textcard_notification(self, announcement: Announcement) -> tuple[str, str, str]:
-        """
-        生成单条公告的文本卡片内容
-
-        新格式示例：
-        ---
-        **北海市涠洲岛旅游区管理委员会关于办公桌的网上超市采购项目成交公告**
-
-        工程类公告 | 北海市涠洲岛旅游区管理委员会 | 2026-01-08 09:28
-        ---
-
-        Args:
-            announcement: 单条公告
-
-        Returns:
-            tuple[str, str, str]: (标题, 描述HTML, URL)
-        """
-        # 标题：公告标题（加粗显示，作为卡片标题）
-        announcement_title = announcement.title
-        if len(announcement_title) > 128:  # 企业微信卡片标题限制128字符
-            announcement_title = announcement_title[:125] + "..."
-        title = announcement_title
-
-        # 公告类型映射（英文枚举值 -> 中文显示名称）
-        type_mapping = {
-            "PURCHASE": "采购公告",
-            "RESULT": "结果公告",
-            "CONTRACT": "合同公告",
-            "CORRECTION": "更正公告",
-            "PRE_ANNOUNCEMENT": "招标文件预公示",
-            "SINGLE_SOURCE": "单一来源公示",
-            "ELECTRONIC_MARKET": "电子卖场公示",
-            "ACCEPTANCE": "履约验收公示",
-            "ENGINEERING": "工程类公告",
-            "FRAMEWORK_AGREEMENT": "框架协议征集公告",
-            "FRAMEWORK_RESULT": "框架协议入围结果公告",
-            "FRAMEWORK_SUMMARY": "框架协议成交结果汇总公告",
-            "INTENTION": "采购意向公开"
-        }
-
-        # 获取公告类型的中文显示名称
-        announcement_type_enum = str(announcement.announcement_type).split('.')[-1]
-        announcement_type_display = type_mapping.get(announcement_type_enum, announcement_type_enum)
-
-        # 确定来源名称
-        source_name = announcement.purchase_name if announcement.purchase_name else announcement.source_name
-        if len(source_name) > 25:  # 限制来源名称长度
-            source_name = source_name[:22] + "..."
-
-        # 时间格式化
-        if announcement.publish_date:
-            time_str = announcement.publish_date.strftime("%Y-%m-%d %H:%M")
-        else:
-            time_str = "时间未知"
-
-        # 生成描述：类型 | 来源 | 时间（使用默认颜色）
-        description = f'<div style="font-size: 14px; margin-top: 8px;">{announcement_type_display} | {source_name} | {time_str}</div>'
-
-        # URL：公告详情链接
-        url = announcement.content_url
-
-        return title, description, url
-
-    def _generate_announcement_notification(self, announcements: List[Announcement],
-                                           max_count: int) -> str:
-        """
-        生成公告通知内容（改进版）
-
-        Args:
-            announcements: 公告列表
-            max_count: 最大显示数量
-
-        Returns:
-            str: Markdown格式的通知内容
-        """
-        if not announcements:
-            return f"""# 🔔 广西政府采购网公告更新
-
-**暂无新公告**
-
----
-
-*更新时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}*
-*点击公告标题查看详情*"""
-
-        # 按日期分组
-        today_announcements = []
-        other_announcements = []
-        today = datetime.now().date()
-
-        for announcement in announcements:
-            if announcement.publish_date and announcement.publish_date.date() == today:
-                today_announcements.append(announcement)
-            else:
-                other_announcements.append(announcement)
-
-        lines = []
-
-        # 标题和概要
-        total_count = len(announcements)
-        lines.append("# 🔔 广西政府采购网公告更新")
-        lines.append("")
-        lines.append(f"📊 **共发现 {total_count} 条新公告**")
-        lines.append("")
-
-        # 今日公告
-        if today_announcements:
-            lines.append(f"## 🔥 今日公告 ({len(today_announcements)}条)")
-            lines.append("")
-            display_today = today_announcements[:max_count//2]
-
-            for i, announcement in enumerate(display_today, 1):
-                # 改进标题显示：保留更多字符，但确保美观
-                title = announcement.title
-                if len(title) > 50:
-                    title = title[:47] + "..."
-
-                # 显示时间
-                time_str = announcement.publish_date.strftime("%H:%M") if announcement.publish_date else "N/A"
-
-                # 添加序号和更好的格式
-                lines.append(f"**{i}.** [{title}]({announcement.content_url})")
-                lines.append(f"   ⏰ {time_str} | 📍 {announcement.source_name}")
-                lines.append("")
-
-            if len(today_announcements) > len(display_today):
-                lines.append(f"⚠️ 还有 {len(today_announcements) - len(display_today)} 条今日公告未显示")
-                lines.append("")
-
-        # 其他公告
-        if other_announcements:
-            lines.append(f"## 📄 其他公告 ({len(other_announcements)}条)")
-            lines.append("")
-            remaining_slots = max_count - len(today_announcements) if today_announcements else max_count
-            display_other = other_announcements[:remaining_slots]
-
-            for i, announcement in enumerate(display_other, 1):
-                title = announcement.title
-                if len(title) > 45:
-                    title = title[:42] + "..."
-
-                date_str = announcement.publish_date.strftime("%m-%d") if announcement.publish_date else "N/A"
-                lines.append(f"**{i}.** [{title}]({announcement.content_url}) - {date_str}")
-
-            if len(other_announcements) > len(display_other):
-                lines.append(f"⚠️ 还有 {len(other_announcements) - len(display_other)} 条历史公告未显示")
-            lines.append("")
-
-        # 统计信息 - 改进版
-        lines.append("## 📈 数据统计")
-        lines.append("")
-
-        # 按来源统计
-        source_stats = {}
-        for announcement in announcements:
-            source = announcement.source_name
-            source_stats[source] = source_stats.get(source, 0) + 1
-
-        # 按类型统计
-        type_stats = {}
-        for announcement in announcements:
-            ann_type = str(announcement.announcement_type).split('.')[-1]  # 获取枚举名称
-            type_stats[ann_type] = type_stats.get(ann_type, 0) + 1
-
-        lines.append("**按来源统计：**")
-        for source, count in sorted(source_stats.items()):
-            lines.append(f"• {source}: {count}条")
-        lines.append("")
-
-        lines.append("**按类型统计：**")
-        for ann_type, count in sorted(type_stats.items()):
-            lines.append(f"• {ann_type}: {count}条")
-        lines.append("")
-
-        # 分割线和时间
-        lines.append("---")
-        lines.append("")
-        lines.append(f"🕒 *更新时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}*")
-        lines.append("💡 *点击公告标题查看详情*")
-
-        return "\n".join(lines)
-
-    def _generate_textcard_notification(self, announcements: List[Announcement],
-                                       max_count: int) -> tuple[str, str, str]:
-        """
-        生成文本卡片格式的通知内容
-
-        Args:
-            announcements: 公告列表
-            max_count: 最大显示数量
-
-        Returns:
-            tuple[str, str, str]: (标题, 描述HTML, URL)
-        """
-        # 按日期分组
-        today_announcements = []
-        other_announcements = []
-        today = datetime.now().date()
-
-        for announcement in announcements:
-            if announcement.publish_date and announcement.publish_date.date() == today:
-                today_announcements.append(announcement)
-            else:
-                other_announcements.append(announcement)
-
-        # 生成标题
-        total_count = len(announcements)
-        title = f"🔔 广西政府采购网公告更新 ({total_count}条)"
-
-        # 生成描述HTML
-        html_parts = []
-
-        # 总统计
-        html_parts.append('<div class="highlight">📊 发现 {total_count} 条新公告</div>'.format(total_count=total_count))
-        html_parts.append("")
-
-        # 今日公告
-        if today_announcements:
-            html_parts.append('<div class="normal">🔥 今日公告 ({count}条)</div>'.format(count=len(today_announcements)))
-
-            display_today = today_announcements[:max_count//2]
-            for i, announcement in enumerate(display_today, 1):
-                # 标题处理
-                ann_title = announcement.title
-                if len(ann_title) > 35:  # 文本卡片标题较短
-                    ann_title = ann_title[:32] + "..."
-
-                # 时间和来源
-                time_str = announcement.publish_date.strftime("%H:%M") if announcement.publish_date else "N/A"
-                source = announcement.source_name[:10]  # 限制来源名称长度
-
-                html_parts.append('{i}. <a href="{url}">{title}</a>'.format(
-                    i=i, url=announcement.content_url, title=ann_title))
-                html_parts.append('<div class="gray">⏰ {time} | 📍 {source}</div>'.format(
-                    time=time_str, source=source))
-
-            if len(today_announcements) > len(display_today):
-                remaining = len(today_announcements) - len(display_today)
-                html_parts.append('<div class="gray">还有 {remaining} 条今日公告...</div>'.format(remaining=remaining))
-
-        # 其他公告
-        if other_announcements:
-            html_parts.append("")
-            html_parts.append('<div class="normal">📄 其他公告 ({count}条)</div>'.format(count=len(other_announcements)))
-
-            remaining_slots = max_count - len(today_announcements) if today_announcements else max_count
-            display_other = other_announcements[:remaining_slots]
-
-            for i, announcement in enumerate(display_other, 1):
-                ann_title = announcement.title
-                if len(ann_title) > 30:
-                    ann_title = ann_title[:27] + "..."
-
-                date_str = announcement.publish_date.strftime("%m-%d") if announcement.publish_date else "N/A"
-                html_parts.append('{i}. <a href="{url}">{title}</a> <span class="gray">({date})</span>'.format(
-                    i=i, url=announcement.content_url, title=ann_title, date=date_str))
-
-            if len(other_announcements) > len(display_other):
-                remaining = len(other_announcements) - len(display_other)
-                html_parts.append('<div class="gray">还有 {remaining} 条历史公告...</div>'.format(remaining=remaining))
-
-        # 统计信息
-        html_parts.append("")
-        html_parts.append('<div class="normal">📈 数据统计</div>')
-
-        # 按来源统计
-        source_stats = {}
-        for announcement in announcements:
-            source = announcement.source_name
-            source_stats[source] = source_stats.get(source, 0) + 1
-
-        html_parts.append('<div class="gray">按来源: {stats}</div>'.format(
-            stats=" | ".join([f"{source}:{count}" for source, count in sorted(source_stats.items())])))
-
-        # 时间戳
-        update_time = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
-        html_parts.append("")
-        html_parts.append('<div class="gray">🕒 更新时间: {time}</div>'.format(time=update_time))
-
-        description = "\n".join(html_parts)
-
-        # 限制描述长度（企业微信文本卡片description不超过512字符）
-        if len(description) > 500:
-            description = description[:497] + "..."
-
-        # 生成跳转URL（可以跳转到公告列表页面或第一条公告）
-        if announcements:
-            url = announcements[0].content_url  # 默认跳转到第一条公告
-        else:
-            url = "https://zfcg.gxzf.gov.cn"  # 默认跳转到网站首页
-
-        return title, description, url
-
-    def send_system_notification(self, title: str, content: str,
-                                message_type: str = "text") -> bool:
-        """
-        发送系统通知
-
-        Args:
-            title: 通知标题
-            content: 通知内容
-            message_type: 消息类型 (text/markdown)
-
-        Returns:
-            bool: 发送是否成功
-        """
-        try:
-            if message_type == "markdown":
-                full_content = f"# {title}\n\n{content}"
-                return self.send_markdown_message(full_content)
-            else:
-                full_content = f"{title}\n\n{content}"
-                return self.send_text_message(full_content)
-
-        except Exception as e:
-            logger.error(f"发送系统通知失败: {str(e)}")
-            return False
-
-    def send_error_notification(self, error_message: str, error_details: Optional[str] = None) -> bool:
-        """
-        发送错误通知
-
-        Args:
-            error_message: 错误消息
-            error_details: 错误详情
-
-        Returns:
-            bool: 发送是否成功
-        """
-        content = f"## ❌ 系统错误\n\n**错误信息**: {error_message}"
-
-        if error_details:
-            content += f"\n\n**错误详情**:\n```\n{error_details}\n```"
-
-        content += f"\n\n*发生时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}*"
-
-        return self.send_markdown_message(content)
-
-    def test_connection(self) -> bool:
-        """
-        测试连接
-
-        Returns:
-            bool: 连接是否正常
-        """
-        try:
-            token = self._get_access_token()
-            return token is not None
-        except Exception as e:
-            logger.error(f"企业微信连接测试失败: {str(e)}")
-            return False
-
-    def get_service_status(self) -> Dict[str, Any]:
-        """
-        获取服务状态
-
-        Returns:
-            Dict[str, Any]: 服务状态信息
-        """
-        return {
-            "service": "wechat",
-            "enabled": self.config.enabled,
-            "corp_id": self.config.corp_id[:10] + "..." if self.config.corp_id else None,
-            "agent_id": self.config.agent_id,
-            "has_token": self._access_token is not None,
-            "token_expires_at": datetime.fromtimestamp(self._token_expires_at).isoformat() if self._token_expires_at > 0 else None,
-            "use_proxy": self.config.use_proxy,
-            "connection_test": self.test_connection() if self.config.enabled else False
-        }
-
-
-class NotificationManager:
-    """通知管理器"""
-
-    def __init__(self):
-        self.wechat = WeChatService()
-        self._services = {
-            "wechat": self.wechat
-        }
-
-    def send_announcement_notification(self, announcements: List[Announcement]) -> Dict[str, bool]:
-        """
-        发送公告通知
-
-        Args:
-            announcements: 公告列表
-
-        Returns:
-            Dict[str, bool]: 各服务发送结果
-        """
-        results = {}
-
-        # 企业微信通知
-        if self.wechat.config.enabled:
-            try:
-                results["wechat"] = self.wechat.send_announcement_notification(announcements)
-            except Exception as e:
-                logger.error(f"企业微信通知失败: {str(e)}")
-                results["wechat"] = False
-        else:
-            results["wechat"] = None  # 未启用
-
-        return results
-
-    def send_system_notification(self, title: str, content: str) -> Dict[str, bool]:
-        """
-        发送系统通知
-
-        Args:
-            title: 通知标题
-            content: 通知内容
-
-        Returns:
-            Dict[str, bool]: 发送结果
-        """
-        results = {}
-
-        if self.wechat.config.enabled:
-            try:
-                results["wechat"] = self.wechat.send_system_notification(title, content, "markdown")
-            except Exception as e:
-                logger.error(f"企业微信系统通知失败: {str(e)}")
-                results["wechat"] = False
-        else:
-            results["wechat"] = None
-
-        return results
-
-    def send_error_notification(self, error_message: str, error_details: Optional[str] = None) -> Dict[str, bool]:
-        """
-        发送错误通知
-
-        Args:
-            error_message: 错误消息
-            error_details: 错误详情
-
-        Returns:
-            Dict[str, bool]: 发送结果
-        """
-        results = {}
-
-        if self.wechat.config.enabled:
-            try:
-                results["wechat"] = self.wechat.send_error_notification(error_message, error_details)
-            except Exception as e:
-                logger.error(f"企业微信错误通知失败: {str(e)}")
-                results["wechat"] = False
-        else:
-            results["wechat"] = None
-
-        return results
-
-    def get_status(self) -> Dict[str, Any]:
-        """
-        获取通知服务状态
-
-        Returns:
-            Dict[str, Any]: 服务状态
-        """
-        return {
-            "services": {
-                name: service.get_service_status() for name, service in self._services.items()
-            }
-        }
-
-
-# 全局通知管理器实例
-_notification_manager = None
-
-
-def get_notification_manager() -> NotificationManager:
-    """
-    获取通知管理器实例
-
-    Returns:
-        NotificationManager: 通知管理器实例
-    """
-    global _notification_manager
-    if _notification_manager is None:
-        _notification_manager = NotificationManager()
-    return _notification_manager
-
-
-def send_announcements_notification(announcements: List[Announcement]) -> bool:
-    """
-    发送公告通知
-
-    Args:
-        announcements: 公告列表
-
-    Returns:
-        bool: 是否至少有一个服务发送成功
-    """
-    manager = get_notification_manager()
-    results = manager.send_announcement_notification(announcements)
-
-    # 检查是否有服务发送成功
-    return any(result for result in results.values() if result is True)
-
-
-def send_system_notification(title: str, content: str) -> bool:
-    """
-    发送系统通知
-
-    Args:
-        title: 通知标题
-        content: 通知内容
-
-    Returns:
-        bool: 是否至少有一个服务发送成功
-    """
-    manager = get_notification_manager()
-    results = manager.send_system_notification(title, content)
-
-    return any(result for result in results.values() if result is True)
-
-
-def send_error_alert(error_message: str, error_details: Optional[str] = None) -> bool:
-    """
-    发送错误警报
-
-    Args:
-        error_message: 错误消息
-        error_details: 错误详情
-
-    Returns:
-        bool: 是否至少有一个服务发送成功
-    """
-    manager = get_notification_manager()
-    results = manager.send_error_notification(error_message, error_details)
-
-    return any(result for result in results.values() if result is True)

gx_gp_monitor/notification/wechat_backup.py

@@ -1,555 +0,0 @@
-"""
-企业微信通知模块
-提供企业微信消息发送功能，支持文本和Markdown格式
-"""
-
-import requests
-import json
-import time
-import hashlib
-from typing import Optional, Dict, Any, List
-from datetime import datetime
-
-try:
-    from ..core.config_manager import get_config
-    from ..core.logger import get_logger
-    from ..core.models import Announcement
-    from ..core.reliability import retry_on_exception, RetryConfig, safe_execute
-    from ..storage.md_generator import AnnouncementMarkdownFormatter
-except ImportError:
-    from core.config_manager import get_config
-    from core.logger import get_logger
-    from core.models import Announcement
-    from core.reliability import retry_on_exception, RetryConfig, safe_execute
-    from storage.md_generator import AnnouncementMarkdownFormatter
-
-
-logger = get_logger(__name__)
-
-
-class WeChatService:
-    """企业微信服务"""
-
-    def __init__(self):
-        self.config = get_config().wechat_app
-        self._access_token = None
-        self._token_expires_at = 0
-
-        logger.info("企业微信服务初始化完成")
-
-    def _get_access_token(self) -> Optional[str]:
-        """
-        获取访问令牌
-
-        Returns:
-            Optional[str]: 访问令牌
-        """
-        current_time = time.time()
-
-        # 检查令牌是否仍然有效
-        if self._access_token and current_time < self._token_expires_at:
-            return self._access_token
-
-        try:
-            # 构建请求URL
-            if self.config.use_proxy and hasattr(self.config, 'proxy_api_url'):
-                url = f"{self.config.proxy_api_url}/cgi-bin/gettoken"
-            else:
-                url = "https://qyapi.weixin.qq.com/cgi-bin/gettoken"
-
-            params = {
-                "corpid": self.config.corp_id,
-                "corpsecret": self.config.secret
-            }
-
-            logger.debug("正在获取企业微信访问令牌")
-
-            response = requests.get(url, params=params, timeout=30)
-            result = response.json()
-
-            if result.get("errcode") == 0:
-                self._access_token = result.get("access_token")
-                # 提前5分钟过期
-                expires_in = result.get("expires_in", 7200) - 300
-                self._token_expires_at = current_time + expires_in
-
-                logger.info("成功获取企业微信访问令牌")
-                return self._access_token
-            else:
-                logger.error(f"获取访问令牌失败: {result}")
-                return None
-
-        except Exception as e:
-            logger.error(f"获取访问令牌异常: {str(e)}")
-            return None
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def send_text_message(self, content: str,
-                         to_user: str = "@all",
-                         to_party: str = "",
-                         to_tag: str = "") -> bool:
-        """
-        发送文本消息
-
-        Args:
-            content: 消息内容
-            to_user: 接收者用户ID，多个用|分隔，@all表示全体
-            to_party: 接收者部门ID，多个用|分隔
-            to_tag: 接收者标签ID，多个用|分隔
-
-        Returns:
-            bool: 发送是否成功
-        """
-        try:
-            access_token = self._get_access_token()
-            if not access_token:
-                logger.error("无法获取访问令牌，发送失败")
-                return False
-
-            # 构建请求URL
-            if self.config.use_proxy and hasattr(self.config, 'proxy_api_url'):
-                url = f"{self.config.proxy_api_url}/cgi-bin/message/send"
-            else:
-                url = "https://qyapi.weixin.qq.com/cgi-bin/message/send"
-
-            params = {"access_token": access_token}
-
-            data = {
-                "touser": to_user,
-                "toparty": to_party,
-                "totag": to_tag,
-                "msgtype": "text",
-                "agentid": self.config.agent_id,
-                "text": {
-                    "content": content
-                }
-            }
-
-            logger.debug(f"发送文本消息: {content[:100]}...")
-
-            response = requests.post(url, params=params, json=data, timeout=30)
-            result = response.json()
-
-            if result.get("errcode") == 0:
-                logger.info("文本消息发送成功")
-                return True
-            else:
-                logger.error(f"文本消息发送失败: {result}")
-                return False
-
-        except Exception as e:
-            logger.error(f"发送文本消息异常: {str(e)}")
-            return False
-
-    @retry_on_exception(RetryConfig(max_retries=3))
-    def send_markdown_message(self, content: str,
-                             to_user: str = "@all",
-                             to_party: str = "",
-                             to_tag: str = "") -> bool:
-        """
-        发送Markdown消息
-
-        Args:
-            content: Markdown格式的消息内容
-            to_user: 接收者用户ID
-            to_party: 接收者部门ID
-            to_tag: 接收者标签ID
-
-        Returns:
-            bool: 发送是否成功
-        """
-        try:
-            access_token = self._get_access_token()
-            if not access_token:
-                logger.error("无法获取访问令牌，发送失败")
-                return False
-
-            # 构建请求URL
-            if self.config.use_proxy and hasattr(self.config, 'proxy_api_url'):
-                url = f"{self.config.proxy_api_url}/cgi-bin/message/send"
-            else:
-                url = "https://qyapi.weixin.qq.com/cgi-bin/message/send"
-
-            params = {"access_token": access_token}
-
-            data = {
-                "touser": to_user,
-                "toparty": to_party,
-                "totag": to_tag,
-                "msgtype": "markdown",
-                "agentid": self.config.agent_id,
-                "markdown": {
-                    "content": content
-                }
-            }
-
-            logger.debug("发送Markdown消息")
-
-            response = requests.post(url, params=params, json=data, timeout=30)
-            result = response.json()
-
-            if result.get("errcode") == 0:
-                logger.info("Markdown消息发送成功")
-                return True
-            else:
-                logger.error(f"Markdown消息发送失败: {result}")
-                return False
-
-        except Exception as e:
-            logger.error(f"发送Markdown消息异常: {str(e)}")
-            return False
-
-    def send_announcement_notification(self, announcements: List[Announcement],
-                                      max_count: int = 20) -> bool:
-        """
-        发送公告通知
-
-        Args:
-            announcements: 公告列表
-            max_count: 最大显示数量
-
-        Returns:
-            bool: 发送是否成功
-        """
-        if not announcements:
-            logger.info("没有新公告，跳过通知")
-            return True
-
-        try:
-            # 生成通知内容
-            notification_content = self._generate_announcement_notification(announcements, max_count)
-
-            # 发送Markdown消息
-            return self.send_markdown_message(notification_content)
-
-        except Exception as e:
-            logger.error(f"发送公告通知失败: {str(e)}")
-            return False
-
-    def _generate_announcement_notification(self, announcements: List[Announcement],
-                                           max_count: int) -> str:
-        """
-        生成公告通知内容
-
-        Args:
-            announcements: 公告列表
-            max_count: 最大显示数量
-
-        Returns:
-            str: Markdown格式的通知内容
-        """
-        # 按日期分组
-        today_announcements = []
-        other_announcements = []
-
-        today = datetime.now().date()
-
-        for announcement in announcements:
-            if announcement.publish_date and announcement.publish_date.date() == today:
-                today_announcements.append(announcement)
-            else:
-                other_announcements.append(announcement)
-
-        lines = []
-
-        # 标题
-        total_count = len(announcements)
-        lines.append(f"# 🔔 广西政府采购网公告更新")
-        lines.append("")
-        lines.append(f"**发现 {total_count} 条新公告**")
-        lines.append("")
-
-        # 今日公告
-        if today_announcements:
-            lines.append(f"## 📅 今日公告 ({len(today_announcements)}条)")
-            lines.append("")
-            display_today = today_announcements[:max_count//2]
-            for announcement in display_today:
-                title = announcement.title
-                if len(title) > 40:
-                    title = title[:40] + "..."
-                publish_time = announcement.publish_date.strftime("%H:%M") if announcement.publish_date else "N/A"
-                lines.append(f"• [{title}]({announcement.content_url}) - {publish_time}")
-
-            if len(today_announcements) > len(display_today):
-                lines.append(f"• ... 还有 {len(today_announcements) - len(display_today)} 条今日公告")
-
-            lines.append("")
-
-        # 其他公告
-        if other_announcements:
-            lines.append(f"## 📄 其他公告 ({len(other_announcements)}条)")
-            lines.append("")
-            remaining_slots = max_count - len(today_announcements) if today_announcements else max_count
-            display_other = other_announcements[:remaining_slots]
-
-            for announcement in display_other:
-                title = announcement.title
-                if len(title) > 40:
-                    title = title[:40] + "..."
-                publish_date = announcement.publish_date.strftime("%m-%d") if announcement.publish_date else "N/A"
-                lines.append(f"• [{title}]({announcement.content_url}) - {publish_date}")
-
-            if len(other_announcements) > len(display_other):
-                lines.append(f"• ... 还有 {len(other_announcements) - len(display_other)} 条公告")
-
-            lines.append("")
-
-        # 统计信息
-        source_stats = {}
-        for announcement in announcements:
-            source = announcement.source_name
-            source_stats[source] = source_stats.get(source, 0) + 1
-
-        lines.append("## 📊 统计信息")
-        lines.append("")
-        for source, count in sorted(source_stats.items()):
-            lines.append(f"• {source}: {count}条")
-
-        lines.append("")
-        lines.append("---")
-        lines.append("")
-        lines.append(f"*更新时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}*")
-        lines.append("*点击公告标题查看详情*")
-
-        return "\n".join(lines)
-
-    def send_system_notification(self, title: str, content: str,
-                                message_type: str = "text") -> bool:
-        """
-        发送系统通知
-
-        Args:
-            title: 通知标题
-            content: 通知内容
-            message_type: 消息类型 (text/markdown)
-
-        Returns:
-            bool: 发送是否成功
-        """
-        try:
-            if message_type == "markdown":
-                full_content = f"# {title}\n\n{content}"
-                return self.send_markdown_message(full_content)
-            else:
-                full_content = f"{title}\n\n{content}"
-                return self.send_text_message(full_content)
-
-        except Exception as e:
-            logger.error(f"发送系统通知失败: {str(e)}")
-            return False
-
-    def send_error_notification(self, error_message: str, error_details: Optional[str] = None) -> bool:
-        """
-        发送错误通知
-
-        Args:
-            error_message: 错误消息
-            error_details: 错误详情
-
-        Returns:
-            bool: 发送是否成功
-        """
-        content = f"## ❌ 系统错误\n\n**错误信息**: {error_message}"
-
-        if error_details:
-            content += f"\n\n**错误详情**:\n```\n{error_details}\n```"
-
-        content += f"\n\n*发生时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}*"
-
-        return self.send_markdown_message(content)
-
-    def test_connection(self) -> bool:
-        """
-        测试连接
-
-        Returns:
-            bool: 连接是否正常
-        """
-        try:
-            token = self._get_access_token()
-            return token is not None
-        except Exception as e:
-            logger.error(f"企业微信连接测试失败: {str(e)}")
-            return False
-
-    def get_service_status(self) -> Dict[str, Any]:
-        """
-        获取服务状态
-
-        Returns:
-            Dict[str, Any]: 服务状态信息
-        """
-        return {
-            "service": "wechat",
-            "enabled": self.config.enabled,
-            "corp_id": self.config.corp_id[:10] + "..." if self.config.corp_id else None,
-            "agent_id": self.config.agent_id,
-            "has_token": self._access_token is not None,
-            "token_expires_at": datetime.fromtimestamp(self._token_expires_at).isoformat() if self._token_expires_at > 0 else None,
-            "use_proxy": self.config.use_proxy,
-            "connection_test": self.test_connection() if self.config.enabled else False
-        }
-
-
-class NotificationManager:
-    """通知管理器"""
-
-    def __init__(self):
-        self.wechat = WeChatService()
-        self._services = {
-            "wechat": self.wechat
-        }
-
-    def send_announcement_notification(self, announcements: List[Announcement]) -> Dict[str, bool]:
-        """
-        发送公告通知
-
-        Args:
-            announcements: 公告列表
-
-        Returns:
-            Dict[str, bool]: 各服务发送结果
-        """
-        results = {}
-
-        # 企业微信通知
-        if self.wechat.config.enabled:
-            try:
-                results["wechat"] = self.wechat.send_announcement_notification(announcements)
-            except Exception as e:
-                logger.error(f"企业微信通知失败: {str(e)}")
-                results["wechat"] = False
-        else:
-            results["wechat"] = None  # 未启用
-
-        return results
-
-    def send_system_notification(self, title: str, content: str) -> Dict[str, bool]:
-        """
-        发送系统通知
-
-        Args:
-            title: 通知标题
-            content: 通知内容
-
-        Returns:
-            Dict[str, bool]: 发送结果
-        """
-        results = {}
-
-        if self.wechat.config.enabled:
-            try:
-                results["wechat"] = self.wechat.send_system_notification(title, content, "markdown")
-            except Exception as e:
-                logger.error(f"企业微信系统通知失败: {str(e)}")
-                results["wechat"] = False
-        else:
-            results["wechat"] = None
-
-        return results
-
-    def send_error_notification(self, error_message: str, error_details: Optional[str] = None) -> Dict[str, bool]:
-        """
-        发送错误通知
-
-        Args:
-            error_message: 错误消息
-            error_details: 错误详情
-
-        Returns:
-            Dict[str, bool]: 发送结果
-        """
-        results = {}
-
-        if self.wechat.config.enabled:
-            try:
-                results["wechat"] = self.wechat.send_error_notification(error_message, error_details)
-            except Exception as e:
-                logger.error(f"企业微信错误通知失败: {str(e)}")
-                results["wechat"] = False
-        else:
-            results["wechat"] = None
-
-        return results
-
-    def get_status(self) -> Dict[str, Any]:
-        """
-        获取通知服务状态
-
-        Returns:
-            Dict[str, Any]: 服务状态
-        """
-        return {
-            "services": {
-                name: service.get_service_status() for name, service in self._services.items()
-            }
-        }
-
-
-# 全局通知管理器实例
-_notification_manager = None
-
-
-def get_notification_manager() -> NotificationManager:
-    """
-    获取通知管理器实例
-
-    Returns:
-        NotificationManager: 通知管理器实例
-    """
-    global _notification_manager
-    if _notification_manager is None:
-        _notification_manager = NotificationManager()
-    return _notification_manager
-
-
-def send_announcements_notification(announcements: List[Announcement]) -> bool:
-    """
-    发送公告通知
-
-    Args:
-        announcements: 公告列表
-
-    Returns:
-        bool: 是否至少有一个服务发送成功
-    """
-    manager = get_notification_manager()
-    results = manager.send_announcement_notification(announcements)
-
-    # 检查是否有服务发送成功
-    return any(result for result in results.values() if result is True)
-
-
-def send_system_notification(title: str, content: str) -> bool:
-    """
-    发送系统通知
-
-    Args:
-        title: 通知标题
-        content: 通知内容
-
-    Returns:
-        bool: 是否至少有一个服务发送成功
-    """
-    manager = get_notification_manager()
-    results = manager.send_system_notification(title, content)
-
-    return any(result for result in results.values() if result is True)
-
-
-def send_error_alert(error_message: str, error_details: Optional[str] = None) -> bool:
-    """
-    发送错误警报
-
-    Args:
-        error_message: 错误消息
-        error_details: 错误详情
-
-    Returns:
-        bool: 是否至少有一个服务发送成功
-    """
-    manager = get_notification_manager()
-    results = manager.send_error_notification(error_message, error_details)
-
-    return any(result for result in results.values() if result is True)

gx_gp_monitor/onu.md

@@ -1,11 +0,0 @@
-# 广西政府采购网公告监控
-
-**更新时间**: 2026-01-08 08:17:34
-
-## 无新公告
-
-当前时间范围内没有找到符合条件的公告。
-
----
-
-*由广西政府采购网公告监控系统生成*

gx_gp_monitor/requirements.txt

@@ -1,41 +0,0 @@
-# 广西政府采购网公告监控系统依赖包
-
-# 核心依赖
-requests>=2.28.0          # HTTP请求库
-psycopg2-binary>=2.9.0    # PostgreSQL数据库驱动
-PyYAML>=6.0               # YAML配置文件解析
-python-dateutil>=2.8.0    # 日期时间处理
-
-# 爬虫相关
-fake-useragent>=1.1.0     # 随机User-Agent生成
-lxml>=4.9.0              # XML/HTML解析（备用）
-
-# 调度器
-schedule>=1.2.0          # 定时任务调度
-croniter>=1.4.0          # Cron表达式解析
-
-# 日志和监控
-logging>=0.4.9.6         # 日志处理（Python内置）
-colorama>=0.4.6          # 控制台颜色输出（可选，用于彩色日志）
-
-# 数据处理
-pandas>=1.5.0            # 数据处理（可选，用于复杂数据分析）
-openpyxl>=3.0.10         # Excel文件处理（可选）
-
-# 加密和安全
-cryptography>=39.0.0     # 加密库（用于微信消息加密）
-pycryptodome>=3.17.0     # 加密算法库
-
-# Web框架（企业微信回调服务器）
-flask>=2.3.0            # Web框架
-
-# 可选依赖（根据需要安装）
-# redis>=4.5.0           # Redis缓存（如果需要）
-# sqlalchemy>=2.0.0      # ORM（如果需要更复杂的数据库操作）
-# celery>=5.3.0          # 分布式任务队列（如果需要）
-
-# 开发依赖（仅开发环境需要）
-# pytest>=7.2.0           # 测试框架
-# black>=23.0.0          # 代码格式化
-# flake8>=6.0.0          # 代码检查
-# mypy>=1.0.0            # 类型检查

gx_gp_monitor/scheduler/__init__.py

@@ -1 +0,0 @@
-"""调度模块"""

gx_gp_monitor/scheduler/scheduler.py

@@ -1,372 +0,0 @@
-"""
-调度器模块
-提供定时任务调度功能，支持cron表达式和间隔执行
-"""
-
-import time
-import threading
-from datetime import datetime, timedelta
-from typing import Callable, Dict, Any, Optional, List
-import schedule
-from croniter import croniter
-
-try:
-    from ..core.config_manager import get_config
-    from ..core.logger import get_logger
-except ImportError:
-    from core.config_manager import get_config
-    from core.logger import get_logger
-
-
-logger = get_logger(__name__)
-
-
-class TaskScheduler:
-    """任务调度器"""
-
-    def __init__(self):
-        self.config = get_config().scheduler
-        self._running = False
-        self._thread = None
-        self._jobs = {}
-        self._job_stats = {}
-
-        # 初始化调度
-        if self.config.enabled:
-            self._load_jobs_from_config()
-
-    def _load_jobs_from_config(self):
-        """从配置加载定时任务"""
-        for job_config in self.config.jobs:
-            if job_config.get("enabled", False):
-                job_name = job_config["name"]
-                cron_expr = job_config["cron"]
-
-                # 根据任务名称创建对应的任务函数
-                if job_name == "daily_crawl":
-                    func = self._create_crawl_job()
-                elif job_name == "data_cleanup":
-                    func = self._create_cleanup_job()
-                else:
-                    logger.warning(f"未知的任务类型: {job_name}")
-                    continue
-
-                self.add_cron_job(job_name, cron_expr, func)
-
-    def _create_crawl_job(self) -> Callable:
-        """创建爬取任务"""
-        def crawl_job():
-            try:
-                logger.info("开始执行定时爬取任务")
-
-                # 导入这里避免循环导入
-                try:
-                    from ..crawler.spider import crawl_announcements
-                    from ..filters.filters import filter_from_config
-                    from ..storage.postgresql import save_announcements_to_storage
-                    from ..storage.md_generator import generate_onu_md
-                    from ..notification.wechat import send_announcements_notification
-                except ImportError:
-                    from crawler.spider import crawl_announcements
-                    from filters.filters import filter_from_config
-                    from storage.postgresql import save_announcements_to_storage
-                    from storage.md_generator import generate_onu_md
-                    from notification.wechat import send_announcements_notification
-
-                # 执行爬取
-                crawl_results = crawl_announcements()
-
-                if not crawl_results:
-                    logger.info("定时爬取任务完成：无数据")
-                    return
-
-                # 收集所有公告
-                all_announcements = []
-                for result in crawl_results:
-                    if result.announcements:
-                        all_announcements.extend(result.announcements)
-
-                if not all_announcements:
-                    logger.info("定时爬取任务完成：无新公告")
-                    return
-
-                # 筛选公告
-                filter_obj = filter_from_config()
-                filtered_announcements, filter_stats = filter_obj.filter(all_announcements)
-
-                logger.info(f"筛选结果: {len(all_announcements)} -> {len(filtered_announcements)}")
-
-                # 保存到数据库
-                saved_count = save_announcements_to_storage(filtered_announcements)
-
-                # 生成Markdown文件
-                generate_onu_md(filtered_announcements)
-
-                # 发送通知
-                if filtered_announcements:
-                    send_announcements_notification(filtered_announcements)
-
-                logger.info(f"定时爬取任务完成：处理 {len(filtered_announcements)} 条公告，保存 {saved_count} 条")
-
-            except Exception as e:
-                logger.error(f"定时爬取任务执行失败: {str(e)}")
-                # 发送错误通知
-                from ..notification.wechat import send_error_alert
-                send_error_alert("定时爬取任务失败", str(e))
-
-        return crawl_job
-
-    def _create_cleanup_job(self) -> Callable:
-        """创建数据清理任务"""
-        def cleanup_job():
-            try:
-                logger.info("开始执行数据清理任务")
-
-                try:
-                    from ..storage.postgresql import cleanup_storage
-                except ImportError:
-                    from storage.postgresql import cleanup_storage
-
-                # 执行清理
-                deleted_count = cleanup_storage()
-
-                logger.info(f"数据清理任务完成：删除 {deleted_count} 条过期数据")
-
-                # 发送通知（如果删除的数据较多）
-                if deleted_count > 0:
-                    try:
-                        from ..notification.wechat import send_system_notification
-                    except ImportError:
-                        from notification.wechat import send_system_notification
-                    send_system_notification(
-                        "数据清理完成",
-                        f"已清理 {deleted_count} 条过期数据"
-                    )
-
-            except Exception as e:
-                logger.error(f"数据清理任务执行失败: {str(e)}")
-
-        return cleanup_job
-
-    def add_cron_job(self, name: str, cron_expr: str, func: Callable) -> bool:
-        """
-        添加cron定时任务
-
-        Args:
-            name: 任务名称
-            cron_expr: cron表达式
-            func: 任务函数
-
-        Returns:
-            bool: 添加是否成功
-        """
-        try:
-            # 验证cron表达式
-            croniter(cron_expr)
-
-            # 添加到schedule
-            schedule.every().day.at("00:00").do(func)  # 临时设置，会被替换
-
-            # 存储任务信息
-            self._jobs[name] = {
-                "func": func,
-                "cron": cron_expr,
-                "next_run": None,
-                "last_run": None,
-                "run_count": 0,
-                "error_count": 0
-            }
-
-            logger.info(f"添加定时任务: {name} ({cron_expr})")
-            return True
-
-        except Exception as e:
-            logger.error(f"添加定时任务失败 {name}: {str(e)}")
-            return False
-
-    def add_interval_job(self, name: str, interval_seconds: int, func: Callable) -> bool:
-        """
-        添加间隔执行任务
-
-        Args:
-            name: 任务名称
-            interval_seconds: 执行间隔（秒）
-            func: 任务函数
-
-        Returns:
-            bool: 添加是否成功
-        """
-        try:
-            schedule.every(interval_seconds).seconds.do(func)
-
-            self._jobs[name] = {
-                "func": func,
-                "interval": interval_seconds,
-                "next_run": None,
-                "last_run": None,
-                "run_count": 0,
-                "error_count": 0
-            }
-
-            logger.info(f"添加间隔任务: {name} ({interval_seconds}秒)")
-            return True
-
-        except Exception as e:
-            logger.error(f"添加间隔任务失败 {name}: {str(e)}")
-            return False
-
-    def remove_job(self, name: str) -> bool:
-        """
-        移除任务
-
-        Args:
-            name: 任务名称
-
-        Returns:
-            bool: 移除是否成功
-        """
-        if name in self._jobs:
-            # 注意：schedule库没有直接的移除方法
-            # 这里只是从我们的记录中移除
-            del self._jobs[name]
-            logger.info(f"移除任务: {name}")
-            return True
-
-        return False
-
-    def start(self):
-        """启动调度器"""
-        if self._running:
-            logger.warning("调度器已经在运行中")
-            return
-
-        if not self.config.enabled:
-            logger.info("调度器已禁用")
-            return
-
-        self._running = True
-        self._thread = threading.Thread(target=self._run_scheduler, daemon=True)
-        self._thread.start()
-
-        logger.info("调度器已启动")
-
-    def stop(self):
-        """停止调度器"""
-        if not self._running:
-            return
-
-        self._running = False
-
-        if self._thread and self._thread.is_alive():
-            self._thread.join(timeout=5)
-
-        logger.info("调度器已停止")
-
-    def _run_scheduler(self):
-        """运行调度器主循环"""
-        logger.info("调度器主循环开始")
-
-        while self._running:
-            try:
-                schedule.run_pending()
-                time.sleep(1)
-            except Exception as e:
-                logger.error(f"调度器运行异常: {str(e)}")
-                time.sleep(5)  # 出错后等待5秒再继续
-
-        logger.info("调度器主循环结束")
-
-    def run_once(self, job_name: Optional[str] = None):
-        """
-        手动执行任务一次
-
-        Args:
-            job_name: 任务名称，如果为None则执行所有任务
-        """
-        if job_name:
-            if job_name in self._jobs:
-                job_info = self._jobs[job_name]
-                logger.info(f"手动执行任务: {job_name}")
-
-                try:
-                    job_info["func"]()
-                    job_info["run_count"] += 1
-                    job_info["last_run"] = datetime.now()
-                    logger.info(f"任务 {job_name} 执行完成")
-                except Exception as e:
-                    job_info["error_count"] += 1
-                    logger.error(f"任务 {job_name} 执行失败: {str(e)}")
-            else:
-                logger.error(f"任务不存在: {job_name}")
-        else:
-            # 执行所有任务
-            for name in list(self._jobs.keys()):
-                self.run_once(name)
-
-    def get_status(self) -> Dict[str, Any]:
-        """
-        获取调度器状态
-
-        Returns:
-            Dict[str, Any]: 状态信息
-        """
-        jobs_status = {}
-        for name, job_info in self._jobs.items():
-            jobs_status[name] = {
-                "enabled": True,
-                "run_count": job_info.get("run_count", 0),
-                "error_count": job_info.get("error_count", 0),
-                "last_run": job_info.get("last_run").isoformat() if job_info.get("last_run") else None,
-                "next_run": job_info.get("next_run").isoformat() if job_info.get("next_run") else None
-            }
-
-        return {
-            "enabled": self.config.enabled,
-            "running": self._running,
-            "timezone": self.config.timezone,
-            "jobs": jobs_status
-        }
-
-    def is_running(self) -> bool:
-        """检查调度器是否正在运行"""
-        return self._running
-
-
-# 全局调度器实例
-_scheduler = None
-
-
-def get_scheduler() -> TaskScheduler:
-    """
-    获取调度器实例
-
-    Returns:
-        TaskScheduler: 调度器实例
-    """
-    global _scheduler
-    if _scheduler is None:
-        _scheduler = TaskScheduler()
-    return _scheduler
-
-
-def start_scheduler():
-    """启动调度器"""
-    scheduler = get_scheduler()
-    scheduler.start()
-
-
-def stop_scheduler():
-    """停止调度器"""
-    scheduler = get_scheduler()
-    scheduler.stop()
-
-
-def run_scheduled_jobs(job_name: Optional[str] = None):
-    """
-    手动执行定时任务
-
-    Args:
-        job_name: 任务名称
-    """
-    scheduler = get_scheduler()
-    scheduler.run_once(job_name)

gx_gp_monitor/storage/__init__.py

@@ -1 +0,0 @@
-"""存储模块"""

gx_gp_monitor/storage/md_generator.py

@@ -1,399 +0,0 @@
-"""
-Markdown生成器模块
-生成公告的Markdown格式输出文件
-"""
-
-import os
-from pathlib import Path
-from typing import List, Dict, Any, Optional
-from datetime import datetime
-from collections import defaultdict
-
-try:
-    from ..core.models import Announcement
-    from ..core.config_manager import get_config
-    from ..core.logger import get_logger
-except ImportError:
-    from core.models import Announcement
-    from core.config_manager import get_config
-    from core.logger import get_logger
-
-
-logger = get_logger(__name__)
-
-
-class MarkdownGenerator:
-    """Markdown生成器"""
-
-    def __init__(self, output_file: Optional[str] = None):
-        """
-        初始化Markdown生成器
-
-        Args:
-            output_file: 输出文件路径
-        """
-        self.config = get_config()
-        self.output_file = output_file or self.config.markdown.output_file
-        self.max_entries = self.config.markdown.max_entries
-        self.include_today_highlight = self.config.markdown.include_today_highlight
-
-        # 确保输出目录存在
-        output_path = Path(self.output_file)
-        output_path.parent.mkdir(parents=True, exist_ok=True)
-
-    def generate_markdown(self, announcements: List[Announcement],
-                         title: str = "广西政府采购网公告监控") -> str:
-        """
-        生成Markdown内容
-
-        Args:
-            announcements: 公告列表
-            title: 文档标题
-
-        Returns:
-            str: Markdown格式的文本
-        """
-        if not announcements:
-            return self._generate_empty_markdown(title)
-
-        # 按来源分组
-        grouped_announcements = self._group_announcements_by_source(announcements)
-
-        # 生成Markdown
-        lines = []
-        lines.append(f"# {title}")
-        lines.append("")
-        lines.append(f"**更新时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
-        lines.append(f"**总公告数**: {len(announcements)}")
-        lines.append("")
-
-        # 生成目录
-        lines.extend(self._generate_toc(grouped_announcements))
-        lines.append("---")
-        lines.append("")
-
-        # 生成各来源的公告
-        for source_name, source_announcements in grouped_announcements.items():
-            lines.extend(self._generate_source_section(source_name, source_announcements))
-            lines.append("")
-
-        return "\n".join(lines)
-
-    def _group_announcements_by_source(self, announcements: List[Announcement]) -> Dict[str, List[Announcement]]:
-        """按来源分组公告"""
-        grouped = defaultdict(list)
-
-        for announcement in announcements:
-            grouped[announcement.source_name].append(announcement)
-
-        # 对每个组内的公告按时间倒序排列
-        for source_name in grouped:
-            grouped[source_name].sort(key=lambda x: x.publish_date, reverse=True)
-
-        return dict(grouped)
-
-    def _generate_toc(self, grouped_announcements: Dict[str, List[Announcement]]) -> List[str]:
-        """生成目录"""
-        lines = ["## 目录", ""]
-
-        for source_name, announcements in grouped_announcements.items():
-            # 创建锚点链接
-            anchor = self._create_anchor(source_name)
-            count = len(announcements)
-            lines.append(f"- [{source_name}](#{anchor}) ({count}条)")
-
-        return lines
-
-    def _generate_source_section(self, source_name: str, announcements: List[Announcement]) -> List[str]:
-        """生成来源章节"""
-        lines = []
-        anchor = self._create_anchor(source_name)
-
-        lines.append(f"## {source_name}")
-        lines.append("")
-        lines.append(f"**共 {len(announcements)} 条公告**")
-        lines.append("")
-
-        # 生成公告列表
-        for i, announcement in enumerate(announcements, 1):
-            lines.extend(self._generate_announcement_item(announcement, i))
-
-        return lines
-
-    def _generate_announcement_item(self, announcement: Announcement, index: int) -> List[str]:
-        """生成单个公告项"""
-        lines = []
-
-        # 公告标题
-        title_prefix = ""
-        if self.include_today_highlight and announcement.is_today:
-            title_prefix = "🆕 **【今日】**"
-
-        title_line = f"### {index}. {title_prefix}[{announcement.title}]({announcement.content_url})"
-        lines.append(title_line)
-        lines.append("")
-
-        # 公告信息
-        info_items = []
-
-        if announcement.publish_date:
-            publish_date = announcement.publish_date.strftime("%Y-%m-%d")
-            info_items.append(f"📅 发布时间: {publish_date}")
-
-        if announcement.purchase_name:
-            info_items.append(f"🏢 发布单位: {announcement.purchase_name}")
-
-        info_items.append(f"📄 来源: {announcement.source_name}")
-
-        if announcement.crawled_at:
-            crawled_time = announcement.crawled_at.strftime("%m-%d %H:%M")
-            info_items.append(f"🤖 爬取时间: {crawled_time}")
-
-        if info_items:
-            lines.append(" | ".join(info_items))
-            lines.append("")
-
-        # 如果是新公告，添加标记
-        if announcement.is_new:
-            lines.append("*🚀 新公告*")
-            lines.append("")
-
-        lines.append("---")
-        lines.append("")
-
-        return lines
-
-    def _generate_empty_markdown(self, title: str) -> str:
-        """生成空内容的Markdown"""
-        lines = [
-            f"# {title}",
-            "",
-            f"**更新时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
-            "",
-            "## 无新公告",
-            "",
-            "当前时间范围内没有找到符合条件的公告。",
-            "",
-            "---",
-            "",
-            f"*由广西政府采购网公告监控系统生成*"
-        ]
-
-        return "\n".join(lines)
-
-    def _create_anchor(self, text: str) -> str:
-        """创建锚点链接"""
-        # 移除特殊字符，替换空格为连字符，转为小写
-        import re
-        anchor = re.sub(r'[^\w\s-]', '', text)
-        anchor = re.sub(r'[-\s]+', '-', anchor)
-        return anchor.lower().strip('-')
-
-    def save_to_file(self, announcements: List[Announcement],
-                    title: Optional[str] = None) -> bool:
-        """
-        保存Markdown到文件
-
-        Args:
-            announcements: 公告列表
-            title: 文档标题
-
-        Returns:
-            bool: 保存是否成功
-        """
-        try:
-            markdown_content = self.generate_markdown(announcements, title)
-
-            with open(self.output_file, 'w', encoding='utf-8') as f:
-                f.write(markdown_content)
-
-            logger.info(f"Markdown文件已保存到: {self.output_file} (共 {len(announcements)} 条公告)")
-            return True
-
-        except Exception as e:
-            logger.error(f"保存Markdown文件失败: {str(e)}")
-            return False
-
-    def append_to_file(self, new_announcements: List[Announcement]) -> bool:
-        """
-        追加新公告到现有文件
-
-        Args:
-            new_announcements: 新公告列表
-
-        Returns:
-            bool: 追加是否成功
-        """
-        if not new_announcements:
-            return True
-
-        try:
-            # 读取现有文件
-            existing_content = ""
-            if os.path.exists(self.output_file):
-                with open(self.output_file, 'r', encoding='utf-8') as f:
-                    existing_content = f.read()
-
-            # 如果文件不存在或为空，创建新文件
-            if not existing_content.strip():
-                return self.save_to_file(new_announcements)
-
-            # 解析现有公告（这里简化处理，实际可能需要更复杂的解析）
-            # 为简单起见，我们重新生成完整文件
-            logger.info("重新生成完整Markdown文件")
-            return self.save_to_file(new_announcements)
-
-        except Exception as e:
-            logger.error(f"追加公告到Markdown文件失败: {str(e)}")
-            return False
-
-    def get_file_stats(self) -> Dict[str, Any]:
-        """获取文件统计信息"""
-        stats = {
-            "file_exists": False,
-            "file_size": 0,
-            "last_modified": None,
-            "announcement_count": 0
-        }
-
-        try:
-            if os.path.exists(self.output_file):
-                file_stat = os.stat(self.output_file)
-                stats["file_exists"] = True
-                stats["file_size"] = file_stat.st_size
-                stats["last_modified"] = datetime.fromtimestamp(file_stat.st_mtime).isoformat()
-
-                # 尝试统计公告数量（简单计数）
-                with open(self.output_file, 'r', encoding='utf-8') as f:
-                    content = f.read()
-                    # 统计###开头的行（每个公告的标题行）
-                    stats["announcement_count"] = content.count("### ")
-
-        except Exception as e:
-            logger.warning(f"获取文件统计信息失败: {str(e)}")
-
-        return stats
-
-
-class AnnouncementMarkdownFormatter:
-    """公告Markdown格式化器"""
-
-    @staticmethod
-    def format_announcement_card(announcement: Announcement) -> str:
-        """格式化单个公告为卡片样式"""
-        lines = []
-
-        # 标题
-        emoji = "🆕" if announcement.is_today else "📄"
-        lines.append(f"### {emoji} {announcement.title}")
-        lines.append("")
-
-        # 链接
-        lines.append(f"[查看详情]({announcement.content_url})")
-        lines.append("")
-
-        # 信息表格
-        lines.append("| 属性 | 值 |")
-        lines.append("|------|-----|")
-
-        if announcement.publish_date:
-            lines.append(f"| 发布时间 | {announcement.publish_date.strftime('%Y-%m-%d %H:%M')} |")
-
-        lines.append(f"| 发布单位 | {announcement.purchase_name or 'N/A'} |")
-        lines.append(f"| 来源 | {announcement.source_name} |")
-        lines.append(f"| 公告类型 | {announcement.announcement_type.value} |")
-
-        if announcement.crawled_at:
-            lines.append(f"| 爬取时间 | {announcement.crawled_at.strftime('%m-%d %H:%M')} |")
-
-        lines.append("")
-
-        return "\n".join(lines)
-
-    @staticmethod
-    def format_announcement_list(announcements: List[Announcement]) -> str:
-        """格式化公告列表"""
-        if not announcements:
-            return "*暂无公告*"
-
-        lines = []
-        for announcement in announcements:
-            emoji = "🆕" if announcement.is_today else "•"
-            publish_date = announcement.publish_date.strftime("%m-%d") if announcement.publish_date else "N/A"
-            line = f"{emoji} [{announcement.title}]({announcement.content_url}) - {publish_date}"
-            lines.append(line)
-
-        return "\n".join(lines)
-
-    @staticmethod
-    def format_notification_message(announcements: List[Announcement],
-                                  max_count: int = 10) -> str:
-        """格式化为通知消息"""
-        if not announcements:
-            return "暂无新公告"
-
-        # 只显示前N条
-        display_announcements = announcements[:max_count]
-        remaining_count = len(announcements) - max_count
-
-        lines = [f"🔔 发现 {len(announcements)} 条新公告：", ""]
-
-        for announcement in display_announcements:
-            title = announcement.title[:50] + "..." if len(announcement.title) > 50 else announcement.title
-            publish_date = announcement.publish_date.strftime("%m-%d") if announcement.publish_date else "N/A"
-            lines.append(f"• {title} ({publish_date})")
-
-        if remaining_count > 0:
-            lines.append(f"... 还有 {remaining_count} 条公告")
-
-        lines.append("")
-        lines.append("*点击公告标题查看详情*")
-
-        return "\n".join(lines)
-
-
-def create_markdown_generator(output_file: Optional[str] = None) -> MarkdownGenerator:
-    """
-    创建Markdown生成器实例
-
-    Args:
-        output_file: 输出文件路径
-
-    Returns:
-        MarkdownGenerator: 生成器实例
-    """
-    return MarkdownGenerator(output_file)
-
-
-def generate_onu_md(announcements: List[Announcement]) -> bool:
-    """
-    生成onu.md文件
-
-    Args:
-        announcements: 公告列表
-
-    Returns:
-        bool: 生成是否成功
-    """
-    generator = create_markdown_generator()
-    return generator.save_to_file(announcements, "广西政府采购网公告监控")
-
-
-def update_onu_md(new_announcements: List[Announcement]) -> bool:
-    """
-    更新onu.md文件，追加新公告
-
-    Args:
-        new_announcements: 新公告列表
-
-    Returns:
-        bool: 更新是否成功
-    """
-    generator = create_markdown_generator()
-
-    # 如果文件不存在，创建新文件
-    if not os.path.exists(generator.output_file):
-        return generator.save_to_file(new_announcements)
-
-    # 否则追加新公告
-    return generator.append_to_file(new_announcements)

gx_gp_monitor/storage/postgresql.py

@@ -1,489 +0,0 @@
-"""
-PostgreSQL存储模块
-实现公告数据的PostgreSQL存储和管理
-"""
-
-from typing import List, Dict, Any, Optional
-from datetime import datetime, timedelta
-import threading
-from dataclasses import asdict
-
-try:
-    from ..core.models import Announcement, CrawlResult
-    from ..core.database import get_database_manager, init_database
-    from ..core.logger import get_logger
-    from ..core.reliability import retry_on_exception, RetryConfig, safe_execute
-except ImportError:
-    from core.models import Announcement, CrawlResult
-    from core.database import get_database_manager, init_database
-    from core.logger import get_logger
-    from core.reliability import retry_on_exception, RetryConfig, safe_execute
-
-
-logger = get_logger(__name__)
-
-
-class PostgreSQLStorage:
-    """PostgreSQL存储管理器"""
-
-    def __init__(self):
-        self.db_manager = get_database_manager()
-        self._lock = threading.Lock()
-
-    def save_announcements(self, announcements: List[Announcement]) -> int:
-        """
-        保存公告列表到数据库（经过筛选的公告）
-
-        Args:
-            announcements: 公告列表
-
-        Returns:
-            int: 成功保存的数量
-        """
-        if not announcements:
-            return 0
-
-        logger.info(f"开始保存 {len(announcements)} 条筛选后公告到数据库")
-
-        try:
-            # 批量保存
-            saved_count = self.db_manager.save_announcements_batch(announcements)
-
-            if saved_count > 0:
-                logger.info(f"成功保存 {saved_count} 条筛选后公告到数据库")
-
-                # 标记新公告
-                self._mark_new_announcements(announcements[:saved_count])
-
-            return saved_count
-
-        except Exception as e:
-            logger.error(f"保存筛选后公告到数据库失败: {str(e)}")
-            # 尝试逐个保存
-            return self._save_announcements_fallback(announcements)
-
-    def save_all_announcements_by_source(self, announcements: List[Announcement],
-                                       max_per_source: int = 100) -> Dict[str, int]:
-        """
-        按来源保存所有公告，每个来源保留最新的max_per_source条
-
-        Args:
-            announcements: 所有公告列表（未经关键词筛选）
-            max_per_source: 每个来源最大保留数量
-
-        Returns:
-            Dict[str, int]: 各来源保存的数量
-        """
-        if not announcements:
-            return {}
-
-        logger.info(f"开始按来源保存 {len(announcements)} 条公告，每个来源最多保留 {max_per_source} 条")
-
-        try:
-            # 按来源分组
-            source_groups = {}
-            for announcement in announcements:
-                source_code = announcement.source_code
-                if source_code not in source_groups:
-                    source_groups[source_code] = []
-                source_groups[source_code].append(announcement)
-
-            saved_stats = {}
-
-            for source_code, source_announcements in source_groups.items():
-                # 对每个来源的公告按发布时间排序（最新的在前）
-                sorted_announcements = sorted(
-                    source_announcements,
-                    key=lambda x: x.publish_date or x.crawled_at or datetime.min,
-                    reverse=True
-                )
-
-                # 取最新的max_per_source条
-                to_save = sorted_announcements[:max_per_source]
-
-                # 为这些公告生成哈希
-                for announcement in to_save:
-                    if not announcement.content_hash:
-                        announcement.generate_content_hash()
-
-                # 批量保存
-                saved_count = self.db_manager.save_announcements_batch(to_save)
-
-                saved_stats[source_code] = saved_count
-
-                # 清理该来源超出限制的旧数据
-                if len(sorted_announcements) > max_per_source:
-                    self._cleanup_old_announcements_by_source(source_code, max_per_source)
-
-                logger.info(f"来源 {source_code} 保存了 {saved_count} 条公告")
-
-            total_saved = sum(saved_stats.values())
-            logger.info(f"按来源保存完成，总计保存 {total_saved} 条公告")
-
-            return saved_stats
-
-        except Exception as e:
-            logger.error(f"按来源保存公告失败: {str(e)}")
-            return {}
-
-    def _cleanup_old_announcements_by_source(self, source_code: str, keep_count: int):
-        """
-        清理指定来源超出限制的旧公告
-
-        Args:
-            source_code: 来源代码
-            keep_count: 保留数量
-        """
-        try:
-            # 使用窗口函数删除超出限制的记录
-            sql = """
-            DELETE FROM announcements
-            WHERE source_code = %s
-              AND id IN (
-                SELECT id FROM (
-                  SELECT id,
-                         ROW_NUMBER() OVER (ORDER BY publish_date DESC, crawled_at DESC) as rn
-                  FROM announcements
-                  WHERE source_code = %s
-                ) ranked
-                WHERE rn > %s
-              )
-            """
-
-            with get_db_cursor() as cursor:
-                cursor.execute(sql, (source_code, source_code, keep_count))
-                deleted_count = cursor.rowcount
-
-                if deleted_count > 0:
-                    logger.debug(f"清理来源 {source_code} 的 {deleted_count} 条旧公告")
-
-        except Exception as e:
-            logger.warning(f"清理来源 {source_code} 旧公告失败: {str(e)}")
-
-        except Exception as e:
-            logger.error(f"按来源保存公告失败: {str(e)}")
-            return {}
-
-    def _save_announcements_fallback(self, announcements: List[Announcement]) -> int:
-        """逐个保存公告的降级方案"""
-        logger.info("使用降级方案逐个保存公告")
-
-        saved_count = 0
-        for announcement in announcements:
-            try:
-                if self.db_manager.save_announcement(announcement):
-                    saved_count += 1
-            except Exception as e:
-                logger.warning(f"保存公告失败: {announcement.title[:50]}..., 错误: {str(e)}")
-                continue
-
-        logger.info(f"降级保存完成，成功保存 {saved_count} 条公告")
-        return saved_count
-
-    def _mark_new_announcements(self, announcements: List[Announcement]):
-        """标记新公告"""
-        # 这里可以添加新公告标记逻辑
-        # 由于我们在爬取时已经标记，这里主要是确保数据库中的标记正确
-        pass
-
-    def save_crawl_results(self, results: List[CrawlResult]) -> int:
-        """
-        保存爬取结果
-
-        Args:
-            results: 爬取结果列表
-
-        Returns:
-            int: 成功保存的数量
-        """
-        if not results:
-            return 0
-
-        saved_count = 0
-        for result in results:
-            try:
-                if self.db_manager.save_crawl_result(result):
-                    saved_count += 1
-            except Exception as e:
-                logger.warning(f"保存爬取结果失败: {result.source.name}, 错误: {str(e)}")
-                continue
-
-        logger.info(f"保存爬取结果完成: {saved_count}/{len(results)}")
-        return saved_count
-
-    def get_recent_announcements(self, hours: int = 24,
-                               limit: int = 100) -> List[Announcement]:
-        """
-        获取最近的公告
-
-        Args:
-            hours: 最近小时数
-            limit: 限制数量
-
-        Returns:
-            List[Announcement]: 公告列表
-        """
-        try:
-            return self.db_manager.get_recent_announcements(hours)
-        except Exception as e:
-            logger.error(f"获取最近公告失败: {str(e)}")
-            return []
-
-    def cleanup_expired_data(self, days: Optional[int] = None) -> int:
-        """
-        清理过期数据
-
-        Args:
-            days: 保留天数，如果为None则使用配置默认值
-
-        Returns:
-            int: 清理的记录数
-        """
-        from ..core.config_manager import get_config
-
-        config = get_config()
-        if days is None:
-            days = config.database.data_retention_days
-
-        logger.info(f"开始清理 {days} 天前的过期数据")
-
-        try:
-            deleted_count = self.db_manager.cleanup_expired_data(days)
-
-            if deleted_count > 0:
-                logger.info(f"成功清理 {deleted_count} 条过期数据")
-            else:
-                logger.info("没有找到需要清理的过期数据")
-
-            return deleted_count
-
-        except Exception as e:
-            logger.error(f"清理过期数据失败: {str(e)}")
-            return 0
-
-    def get_statistics(self) -> Dict[str, Any]:
-        """
-        获取存储统计信息
-
-        Returns:
-            Dict[str, Any]: 统计数据
-        """
-        try:
-            stats = self.db_manager.get_statistics()
-            stats.update({
-                "storage_type": "postgresql",
-                "last_cleanup": datetime.now().isoformat()
-            })
-            return stats
-        except Exception as e:
-            logger.error(f"获取存储统计信息失败: {str(e)}")
-            return {
-                "storage_type": "postgresql",
-                "error": str(e),
-                "total_announcements": 0,
-                "last_cleanup": datetime.now().isoformat()
-            }
-
-    def search_announcements(self, keyword: Optional[str] = None,
-                           source_code: Optional[str] = None,
-                           start_date: Optional[datetime] = None,
-                           end_date: Optional[datetime] = None,
-                           limit: int = 50) -> List[Announcement]:
-        """
-        搜索公告
-
-        Args:
-            keyword: 关键词
-            source_code: 来源代码
-            start_date: 开始日期
-            end_date: 结束日期
-            limit: 限制数量
-
-        Returns:
-            List[Announcement]: 搜索结果
-        """
-        # 这里可以实现更复杂的搜索逻辑
-        # 目前使用现有的查询方法
-        try:
-            return self.db_manager.get_announcements(
-                source_code=source_code,
-                start_date=start_date,
-                end_date=end_date,
-                limit=limit
-            )
-        except Exception as e:
-            logger.error(f"搜索公告失败: {str(e)}")
-            return []
-
-    def is_healthy(self) -> bool:
-        """
-        检查存储健康状态
-
-        Returns:
-            bool: 是否健康
-        """
-        try:
-            # 尝试执行一个简单的查询
-            stats = self.get_statistics()
-            return "error" not in stats
-        except Exception as e:
-            logger.error(f"存储健康检查失败: {str(e)}")
-            return False
-
-    def optimize_storage(self):
-        """优化存储性能"""
-        # 这里可以添加数据库优化逻辑，如重建索引、清理碎片等
-        logger.info("开始优化存储性能")
-
-        try:
-            # 执行一些基本的优化操作
-            # 注意：实际的优化命令取决于PostgreSQL版本和配置
-
-            # 这里可以添加具体的优化SQL
-            # 例如：VACUUM, REINDEX等
-
-            logger.info("存储优化完成")
-        except Exception as e:
-            logger.error(f"存储优化失败: {str(e)}")
-
-    def backup_data(self, backup_path: Optional[str] = None) -> bool:
-        """
-        备份数据
-
-        Args:
-            backup_path: 备份文件路径
-
-        Returns:
-            bool: 备份是否成功
-        """
-        # 这里可以实现数据备份逻辑
-        # 可以使用pg_dump或其他备份工具
-
-        logger.info("开始备份数据")
-
-        try:
-            # 实现备份逻辑
-            # 注意：这需要系统权限来执行pg_dump
-
-            logger.info("数据备份完成")
-            return True
-
-        except Exception as e:
-            logger.error(f"数据备份失败: {str(e)}")
-            return False
-
-
-class StorageManager:
-    """存储管理器"""
-
-    def __init__(self):
-        self.postgresql = PostgreSQLStorage()
-        self._current_storage = self.postgresql  # 默认使用PostgreSQL
-
-    def save_announcements(self, announcements: List[Announcement]) -> int:
-        """保存筛选后的公告"""
-        return self._current_storage.save_announcements(announcements)
-
-    def save_all_announcements_by_source(self, announcements: List[Announcement],
-                                       max_per_source: int = 100) -> Dict[str, int]:
-        """按来源保存所有公告"""
-        return self._current_storage.save_all_announcements_by_source(announcements, max_per_source)
-
-    def save_crawl_results(self, results: List[CrawlResult]) -> int:
-        """保存爬取结果"""
-        return self._current_storage.save_crawl_results(results)
-
-    def get_recent_announcements(self, hours: int = 24, limit: int = 100) -> List[Announcement]:
-        """获取最近公告"""
-        return self._current_storage.get_recent_announcements(hours, limit)
-
-    def cleanup_expired_data(self, days: Optional[int] = None) -> int:
-        """清理过期数据"""
-        return self._current_storage.cleanup_expired_data(days)
-
-    def get_statistics(self) -> Dict[str, Any]:
-        """获取统计信息"""
-        return self._current_storage.get_statistics()
-
-    def is_healthy(self) -> bool:
-        """检查健康状态"""
-        return self._current_storage.is_healthy()
-
-    def optimize(self):
-        """优化存储"""
-        self._current_storage.optimize_storage()
-
-    def backup(self, backup_path: Optional[str] = None) -> bool:
-        """备份数据"""
-        return self._current_storage.backup_data(backup_path)
-
-
-# 全局存储管理器实例
-_storage_manager = None
-_storage_lock = threading.Lock()
-
-
-def get_storage_manager() -> StorageManager:
-    """
-    获取存储管理器实例
-
-    Returns:
-        StorageManager: 存储管理器实例
-    """
-    global _storage_manager
-    if _storage_manager is None:
-        with _storage_lock:
-            if _storage_manager is None:
-                _storage_manager = StorageManager()
-    return _storage_manager
-
-
-def init_storage():
-    """初始化存储"""
-    try:
-        init_database()
-        logger.info("存储初始化完成")
-    except Exception as e:
-        logger.error(f"存储初始化失败: {str(e)}")
-        raise
-
-
-def save_announcements_to_storage(announcements: List[Announcement]) -> int:
-    """
-    保存筛选后的公告到存储
-
-    Args:
-        announcements: 公告列表
-
-    Returns:
-        int: 保存成功的数量
-    """
-    return get_storage_manager().save_announcements(announcements)
-
-
-def save_all_announcements_by_source_to_storage(announcements: List[Announcement],
-                                              max_per_source: int = 100) -> Dict[str, int]:
-    """
-    按来源保存所有公告到存储
-
-    Args:
-        announcements: 所有公告列表
-        max_per_source: 每个来源最大保留数量
-
-    Returns:
-        Dict[str, int]: 各来源保存的数量
-    """
-    return get_storage_manager().save_all_announcements_by_source(announcements, max_per_source)
-
-
-def cleanup_storage(days: Optional[int] = None) -> int:
-    """
-    清理存储中的过期数据
-
-    Args:
-        days: 保留天数
-
-    Returns:
-        int: 清理的记录数
-    """
-    return get_storage_manager().cleanup_expired_data(days)