Enhance announcement storage functionality by adding a method to save all announcements by source, retaining the latest 100 per source. Update related logging and comments for clarity. Adjusted the main script to utilize the new storage method and log the saving process. Updated the markdown file timestamp.

gx_gp_monitor/logs/gx_gp_monitor.log

@@ -797,3 +797,62 @@
 2026-01-07 17:36:12 - __main__ - INFO - 筛选后剩余 0 条公告
 2026-01-07 17:36:12 - storage.md_generator - INFO - Markdown文件已保存到: onu.md (共 0 条公告)
 2026-01-07 17:36:12 - __main__ - INFO - 爬取任务完成: {'success': True, 'total_crawled': 1300, 'filtered': 0, 'saved': 0, 'markdown_generated': True, 'notification_sent': False, 'filter_stats': {'keyword_filtered': 1282, 'date_filtered': 0, 'duplicate_filtered': 18, 'source_filtered': 0}}
+2026-01-08 08:17:30 - __main__ - INFO - === 广西政府采购网公告监控系统启动 ===
+2026-01-08 08:17:30 - __main__ - INFO - 版本: 1.0.0
+2026-01-08 08:17:30 - __main__ - INFO - 配置文件: 默认配置
+2026-01-08 08:17:30 - core.reliability - INFO - 健康检查通过
+2026-01-08 08:17:30 - core.database - INFO - 数据库连接池初始化成功
+2026-01-08 08:17:30 - core.database - INFO - 开始初始化数据库表结构
+2026-01-08 08:17:30 - core.database - INFO - 数据库表结构初始化完成
+2026-01-08 08:17:30 - storage.postgresql - INFO - 存储初始化完成
+2026-01-08 08:17:30 - __main__ - INFO - 应用初始化完成
+2026-01-08 08:17:30 - __main__ - INFO - 开始执行爬取任务
+2026-01-08 08:17:30 - crawler.spider - INFO - 开始爬取 13 个公告来源
+2026-01-08 08:17:30 - gx_gp_monitor - INFO - 开始爬取 采购公告
+2026-01-08 08:17:31 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:31 - gx_gp_monitor - INFO - 采购公告 爬取完成，共获取 100 条公告，耗时 0.53秒
+2026-01-08 08:17:31 - gx_gp_monitor - INFO - 开始爬取 结果公告
+2026-01-08 08:17:31 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:31 - gx_gp_monitor - INFO - 结果公告 爬取完成，共获取 100 条公告，耗时 0.19秒
+2026-01-08 08:17:31 - gx_gp_monitor - INFO - 开始爬取 合同公告
+2026-01-08 08:17:31 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:31 - gx_gp_monitor - INFO - 合同公告 爬取完成，共获取 100 条公告，耗时 0.20秒
+2026-01-08 08:17:31 - gx_gp_monitor - INFO - 开始爬取 更正公告
+2026-01-08 08:17:31 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:31 - gx_gp_monitor - INFO - 更正公告 爬取完成，共获取 100 条公告，耗时 0.28秒
+2026-01-08 08:17:31 - gx_gp_monitor - INFO - 开始爬取 招标文件预公示
+2026-01-08 08:17:31 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:31 - gx_gp_monitor - INFO - 招标文件预公示 爬取完成，共获取 100 条公告，耗时 0.18秒
+2026-01-08 08:17:31 - gx_gp_monitor - INFO - 开始爬取 单一来源公示
+2026-01-08 08:17:32 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:32 - gx_gp_monitor - INFO - 单一来源公示 爬取完成，共获取 100 条公告，耗时 0.16秒
+2026-01-08 08:17:32 - gx_gp_monitor - INFO - 开始爬取 电子卖场公示
+2026-01-08 08:17:32 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:32 - gx_gp_monitor - INFO - 电子卖场公示 爬取完成，共获取 100 条公告，耗时 0.34秒
+2026-01-08 08:17:32 - gx_gp_monitor - INFO - 开始爬取 履约验收公示
+2026-01-08 08:17:32 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:32 - gx_gp_monitor - INFO - 履约验收公示 爬取完成，共获取 100 条公告，耗时 0.48秒
+2026-01-08 08:17:32 - gx_gp_monitor - INFO - 开始爬取 工程类公告
+2026-01-08 08:17:33 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:33 - gx_gp_monitor - INFO - 工程类公告 爬取完成，共获取 100 条公告，耗时 0.23秒
+2026-01-08 08:17:33 - gx_gp_monitor - INFO - 开始爬取 框架协议征集公告
+2026-01-08 08:17:33 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:33 - gx_gp_monitor - INFO - 框架协议征集公告 爬取完成，共获取 100 条公告，耗时 0.18秒
+2026-01-08 08:17:33 - gx_gp_monitor - INFO - 开始爬取 框架协议入围结果公告
+2026-01-08 08:17:33 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:33 - gx_gp_monitor - INFO - 框架协议入围结果公告 爬取完成，共获取 100 条公告，耗时 0.23秒
+2026-01-08 08:17:33 - gx_gp_monitor - INFO - 开始爬取 框架协议成交结果汇总公告
+2026-01-08 08:17:33 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:33 - gx_gp_monitor - INFO - 框架协议成交结果汇总公告 爬取完成，共获取 100 条公告，耗时 0.16秒
+2026-01-08 08:17:33 - gx_gp_monitor - INFO - 开始爬取 采购意向公开
+2026-01-08 08:17:33 - crawler.parsers - INFO - 成功解析 100/100 条公告记录
+2026-01-08 08:17:33 - gx_gp_monitor - INFO - 采购意向公开 爬取完成，共获取 100 条公告，耗时 0.19秒
+2026-01-08 08:17:33 - crawler.spider - INFO - 爬取完成: 共处理 13 个来源，成功 13 个，失败 0 个，获取 1300 条公告
+2026-01-08 08:17:33 - __main__ - INFO - 爬取到 1300 条原始公告
+2026-01-08 08:17:33 - filters.filters - INFO - 来源筛选: 1300 -> 1300 条公告
+2026-01-08 08:17:34 - filters.filters - INFO - 去重筛选: 移除了 17 条重复公告
+2026-01-08 08:17:34 - filters.filters - INFO - 关键词筛选: 1283 -> 0 条公告
+2026-01-08 08:17:34 - filters.filters - INFO - 筛选完成: 总数 1300 -> 筛选后 0 (关键词: 1283, 日期: 0, 去重: 17, 来源: 0)
+2026-01-08 08:17:34 - __main__ - INFO - 筛选后剩余 0 条公告
+2026-01-08 08:17:34 - storage.md_generator - INFO - Markdown文件已保存到: onu.md (共 0 条公告)
+2026-01-08 08:17:34 - __main__ - INFO - 爬取任务完成: {'success': True, 'total_crawled': 1300, 'filtered': 0, 'saved': 0, 'markdown_generated': True, 'notification_sent': False, 'filter_stats': {'keyword_filtered': 1283, 'date_filtered': 0, 'duplicate_filtered': 17, 'source_filtered': 0}}

gx_gp_monitor/main.py

@@ -20,7 +20,7 @@ try:
     from .core.reliability import check_system_health
     from .crawler.spider import crawl_announcements
     from .filters.filters import filter_from_config
-    from .storage.postgresql import init_storage, save_announcements_to_storage, cleanup_storage
+    from .storage.postgresql import init_storage, save_announcements_to_storage, save_all_announcements_by_source_to_storage, cleanup_storage
     from .storage.md_generator import generate_onu_md
     from .notification.wechat import send_announcements_notification, send_system_notification
 except ImportError:
@@ -31,7 +31,7 @@ except ImportError:
         from core.reliability import check_system_health
         from crawler.spider import crawl_announcements
         from filters.filters import filter_from_config
-        from storage.postgresql import init_storage, save_announcements_to_storage, cleanup_storage
+        from storage.postgresql import init_storage, save_announcements_to_storage, save_all_announcements_by_source_to_storage, cleanup_storage
         from storage.md_generator import generate_onu_md
         from notification.wechat import send_announcements_notification, send_system_notification
     except ImportError as e:
@@ -98,13 +98,18 @@ class GXGPMonitorApp:
 
             logger.info(f"爬取到 {len(all_announcements)} 条原始公告")
 
+            # 先保存所有公告（按来源分组，每源保留最新100条）
+            all_saved_stats = save_all_announcements_by_source_to_storage(all_announcements, max_per_source=100)
+            all_saved_count = sum(all_saved_stats.values())
+            logger.info(f"保存所有公告完成：共保存 {all_saved_count} 条，按来源统计: {all_saved_stats}")
+
             # 筛选公告
             filter_obj = filter_from_config()
             filtered_announcements, filter_stats = filter_obj.filter(all_announcements)
 
             logger.info(f"筛选后剩余 {len(filtered_announcements)} 条公告")
 
-            # 保存到数据库
+            # 保存筛选后的公告（用于标记关键词匹配等）
             saved_count = save_announcements_to_storage(filtered_announcements)
 
             # 生成Markdown文件

gx_gp_monitor/onu.md

@@ -1,6 +1,6 @@
 # 广西政府采购网公告监控
 
-**更新时间**: 2026-01-07 17:36:12
+**更新时间**: 2026-01-08 08:17:34
 
 ## 无新公告
 

gx_gp_monitor/storage/postgresql.py

@@ -32,7 +32,7 @@ class PostgreSQLStorage:
 
     def save_announcements(self, announcements: List[Announcement]) -> int:
         """
-        保存公告列表到数据库
+        保存公告列表到数据库（经过筛选的公告）
 
         Args:
             announcements: 公告列表
@@ -43,14 +43,14 @@ class PostgreSQLStorage:
         if not announcements:
             return 0
 
-        logger.info(f"开始保存 {len(announcements)} 条公告到数据库")
+        logger.info(f"开始保存 {len(announcements)} 条筛选后公告到数据库")
 
         try:
             # 批量保存
             saved_count = self.db_manager.save_announcements_batch(announcements)
 
             if saved_count > 0:
-                logger.info(f"成功保存 {saved_count} 条公告到数据库")
+                logger.info(f"成功保存 {saved_count} 条筛选后公告到数据库")
 
                 # 标记新公告
                 self._mark_new_announcements(announcements[:saved_count])
@@ -58,10 +58,112 @@ class PostgreSQLStorage:
             return saved_count
 
         except Exception as e:
-            logger.error(f"保存公告到数据库失败: {str(e)}")
+            logger.error(f"保存筛选后公告到数据库失败: {str(e)}")
             # 尝试逐个保存
             return self._save_announcements_fallback(announcements)
 
+    def save_all_announcements_by_source(self, announcements: List[Announcement],
+                                       max_per_source: int = 100) -> Dict[str, int]:
+        """
+        按来源保存所有公告，每个来源保留最新的max_per_source条
+
+        Args:
+            announcements: 所有公告列表（未经关键词筛选）
+            max_per_source: 每个来源最大保留数量
+
+        Returns:
+            Dict[str, int]: 各来源保存的数量
+        """
+        if not announcements:
+            return {}
+
+        logger.info(f"开始按来源保存 {len(announcements)} 条公告，每个来源最多保留 {max_per_source} 条")
+
+        try:
+            # 按来源分组
+            source_groups = {}
+            for announcement in announcements:
+                source_code = announcement.source_code
+                if source_code not in source_groups:
+                    source_groups[source_code] = []
+                source_groups[source_code].append(announcement)
+
+            saved_stats = {}
+
+            for source_code, source_announcements in source_groups.items():
+                # 对每个来源的公告按发布时间排序（最新的在前）
+                sorted_announcements = sorted(
+                    source_announcements,
+                    key=lambda x: x.publish_date or x.crawled_at or datetime.min,
+                    reverse=True
+                )
+
+                # 取最新的max_per_source条
+                to_save = sorted_announcements[:max_per_source]
+
+                # 为这些公告生成哈希
+                for announcement in to_save:
+                    if not announcement.content_hash:
+                        announcement.generate_content_hash()
+
+                # 批量保存
+                saved_count = self.db_manager.save_announcements_batch(to_save)
+
+                saved_stats[source_code] = saved_count
+
+                # 清理该来源超出限制的旧数据
+                if len(sorted_announcements) > max_per_source:
+                    self._cleanup_old_announcements_by_source(source_code, max_per_source)
+
+                logger.info(f"来源 {source_code} 保存了 {saved_count} 条公告")
+
+            total_saved = sum(saved_stats.values())
+            logger.info(f"按来源保存完成，总计保存 {total_saved} 条公告")
+
+            return saved_stats
+
+        except Exception as e:
+            logger.error(f"按来源保存公告失败: {str(e)}")
+            return {}
+
+    def _cleanup_old_announcements_by_source(self, source_code: str, keep_count: int):
+        """
+        清理指定来源超出限制的旧公告
+
+        Args:
+            source_code: 来源代码
+            keep_count: 保留数量
+        """
+        try:
+            # 使用窗口函数删除超出限制的记录
+            sql = """
+            DELETE FROM announcements
+            WHERE source_code = %s
+              AND id IN (
+                SELECT id FROM (
+                  SELECT id,
+                         ROW_NUMBER() OVER (ORDER BY publish_date DESC, crawled_at DESC) as rn
+                  FROM announcements
+                  WHERE source_code = %s
+                ) ranked
+                WHERE rn > %s
+              )
+            """
+
+            with get_db_cursor() as cursor:
+                cursor.execute(sql, (source_code, source_code, keep_count))
+                deleted_count = cursor.rowcount
+
+                if deleted_count > 0:
+                    logger.debug(f"清理来源 {source_code} 的 {deleted_count} 条旧公告")
+
+        except Exception as e:
+            logger.warning(f"清理来源 {source_code} 旧公告失败: {str(e)}")
+
+        except Exception as e:
+            logger.error(f"按来源保存公告失败: {str(e)}")
+            return {}
+
     def _save_announcements_fallback(self, announcements: List[Announcement]) -> int:
         """逐个保存公告的降级方案"""
         logger.info("使用降级方案逐个保存公告")
@@ -279,9 +381,14 @@ class StorageManager:
         self._current_storage = self.postgresql  # 默认使用PostgreSQL
 
     def save_announcements(self, announcements: List[Announcement]) -> int:
-        """保存公告"""
+        """保存筛选后的公告"""
         return self._current_storage.save_announcements(announcements)
 
+    def save_all_announcements_by_source(self, announcements: List[Announcement],
+                                       max_per_source: int = 100) -> Dict[str, int]:
+        """按来源保存所有公告"""
+        return self._current_storage.save_all_announcements_by_source(announcements, max_per_source)
+
     def save_crawl_results(self, results: List[CrawlResult]) -> int:
         """保存爬取结果"""
         return self._current_storage.save_crawl_results(results)
@@ -343,7 +450,7 @@ def init_storage():
 
 def save_announcements_to_storage(announcements: List[Announcement]) -> int:
     """
-    保存公告到存储
+    保存筛选后的公告到存储
 
     Args:
         announcements: 公告列表
@@ -354,6 +461,21 @@ def save_announcements_to_storage(announcements: List[Announcement]) -> int:
     return get_storage_manager().save_announcements(announcements)
 
 
+def save_all_announcements_by_source_to_storage(announcements: List[Announcement],
+                                              max_per_source: int = 100) -> Dict[str, int]:
+    """
+    按来源保存所有公告到存储
+
+    Args:
+        announcements: 所有公告列表
+        max_per_source: 每个来源最大保留数量
+
+    Returns:
+        Dict[str, int]: 各来源保存的数量
+    """
+    return get_storage_manager().save_all_announcements_by_source(announcements, max_per_source)
+
+
 def cleanup_storage(days: Optional[int] = None) -> int:
     """
     清理存储中的过期数据