|
|
@@ -17,373 +17,124 @@ Docker Compose:
|
|
|
docker compose up --build
|
|
|
```
|
|
|
|
|
|
-Default SSE mount (FastMCP):
|
|
|
-- `http://127.0.0.1:8506/mcp/sse`
|
|
|
+Endpoints:
|
|
|
+- MCP: `http://127.0.0.1:8506/mcp/sse`
|
|
|
+- Health: `http://127.0.0.1:8506/health`
|
|
|
+- Dashboard: `http://127.0.0.1:8506/dashboard/`
|
|
|
|
|
|
-Health:
|
|
|
-- `http://127.0.0.1:8506/health`
|
|
|
+## What it does
|
|
|
|
|
|
-## What this server provides
|
|
|
-- Fetches from one or more configured news feeds (`NEWS_FEED_URL` / `NEWS_FEED_URLS`)
|
|
|
-- Deduplicates articles into clusters (v1 fuzzy title similarity)
|
|
|
-- Enriches clusters with configurable LLM providers/models (topic/entities/sentiment/keywords)
|
|
|
-- Applies a case-insensitive entity blacklist after extraction
|
|
|
-- Caches clusters + LLM fields in SQLite
|
|
|
-- Resolves entities in-process via Google Trends suggestions (no `trends-mcp` hop required for entity resolution)
|
|
|
+- Fetches from configured news feeds (`NEWS_FEED_URLS`)
|
|
|
+- **Three-layer dedup**: feed hash → article URL → content hash (detects in-place updates)
|
|
|
+- Clusters articles via title similarity (≥0.75), Jaccard (≥0.55), dual-signal, or embeddings
|
|
|
+- Enriches clusters with LLM: topic, entities, sentiment, keywords, summary
|
|
|
+- Resolves entities via Google Trends suggestions
|
|
|
+- Dashboard with runtime Config page
|
|
|
|
|
|
## Tools (MCP)
|
|
|
|
|
|
-1) `get_latest_events(topic, limit, include_articles=false)`
|
|
|
-- `topic` is a coarse category: `crypto | macro | regulation | ai | other`
|
|
|
-- when `include_articles=true`, includes `articles[].url` + minimal fields per returned cluster
|
|
|
-
|
|
|
-2) `get_events_for_entity(entity, limit, timeframe="24h", include_articles=false)`
|
|
|
-- substring, case-insensitive match over extracted `entities`
|
|
|
-- uses the requested timeframe as the scan window; `limit` is the cap within that window
|
|
|
-- when `include_articles=true`, includes `articles[].url` + minimal fields per returned cluster
|
|
|
-
|
|
|
-3) `get_event_summary(event_id, include_articles=false)`
|
|
|
-- LLM-written compressed narrative for a given `cluster_id`
|
|
|
-- when `include_articles=true`, includes the underlying `articles` list (with `url`) from the stored cluster
|
|
|
-
|
|
|
-4) `detect_emerging_topics(limit)`
|
|
|
-- derives “emerging” signals from recent cached clusters
|
|
|
-
|
|
|
-5) `get_news_sentiment(entity, timeframe)`
|
|
|
-- aggregates sentiment around an entity from cached enriched clusters
|
|
|
-
|
|
|
-6) `get_related_recent_entities(subject, timeframe, limit, include_trends=true)`
|
|
|
-- merges recent co-occurrence data from cached clusters with Google Trends suggestions and returns
|
|
|
- related entities (with `mid` when available) plus source/score metadata
|
|
|
-
|
|
|
-7) `get_capabilities()`
|
|
|
-- describes the server’s tool surface, composition recipes, and output conventions for agents
|
|
|
-
|
|
|
-### Entity aliasing
|
|
|
-
|
|
|
-The server keeps a conservative alias map in `config/entity_aliases.json` for obvious shorthands
|
|
|
-like `btc -> Bitcoin`, `eth -> Ethereum`, and `ether -> Ethereum`. Keep this map tight; it is meant
|
|
|
-to reduce false misses, not to rewrite every possible name variant.
|
|
|
-
|
|
|
-## Configuration
|
|
|
-
|
|
|
-See `news-mcp/.env`.
|
|
|
-Key variables:
|
|
|
-- `NEWS_EXTRACT_PROVIDER`, `NEWS_EXTRACT_MODEL`
|
|
|
-- `NEWS_SUMMARY_PROVIDER`, `NEWS_SUMMARY_MODEL`
|
|
|
-- `GROQ_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY`
|
|
|
-- `ENTITY_BLACKLIST` (comma-separated, case-insensitive patterns; wildcards are supported)
|
|
|
-- `NEWS_PROMPTS_DIR` (override prompt directory)
|
|
|
-- `NEWS_ENTITY_ALIASES_FILE` (override entity alias JSON file)
|
|
|
-- `NEWS_FEED_URL` (single feed fallback)
|
|
|
-- `NEWS_FEED_URLS` (comma-separated feed URLs; overrides `NEWS_FEED_URL`)
|
|
|
-- `NEWS_FEED_ITEMS_PER_POLL` (per-feed fetch cap per poll; default 50)
|
|
|
-- `NEWS_REFRESH_INTERVAL_SECONDS` (default 900)
|
|
|
-- `NEWS_BACKGROUND_REFRESH_ON_START` (default true)
|
|
|
-- `NEWS_BACKGROUND_REFRESH_ENABLED` (default true)
|
|
|
-- `NEWS_DEFAULT_LOOKBACK_HOURS` (freshness window for reads; older rows are ignored by queries)
|
|
|
-- `NEWS_PRUNING_ENABLED` (default true; if false, no rows are physically deleted)
|
|
|
-- `NEWS_RETENTION_DAYS` (physical delete threshold for stored clusters)
|
|
|
-- `NEWS_PRUNE_INTERVAL_HOURS` (how often in-server pruning may run)
|
|
|
-- `ENRICH_OTHER_TOPICS_ONLY` (default false; set true to only LLM-enrich "other" topic clusters)
|
|
|
-- `ENRICHMENT_MAX_PER_REFRESH` (default 0 = no limit; max clusters to LLM-enrich per refresh cycle)
|
|
|
-- `NEWS_LLM_DEBUG` (default false; enable debug logging for LLM calls)
|
|
|
-- `NEWS_LLM_CONCURRENCY_<PROVIDER>` (e.g. `NEWS_LLM_CONCURRENCY_GROQ`; max concurrent outbound LLM calls per provider; overrides the built-in defaults: groq=8, openai=5, openrouter=2)
|
|
|
-- `NEWS_LLM_RATE_LIMIT_<PROVIDER>` (e.g. `NEWS_LLM_RATE_LIMIT_GROQ`; max LLM calls per second per provider. Set to `0` to disable rate limiting. Built-in defaults: groq=1.0, openai=5.0, openrouter=2.0)
|
|
|
-- `NEWS_EMBEDDINGS_ENABLED` (default false; enables Ollama embeddings for clustering)
|
|
|
-- `OLLAMA_BASE_URL` / `OLLAMA_URL` (default `http://127.0.0.1:11434`)
|
|
|
-- `OLLAMA_EMBEDDING_MODEL` (default `nomic-embed-text`)
|
|
|
-- `NEWS_EMBEDDING_SIMILARITY_THRESHOLD` (default `0.885`)
|
|
|
-- `NEWS_CLUSTER_MAX_AGE_HOURS` (default `4`; cross-cycle merge window. Set `0` to disable)
|
|
|
-
|
|
|
-When embeddings are enabled, news-mcp tries Ollama first and falls back to the existing heuristic clustering path if Ollama is unavailable.
|
|
|
-
|
|
|
-## Clustering
|
|
|
-
|
|
|
-The clustering pipeline has two modes:
|
|
|
-
|
|
|
-**In-cycle dedup** (every poll): new articles are compared against each other and against recently loaded existing clusters. A match merges into the existing cluster; no match creates a new cluster.
|
|
|
-
|
|
|
-**Cross-cycle merge** (controlled by `NEWS_CLUSTER_MAX_AGE_HOURS`): before clustering, the poller loads recent clusters from the DB and seeds them as merge targets. This means an article that arrives in poll N+1 can merge into a cluster created in poll N, even if the article's title is different enough that it wouldn't match against the cluster's original seed article. Set to `0` to disable.
|
|
|
-
|
|
|
-**Stable cluster IDs**: cluster IDs are derived from the topic and the lexicographically smallest article key in the cluster, not from the first article's title. This means the same set of articles always resolves to the same `cluster_id` regardless of processing order or polling cycle.
|
|
|
-
|
|
|
-**Orphan merge**: a post-clustering pass detects clusters that share article keys (via Union-Find) and merges them. This catches cases where two articles about the same event didn't match during the main loop (e.g. embeddings were temporarily unavailable).
|
|
|
-
|
|
|
-**Signal cascade**: each new article is compared against all articles in a candidate cluster (not just the seed). The matching cascade is: cosine similarity → title similarity → token Jaccard → consensus (cosine + title/jaccard). The first signal that clears its threshold wins.
|
|
|
-
|
|
|
-## Persistence and migration
|
|
|
-
|
|
|
-The default database path is project-relative:
|
|
|
-
|
|
|
-- `NEWS_MCP_DATA_DIR=./data`
|
|
|
-- `NEWS_MCP_DB_PATH=./data/news.sqlite`
|
|
|
-
|
|
|
-That keeps persistence inside the repository tree in both local and Docker runs.
|
|
|
-
|
|
|
-Recommended workflow:
|
|
|
-
|
|
|
-1. Keep **code** in Git.
|
|
|
-2. Keep the **data directory** outside Git but inside the project tree.
|
|
|
-3. Use `rsync` for the initial data transfer to a remote server.
|
|
|
-4. After that, move code with Git and move data only when you actually need a fresh copy.
|
|
|
-
|
|
|
-Example initial transfer:
|
|
|
-
|
|
|
-```bash
|
|
|
-rsync -a ./data/ user@remote:/srv/news-mcp/data/
|
|
|
-```
|
|
|
-
|
|
|
-If you change the location later, override the defaults with:
|
|
|
-
|
|
|
-- `NEWS_MCP_DATA_DIR`
|
|
|
-- `NEWS_MCP_DB_PATH`
|
|
|
-
|
|
|
-## TTL vs pruning
|
|
|
-
|
|
|
-These are intentionally different:
|
|
|
-
|
|
|
-- `NEWS_DEFAULT_LOOKBACK_HOURS` controls **read freshness** only. Older rows remain in SQLite but do not appear in normal "latest" queries.
|
|
|
-- `NEWS_PRUNING_ENABLED` controls whether the server is allowed to **physically delete** old rows.
|
|
|
-- `NEWS_RETENTION_DAYS` controls how old rows may get before they are deleted.
|
|
|
-- `NEWS_PRUNE_INTERVAL_HOURS` controls how often the server checks whether deletion is due.
|
|
|
-
|
|
|
-Pruning is self-contained inside the server:
|
|
|
-- on startup
|
|
|
-- after refresh cycles (prune-if-due)
|
|
|
-
|
|
|
-If `NEWS_PRUNING_ENABLED=false`, no pruning occurs and old rows are retained indefinitely.
|
|
|
-
|
|
|
-## Live extraction smoke test
|
|
|
-
|
|
|
-Run a standardized, fabricated extraction test against the currently configured provider/model:
|
|
|
-
|
|
|
-```bash
|
|
|
-./live_tests.sh
|
|
|
-```
|
|
|
-
|
|
|
-The script reads `./.env`, selects OpenAI or Groq based on the configured keys, and checks that the core expected entities are extracted.
|
|
|
-
|
|
|
-## mcporter examples (all news-mcp calls)
|
|
|
-
|
|
|
-Use your existing config path:
|
|
|
-
|
|
|
-```bash
|
|
|
-CONFIG=/home/lucky/.openclaw/workspace/config/mcporter.json
|
|
|
-```
|
|
|
-
|
|
|
-Inspect server + tools:
|
|
|
-
|
|
|
-```bash
|
|
|
-mcporter --config "$CONFIG" list news --schema
|
|
|
-```
|
|
|
-
|
|
|
-### 1) Latest events
|
|
|
-
|
|
|
-```bash
|
|
|
-mcporter --config "$CONFIG" call news.get_latest_events topic=crypto limit=10
|
|
|
-mcporter --config "$CONFIG" call news.get_latest_events topic=macro limit=5
|
|
|
-```
|
|
|
-
|
|
|
-### 2) Events for an entity
|
|
|
-
|
|
|
-```bash
|
|
|
-mcporter --config "$CONFIG" call news.get_events_for_entity entity=Bitcoin timeframe=24h limit=10
|
|
|
-mcporter --config "$CONFIG" call news.get_events_for_entity entity=ETH timeframe=3d limit=10
|
|
|
-mcporter --config "$CONFIG" call news.get_events_for_entity entity=ETF timeframe=7d limit=10
|
|
|
-```
|
|
|
-
|
|
|
-### 3) Event summary (by cluster_id)
|
|
|
-
|
|
|
-```bash
|
|
|
-# First fetch an event id
|
|
|
-mcporter --config "$CONFIG" call news.get_latest_events topic=crypto limit=1
|
|
|
-
|
|
|
-# Then summarize it
|
|
|
-mcporter --config "$CONFIG" call news.get_event_summary event_id=<cluster_id>
|
|
|
-```
|
|
|
-
|
|
|
-### 4) Emerging topics
|
|
|
-
|
|
|
-```bash
|
|
|
-mcporter --config "$CONFIG" call news.detect_emerging_topics limit=10
|
|
|
-```
|
|
|
-
|
|
|
-### 5) Sentiment for an entity
|
|
|
-
|
|
|
-```bash
|
|
|
-mcporter --config "$CONFIG" call news.get_news_sentiment entity=Bitcoin timeframe=24h
|
|
|
-mcporter --config "$CONFIG" call news.get_news_sentiment entity=Ethereum timeframe=72h
|
|
|
-```
|
|
|
-
|
|
|
-### 6) Related entities (recent neighborhood + trends blending)
|
|
|
-
|
|
|
-```bash
|
|
|
-# Iran: blend local co-occurrence with Google Trends related topics
|
|
|
-mcporter --config "$CONFIG" call news.get_related_recent_entities subject=Iran timeframe=72h limit=12 include_trends=true
|
|
|
-
|
|
|
-# Another seed phrase
|
|
|
-mcporter --config "$CONFIG" call news.get_related_recent_entities subject="iran war" timeframe=72h limit=12 include_trends=true
|
|
|
-```
|
|
|
-
|
|
|
-### 7) Capabilities / composition guidance
|
|
|
-
|
|
|
-```bash
|
|
|
-mcporter --config "$CONFIG" call news.get_capabilities
|
|
|
-```
|
|
|
-
|
|
|
-Use this when you want the server to explain how to chain the tools together, which fields to keep hidden (e.g. `cluster_id`), and how to present sources/timestamps consistently.
|
|
|
-
|
|
|
-## Blacklist enforcement (optional back-clean)
|
|
|
-
|
|
|
-If you change `ENTITY_BLACKLIST`, existing clusters in `news.sqlite` may still
|
|
|
-contain entities/keywords that would now be filtered at extraction time.
|
|
|
-
|
|
|
-For one-off cleanup, run:
|
|
|
-
|
|
|
-```bash
|
|
|
-./.venv/bin/python scripts/enforce_news_blacklist.py --dry-run --limit 200
|
|
|
-./.venv/bin/python scripts/enforce_news_blacklist.py --limit 1000
|
|
|
-```
|
|
|
-
|
|
|
-This enforces `ENTITY_BLACKLIST` inside stored clusters by removing matching
|
|
|
-entries from `payload.entities` and `payload.keywords` and (if needed) setting
|
|
|
-`payload.topic = "other"`.
|
|
|
-
|
|
|
-## Embeddings backfill (optional)
|
|
|
-
|
|
|
-If `NEWS_EMBEDDINGS_ENABLED=true`, you can precompute cluster embeddings for
|
|
|
-older rows before restarting the server:
|
|
|
-
|
|
|
-```bash
|
|
|
-./.venv/bin/python scripts/backfill_news_embeddings.py --dry-run --limit 200
|
|
|
-./.venv/bin/python scripts/backfill_news_embeddings.py --limit 1000
|
|
|
-```
|
|
|
-
|
|
|
-This stores a cluster-level `embedding` and `embedding_model` inside the SQLite
|
|
|
-payload so the Ollama-first clustering path has data ready to use.
|
|
|
-
|
|
|
-## Embedding merge analysis (optional)
|
|
|
-
|
|
|
-To inspect likely cluster merges at different cosine thresholds without writing
|
|
|
-anything back to the DB:
|
|
|
-
|
|
|
-```bash
|
|
|
-./.venv/bin/python scripts/analyze_cluster_embedding_merges.py --thresholds 0.82 0.85 0.88 --limit 200
|
|
|
-```
|
|
|
-
|
|
|
-This prints candidate pairs per threshold so you can decide whether a merge
|
|
|
-script is worth adding next.
|
|
|
-
|
|
|
-## Embedding merge pass (optional, destructive)
|
|
|
-
|
|
|
-After inspecting the analysis output, you can merge clusters above a chosen
|
|
|
-threshold. Start with dry-run:
|
|
|
-
|
|
|
-```bash
|
|
|
-./.venv/bin/python scripts/merge_cluster_embeddings.py --dry-run --threshold 0.90
|
|
|
-```
|
|
|
-
|
|
|
-If the groupings look right, run wet:
|
|
|
-
|
|
|
-```bash
|
|
|
-./.venv/bin/python scripts/merge_cluster_embeddings.py --threshold 0.90
|
|
|
-```
|
|
|
-
|
|
|
-This merges embedding-similar clusters within the same topic and removes the
|
|
|
-absorbed duplicates from SQLite.
|
|
|
-
|
|
|
-## Article dedup cleanup (optional)
|
|
|
-
|
|
|
-Some stored clusters may contain repeated article entries for the same
|
|
|
-underlying article id / URL path. To clean existing rows:
|
|
|
-
|
|
|
-```bash
|
|
|
-./.venv/bin/python scripts/dedup_articles_in_clusters.py --dry-run
|
|
|
-./.venv/bin/python scripts/dedup_articles_in_clusters.py
|
|
|
-```
|
|
|
-
|
|
|
-The live clustering path also deduplicates article entries when new data comes in.
|
|
|
-
|
|
|
-As of the latest hardening, the server/storage write path also self-heals `payload.articles` by deduplicating before persisting (so historical rows can be fixed via the cleanup script, and future writes won’t reintroduce duplicates).
|
|
|
-```
|
|
|
-
|
|
|
-## Dashboard (new)
|
|
|
-
|
|
|
-A browser-based monitoring dashboard is available at:
|
|
|
-```
|
|
|
-http://127.0.0.1:8506/dashboard/
|
|
|
-```
|
|
|
-
|
|
|
-**Views:**
|
|
|
-- **Health** — cluster/entity counts, freshness indicator, topic distribution (doughnut chart), sentiment overview, feed activity
|
|
|
-- **Clusters** — filterable/sortable table with topic, sentiment, importance, entity chips; search by keyword
|
|
|
-- **Sentiment** — time-series chart (avg sentiment per configurable time bucket) with cluster count overlay
|
|
|
-- **Entities** — top entities by mention frequency, horizontal bar chart, click for detail
|
|
|
-- **Detail** — click any cluster row or search by cluster ID for full drill-down (summary, key facts, articles, keywords, entities)
|
|
|
-
|
|
|
-**Tech:** Pure static HTML/JS with Chart.js for visualizations. Served from the same FastAPI process at `/dashboard`.
|
|
|
-
|
|
|
-**Configuration defaults:** The dashboard's default lookback window follows `NEWS_DEFAULT_LOOKBACK_HOURS` (configured via `.env`, default 144h).
|
|
|
+| Tool | Description |
|
|
|
+|------|-------------|
|
|
|
+| `get_latest_events(topic, limit, include_articles)` | Latest clusters by topic |
|
|
|
+| `get_events_for_entity(entity, limit, timeframe, include_articles)` | Clusters matching entity |
|
|
|
+| `get_event_summary(event_id, include_articles)` | LLM narrative for cluster |
|
|
|
+| `detect_emerging_topics(limit)` | Emerging signals from recent clusters |
|
|
|
+| `get_news_sentiment(entity, timeframe)` | Aggregated sentiment |
|
|
|
+| `get_related_recent_entities(subject, timeframe, limit)` | Co-occurrence + Trends blend |
|
|
|
+| `get_feeds()` / `toggle_feed(url, enabled)` | Feed management |
|
|
|
+| `debug_dedup(url, title?)` | Inspect dedup decisions & similarity signals |
|
|
|
+| `get_capabilities()` | Tool surface documentation |
|
|
|
|
|
|
## REST API
|
|
|
|
|
|
-The following read-only endpoints are available for programmatic access (in addition to the MCP SSE tools):
|
|
|
-
|
|
|
| Method | Path | Description |
|
|
|
|--------|------|-------------|
|
|
|
-| GET | `/api/v1/health` | Extended health: cluster/entity counts, freshness, feed state, pruning config |
|
|
|
-| GET | `/api/v1/clusters` | Paginated clusters. Params: `topic`, `hours`, `limit`, `offset` |
|
|
|
-| GET | `/api/v1/sentiment-series` | Sentiment time-series. Params: `topic`, `hours`, `bucket_hours` |
|
|
|
-| GET | `/api/v1/entities` | Top entities by frequency. Params: `hours`, `limit` |
|
|
|
-| GET | `/api/v1/cluster/{cluster_id}` | Full cluster detail with summary, facts, articles |
|
|
|
-
|
|
|
-## Startup behavior
|
|
|
-
|
|
|
-The server uses a lifespan-based startup (FastAPI ≥0.111). Background feed refresh and pruning run as fire-and-forget coroutines, so the HTTP API and dashboard are available immediately — no blocking on the first fetch cycle.
|
|
|
-
|
|
|
-Important env vars controlling background behavior:
|
|
|
-- `NEWS_BACKGROUND_REFRESH_ENABLED=true` — enable/disable background loop
|
|
|
-- `NEWS_BACKGROUND_REFRESH_ON_START=true` — fetch immediately on startup (previously `false`; changed to `true` for faster first data)
|
|
|
-- `NEWS_REFRESH_INTERVAL_SECONDS=900` — polling interval between refresh cycles
|
|
|
-
|
|
|
-## Dashboard (updated)
|
|
|
-
|
|
|
-A browser-based monitoring dashboard is available at:
|
|
|
-```
|
|
|
-http://<your-host>:8506/dashboard/
|
|
|
-```
|
|
|
-
|
|
|
-**Views:**
|
|
|
-| View | What it shows |
|
|
|
-|---|---|
|
|
|
-| Health | Cluster/entity counts, freshness badge, topic doughnut, sentiment overview, feed activity |
|
|
|
-| Clusters | Filterable table — topic, sentiment, importance, entity chips, full-text search |
|
|
|
-| Sentiment | Time-series line chart (avg sentiment per configurable bucket) + cluster count overlay |
|
|
|
-| Entities | Top entities by mention frequency, bar chart, click-through to matching clusters |
|
|
|
-| Detail | Click any cluster row or paste a cluster_id — full drill-down with summary, key facts, articles, keywords, entities |
|
|
|
+| GET | `/api/v1/health` | Stats, freshness, feed state, pruning |
|
|
|
+| GET | `/api/v1/clusters` | Paginated clusters |
|
|
|
+| GET | `/api/v1/sentiment-series` | Sentiment time-series |
|
|
|
+| GET | `/api/v1/entities` | Top entities by frequency |
|
|
|
+| GET | `/api/v1/keywords` | Top keywords by frequency |
|
|
|
+| GET | `/api/v1/clusters/by-entity` | Entity search (SQL) |
|
|
|
+| GET | `/api/v1/clusters/by-keyword` | Keyword search (SQL) |
|
|
|
+| GET | `/api/v1/cluster/{id}` | Full cluster detail |
|
|
|
+| GET | `/api/v1/feeds` | Feed state list |
|
|
|
+| POST | `/api/v1/feeds/toggle` | Enable/disable feed |
|
|
|
+| GET | `/api/v1/config` | All config parameters |
|
|
|
+| POST | `/api/v1/config/update` | Update a parameter |
|
|
|
+| POST | `/api/v1/config/reset` | Reset to defaults |
|
|
|
|
|
|
-**Tech stack:** Vanilla HTML/JS + Chart.js, served as static files from the same FastAPI process at `/dashboard`. No extra dependencies.
|
|
|
-
|
|
|
-The dashboard default lookback window follows `NEWS_DEFAULT_LOOKBACK_HOURS` (configured via `.env`, default: 144h).
|
|
|
-
|
|
|
-## REST API
|
|
|
-
|
|
|
-Five read-only JSON endpoints for programmatic access:
|
|
|
-
|
|
|
-| Method | Path | Params | Returns |
|
|
|
-|--------|------|--------|---------|
|
|
|
-| GET | `/api/v1/health` | — | stats, freshness, feed state, pruning config |
|
|
|
-| GET | `/api/v1/clusters` | `topic`, `hours`, `limit`, `offset` | paginated cluster list |
|
|
|
-| GET | `/api/v1/sentiment-series` | `topic`, `hours`, `bucket_hours` | time-series data for Chart.js |
|
|
|
-| GET | `/api/v1/entities` | `hours`, `limit` | top entities by mention count |
|
|
|
-| GET | `/api/v1/cluster/{id}` | — (path param) | full cluster detail |
|
|
|
-
|
|
|
-## Startup
|
|
|
-
|
|
|
-Uses FastAPI `lifespan` — **HTTP API available in <0.3s** regardless of feed/LLM latency. Background refresh + pruning are fire-and-forget coroutines.
|
|
|
+## Configuration
|
|
|
|
|
|
-Key `.env` vars:
|
|
|
-- `NEWS_BACKGROUND_REFRESH_ON_START=true` — fetch immediately on boot
|
|
|
-- `NEWS_BACKGROUND_REFRESH_ENABLED=true` — enable/disable the background loop
|
|
|
-- `NEWS_REFRESH_INTERVAL_SECONDS=900` — polling interval
|
|
|
+All parameters are stored in the `site_config` DB table and editable via the dashboard Config page.
|
|
|
+On first startup, seeded from `.env` or built-in defaults.
|
|
|
+
|
|
|
+Key `.env` vars (seeded into site_config):
|
|
|
+
|
|
|
+| Variable | Default | Purpose |
|
|
|
+|----------|---------|---------|
|
|
|
+| `NEWS_FEED_URLS` | — | Comma-separated feed URLs |
|
|
|
+| `NEWS_REFRESH_INTERVAL_SECONDS` | 300 | Polling interval |
|
|
|
+| `NEWS_DEFAULT_LOOKBACK_HOURS` | 24 | Read freshness window |
|
|
|
+| `NEWS_RETENTION_DAYS` | 10 | Prune threshold |
|
|
|
+| `NEWS_PRUNE_INTERVAL_HOURS` | 12 | Prune check interval |
|
|
|
+| `NEWS_CLUSTER_MAX_AGE_HOURS` | 6 | Cross-cycle merge window |
|
|
|
+| `NEWS_EMBEDDINGS_ENABLED` | true | Enable Ollama embeddings |
|
|
|
+| `NEWS_EMBEDDING_SIMILARITY_THRESHOLD` | 0.885 | Cosine threshold |
|
|
|
+| `OLLAMA_BASE_URL` | `http://192.168.0.200:11434` | Ollama API URL |
|
|
|
+| `NEWS_EXTRACT_PROVIDER` / `NEWS_SUMMARY_PROVIDER` | groq | LLM provider |
|
|
|
+| `NEWS_EXTRACT_MODEL` / `NEWS_SUMMARY_MODEL` | llama4-16e | LLM model |
|
|
|
+| `GROQ_API_KEY` / `OPENAI_API_KEY` / `OPENROUTER_API_KEY` | — | API keys |
|
|
|
+| `ENTITY_BLACKLIST` | — | Comma-separated entity patterns |
|
|
|
+| `ENRICHMENT_MAX_PER_REFRESH` | 0 (unlimited) | LLM enrichments per cycle |
|
|
|
+
|
|
|
+Clustering thresholds (also in site_config):
|
|
|
+- `title_threshold`: 0.75
|
|
|
+- `jaccard_threshold`: 0.55
|
|
|
+- `dual_title_floor`: 0.55
|
|
|
+- `dual_jaccard_floor`: 0.25
|
|
|
+
|
|
|
+## Clustering pipeline
|
|
|
+
|
|
|
+1. **Fetch** all feeds concurrently
|
|
|
+2. **Feed hash** — skip unchanged feeds entirely
|
|
|
+3. **Retention filter** — drop articles older than `NEWS_RETENTION_DAYS`
|
|
|
+4. **Seen articles** — `filter_already_seen()` splits into:
|
|
|
+ - `new` → never seen, full processing
|
|
|
+ - `unchanged` → same URL, same content hash → skip
|
|
|
+ - `changed` → same URL, different content hash → re-cluster + re-enrich
|
|
|
+5. **Cluster** — title similarity, Jaccard, embeddings, dual-signal merge
|
|
|
+6. **Enrich** — LLM extraction, summarization, sentiment
|
|
|
+7. **Prune** — delete clusters older than retention window
|
|
|
+
|
|
|
+## Content-change detection
|
|
|
+
|
|
|
+When an article is updated in-place at the same URL (e.g. FT's "More to come..." → real content):
|
|
|
+1. `content_hash = SHA1(title|summary)` is computed
|
|
|
+2. Compared against `seen_articles.content_hash`
|
|
|
+3. If different → article is re-clustered into its existing cluster
|
|
|
+4. `enriched_at` is cleared → next cycle re-enriches with updated content
|
|
|
+
|
|
|
+## Persistence
|
|
|
+
|
|
|
+- SQLite at `./data/news.sqlite` (local) or `/app/data/news.sqlite` (Docker)
|
|
|
+- Schema auto-migrates on startup (ALTER TABLE for new columns)
|
|
|
+- Backfill script for seeding `seen_articles` from existing clusters:
|
|
|
+ ```bash
|
|
|
+ docker exec -it news-mcp python3 scripts/backfill_seen_articles.py
|
|
|
+ ```
|
|
|
+
|
|
|
+## Dashboard
|
|
|
+
|
|
|
+`http://<host>:8506/dashboard/`
|
|
|
+
|
|
|
+- **Health** — stats, charts, feed status
|
|
|
+- **Feeds** — toggle on/off
|
|
|
+- **Clusters** — filterable table, click for drill-down modal
|
|
|
+- **Sentiment** — time-series chart
|
|
|
+- **Entities** — top entities, frequency chart
|
|
|
+- **Keywords** — top keywords, frequency chart
|
|
|
+- **Config** — runtime parameter tuning (new in v0.5.0)
|
|
|
+
|
|
|
+## Version
|
|
|
+
|
|
|
+See `./version-hash.sh` for the current content hash.
|
