meshcore-bot/docs/FEEDS.md

Feed Management

The Feed Management system allows the bot to subscribe to RSS feeds and REST APIs, automatically polling for new content and posting updates to specified mesh channels.

Overview

The feed manager supports two feed types:

• RSS Feeds: Standard RSS/Atom feeds
• API Feeds: REST API endpoints returning JSON data

Both feed types support:

• Configurable polling intervals
• Custom message formatting
• Item filtering
• Sorting
• Automatic deduplication
• Rate limiting

Configuration

Global Settings

Configure feed manager behavior in config.ini:

[Feed_Manager]
# Enable/disable feed manager
feed_manager_enabled = true

# Default check interval (seconds)
default_check_interval_seconds = 300

# Maximum items to process per check
max_items_per_check = 10

# HTTP request timeout (seconds)
feed_request_timeout = 30

# User agent for HTTP requests
feed_user_agent = MeshCoreBot/1.0 FeedManager

# Rate limit between requests to same domain (seconds)
feed_rate_limit_seconds = 5.0

# Maximum message length (characters)
max_message_length = 130

# Default output format
default_output_format = {emoji} {body|truncate:100} - {date}\n{link|truncate:50}

# Default interval between sending queued messages (seconds)
default_message_send_interval_seconds = 2.0

# Shorten item link URLs via [External_Data] short_url_website (v.gd / is.gd)
shorten_urls = false

# Or shorten only where the format says {link|shorten} (see placeholders below)

Per-output-format URL shortening: use {link|shorten} for a single shortened link, or {link|shorten|truncate:N} to shorten then cap length. shorten_urls = true shortens every plain {link}.

RSS Feed Configuration

The web interface provides separate input fields for each configuration option. Below are examples showing the values to enter in each field.

Basic RSS Feed

Feed Type: rss
Feed URL: https://example.com/rss.xml
Channel: #alerts
Feed Name (Optional): Example RSS Feed
Check Interval (seconds): 300
Output Format: (leave empty to use default)
Message Send Interval (seconds): 2.0
Filter Configuration: (leave empty)
Sort Configuration: (leave empty)

RSS Feed with Custom Format

Feed Type: rss
Feed URL: https://example.com/rss.xml
Channel: #alerts
Feed Name (Optional): Emergency Alerts
Check Interval (seconds): 60
Output Format:

{emoji} {title|truncate:80}
{body|truncate:100}
{date}

Message Send Interval (seconds): 2.0
Filter Configuration: (leave empty)
Sort Configuration: (leave empty)

API Feed Configuration

Basic API Feed

Feed Type: api
Feed URL: https://api.example.com/alerts
Channel: #alerts
Feed Name (Optional): API Alerts
Check Interval (seconds): 300
Output Format: (leave empty to use default)
Message Send Interval (seconds): 2.0
API Configuration (JSON):

{
  "method": "GET",
  "headers": {},
  "params": {
    "api_key": "your-api-key"
  },
  "response_parser": {
    "items_path": "data.alerts",
    "id_field": "id",
    "title_field": "title",
    "description_field": "description",
    "timestamp_field": "created_at"
  }
}

Filter Configuration: (leave empty)
Sort Configuration: (leave empty)

WSDOT Highway Alerts Example

Feed Type: api
Feed URL: https://wsdot.wa.gov/Traffic/api/HighwayAlerts/HighwayAlertsREST.svc/GetAlertsAsJson
Channel: #traffic
Feed Name (Optional): WSDOT Highway Alerts
Check Interval (seconds): 300
Output Format:

{emoji} [{raw.Priority|switch:highest:🔴:high:🟠:medium:🟡:⚪}] {title|truncate:80}
{raw.EventCategory} | {raw.Region} | {raw.EventStatus}
{body|truncate:70}

Message Send Interval (seconds): 2.0
API Configuration (JSON):

{
  "method": "GET",
  "headers": {},
  "params": {
    "AccessCode": "your-access-code"
  },
  "response_parser": {
    "items_path": "",
    "id_field": "AlertID",
    "title_field": "HeadlineDescription",
    "description_field": "ExtendedDescription",
    "timestamp_field": "LastUpdatedTime"
  }
}

Filter Configuration (JSON):

{
  "conditions": [
    {
      "field": "raw.EventCategory",
      "operator": "in",
      "values": ["Alert", "Closure"]
    }
  ],
  "logic": "OR"
}

Sort Configuration (JSON):

{
  "field": "raw.LastUpdatedTime",
  "order": "desc"
}

Output Format

The output format string controls how feed items are formatted before sending to channels.

Placeholders

• {title} - Item title
• {body} - Item description/body text
• {date} - Relative time (e.g., "5m ago", "2h 30m ago")
• {link} - Item URL
• {emoji} - Auto-selected emoji based on feed name (📢, 🚨, ⚠️, ℹ️)
• {raw.field} - Access raw API data fields (API feeds only)
• {raw.nested.field} - Access nested API fields (e.g., {raw.StartRoadwayLocation.RoadName})

Shortening Functions

Apply functions to placeholders using the pipe operator:

• {field|auto} - Use the remaining characters up to max_message_length (from [Feed_Manager]). The format string is read left to right: every placeholder before {field|auto} is rendered, then every placeholder after it; the space left in the message is filled with that field’s text. If the text is longer than that space, it is cut with ... (same idea as truncate:N). Use at most one {field|auto} per format. If more than one appears, the bot logs a warning, only the first expands, and any extra {field|auto} render empty. If the fixed prefix and suffix already exceed max_message_length, the auto segment is empty and the normal end-of-message truncation may still run.

• {field|truncate:N} - Truncate to N characters

• {field|word_wrap:N} - Wrap at N characters, breaking at word boundaries

• {field|first_words:N} - Take first N words

Examples:

{title|truncate:60}
{body|word_wrap:100}
{body|first_words:20}

Regex Extraction

Extract specific content using regex patterns:

• {field|regex:pattern} - Extract using regex (uses first capture group)
• {field|regex:pattern:group} - Extract specific capture group (0 = whole match, 1 = first group, etc.)

Examples:

{body|regex:Temperature:\s*([^\n]+):1}
{body|regex:Conditions:\s*([^\n]+):1}

Conditional Formatting

• {field|if_regex:pattern:then:else} - If pattern matches, return "then", else return "else"
• {field|switch:value1:result1:value2:result2:...:default} - Multi-value conditional

Examples:

{raw.Priority|switch:highest:🔴:high:🟠:medium:🟡:⚪}
{body|if_regex:No restrictions:👍:Restrictions apply}

Extract and Check

• {field|regex_cond:extract_pattern:check_pattern:then:group} - Extract text, check if it matches pattern, return "then" if match, else return extracted text

Example:

{body|regex_cond:Northbound\s*\n([^\n]+):No restrictions:👍:1}

Filter Configuration

Filter configuration determines which items are sent to channels.

Filter Structure

{
  "conditions": [
    {
      "field": "raw.Priority",
      "operator": "in",
      "values": ["highest", "high"]
    },
    {
      "field": "raw.EventStatus",
      "operator": "equals",
      "value": "open"
    }
  ],
  "logic": "AND"
}

Operators

• equals - Exact match
• not_equals - Not equal
• in - Value in list
• not_in - Value not in list
• matches - Regex match
• not_matches - Regex does not match
• contains - String contains value
• not_contains - String does not contain value
• within_days - Item timestamp is within the last N calendar days (rolling window from now, UTC)
• within_weeks - Same as within_days with N × 7 days (e.g. four weeks ≈ within_weeks 4 or within_days 28)

within_days / within_weeks require a field pointing at a date (same paths as sort: published for RSS, or raw.SomeTimeField for APIs). They also require days or weeks respectively.

If the date is missing or cannot be parsed, the condition fails (item is excluded). Set "include_if_missing": true on that condition to treat missing/unparseable dates as a pass instead.

Examples:

{
  "conditions": [
    {
      "field": "published",
      "operator": "within_days",
      "days": 28
    }
  ],
  "logic": "AND"
}

{
  "conditions": [
    {
      "field": "raw.LastUpdatedTime",
      "operator": "within_weeks",
      "weeks": 4
    }
  ],
  "logic": "AND"
}

Date parsing for filters matches sorting: ISO strings, Microsoft /Date(...)/, Unix timestamps (seconds or milliseconds), and common string formats.

Logic

• AND - All conditions must match (default)
• OR - Any condition matches

Field Paths

For API feeds, use raw.field or raw.nested.field to access API response fields:

• raw.Priority
• raw.EventStatus
• raw.StartRoadwayLocation.RoadName

For RSS feeds, use published for the item publication time in within_days / within_weeks conditions.

Sort Configuration

Sort items before processing:

{
  "field": "raw.LastUpdatedTime",
  "order": "desc"
}

Sort Options

• field - Field path to sort by (e.g., raw.LastUpdatedTime, raw.Priority, published)
• order - asc (ascending) or desc (descending)

Date Format Support

The sort function supports:

• ISO format dates
• Microsoft JSON date format: /Date(timestamp-offset)/ (e.g., WSDOT API)
• Unix timestamps
• Common date string formats

Message Queuing

Messages are queued and sent at configured intervals to prevent rate limiting:

• message_send_interval_seconds - Time between sending messages from the same feed (default: 2.0 seconds)
• Messages are automatically queued and processed in order
• Each feed maintains its own send interval

Deduplication

The system automatically prevents duplicate posts:

• Items are tracked by ID in the database
• Previously processed items are skipped
• Works correctly even when sorting changes item order
• Database-backed deduplication ensures reliability across restarts

Rate Limiting

The feed manager implements rate limiting:

• Per-domain rate limiting (default: 5 seconds between requests to same domain)
• Configurable via feed_rate_limit_seconds
• Prevents overwhelming feed sources

Web Interface

The feed management system includes a web interface accessible at /feeds:

• View all feed subscriptions
• Add/edit/delete feeds
• Preview output format with live feed data
• View feed statistics and activity
• Monitor errors

Command Interface

Feeds can be managed via mesh commands. The feed command requires admin access and must be sent as a direct message (DM) to the bot. The command is enabled by default.

Command Format: feed <subcommand> [arguments] (DM only)

Available Commands

• feed subscribe <rss|api> <url> <channel> [name] [api_config] - Subscribe to a feed
• feed unsubscribe <id|url> [channel] - Unsubscribe from a feed (by ID or URL)
• feed list [channel] - List all feed subscriptions (optionally filtered by channel)
• feed status <id> - Show detailed status for a feed
• feed enable <id> - Enable a feed subscription
• feed disable <id> - Disable a feed subscription
• feed update <id> [interval_seconds] - Update feed settings
• feed test <url> - Test/validate a feed URL

Examples

feed subscribe rss https://alerts.example.com/rss emergency "Emergency Alerts"
feed subscribe api https://api.example.com/alerts emergency "API Alerts" '{"headers": {"Authorization": "Bearer TOKEN"}}'
feed list
feed list #alerts
feed status 1
feed enable 1
feed disable 1
feed unsubscribe 1
feed update 1 60

Note: The feed command requires admin access. API feeds require JSON configuration as the last argument when subscribing.

Best Practices

1. Check Intervals: Set appropriate intervals based on feed update frequency (60-300 seconds typical)

2. Message Formatting: Keep messages under 130 characters for mesh compatibility

3. Filtering: Use filters to reduce noise and only send relevant items

4. Rate Limiting: Respect feed source rate limits by configuring appropriate intervals

5. Error Handling: Monitor feed errors in the web interface and adjust configuration as needed

6. Testing: Use the preview feature in the web interface to test output formats before enabling feeds

Troubleshooting

Feeds Not Polling

• Verify feed_manager_enabled = true in config
• Check that feeds are enabled in the database
• Review bot logs for errors
• Ensure bot is connected to mesh network

Items Not Appearing

• Check filter configuration - items may be filtered out
• Verify output format is correct
• Check feed activity log in web interface
• Review error log for parsing issues

Duplicate Messages

• Deduplication is automatic - check if item IDs are changing
• Verify last_item_id is being updated correctly
• Check database for processed items

Rate Limiting Issues

• Increase feed_rate_limit_seconds in config
• Increase message_send_interval_seconds for specific feeds
• Reduce check_interval_seconds to poll less frequently