mirror of
https://github.com/agessaman/meshcore-bot.git
synced 2026-10-15 02:29:55 +00:00
The landing page re-ran ~50 aggregate queries five times per load, then repeated the whole sequence every 30 seconds forever — including in backgrounded tabs. Against the live 1.44 GB database that was roughly 20 seconds of SQLite work per page load. Move the work off the request path. A refresher thread in the viewer process (which already runs migrations, so it works for a split-DB install) writes two tables: daily_rollup, one row per local date, and dashboard_snapshot, a single JSON row. A page load now reads one row. Measured on the live database: first paint 6 requests -> 2, /api/dashboard/summary p50 1.1 ms (304 in 0.8 ms), /api/stats 130 ms, and a 0.32 s refresh once a minute in the background. Make the numbers mean what they say: - Window selectors are built from each source's retention. The page offered "30d" and "All" against tables pruned at 7 days, so three of four choices returned the same figure under a label that denied it. - The incoming-packet chart reports its measured window instead of claiming 7 days for a table pruned at 3 — it sat beside a genuine 7-day contacts chart inviting an invalid comparison. - Days with no source data store NULL and render as gaps. Writing 0 would put a fake cliff at every retention boundary. - Signal metrics are stored as sums and counts, never means, so any window re-aggregates correctly. - Delta chips compare the last two complete calendar days and say so; the headline above them is a rolling 24 hours. - Unmapped role ordinals (type0..type15) bucket into "Unknown". - SNR comes from message_stats, where it is populated on every row, not from complete_contact_tracking, where it is populated on 7%. Kill the json_extract scans: packet_stream gains denormalized route_type_name, payload_type_name, path_len and bytes_per_hop, written at capture time. Aggregating those from JSON cost 3-6 s per query. Existing rows convert a bounded batch per tick rather than in one migration that would rewrite ~180 MB into the WAL and stall bot startup. A partial index serves as the backfill worklist — without it the "any rows left?" probe is a full scan costing 4.6 s per tick, and it costs that after the backfill finishes, because finding nothing still means reading everything. Also: replace the per-contact hop-prefix scan with the existing bucketed matcher and memoize the 7-day chunk set (264 ms -> 35 ms on a synthetic 100k-row database, regression-locked by a test); move the dashboard's JS and CSS to static files, which removes the CSP nonce requirement for the bulk of the page; and give cleanup_old_stats a future-timestamp guard, without which rows dated 2103 are never older than the cutoff and so live forever. Deletes the orphaned /stats page, unreachable from the nav and rendering stub charts that never populated. /api/stats stays as a shim with every key name intact plus Deprecation and Sunset headers. All schema changes are additive, so a downgraded codebase can read the data; it would however need the new schema_version rows removed, since MigrationRunner rejects versions it does not know.
MeshCore Bot Test Suite
This directory contains the test suite for the MeshCore Bot, focusing on graph-based path guessing functionality.
Structure
tests/
├── README.md # This file
├── conftest.py # Pytest fixtures and configuration
├── helpers.py # Test data factories and helper functions
├── unit/ # Unit tests (isolated, with mocks)
│ ├── test_mesh_graph_edges.py
│ ├── test_mesh_graph_validation.py
│ ├── test_mesh_graph_scoring.py
│ ├── test_mesh_graph_multihop.py
│ └── test_path_command_graph_selection.py
└── integration/ # Integration tests (with real database)
└── test_path_resolution.py
Running Tests
Run all tests
pytest
Run only unit tests
pytest tests/unit/
Run only integration tests
pytest tests/integration/
Run specific test file
pytest tests/unit/test_mesh_graph_edges.py
Run specific test
pytest tests/unit/test_mesh_graph_edges.py::TestMeshGraphEdges::test_add_new_edge
Run with coverage
pytest --cov=modules --cov-report=html --cov-report=term-missing
Run with verbose output
pytest -v
Run with markers
pytest -m unit # Run only unit tests
pytest -m integration # Run only integration tests
pytest -m slow # Run slow tests
Test Coverage
Unit Tests
test_mesh_graph_edges.py (15 tests)
Tests for MeshGraph edge management:
- Adding new edges
- Updating existing edges
- Public key handling
- Hop position tracking
- Geographic distance
- Edge queries (get, has, outgoing, incoming)
- Prefix normalization
test_mesh_graph_validation.py (12 tests)
Tests for path validation:
- Path segment validation
- Confidence calculation
- Recency checks
- Bidirectional edge validation
- Full path validation
- Minimum observations filtering
test_mesh_graph_scoring.py (11 tests)
Tests for candidate scoring:
- Score calculation with various edge combinations
- Bidirectional bonuses
- Hop position matching
- Geographic distance bonuses
- Minimum observations filtering
test_mesh_graph_multihop.py (12 tests)
Tests for multi-hop path inference:
- 2-hop and 3-hop path finding
- Intermediate node discovery
- Minimum observations filtering
- Bidirectional path bonuses
- Score reduction for longer paths
test_path_command_graph_selection.py (8 tests)
Tests for PathCommand._select_repeater_by_graph:
- Direct edge selection
- Stored public key bonus
- Star bias multiplier
- Multi-hop inference
- Confidence conversion
Integration Tests
test_path_resolution.py (5 tests)
End-to-end tests for full path resolution:
- Path resolution with graph edges from database
- Prefix collision resolution using graph data
- Edge persistence across graph restarts
- Graph vs geographic selection
- Real-world multi-hop scenarios
Test Fixtures
Fixtures are defined in conftest.py:
mock_logger: Mock logger for testingtest_config: Test configuration with Path_Command settingstest_db: In-memory SQLite database for testingmock_bot: Mock bot instance with all necessary attributesmesh_graph: CleanMeshGraphinstance for testingpopulated_mesh_graph:MeshGraphinstance with sample edges
Test Helpers
Helper functions in helpers.py:
create_test_repeater(): Factory for creating test repeater datacreate_test_edge(): Factory for creating test edge datacreate_test_path(): Factory for creating test path datapopulate_test_graph(): Helper to populate a graph with test edges
Writing New Tests
Unit Test Example
import pytest
from tests.helpers import create_test_edge
@pytest.mark.unit
class TestMyFeature:
def test_my_feature(self, mesh_graph):
"""Test description."""
mesh_graph.add_edge('01', '7e')
assert mesh_graph.has_edge('01', '7e')
Integration Test Example
import pytest
@pytest.mark.integration
class TestMyIntegration:
def test_my_integration(self, mock_bot, test_db):
"""Test description."""
# Use real database and bot components
pass
Test Markers
Tests are marked with:
@pytest.mark.unit: Unit tests (isolated, with mocks)@pytest.mark.integration: Integration tests (with real database)@pytest.mark.slow: Slow-running tests
Dependencies
Test dependencies are in requirements.txt:
pytest>=7.0.0pytest-asyncio>=0.21.0pytest-mock>=3.10.0pytest-cov>=4.0.0
Configuration
Pytest configuration is in pytest.ini:
- Test discovery patterns
- Async test support
- Output options
- Test markers
Notes
- Unit tests use in-memory SQLite databases for speed
- Integration tests may use real database connections
- All tests should be deterministic and not depend on external services
- Tests should clean up after themselves (fixtures handle this automatically)