manba generic
The reference architecture. Domain-neutral; every column below instantiates it.
service/app/engine.py · pipeline.py · canon.py
aonav.ai live
U.S. employment & careers. ~1,750 companies · 1,016 O*NET occupations · 561 clusters · ~55k pairs.
EC2 t3.large · 35.80.211.12 · nginx :443→:8081 · v0.15.1-beta
mrayeh / MFF live
Mamnoon Fine Foods — wholesale food sales analytics (dips, spreads, wraps, chips).
GCE e2-small · boh.mamnoonff.com · systemd+Uvicorn+nginx · app 2.0.0 · schema 2026.06.09
manba.ai service MVP
Autonomous publishing: point at a dataset + intent → a certified governed instance, no hand-written grammar.
service/app (FastAPI) · SQLite metastore · reuses the generic engine
Consumerswho / what asks
AnalystsAgentsPublic LLMsPipelinesDashboards
Humans and machines, expert and non-expert — funneled to domain precision.
career agentsCursor / Claude (MCP)Open WebUI chatanalystscontractor apps
owner / execsGemini chatCursor / Claude (MCP)investor deckbuyer portal
Customer brings data + intent → the service authors the instance; then agents / analysts consume the generated membrane on first contact (coached in-band).
Surfacesmembrane doors
DSLMCPRESTNLCLIHTTP/HTML
HTTP/HTML = independent validation URL: reproduce any query, read the certificate.
REST /api/* · DSL /api/dsl · NL /api/translate
MCP /mcp/v1, /mcp, stdio · OpenAI-compat /openai/v1
HTML /ui /pres /admin · Open WebUI :3443
— hybrid_server_mcp.py (aiohttp); one tool aoi_dsl_query
MCP FastMCP("mrayeh") stdio — mrayeh_query · _trend · _compare · _customer_lifecycle · _deck · _schema
DSL /api/dsl (+batch/schema/validate) · chat /api/chat (Gemini+query_data)
Web /dashboard /portal /boh /mffpres · legacy REST
Generated day-one: DSL /i/{iid}/dsl · MCP /i/{iid}/mcp (coach via initialize.instructions)
HTML mount → checkpoint → playground → certificate · query URL /i/{iid}/q/{b64} · /i/{iid}/help/{topic}
LanguageTier 2 over Tier 1
Tier 1 (fixed): LIST GET COUNT SUM RESOLVE HELP
ops IS · IS_NOT · CONTAINS · GT/GTE/LT/LTE · AND/OR; mods ORDER BY · LIMIT · OFFSET · BY
Tier 2 (per-instance): generated grammar.json — entities, fields, enums, relations, vocabulary.
Uppercase SQL-like. LIST GET COUNT SEARCH ANALYZE (+admin UPDATE/REMOVE/ADD, SUBMIT, HELP)
Tier 2 verbs: TITLECONVERT · COMPANYCONVERT · CLUSTERCONVERT
LIST COMPANIES WHERE INDUSTRY IS "Retail"
AND OVERALL_BADGE IS "Platinum"
TITLECONVERT "software engineer" EXPAND CLUSTER
Lowercase verb subject modifier:value. Verbs: get count sum search trend compare export lifecycle report
sum orders by:channel date:ytd authoritative:true
compare orders by:customer date:last-30d
vs:previous sort:revenue_delta:desc limit:20
lifecycle = new/returning/lapsed segmentation; report → cal-finance, deck.
Emits the same Tier 2 grammar.json the generic engine consumes — P3. Verbs/ops identical to generic; entities/enums/relations inferred from the data (P1→P2).
Engine code is shared & unchanged; only the grammar config differs per instance.
Entitiesdomain model
Discovered per instance — every entity/field/enum traces to a source column (gate). Name fields become identity-resolved.
COMPANIESOCCUPATIONSCLUSTERSWAGESAI_IMPACTPATHWAYSHIRINGRETENTIONBADGESTITLE
FOR-clause joins: LIST OCCUPATIONS FOR COMPANY "…"
ordersorder_linescustomersproductschannelsnotesactivationscustomer_groups
channels:
psfh · crown · direct · tonys. Trust:
authoritative:true → order_canonical (is_authoritative).
P2 domain_model.json — entities (table→UPPER), fields (type-inferred), enums (≤12 distinct), relations (name/_ID match), vocabulary (customer aliases), derived products, resolution candidates.
Gate: every claim traces back to P1.
Canonical corestrata · epochs
① observation log (append-only, bitemporal: source + ingest time)
② resolution assertions (confidence-scored, non-destructive; auto ≥0.93 / fuzzy ≥0.82 → review)
③ canonical store (rebuilt from ①+② · epoch-stamped) → ④ optimization
Epoch advances on sync / resolution-commit / regen. same query + same epoch = same answer
Disambiguation tiers: PG FTS → pgvector RAG (1024-dim HNSW) → LLM (disambiguation_service.py)
In-memory CompanyMatcher prefix index · title_conversion ~140k · company_aliases
order_canonical / line_canonical · customer_entities · product_entities · channel_entities (+ name aliases) · exclusion_rules
AttributionService.run_full_attribution() rebuilds canonical after each import.
ingest.py → observations (epoch++) · resolution.py → resolution_proposals · canon.py rebuild → canon (epoch-stamped)
review-commit / resync → new epoch; certified epochs stay replayable.
Ingestion /
generationhow it's built & fed
Either hand-authored (instance-zero) or generated by the service. Data enters via mount(source, adapter); each sync → a new epoch.
Bulk loaders → RDS. SQL migrations (sql/*.sql), scripts/import_*.py, scripts/seed_pgvector.py (embeddings). No live ingestion bus — periodic dataset refreshes.
Multi-source, live. Drivers:
gmail · google_sheets · csv · mbox + PSFH
Playwright browser_worker.
CONNECTDETECTEXTRACTNORMALIZEPERSIST
→ attribution → canonical. Email classifier (PSFH_AGGREGATE, CROWN_PACIFIC_PO…). Scheduler 6am daily.
Anomaly agent → pipeline_anomalies/improvements. Tracer: import_runs/logs.
The new generation pipeline (P1–P6) — LLM drafts, deterministic gates validate:
ingestCSV / Postgres → observation log (epoch++)
P1discover schema_map — types, keys, cardinalities
P2domain model ✓ trace-to-P1
P1.5intent + coverage (expressible / needs-derived / unanswerable+lead)
P3grammar.json ✓ closure: parse+exec
P45 context layers ✓ lint
P5benchmark (Axis1 grammar + Axis2 intent)
P6certify ✓ score ≥ 0.90 → certificate pinned to epoch
mount → checkpoint (human confirm) → generate → certify → go-live.
Trust &
contexthonesty · coaching
Invariants: I1 every response _meta{provenance, epoch}; I2 structured errors {unknown_fields, valid_fields, suggestion, example}; never fail silently.
5 context layers: L1 coach → MCP initialize.instructions · L2 HELP lessons · L3 schema self-correct · L4 _meta · L5 errors.
Benchmark: 2 axes (model × system) + delivery check → gate.
L1 agent_behavior.md → MCP instructions · L2 16 HELP topics · L3 schema.md · L5 errors -32602
Benchmark 96.4% Grade A (200.5/208, 20-case × 5-tier baseline); suite now 25 cases × 6 tiers. TITLECONVERT golden 98.0%.
Single source src/agent/context.py (GLOSSARY, DOMAIN, DSL_REFERENCE, MCP_INSTRUCTIONS, CHAT_SYSTEM_PROMPT)
Response meta.hints/related/canonical/schema_version · smoke_test_dsl · benchmark_api.
Inherited unchanged from the generic engine: provenance+epoch, structured QueryError, never-silent, determinism.
P4 generates all five layers in lockstep with the model (lint-gated). Certificate = public, reproducible, epoch-pinned.
Data storespersistence
Pluggable per instance — relational · vector · search · cache · files — behind the mount adapter. The canonical core is store-agnostic.
AWS RDS PostgreSQL 16.6aoi_data_v6pgvector 1024-HNSWPG FTSRedis
Auth: PG tables +
Auth0 SSO + casbin. Containers:
aoi-mcp-split-pg ·
redis-split ·
open-webui-split.
DuckDB (embedded)data/mrayeh.duckdbChromaDB (notes, opt)Redis (cache, opt)
No external RDBMS — single-file analytical store.
SQLite metastore (db.py)
tables: instances · syncs · observations · resolution_proposals · canon · artifacts · benchmark_cases/runs · config + uploads dir.
Sources & LLMupstream · models
Any tabular / relational feed + external APIs. LLMs are used to author & assist, never to generate governed answers directly.
Sources: WYWM (Burning Glass Inst · Schultz FF) · LinkedIn · Glassdoor · Lightcast · O*NET · BLS OEWS.
LLM: AWS Bedrock qwen3-32b → ollama/gemini/openai · embeddings Bedrock Titan v2.
Sources: Google Sheets (mezze weekly-order tabs) · Gmail (label mrayeh, Crown Pacific) · CSV PSFH (localfoodconnect) · MBOX (finefoods.mbox) · PSFH portal.
LLM: Gemini (chat + query planning).
Sources: any CSV upload or Postgres connection (via mount).
LLM: optional, env-gated enrichment (P2 descriptions) — output passes the same gates; the pipeline is fully functional without it.