CivicSpark AI

Built with community journalist and author/activist, Carlos Moreno, with support from Builders & Backers, in consultation with the Tulsa City Auditor’s Office and local community organizations. User research with 80+ residents, journalists, Mayor’s-office staffers, and organizers shaped the product surface.

Repo: kaizengrowth/CityCamp_AI: React/FastAPI, hybrid retrieval over budgets, legislation, and minutes; topic subscriptions; district lookup + email draft. The original PoC demo is still viewable.

CivicSpark has moved in stages. We built the proof of concept, ran an open public testing period with Tulsa residents, and folded what we heard into the design. Then came a necessary pause: seeking additional funding, and doing the slower work of community engagement. I strongly believe that a civic tool’s most important building blocks are not pieces of software but the community relationships that inform what should be built.

With sponsorship from Builders & Backers and the George Kaiser Family Foundation, we have been in a process of review and community-engaged evaluation, working towards a golden eval set written with the community. Development on the platform has continued in the iterations described below. Currently, we’ve taken down the website that was used in testing online to save on computing costs; this is now a community project in active testing on a stripped down local deployment.

The sketch below started as the ideal next-architecture for CivicSpark: what a production civic system should look like in 2025, using methods that hold up on municipal corpora. Much of what we planned out in early stages of architecture diagramming now exists in the code base, with many other layers we did not anticipate. The joy of working on this project has been what we’ve learned along the way, talking with community members.

The values that motivate this project

City council is the level of government with the most direct claim on daily life, the one deciding bus routes, curb cuts, zoning variances, and library hours, and it is also the hardest to follow. Agendas arrive as long PDFs posted days before a vote. Minutes sometimes only surface weeks later. A resident who wants to know what happened to the rezoning case on their block, or advocate for funding for a community center in their neighborhood, has to learn the city’s filing conventions before they can even ask the question.

The accessibility gap that prohibits true participatory democracy on the local level is a question that I’m passionate about, and why I built CivicSpark with my friend, Carlos Moreno. People didn’t come asking for a chatbot for City Government. The City Auditor, Nathan Pickard, wanted a way to better engage residents in each community district for input on city decisions. Residents we talked to said they have never been to a City Council meeting and didn’t know they were accessible to the public, yet expressed concerns about when a decision would land, whom to call about it, and whether showing up to speak would change anything. A journalist wanted the motion text for a case. An organizer wanted to know the moment her intersection of activist priorities appeared on any city agenda.

Underneath every feature request is a cautious hope that technological tools can poentially help eliminate some of the friction and inequity in civic participation. This is only possible if the tools are designed in close consultation with community members who want to help make city government more accessible, transparent, responsive, and accountable.

I go back and forth on whether language models help or hurt here. The potential with AI is that accessibility gains that were previously unaffordable, especially around language and literacy / reading levels, are now possible at a low cost. This could allow for more engagement by residents in local government through their phone, allowing them to participate in decisions that impact their neighborhoods and their daily lives. Most concerned residents do not have the time to search for answers in complex municipal codes and meeting minutes, especially if these codes are not in their native language, or are filled with legal jargon that is only understandable by specialists. A well-designed interface can make it easier for residents to get the information they need and send their concerns and requests to the right city department.

The case against: models fabricate with confidence, and a fabricated budget number can really damage community trust. A tool like this can widen participation, but if it is not designed properly, it can also automate an appearance of participation while the same people go unheard.

This is why we proritized taking time to engage in community design with the people who face the most barriers in government inclusion: community groups in low-income neighborhoods, homeless support services, immigrant community centers, organizations fighting for police accountability, groups raising funds for affordable housing.

As a new resident in Tulsa through the Tulsa Remote program, this process of getting to know the various organizations doing good work in Tulsa has been especially meaningful for me.

Responsible AI for civic systems

A commitment to human values for Civic Spark comes before any software architecture decisions or code.

The govtech frameworks I keep returning to (NIST’s AI Risk Management Framework, the public-sector chatbot literature in the references) converge on a handful of duties, listed below:

1. Accuracy lies in community-engaged evaluation systems, and it’s the most important factor in building any AI system. A wrong interpretation of nuanced policy can lead a resident to file for the wrong paperwork under tight deadlines, that can impost cost on her family. A wrong budget number parsed from a city document can cause citizen journalists to misrepresent work at the city. That is why refuse-over-invent is a hard rule rather than a tuning preference, why numbers come from structured tables instead of model memory, and why the claim verifier will delete sentences the evidence can’t carry. A system that answers less but is right better serves residents better than a fluent one that guesses.

2. People keep the decisions. The email drafts to your representative are sent by the resident with suggested language for effective communication, but never sent on behalf of the resident. Entity merges get proposed by the model and confirmed by a human. Nothing in the platform acts on a resident’s behalf without that resident’s explicit permission and action, and no one receives text messages about City Council decisions without opting in.

3. Configuration is political. How the AI handles contested topics (housing, policing, budget fights) is a policy choice someone makes. We need to be able to build explainability into our systems, and fine-tune with bias testing in mind. Choices on what kinds of documents are stored in the system, how we prompt each agent to search for answers, and hand off decisions between agents, are incredibly important in determining what kinds of responses are given. Each step in the configuration is inherently political, and should be handled carefully, in consultation with local groups and experts.

4. Collect less, keep less. The training corpus we use is public documents, and it stays that way; no quiet indexing of staff material or storing of resident data on the platform. Consent for alerts lives apart from account auth, chat logs are minimized and separated from analytics, and nothing profiles individual residents.

5. Accessibility is the most critical feature. If the tool only works for confident English readers on fast connections with modern browsers, it widens the participation gap it set out to close. Plain language as the default register, Spanish through the same pipeline as English, SMS alongside the web app, and WCAG conformance treated as a launch requirement.

6. Build for the budget that survives. Our first version of the platform ran on expensive managed services on AWS. We quickly realized that this is not sustainable for a community platform. Now the transcription pipeline runs on free GitHub Actions schedule pipeline with a free but high quality, and fine-tuned LLM. This is a deliberate ethical position rather than thrift. Civic tools often die when the grant ends. A platform a small nonprofit can keep running is more responsible than a better-funded one that disappears, taking residents’ subscriptions and trust with it.

Related projects

CivicSpark is not alone. There are also many wonderful projects in other cities that are exploring very interesting ideas in the civic tech space for participatory democracy. Here are a few I referenced and researched while building this project:

Positioning for Tulsa: CivicSpark’s MVP v0 proof of concept combined explorer + RAG chat + topic watch + district email, closer to View Royal / Engagic’s product shape than to a pure search archive like Civic Band. The gap is structure and verification: Engagic’s matters-first model and Civic Band’s “index is the product” discipline are the iterative north stars; official city chatbots are what we must not become.

Challenges and frequent mistakes

These show up across the projects above, in public-sector eval literature, and in our own PoC.

1. Treating the chatbot as the product

Mistake: Ship a chat box over scraped PDFs and call transparency done.
Reality: Residents, journalists, and organizers need browse, search, deep links, and alerts. Chat is one interface over an evidence layer. Civic Band’s durability comes from searchable minutes; Engagic’s from matters that persist across meetings.
Improve: Make the meeting/item explorer and watch digests correct even if the LLM is offline. Chat degrades gracefully to “here are the three agenda items” with links.

2. Chunk soup (losing legislative identity)

Mistake: Fixed-size embedding chunks with no meeting_id / item_id / ordinance section.
Reality: “What did Council decide on X?” is an identity query. Engagic reports ~97% item-level structured agendas where vendors allow it; the 3% PDF-monolith path is explicitly a fallback. Our PoC’s 512/50 chunking is that fallback as the default.
Improve: Parse agenda trees first; chunk by item; keep parent metadata on every vector.

3. Hallucinated certainty on budgets and code

Mistake: Free-text answers from nearest neighbors for dollar amounts and “what the law says.” GRASP’s whole point: naked frontier models lose to grounded+tool agents on municipal budget questions. Legal RAG evals (Stanford RegLab & HAI line) still show double-digit unsupported-answer rates even in commercial tools.
Improve: Numbers → structured tools; legal/code claims → cite-then-verify; refuse > invent. Never present a figure without a source span or table cell.

4. Contested civic issues and narrative capture

Mistake: Vendor defaults that redirect, soft-pedal, or refuse contested topics (housing, policing, accountability) while sounding authoritative, documented in Route Fifty’s tests of live city bots.
Improve: Explicit policy for contested queries: retrieve primary sources on all sides that exist in the corpus, label what Council did vs what advocates claim, and surface staff reports + public comment packets when present. Configuration choices are political; log who set them.

5. Alerts that spam or summarize away the source

Mistake: LLM digests as the only notification body; weak keyword match → unsubscribe death spiral.
Improve: Dual-track matching (Engagic-style: keyword ∪ matter/entity); item deep link mandatory; summary optional; digest vs immediate; measure precision by retention, not open rate vanity.

6. Auto-action and consent theater

Mistake: Auto-send mail, pre-checked SMS, or burying unsubscribe.
Improve: LegalCheck pattern: draft, show evidence, human sends. Separate consent store; quiet hours; no dark patterns.

7. Scraper fragility and silent staleness

Mistake: One brittle HTML scraper; missed packets; bot answers from last month with high confidence. Open Council’s docs are honest: adapting the scraper is the main work per city.
Improve: Vendor-aware adapters (Engagic’s lesson); content-hash + retrieved_at; freshness SLO dashboards; “index as of …” in the UI.

8. No eval loop / procurement without edge cases

Mistake: Demo happy-path, deploy, no gold set, no challenge process for residents (Route Fifty’s critique of city procurements).
Improve: Frozen civic eval set from the 80-person research day; regression CI; thumbs-down → human review queue (Toscana pattern); publish accuracy limits in plain language.

9. One voice for every audience

Mistake: Same register for a first-time resident and a beat reporter.
Improve: Intent + audience routing; plain-language default; “show me the motion text” mode for journalists.

Iteration loop (PoC → v2 → ongoing)

CivicSpark should improve the way mature civic projects do: structure the corpus, measure failure, then add model cleverness, not the reverse.

Iteration 0: Done (PoC)

• Surfaces: minutes UI, chatbot, topic tags, district email draft, AWS deploy.
• Research: Auditor’s Office + community orgs + public day.
• Honest limit: demo-grade RAG (fixed chunks, Chroma/FAISS, no claim verifier).

Iteration 1: Evidence layer (built)

Borrow Engagic/Open Council discipline:

1. Tulsa-specific adapter(s) for whatever portal publishes agendas/minutes (treat Legistar/Granicus/etc. as first-class if applicable).
2. Item-level JSON as source of truth; markdown/HTML views are projections.
3. Hybrid BM25 + dense index; Civic Band–style FTS as the offline-capable baseline.
4. Provenance on every chunk (source_url, hash, retrieved_at, page/bbox).

Exit criteria: a journalist can deep-link an agenda item without the chat; “as of” timestamp visible.

Iteration 2: Grounded Q&A (built)

Borrow GRASP + legal-RAG:

1. Intent router; hybrid retrieve + rerank; mandatory citations.
2. Claim verification; structured budget tools.
3. Contested-issue policy documented with city partners.
4. Gold set v1 from research transcripts + staff FAQs; CI fails the build on groundedness regressions.

Exit criteria: publish a simple public scorecard (e.g. groundedness % on the frozen set); refusals preferred to wrong answers.

Iteration 3: Watches and outreach that earn trust (built)

Borrow Engagic alerts + LegalCheck drafting + Toscana feedback:

1. Ingest-time watch matching (keyword ∪ entity/matter); deep links first.
2. Email: retrieve → draft → cite → user sends.
3. Thumbs-down queue reviewed weekly; each miss becomes a corpus or prompt fix.

Exit criteria: alert retention stable; zero auto-sends; documented consent.

Iteration 4: Longitudinal “matters” and optional media (built)

Borrow View Royal / Engagic matters graph; OpenCouncil.gr media only if Tulsa publishes A/V:

1. Track the same legislative matter across meetings (intro → amend → vote).
2. Optional: transcript segments synced to video if the city releases it.
3. Councillor/topic stance summaries only with evidence spans and confidence, never as unlabeled fact.

Exit criteria: “Where is matter X in the process?” answered from the graph, not a vibe.

Continuous: Observation as product management

From Regione Toscana’s production practice: every negative signal is a trigger for manual review (wrong answer, bad source, missing doc). That queue is the roadmap, not a backlog of model upgrades in the abstract.

Problem constraints (why “chat over PDFs” fails)

Municipal Q&A is closer to legal research than consumer chatbot use:

1. Answers must be grounded and citable. Unsupported claims about budgets or ordinances destroy trust faster than a slow UI. Stanford RegLab-style evals still show non-trivial hallucination even in commercial legal RAG, so verification is a first-class stage, not a nice-to-have.
2. Documents are structured and temporal. An agenda item, a minute entry, an ordinance section, and a budget line are different object types with different identity keys. “What did Council decide on X?” needs meeting date + item ID + disposition, not nearest embedding.
3. Expertise reversal in the audience. Residents, journalists, organizers, and staffers ask different questions of the same corpus. One undifferentiated answer style fails all of them.
4. Action has consequences. Drafting email to a representative or sending SMS alerts is an action system, not a retrieval system: confirmation, audit logs, and opt-in are mandatory.

System overview

Five product surfaces share one evidence layer:

Ideal ingestion pipeline

PoC path was roughly: upload → extract text → fixed 512-token chunks / 50 overlap → text-embedding-3-small → Chroma/FAISS. That works for demos; it loses agenda hierarchy and table structure.

Target pipeline:

1. Source adapters per feed (council agenda HTML/PDF, minutes, budget books, municipal code, auditor reports). Prefer official URLs with content-hash + retrieved_at; never index without provenance.
2. Layout-aware parse. Preserve agenda trees (item → subitem → attachment), table cells for budgets, and speaker turns in minutes where available. OCR only when needed; keep page/bbox for citation deep-links.
3. Canonical entities. Resolve councilors, districts, ordinance numbers, fund codes, neighborhood names into stable IDs (string matching + gazetteer + LLM-assisted proposal with human merge for new entities).
4. Structure-aware chunking. Chunk by agenda item or code section, not arbitrary token windows. Parent context (meeting date, body, item number) is stored as metadata on every chunk.
5. Versioning. Municipal code and budgets are time-indexed. Retrieval defaults to current effective text; historical questions must filter on as_of date explicitly.
6. Topic taxonomy. Multi-label classify into a fixed civic taxonomy (~40 labels from the PoC, e.g. housing, transportation, public safety) with a human-reviewed label set, used for both browse and watch subscriptions.

Ideal retrieval and grounded generation

SOTA for this domain is hybrid retrieval + rerank + cite-then-verify, closer to GRASP-style municipal budget agents and legal RAG than to vanilla “stuff top-k into the prompt.”

Retrieval stack

1. Query understanding: classify intent (budget_fact, meeting_outcome, code_lookup, process_how_to, contact_rep, subscribe_topic); extract entities and time bounds.
2. Hybrid candidate generation: dense vectors (semantic) ∪ BM25/keyword (ordinance numbers, fund codes, exact titles). Metadata filters: doc type, date range, district, topic.
3. Cross-encoder / late-interaction rerank over the fused candidate set (keep top 8–12 for generation).
4. Graph expansion (optional second hop): from a retrieved agenda item, pull linked ordinance, prior meeting dispositions, and responsible committee, via the civic KG, not another vague embedding search.
5. Generation with mandatory citations: every factual sentence maps to ≥1 chunk ID; UI renders footnotes that open the source span.
6. Claim verification pass: a separate checker drops or hedges any claim not entailed by retrieved text (refuse > invent). For budgets/numbers, prefer tool reads from structured tables over free-text generation.

Agentic tools (when RAG alone is wrong)

Some questions need actions, not paragraphs, a pattern from municipal GRASP / ReAct agents:

The agent plans tool calls; the LLM never “remembers” a dollar amount that didn’t come from a tool or cited chunk.

Topic watch and notifications

Subscriptions are a retrieval job on new agenda ingest, not a chatbot feature:

1. On each new agenda parse, run topic classifiers + keyword/entity match against active watches.
2. Rank matches by specificity (entity hit > topic label > weak semantic).
3. Notify via SMS/email with deep link to the item, not a model summary as the only content (summary optional, item link mandatory).
4. Quiet hours, digest vs immediate, and one-click unsubscribe. Store consent + channel separately from account auth.

Representative outreach (human-gated)

Email generation is Context-Augmented Generation in the LegalCheck sense: retrieve relevant agenda/code context, accept user steering (“I oppose rezoning at …”), draft a letter, show citations, require explicit Send. Never auto-send. Log drafts for abuse review without storing message body longer than policy requires.

Evaluation (non-negotiable for civic deploy)

Ship an eval harness before claiming reliability:

Gold questions should be written with journalists and organizers, not only engineers; the 80-person research day is the seed for that set.

Security, privacy, accessibility

• Public corpus by default; no silent indexing of non-public staff documents.
• PII minimization in logs; separate analytics from chat transcripts.
• Prompt-injection hardening on retrieved HTML/PDF text (treat retrieved content as untrusted data).
• WCAG 2.2 AA for explorer + chat; SMS fallback for users who won’t use a SPA.
• Model/provider choice is a policy decision (data residency, retention). Ideal: embeddings + rerank can be self-hosted; generation can be API with a redaction pass; document the tradeoff explicitly for city partners.

PoC vs ideal (gap map)

Kept as written in the original sketch. As of July 2026 the “Ideal” column describes what the test system does for every row except rerank, eval breadth, and the alert retention metric.

What works well (from the PoC + research)

• Real stakeholder loop (Auditor’s Office, organizers, journalists) before architecture cosplay.
• Topic taxonomy + meeting analytics as a resident-facing affordance, not only a chatbot.
• District lookup + draft email as the action path people actually wanted in research.
• RAG framing (cite city docs, don’t invent policy) as the product promise; keep that promise stricter in v2.

Where the build stands (July 2026)

Progress has come in stages rather than a straight line: build, public testing, pause for funding and community engagement, build again. Through the most recent stretch I kept working through the iterations above as a personal project, and the commits track this page almost section by section (“Iteration 2: grounded Q&A”, “Iteration 3: watches and outreach that earn trust”, “Iteration 4: matters graph”). None of it is publicly deployed; it runs in testing. But most of what this sketch specs now exists in code.

Evidence layer (Iteration 1). One Postgres database is the system of record, with pgvector for embeddings instead of the old Chroma/FAISS split. Every indexed document carries a source hash and retrieval time. Retrieval is hybrid: full-text search unioned with dense vectors, fused by rank, and it works with no LLM configured at all. Search before chat, in practice. Embeddings default to Jina v3 with a numpy fallback that needs no extensions.

Grounded Q&A (Iteration 2). An intent router sends queries through a multi-tool agent loop (budget tools included), and a claim-verification pass runs at temperature 0 against the retrieved evidence. Unsupported claims get removed or hedged; if the core answer has no support, the bot refuses and points at official sources. The refuse-over-invent rule survived contact with implementation.

Watches and outreach (Iteration 3). Alert matching happens at ingest time, exactly as specced: item-level keyword hits outrank topic labels, which outrank general meeting-type interest. Every alert leads with a deep link to the meeting item. Unsubscribe is one click via an HMAC-signed expiring token, no login required. Quiet hours, digest mode, and per-day caps sit on each subscription, and there are zero auto-sends anywhere in the system. Constituent email remains draft-review-send, with the resident doing the sending.

Matters graph and media (Iteration 4). Official identifiers (zoning applications like Z-7642, PUDs, BOA cases, ordinance and resolution numbers) are extracted from agenda items and resolved into persistent matter records. Each sighting records an inferred action: introduced, discussed, amended, postponed, voted. Terminal outcomes are never downgraded by a later stale mention, and a track_matter tool answers “where is Z-7642 in the process?” from the graph rather than from vibes. On the media side, a daily job syncs Granicus feeds, fans meeting video out to faster-whisper transcription running on free GitHub Actions, stores timestamped segments, and links quotes back to the exact video moment. Summaries and Spanish translations ride the same pipeline.

Evaluation plumbing. CI carries a frozen gold set with deterministic embeddings (no network calls), and recall regressions fail the build. Thumbs-down feedback lands in a chat_feedback review queue tagged by routed intent, so every miss maps to a corpus fix, a prompt fix, or a routing fix. This is the Toscana pattern from the sketch: the complaint queue is the roadmap.

Stack and provider changes. The test deployments target Vercel (React 18/TypeScript frontend) and Render (FastAPI backend), replacing the AWS ECS footprint. The OpenAI dependency is gone: generation is provider-agnostic behind an env var, defaulting to Llama 3.3 70B on Groq’s free tier, with OpenRouter, Together, or local Ollama as drop-ins. For a civic tool this matters beyond cost. An open-weight default means a future city partner could self-host the whole path if data residency ever demands it.

Still open, and deliberately so: cross-encoder reranking, digest dispatch, alert-retention metrics, councilor stance summaries (deferred until they can carry evidence spans and confidence labels), and a migration chain that can’t yet bootstrap a fresh database, an inheritance from the PoC. The biggest gap is the one the evaluation process below is closing: the CI gold set was written by engineers, and the one written with the community is what replaces it as the benchmark that counts.

Current status

The failure modes above are the acceptance tests: if v2 still answers budget questions from chunk soup without citations, it is just a prettier PoC.

Community-engaged evaluation (underway)

The frozen gold set in CI keeps retrieval honest, but engineers wrote it, and engineers ask engineer questions. The 80-person research day left us transcripts full of the questions Tulsans actually bring to city government, in the words they actually use. That is the raw material for something better, and it is now getting built: through a process of review and community-engaged evaluation, sponsored by Builders & Backers and the George Kaiser Family Foundation, we are working towards a real golden set of community-driven evals. The sponsorship matters for a reason beyond money. It pays participants for their time and puts structure around the review process, which is what separates community engagement from asking favors of the same organizers twice a year.

What the process covers:

1. Gold questions authored by residents, journalists, and organizers. Working sessions where participants write questions in their own phrasing, including Spanish, and agree on what a correct answer has to contain before anyone looks at what the bot says. Their questions become the public benchmark; ours stay as regression plumbing.
2. A public scorecard in plain language. Groundedness, citation precision, and refusal quality on the community-authored set, published on the site with one readable sentence per number. The test of the format: a partner at the Auditor’s Office could paste it into a council presentation without translation.
3. A visible challenge path. Any answer in the UI should be disputable in about two clicks, landing in the chat_feedback queue that already exists. The queue gets reviewed weekly with a community partner in the room, and the log of what changed as a result gets published too. A correction nobody can see is indistinguishable from no correction.
4. Retention as the alert metric that counts. An alert program is working when people who signed up in March still have it on in September. Open rates flatter; unsubscribes tell the truth. This metric is still unbuilt, and it should be the first one on the scorecard once digest dispatch lands.
5. Access testing by the people it’s for. The WCAG 2.2 AA target gets verified with residents who use screen readers and residents on SMS-only phones, not just with an audit tool. Same for the Spanish path: a translation pipeline is not language access until Spanish-speaking residents say the answers read naturally.
6. Close the loop with the research participants. Everyone interviewed in the original research gets a short report: here is what you told us, here is what got built because of it, here is the scorecard. Being studied for a tool and never hearing what came of it is its own small civic failure, and an avoidable one.

Screenshots (PoC UI)

References

Peer systems

• Engagic: multi-city, matters-first agendas and alerts
• explorience/open-council: scrape → JSON → chatbot pattern
• schemalabz/opencouncil / opencouncil.gr: transcription, search, notifications
• Civic Band: OCR + SQLite FTS across hundreds of municipalities
• kpeatt/viewroyal: meeting schema, votes, multi-tool RAG agent

Methods and critiques

• GRASP: municipal budget RAG + agents (arXiv:2503.23299)
• LegalCheck: municipal legal drafting with human refinement (arXiv:2605.12012)
• Regione Toscana RAG architecture: official sources, feedback-driven ops (Net7 writeup)
• Route Fifty: field test of city AI chatbots on contested issues (2026)
• Frontiers: public-service chatbot architectures, safety, and hallucination risk (2025)
• Stanford RegLab / HAI: legal AI hallucination measurement (groundedness baseline for citation-heavy domains)
• NIST AI Risk Management Framework (AI RMF 1.0): govern/map/measure/manage structure behind the responsible-AI section above