Repository navigation
Description
Goal
Add an in-app AI assistant for Exceptionless that lets users ask project questions in plain English and powers a Fix It / Explain button on events and stacks.
The in-app assistant should reuse the same core capabilities as the hosted MCP server, but it should be implemented as a first-party backend feature rather than calling MCP over HTTP from the app.
Product rules
- AI is available only on selected paid plans.
- Usage is token-based because provider billing is token-based.
- Start with included monthly token budgets only.
- Do not support overage billing in v1.
- Track provider cost internally from day one.
- Start read-only; any write actions must require explicit user confirmation.
Initial plan fields:
ai_enabledai_monthly_token_limitai_max_tokens_per_requestai_max_tool_calls_per_requestai_max_context_bytesai_allowed_modelsai_overage_enabled = false
Phase 1: Shared AI tool layer
Extract reusable backend tool logic so MCP and in-app AI use the same permission checks, validation, result shaping, and truncation behavior.
Suggested services:
AiToolCatalogAiToolExecutorAiContextBuilderAiUsageTrackerAiPlanLimitService
The in-app chat should call shared internal services directly. It should not call /mcp over HTTP.
Phase 2: Usage metering and limits
Add persistent usage tracking per organization.
Track:
- organization id
- user id
- operation:
chat,fix_it,explain_event,explain_stack - model
- input tokens
- output tokens
- total tokens
- estimated provider cost
- created date / month bucket
Enforce before and during requests:
- plan allows AI
- monthly token budget remaining
- per-request max token cap
- max tool calls
- max context size
Use a reservation or capped request budget so a single request cannot materially overspend the remaining monthly allowance.
Phase 3: Backend chat API
Add API endpoints such as:
POST /api/v2/ai/conversationsGET /api/v2/ai/conversationsGET /api/v2/ai/conversations/{id}POST /api/v2/ai/conversations/{id}/messages- optional streaming endpoint for assistant responses
The backend owns:
- permission checks
- plan checks
- usage reservation
- model calls
- tool calls
- persisted conversation history
Phase 4: Chat UI
Add a project-scoped AI chat surface in the app.
Potential entry points:
- Account AI Tools page
- project settings
- client setup page
- profile menu
- later: sidebar/global command menu
UI should show:
- current project / organization context
- token usage this month
- remaining token budget
- limit reached state
- conversation history
Phase 5: Fix It / Explain buttons
Add Fix It / Explain buttons on stack and event detail pages.
Clicking starts a conversation with preloaded context:
- project
- stack/event id
- stack trace
- request info
- environment
- recent occurrences
- version/release info if available
- related stacks/events if useful
Assistant response should be structured:
- likely cause
- evidence
- suggested fix
- assumptions
- next debugging steps
- related Exceptionless links
Phase 6: Confirmed write actions later
After read-only chat is useful, add confirmed write actions:
- mark fixed in version
- snooze
- mark future events critical
- add reference link
The assistant may propose an action, but the UI should execute it only after explicit user confirmation. All write actions should be audited.
Phase 7: Overage billing later
After real usage data exists, add pay-as-you-go.
Add:
ai_overage_enabledai_monthly_spend_limit_centsai_overage_price_per_million_tokens- internal provider cost tracking
- markup calculation
User options:
- stop at included token limit
- allow overage up to a monthly cap
Recommended first implementation slice
Start with foundation work only:
- Add plan AI limit fields.
- Add AI usage tracking.
- Add backend limit enforcement service.
- Add provider abstraction, with model calls behind config/feature flag.
- Keep the first slice read-only.
- Add minimal admin/system visibility for usage and limits.
A later PR can add the first user-facing read-only chat and Fix It experience.
Activity