AI incident investigation
AI incident investigation before anyone is paged
When an alert arrives, Oncallz can ask an AI model to read it and write a short summary, a probable cause and a suggested first step, attached to the incident before the on-call engineer is paged.
It runs on your own AI provider key, fails open when the provider is slow, and never silences anyone unless you explicitly let it skip the phone call for clear noise.
Context when the page lands
A summary, probable cause, suggested first step and a 1–10 confidence score appear on the incident and in Slack or Microsoft Teams messages.
Never blocks paging
If AI is off, unconfigured, slow or failing, the incident is paged normally. The analysis waits 15 seconds at most by default.
Your key, your provider
Analysis uses your own provider API key. Every request, response, token count and estimated cost is logged for you to review.
What does the AI investigation produce?
For each new incident, the model receives the title, description, severity, source and raw alert payload, and returns a structured verdict. It is instructed not to invent details the alert does not support and to recommend paging whenever in doubt.
The analysis panel on the incident is marked as AI-generated, with a reminder to verify before acting.
- Summary — one or two sentences on what the alert means.
- Probable cause — the most likely root cause, or nothing if the alert carries too little signal.
- Suggested first step — the first concrete thing the responder should check.
- Recommendation — page normally, or treat the event as noise.
- Confidence — a score from 1 to 10.
Can AI stop a noisy alert from calling me?
Only if you turn that on. By default the AI only analyses: it never silences anyone and never changes an escalation policy.
A separate, opt-in setting lets it skip the phone call for events it judges to be clear noise, such as a self-recovered blip, a heartbeat or test ping, or an obvious false positive.
Even then the incident is still created and the on-call engineer is still notified by push, SMS and email; only the call is spared. Every muted call is recorded on the incident timeline as “AI muted the phone call”, so you can audit each decision.
Which AI provider does Oncallz use, and where does my data go?
Oncallz uses your own API key; there is no shared platform key, so AI features stay off until you add one. Today the supported provider is DeepSeek. You can leave the model empty to use the provider’s default or name a specific one.
The key is verified with the provider when you save it and stored encrypted.
Be clear about what that means: with investigation enabled, each incident’s title, description, severity, source and raw alert payload are sent to the AI provider under your account and its terms.
Strip secrets or personal data you do not want to leave your monitoring stack from alert payloads, or keep AI investigation off. Every request and response is visible in the AI request history.
What else does the AI do?
The same key powers three more features, all run on demand or on a schedule you control:
- Duplicate incident analysis — on the analytics page, the AI reads the alert titles in a date range, clusters the ones that describe the same problem and proposes a grouping template with * wildcards, so they fold into one incident from then on. You review and adopt each template yourself.
- Team scheduling analysis — reviews a team’s last 30 days and next 14 days of on-call for coverage gaps, overloaded people, override churn and escalation problems.
- Daily digest — a morning email to owners and admins with an AI briefing on the previous day’s incidents.
How do you know whether the AI is any good?
Responders rate each analysis as accurate, helpful or incorrect, with an optional comment, and the rating stays on the incident. Over time that shows whether the analyses are worth reading for your alerts.
The request history also lists latency, token usage and an estimated cost for each call; your provider’s invoice remains the authoritative figure.
Frequently asked questions
Does AI investigation delay paging?
Analysis runs before notifications go out, with a timeout of 15 seconds by default. If the provider answers in time, its verdict is attached to the incident before anyone is paged. If it is slow, returns an error or is not configured, Oncallz logs the failure and pages normally.
AI never stands between an alert and the on-call engineer.
Will the AI ever ignore a real outage?
It cannot silence an incident. At most, and only if you enable the optional setting, it can skip the phone call for an event it judges to be clear noise; the incident is still created and push, SMS and email still go out.
The model is told never to suppress anything that could be a real, ongoing outage, and to page whenever in doubt.
Which AI provider do I need?
Currently DeepSeek. You add your own DeepSeek API key in the AI settings, optionally with a specific model. Oncallz has no platform-wide key, so AI features stay off until you add yours.
Usage is billed by the provider to your own account, and Oncallz shows an estimated cost per request based on token counts.
Is my alert data sent to a third party?
Yes, when AI features are on. The incident’s title, description, severity, source and raw payload are sent to your AI provider, currently DeepSeek, using your key and under your agreement with them. Team scheduling analysis and the daily digest send schedule and incident data the same way.
If that is not acceptable for some alerts, keep AI off or strip sensitive fields at the source.
Is AI investigation an extra cost?
Not from Oncallz. It is included in the $4 per user per month plan, like every other feature. You pay your AI provider for the tokens you use, at their rates, on your own account. The request history shows each call’s token usage and an estimated cost, so you can keep an eye on spend.
Related
- Phone & SMS alerting Page on-call engineers by phone call, SMS, push and email by severity — and let them acknowledge or escalate from the phone keypad.
- Post-mortems Write blameless post-mortems with timelines and action items, and publish a customer-facing incident report with one click.
- Graylog Turn Graylog event notifications into on-call incidents with severity from the log level, grouped by title and paged by phone, SMS or push.
Ready when the pager goes off.
Create your workspace in minutes. Free to start, no credit card — $4 per user a month when you’re ready.