AlertRouting

1mission AlertRouting
2use product
3use delivery
4
5title "Route incident alerts to on-call and escalate the unacknowledged"
6for OnCallEngineer
7
8goal
9  Get every incident to someone who can act, and escalate before it is forgotten.
10
11guarantee a triggered alert is acknowledged or escalated, never silently dropped
12  because an unrouted incident is an outage nobody is fixing
13  verify escalation-on-timeout test
14
15never page an engineer for a non-actionable informational event
16  verify alert-classification test
17never send incident details over an unauthenticated channel
18  verify transport-security scan
19
20outcome FasterAcknowledgement
21  "on-call acknowledges high-severity alerts before they escalate"
22
23metric ack_within_sla_rate
24  baseline 68%
25  target 90%
26  window 30 days after release
27
28# How an incoming alert is routed. Run it: intent run AlertRouting.intent
29#   --inputs '{"severity":"high","primaryAvailable":true,"businessHours":false}'
30decision RouteAlert
31  inputs
32    severity
33    primaryAvailable
34    businessHours
35  rule pagePrimary
36    when severity == "high" and primaryAvailable == true
37    return PagePrimary
38  rule escalate
39    when severity == "high" and primaryAvailable == false
40    return Escalate
41  rule emailDaytime
42    when severity == "medium" and businessHours == true
43    return EmailPrimary
44  default
45    return QueueForReview
46
47# The life of an alert. Walk it: intent simulate AlertRouting.intent --events notify,ack,resolve
48lifecycle AlertLifecycle
49  state Triggered
50  state Notified
51  state Acknowledged
52  state Escalated
53  state Resolved
54  transition notify
55    from Triggered
56    to Notified
57  transition ack
58    from Notified
59    to Acknowledged
60  transition escalate
61    from Notified
62    to Escalated
63  transition escalatedAck
64    from Escalated
65    to Acknowledged
66  transition resolve
67    from Acknowledged
68    to Resolved
69  terminal Resolved
70
71# An incident is published to consumers; the payload carries no secrets.
72event IncidentRaised
73  payload
74    incidentId: IncidentId
75    severity: Severity
76    service: ServiceName
77    summary: Text
78  publishedBy AlertRouting
79  consumedBy OnCallDashboard
80
81# Paging is a remote call: it must be idempotent and time-bounded, and compensated on failure.
82command PageOnCall
83  idempotency_key incidentId
84  timeout 30 seconds
85  retry 3 times
86on PageOnCallFailed
87  compensate escalate to secondary rotation
88
89# The commitment, and the delivery result that measures it: intent outcomes AlertRouting.intent
90outcome_contract AckSpeed
91  outcome FasterAcknowledgement
92  metric ack_within_sla_rate
93  baseline 68%
94  target 90%
95  window 30 days after release
96  owner ReliabilityLead
97
98result Q3Acknowledgement
99  measures FasterAcknowledgement
100  metric ack_within_sla_rate
101  value 92%
102  baseline 68%
103
104# The spec proves itself: intent test AlertRouting.intent
105test RouteAlert
106  case high severity pages primary
107    given severity "high", primaryAvailable true, businessHours false
108    expect PagePrimary
109  case high severity with no primary escalates
110    given severity "high", primaryAvailable false, businessHours false
111    expect Escalate
112  case medium in hours emails
113    given severity "medium", primaryAvailable true, businessHours true
114    expect EmailPrimary
115  case low severity is queued
116    given severity "low", primaryAvailable true, businessHours true
117    expect QueueForReview
118
119test AlertLifecycle
120  scenario acknowledged directly
121    events notify, ack, resolve
122    expect Resolved
123    valid
124  scenario escalated then resolved
125    events notify, escalate, escalatedAck, resolve
126    expect Resolved
127    valid
128  scenario cannot resolve before acknowledging
129    events notify, resolve
130    invalid