IngestClickstream

1# IngestClickstream.intent
2# Domain: data pipeline. Ingest raw events, validate schema, drop duplicates, and
3# route bad or sensitive records instead of loading them blindly. Exactly-once into
4# the warehouse; PII never lands in the clear. The routing rule runs deterministically.
5
6mission IngestClickstream
7use product
8
9title "Ingest, validate, dedupe, and route clickstream events"
10for DataPlatformTeam
11
12goal
13  Turn a raw event stream into clean, deduplicated, schema-valid rows in the
14  warehouse, quarantining what does not qualify.
15
16why
17  Silent bad data is worse than missing data: it corrupts every metric downstream
18  and is expensive to unwind once dashboards depend on it.
19
20requires
21  EventSchema
22  Warehouse
23
24input
25  eventId: EventId
26  schemaValid: Flag
27  alreadyLoaded: Flag
28  containsRawPii: Flag
29
30output
31  disposition: RecordDisposition
32
33guarantee each event is loaded at most once
34  because at-least-once delivery replays events, and double-counting skews metrics
35  verify exactly once load test
36
37guarantee schema-invalid records are quarantined, never loaded
38  because one malformed row can break every query that reads the table
39  verify schema validation test
40
41never
42  load a record that fails schema validation
43  write raw personal data to the warehouse in the clear
44
45never load a record that fails schema validation
46  because unvalidated data poisons every downstream consumer at once
47  verify schema validation test
48
49never write raw personal data to the warehouse in the clear
50  because unmasked PII in analytics storage is a compliance and breach risk
51  verify pii masking test
52
53# One deterministic routing decision per record. `intent run` / `intent test`.
54decision RouteRecord
55  inputs
56    schemaValid
57    alreadyLoaded
58    containsRawPii
59  rule duplicate
60    when alreadyLoaded == true
61    return Drop
62  rule invalid
63    when schemaValid == false
64    return Quarantine
65  rule sensitive
66    when containsRawPii == true
67    return Mask
68  default
69    return Load
70
71target
72  Tests
73
74test RouteRecord
75  case clean new record loads
76    given schemaValid true, alreadyLoaded false, containsRawPii false
77    expect Load
78  case duplicate is dropped
79    given schemaValid true, alreadyLoaded true, containsRawPii false
80    expect Drop
81  case invalid record is quarantined
82    given schemaValid false, alreadyLoaded false, containsRawPii false
83    expect Quarantine
84  case pii record is masked before load
85    given schemaValid true, alreadyLoaded false, containsRawPii true
86    expect Mask
87  case duplicate wins even when invalid
88    given schemaValid false, alreadyLoaded true, containsRawPii false
89    expect Drop