Core idea Infra already has telemetry: CPU, memory, storage, network, latency, availability, capacity, cost, incidents, predictability. Data needs the same thing. Not just: “Is the data quality good?” But: “Where did the data come from, how did it move, how was it transformed, who used it, what broke, what changed, what is trusted, what is risky, and what business process depends on it?” This becomes a Data Telemetry Observatory. 1. What data telemetry should cover End-to-end flow Source ↓ Ingestion ↓ Landing / Raw / T1 ↓ Standardization / Integration / T2 ↓ Serving / Analytics / T3 ↓ Distribution ↓ Consumption ↓ Business outcome / report / model / decision For every hop, capture telemetry. 2. Telemetry categories to capture 1. Data movement telemetry This answers: “How is data physically moving?” Track: Area What to capture Source system Which app/system produced the data Feed/API/file/topic Mode of movement Schedule Real-time, near-real-time, batch, ad hoc Volume Rows, files, messages, size Latency Source-to-landing, landing-to-serving SLA Expected vs actual arrival Failed loads Count, reason, retry status Reprocessing How often data is replayed Duplicate movement Same data moving through multiple routes Manual movement Excel, email, SFTP, manual uploads Good cop: This creates operational control over data like infra monitoring. Bad cop: If we only capture pipeline logs, we get technical telemetry, not business data telemetry. 2. Data freshness telemetry This answers: “Is the data current enough for the business process?” Track: Area What to capture Last source update When source changed Last successful ingestion When platform received it Last transformation When it was processed Last publish time When consumers received it Freshness gap Difference between source and consumer SLA breach Late beyond agreed threshold Staleness risk Report/model using old data Business impact Which process/report is affected Example: Customer Risk Rating source updated at 9:00 AM T1 received at 9:05 AM T2 processed at 10:30 AM T3 report refreshed at 4:00 PM Telemetry says: Freshness gap = 7 hours Business impact = AML dashboard using stale rating 3. Data quality telemetry This is needed, but it is only one part. Track: Dimension Example Completeness Customer ID missing Validity Invalid product code Accuracy Wrong customer risk rating Consistency Different rating across systems Timeliness Late arrival Uniqueness Duplicate customer/account Referential integrity Account has no valid customer Reconciliation Source count vs target count mismatch Drift Sudden change in distribution/pattern Good cop: This makes data trustworthy. Bad cop: DQ alone does not explain lineage, usage, business impact, ownership, or root cause. 4. Transformation telemetry This answers: “What happened to the data between source and consumption?” Track: Area What to capture Transformation rules Business logic applied Mapping Source field to target field Joins Which datasets were combined Filters What was excluded Aggregations How metrics were calculated Derivations How new fields were created Code version Which code version produced output Rule owner Who owns the business logic Change history What changed, when, why Exception logic Hardcoded overrides, local fixes Example: Customer Risk Rating = max(AML score, credit risk grade, fraud risk flag) Telemetry should show: Source fields used Rule applied Transformation code Business owner Last changed by Downstream consumers affected 5. Lineage telemetry This answers: “Where did this data come from and where did it go?” Track: Level Example System lineage Core Banking → ODS → EDW → Risk Report Dataset lineage CUSTOMER_MASTER → CUSTOMER_RISK_PROFILE Field lineage SRC_RISK_CD → risk_rating Business lineage Customer onboarding → AML screening → Risk dashboard Report lineage Which dashboards use the field AI lineage Which models/prompts/agents use this data This is where data telemetry becomes powerful. Not just: Table A feeds Table B But: Customer Risk Rating in AML Dashboard comes from Source X, transformed by Rule Y, published through Dataset Z, owned by Team A, used by Process B, and breached SLA twice this week. 6. Data contract telemetry This answers: “Did producers deliver what consumers were promised?” Track: Area What to capture Schema Expected fields, types, nullable rules Semantics Business meaning of each field SLA Delivery frequency and timing Quality rules Mandatory checks Volume expectation Normal row/file/message count Breaking changes Added, removed, renamed fields Contract owner Producer and consumer owner Version Contract version Exceptions Approved deviations Example contract telemetry: Dataset: Customer Risk Rating Producer: AML Platform Consumer: Enterprise Risk Mart Expected SLA: T+0 by 10 AM Actual: 11:45 AM Status: Breached Reason: Source delay Impact: 3 dashboards, 1 regulatory extract 7. Usage telemetry This answers: “Who is using the data, how often, and for what?” Track: Area What to capture Consumers Reports, apps, APIs, users, models Query frequency How often data is accessed Popular fields Most used columns Unused datasets Dead or stale data products Critical datasets High dependency datasets Cost-to-serve Compute/storage/query cost Business process What process depends on this data Regulatory use Which data supports regulatory reporting AI use Which agents/models use this data Important: Usage telemetry helps identify what is actually important. A dataset with 1,000 fields may look big. But telemetry may show: Only 43 fields are actively used 12 fields feed regulatory reports 5 fields feed customer decisioning 3 fields drive AI models 8. Ownership and accountability telemetry This answers: “Who owns the data and who fixes it?” Track: Area What to capture Business owner Owns meaning and rules Technical owner Owns pipeline/platform Data steward Owns quality and definition Application owner Owns source system Consumer owner Owns downstream usage Support group Owns incident resolution Escalation path Who gets alerted RACI Producer/consumer accountability Without this, dashboards become passive. With this, telemetry becomes actionable. 9. Incident and root-cause telemetry This answers: “When data breaks, can we diagnose fast?” Track: Area What to capture Incident type Late, missing, wrong, duplicate, schema change First detected at Source, ingestion, transformation, report Root cause Source issue, mapping issue, infra issue, rule issue Blast radius Affected datasets/reports/models Time to detect MTTD Time to recover MTTR Recurrence Same issue repeating Preventive control What check was added This mirrors infra incident management. But for data. 10. Change telemetry This answers: “What changed in the data landscape?” Track: Change type Example Schema change Field added/removed Rule change Risk rating logic changed Source change New upstream system Pipeline change Batch moved to streaming Consumer change New report/model added Ownership change New data owner SLA change T+1 changed to near-real-time Data contract change Version upgraded Access change New user/group consuming data This is critical because most data issues come from silent change. 3. The telemetry model You can structure telemetry around 8 core entities: Business Process Data Product Dataset Field System Pipeline Data Contract Consumer And the relationships: Business Process uses Data Product Data Product contains Dataset Dataset contains Field Field comes from Source Field Dataset moves through Pipeline Pipeline runs on System Dataset governed by Data Contract Consumer uses Dataset This gives you a graph view. Not just dashboard tables. 4. What dashboards can be built Executive dashboard For CDO/CIO/COO. Shows: Critical data health SLA breaches Top business-impacting issues Regulatory data risk Data product adoption Duplicate data movement Unowned critical datasets High-risk data flows Example: Metric View Critical datasets healthy 82% SLA breaches this week 17 Reports impacted 23 Unowned datasets 41 Duplicate feeds 128 Manual data movements 36 Regulatory flows at risk 9 Business process dashboard For business heads. Shows: Customer onboarding data health Credit approval data health AML monitoring data health Finance close data health Regulatory reporting data health Example: Business Process: Customer Risk Review Data Products: - Customer Master - Account Master - Transaction History - Risk Rating - KYC Profile Telemetry: - Freshness: Amber - Quality: Green - SLA: Red - Ownership: Green - Usage: High - Incidents: 3 this month Data product dashboard For data product owners. Shows: Who uses my data? Is it fresh? Is quality good? Which fields are most used? Which contracts are breached? Which consumers are impacted? Pipeline dashboard For engineering teams. Shows: Pipeline runs Failures Latency Volume anomaly Schema drift Retry count Reprocessing Dependency failures Contract dashboard For producer-consumer governance. Shows: Contract compliance Schema compatibility SLA adherence Quality rule pass/fail Consumer impact Version mismatch Breaking changes Lineage and blast-radius dashboard For architecture/governance. Shows: If this source breaks, what is impacted? If this field changes, which reports break? If this rule changes, which models are affected? Example: Source Field: CUSTOMER.RISK_CODE Impacts: - T1_CUSTOMER_RAW - T2_CUSTOMER_RISK_PROFILE - T3_AML_DASHBOARD - Regulatory Risk Extract - Customer Risk Agent - 4 reports - 2 APIs 5. What should be captured at each stage Source stage Capture: System of record Business owner Technical owner Dataset produced Schema Update frequency Known data issues Criticality Source SLA Source change calendar Ingestion stage Capture: Feed type Arrival time Volume File/message count Rejected records Schema mismatch Duplicate records Load duration Retry count Ingestion SLA Storage stage Capture: Zone: T1/T2/T3 Table/dataset location Partitioning Retention Encryption Classification PII/confidentiality Access groups Storage growth Unused/stale data Transformation stage Capture: Mapping Business rules Join logic Filter logic Aggregation logic Code version Rule owner Reconciliation result DQ checks Transformation latency Distribution stage Capture: Published dataset/API/report/feed Consumer list Contract version SLA Delivery status Access pattern Failure status Downstream acknowledgment Usage stage Capture: User/app/report/model consuming data Query/API usage Last used date Most used fields Criticality Cost Business process Regulatory relevance AI/model dependency 6. The “so what” The real value is not more dashboards. The value is: 1. Predictability Know before the business complains. Risk report will miss SLA because upstream feed is late. 2. Root cause Find issue fast. Report is wrong because T2 transformation used old mapping. 3. Impact analysis Know blast radius. Changing customer risk field affects 12 reports, 3 APIs, 1 model. 4. Governance Move from policy to control. Every critical dataset must have owner, contract, lineage, SLA, DQ and usage telemetry. 5. Rationalization Remove duplication. Same customer data is distributed through 8 routes. Consolidate through governed T2/T3 data product. 6. Trust Business knows which data is fit for use. This dataset is certified for regulatory use. This one is local-use only. 7. Proposed capability name A few options: Name Positioning Data Telemetry Observatory Best overall Enterprise Data Control Tower Senior leadership friendly Data Reliability Command Center Operational focus Data Movement Observatory Strong for flow/lineage Data Product Health Dashboard Product-centric Data Supply Chain Telemetry Best metaphor My pick: Enterprise Data Telemetry Observatory — the control tower for data movement, trust, usage, and business impact. 8. Good cop view This is a very strong idea. Infra has already proven the model: Telemetry → Observability → Predictability → Control → Optimization Data should follow the same path. This can help create: Data SRE Data product operations Producer-consumer accountability Automated governance Real-time SLA and DQ monitoring Business-impact lineage It also gives Enterprise Data Architecture a stronger role. Not just drawing data flows. But operating the enterprise data nervous system. 9. Bad cop view The risk is that this becomes yet another metadata dashboard. Common failure modes: Too much passive metadata No ownership No business process linkage No automated capture No real-time telemetry No action workflow No enforcement No clear “so what” Also, many firms confuse: Catalog ≠ Telemetry Lineage ≠ Control DQ dashboard ≠ Data observability Architecture diagram ≠ Operating model The biggest trap: Building a beautiful dashboard that nobody uses during incidents, change approvals, SLA reviews, or architecture decisions. 10. What makes this different from data catalog / lineage tools Data catalog says: This dataset exists. Lineage tool says: This dataset flows from A to B. DQ tool says: This dataset passed/failed checks. Data telemetry says: This dataset is late, the reason is upstream source delay, the impacted business process is AML review, 3 reports and 1 model are affected, the owner is X, the SLA breach is recurring, and the recommended action is to fix the source contract. That is the difference. 11. Practical MVP Do not start with the entire enterprise. Start with 1 critical data domain. Example: Customer Risk Rating Build telemetry across: Source apps T1 raw ingestion T2 integrated/customer risk profile T3 risk mart/report/API Downstream AML/credit/regulatory consumers MVP telemetry: Capability MVP output Source-to-consumption lineage End-to-end graph Freshness Source vs T1 vs T2 vs T3 timestamp SLA Expected vs actual delivery Quality Key rule pass/fail Contract Schema and SLA compliance Usage Reports/APIs/models using it Ownership Business + tech owner Incident Failure + blast radius 12. One sharp executive framing We have infra observability, but we do not have data observability at the same maturity. The next step is to build a Data Telemetry Observatory — a control tower that tracks data from source to ingestion, transformation, storage, distribution and usage. This is not just DQ. It combines lineage, freshness, SLA, contracts, quality, ownership, usage, change and business impact. The outcome is simple: know what data moved, whether it is trusted, who used it, what broke, who owns it, and which business process is impacted.