Core idea
Infra already has telemetry: CPU, memory, storage, network, latency, availability, capacity, cost, incidents, predictability.
Data needs the same thing.
Not just:
“Is the data quality good?”
But:
“Where did the data come from, how did it move, how was it transformed, who used it, what broke, what changed, what is trusted, what is risky, and what business process depends on it?”
This becomes a Data Telemetry Observatory.
1. What data telemetry should cover
End-to-end flow
Source
↓
Ingestion
↓
Landing / Raw / T1
↓
Standardization / Integration / T2
↓
Serving / Analytics / T3
↓
Distribution
↓
Consumption
↓
Business outcome / report / model / decision
For every hop, capture telemetry.
2. Telemetry categories to capture
1. Data movement telemetry
This answers:
“How is data physically moving?”
Track:
[TABLE]
| Area | What to capture |
| Source system | Which app/system produced the data |
| Feed/API/file/topic | Mode of movement |
| Schedule | Real-time, near-real-time, batch, ad hoc |
| Volume | Rows, files, messages, size |
| Latency | Source-to-landing, landing-to-serving |
| SLA | Expected vs actual arrival |
| Failed loads | Count, reason, retry status |
| Reprocessing | How often data is replayed |
| Duplicate movement | Same data moving through multiple routes |
| Manual movement | Excel, email, SFTP, manual uploads |
[/TABLE]
Good cop: This creates operational control over data like infra monitoring.
Bad cop: If we only capture pipeline logs, we get technical telemetry, not business data telemetry.
2. Data freshness telemetry
This answers:
“Is the data current enough for the business process?”
Track:
[TABLE]
| Area | What to capture |
| Last source update | When source changed |
| Last successful ingestion | When platform received it |
| Last transformation | When it was processed |
| Last publish time | When consumers received it |
| Freshness gap | Difference between source and consumer |
| SLA breach | Late beyond agreed threshold |
| Staleness risk | Report/model using old data |
| Business impact | Which process/report is affected |
[/TABLE]
Example:
Customer Risk Rating source updated at 9:00 AM
T1 received at 9:05 AM
T2 processed at 10:30 AM
T3 report refreshed at 4:00 PM
Telemetry says:
Freshness gap = 7 hours
Business impact = AML dashboard using stale rating
3. Data quality telemetry
This is needed, but it is only one part.
Track:
[TABLE]
| Dimension | Example |
| Completeness | Customer ID missing |
| Validity | Invalid product code |
| Accuracy | Wrong customer risk rating |
| Consistency | Different rating across systems |
| Timeliness | Late arrival |
| Uniqueness | Duplicate customer/account |
| Referential integrity | Account has no valid customer |
| Reconciliation | Source count vs target count mismatch |
| Drift | Sudden change in distribution/pattern |
[/TABLE]
Good cop: This makes data trustworthy.
Bad cop: DQ alone does not explain lineage, usage, business impact, ownership, or root cause.
4. Transformation telemetry
This answers:
“What happened to the data between source and consumption?”
Track:
[TABLE]
| Area | What to capture |
| Transformation rules | Business logic applied |
| Mapping | Source field to target field |
| Joins | Which datasets were combined |
| Filters | What was excluded |
| Aggregations | How metrics were calculated |
| Derivations | How new fields were created |
| Code version | Which code version produced output |
| Rule owner | Who owns the business logic |
| Change history | What changed, when, why |
| Exception logic | Hardcoded overrides, local fixes |
[/TABLE]
Example:
Customer Risk Rating = max(AML score, credit risk grade, fraud risk flag)
Telemetry should show:
Source fields used
Rule applied
Transformation code
Business owner
Last changed by
Downstream consumers affected
5. Lineage telemetry
This answers:
“Where did this data come from and where did it go?”
Track:
[TABLE]
| Level | Example |
| System lineage | Core Banking → ODS → EDW → Risk Report |
| Dataset lineage | CUSTOMER_MASTER → CUSTOMER_RISK_PROFILE |
| Field lineage | SRC_RISK_CD → risk_rating |
| Business lineage | Customer onboarding → AML screening → Risk dashboard |
| Report lineage | Which dashboards use the field |
| AI lineage | Which models/prompts/agents use this data |
[/TABLE]
This is where data telemetry becomes powerful.
Not just:
Table A feeds Table B
But:
Customer Risk Rating in AML Dashboard comes from Source X,
transformed by Rule Y,
published through Dataset Z,
owned by Team A,
used by Process B,
and breached SLA twice this week.
6. Data contract telemetry
This answers:
“Did producers deliver what consumers were promised?”
Track:
[TABLE]
| Area | What to capture |
| Schema | Expected fields, types, nullable rules |
| Semantics | Business meaning of each field |
| SLA | Delivery frequency and timing |
| Quality rules | Mandatory checks |
| Volume expectation | Normal row/file/message count |
| Breaking changes | Added, removed, renamed fields |
| Contract owner | Producer and consumer owner |
| Version | Contract version |
| Exceptions | Approved deviations |
[/TABLE]
Example contract telemetry:
Dataset: Customer Risk Rating
Producer: AML Platform
Consumer: Enterprise Risk Mart
Expected SLA: T+0 by 10 AM
Actual: 11:45 AM
Status: Breached
Reason: Source delay
Impact: 3 dashboards, 1 regulatory extract
7. Usage telemetry
This answers:
“Who is using the data, how often, and for what?”
Track:
[TABLE]
| Area | What to capture |
| Consumers | Reports, apps, APIs, users, models |
| Query frequency | How often data is accessed |
| Popular fields | Most used columns |
| Unused datasets | Dead or stale data products |
| Critical datasets | High dependency datasets |
| Cost-to-serve | Compute/storage/query cost |
| Business process | What process depends on this data |
| Regulatory use | Which data supports regulatory reporting |
| AI use | Which agents/models use this data |
[/TABLE]
Important: Usage telemetry helps identify what is actually important.
A dataset with 1,000 fields may look big.
But telemetry may show:
Only 43 fields are actively used
12 fields feed regulatory reports
5 fields feed customer decisioning
3 fields drive AI models
8. Ownership and accountability telemetry
This answers:
“Who owns the data and who fixes it?”
Track:
[TABLE]
| Area | What to capture |
| Business owner | Owns meaning and rules |
| Technical owner | Owns pipeline/platform |
| Data steward | Owns quality and definition |
| Application owner | Owns source system |
| Consumer owner | Owns downstream usage |
| Support group | Owns incident resolution |
| Escalation path | Who gets alerted |
| RACI | Producer/consumer accountability |
[/TABLE]
Without this, dashboards become passive.
With this, telemetry becomes actionable.
9. Incident and root-cause telemetry
This answers:
“When data breaks, can we diagnose fast?”
Track:
[TABLE]
| Area | What to capture |
| Incident type | Late, missing, wrong, duplicate, schema change |
| First detected at | Source, ingestion, transformation, report |
| Root cause | Source issue, mapping issue, infra issue, rule issue |
| Blast radius | Affected datasets/reports/models |
| Time to detect | MTTD |
| Time to recover | MTTR |
| Recurrence | Same issue repeating |
| Preventive control | What check was added |
[/TABLE]
This mirrors infra incident management.
But for data.
10. Change telemetry
This answers:
“What changed in the data landscape?”
Track:
[TABLE]
| Change type | Example |
| Schema change | Field added/removed |
| Rule change | Risk rating logic changed |
| Source change | New upstream system |
| Pipeline change | Batch moved to streaming |
| Consumer change | New report/model added |
| Ownership change | New data owner |
| SLA change | T+1 changed to near-real-time |
| Data contract change | Version upgraded |
| Access change | New user/group consuming data |
[/TABLE]
This is critical because most data issues come from silent change.
3. The telemetry model
You can structure telemetry around 8 core entities:
Business Process
Data Product
Dataset
Field
System
Pipeline
Data Contract
Consumer
And the relationships:
Business Process uses Data Product
Data Product contains Dataset
Dataset contains Field
Field comes from Source Field
Dataset moves through Pipeline
Pipeline runs on System
Dataset governed by Data Contract
Consumer uses Dataset
This gives you a graph view.
Not just dashboard tables.
4. What dashboards can be built
Executive dashboard
For CDO/CIO/COO.
Shows:
Critical data health
SLA breaches
Top business-impacting issues
Regulatory data risk
Data product adoption
Duplicate data movement
Unowned critical datasets
High-risk data flows
Example:
[TABLE]
| Metric | View |
| Critical datasets healthy | 82% |
| SLA breaches this week | 17 |
| Reports impacted | 23 |
| Unowned datasets | 41 |
| Duplicate feeds | 128 |
| Manual data movements | 36 |
| Regulatory flows at risk | 9 |
[/TABLE]
Business process dashboard
For business heads.
Shows:
Customer onboarding data health
Credit approval data health
AML monitoring data health
Finance close data health
Regulatory reporting data health
Example:
Business Process: Customer Risk Review
Data Products:
- Customer Master
- Account Master
- Transaction History
- Risk Rating
- KYC Profile
Telemetry:
- Freshness: Amber
- Quality: Green
- SLA: Red
- Ownership: Green
- Usage: High
- Incidents: 3 this month
Data product dashboard
For data product owners.
Shows:
Who uses my data?
Is it fresh?
Is quality good?
Which fields are most used?
Which contracts are breached?
Which consumers are impacted?
Pipeline dashboard
For engineering teams.
Shows:
Pipeline runs
Failures
Latency
Volume anomaly
Schema drift
Retry count
Reprocessing
Dependency failures
Contract dashboard
For producer-consumer governance.
Shows:
Contract compliance
Schema compatibility
SLA adherence
Quality rule pass/fail
Consumer impact
Version mismatch
Breaking changes
Lineage and blast-radius dashboard
For architecture/governance.
Shows:
If this source breaks, what is impacted?
If this field changes, which reports break?
If this rule changes, which models are affected?
Example:
Source Field: CUSTOMER.RISK_CODE
Impacts:
- T1_CUSTOMER_RAW
- T2_CUSTOMER_RISK_PROFILE
- T3_AML_DASHBOARD
- Regulatory Risk Extract
- Customer Risk Agent
- 4 reports
- 2 APIs
5. What should be captured at each stage
Source stage
Capture:
System of record
Business owner
Technical owner
Dataset produced
Schema
Update frequency
Known data issues
Criticality
Source SLA
Source change calendar
Ingestion stage
Capture:
Feed type
Arrival time
Volume
File/message count
Rejected records
Schema mismatch
Duplicate records
Load duration
Retry count
Ingestion SLA
Storage stage
Capture:
Zone: T1/T2/T3
Table/dataset location
Partitioning
Retention
Encryption
Classification
PII/confidentiality
Access groups
Storage growth
Unused/stale data
Transformation stage
Capture:
Mapping
Business rules
Join logic
Filter logic
Aggregation logic
Code version
Rule owner
Reconciliation result
DQ checks
Transformation latency
Distribution stage
Capture:
Published dataset/API/report/feed
Consumer list
Contract version
SLA
Delivery status
Access pattern
Failure status
Downstream acknowledgment
Usage stage
Capture:
User/app/report/model consuming data
Query/API usage
Last used date
Most used fields
Criticality
Cost
Business process
Regulatory relevance
AI/model dependency
6. The “so what”
The real value is not more dashboards.
The value is:
1. Predictability
Know before the business complains.
Risk report will miss SLA because upstream feed is late.
2. Root cause
Find issue fast.
Report is wrong because T2 transformation used old mapping.
3. Impact analysis
Know blast radius.
Changing customer risk field affects 12 reports, 3 APIs, 1 model.
4. Governance
Move from policy to control.
Every critical dataset must have owner, contract, lineage, SLA, DQ and usage telemetry.
5. Rationalization
Remove duplication.
Same customer data is distributed through 8 routes. Consolidate through governed T2/T3 data product.
6. Trust
Business knows which data is fit for use.
This dataset is certified for regulatory use.
This one is local-use only.
7. Proposed capability name
A few options:
[TABLE]
| Name | Positioning |
| Data Telemetry Observatory | Best overall |
| Enterprise Data Control Tower | Senior leadership friendly |
| Data Reliability Command Center | Operational focus |
| Data Movement Observatory | Strong for flow/lineage |
| Data Product Health Dashboard | Product-centric |
| Data Supply Chain Telemetry | Best metaphor |
[/TABLE]
My pick:
Enterprise Data Telemetry Observatory — the control tower for data movement, trust, usage, and business impact.
8. Good cop view
This is a very strong idea.
Infra has already proven the model:
Telemetry → Observability → Predictability → Control → Optimization
Data should follow the same path.
This can help create:
Data SRE
Data product operations
Producer-consumer accountability
Automated governance
Real-time SLA and DQ monitoring
Business-impact lineage
It also gives Enterprise Data Architecture a stronger role.
Not just drawing data flows.
But operating the enterprise data nervous system.
9. Bad cop view
The risk is that this becomes yet another metadata dashboard.
Common failure modes:
Too much passive metadata
No ownership
No business process linkage
No automated capture
No real-time telemetry
No action workflow
No enforcement
No clear “so what”
Also, many firms confuse:
Catalog ≠ Telemetry
Lineage ≠ Control
DQ dashboard ≠ Data observability
Architecture diagram ≠ Operating model
The biggest trap:
Building a beautiful dashboard that nobody uses during incidents, change approvals, SLA reviews, or architecture decisions.
10. What makes this different from data catalog / lineage tools
Data catalog says:
This dataset exists.
Lineage tool says:
This dataset flows from A to B.
DQ tool says:
This dataset passed/failed checks.
Data telemetry says:
This dataset is late, the reason is upstream source delay,
the impacted business process is AML review,
3 reports and 1 model are affected,
the owner is X,
the SLA breach is recurring,
and the recommended action is to fix the source contract.
That is the difference.
11. Practical MVP
Do not start with the entire enterprise.
Start with 1 critical data domain.
Example:
Customer Risk Rating
Build telemetry across:
Source apps
T1 raw ingestion
T2 integrated/customer risk profile
T3 risk mart/report/API
Downstream AML/credit/regulatory consumers
MVP telemetry:
[TABLE]
| Capability | MVP output |
| Source-to-consumption lineage | End-to-end graph |
| Freshness | Source vs T1 vs T2 vs T3 timestamp |
| SLA | Expected vs actual delivery |
| Quality | Key rule pass/fail |
| Contract | Schema and SLA compliance |
| Usage | Reports/APIs/models using it |
| Ownership | Business + tech owner |
| Incident | Failure + blast radius |
[/TABLE]
12. One sharp executive framing
We have infra observability, but we do not have data observability at the same maturity. The next step is to build a Data Telemetry Observatory — a control tower that tracks data from source to ingestion, transformation, storage, distribution and usage. This is not just DQ. It combines lineage, freshness, SLA, contracts, quality, ownership, usage, change and business impact. The outcome is simple: know what data moved, whether it is trusted, who used it, what broke, who owns it, and which business process is impacted.