Connected-vehicle telemetry: seven data problems that break pipelines first

A connected vehicle reports a stream of signals: location, speed, odometer, battery or fuel level, fault codes, and many more read from the vehicle's internal network. On a whiteboard this is a simple pipeline: vehicles send events, a gateway receives them, a stream processor cleans them, and a lakehouse stores them. In practice, the data breaks that pipeline in predictable ways long before anyone looks at a dashboard.

1. Data arrives late, and out of order

Vehicles lose connectivity in underground car parks, tunnels and areas with poor coverage, and many stop transmitting when the ignition is off. They buffer readings and upload them later, sometimes hours later, sometimes days, and not always in the order they were recorded.

What we do:

• Process on event time (when the reading was taken), not arrival time, using watermarks in the stream processor to decide when a time window is complete enough to emit.
• Set allowed lateness per use, not globally. A live alert cannot wait an hour; a daily mileage report can wait days.
• Send events that arrive after their window has closed to a correction path that recomputes only the affected day and vehicle, instead of dropping them or reprocessing everything. Partitioning the lakehouse tables by event date keeps those recomputations small.
• Publish completeness alongside the numbers, so users know whether yesterday is final or still filling in.

2. Clocks that cannot be trusted

A reading can carry several times: the device's own clock, a GPS time, and the time the gateway received it. Device clocks drift, and some reset to a default date after the battery is disconnected or a control unit is replaced. Suddenly a vehicle reports from 1970, or from next year.

What we do:

• Keep every timestamp you are given, raw, alongside the one you decide to use.
• Apply sanity bounds: nothing before the vehicle was built, nothing later than the time it was received plus a small tolerance.
• Prefer GPS time when it is present and valid. Otherwise, estimate the clock offset for each upload session from the receive time and correct the whole session consistently.
• Flag every corrected row, so analysts and models can include or exclude corrected data deliberately.

3. The same reading, delivered twice

Telemetry protocols and gateways are built to deliver at least once. With MQTT, for example, a message sent with quality-of-service level 1 is re-sent until it is acknowledged, so a dropped acknowledgement means a duplicate. Gateways retry whole uploads after timeouts. Duplicates are normal, not an error.

What we do:

• Build a deterministic event ID from the vehicle identifier, the signal, the event time and a device sequence number where one exists. The same reading always gets the same ID.
• Remove duplicates in the stream using that ID, with state kept only as long as the longest realistic retry window, so memory stays bounded.
• Make every downstream write an upsert on the event ID (a MERGE in the lakehouse), so a replay or a duplicate that slips past the stream does no harm.

4. Every model and firmware version speaks differently

The same physical quantity can arrive with a different signal name, unit, scaling factor or valid range depending on the model, model year, supplier or firmware version. Raw values from the vehicle network also need decoding: a raw integer becomes a speed or a temperature only after a scale and an offset are applied.

What we do:

• Maintain a versioned signal catalogue: for each model and firmware version, the mapping from raw signal to canonical name, unit, scale, offset and valid range.
• Decode at ingestion, and record the catalogue version used on every row, so a correction to the catalogue can be replayed precisely.
• Quarantine signals the catalogue does not recognise instead of dropping them. A new firmware release should produce an alert and a backlog item, not a silent gap.
• Register event schemas with a compatibility rule, so a producer change that would break consumers is rejected before it ships.

5. Signals sampled at different rates

Some signals are sent every second, some only when they change, some once per trip. Joining them on exact timestamps produces either mostly empty rows or wrong pairings: a speed reading matched with a fuel level from twenty minutes earlier.

What we do:

• Use as-of joins: for each reading, take the latest known value of the other signal at or before that moment.
• Give every signal a maximum staleness. A tyre pressure reading from ten minutes ago may still be valid; a location from ten minutes ago, at motorway speed, is not.
• Resample to fixed intervals only for specific analytical uses, never in the base tables.

6. Trips are not in the data

Business questions are about trips, journeys and shifts, but vehicles send readings, not trips. Ignition events are sometimes missing or duplicated, and a trip can cross midnight, a partition boundary or an upload gap.

What we do:

• Build trips with session windows: readings belong to the same trip until there is a gap longer than a set timeout, refined by ignition and speed signals when they are present.
• Generate deterministic trip IDs from the vehicle and the trip's start, so recomputing a day after late data arrives produces the same IDs instead of new ones.
• Recompute trips only around the late data that changed them, using the correction path from problem 1.

7. Location is personal data

A vehicle's location history says where someone lives, works and spends their evenings. Regulators treat it accordingly: the GDPR in Europe, India's Digital Personal Data Protection Act, 2023, and the European Data Protection Board's specific guidance on connected vehicles all apply to data like this. Consent can also be withdrawn.

What we do:

• Treat consent as data: a table, kept current, joined at ingestion, so data without a valid basis is never stored in the first place.
• Minimise by default: coarsen precise locations when the use does not need them, and keep the link between a vehicle and a person in a separate, restricted store.
• Build a deletion path before you need it: one job that finds and removes a vehicle's data across raw and derived tables, then expires the old table snapshots so the deleted data is actually gone, not just hidden from the latest version.

What this means for agents

The next users of vehicle data are agents: a service advisor's assistant that checks a vehicle's fault codes before it arrives, or a fleet assistant that answers "where is truck 14 and is it due a service?" For these, the latest state of each vehicle is kept in a context store keyed by vehicle, fed from a compacted Kafka topic, and every value carries its age.

That age matters more here than almost anywhere. "Last reported at the depot three days ago" and "at the depot now" are very different answers, and only the first is honest if the vehicle has been offline since. The seven patterns above are what make that age trustworthy: correct event times, no duplicates, consistent signal meanings and complete trips.