Technical article
Stale Readings After a Restart? Fixing MQTT Message Replay in ChirpStack, Telegraf and InfluxDB
A common LoRaWAN pipeline is ChirpStack → Mosquitto → Telegraf → InfluxDB. If readings jump or duplicate after a restart, the cause is usually how messages are queued and timestamped.
By Faraz, Co-Founder, CTO & Head of R&D, MatrixAI · · 8 min read
The symptom
After Telegraf restarts, or after a short network outage between Telegraf and the broker, a burst of meter readings lands in InfluxDB at once. Charts show spikes, consumption looks wrong for that hour, and some readings appear twice. The devices themselves reported normally.
Cause 1: persistent sessions replay queued messages
Telegraf's MQTT consumer can use a persistent session: with a fixed client ID and QoS 1, the broker queues messages while Telegraf is disconnected and delivers them all when it reconnects. That is useful — it prevents data loss — but those messages arrive minutes or hours late.
If Telegraf timestamps each point when it receives the message, the whole backlog is written with the reconnect time instead of the time each reading was taken. That produces the spike and the wrong consumption.
Cause 2: retained messages
If anything publishes to the event topics with the retain flag, the broker stores the last message per topic and hands it to every new subscriber. Each time Telegraf subscribes, it receives one old reading per device. Check with a subscriber that prints only retained messages, for example: mosquitto_sub -v -t 'application/#' --retained-only.
Cause 3: broker persistence after a broker restart
With persistence enabled, Mosquitto saves queued messages for persistent clients to disk and restores them when it restarts. If Telegraf was offline for a long time, a large backlog can be replayed in one go.
The fix: timestamp from the payload
ChirpStack v4 includes the uplink time in every event as the time field. Make Telegraf use it, so late messages are written at the time the reading was taken:
[[inputs.mqtt_consumer]]
servers = ["tcp://127.0.0.1:1883"]
topics = ["application/+/device/+/event/up"]
qos = 1
client_id = "telegraf-lorawan"
persistent_session = true
data_format = "json_v2"
[[inputs.mqtt_consumer.json_v2]]
measurement_name = "lorawan_uplink"
timestamp_path = "time"
timestamp_format = "2006-01-02T15:04:05Z07:00"
[[inputs.mqtt_consumer.json_v2.tag]]
path = "deviceInfo.devEui"
rename = "dev_eui"
[[inputs.mqtt_consumer.json_v2.object]]
path = "object"With payload timestamps, a replayed message is written at its original time. InfluxDB treats points with the same measurement, tags and timestamp as the same point, so a message delivered twice overwrites itself instead of creating a duplicate.
Further hardening
- Keep persistent_session = true with a fixed client_id if you want no data loss, now that replays are harmless.
- Limit the backlog in Mosquitto with max_queued_messages, and expire abandoned sessions with persistent_client_expiration.
- Make sure no integration publishes events with the retain flag.
- Use a separate client ID for every consumer. Two consumers sharing an ID will disconnect each other repeatedly.
- Monitor message rate per device: a sudden burst after a quiet period is the signature of a replay.
Frequently asked questions
Should Telegraf use a persistent MQTT session?
Yes if you want to avoid losing data while Telegraf is offline, but only together with payload timestamps, so replayed messages are written at the time they were measured.
Why do I see duplicate points in InfluxDB?
Duplicates appear when the same reading is written with two different timestamps, typically ingest time. Using the reading's own timestamp makes repeats overwrite the same point.