Learn
Store and forward at the edge
Store and forward is the technique of recording data locally at a site and sending it onward when the link allows. An edge buffer writes every reading into a local buffer first, hands batches upstream as connectivity permits, and keeps collecting while the connection is down, within whatever size and age limits the buffer was given.
Almost every description of edge buffering stops at the reassuring half: it keeps collecting when the link drops. That half is true and it is not the interesting part. The interesting part is what the buffer does under pressure, in what order things come back, and what happens when an outage lasts longer than the buffer can hold. This page is about the second half, because that is the half you will eventually meet.
What is store and forward, and why does an industrial site need it?
Store and forward means the collector writes locally first and sends second. The two steps are decoupled on purpose, so that the ability to record does not depend on the ability to transmit. Without that decoupling, a reading is only as durable as the network at the instant it was taken.
Industrial sites need it more than most systems do, for reasons that are not going away.
- The sites are where the resource is, not where the connectivity is. Wind sits on ridgelines, solar sits in deserts, and the backhaul was an afterthought in the capital plan.
- The uplink is frequently a single link with no redundancy, because a second one costs real money for an asset whose economics were modeled without it.
- Outages are correlated with the events you most want data for. Storms take out both the link and the interesting turbine behavior in the same hour.
- Equipment does not pause. A controller keeps producing values whether anything is listening or not, and a value not read at the moment it existed is gone.
A collector that lives in the cloud and polls across the wide-area link has none of this. When the link is down it is not collecting slowly, it is not collecting at all, and the resulting gap is exactly as wide as the outage. That is the failure store and forward exists to remove, and it is why the collector has to run at the site rather than near the analytics.
Why does the buffer have two lanes instead of one?
Because the normal case and the outage case want opposite things from storage, and a single lane has to compromise on one of them.
In the normal case a reading lives in the buffer for a fraction of a second before it is handed upstream. Writing it to disk is pure cost: a disk write, a sync, and wear on the flash in a gateway that has to survive years in a cabinet with no one visiting it. In the outage case the opposite is true. A reading may need to survive for hours and across a restart, and memory offers no protection at all against either.
So Fleetera's runtime keeps a dual hot and cold buffer. The hot lane is memory-backed and carries the steady state. The cold lane is file-backed on the gateway's persistent volume and carries anything that has to outlive the moment. Both are drained by the same loop and both feed the same upstream path, so nothing downstream needs to know which lane a reading came through.
- Hot lane
- Memory-backed, used while the upstream link is healthy. Fast, no disk cost, and deliberately not durable, because in the steady state a reading is in it for milliseconds.
- Cold lane
- File-backed on the persistent volume, used during and after trouble. It survives a process crash and a container recreate, which is the entire reason it exists.
- The drain loop
- A single background worker that claims a batch of readings, sends them upstream, and either confirms them or returns them for retry. It also owns the decision about which lane is active.
What makes the buffer switch from the fast lane to the durable one?
Two things, and the second one exists because the first is not sufficient.
The obvious trigger is a failed send. When a batch cannot be delivered upstream, the drain loop returns that batch to the buffer, switches new writes to the durable lane, and backs off before trying again. The backoff doubles on each consecutive failure up to a ceiling, so a long outage settles into periodic retries rather than hammering a dead link.
The second trigger is subtler and was added after it was observed in practice. A send does not always fail; sometimes it simply never returns. A network path that black-holes traffic, a middlebox holding a connection open, a broker that accepts a connection and then stops responding, all produce a send that is neither succeeding nor erroring. During that hang the buffer would happily keep filling the memory lane, and the oldest readings in memory would start being discarded to make room while nothing was ever marked as failed.
So there is a stall watchdog. If a single send has been in flight for longer than a configured threshold, the buffer is forced onto the durable lane regardless of whether an error has been reported. Incoming telemetry then lands on disk instead of quietly evicting the memory lane, which converts a silent loss into a recoverable backlog.
Why a timeout is not enough on its own
A hung send has no error to react to. Any design that only switches lanes on failure is blind to the case where nothing ever fails, and that case is the one that loses data without logging anything.
Coming back is deliberately more conservative than going out. The buffer only returns to the fast lane after several consecutive drain cycles in which the upstream is reachable and both lanes are empty. A single successful send is not evidence of recovery, and flapping between lanes during an unstable link would be worse than staying on the durable one.
What happens when the buffer fills up?
It loses data, and this is the part almost nobody writes down. A buffer is finite. The durable lane has a configured size on disk and a configured maximum age, and when either limit is reached the oldest messages are discarded to make room for new ones. If those messages had never been delivered, that is a real, permanent loss of readings.
It is worth being exact about which limit bites first, because the two behave differently.
- The size limit is reached when the outage lasts long enough that the readings collected during it exceed the space allowed. How long that takes depends entirely on your reading rate, which is a property of your site, not of the software.
- The age limit is reached when readings sit in the buffer for longer than the configured maximum age, whether or not there is space. It is a backstop against data that can never be delivered occupying the volume indefinitely.
- There is a third, smaller loss path worth knowing about: an in-process queue between the drivers and the buffer. If readings arrive faster than the buffer can absorb them, that queue fills and readings are dropped before they ever reach a lane. It is counted separately, because it means something different: the site is producing faster than the local pipeline is configured to take, which is a sizing problem rather than a connectivity one.
What we can say honestly is that the runtime does not hide any of this. An eviction monitor compares what has left the buffer against what was actually confirmed as delivered, and the difference is undelivered data that was discarded. That figure is counted, logged as a warning that names it as data loss, and reported to the cloud alongside the rest of the gateway's health, so a site that has started losing readings is visible as a number rather than as a gap somebody notices in a report six months later.
The claim we do not make
An outage that outlasts the buffer costs you data. Anyone telling you their edge buffer means you never lose a reading is describing a buffer with no size limit, and no such thing exists.
What order does the buffered data arrive in?
Approximately the order it was collected, and approximately is the honest word. It is worth understanding why, because the difference between approximate and guaranteed is exactly the difference between an analysis that works and one that quietly does not.
The drain loop claims from the fast lane first and only then from the durable lane. During recovery that ordering is the useful one: anything still sitting in memory from before the trouble started goes out ahead of the backlog that accumulated during it. But it is a heuristic about which lane to read from, not a sort. Readings that crossed the lane switch can be interleaved on the way out.
The guarantee that survives all the way downstream is narrower and more useful to state precisely: the pipeline preserves order per asset. Readings from one piece of equipment stay in sequence relative to each other. Readings from two different assets have no ordering guarantee between them, and anything that needs a cross-asset sequence has to take it from the timestamps rather than from arrival order.
Does the same reading ever arrive twice?
Yes, and it is a design decision rather than a defect. Delivery is at-least-once by design. When a send fails after the data may already have been accepted upstream, the batch is returned and sent again, because the alternative, dropping a batch whose fate is unknown, loses data to avoid duplicating it. So the same reading can arrive more than once, and any consumer that counts rather than averages should be built expecting that.
The situation this exists for is narrow and unavoidable. A batch leaves the gateway, and the acknowledgement does not come back. From inside the gateway those two cases are identical: the batch was accepted and the acknowledgement was lost, or the batch never landed at all. There is no third signal to distinguish them, so the choice is between sending again, which can duplicate, and giving up, which can lose. Anything claiming to avoid both is claiming a guarantee the network does not offer.
The stall case resolves the same way. When the watchdog forces the buffer onto the durable lane because a send has been in flight too long, that batch is returned for retry rather than being assumed delivered. A duplicate is a visible, correctable nuisance. A silently missing reading is neither.
Downstream, a repeat is a second copy of the same reading: same asset, same variable, same timestamp, same value. Averages and last-value reads are barely affected by that. Anything that sums or counts readings is, so build those to expect a repeat rather than assuming each stored row is a distinct measurement.
State this the way the system behaves
Per-asset ordering, at-least-once delivery, and approximate global ordering during recovery. All three are ordinary properties of a durable pipeline, and all three are much cheaper to design around than to discover.
What survives a restart, and what does not?
The answer depends on which lane a reading was in, which is the practical consequence of the two-lane design and the thing most worth internalizing.
- Readings in the durable lane survive a process crash, a container restart and a container recreate, because that lane is files on a volume that outlives the container.
- Readings in the fast lane do not survive a restart. In the steady state that is a fraction of a second of data, which is the trade the lane exists to make.
- Readings that were claimed by the drain loop but never confirmed are recovered on startup and returned for retry, rather than being left marked as in flight forever by a process that no longer exists.
One operational consequence deserves emphasis because it is easy to get wrong at deployment time and expensive to discover later. The buffer's volume has to persist across container recreates. If the gateway's data directory is created fresh each time the container is replaced, the durable lane is not durable, and the design's entire protection against restarts is silently absent. Nothing errors. It simply behaves like a single memory lane whenever it matters most.
The same volume holds the gateway's identity and its cached configuration, which is what allows a site to start up and resume collecting after a reboot even if the cloud is unreachable at that moment. A gateway that comes back without its volume comes back as a stranger.
The runtime this describes
One binary per site, with the two-lane buffer and its volume built in rather than assembled.
How do you size an edge buffer?
By working backwards from the outage you intend to survive, rather than by accepting a default and hoping. The arithmetic is simple and the inputs are all things you can measure at your own site.
The sizing arithmetic in five steps, with the assumption each step exists to stop you making.
| Step | What goes into it | Where it goes wrong |
|---|---|---|
| 1. Count the signals | The signals actually being collected at the site, not the ones available on the equipment | Only bound signals produce readings, so the equipment's capability is the wrong number |
| 2. Multiply by the rate | The rate those signals are collected at | A one-second poll on a few thousand signals is a very different volume from a one-minute poll on the same set |
| 3. Multiply by the size of a reading | The on-disk size, which includes identity and timestamp, not just the value | Measure it rather than assuming it |
| 4. Multiply by the outage | The longest outage you are willing to survive without loss | Be honest about that number rather than optimistic |
| 5. Compare against the gateway | The space the gateway actually has, and the configured limits | If the numbers disagree, one of the four inputs above has to change. Pretending otherwise just moves the decision to the day of the outage |
This calculation usually produces one of two useful surprises. Either the buffer comfortably covers anything realistic, in which case you can stop worrying about it, or it covers far less than people assumed, in which case the sensible responses are more disk, fewer or slower signals, or an explicit acceptance that outages beyond a stated length will lose data. All three are defensible. Not knowing which one applies to you is not.
What should you monitor on a store and forward buffer?
Four numbers, and only one of them is the one people usually watch.
The four numbers worth an alert, and which of them describes something that can still be undone.
| What to watch | What it tells you | Does it recover? |
|---|---|---|
| Buffer depth | How much is waiting to be sent. Rising depth means the site is producing faster than it is draining, which during an outage is expected and outside one is a warning | Yes, as soon as the drain catches up |
| Age of the oldest unsent reading | More informative than depth, because it converts directly into how much history is at risk. Depth in bytes means nothing without knowing how long it represents | Yes, and it falls faster than depth does |
| Discarded readings | The count of readings dropped without ever being delivered. This is the one that matters and the one nobody watches, because it stays at zero for months and then does not | No. The reading is gone |
| Which lane is active | A gateway sitting on the durable lane is a gateway that has had trouble recently. It is a leading indicator, and it is visible before any data is lost | Yes, the lane flips back |
The reason to watch the discard count specifically is that it is the only one of the four that reports an irreversible event. Depth recovers, age recovers, the lane flips back. A discarded reading is gone, and the difference between finding out today and finding out at the next quarterly report is the difference between an operational incident and a hole in a dataset somebody has already published a conclusion from.
Common questions
Terms on this page
The vocabulary this guide uses, defined plainly in the industrial data glossary. Each one opens at its own entry.