Offline-first isn't a feature, it's an architecture — a driver app post-mortem

The app is for last-mile delivery drivers: 900 of them, across eight cities, mostly tier-2. They scan a package, capture a signature and a photo, mark a delivery, sometimes collect cash. They do this in basements, in lifts, inside warehouses with metre-thick walls, and in areas where the network exists in the same way that a rumour exists.

Version one worked beautifully in our office. In the field it lost 4.7% of proof-of-delivery records over four months. Drivers re-entered them, ops chased them on WhatsApp, and one customer's SLA report showed deliveries that the system swore had never happened.

This is what was wrong, structurally, and what we did about it.

What version one was

An online app with a retry queue. The driver taps "delivered"; the app POSTs to the API; if the POST fails, it goes into a local list and we try again later. This is what most teams mean when they say "we support offline", and it is not offline-first. It is an online app that hopes.

Three things were structurally broken.

The IDs came from the server. A delivery record did not have an identity until the server gave it one. So while offline, the app was holding objects that did not properly exist. Anything referencing them — a photo, a cash collection, a subsequent status change — had nothing stable to point at. When the retry finally fired, we had to invent a correlation key on the fly, and where we got that wrong, records duplicated or vanished.

The UI assumed a round trip. The success screen was rendered on a 200 response. Offline, the driver saw a spinner, then an error, then tapped again, because the package was in the customer's hands and the app was insisting it was not. Every one of those double taps was a potential duplicate.

Conflicts were undetectable. If ops cancelled an order in the dashboard while a driver was offline delivering it, then the driver came back online, the last write won. Sometimes that meant a cancelled order was marked delivered. Sometimes a completed delivery got wiped by a stale status. We could not even tell which had happened, because we had no record of when the driver's action occurred — only when it synced.

The rewrite: the phone is the source of truth

The reframing that made the rest fall out:

The local database is the system of record for anything the driver does. The server is a replica that eventually catches up. The network is an optimisation, not a dependency.

Concretely, in Flutter with a SQLite database via Drift:

1. Client-generated IDs

Every entity gets a UUIDv7 the instant it is created on the device. It never changes. The server accepts it as the primary key. This one decision eliminates an entire class of bugs, because a record's identity no longer depends on connectivity.

UUIDv7 specifically, not v4, because it is time-ordered — which means it indexes well as a primary key and does not shred your B-tree the way random v4s do. That matters at a few million rows.

2. An append-only outbox

Every mutation writes two things in one local transaction: the new state, and an event describing the change.

class Outbox extends Table {
  TextColumn get id => text()();                 // uuidv7 of the EVENT
  IntColumn  get seq => integer().autoIncrement();
  TextColumn get entityType => text()();         // 'delivery' | 'scan' | 'cash'
  TextColumn get entityId => text()();           // uuidv7 of the entity
  TextColumn get op => text()();                 // 'create' | 'update'
  TextColumn get payload => text()();            // JSON
  DateTimeColumn get occurredAt => dateTime()(); // device clock, advisory only
  IntColumn  get attempts => integer().withDefault(const Constant(0))();
  TextColumn get lastError => text().nullable()();
}

The UI writes to the local tables and returns immediately. It never awaits the network. Sync is a background process that drains the outbox in seq order and applies whatever came back from the server.

The autoincrementing seq is doing important work: it preserves causal order. "Scanned at pickup" must reach the server before "delivered", even though both were created offline and both are being flushed in the same burst. Ordering by device timestamp would not be safe — see the clock section below.

3. Idempotency on the server

Each outbox event carries its own UUID, and the server stores processed event IDs. A duplicate delivery of the same event — which will happen, because the phone can send an event, lose the network before receiving the ACK, and retry — is recognised and acknowledged without being reapplied.

public function ingest(Request $r)
{
    $events = $r->validate([...])['events'];

    foreach ($events as $e) {
        // The unique index on event_id makes this safe under concurrency.
        $fresh = ProcessedEvent::query()
            ->insertOrIgnore(['event_id' => $e['id'], 'driver_id' => $r->user()->id]);

        if ($fresh === 0) {
            continue;   // already applied — ACK and move on
        }

        DeliveryEvent::apply($e);
    }

    return response()->json([
        'acked'  => collect($events)->pluck('id'),
        'cursor' => ChangeLog::currentCursor(),
    ]);
}

4. A conflict policy per entity — chosen deliberately

There is no general answer to "what happens when both sides changed". There is only a per-entity answer, and it must be written down:

• Scan events: append-only. A scan is a fact that happened at a place and time. Facts do not conflict. Both sides keep both.
• Assignment (which driver owns which order): server wins, always. Dispatch is the authority. If the server reassigned the order while the driver was offline, the driver's local state is stale and gets overwritten — with a loud, blocking notification, because the driver is possibly holding a package they must now hand back.
• Delivery status: client wins, with a caveat. The driver was physically there. Their word beats a dashboard. The caveat: if the server has a terminal state (cancelled by customer, returned to origin) with a timestamp that provably precedes the driver's action, the event is routed to an exception queue for a human, not silently discarded.
• Driver profile: last-write-wins. Nobody cares.

Writing those four lines down was a two-hour meeting and it is the most valuable artefact of the project.

The hard parts nobody warns you about

Device clocks lie

Drivers change their phone's time. Sometimes to game an attendance feature; often by accident. We saw timestamps in 2019 and in 2031. Never make a correctness decision based on a device timestamp. We keep occurredAt for display and forensics, use the monotonic outbox seq for ordering within a device, and let the server stamp authoritative time on ingest, recording the skew so we can flag a device whose clock is wild.

Photos are a separate problem

A proof-of-delivery photo is 2–4 MB. Putting it in the outbox JSON is a mistake: one failed 4 MB upload on a 200 kbps link will block every text event behind it. Photos go into their own queue, compressed on-device to roughly 200 KB (which is more than enough to see a doorstep and a package), uploaded with resumable multipart, and — critically — the delivery event syncs without waiting for its photo. The record is complete for ops; the image attaches when it can.

Android will kill your background sync

This consumed more engineering time than the entire sync protocol. WorkManager is the correct API, and it is not sufficient, because several OEM Android skins — Xiaomi, Vivo, Oppo, Realme, which is most of this fleet — aggressively kill background work to protect battery. Our answers:

• A foreground service with a persistent notification while a shift is active. Users accept it; the notification also shows pending-sync count, which turns out to be reassuring rather than annoying.
• An in-app prompt walking the driver through the OEM's battery-optimisation whitelist, per manufacturer, with screenshots. Yes, really. It is the difference between a sync that runs and one that does not.
• Sync opportunistically on any app foreground, and on connectivity-regained.
• Never assume a scheduled job ran. The outbox is durable; if it did not run, it will run later, and nothing is lost.

Storage fills up

These phones have under 2 GB free. Cap the local database, prune synced records older than 14 days, and hard-cap the pending photo queue with a visible warning to the driver. An app that fills the phone gets uninstalled.

Results, four months after the rewrite

• POD loss: 4.7% → 0.02% (and the residual is genuine device loss/theft)
• Median time from delivery tap to server visibility: 11 seconds when online; the p95 is dominated by drivers who are in a basement, which is correct behaviour, not a bug
• Ops WhatsApp escalations about missing deliveries: down ~60%
• Driver-reported "app hangs on delivery": effectively zero, because the app no longer waits for the network to confirm what the driver can see with their own eyes

What we would tell ourselves at the start

Offline-first is not a checkbox you can add in sprint 9. It determines your ID strategy, your API shape, your conflict semantics and your UI's relationship with time. If you might need it, design for it on day one — the cost then is maybe 15% more effort, and the cost of retrofitting it is a rewrite.

And the tell that you have not actually done it: if any screen in your app can show a spinner that is waiting on the network before confirming an action the user already physically performed, you are still online-first.