Systems engineering

Inside LIFT & FIT Private Space: coordinating money, time, and a real door.

A production system where a checkout can arrive late, a queue can disappear after commit, and a stale background job can affect physical access. The architecture is built around those failure modes.

Full-stack Payments Access control Operations
Screenshot of the Lift & Fit Private Space booking platform homepage.
Customer-facing product A short booking journey backed by separate booking, payment, and access state machines. Production

LIFT & FIT Private Space presents a deliberately short customer journey: choose an exclusive time slot, pay or redeem a pass, receive temporary access, use the studio, and leave. The implementation cannot be equally linear. It has to coordinate a relational database, a payment gateway, a job queue, email delivery, and a smart-lock service, none of which share a transaction.

That makes this less like a calendar and more like a compact distributed system. The interesting question is not whether the happy path works. It is whether the system remains coherent when the customer retries checkout, a callback arrives twice, Redis is unavailable after a database commit, a payment settles after cancellation, or an old revoke job wakes up after a reschedule.

3independent domain state machines
DBdurable coordination point
≥1side-effect delivery semantics
Closedphysical-access failure default
Live product: LIFT & FIT Private Space The public interface keeps the complexity out of the customer's way. This article focuses on the state and failure semantics behind that interface.

The domain is a distributed transaction without distributed commit.

No database transaction can atomically commit a local booking, capture money at an external gateway, deliver an email, and create a provider-side door credential. Pretending otherwise produces familiar split-brain states: a paid customer with no booking, a confirmed booking with no access job, or a cancelled slot whose old PIN still works.

The architecture therefore gives each concern its own state machine and uses the database as the durable coordination point. Cross-system work is expressed as intent, then reconciled until local and remote state agree.

State axisRepresentative statesInvariant
SlotAvailable, held, booked, closedOnly one live booking can own an exclusive interval.
BookingPending, confirmed, cancelled, completed, refundedConfirmation requires a valid payment or pass transition.
PaymentPending, paid, refund pending, refunded, failedProvider state is accepted only after server-side reconciliation.
AccessPlanned, provisioning, disclosed, revoke pending, revokedA credential belongs to one booking, one window, and one current provision.
OutboxPending, dispatched, failedA committed domain transition cannot silently lose its required side effect.
01Claim intentA browser attempt is reduced to a stable, non-reversible idempotency key.
02Hold atomicallyThe slot, booking, payment snapshot, and timeout intent commit together.
03Pay externallyThe gateway owns money movement; local state remains explicitly pending.
04ReconcileThe backend fetches authoritative payment detail and validates the contract.
05Confirm locallyBooking and slot transition together, with an access intent in the same commit.
06Issue and revokeWorkers converge provider access while rejecting stale operations.

Booking begins with concurrency control, not form validation.

Two customers can select the same slot, and one customer can submit the same checkout twice. Those are different races. The first is contention for a scarce resource; the second is idempotency for one logical request. They need separate guards.

A short-lived browser token is transformed with a keyed hash before storage. Retries with the same token resolve to the existing booking only if the requested slot, identity context, party size, and payment mode still match. Inside the transaction, a database-level lock serializes the attempt, another serializes the slot, and pass inventory is locked independently when it is consumed.

# Reduced transaction contract; not production source.
lock(checkout_attempt)
existing = booking_by_attempt(checkout_attempt)

if existing:
    require same_request(existing, request)
    return existing

lock(slot)
require slot.state == AVAILABLE

create booking(state=PENDING, price=snapshot(slot.price))
create payment(state=PENDING, amount=snapshot(slot.price))
update slot(state=HELD)
append outbox(RELEASE_IF_UNPAID)

commit

The price is copied into the booking and payment records rather than read later from a mutable slot. The same snapshot principle applies to terms, ownership context, and the access window. A future configuration edit must not rewrite the meaning of an in-flight transaction.

A payment callback is a hint, not proof.

The browser return page is user experience, not a trust boundary. A gateway notification is also insufficient on its own: it may be duplicated, delayed, forged outside the expected channel, or refer to a payment whose local context changed. The backend uses the notification to fetch the provider's current payment detail server to server.

Reconciliation first locates the local payment, then checks amount and currency against the immutable local snapshot. Only after that does it map the provider state into the local payment machine. Booking and slot transitions are conditional, so a repeated callback cannot confirm the same pending booking twice.

// Conceptual reconciliation; provider details are intentionally omitted.
const remote = await gateway.getPayment(providerPaymentId);

require(remote.amount === payment.amountSnapshot);
require(remote.currency === payment.currencySnapshot);

if (remote.state === "paid" && booking.state === "pending") {
  transaction(() => {
    payment.markPaid();
    booking.confirmIfPending();
    slot.bookIfHeld();
    outbox.append("provision-access", booking.id);
  });
} else if (remote.state === "paid" && booking.state === "cancelled") {
  transaction(() => {
    payment.markRefundPending();
    outbox.append("request-refund", payment.id);
  });
}

The second branch is easy to miss and operationally important. Payment and cancellation race across different systems. If money settles after the local hold expired or the booking was cancelled, resurrecting the booking would violate slot ownership. The safe transition is a refund workflow with a visible intermediate state.

The outbox closes the commit-to-queue failure window.

Writing a booking and then calling a queue creates a gap: the database can commit just before the queue call fails. Calling the queue first only reverses the problem. The platform records each required side effect as an outbox row in the same database transaction as the domain change.

domain transaction
  update booking and slot
  insert outbox(effect_id, queue, payload, retry_policy)
commit

dispatcher
  read pending outbox rows
  queue with job_id derived from effect_id
  mark dispatched after queue acknowledgement

consumer
  verify current domain state
  perform an idempotent or recoverable effect

This gives at-least-once delivery, not magical exactly-once execution. If the dispatcher crashes after the queue accepts a job but before the outbox row is marked, the same deterministic job identifier suppresses the duplicate dispatch. Consumers still have to tolerate replay because a failure can occur after the provider accepted a request but before the worker observed the response.

Retries are bounded. A permanently invalid payload should become an operator-visible failed event, not an immortal loop. Recovery can advance a dispatch generation so an old failed queue record does not suppress a deliberate replay.

Physical access must reject stale work.

Access is planned locally before it is activated remotely. The plan ties a booking to a slot, a validity window, an access grant, and a concrete provision record. Sensitive credential material is encrypted at rest while needed and removed after revocation. Provider identifiers are kept separately from the customer-visible credential.

Deferred activation

Plan early; activate only inside the allowed window.

A confirmed booking can have durable access intent without keeping a remote credential active for longer than necessary.

Ambiguous response

Recover by stable identity before creating again.

If the provider may have accepted a request before a timeout, the worker looks for one exact alias match. Zero and multiple matches are different failures.

Finalization guard

Remote success is committed only if the booking is still current.

The booking must still be confirmed, the slot booked, and grant IDs plus validity window unchanged. Otherwise the new remote credential is cleaned up.

Stale revoke guard

Every destructive job carries the state it expects to destroy.

Grant, provision, provider credential, and validity window are compared before deletion. A revoke from an old schedule cannot erase the replacement credential.

The sharpest invariant appears during cancellation or rescheduling. If a credential has already been disclosed, the old slot is not immediately made sellable. It moves to a closed state until provider-side revocation is confirmed. Availability is sacrificed temporarily because selling the interval while an old credential may still work is a physical security bug, not a harmless delay.

FailureSystem responseWhy
Duplicate checkout submitReturn the matching prior attempt.Prevents duplicate bookings without accepting a changed request under the same key.
Queue unavailable after commitLeave the outbox event pending.The required side effect remains durable and observable.
Paid after cancellationEnter refund-pending workflow.Money settling late must not reclaim an already released slot.
Access-provider timeoutRecover by stable remote identity before retrying creation.A timeout does not prove that the provider rejected the first request.
Cancelled after credential disclosureClose the slot until remote revoke succeeds.Prevents resale while old physical access may remain valid.
Old revoke job after rescheduleSkip when expected IDs or window no longer match.Delayed work must not mutate a newer access generation.

Operations is part of the consistency model.

Some inconsistencies cannot be repaired safely without context. Monitoring therefore looks for stale payment states, failed outbox rows, access plans that missed their activation window, revoke-pending credentials, and mismatches between local and provider inventory. Safe repairs are bounded and idempotent; ambiguous cases are surfaced to an operator with an audit trail.

Email has its own ambiguity. A network timeout after a provider accepts a message is not the same as a definite send failure. Blindly switching providers can deliver two access emails. Delivery attempts need a stable key, durable status, and an explicit policy for outcomes where the send may have succeeded.

The admin interface is consequently not just CRUD. It is an operational view over state transitions: which component owns the current truth, which effect is pending, what can be retried, and what requires a deliberate human decision.

Five engineering lessons that transfer beyond booking software.

Lesson 01

Model independent truths independently.

Booking, payment, availability, and access should not be collapsed into one convenient status field. Their disagreements are the failure states the system must represent.

Lesson 02

Idempotency must include request equivalence.

Returning any object for a repeated key is unsafe. The replay has to describe the same logical request, or it is a conflict.

Lesson 03

Persist intent beside the state transition.

A transactional outbox turns a side effect from an unreliable function call into durable work that can be observed, retried, and audited.

Lesson 04

Retries require identity and postconditions.

A provider timeout is an unknown outcome. Stable remote identity and a local postcondition are what make recovery safer than blind repetition.

Lesson 05

Fail closed where software controls the physical world.

Temporary unavailability is preferable to selling access while credential revocation is uncertain. Product availability and security do not always optimize in the same direction.

What this article intentionally omits.

We do not publish provider credentials, callback-verification material, lock or device identifiers, credential formats, encryption keys, raw payment payloads, production routes, customer data, infrastructure addresses, exact operational thresholds, or recovery runbooks. The snippets above are reduced contracts written for this article, not source code.

The transferable value is in the invariants: local snapshots before external work, conditional state transitions, server-side reconciliation, durable side-effect intent, stale-job rejection, and explicit uncertainty when a provider outcome is ambiguous.

Where we take the system next.

The next useful work is not another dashboard widget. It is stronger evidence about correctness: property-based tests over interleaved payment and cancellation events, model-based tests for booking/access transitions, provider contract tests around ambiguous timeouts, and drift monitors that explain rather than merely count inconsistencies.

Systems like this become trustworthy by making bad states representable, detectable, and recoverable. The customer should never need to know that a payment callback raced a cancellation or that a worker restarted between two writes. The architecture exists so those events remain engineering problems instead of becoming a locked door at the start of a booked session.

Back to Journal Research notes, systems writeups, and technical implementation essays.

Have a technical system to build?

Tell us what you are solving. We will come back with a concrete next step.

Contact Gloryck