Normal runs hide what remains when recording fails after an effect. Taking questions from Northstar's storage and cancellation tests, I made a small lab with no external transmission. It records the requested fault, whether execution actually reached it, and how many synthetic sends occurred.
Choose a failure point from a fixed vocabulary
Lab accepts fixed faults for preparation recording, dispatch recording, native reply, verification and final recording. Cancellation can occur after preparation, immediately before sending or after sending. It accepts neither an arbitrary callback nor an actual file path. One run object can be started only once.
The restriction identifies an observation experiment rather than an automatic retry system. Even a pre-dispatch failure cannot restart the same object. When a new operational request is appropriate remains a separate decision based on current results and human intent. This package grants no such authority. Its small input vocabulary makes the effect of each declared interruption inspectable.
Separate checkpoint labels from the number of sends
A memory checkpoint can say dispatched while cancellation immediately before sending leaves send_count at zero. Conversely, native_reply failure happens after one simulated receiver effect but before a useful result read. A single status word cannot distinguish those opposite situations.
The synthetic receiver only increments a fixed effect count and uses no network. Read observations are declared match, mismatch or unavailable conditions. Expected_result_observed means the model's read condition matched; it proves no live save or payment. Stating where both the effect and its evidence come from prevents a teaching experiment from lending unsupported credibility to another product.
A failed final write does not erase a matching read
With final_write failure, evidence of a matching read remains, but final_receipt_written is false and status is needs_review. A successful read does not imply persistence succeeded. The absent final record also does not authorize another synthetic send.
Inspect returns a deep copy of current results. Repeated reads do not increase send_count, and mutating the returned trace cannot change the internal one. Memory_checkpoints is not a disk journal. Durable_receipt_verified stays false even on normal success. Those distinctions retain result observation, recording and authority to resend as separate questions rather than make one successful flag answer all three.
Explore the implementation
Complete independent synthetic implementation
"""Deterministic, in-memory failure teaching lab. No I/O, timers or replay."""
from copy import deepcopy
FAULTS = frozenset({'none','prepare_write','dispatch_write','native_reply','verification','final_write'})
CANCELLATIONS = frozenset({'none','after_prepare','before_send','after_send'})
OBSERVATIONS = frozenset({'match','mismatch','unavailable'})
class Fault(Exception):
pass
class Lab:
def __init__(self, *, fault='none', cancel_at='none', observation='match'):
for value, allowed in [(fault,FAULTS),(cancel_at,CANCELLATIONS),(observation,OBSERVATIONS)]:
if type(value) is not str or value not in allowed:
raise ValueError('INVALID_SCENARIO')
self.fault, self.cancel_at, self.observation = fault, cancel_at, observation
self._started = False
self._state = dict(status='ready', stage='ready', trace=[], memory_checkpoints=[],
send_count=0, simulated_effect_count=0, read_count=0,
expected_result_observed=False, final_receipt_written=False,
cancellation_observed=False, fault_requested=fault,
fault_exercised=False, outcome='not_dispatched', error=None,
real_network_used=False, real_filesystem_used=False,
durable_receipt_verified=False, automatic_replay=False)
def inspect(self):
"""Read only; even repeated inspection cannot send or certify persistence."""
return deepcopy(self._state)
def _point(self, stage):
self._state['stage'] = stage
self._state['trace'].append(stage)
if stage == self.fault:
self._state['fault_exercised'] = True
raise Fault(stage)
def _cancel(self, stage):
if stage == self.cancel_at:
self._state['cancellation_observed'] = True
self._state['trace'].append('cancel:'+stage)
raise Fault('cancelled')
def run(self):
if self._started:
raise ValueError('RUN_ALREADY_STARTED')
self._started = True
s = self._state
s['status'] = 'running'
try:
self._point('prepare_write')
s['memory_checkpoints'].append('prepared')
self._cancel('after_prepare')
self._point('dispatch_write')
s['memory_checkpoints'].append('dispatched')
self._cancel('before_send')
# This is a synthetic receiver, not an external operation.
s['send_count'] += 1
s['simulated_effect_count'] += 1
s['outcome'] = 'unknown'
s['trace'].append('synthetic_send')
self._cancel('after_send')
self._point('native_reply')
s['read_count'] += 1
self._point('verification')
if self.observation == 'match':
s['expected_result_observed'] = True
s['outcome'] = 'observed_match'
else:
s['outcome'] = 'observed_mismatch' if self.observation == 'mismatch' else 'unknown'
raise Fault('result_not_verified')
self._point('final_write')
s['memory_checkpoints'].append('succeeded')
s['final_receipt_written'] = True
s['status'] = 'succeeded'
s['stage'] = 'finished'
except Fault as e:
s['error'] = str(e)
s['status'] = 'needs_review' if s['send_count'] else ('cancelled' if s['cancellation_observed'] else 'failed')
self.assert_invariants()
return self.inspect()
def assert_invariants(self):
s = self._state
assert s['send_count'] in {0,1}
assert s['simulated_effect_count'] == s['send_count']
assert s['read_count'] in {0,1}
if s['status'] == 'succeeded':
assert s['send_count']==1 and s['read_count']==1
assert s['expected_result_observed'] and s['final_receipt_written']
assert not s['cancellation_observed'] and s['error'] is None
if not s['send_count']:
assert s['outcome']=='not_dispatched' and not s['expected_result_observed']
assert not s['automatic_replay'] and not s['durable_receipt_verified']
Record the requested fault and the one actually exercised
Requesting final-write failure does not exercise it if cancellation already ends the run after preparation. Fault_exercised then remains false. Naming a fault in a test is not evidence that execution reached it. Without reachability, an early exit can misleadingly resemble resilience against every failure.
Twenty-eight independent tests cover at-most-one send across all declared combinations, the read-and-receipt requirements for normal success, deep-copy inspection and refusal to restart. Combinations within a test are not added to the test count. Five records were saved from actual local function calls; the selector reads them. No original application or external-service test was newly executed.
An operational adaptation still needs real persistence and recovery
Code, tests, record generation and explanation permit modification and reuse. The model has no asynchronous threads, clock, filesystem, database, browser, authentication or forced-exit recovery. Explicit event ordering compares scenarios; it does not explore every timing of a real concurrent system.
Operations need additional guarantees around persistent-write atomicity, server-side read evidence and state after restart. The value of this example is a precise answer to a small question. Retaining where failure occurred, how many sends happened and what remains unverified exposes design issues hidden by repeated normal success. Implement and test the missing operational layers before applying its conclusions beyond the model.