After building a runner that returns a short result, I also needed an honest explanation of what became smaller. Total tokens, uncached input, transmitted bytes and development usage answer different questions. I made an independent small ledger so those distinctions can be exercised rather than left as reporting advice.
This ledger is not a usage-collection connector
Inputs are caller-declared synthetic records: a call ID, development or normal-run bucket, completed/failed/cancelled status, and token and byte fields. The code reads no account or session and makes no model call. It does not authenticate a record against a provider response, so provider_authenticated always remains False.
The original internal CLI reviewed as source is already retired historical implementation. Retained accounting schemas and tests are not restored as a current model runner. This example's strict record contract is an independent choice for understanding units, not a universal response schema for every service. Currency cost and subscription allowance are outside its scope.
Do not add subset counts to the total again
Within this contract, cached is a subset of input and reasoning a subset of output. A known total must equal input plus output; contradictory records that add subsets again are rejected. If total is absent but both components exist, an effective total is derived and labeled separately from a reported total.
Uncached input is input minus cached. Uncached input plus output is another explicitly named measure, never renamed total. Unknown cache makes uncached input unknown as well. Values are not filled with zero just to make subtraction possible. The calculation conditions and metric name travel together so two readers of the same table can tell which quantity they are comparing.
A sum with unavailable calls is visibly partial
Every metric returns known_sum, unknown_calls and complete. Known values can be added, but missing calls prevent a complete aggregate. A reported total without input/output details is handled separately: a complete total metric does not imply known input, cache or reasoning details.
An explicit zero remains known. Failed and cancelled calls stay in the ledger, including their unavailable usage. Wire bytes are aggregated independently and never converted into tokens. Instead of dropping unknown rows to produce a more flattering successful-run table, the result shows which parts remain unknown. Different completeness flags can therefore coexist legitimately within one summary.
Explore the implementation
Complete independent synthetic implementation
"""Strict synthetic usage ledger. Counts are caller declarations, not billing evidence."""
from copy import deepcopy
TOKEN_FIELDS = ('input', 'cached', 'output', 'reasoning', 'total')
BYTE_FIELDS = ('request_bytes', 'response_bytes')
FIELDS = {'id', 'bucket', 'status', *TOKEN_FIELDS, *BYTE_FIELDS}
def count(value):
if value is not None and (type(value) is not int or value < 0 or value > 9007199254740991):
raise ValueError('NONNEGATIVE_INTEGER_OR_UNKNOWN_REQUIRED')
return value
def normalize(row):
if type(row) is not dict or set(row) != FIELDS:
raise ValueError('EXACT_RECORD_FIELDS_REQUIRED')
if type(row['id']) is not str or not row['id'].strip() or len(row['id']) > 120:
raise ValueError('CALL_ID_REQUIRED')
if row['bucket'] not in ('development', 'normal_run') or row['status'] not in ('completed', 'failed', 'cancelled'):
raise ValueError('RECORD_CATEGORY_INVALID')
for key in TOKEN_FIELDS + BYTE_FIELDS:
count(row[key])
i, c, o, r, t = (row[k] for k in TOKEN_FIELDS)
if i is not None and c is not None and c > i:
raise ValueError('CACHE_IS_A_SUBSET_OF_INPUT')
if o is not None and r is not None and r > o:
raise ValueError('REASONING_IS_A_SUBSET_OF_OUTPUT')
if t is not None:
if (i is not None and o is not None and t != i + o) or t < (i or 0) + (o or 0):
raise ValueError('TOTAL_INCONSISTENT_WITH_INPUT_OUTPUT')
return deepcopy(row)
def derived(row):
i, c, o, r, t = (row[k] for k in TOKEN_FIELDS)
uncached = None if i is None or c is None else i - c
total = t if t is not None else None if i is None or o is None else i + o
return {**{k: row[k] for k in TOKEN_FIELDS + BYTE_FIELDS},
'effective_total': total,
'uncached_input': uncached,
'uncached_input_plus_output': None if uncached is None or o is None else uncached + o,
'total_source': 'reported' if t is not None else 'derived' if total is not None else 'unknown'}
def aggregate(rows):
values = [derived(row) for row in rows]
keys = TOKEN_FIELDS + BYTE_FIELDS + ('effective_total', 'uncached_input', 'uncached_input_plus_output')
result = {'calls': len(rows), 'status_counts': {s: sum(r['status'] == s for r in rows) for s in ('completed', 'failed', 'cancelled')},
'metrics': {}, 'total_sources': {s: sum(v['total_source'] == s for v in values) for s in ('reported', 'derived', 'unknown')},
'provider_authenticated': False, 'currency_cost_calculated': False}
for key in keys:
known = [v[key] for v in values if v[key] is not None]
result['metrics'][key] = {'known_sum': sum(known), 'unknown_calls': len(rows) - len(known), 'complete': len(rows) == len(known)}
return result
class UsageLedger:
def __init__(self):
self._rows = {}
def add(self, row):
clean = normalize(row)
old = self._rows.get(clean['id'])
if old is not None:
if old != clean:
raise ValueError('CALL_ID_CONFLICT')
return False
self._rows[clean['id']] = clean
return True
def rows(self):
return deepcopy(list(self._rows.values()))
def summary(self, bucket=None):
if bucket is not None and bucket not in ('development', 'normal_run'):
raise ValueError('BUCKET_INVALID')
return aggregate([r for r in self._rows.values() if bucket is None or r['bucket'] == bucket])
def compare_samples(self, before_ids, after_ids):
for ids in (before_ids, after_ids):
if type(ids) is not list or not ids or any(type(x) is not str for x in ids) or len(ids) != len(set(ids)):
raise ValueError('DISTINCT_NONEMPTY_CALL_LIST_REQUIRED')
if any(x not in self._rows for x in ids):
raise ValueError('CALL_NOT_FOUND')
if set(before_ids) & set(after_ids):
raise ValueError('OVERLAPPING_SAMPLES')
rows = [self._rows[x] for x in before_ids + after_ids]
if any(r['bucket'] != 'normal_run' for r in rows):
raise ValueError('COMPARE_NORMAL_RUNS_ONLY')
before, after = (aggregate([self._rows[x] for x in ids]) for ids in (before_ids, after_ids))
changes = {}
for key in ('effective_total', 'uncached_input_plus_output'):
a, b = before['metrics'][key], after['metrics'][key]
complete = a['complete'] and b['complete']
changes[key] = {'before': a, 'after': b,
'difference': b['known_sum'] - a['known_sum'] if complete else None,
'reduction_percent': round(100 * (a['known_sum'] - b['known_sum']) / a['known_sum'], 4) if complete and a['known_sum'] else None}
return {'sample_changes': changes, 'controlled_experiment_verified': False,
'general_savings_guaranteed': False, 'currency_cost_calculated': False}
Separate redelivery of a receipt from another call
An identical ID and record delivered again is not counted as another call. The same ID with different usage is rejected as a conflict rather than silently replacing history. IDs are caller-supplied, not independently authenticated provider request IDs; an operational system still needs the original evidence connection.
Development and normal_run aggregate separately. Improvement work including failures and one normal repetition are different scopes. The comparison function accepts normal-run samples only. That prevents this teaching API from using large development usage as the baseline for a dramatic normal-execution savings claim, while still retaining both buckets in the complete ledger.
A comparison calculates only the difference between samples
Call sets must be nonempty and disjoint. A metric needs complete values on both sides before difference and reduction percentage are calculated, and a zero baseline produces no percentage. Labelling both samples normal runs does not establish matching tasks, models, cache, context length or execution order.
Controlled_experiment_verified and general_savings_guaranteed therefore stay False. A displayed reduction is arithmetic over the supplied samples. It does not turn a successful run or historical comparison into fixed savings for every customer. I made those limits output fields because they mattered while interpreting the original development measurements as well.
Keep tests and execution records in the same stated scope
Thirty-two tests run against the independent source and again after ZIP extraction. They cover double-counted cache/reasoning, unknown versus zero, missing component detail, retained failed/cancelled usage, ID conflict, development-baseline refusal and zero-denominator percentages. Five records are generated by actual local function calls; the selector reads those saved results.
Code, tests, record generation and documentation permit modification and reuse. Persistent storage, concurrent writers, automatic collection and pricing integration are not implemented. Even after adding operational features, the unit and measurement interval must remain explicit. I share a contract for producing comparable figures rather than treating the word savings as sufficient evidence.