A resumed request can fail even when an earlier artifact remains useful. Reading the supplemental import code again led me to separate the decision about which artifact to retain from the account of what the new attempt actually achieved.
A resumed job carries two expectations
The historical conversation contains both a question about whether everything had been downloaded and a request to resume. The recorded answers distinguished saved material, queued work and failures. For this article I read the supplemental import implementation behind that workflow. Those old progress figures are historical evidence; they do not describe the current condition of any external site.
A person asking to resume expects earlier work to remain useful. They also expect to learn when the new attempt fails. Those expectations can both be reasonable for the same resource. An earlier artifact may still be available while a more recent attempt to obtain it has failed. A single success badge has too little vocabulary for that situation.
If the newest row always replaces the previous row, a failed retry can remove the earlier artifact from the inventory even though it was previously saved. If only the older success survives, the inventory can make the retry appear successful. My design question was how to select the artifact we can use while keeping a separate account of what the latest attempt actually did.
Newer metadata does not automatically win
The inspected implementation first indexes the previous records by resource address. A supplemental row replaces its predecessor when it has a saved path. It also remains when there was no earlier row, or when the earlier row had no path either. An earlier row with a path survives a new row without one. That is a preference for a recorded artifact, rather than an unconditional preference for recency.
In ordinary language: use the newly saved material when it exists; otherwise keep the earlier saved record; when no usable record has existed, retain the new failure as a failure. Keeping the earlier record must not turn the failed retry into a successful attempt. The selection answers which recorded material remains available to the workflow, rather than rewriting the history of the retry.
The actual source condition uses the presence of a path field. It does not independently establish that the file still exists, that its bytes are intact, or that its contents match the current remote resource. This distinction matters when explaining the design. A metadata preference can preserve a useful reference, but a later file check is still needed before claiming that the referenced artifact is presently readable.
Record the attempt before folding it into the inventory
Before merging, the supplemental importer writes the full records for that attempt into a separate report. It then combines the earlier records with the supplemental ones and writes the consolidated reports. Reading the two together preserves an explanation for a retained artifact: the earlier row was selected even though the latest attempt did not save a replacement.
Consider an invented instruction document that was saved earlier and could not be obtained on the next attempt. The consolidated inventory can still select the earlier row. The attempt report still contains the failure. A useful interface could say “Earlier saved copy retained; latest import failed.” Calling it “updated to the latest copy” would promise an event that did not happen. This wording is a proposal developed here, not a claim that I added it to the original interface.
A separate report is also not automatically a permanent audit history. The implementation writes to a fixed supplemental report name; that alone is not a distinct, retained record for every execution. A durable history would need additional decisions about execution identifiers, timestamps, retention and reconstruction. I would establish those contracts before presenting the report as a complete account of every attempt.
Inventory counts and attempt results answer different questions
After merging, the code recalculates saved and failed counts from the selected consolidated records. The job status, however, was assigned before that merge according to failures in the supplemental attempt. The exit code also follows the supplemental failure count. Consequently, the inventory can retain an older saved record while the latest execution still reports a partial outcome or returns a failure exit code.
Forcing these values to match can remove the very signal an operator needs. The inventory count describes the selected records. The supplemental failure count describes requests the current attempt did not resolve. Labels such as “selected saved records” and “failed attempts in this run” make that scope visible. They allow an agent or a person to decide what to retry without throwing away the usable earlier selection.
Another inspected path in the crawler returns already saved HTML bytes when a local cached page exists. A success status in that path should not be generalized into proof of a new network response. Cached material, a retained earlier row and a newly obtained artifact have different provenance. Resumption can make good use of all three, provided its report does not erase the distinction.
The reusable example records bytes and selection provenance
I built an independent in-memory example to make this decision inspectable in another program. It accepts invented resource identifiers, saved/failed/skipped outcomes and supplied synthetic bytes. It accepts no resource addresses or storage paths. Duplicate identifiers within either input list are rejected instead of silently taking the last row. This is a deliberately stricter teaching contract, not a reproduction of the original address-based importer.
A saved input must contain a bytes value and passes through the existing byte receipt comparison function. The selected artifact receives the length and SHA calculated from those supplied bytes. A latest saved input wins; otherwise an earlier saved input wins when present. The output retains both outcomes, the side selected and whether an earlier artifact was retained after a failed attempt. A skipped attempt remains distinct from a failed one.
That receipt compares supplied bytes with a supplied length. Because the caller supplies both, it is not independent observation of a remote transfer. The helper locates no file, writes nothing and proves neither freshness, remote origin nor survival after power loss. Each input list is limited to 64 records, and each byte value follows the existing module’s 64 KiB bound. Larger artifacts and real address identity rules require an additional integration design.
Explore the implementation
Complete independent synthetic merge helper
"""Independent synthetic merge; place next to file_receipt.py from linked ZIP."""
from copy import deepcopy
import re
from file_receipt import compare_download_bytes
def merge_inventory(previous, latest):
def index(rows):
if type(rows) is not list or len(rows) > 64:
raise ValueError('BOUNDED_RECORD_LIST')
out = {}
for row in rows:
if type(row) is not dict or set(row) != {'id', 'outcome', 'data'}:
raise ValueError('EXACT_RECORD_FIELDS')
key = row['id']
if type(key) is not str or not re.fullmatch(r'[a-z][a-z0-9_-]{0,39}', key):
raise ValueError('RESOURCE_ID')
if key in out:
raise ValueError('DUPLICATE_RESOURCE_ID')
if row['outcome'] not in ('saved', 'failed', 'skipped'):
raise ValueError('OUTCOME')
if row['outcome'] == 'saved':
if type(row['data']) is not bytes:
raise ValueError('SAVED_BYTES_REQUIRED')
compare_download_bytes(row['data'], len(row['data']), True)
elif row['data'] is not None:
raise ValueError('FAILED_OR_SKIPPED_BYTES_MUST_BE_ABSENT')
out[key] = row
return out
a, b = index(previous), index(latest)
result = []
for key in sorted(a.keys() | b.keys()):
old, new = a.get(key), b.get(key)
chosen = (new if new and new['outcome'] == 'saved'
else old if old and old['outcome'] == 'saved' else None)
source = ('latest' if chosen is not None and chosen is new
else 'previous' if chosen is not None else 'none')
receipt = (compare_download_bytes(chosen['data'], len(chosen['data']), True)
if chosen else None)
result.append({
'id': key,
'selected_from': source,
'selected_receipt': receipt,
'previous_outcome': old['outcome'] if old else None,
'latest_outcome': new['outcome'] if new else None,
'retained_after_failed_attempt': (
source == 'previous' and new is not None and new['outcome'] == 'failed'
),
'fresh_network_response_verified': False,
'performs_fetch_or_write': False,
})
return deepcopy(result)
Test failure, equal-length changes and skipped attempts
I downloaded the existing public ZIP, checked its 5,859 bytes and SHA, extracted it and ran its 36 packaged tests. I then ran 22 separate tests for the new merge helper that uses that module. These counts describe synthetic code checks. They are neither original crawler tests nor successful collections from external sites.
The new checks confirm that an earlier digest survives a later failure and that different bytes of the same length replace the earlier digest when the new input is saved. They cover recovery from an earlier failure, a new failure with no available artifact, no new attempt and a skipped attempt. They also cover duplicate identifiers, invalid outcomes, unexpected fields, the byte limit, empty saved bytes, unchanged inputs and exclusion of raw byte content from the output.
Five illustrative scenarios were executed. The excerpt below is the actual synthetic output for an earlier saved input followed by a failed attempt. It selects previous while retaining failed as the latest outcome. The flags for a verified fresh network response and for fetching or writing remain false. That coexistence is the behavior under test: preserving an artifact must not conceal what happened in the latest attempt.
Explore the implementation
Actual synthetic execution output
[
{
"id": "asset",
"selected_from": "previous",
"selected_receipt": {
"verified": true,
"reason": "supplied_bytes_match",
"size": 3,
"sha256": "cba06b5736faf67e54b07b561eae94395e774c517a7d910a54369e1263ccfbd4",
"expected_digest_checked": false,
"network_origin_verified": false,
"power_loss_durability_verified": false
},
"previous_outcome": "saved",
"latest_outcome": "failed",
"retained_after_failed_attempt": true,
"fresh_network_response_verified": false,
"performs_fetch_or_write": false
}
]Decide the integration contract before using the rule
Take file_receipt.py from the existing byte receipt ZIP linked below and save the complete helper above as merge_inventory.py beside it. The merge helper is an inline addition to this article. It has not been inserted into or published as a new version of the existing ZIP. The small invocation below uses invented bytes and accesses no site or file. Start by reading the selected side and the latest outcome together.
A real integration still needs a resource identity policy, a point at which to read and check saved files, and a retention policy for attempt reports. Address normalization needs an explicit rule when several addresses can identify the same material. Whether an earlier artifact is acceptable also depends on the work: an archived design reference and a current price or permission check have different freshness requirements. The example’s selection policy is a tool for that decision, not a universal instruction to reuse old material.
The earlier archive article separated a readable ZIP from a complete set of referenced artifacts. This article follows a different question: when resuming, which record should remain selected and where should the latest failure survive? I did not rerun the original collection or browser, and this work does not repair the earlier missing reference. The two articles and the reusable comparison module provide a starting point for reports that preserve earlier work while making unresolved new attempts visible.
Explore the implementation
Minimal reproducible invocation
from merge_inventory import merge_inventory
previous = [{'id': 'guide', 'outcome': 'saved', 'data': b'earlier'}]
latest = [{'id': 'guide', 'outcome': 'failed', 'data': None}]
result = merge_inventory(previous, latest)[0]
assert result['selected_from'] == 'previous'
assert result['latest_outcome'] == 'failed'
assert result['retained_after_failed_attempt'] is True
print(result)