A healthy output is difficult to trust without knowing which failures a checker can skip. I developed this small public HTML probe after reading K-POP PULSE’s historical HEAD and feed-comparison records. The code and reproducible checks are shared so it can be adapted to another website.
It compares one public server document with specified text. It does not certify every body structure, browser behavior or factual claim. I describe that limited scope while requiring an unreadable result to remain distinct from success.
Specify the expected document before checking
An address alone can establish only that something responded. Supply an expected heading and a distinctive sentence from the article. A generic menu label may occur on the wrong page; choose content that should belong to this particular manuscript.
The program extracts server-document text excluding script, style and title content, then normalizes entities and whitespace. It does not interpret CSS visibility or content later created by JavaScript. Checking the precise locations and structure of expected prose requires a separate comparison.
Explore the implementation
Replace PUBLIC_URL with the canonical public address and choose distinctive expected text.
python -X utf8 publication_probe.py PUBLIC_URL --contains "Article heading" --contains "A unique body sentence"Read outcomes and exits with the same meaning
A 200 HTML response containing all specified text produces passed and exit 0. A 404 or absent expected text produces missing or mismatch and exit 1. Those outcomes are reasons to compare the publication state or intended content again.
Transport failure, unexpected responses and documents that cannot be inspected because of type, encoding or size produce unverified and exit 2. This does not establish that the article was lost. It means the required inspection could not be completed. An invalid invocation is not reported as success either.
Check the request, response type and limits together
The module uses GET to read the actual document. A 200 JSON response is not accepted as an inspected HTML body. Exceeding the size limit does not permit the beginning of the document to pass as a complete inspection. Invalid character encoding follows the same rule.
Redirects are not automatically followed. Decide whether to inspect the canonical address or verify a redirect with a different contract. This does not label every redirect wrong; it prevents the intended check from silently becoming another request.
Implementation and actual reproducible checks
The complete shared program is below. It uses the Python standard library and performs no manuscript storage or publishing requests. The downloadable package includes a synthetic local HTTP server and CLI-execution checks, allowing failures to be reproduced without changing production.
The actual tests cover successful GET, unsupported HEAD, a different 200 document,404, 405, redirects, wrong type, invalid encoding, size limits and real subprocess exits. They also distinguish entity-encoded prose from a marker present only in script text. These are tests of the new program, not an inspection of every current state in the historical service.
Explore the implementation
A newly written public HTML probe, not a recovered historical checker.
"""A new public HTML probe. It does not publish, authenticate or verify source truth."""
import argparse
import json
from html.parser import HTMLParser
from urllib.error import HTTPError, URLError
from urllib.parse import urlsplit
from urllib.request import HTTPRedirectHandler, ProxyHandler, Request, build_opener
MAX_BYTES=1024*1024
class DocumentText(HTMLParser):
def __init__(self):
super().__init__(convert_charrefs=True);self.skip=0;self.chunks=[]
def handle_starttag(self,tag,attrs):
if tag in ['script','style','title']:self.skip+=1
def handle_endtag(self,tag):
if tag in ['script','style','title'] and self.skip:self.skip-=1
def handle_data(self,value):
if not self.skip:self.chunks.append(value)
def text(self):return ' '.join(' '.join(self.chunks).split())
def inspect_response(status, content_type, body, markers, max_bytes=MAX_BYTES):
if not isinstance(markers,(list,tuple)) or not markers or any(not isinstance(m,str) or not m.strip() for m in markers):
raise ValueError('Provide at least one nonempty expected text marker')
if not isinstance(max_bytes,int) or max_bytes<1:raise ValueError('Invalid body-size limit')
if not isinstance(body,bytes):raise TypeError('Response body must be bytes')
result={'state':'unverified','http_status':status,'reason':'','missing_indices':[],'bytes_read':len(body)}
if status==404:result.update(state='missing',reason='Page not found');return result
if status!=200:result['reason']='Unexpected HTTP response';return result
if len(body)>max_bytes:result['reason']='Body exceeds the inspection limit';return result
if str(content_type).split(';',1)[0].strip().lower() not in ['text/html','application/xhtml+xml']:
result['reason']='Expected an HTML response';return result
try:text=body.decode('utf-8')
except UnicodeDecodeError:result['reason']='Response is not valid UTF-8';return result
parser=DocumentText();parser.feed(text);document=parser.text()
missing=[i for i,marker in enumerate(markers) if ' '.join(marker.split()) not in document]
if missing:result.update(state='mismatch',reason='Expected text is absent',missing_indices=missing);return result
result.update(state='passed',reason='HTTP and expected server-document text matched');return result
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self,request,fp,code,message,headers,newurl):return None
def probe(url,markers,timeout=10,max_bytes=MAX_BYTES):
# Validate the comparison contract before making a network request.
inspect_response(0,'',b'',markers,max_bytes)
parsed=urlsplit(url)
if parsed.scheme not in ['http','https'] or not parsed.hostname or parsed.username or parsed.password:
raise ValueError('Use a public HTTP(S) address without embedded credentials')
opener=build_opener(ProxyHandler({}),NoRedirect())
request=Request(url,headers={'User-Agent':'Public-HTML-Probe/1.0'},method='GET')
try:
with opener.open(request,timeout=timeout) as response:
status=response.status;kind=response.headers.get('Content-Type','');body=response.read(max_bytes+1)
except HTTPError as error:
with error:
status=error.code;kind=error.headers.get('Content-Type','');body=error.read(max_bytes+1)
except (URLError,OSError,TimeoutError) as error:
return {'state':'unverified','http_status':None,'reason':'Transport failure: '+type(error).__name__,'missing_indices':[],'bytes_read':0}
return inspect_response(status,kind,body,markers,max_bytes)
def exit_code(result):
return 0 if result['state']=='passed' else 1 if result['state'] in ['missing','mismatch'] else 2
def main():
parser=argparse.ArgumentParser(description='Check one public HTML response and explicit expected text. No writes.')
parser.add_argument('url');parser.add_argument('--contains',action='append',required=True)
parser.add_argument('--timeout',type=float,default=10);parser.add_argument('--max-bytes',type=int,default=MAX_BYTES)
args=parser.parse_args()
if args.timeout<=0:parser.error('Timeout must be positive')
try:result=probe(args.url,args.contains,args.timeout,args.max_bytes)
except (ValueError,TypeError) as error:parser.error(str(error))
print(json.dumps(result,ensure_ascii=False))
return exit_code(result)
if __name__=='__main__':raise SystemExit(main())
Adapt the comparison conditions
If a document exceeds the default limit, inspect its actual requirements before changing max-bytes. Adjusting a justified limit is different from silently inspecting only a truncated beginning. Set timeout according to the service’s conditions as well.
Check each language’s address with text from that language. After an edit, derive the markers from the currently intended manuscript. A complete body requirement calls for comparing every paragraph, list, code block and file against the reviewed content. Multiple markers alone do not establish identical meaning throughout.
Preserve failures in the surrounding automation
If a parent runner ignores exit 2and prints overall success, the module’s distinction disappears. Keep each address’s result, and retain incomplete verification in the final report. Confirmed absence and failure to inspect also call for different follow-up work.
The download contains the program, reproducible checks and Korean and English instructions. You may copy and adapt it for your website. Use it as a starting point for keeping successful deployment, a matching document, faithful body delivery and actual screen behavior as separate kinds of evidence.