technical note 05 · network clients, text extraction & human factors
Fixed-Frame Terminal Reading: TLS Handshake Escalation, Boilerplate Stripping, and Bounded Discovery in TermBrow
Why passive CDN TLS ClientHello fingerprinting blocks standard Python HTTP clients, how TermBrow escalates cleanly from honest User-Agents to curl_cffi BoringSSL impersonation and Wayback snapshots, defuses 29 classes of tracker links, and budgets local 0.98-decay topic curation without opaque engagement ranking.
1. Visual Re-Orientation Cost Across Heterogeneous Documentation
Reading across twenty unrelated technical references in a conventional web browser — an IETF RFC, a Substack post, an academic preprint landing page, a documentation wiki, and a news archive — forces continuous perceptual re-calibration. Column widths swing from 48ch to full-bleed 160ch; sticky headers, newsletter modals, and related-article carousels compete for peripheral attention; body contrast and font metrics change on every click.
TermBrow (a Python / Textual terminal browser, and the exact UI frame powering termbrow.org) treats layout invariance as a client-side invariant: one background, one accent color, one centered 84ch reading column, and identical peripheral chrome in the same screen coordinates regardless of origin domain. Every retrieved document is reduced to clean structured Markdown (headings, paragraphs, tables, code blocks, and numbered links) before rendering.
2. Passive TLS Fingerprinting and the Four-Tier Retrieval Ladder
Building a terminal reader in Python quickly hits a transport-layer wall that has nothing to do with HTML parsing. A polite request sent via Python's httpx or urllib (linked against system OpenSSL) routinely receives HTTP 403 Forbidden or an interactive JavaScript challenge from CDN edge proxies (Cloudflare, Akamai, DataDome), even on public documentation and reference articles — and even if the caller copies a verbatim Chrome User-Agent string.
The discriminator is passive transport fingerprinting prior to HTTP header inspection:
- TLS
ClientHello(JA3 / JA4): OpenSSL's default cipher-suite list, extension permutation order, elliptic-curve groups, GREASE absence, and ALPN advertisement differ visibly from BoringSSL (Chrome) and NSS (Firefox). - HTTP/2 frame negotiation: Initial
SETTINGSwindow sizes,WINDOW_UPDATEincrements, and pseudo-header ordering (:method,:authority,:scheme,:path) identify non-browser HTTP stacks on the first flight of packets.
Always impersonating Chrome on every request would be needlessly dishonest toward origin servers that welcome lightweight readers; never impersonating Chrome makes a large fraction of the public web unreadable from a terminal. In termbrow/fetch.py, fetch_page(url) resolves this via a strict four-tier escalation ladder gated on a minimum extracted prose threshold (_MIN_READABLE = 280 Markdown characters after boilerplate removal):
_MIN_READABLE = 280 # extracted Markdown chars required to accept a tier
# Tier 1: Honest terminal reader UA over async httpx + per-domain cookie jar
# User-Agent: "TermBrow/1.0 (terminal reading browser) Mozilla/5.0"
# Tier 2: Standard desktop Chrome Sec-Fetch-* & Accept-Language header set
# Tier 3: BoringSSL TLS ClientHello + HTTP/2 impersonation via curl_cffi
# curl_cffi.requests.get(url, impersonate="chrome", timeout=20)
# Tier 4: Internet Archive Wayback Availability API fallback
# GET https://archive.org/wayback/available?url=<encoded>
Two implementation details keep this escalation predictable and privacy-preserving:
- Extraction-gated fallback rather than status-code-only fallback: Many soft-paywall or bot-wall responses return
HTTP 200 OKcontaining a 140-character "Please enable JavaScript and cookies to continue" shell. Because each tier passes its HTML through content extraction before checkinglen(markdown) >= _MIN_READABLE, soft 200-status shells automatically escalate to Tier 3 (browser-impersonation) or Tier 4 (wayback), and the status bar explicitly reports which retrieval tier succeeded (via: direct,via: browser-impersonation, orvia: wayback). - Strict
.onioncircuit isolation: Whentor.is_onion(url)detects a v3 onion service hostname,fetch_pagebypasses all clearnet tiers and the Wayback fallback completely, routing exclusively through the local Tor SOCKS5 proxy (127.0.0.1:9050or9150) so onion hostnames are never leaked to clearnet DNS resolvers orarchive.org.
3. Boilerplate Stripping and Ad/Tracker Link Defusing
Raw DOM trees are converted to Markdown via trafilatura.extract(..., output_format="markdown", include_links=True, include_comments=False, include_tables=True, favor_recall=True), with a secondary BeautifulSoup structural extractor if trafilatura yields an empty tree. However, preserving inline article hyperlinks introduces a residual hazard: inline affiliate redirects and ad-network click-wrappers embedded inside body paragraphs.
Rather than running a heavyweight filter list engine, fetch.py applies a deterministic post-extraction pass (_defuse_ad_links) across 29 ad-tech and click-tracker host substrings (doubleclick.net, googlesyndication.com, googleadservices.com, taboola.com, outbrain.com, criteo.com, adsrvr.org, app.adjust.com, onelink.me, etc.):
_MD_LINK_RE = re.compile(r"\[([^\]]+)\]\((https?://[^)\s]+)\)")
def _defuse_ad_links(md: str) -> str:
def _check(m: re.Match) -> str:
text, href = m.group(1), m.group(2)
host = urlparse(href).netloc.lower()
if any(ad in host for ad in _AD_HOSTS):
return text # preserve readable prose, strip outbound tracker URL
return m.group(0)
return _MD_LINK_RE.sub(_check, md)
Rewriting [anchor](https://tracker.example/...) to plain anchor keeps sentences grammatically intact while making tracker navigation impossible.
4. Inspectable 0.98-Decay Topic Weights and Bounded Discovery
Pure chronological RSS readers exhibit zero cross-domain serendipity; commercial recommendation feeds optimize opaque engagement objectives on remote servers. TermBrow implements a local, inspectable middle ground stored entirely as plain JSON under ~/.termbrow/{history,keywords,library,prefs}.json (written crash-safely via .tmp write + POSIX/NTFS atomic Path.replace()).
On every visited article, extract_keywords(title, markdown, top_n=8) tokenizes alphabetic words of length $\ge 4$ outside a 110-word English/web stopword set, weights title tokens $3\times$ against body tokens $1\times$, and updates keywords.json via exponential decay:
A multiplicative decay factor of _DECAY = 0.98 per page visit gives every topic weight a deterministic half-life of $\ln(0.5)/\ln(0.98) \approx 34.3$ page reads: sustained research threads rise naturally above 3.0–8.0, while one-off detours decay below the 0.05 pruning floor without manual cleanup.
When populating the horizontal For You · Explore strip (both in the CLI and at the top of termbrow.org — press Alt+E or run :keywords in the bar above to inspect your live local vector), curate.py partitions limit slots strictly according to the user-selected exploration ratio:
Focus (15% explore): 85% direct keyword-affinity matches, 15% discovery slots (marked with◇).Balanced (40% explore): 60% affinity matches, 40% anchored discovery slots.Discover (65% explore): 35% affinity matches, 65% cross-domain discovery slots.
Crucially, discovery slots in the desktop browser are anchored rather than random: _wiki_hop fetches outgoing main-namespace (plnamespace=0) links from a Wikipedia article matching one of the reader's top weighted topics (filtering out List of ... and Index of ... meta-pages), while _cross_pollinate queries pairwise intersections of distinct interest clusters, subject to per-host diversity caps (max_per_host = 2) and uniform deterministic interleaving at slot indices round((i + 1) * n_disc / limit).