Retiring a claim we couldn’t measure

Until this week, 4DA’s one-sentence description ended with a promise: it “gets sharper every day.” The follow-up line explained how: “it learns from how you engage with what it shows you.” That promise is now retired. Not softened — removed, from the app, the installer metadata, the site’s structured data, and a published npm README, with a build check that fails if the phrase comes back.

This is the engineering record of why. The short version: an audit of the learning loop behind the promise found four incidents in two months, and a data-starvation problem that meant the loop’s benefit was never measurable in the first place. A mechanism whose costs we could measure and whose benefit we couldn’t doesn’t keep authority over scoring. And a claim we can’t measure doesn’t stay on the homepage.

What the promise was, mechanically

The engine watched how you engaged with what it surfaced — what you read, dismissed, scrolled past — and fed that back into relevance scoring: a per-topic affinity multiplier with ×0.3–1.7 authority over an item’s score, a learned axis in the confirmation gate, learned source-quality weights, auto-tuned thresholds, and a calibration curve that reshaped the LLM judge’s output. Yesterday’s noise becomes tomorrow’s signal — via your behavior.

What two months of it actually did

Four incidents, in the order we found them:

1. It learned to suppress the user’s own stack. Passive scrolling was recorded as negative engagement. On a Rust + Tauri + TypeScript codebase, the loop drove the topics rust, tauri, and typescript to −1.0 affinity — the floor — and with ×0.3–1.7 authority over scores, the engine started suppressing exactly the content the product exists to surface. The promise, running in reverse.
2. A boost bypassed a cap. The pipeline puts a categorical ceiling on look-alike package releases — name-similar crates that resemble your dependencies but aren’t them. An attention-gap boost (“you haven’t seen much of this topic lately”) applied after the cap and pushed 145 of them back over the relevance threshold and into the feed.
3. The capture wiring was dead — and when it wasn’t, it disagreed with itself. The feedback-capture path was silently broken for weeks before anything noticed, which is its own indictment: a learning loop whose input can flatline without a symptom isn’t load-bearing. When it was alive, three code paths recorded feedback on three incompatible strength scales. One of them recorded dismissals as +0.3 — a positive.
4. A calibration curve outlived its own training data. The most instructive one. A curve fit in June from 50 mislabeled samples was degenerate — every input bucket mapped to certainty. A later database reset wiped the training tables, but the curve survived, because it lived as a file on disk, outside the database: the reset destroyed the evidence and kept the conclusion. Every cycle after that, the LLM judge’s honest 1-out-of-5 relevance scores were remapped to 5-out-of-5, a reconciler added +0.15 to 48 items per cycle, and each remapped score was re-persisted as a “raw” training sample for the next fit. A fitted artifact certifying its own output as ground truth.

The number that decided it

Explicit feedback signals in the entire instance: 7. Corpus scored against: roughly 200,000 items. You cannot fit a per-user model on seven labels — and you cannot measure whether it helped, either. The loop wasn’t underperforming; it was unmeasurable. Every cost above is a number in an incident record. The benefit never produced a number at all.

That starvation isn’t an implementation bug. A local-first, zero-telemetry app has exactly one user’s signals to learn from, and one developer produces single-digit explicit labels while the engine scores hundreds of thousands of items. The standard fix — pool engagement data across users — is the thing we refuse to build. So the honest conclusion is that this mechanism, in this product, cannot currently earn its authority.

One mitigating fact, stated plainly: 4DA is pre-launch, and the instance in every incident above is our own. The only feed this corrupted was ours. That made now the cheap moment to fix it — after launch, this post would be a retraction.

What we changed

Two decisions, in order.

First, the mechanism lost its authority (pipeline v19). Engagement-derived signals no longer move scores or verdicts — the learned-behavior weight is pinned at 0.0 by a written invariant. The attention-gap boost is deleted. The threshold auto-tuners are frozen at the neutral default. Calibration curves are refused if degenerate, at save and at load, and samples now persist the raw pre-transform score so a bad curve can’t launder its output into future training data; the 3,028 poisoned samples were purged. Score ceilings are re-asserted in the one pass that runs after every writer, so nothing downstream can un-cap an item again. What stayed: everything user-controlled and visible — the preferences panel (pin, forget, reset), the engagement dashboards, and all of the codebase-context learning, which was never the problem.

Then the promise itself was retired. Around 30 in-app strings, the installer metadata, the homepage’s JSON-LD, and an npm README still asserted the removed mechanism — several inside the score-explainability surface, where describing a scoring input that no longer exists isn’t marketing taste, it’s a false explanation. All of it is replaced with what is permanently true and verifiable: relevance is derived from your manifests, dependencies, and commits; when the engine improves, it re-judges the whole corpus. “Yesterday’s noise becomes tomorrow’s signal” survives, because that re-judging mechanism actually exists — it’s just no longer attributed to your engagement. And a check in the build now fails on the retired phrases anywhere outside the historical record.

What did not change

• Local-first. Everything runs on your machine. Raw data never leaves it.
• BYOK. Your API keys, your models, never stored remotely.
• Zero telemetry. We don’t collect engagement data across users — which, as covered above, is precisely why the learning loop was starved. We’re keeping the constraint and dropping the claim, not the reverse.
• Dependency grounding. Relevance was always primarily driven by your actual manifests, lockfiles, and commit history. That part was never the learned behavior, and it’s what carries the product.
• The benchmark. The scoring pipeline’s published result — 93% of content filtered as noise, 98.9% of actual noise correctly rejected, measured across 9 simulated developer personas on a 245-item labeled corpus — stands, with its methodology attached, in body copy where a number can carry its context. Never as a headline.

The bar for ever saying it again

“Improves with use” is a measurable claim, so the bar for making it is a measurement. The decision record spells it out: one unified feedback scale with no positive-valued negative gestures; degeneracy guards on every fitted artifact; fitted state bound to the corpus it was fit on, so no artifact outlives its data again; a user-visible off switch; and — the actual gate — a measured lift over the neutral baseline on labeled data, published. If that lift ever exists, the claim returns for the first time with evidence behind it.

We’ll make the claim when we can prove it.