Most local-LLM benchmarks begin with a number. Cloudlight began with a deployment question: can this browser on this machine run a local language model well enough to support a web application?
That question ruled out the usual benchmark shape. A leaderboard can afford to be broad. A deployment tool has to be specific about the workload, runtime, and failure conditions behind its result.
Close the experiment before measuring it
An arbitrary-model launcher is attractive until somebody tries to compare the results. Similar parameter counts conceal different quantization, context limits, artifact layouts, memory requirements, and runtime behavior. The result is a collection of anecdotes with a table around it.
Cloudlight uses a small, versioned ladder instead: WebLLM workloads from a 0.5B Qwen through a 3B Qwen, plus TinyLlama as a 1.1B compatibility baseline. It also has a pinned 0.5B Qwen workload through Transformers.js. The runtime is in the workload identity because it changes the experiment. A WebLLM result and a Transformers.js result describe different executable systems, even when the model names are close.
This makes a smaller product. It also makes a result that can be rerun on a second machine without guessing what changed. The same logic keeps an 8B or 9B tier out for now. It needs an explicit high-memory failure story before it belongs beside workloads intended to be routinely runnable.
A capability check cannot promise capacity
WebGPU preflight is necessary, but it is a poor compatibility score. A browser can expose navigator.gpu and still fail to acquire an adapter. A working adapter may still reject a larger model. Processor counts and device-memory hints are useful context, not a description of allocatable GPU memory.
Cloudlight uses the preflight to avoid a clearly unsupported path, then treats a successful model load as the admission test for that workload. This is the important product boundary. The page does not pronounce a laptop ready for local AI. It records whether the selected runtime loaded the selected model in the environment that actually exists.
The same rule changed how loading is reported. WebLLM offers progress but no portable byte-level cache signal, so Cloudlight reports first load in this session or loaded earlier in this session. The displayed download sizes are estimates. Calling either one a cache hit or a memory guarantee would make the UI feel more precise while making the report worse.
Reliability had to be runtime-aware
Speed is only part of a browser integration. Cloudlight runs bounded probes for cold and warm time to first token, prompt processing, decode speed, duration, and repeated runs. It also checks structured JSON and cancellation when the runtime exposes those operations.
WebLLM supports the full probe suite. The current Transformers.js path does not yet expose equivalent JSON and cancellation controls, so its report says Not supported. That's more useful than an invented pass or failure because it preserves the reason the comparison is incomplete.
One bug made the distinction concrete. The initial Transformers.js benchmark inherited the playground's 2,048-token generation limit. Small benchmark probes looked stalled because an interactive default had leaked into a controlled measurement. The fix was to bound each probe independently. The playground can remain generous; the benchmark needs to finish predictably.
The architecture followed the privacy claim
Cloudlight has no benchmark backend, account, API key, or application telemetry. The chosen runtime fetches its model assets into the browser and a dedicated worker runs the load and inference work away from the UI thread. That is both a privacy boundary and a usability decision: an interface that freezes while a model initializes has already failed the deployment test.
Sharing follows the same shape. A report can live in the URL fragment and be rendered without sending it to Cloudlight. Optional published links use the user's NIP-07 signing extension and the URLattice publishing contract. Session comparison still disappears on refresh, and there is no global leaderboard. Persistent history and cross-device aggregation need their own storage, authority, and privacy decisions.
What the result is for
Cloudlight cannot predict every application's behavior. Real products bring their own prompts, context windows, interaction patterns, and failure budget. It can answer the next useful question: did this known browser-local workload load, run, and produce a measured profile on this machine?
That is sufficient to decide whether to try a browser-native local model, keep the workload small, investigate a different runtime, or stop before building around a capability the device does not have.
The one-line version
A benchmark earns trust when its result is no broader than the experiment that produced it.