Why domain-specific LLM evaluation cannot stop at loss

Loss is not a task

When teams adapt language models to specialised domains, training loss is often treated as a proxy for progress. It is a useful optimisation signal. It is a weak proxy for whether the system can complete real work correctly.

Domain correctness is local. A semiconductor, legal, clinical or industrial workflow has its own definitions of a valid answer, a dangerous answer and a merely incomplete answer. Those definitions rarely appear cleanly in a generic leaderboard.

What evaluation should cover

A domain evaluation suite should, at minimum, address:

1. Task performance — Can the model complete the jobs that matter?
2. Failure modes — How does it fail when it is wrong?
3. Format and structure — Does output respect required schemas, units or conventions?
4. Regression risk — Did a new checkpoint break something previously stable?
5. Operational behaviour — Latency, cost and stability under realistic load.

Without these, a low loss curve can coexist with a system that is not safe to deploy.

Practical design notes

Start with a small but honest golden set. Prefer fewer high-quality cases with clear scoring rules over large noisy collections that no one trusts. Include adversarial and edge cases that operators already know cause trouble. Separate automatic scoring from human review where judgements are subtle.

Build evaluation early enough that it can influence data design and training choices. Waiting until after an expensive continued-pretraining run is a common and expensive mistake.

Evidence before theatre

A domain model is not dependable because it sounds fluent in the domain language. It is dependable when evaluation shows consistent behaviour on the tasks that define success. That is the standard Asthra uses when designing adaptation and reliability work.