OncoNoteBERT Speaks Oncology. That Does Not Yet Make It Clinically Proven

What happened

A team led by Wuraola Oyewusi has developed two language models for a specialised kind of clinical text: real-world outpatient oncology notes. The work is an arXiv preprint accepted at the AI at Scale for Clinical Impact: Cancer Pathology Foundation Models Workshop at NeurIPS 2026.

The OncoNoteBERT preprint, written with Eliana Vasquez Osorio, Goran Nenadic and Gareth Price, uses 290,026 notes from 21,564 patients treated for lung and head-and-neck cancers at a specialist oncology centre in the United Kingdom. The notes span diagnosis, staging, imaging, treatment, toxicity, supportive care and the compressed language clinicians use in everyday records.

Oyewusi shared the preprint publicly on X, where it became a research lead for Haivian Discovery News. The social post identifies the story; the preprint carries the evidential weight.

The results below are reported by the authors. Haivian has not audited the clinical data or models and has not independently reproduced the experiments.

The researchers compared two established models from neighbouring clinical domains—RadBERT and PathologyBERT—with two models adapted to the local oncology corpus. OncoNote-RadBERT continued training an existing radiology model on the oncology notes. OncoNoteBERT began with randomly initialised BERT-style parameters and a new vocabulary built for the oncology text itself.

Both local approaches fit the held-out oncology material much better than the unadapted comparators. The lowest reported perplexity was 2.10 for OncoNote-RadBERT, followed by 2.83 for OncoNoteBERT. RadBERT scored 113.04 and PathologyBERT 2,035.03 in the paper's zero-shot evaluation.

Those are dramatic differences. They are also easy to overread. Perplexity answers a narrow question about how well a language model predicts text drawn from a particular distribution. It does not show that a model can safely inform cancer treatment, extract clinical facts accurately at another hospital, or improve patient outcomes.

What the evidence shows

In masked-language modelling, parts of a sentence are hidden and the model learns to predict them. Perplexity is derived from the probability the model assigns to the held-out text. Lower is better within a properly controlled comparison: it means the text was less surprising to the model.

For OncoNote-RadBERT, the comparison with RadBERT is especially informative because the models share a tokenizer lineage. Continued training on outpatient oncology notes substantially improved the model's fit to that local language.

OncoNoteBERT demonstrated a different benefit. Its oncology-specific tokenizer represented terms such as “ECOG”, “T3N2M0”, “chemoradiotherapy”, “cisplatin” and “pembrolizumab” more compactly. On a shared document sample, it required about 23.6% fewer tokens than RadBERT. That can preserve more useful text within a fixed context window and reduce needless fragmentation of specialist terms.

It also returned a prespecified acceptable answer among its top five predictions for 12 of 13 masked-token probes. OncoNote-RadBERT did so for seven. The result suggests that vocabulary design and overall corpus fit are measuring different properties.

The authors are notably careful about this distinction. A one-token answer is easier for a model whose vocabulary already contains the whole clinical expression. The 13 prompts were therefore a qualitative audit, not an independent measurement of medical knowledge.

What remains uncertain

The largest uncertainty is external validity. All notes came from one specialist UK centre and covered two cancer groups. Documentation conventions, abbreviations, patient populations and de-identification systems differ across hospitals and countries.

The models were each trained once, so the reported results are point estimates rather than averages across repeated runs or alternative settings. Some perplexity comparisons also combine two things at once: differences in model parameters and differences in tokenization. Perplexity values produced with different tokenizers should not be treated as a clean league table of clinical capability.

Most importantly, the study deliberately evaluates intrinsic representation, not a clinical task. It does not establish performance on extracting tumour stage, classifying treatment response, identifying toxicity, predicting survival or supporting treatment decisions. It does not measure errors across demographic groups or test whether clinicians using the models make better decisions.

The paper says this plainly: more compact representation does not guarantee better downstream performance, and the models' clinical value remains to be established.

Why independent scrutiny matters

OncoNoteBERT is valuable precisely because it separates an important engineering achievement from claims that have not yet been tested. The new models appear better adapted to one governed oncology corpus. That is a foundation for further evaluation, not a certificate of clinical readiness.

A rigorous next stage would define a specific task, freeze the evaluation protocol, test on untouched data from other institutions, report confidence intervals and clinically meaningful error categories, and compare performance with existing systems and qualified human workflows. Safety analysis would need to examine not only average accuracy but the consequences of omissions, false extraction and subgroup failures.

This distinction matters far beyond oncology. A model can reproduce the statistical texture of specialist language while still misunderstanding the facts needed for a real decision. Low perplexity means the text looks familiar. External clinical validation asks whether the system works, for the intended purpose, on patients and settings it did not learn from.

Haivian provides bounded independent research audits for teams that need claims, methods and evidence examined before publication or deployment. An audit does not replace clinical validation, but it can identify where a result is strong, where its boundary lies, and what evidence is still missing.