HalleluBERT – a large Hebrew RoBERTa model achieving state-of-the-art performance

HalleluBERT

HalleluBERT – A Compact Contribution with a State-of-the-Art Impact

HalleluBERT was designed as a focused, compact research contribution, a small paper with a clear goal: to build a strong Hebrew RoBERTa-style model and evaluate how far pre-training and scaling would push performance in a low-resource setting.

Although modest in scope, the project delivered something remarkable: the first large Hebrew RoBERTa model trained with a modern pre-training setup, achieving state-of-the-art performance across the evaluation benchmarks we designed.

The workflow followed the pattern established in earlier projects.
The pre-training was executed on a TPUv4-128 pod, while the entire downstream evaluation was performed locally on private basement hardware, using the same workstation that powered GeistBERT and parts of ChristBERT. In that sense, HalleluBERT represents the final chapter of a long hardware-driven research arc, the last model trained with this infrastructure setup.

From a historical perspective, HalleluBERT is not the most complex or emotionally loaded project in the family. But scientifically, it closes a loop: it rounds off the sequence GottBERT → GeistBERT → ChristBERT with a clean, technically sharp contribution that stands on its own.

The work is publicly available as an arXiv preprint and currently under review for LREC-COLING 2026.

Links & Resources

NLP Language Models Hebrew Deep Learning Model Pretraining Low-Resource NLP Open Source