Research & More

We enrich the self-supervision strategy employed in BERT by applying self-supervision on the Word-Sense level. Our model, SenseBERT, achieves significantly improved lexical disambiguation abilities, setting state-of-the-art results on the Word in Context (WiC) task.

We present a masking strategy that makes bidirectional masked-LM training more educating.

Exemplar Guided Active Learning (EGAL) is a method we developed to economically annotate training data for tasks with extremely skewed label distributions, such as disambiguating rare word senses.

We designed a simple and efficient method, called Auxiliary Tuning, for adapting a pre-trained Language Model (LM) to a novel task, and demonstrated the approach on the task of conditional text generation.