Understanding Multilingual Language Models: Training, Representation and Architecture

Abstract

The field of natural language processing has seen numerous advancements since its inception during the Cold War, when attempts were made to automate translating Russian text to English. The 2010s have been a particularly fertile decade, going by sheer research volume – the use of a class of machine learning models known as neural networks has led to substantial improvements in performance on a broad range of language-related tasks, ranging from classification, to summarisation, to machine translation. In addition, some of these models have demonstrated an excellent multilingual capacity, where they can be trained to process text in English, and with only a bit of extra data, show solid results across languages. Yet these improvements, rapid as they have been, are not without their drawbacks. One such drawback is that we lack the tools to rationalise how and why these models function, particularly given the many (often arbitrary) decisions that went into their design. This dissertation adopts multiple analytical frameworks in an effort to answer questions surrounding the behaviour of multilingual neural models. How does the data they were trained on affect their performance? What linguistic capabilities can they exhibit? And finally, how do different model components help enable multilinguality?