Will language models eventually supplant knowledge bases?

Saurabh Verma
3 min readJul 3, 2021

--

With the pace at which language models in NLP are getting improved w.r.t several downstream task people have started to question the use case of well-known knowledge bases. For those whom the above line seemed like a riddle, let me quickly introduce you all to what exactly we mean by language models. A language model is the core component of modern Natural Language Processing (NLP). According to Wikipedia,

A statistical language model is a probability distribution over sequences of words. Given such a sequence, say of length m, it assigns a probability to the whole sequence.

In simpler words, Language Models determine the probability of the next word by analyzing the text in data.

It seems that the language models may be storing relational knowledge present in the training data and may be able to answer queries structured as “fill in the blanks” kind of statements also known as cloze-style questions in the NLP community. So, why are we comparing this with existing traditional knowledge bases? The reason is very strong! We know that it takes a lot of time, effort, and cost to make a rule-based knowledge base. Moreover, not to ignore the fact that the knowledge base is well structured and thus requires schema engineering.

Petroni et al., have shown that language models like BERT (Devlin et al., 2018a) and ELMo(Peters et al., 2018a) do have relational knowledge to some extent needed to answer several factual based questions without applying any fine-tuning. But here are several questions that pop immediately, like, How much knowledge do a particular language model store? Is the amount of knowledge sufficient to answer anything and everything? How does this differ for different types of knowledge such as facts about entities, common sense, and general QA? All these questions are something that is pushing the NLP community to come up with an explanation.

Let’s consider a scenario, Suppose a model is trained on millions of articles related to several banks (a bank is the land alongside a body of water) as well as numerous banks (A bank is a financial institution licensed to receive deposits and make loans). Confused? Now, if someone asks a question related to the bank, how difficult it would be for the model to come up with the desired answer as different article has different information related to Bank and which one is desired? Forget about the model, how difficult it would be for a normal human to answer the question? You are right, we humans also need some kind of context to frame our answer, and Kelvin Guu et al., have shown that even if we consider a model trained on millions of documents and facts in Wikipedia and other textual resources, it is unlikely that a language model with a fixed number of parameters can reliably store and retrieve factual knowledge with sufficient precision.

Later, Petroni et al., acknowledge the fact that it is very difficult for a language model trained on such a big corpus of text to come up with answers with high precision and came up with the idea of providing context along with the cloze-style questions to the model. According to their experiments, they concluded that enriching cloze-style questions with relevant context dramatically improves the performance of language models like BERT with respect to using context-free questions. Also, It is important to note that the pre-trained language model was not fined-tuned in any manner and performed comparatively well just with the presence of relevant context.

So, what do you think about whether language models have become superior to Knowledge bases or it is still too early to comment?

Thanks for reading the article. I hope you liked it.

References:

  1. https://openreview.net/pdf?id=025X0zPfn
  2. https://aclanthology.org/D19-1250.pdf

--

--

Saurabh Verma
Saurabh Verma

Written by Saurabh Verma

NLP Enthusiast currently working towards Explainable NLP

No responses yet