DeepMind: Why is AI so good at language? It’s something in language itself

How is it that a program such as OpenAI’s GPT-3 neural network can answer multiple choice questions, or write a poem in a particular style, despite never being programmed for those specific tasks?

It may be because the human language has statistical properties that lead a neural network to expect the unexpected, according to new research by DeepMind, the AI unit of Google.   

Natural language, when viewed from the point of view of statistics, has qualities that are “non-uniform,” such as words that can stand for multiple things, known as “polysemy,” like the word “bank,” meaning a place where you put money or a rising mound of earth. And words that sound the same can stand for different things, known as homonyms, like “here” and “hear.” 

Those qualities of language are the focus of a paper posted on arXiv this month, “Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers,” by DeepMind scientists Stephanie C.Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill.

The authors started by asking how programs such as GPT-3 can solve tasks where they are presented with kinds of queries for which they have not been explicitly trained, what is known as “few-shot learning.” 

For example, GPT-3 can answer multiple choice questions without ever having been explicitly programmed to answer such a form of a question, simply by being prompted by a human user typing an example of a multiple choice question and answer pair. 

“Large transformer-based language models are able to perform few-shot learning (also known as in-context learning), without having been explicitly trained for it,” they write, referring to the wildly popular Transformer neural net from Google that is the basis of GPT-3 and Google’s BERT language program. 

As they explain, “We hypothesized that specific distributional properties of natural language might drive this emergent phenomenon.”

Read more: ZD Net

Leave a Reply

Your email address will not be published.

15 − 14 =