In February 2019, OpenAI disclosed a language generation algorithm called GPT-2. It did only one thing: predict the next word given all previous words in the text. And, while not perfect, it does this very well. When prompted with:
In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.
it responds with:
The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.
Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.
(The text continues.)
GPT-2 is a transformer-based neural network with 1.5 billion parameters trained on a dataset of 8 million web pages. Transformer-based networks were introduced by Google researchers in 2017 primarily for the purpose of language translation. They work on language by figuring out how much attention to pay to which words. Some words have more semantic value than others, and transformer-based neural networks can learn how to value different words with large amounts of training data. The biggest benefit of a transformer-based network is that the computation can be easily performed in parallel, in contrast to the more traditional and sequential RNN models used for language translation.
In a controversial move, OpenAI originally declined to make the GPT-2 model available to researchers, citing concerns about it being used to create “deceptive, biased, or abusive language at scale . . . .” Recently, however, they have released a smaller, less capable version of the model, and are considering other ways to share the research with AI partners.
Anyways…. now you can play with the smaller GPT-2 model at TalkToTransformer.com.