Generative Pre-Trained Transformer 3 (GPT-3) is the third in a series of AI language models released by the OpenAI lab in San Fransisco. GPT-3 was made available earlier this year and represents a big step forward in AI natural language processing. Compared to its predecessor, GPT-2, GPT-3 is a much more sophisticated and convincing imitation of not only human language, but other human skills as well.
The main thing that sets GPT-3 apart from its previous generation is the sheer amount of computing power available to it. The full version of GPT-3 has 125 billion machine learning parameters, compared to the only 1.5 billion in GPT-2. Before the advent of GPT-3, some theorized that merely scaling up a language model in this way would have diminishing returns in terms of performance. However, GPT-3 has shown that, at least for now, this isn’t the case. It begs the question, how much bigger can we go before the impact starts to be reduced?
GPT-3 can do a number of things better than its earlier counterparts. For one thing, humans can only accurately identify short articles written by GPT-3 about 52% of the time, when comparing them with human-written articles. Additionally, GPT-3 is much better at simple arithmetic than other language models, regularly able to answer simple sums and differences. This is important because it implies a degree of general knowledge, i.e. the AI performing broad tasks beyond the language processing it was trained for. It also has done things like making jokes, providing translations, generating plots, and writing its own code.
Here are a few interesting things people have done with the GPT-3 API:
- gwern.net has a collection of creative fiction in many styles, all generated by GPT-3
- AI Dungeon uses GPT-3 to create interactive RPG-style stories along with the user
- GiftGenius is a gift recommendation engine powered by GPT-3
GPT-3 is very powerful, but don’t worry, we haven’t reached the singularity just yet. It still struggles with certain tasks, like maintaining a narrative thread over a longer work or identifying and appropriately answering nonsensical questions. Regardless, GPT-3 gives us a glimpse at what the future of AI might be and how we could use it to reach goals that seemed impossible before.
