The age of large language models: A shift to a much smaller scale

• Author:
• Author name

  Quantumrun Foresight
• July 27, 2023

Insight highlights

Large Language Models (LLMs), such as ChatGPT, employ deep learning and large data sets to create and comprehend text, promising a future with superpowered artificial intelligence (AI) assistants. Their growing influence holds disruptive economic potential and significant societal implications, including increased productivity, the potential for job displacement, and the reshaping of social norms. However, due to diminishing returns from scaling up models and rising costs, OpenAI's CEO, Sam Altman, suggests a shift towards smaller, business-specific models.

The age of large language models context

An LLM is a type of AI that uses deep learning and massive data sets to comprehend and generate text-based content, also classified as generative AI. Drawing from the human development of spoken languages for communication, AI employs language models to create and convey new concepts. These models originated with early AI innovations like the ELIZA language model in 1966. Language models are trained on a data set, then use various techniques to infer relationships and generate fresh content, often utilized in natural language processing (NLP) applications. LLMs represent an advanced stage in this concept, training on larger data sets with a billion or more parameters, significantly enhancing the AI's capabilities.

The LLM powering ChatGPT and its competitors is evolving quickly. While they are far from what some people call artificial general intelligence (AGI), they give the public an early glimpse of what life might look like with access to a truly superpowered personal assistant. For example, Auto-GPT was released in March 2023, an application that mimics some of an AGI's theoretical functionality, such as the ability to understand the world like a human and an equivalent capacity for learning and performing a wider range of tasks. Auto-GPT is an experimental, open-source Python program that uses GPT-4 for autonomous operations, which means that Auto-GPT can self-prompt to execute assigned tasks with minimal human involvement.

Disruptive impact

Although still experimental, these LLM agents may boost productivity gains aided by declining AI hardware and software costs. ARK's research shows that AI software might make as much as USD $14 trillion in revenue and contribute to a USD $90 trillion enterprise value by 2030. As GPT-4 and similar large models improve, many businesses are making their own smaller, business-specific models with lower costs. Even OpenAI's CEO, Sam Altman, thinks that the future of LLMs will be much smaller.

Altman suggests GPT-4 might be the final significant progress resulting from their current approach of enlarging the models and increasing data input. While the specifics of potential future research strategies or techniques weren't outlined, Altman pointed out that their research on GPT-4 indicated diminishing returns on scaling up a model size. He also noted practical constraints, such as the number of data centers OpenAI can construct and the rate at which they can be built.

Training these models is also becoming more expensive than ever. A 530-billion parameter model on enormous text datasets was accomplished by Microsoft and NVIDIA through the deployment of hundreds of DGX A100 multi-GPU servers, each costing USD $199,000. When you consider the costs associated with networking equipment, hosting, and other related expenses, replicating such an experiment would likely approach a staggering USD $100 million.

Implications of the age of large language models

Wider implications of the age of large language models may include: 

• Increased dependence on AI for communication, problem-solving, and decision-making. The influence of AI in shaping social norms and opinions could raise concerns about individual agency and the potential for manipulation by AI-driven narratives.
• Significant improvements in productivity and cost savings for businesses. Automation can make many industries more efficient, but it could also worsen income inequality, as highly skilled workers and AI owners may reap more benefits.
• LLMs being used to analyze political sentiment, develop policy recommendations, or even draft legislation. However, there is a risk of AI-generated content being used for misinformation campaigns or to manipulate public opinion.
• Older people potentially enjoying AI-powered healthcare and companionship services, while younger workers could face new job challenges because of widespread automation.
• Further advances in AI research and applications, leading to the creation of new technologies and industries. However, AI technologies also raise concerns about potential misuse, cybersecurity threats, and the need for adequate regulation and oversight.
• Significant job displacement, particularly in industries that involve repetitive tasks or data analysis. While new job opportunities may become available, there may also be a need for extensive retraining and support for displaced workers to transition into new roles. Society will need to adapt to these changes, potentially rethinking education, social safety nets, and labor policies.

Questions to consider

• How else do you think LLMs are going to evolve?
• If you work in the AI industry, what are some developments in LLM use cases?