# The Bursting AI Bubble
> As the AI bubble might be bursting, there are still many things to look forward to
By Avid Fayaz | 25 August 2024 | https://www.avidfayaz.com/writings/PopCulture/pop-culture-ai-hype-and-what-follows
In 2001, tech stocks of internet companies that promised to revolutionize the digital future soared to unprecedented heights. Investors were convinced that the internet would usher in an era of unlimited growth and prosperity. However, reality turned out quite differently. The bubble burst when many of the companies that were expected to lead this new digital age posted disappointing results, revealing that despite years of multi-million-dollar investments, they had failed to generate any substantial profits.
What followed can be best compared to the story of the Emperor’s new clothes, after the little girl points out that the emperor is naked. Although it had been years without significant measurable growth among digital and technology companies, dot-com companies' valuations continued to soar—until a small burst that, once seen, could no longer be ignored. The bubble burst, and within a few weeks, a profusion of companies with no solid financial or technological grounding were wiped out. Yet, as we can see today, many of the internet-based companies founded during and after the dot-com rush are still standing, eventually delivering on many of the promises made during that era.
In many ways, the AI hype can be compared to the dot-com bubble. With the promise of a new golden age of productivity and innovation, creating growth that could eventually lead to a cycle of self-sustaining growth and productivity. However, the AI hype is more complex, as the promises and issues arise within the confusing nature of how Artificial Intelligence is defined.
Since the dawn of the computer age, the idea of creating hardware and software that could replicate human intelligence has fascinated those in computer science and science fiction. Reality, however, has been different with many AI winters and springs ensuing. Yet, due to the nature of recent AI breakthroughs, there have been many arguments—both inside and outside the field—that an Artificial Super-Intelligence might soon be within reach.
### A Question of Scale
A closer look at these models' inner workings reveals their strengths and weaknesses. At its core, a transformer model, the force behind large language models (LLMs), is trained on large corpora of text, which it tokenizes into smaller parts (mostly single words). The model then learns to identify relationships between these tokens. By doing this on large enough corpora, these models develop an impressive “understanding” of language. As these corpora expanded to include vast data from the internet, the models began demonstrating capabilities that many regard as reasoning.
However, the generative capabilities of these models are still grounded in and bound by the old architectural limitations of transformers. In essence, they generate sentences based on the statistical likelihood of a token being the best fit in a given context.
### To Err is Human
Partly the misassessment of LLMs’ capabilities is perhaps due to the all-too-human error of anthropomorphization—the tendency to assign human traits to inanimate objects, animals, or in this case, software. This was evident in people’s reactions to ELIZA (developed by Joseph Weizenbaum in the 1960s), the very first chatbot that made many who interacted with it believe they were interacting with a sentient computer program.
Some have argued that if the current generative models were looped to continuously repeat and improve themselves, they might accidentally accumulate some form of human-like intelligence. While these arguments cannot be entirely dismissed —since evolution is riddled with accidents— the belief that models trained to produce the next token could somehow overcome their inherent structural limitations is easily dismissible.
For instance, asking ChatGPT or one of its rivals whether 3 is a prime number will yield a confident 'yes,' as it has processed texts affirming this. However, with larger, less familiar prime numbers, the model may start providing incorrect answers due to a lack of specific training data. This occurs because it makes a statistical assessment that a large number is less likely to be prime. Once the model commits to a specific token —whether "yes" or "no"— it will continue its reasoning from that point, presenting an elegant yet entirely incorrect explanation.
This problem is one of the first exercises first-year computer science students learn to solve using the Sieve of Eratosthenes, an ancient algorithm from the 3rd century BCE. While ChatGPT could provide this algorithm in any programming language, as it is readily available on many public repositories included in its training data, it does not apply this logic to ensure its own answers are correct.
Without delving into the philosophical aspects behind the definition of mathematics, insofar as defining logic as the underlying structure for mathematical reasoning, it can be argued that logical reasoning is essential for our subjective attempts to describe the formula for truths and to establish consistent relationships with the laws of thought. A model that is incapable of reasoning to perform the simplest arithmetic and division, despite all the computational resources behind it cannot be regarded as a step closer to an artificial general intelligence that could evolve into a superintelligence.
So while many, if not most, agree that an Artificial General Intelligence would indeed pose a threat to humanity, as it challenges our status as the planet's superior intelligence, models without inherent logical and reasoning capabilities that can’t consistently answer how many "r"s are in "strawberry" are unlikely to present that challenge.
### The Discovery of Fire
The overestimation of LLMs' capabilities and the belief that further scaling would lead to significantly greater advancements have been major drivers of the AI hype. To the extent that at its very peaks the CEO of one of the major AI developers compared it to the discovery of fire. However, setting aside the dreams of AGI and analyzing LLMs solely as a technological leap in language capabilities, we confront the reality that not only has the continuous exponential growth many anticipated not materialized, but many of the major companies developing LLMs are also struggling to monetize them efficiently.
Since the initial release of ChatGPT in November of 2022, the valuation of many companies involved in developing both the hardware and software behind LLMs has skyrocketed, with the valuation of Nvidia, the developer of the GPUs that are used to train the frontier AI language models, going from 400 billion dollars to over 3 trillion dollars within less than 18 months and at its peak supplanting both Apple and Microsoft as the most valuable company in the world.
Nvidia's sudden and meteoric rise can be justified in many ways, as it was one of the first companies in the chip industry to explore the uses of GPUs beyond graphics and gaming. GPUs are uniquely suited for performing massive amounts of parallel processing, a core requirement for training and maintaining LLMs. Moreover, Nvidia’s proprietary CUDA platform has become the de facto industry standard for software designed to run on its GPUs, further entrenching its position over its rivals in the AI chips race.
Despite Nvidia’s near-monopolistic hold on the hardware behind the most advanced LLM models, it is unlikely that this dominance can be maintained over the long term. Rivals like AMD are adopting similar modular architectures in their chip designs, and efforts are underway to port software written in CUDA to other platforms like AMD’s ROCm. Additionally, a significant number of startups with strong financial resources, have started designing chips specifically tailored for LLM training and processing, with the goal of leapfrogging the industry giants.
All of this would be of little concern if the demand for more powerful AI chips were expected to continue growing as it has over the past year. Many companies, from tech giants like Google, Amazon, and Microsoft to state-backed firms in Europe and Asia, have pledged to invest billions of dollars in GPU farms and data centers to develop stronger and more capable models. However, several factors could potentially halt the current surge in GPU demand.
The most critical factor is scaling. As mentioned earlier, scaling has been the key driver behind the advancements in LLM capabilities. The issue, however, is that the most advanced models currently available have already consumed all the accessible data for training. While developers like OpenAI and Google are striking deals with media companies to access data behind paywalls, it's becoming apparent that scaling might be reaching its limits. For example, the release of LLaMA 3.1, Meta’s open-source model, demonstrated that the leap from 70 to 405 billion parameters did not produce gains as significant as the jump from 8 to 70 billion parameters on most major benchmarks.
The second factor is the surfeit of AI companies developing and training LLMs. Currently, there are at least 10 companies working on developing frontier large language models, most of which rely on cloud providers for their training compute, with Meta and X AI being notable exceptions. Cloud providers like Microsoft, Google, and Amazon have committed substantial resources to expanding their GPU clusters. However, with advancements in AI capabilities no longer as significant as they were in the past two years, the market is likely to become oversaturated with too many LLM developers offering similar services. And that is without factoring Meta’s open-source strategy further undercutting the market by making high-quality models widely accessible.
Lastly, the disruptions anticipated in the stock market from the AI revolution have yet to translate into significant profits. The rate of monetization and adoption in major industries has fallen short of initial expectations, partially due to concerns over accuracy and security. But also becasue many companies struggling to find or implement constructive use cases of language models in their operations.
As a result, some of the largest AI developers are facing significant financial losses. All these factors lead to the conclusion that it is highly unlikely that demand for GPUs will continue to be in such high volumes, meaning that the massive jumps in stock prices of AI-related companies based on AI-related growth are overhyped and in a bubble.
### Strawberry fields forever 🍓
Despite this bubble, it is important not to lose sight of how profound the advancements in LLMs are. For over half a century, computer scientists tried to create machines that could understand and generate human language with fluency and context awareness. The progress made in the past decade, particularly with the advent of LLMs, represents a significant leap forward in this endeavor. These models have transcended earlier limitations, enabling a wide range of applications that were once thought by many in the industry to be in the realm of science fiction.
Already their use cases in entertainment, education, drug discovery, and crystallography have become apparent. In entertainment, LLMs are transforming content creation, generating scripts, stories, and even music. In education, they serve as personalized tutors, capable of adapting to the unique learning styles and needs of students. In drug discovery and crystallography, LLMs are accelerating research by predicting molecular structures and interactions, leading to faster and more efficient development of new therapies. So, as the industry matures, many more use cases will emerge, potentially delivering on the promised AI revolution and transforming numerous sectors in ways that are yet to be conceived.
And still more advanced models with greater multimodal capabilities and improved reasoning abilities are likely in the pipeline, which could push the boundaries of what LLMs can achieve even further. These models could integrate text, images, audio, and video, developing a more holistic representation of the world. So as we continue to refine and develop these technologies, their impact on society will be profound, extending far beyond what can currently be speculated.
It is therefore, not unjustified to compare the AI hype to the dotcom bubble, as both are characterized by significant overestimation of short-term gains while underestimating the long-term potential. Once again, markets are in a state of overexcitement about a technology that will undoubtedly play a significant role in our lives, yet they are miscalculating how quickly these gains will become evident in people's daily lives.
So, when at some point this bubble bursts, it is important to remember that the concepts and ideas upon which it was created are on a solid grounding and, with time, will eventually materialize.
### Related Articles
- [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
- [AI models collapse when trained on recursively generated data](https://www.nature.com/articles/s41586-024-07566-y)
- [A Sentence is Worth a Thousand Pictures](https://arxiv.org/pdf/2308.00109)
- [FT: Elliott says Nvidia is in a 'bubble' and AI is 'overhyped'](https://www.ft.com/content/24a12be1-a973-4efe-ab4f-b981aee0cd0b)
- [The Economist: What happened to the artificial-intelligence revolution?](https://www.economist.com/finance-and-economics/2024/07/02/what-happened-to-the-artificial-intelligence-revolution)
- [Scaling deep learning for materials discovery](https://www.nature.com/articles/s41586-023-06735-9)
- [AlphaFold predictions are valuable hypotheses and accelerate but do not replace experimental structure determination](https://www.nature.com/articles/s41592-023-02087-4)
- [The future of learning: How AI is revolutionizing education 4.0](https://www.weforum.org/agenda/2024/04/future-learning-ai-revolutionizing-education-4-0/)