Energy Consumption and AI
How many lightbulbs does it take to power GenAI ?
So, a colleague recently posted the above image of IBM’s Watson roadmap. I think it’s fair to say we have had a great start to our 2024 ambitions, focusing on integrating trust guardrails throughout the AI foundation models’ lifecycle and implementing AI governance at the organizational level. Data representations are being optimized across privacy, fairness, explainability, robustness, etc., with the commercial release of Watson.X Governance back in November 2023. However, it was the 2025 ambition that was gaining much of the focus: ‘In 2025, we will improve the energy and cost efficiency of foundation model training and inference by 5x and bring 200B+ parameter foundation models to enterprises.’ It’s all about making them more powerful, useful, and practical.
This led to several interesting interactions, and I ended up with three questions that I am going to attempt to answer:
- How much energy does AI use anyway?
- Why do I care how much energy my AI is using?
- How do you make AI’s power consumption more efficient?
How much energy does AI use anyway?
So, I started by speaking to a geek friend of mine who works in energy generation. She pointed me to a peer-reviewed analysis published in the journal Joule, which is one of the first to quantify the demand that is quickly materializing. A continuation of the current trends in AI capacity and adoption is set to lead to NVIDIA shipping 1.5 million AI server units per year by 2027. These 1.5 million servers, running at full capacity, would consume at least 85.4 terawatt-hours of electricity annually. The chart shows the estimated energy consumption by these against the energy consumption of countries as per the IEA 2023 Energy Survey.”
So broadly speaking energy is used in two areas of AI the huge compute effort to create the models and ingest the data and then secondly the interactions with the models. While the training process of these LLMs typically receives the brunt of environmental concern — models can consume many terabytes of data and use over 1,000 megawatt-hours of electricity de Vries’s report highlights that in some cases electricity consumed while making inferences may be even higher.
For me really revealing part was just how much energy is used by a single interaction.
“You could say that a single LLM interaction may consume as much power as leaving a low-brightness LED lightbulb on for one hour,” de Vries says.
Something for us all to think about when playing with ChatGPT.
Why do I care how much energy my AI using ?
For a start AI usage is going to add to your organisations carbon footprint something many organisations are pro-activly trying reduce. In the HR Context a IBM survey reported that 71 per cent of employees believe environmentally sustainable companies are more attractive employers.
We also know that the new EU AI Act considers any general purpose AI models that were trained using a total computing power of more than 10²⁵ FLOPs are considered to carry systemic risks, given that models trained with larger compute tend to be more powerful. The AI Office (established within the Commission) may update this threshold in light of technological advances, and may furthermore in specific cases designate other models as such based on further criteria (e.g. number of users, or the degree of autonomy of the model).
Providers of models with systemic risks are therefore mandated to assess and mitigate risks, report serious incidents, conduct state-of-the-art tests and model evaluations, ensure cybersecurity and provide information on the energy consumption of their model.
How do you make AI’s power consumption more efficient?
Roberto Verdecchia is an assistant professor at the University of Florence and the first author of a paper published earlier this year on developing green AI solutions. He says that de Vries’s predictions may even be conservative when it comes to the true cost of AI.
In his paper, published in the journal WIREs Data Mining and Knowledge Discovery, Verdecchia and colleagues highlight several algorithmic approaches that experts are taking instead. These include improving data-collection and processing techniques, choosing more-efficient libraries, and improving the efficiency of training algorithms.
The second approach to energy efficient AI is the delivery of more efficient hardware. IBM’s new chip NorthPole, is more than 20 times as fast as — and roughly 25 times as energy efficient as — any microchip currently on the market when it comes to artificial intelligence tasks. According to a study from IBM, applications for the new silicon chip may include autonomous vehicles and robotics. You can read more about it’s technical details here.
Of course, both of these potentially make AI more attractive, and I guess for me, that shows the third principle has to be human regulation. Just because you can use AI, does it mean you should? This is meant to be an overview of some of the issues but more the start of the conversation.
