AI’s Compute Scarcity Problem Will Not Be Solved by Data Centers Alone

For the past couple years, companies have been treating the high demand and explosive growth for AI inferencing as a construction project rather than a software engineering problem. Their answer has been to purchase as many GPUs as possible, build more data centers, and acquire as much power as possible. However, construction scales linearly, not exponentially, so without an equal focus on efficient token usage there won’t be sufficient supply no matter how many data centers are created.

The physical limitations of adding more data center compute are already showing up in headlines, such as the report from energy analytics firm Wood Mackenzie that shows construction engineers only added 25 gigawatts of power, less than half the capacity created the quarter before, showing how the market is struggling to keep pace with new commitments. At the same time, communities are also pushing back on new data centers, with lawmakers in more than 10 states proposing temporary bans on data-center construction this year.

This is why the AI energy debate needs to focus on not only increasing capacity, but also architecting for efficient token usage. Hardware, grid investment and new capacity matter, but companies can’t control zoning battles, transmission upgrades or multiyear construction schedules. They can control how efficient their systems are at using AI inferencing by architecting systems that are designed to give accurate, efficient, and explainable results. Building software systems that inject the optimal context into each LLM call reduces the amount of re-prompting, while also optimizing token usage.

Where LLM Models Fall Short

Large language models can summarize, synthesize, translate and generate text and code efficiently and in high volume, opening up new use cases. However, they don’t work equally well for every type of job and are not tailored and trained on domains that require enterprise knowledge. Some enterprise problems call for exact answers, current system state and clear lineage. In those cases, sending every request to a model can add cost, latency and uncertainty.

Research published in Environmental Science and Ecotechnology shows that optimized prompting can reduce LLM inference energy and CO2 emissions by 32–48%. The study found that ambiguous prompts often required follow-up queries which dramatically increased energy and token usage. Instead, the paper recommends using concise prompts with high semantic precision to help reduce the environmental impact of LLM inferences.

One way to help focus prompts on the most crucial information is to utilize a knowledge graph that explicitly tracks the relationship between relevant data. Think of this as a knowledge graph first, model on demand. This moves AI outputs from statistical guesses by an LLM to a more focused answer that’s the result of knowing exactly what data the model should focus on, while looking for ways to find the path to a solution without having to call a model at all.

A knowledge graph can often resolve a question directly because the relationship clearly leads to the answer. A graph can provide the precise context that a model needs, so the model can do high-value reasoning without chewing through piles of irrelevant material. Even a lightweight AI model can translate a user’s question into a graph query, fetch the right information and return a clean answer. That is a much more disciplined way to build AI systems than flooding a model with information and hoping the LLM will make the correct connections when generating results.

We’re seeing this play out in the real world as well. Aakash Gupta found that longer prompts do not automatically lead to better outcomes. In many cases the more irrelevant material you send to the system, the more work it has to do to find what matters. That drives up cost, latency and energy use, and it can still leave you with a weaker answer.

Where Graph Technology Becomes Especially Useful

A traditional relational database system often has to reconstruct relationships over and over through SQL joins and index lookups. A knowledge graph stores those connections explicitly, allowing the system to traverse them directly. That changes the economics of connected data problems. The more a task depends on multi-hop relationships, the more that difference starts to matter.

One example of how a company has been able to reduce its server load comes from Adobe, for its Behance platform for showcasing and discovering creative work, artists and designers. The team moved from 48 nodes of a wide-column NoSQL database to three nodes on a property graph database for its activity feed. Storage dropped from 50TB to 50GB, requiring much less maintenance. Marriott was able to reduce average processing time in its hotel room pricing engine from more than four minutes to 13 seconds, while reducing infrastructure cost by about 50% using graph technology.

These wins show what happens when you stop scaling inefficient patterns and start changing the architecture to find more accurate and efficient answers.

None of this means companies should stop building data centers or stop improving chips. Demand for AI is still growing, and efficiency gains often expand the market rather than shrink it (thanks to Jevons Paradox). Lower costs tend to create new use cases and more demand. But focusing on the right data will help deliver better results regardless of the price of tokens or number of data centers in the world.

Software Architecture Must Play a Part in Near-Term Sustainability Strategy

Focusing AI on the right data to deliver better solutions is the best approach I see in the short-term, until data center construction catches up with demand. To improve things today, companies can look at things like prompt caching so they do not have to recompute the same context again and again. Implement persistent decision traces so systems do not rerun the same reasoning from scratch (with different results).

Companies need to route questions that require exact answers to deterministic systems rather than probabilistic LLMs. They need to use models where they add clear value, and use graphs and other tools where connected data, precision and explainability matter more.

The companies that win the next phase of enterprise AI will not be the ones that send the most tokens through the biggest clusters. They will be the ones that treat architecture as a competitive advantage and make smarter choices about what to compute, when to compute it and which tool should do the work.

Power is getting harder to secure. Community acceptance is getting harder to win. Capital is getting more closely watched. Under those conditions, just using more tokens is not a strategy.

Helping companies succeed with AI while avoiding energy waste can address scarcity problems and lower costs. It will also save companies from trying to use data that will just confuse outputs and deliver less precise results. The objective is to focus AI inputs on the most relevant data to achieve the most accurate and efficient results.