Why Amazon and Replit are abandoning AI token usage rankings

Amazon and other technology companies are moving away from internal dashboards that rank employees by their AI token consumption, the practice known as a "token leaderboard."

Speaking at Web ...

"Token leaderboards are very dystopian," the head of AI at Replit said at a tech conference Monday. The executive called the metric a poor measure for evaluating employee performance or productivity.

Companies built the dashboards to track how many AI tokens each employee used, treating volume as a proxy for how quickly the workforce was adopting large language models. Replit's AI chief said that misses the point.

"A single good answer that saves hours of work might consume fewer tokens than dozens of shallow queries," the executive said. The metrics have to reward the outcome, not the process.

Amazon already shifted its approach. The retailer previously tracked token consumption across teams but now measures task completion and user satisfaction instead, according to people familiar with the matter.

The change highlights a central problem in enterprise AI: how to measure return on investment when the output resists easy quantification. Token counts are simple to collect. They say nothing about whether the AI actually solved a problem.

Replit, which runs an online coding platform with built-in AI assistants, adopted internal measures that track developer satisfaction and project completion rates, the AI chief said. The company no longer watches tokens per employee.

Not everyone agrees. Some executives still defend the leaderboards as a straightforward way to push adoption across the organization. The Replit and Amazon pushback suggests the approach is losing ground in parts of the industry.