Cheapest per token is not cheapest per finished job: what 2,400 runs show
A model with a 33% pass rate turned out to be the cheapest way to finish a task in a recent 10-model benchmark. The same benchmark shows why that result is less of a free lunch than it looks. Here is the metric behind it and how to compute it on your own prompts. The metric Cost per successful task = everything the model spent across every attempt (failed, retried, timed out) divided by the number of attempts that passed. Per-token price cannot see retries, malformed outputs or runs that never f
