What is the best benchmarking tool for graphics accelerators?

I recently read a question on CGSociety forums on how to evaluate graphic card using benchmarks. Let me share an edited version of the question and then provide what I consider the most realistic and valuable answer.

Answer:
Benchmarks are a single metric for evaluating any product, including graphics cards. They test only what they are designed to test. It is virtually impossible to create a benchmark that will cover all possible user cases. Benchmarks are based on specific workloads and actions to run. And while you can look specifically at those benchmarks that best align with what you do on a daily basis, the workloads and tasks are never truly personalized.

The best benchmarking tool is to try the cards out to see which performs better for your particular needs (you should check the return policy for particular web vendors to make sure you can return a card). There is a lot of brand loyalty out there for many products, including graphics cards, which typically is not based on any kind of comparison against the competition. What may have been true for accelerators from a year ago, is not likely true today, as the drivers are constantly enhanced and optimized.

With respect to published benchmark scores, FireGL cards have shown a performance advantage over Quadro in SPECViewperf for a while now, particularly with Maya. It may be that the graphics architecture for FireGL is better suited to run Maya. Or maybe the FireGL graphic drivers are simply better tuned. Even though others may be recommending to go with Quadro, it would be unwise to completely ignore documented benchmarks performance advantages. The difference just means you need to look more closely.

1. Published benchmarks are certainly a good starting place for your own evaluation.
2. Recommendations from colleagues can also be useful. But make sure they have actually tested the alternatives (i.e. they are not just speaking as a fanboy), that you know what version of driver they tested, and that you understand what their workflow is like relative to your own.
3. The best benchmark is for you to compare the cards yourself, using your data and your workflows, if possible.

An informed decision is always the best decision.