01
Compare the product contract before comparing outputs.
AI services can look similar in a chat window while differing in public model identity, routing control, compatible endpoints, key permissions, data handling, rate limits, and billing. Write down the requirements your application actually depends on before testing a favorite prompt.
| Dimension | Question to answer | Evidence |
|---|---|---|
| Product identity | Stable service alias or explicit model catalog? | Models endpoint and official docs |
| API surface | Which requests, streams, tools, and formats work? | Wire-level integration tests |
| Behavior | How often is the task attempted and completed correctly? | Dated prompt set with human review |
| Operations | What are latency, rate limits, failures, and support like? | Production-shaped load and error tests |
| Economics | What does representative input and output cost? | Token usage and current pricing |
| Data | What is stored, logged, routed, or deleted? | System, privacy, terms, and deployment contract |
02
Hold the task and measurement rules constant.
- Define task categories, allowed content, and acceptance thresholds before collecting outputs.
- Record each service configuration, date, prompt, parameters, and output limit.
- Score attempt or refusal separately from factual correctness, completeness, and instruction following.
- Measure first-token latency, total latency, token usage, cost, and failure rate over repeated samples.
- Exercise authentication, context, rate-limit, streaming interruption, and temporary outage paths.
03
Current comparison guides.
The guides below compare product structure and give you a decision framework. Third-party catalogs, prices, policies, and terms can change, so each guide directs readers back to current official documentation rather than freezing volatile facts as permanent claims.