A Simple Rubric for Comparing Model Answers

Two model answers can look equally convincing while differing in ways that matter. One may be correct but difficult to use. Another may read beautifully while missing a requirement. A small scoring rubric helps you compare those differences without turning a personal project into a research study.

The aim is not to find the best model for everything. It is to find an answer you can trust for a particular job, at a cost you can justify.

Choose a task you can actually check

Start with one task from your own work, such as explaining a course concept, reviewing a short function, or summarizing a document. Prefer something where you can verify the important details yourself.

Before requesting any answers, write down what success means. For a summary, that might mean preserving three key facts, staying within a word limit, and adding no unsupported claims. For a programming explanation, it might mean identifying the bug and explaining why the proposed correction works.

Keep a reference nearby: your source document, course notes, or a test result. An answer should earn points because it matches evidence, not because it sounds confident.

Turn your expectations into four scoring criteria

Use four criteria, each scored from zero to two. Define the scale before reading the answers so that you are less tempted to reward whichever writing style you happen to prefer.

• Correctness. Award two points when the important claims check out, one when a minor error needs correction, and zero when a central claim is wrong or unsupported.
• Instruction following. Award two points when all explicit requirements are met, one for a small miss, and zero when the answer ignores an essential constraint.
• Usefulness. Award two points when you can apply the answer with little extra work, one when it needs meaningful editing, and zero when it does not solve the task.
• Clarity. Award two points when the explanation is easy to follow, one when it requires rereading, and zero when ambiguity makes it difficult to use.

This gives each answer a score out of eight. Also define a rejection rule: a serious factual error or an unsafe recommendation makes an answer unacceptable regardless of its total. Polished prose should not cancel out a broken solution.

Compare answers under the same conditions

Send the same prompt and supporting material to two candidates. Keep shared settings similar where possible, including the response length allowance. Start fresh conversations so that earlier messages do not give one candidate extra context.

Hide the model names and mark the outputs as Answer A and Answer B before scoring. This will not remove every bias, but it reduces the temptation to favor a familiar name or a higher price.

Read each answer against the reference first, then apply the rubric. Add a sentence explaining every lost point. Those notes make your judgment easier to revisit than a bare number.

Record cost separately from answer quality

Do not fold price into the correctness score. Record the quality score, request cost, and approximate editing time as separate observations. This lets you see whether a cheaper answer genuinely saves money or merely moves the work back to you.

MyRelay shows the tokens and cost of each request, which helps you compare actual spending rather than estimating from answer length. If you want to run this comparison through one personal budget, create a MyRelay account and set a small spending limit for your experiment.

Repeat the test before choosing your default

One answer is a sample, not a dependable ranking. Try three to five representative tasks, and repeat any close comparison if your budget allows. Set the number of requests beforehand so that testing does not become an open-ended expense.

Look for the lowest-cost candidate that consistently clears your quality threshold without rejection-level mistakes. Keep a stronger alternative for tasks where your default struggles. Save your prompts, scores, costs, and brief notes so you can rerun the same comparison when your needs change.