Calibrating the AI

A rubric tells the AI what earns points, but not how strict to be. Is "42" without the book title worth half the points, or none? Calibration answers settle questions like that: previously graded answers to the same question, which the AI uses as examples of how your teachers apply the rubric.

The AI still grades every answer on its own merits. It uses the examples to judge how strictly to apply the rubric and when to give partial credit.

Adding calibration answers

Add calibration answers per question, after you've created the exam and its questions and before you import the answers you want graded. Each one is an answer with the points a teacher gave it:

[
  {
    "value": "42, from The Hitchhiker's Guide to the Galaxy.",
    "pointsAwarded": 1,
    "reasoning": "Names both the number and the book."
  },
  {
    "value": "The answer is 42.",
    "pointsAwarded": 0.5,
    "reasoning": "Correct number, but doesn't say where it comes from."
  },
  {
    "value": "To be happy.",
    "pointsAwarded": 0,
    "reasoning": "A personal answer, not the one from the book."
  }
]

The endpoint takes an array of up to 1,000 answers per request, and returns 201 Created without a body.

For example, an answer to the analytical rubric from the developer guide can say which level the teacher picked:

{
  "value": "The answer is 42.",
  "pointsAwarded": 0.5,
  "scoringCriteria": [
    { "id": "c1", "selectedLevelId": "c1l2", "pointsAwarded": 0.5 }
  ],
  "reasoning": "Correct number, but doesn't say where it comes from."
}

Choosing good examples

A handful of well-chosen examples does more than a large pile of similar ones:

• Cover the range. Include weak, middling and strong answers. Answers in the middle are the most informative, since that's where partial credit gets decided.
• Explain the score. A short reasoning shows the AI what made the difference, which is more useful than the score alone.
• Keep them representative. Pick answers that look like the ones your students actually write, rather than unusually long or unusual ones.

Equally, however, our system will automatically choose the most representative examples for the grading operation from the set you provide, so feel free to upload as many examples as you can.

Different point scales

Calibration answers are usually graded out of the question's maximum points. If yours were graded on a different scale, for example out of 10 in your own system, set pointsToEarn to that maximum. Examplary then rescales the score to the question's current maximum, so { "pointsAwarded": 7, "pointsToEarn": 10 } counts as 70% of the points. This also keeps calibration answers useful if you change the question's points later.