Assessment Quality with AI: How to Control Hallucinations, Bias, and Difficulty Level in a Question Bank

In a multiple-choice question, a single word ("most appropriate", "first", "which is not") can change what you measure; assessment is sometimes more grammar than content. In Calvino’s Invisible Cities, cities are built from the same material yet run by completely different rules—my question banks are like that too: same topic, same target behavior; but when the stem and distractors change, a different “city” emerges (Calvino, Le città invisibili, 1972).

That’s why generating questions with AI is both great and dangerous. Great, because production speed increases. Dangerous, because quality often deteriorates silently: nobody raises an alarm saying “we measured it wrong”; scores just get weird, people say “that exam was nonsense,” and then the topic closes.

In this article, I’ll propose a practical quality assurance (QA) framework to keep a question bank reliable, fair, and leveled: question generation → quality gates → publishing approval. On the Nextrain side, you can think of the mental model as “AI Gates”: content that can’t pass a gate doesn’t move to the next stage; it goes back, gets fixed, and is tried again.

“Not everything that counts can be counted, and not everything that can be counted counts.” [William Bruce Cameron, 1963]
I translate this to exams like this: You can score everything; but not every score is a measurement.

1) What does “quality” mean in a question bank? (Three axes: accuracy, fairness, level)

I see teams trying to manage question quality with a single “good/bad” label. It’s a simplification the human mind loves. But the question bank quality problem happens on three separate axes:

1. Accuracy (validity + technical correctness):

   • Is there a single correct answer?
   • Is the correct answer truly correct?
   • Is the stem vague?
   • Does it require out-of-context knowledge?

2. Fairness (bias + inclusivity):

   • Is the language level appropriate for the role?
   • Do cultural references advantage one group?
   • Are the things assumed to be “common knowledge” really known by everyone?

3. Level (difficulty + discrimination):

   • Is it too easy/too hard?
   • Does it separate those who know well from those who know less?
   • Are distractors realistic, or “funny wrong”?

What’s interesting is this: people usually debate #1 (“is it right or wrong?”), talk about #2 cautiously (the word “bias” tenses up the meeting), and almost never measure #3 (“we already tested at the end of the training”). Yet deterioration in #3 invisibly corrodes #1 and #2 as well.

2) Typical error classes in AI-generated questions (and quick diagnosis)

The mistakes AI makes while generating questions aren’t “random”; there are recurring classes. In question banks, I most often see these four errors:

A) Ambiguous stem

Symptom: Two different interpretations are possible, two different correct answers are possible.

• Bad example: “Which approach is appropriate for data processing under GDPR?”
  Appropriate according to what—consent, contract, legitimate interest? There’s no context.

• Quick fix standard:

  • The stem must provide a single context: role, situation, constraint.
  • One verb only: “choose”, “identify”, “apply” (not all at once).

B) Multiple correct

Symptom: Two options are correct; the exam measures not “knowledge” but “the question writer’s intent.”

• Diagnostic question: “Could I write the correct answer without seeing the options?”
  If you can’t, the stem is probably missing something; if you can, but two options fit, the options are the problem.

C) Out-of-context knowledge

Symptom: A detail is asked that isn’t in the training content, isn’t in company policy, and may even differ by country’s legislation.

This is something AI does when it tries to help with “general world knowledge.” Like the ocean in Lem’s Solaris: it gives you what you want, but not from your context (Lem, Solaris, 1961). In an exam, this is called: hallucination.

• Quick fix standard:
  • Mandatory “evidence linking” for every question (I’ll explain below).

D) Misleading distractor

Symptom: The distractor isn’t realistic; it’s either obviously wrong or it’s trying to be “tricky.”

• Obviously wrong distractor: Doesn’t discriminate learners; it only measures speed.
• Tricky distractor: Punishes well-intentioned learners; feels like a “joke.”

My small rule: A distractor should be written in the language of a common real-world mistake. It should be “wrong but understandable,” not “wrong and weird.”

3) The most boring but most effective way to reduce hallucination: the “evidence linking” standard

Eliminating AI hallucination completely… sounds great in theory, expensive in practice. A better strategy: prevent hallucination from breaking the exam.

For this, I set a mini “evidence package” standard for every question. Every item that goes live carries these three things:

• Rationale for the correct answer (1–2 sentences): “Why is it correct?”
• Rationale for the wrong answers (for at least 1 critical distractor): “Why is it tempting but wrong?”
• Reference (internal document / training section / regulation heading): “What is it based on?”

When you write this package, two things happen:

1. The question writer (including me) catches the “actually I’m not sure” moment.
2. When an audit comes (GDPR/HSE, etc.), the question bank becomes defensible.

You can use a template like this:

Soru ID: KVKK-ONB-014
Hedef davranış: Veri minimizasyonu ilkesini uygulama
Kök: (senaryo + rol + kısıt)
Doğru seçenek: B
Gerekçe (doğru): ...
Çeldirici notu (A): ... (neden yanlış ama cazip)
Referans: KVKK farkındalık eğitimi / Bölüm 2.3 + kurum politikası madde X
Dil seviyesi: Operasyon
Risk etiketi: KVKK

Let me make a small correction here: I just said “at least 1 critical distractor”; in high-risk areas (GDPR, HSE, ethics) I actually want a short rationale for all options. Because in these areas, “wrong information” doesn’t just break measurement; it can break behavior.

4) Bias and inclusivity: language, culture, role… and invisible thresholds

Bias is perceived in many teams as “bad intent.” What I see more often is: laziness in language and context.

Three types of risk:

1) Language bias (literacy threshold)

The same knowledge can be asked with different language load.

• Bad example (high language load):
  “Which of the following is a holistic representation of the minimum elements that the data controller must provide to the data subject within the scope of the obligation to inform?”

This sentence isn’t an assessment question; it’s a breathing test.

• More inclusive version:
  “Which information must definitely be included in a GDPR privacy notice?”

Same topic, lower language barrier.

2) Cultural bias (reference barrier)

Examples you assume “everyone knows” may be unfamiliar to some groups: certain cities, certain social practices, certain family roles.

My rule: examples should come from the work context. If it’s sales, then sales; if it’s a warehouse, then a warehouse; if it’s a call center, then a call center. A question bank isn’t a literature club. (It’s ironic for me to say this because I just cited Calvino. But I use the reference for myself, not for the reader; not in the questions.)

3) Role bias (work reality barrier)

Not everyone can apply the same concept with the same authority.

• Bad example: “The employee must stop the data processing activity.”
  Which employee? With what authority?

• Fixed: “Before approving a suspicious data sharing request, which unit should the employee escalate to?”
  This measures an action the role can actually take.

One user behavior still surprises me: the same organization, in the same week, says both “the questions are too easy, nobody takes it seriously” and “the operations team can’t understand the text.” Both can be true at the same time—but the solution isn’t “make it harder”; it’s to reduce language load and clarify the cognitive target.

5) Difficulty and discrimination: a quick checklist with basic item analysis

As the question bank grows, “gut feeling” isn’t enough. I recommend a quick item analysis checklist. Don’t let the statistics part scare you; the core signals are simple.

The table below has practical thresholds (adjust by organization):

Because I know Nextrain has event-level tracking on the analytics side like “Answer” and “Time,” I especially like these two signals: correct rate alone can be misleading; combined with time, you start separating whether a question is “hard” or “unclear.”

There’s a small trap here: very hard questions aren’t always bad. If the target behavior is “expert differentiation,” p=0.25 can be perfectly fine. What’s bad is the question being misaligned with the target level.

6) Flow design: Question generation → quality gates → publishing approval (the AI Gates mindset)

Question bank quality work is less about fixing questions one by one and more about process design. I set it up as a three-gate flow. I like the term “AI Gates” because it’s mechanical: if you can’t pass, you can’t pass.

Gate 1 — Structural validation (format + single correctness)

Goal: Catch the cheapest errors as early as possible.

Checklist:

• Does the stem ask for a single action?
• Is there a single correct answer?
• Are the options grammatically aligned with the stem?
• Are options like “All of the above / None of the above” avoided unless necessary?

Gate 2 — Evidence and context (hallucination + organizational alignment)

Goal: Link “why the correct answer is correct.”

Checklist:

• Is there a reference? (training section / policy / regulation)
• Is a rationale written?
• Is there a risk of misinformation in areas like GDPR/HSE?
• Does the scenario fit organizational reality?

At this point, Saadet, our “We’ll Handle It” Specialist, usually asks the best question: “When someone in the field reads this, do they say ‘okay, but what am I supposed to do?’” Her reflex catches bias and role mismatch early—because the customer problem is often not “the information is wrong,” but “the action is unclear.”

Gate 3 — Statistical health (difficulty + discrimination)

Goal: Keep the question bank alive; catch deterioration early.

Checklist:

• Is the p-value in the target range?
• Is discrimination negative?
• Is a distractor never selected?
• Is there a time anomaly?

This gate doesn’t have to be “pre-publish.” I often publish first, then run this gate with real user data. Because some questions look perfect on paper and collapse in the field. People misunderstand in creative ways.

7) Practical example: Improving the same question four times (ambiguity → evidence → bias → level)

Let’s walk through an example. Topic: a basic behavior in HSE (a general example; customize for your organization).

Version 0 (AI’s quick generation)

Question: “Which behavior is correct in terms of HSE?”
A) Being careful
B) Following the rules
C) Reducing risks
D) Getting training

This isn’t a question; it’s a motivational poster.

Version 1 (fix the ambiguous stem)

Question: “You see a liquid spill on the workshop floor. What should you do first?”
A) Jump over it and continue
B) Mark the area and inform the relevant person
C) Take a photo and post it to the group
D) Clean it when the shift ends

Now there’s a behavior and a “first step.”

Version 2 (link evidence)

• Correct: B
• Rationale: Slip/fall risk is immediate; making the area safe and reporting is the priority.
• Reference: HSE basic training / “Hazard reporting and area safety” section

Here, hallucination risk decreases: “take a photo and post it to the group” may be encouraged in some organizations—but not as the “first step.” The reference anchors it.

Version 3 (reduce bias: role and authority)

Suppose you notice some employees don’t know who the “relevant person” is.

Revised stem: “... What should you do first?”
B) “Mark the area and inform the shift supervisor / HSE representative”

This brings the action down to role reality.

Version 4 (level tuning: make distractors realistic)

“Jump over it and continue” is too cartoonish. People usually make more reasonable mistakes: they say “I’ll clean it.”

A) “Quickly clean it yourself so you don’t bother anyone”
D) “If you don’t have the right equipment, leave it to the cleaning team at the end of the shift”

Now the wrong options are more “human”; discrimination increases.

Closing: A question bank is not a content warehouse; it’s a measurement instrument

In Borges’s “map” story, the empire makes a map at a one-to-one scale of the country, and the map eventually rots and blends into the desert (Borges, “Del rigor en la ciencia”, 1946). Question banks can become like that too: they grow and grow and grow—then nobody trusts them, nobody uses them, and they rot under their own weight.

The framework I’m proposing isn’t about keeping the question bank small; it’s about keeping it healthy:

• Name the error classes,
• Set an evidence linking standard,
• Treat bias control as a design issue, not an “intent” issue,
• Monitor difficulty and discrimination with simple statistics,
• And run it all through a gated process.

AI accelerated question generation. Protecting measurement quality is still an engineering job: boring checks, clear standards, repeated gates. Boring things usually work. (As I write this, I used the word “usually” a bit too comfortably; some boring things are just boring. But quality gates—yes, they work.)

Notes

1. Italo Calvino, Le città invisibili (Invisible Cities), 1972.
2. Stanisław Lem, Solaris, 1961.
3. William Bruce Cameron, “Not everything that counts can be counted...”, 1963 (common quote).
4. Jorge Luis Borges, “Del rigor en la ciencia” (in El Hacedor), 1946.