EMS-MCQA: 18,602 exam-style questions, and why ours stay internal

Writing a good assessment question is slow. The stem has to test one decision, the wrong answers have to be believable, and the explanation has to hold up with a medical director. An educator facing a blank page could use a few thousand examples to start from. That's the job EMS-MCQA does for us, and it's also as far as we let it go.

#What EMS-MCQA is

EMS-MCQA is an open multiple-choice dataset for emergency medical services, published by a university research group on Hugging Face. The open release contains:

• 18,602 questions, split into training (13,021), validation (1,860) and test (3,721) sets for AI research.
• Up to six answer choices per question. Most have a single correct answer, and some have several.
• Tags for certification level (EMR, EMT, AEMT, paramedic) and topic: airway and ventilation, anatomy, assessment, cardiology and resuscitation, EMS operations, medical and obstetrics, paediatrics, pharmacology and trauma.
• An optional explanation and a link to where each question was found.

The researchers built it to test how well AI language models answer EMS questions. The authors describe the published set as an open subset of a larger collection.

#The licensing problem

The dataset is labelled MIT, which is a permissive licence. But the source links show where the text came from: many questions were collected from quiz and flashcard sites and exam-prep providers. A dataset author can't relicense text they didn't write. So the MIT label covers the dataset's packaging, not the questions themselves.

For a research paper that's a manageable risk. For a commercial training product sold to paramedic services, it isn't. Our position is simple: no EMS-MCQA question reaches a paramedic in its original form.

#How Imedica uses EMS-MCQA

Inside Imedica, administrators can import EMS-MCQA into an internal question bank:

1. Import. We download the dataset when the import runs. We don't copy it into our public code.
2. Clean. We keep only questions our quiz format can show: one correct answer and well-formed choices. Questions with several correct answers or broken formatting are skipped. We also tidy the text (capitalisation, blanks marked with invisible characters).
3. Browse. Educators search by level, topic and keyword. Every question shows its source.
4. Copy as a starting point. An educator can copy a question into a draft assessment. From there it follows the same rule as everything else we publish: a physician rewrites it against current directives, and a second person approves it before it can go live.

The whole feature sits behind an internal switch. Customer accounts can't see the question bank, and its API answers "not found" when the switch is off.

#What it's good for, and what it isn't

Good for:

• Finding gaps: if we have no assessment questions on, say, obstetric emergencies, the bank shows what other curricula ask.
• Wording ideas for distractors, the wrong answers that tempt someone who half-knows the material.
• Calibrating difficulty across certification levels.

Not good for:

• Canadian accuracy. The questions follow US exam content. Scope of practice, drug choices and protocols often differ from Ontario's ALS/BLS standards or other provinces' directives.
• Currency. Some quiz-site questions are out of date or just wrong. An answer key isn't evidence.
• Measuring decisions. A multiple-choice question tests recall. Imedica's main measure is what a paramedic does inside a scenario, and quiz items are only a small part of an assessment.

#For training leads

If you build quizzes from free online question banks, check where each question came from and whether it fits your service's medical directives. A wrong answer key teaches the wrong thing with confidence, and copying commercial prep material can create licensing trouble for your service.

#The takeaway

EMS-MCQA is a big, useful reference for drafting EMS assessments. We use it inside the company as a source of ideas. Because of where its text came from and the US focus of its content, every question that reaches a paramedic is rewritten and approved by a physician first.

Written by the Imedica team. Dataset figures checked against the EMS-MCQA documentation on Hugging Face, October 2026.