Assessment Quality Is Not an Item-Production Problem

Assessment leaders still make one costly mistake: they treat assessment quality as a content-production problem. If enough items are written and the form goes out on schedule, the program is assumed to be in good shape.

In my experience, that is where problems begin.

High-quality assessment is not built by volume. It is built by making disciplined decisions about what the assessment is meant to measure, why it exists, who it serves, and how the results will be used. When those decisions are weak, the damage shows up later in rework, questionable results, and assessments that are hard to defend.

That is the distinction more leaders need to make. Assessment is a measurement system. In this blog, we will discuss why the quality of that system is set long before item writing begins.

Define Assessment Quality Before You Try to Scale It

I use three non-negotiables when I think about assessment quality.

1. Assessment Validity with a Purpose

An assessment has to measure what it claims to measure, and that claim has to be tied to a clear use. A formative check, a summative assessment, a placement tool, and a skills assessment do not serve the same purpose. If the intended use is vague, the assessment will be vague too.

2. Assessment Representation

A strong assessment reflects the students or test takers it is built for. Cultural responsiveness is not a late-stage review note. It is a design requirement. If the assessment does not represent the population it serves, then quality is already compromised.

3. Assessment  Accessibility

If students are being blocked by the design rather than challenged by the construct, the assessment has failed before scoring ever begins.

Those three standards sound basic, but they are where many programs go off course. Teams often move too quickly to production and assume quality will emerge from scale. It does not. Scale only multiplies the strengths or weaknesses already built into the design.

Build a Defensible Assessment Lifecycle

Once the purpose of your assessment is clear, the work becomes operational.

First, define the construct. Be precise about what the assessment is intended to measure. Then build the blueprint. The blueprint is the structure that keeps the assessment aligned to its purpose. Without it, item writing becomes less of a measurement exercise and more of a content exercise.

From there, development can move forward: write the items or tasks, review them carefully, refine them, and prepare them for use. But a defensible assessment needs to be piloted with actual students where possible. It needs a psychometric review and statistical analysis, including differential item functioning analysis, to identify whether items may be introducing bias. And it needs revision, not once, but iteratively.

That sequence often causes weak assessments to reveal themselves early, not when students interact with them. Pilot testing and statistical review are how teams learn whether an assessment is functioning as intended. They are also how teams avoid a familiar and damaging mistake: assuming poor performance reflects a problem in the learner when the problem is actually in the test design.

Make Your Assessment Results Useful, Not Just Reportable

A score alone is not much of an achievement.

Assessment becomes instructionally valuable when it gives students feedback they can use to move forward and gives educators information they can use to adjust instruction. That is a much better standard than treating assessment as a simple gatekeeping tool.

For too long, assessments were asked to sort students and move on. That model still shows up everywhere. Learners sit down, take the test, get the score, and accept that result as the main story. But a single test event is a thin basis for understanding what a learner knows or can do. One score should not carry that much weight.

Leaders should push for assessment systems that produce evidence that is useful for decision-making, not just easy to report. If the goal is placement, intervention, readiness, or support, the assessment must generate evidence that actually helps inform those decisions. Otherwise, the program is creating measurement theater: a lot of process and not enough value.

Know How Assessment Programs Drift

Assessment programs can start to drift when schedule and output become the dominant measures of success. A team starts with a clear construct, a sound purpose, and a strong design vision. Then the pressure shifts. The conversation becomes about how many items must be completed by when. Once that happens, the assessment starts serving the production calendar instead of the learner.

Leaders should absolutely care about timelines and throughput. But those should not be the only health indicators.

A healthier set of signals includes alignment to purpose, overall quality, accessibility, cultural responsiveness, and whether the workflow is transparent and improving over time. These are the indicators that tell you whether the program is becoming more defensible or simply more efficient at producing questionable content.

Use Technology Carefully, and Change the Format Where the Claim Demands It

Technology can improve the development workflow. Used well, it can help teams move faster and create a starting point for development. But it does not change the standards for good assessment.

Human judgment remains central because the hard part of assessment is deciding whether your content is aligned, fair, accessible, and meaningful. This work still belongs to experienced professionals. Teams need a clear process for how writers and reviewers use technology, where human review is required, and where judgment cannot be outsourced.

The more important design question is whether the format matches the claim.

This is where I see the next meaningful shift for publishers, edtech companies, higher education, and workforce programs. We still rely too heavily on selected-response formats for claims that they were never designed to support. Those formats remain useful when the goal is efficient measurement of specific knowledge or standards. They are quick, and they can be psychometrically reliable. But they are not enough when the real question is whether someone can apply knowledge, exercise judgment, or demonstrate a skill in context.

Why Scenario-Based Assessment Is the Stronger Next Step

If we are serious about measuring what learners can actually do, then we have to admit something uncomfortable: too many assessments still rely on formats that would look familiar nearly a century ago. Traditional selected-response items are efficient, and there are valid reasons they remain part of the landscape. They are quick, scalable, and often statistically dependable. But efficiency is not the same thing as completeness.

Scenario-based assessment offers a better direction because it can produce richer evidence of performance, decision-making, and applied understanding. Instead of asking learners to recognize the right answer from a list, it can ask them to work through a situation, apply judgment, and demonstrate what they can do over time. That is much closer to how knowledge operates in the real world.

This matters especially in higher education and workforce contexts, where the real question is not whether someone can pick A, B, C, or D, but whether the assessment aligns with real-world skills and uses clear rubrics to generate meaningful scores.

In those environments, scenario-based approaches are more defensible because they can provide more useful insights into placement, readiness, and support.

I am not arguing that every fixed-form item should disappear tomorrow. We should stop mistaking item output for assessment quality. We should stop confusing fast scoring with full understanding. And we should stop assuming one score tells the whole story of a learner.

What Assessment Leaders Should Do Now

Start by asking harder questions earlier.

• What is this assessment for?
• What decision will it support?
• Does the design reflect the students it serves?
• Is accessibility built in from the start?
• Has the assessment been piloted, reviewed statistically, and revised?
• Does the format match the claim, or are we forcing everything through the easiest item type to produce?

Those are the foundation of assessment quality.

If leaders want assessments that are credible, useful, and worth the decisions attached to them, they need to stop treating assessment quality as a production issue. Assessment quality begins with purpose, is strengthened by disciplined design, and is proven through evidence.