# How to Evaluate UX Agency Case Study Evidence

> A practical way to assess UX agency case studies: check project fit, agency role, research, delivery, source quality, and outcome claims.

Published: 2026-11-02
Author: UXComparation editorial desk

## In brief

A useful case study lets you follow a chain: the user or business problem, the agency’s contribution, the work that reached people, and the evidence behind any result.

## Treat the case study as a starting record

An agency case study is written to explain selected work. It can be a valuable source for understanding how the agency describes a project, its role, and its results. It is still one account, shaped by what the agency can publish and chooses to emphasize.

A client name or familiar logo establishes very little about the engagement on its own. Look for a named project, a description of the actual product or service, the users involved, and the work the agency says it performed. When a project is confidential, ask the agency for a private reference or a fuller explanation under its disclosure rules. [1](#source-1) Related: [our source standards](https://uxcomparation.com/sources/).

- A logo or client list supports a lead for further research; it does not establish the scope of UX work.
- A portfolio page supports the agency’s published account. A client source may independently confirm the relationship, role, launch, or outcome it describes.
- A review platform documents the reviews and profile fields shown there. It does not provide a standardized measure of design quality across firms.

## Check whether the project matches your problem

Start with the decision your team needs to make. “Redesign the dashboard” names a deliverable. A more useful brief explains who uses the dashboard, what they are trying to do, where the current experience gets in the way, and what constraints shape a change.

Then compare that context with the case study. Similar screens can hide very different work. A consumer onboarding flow, a clinical workflow, and a multi-role enterprise system each carry different user needs and delivery constraints.

Look for a problem statement specific enough to distinguish this project from a generic redesign. Useful context may include the user’s task, the existing workflow, the platform, the business or service environment, and the product stage. A case study does not need to reveal confidential details to explain the shape of the problem. It can describe user roles and constraints without naming individuals or exposing private data.

- Which users and tasks appear in the case?
- Does the product, workflow, or operating environment resemble yours?
- Does the case explain the starting problem, or only show the finished interface?
- Which important constraints—technical, regulatory, accessibility, or organizational—are described?

## Find the agency’s specific role

A project may involve the client’s product team, an engineering partner, research vendors, and several design firms. Identify what this agency owned: discovery, participant research, product strategy, interaction design, visual design, prototyping, implementation, or ongoing optimization.

Pay attention to verbs and attribution. “We partnered on the redesign” gives less detail than a source that names the research plan, tested workflows, design system, or release the agency delivered. If multiple teams contributed, keep the boundary visible instead of assigning the whole result to one firm.

Also distinguish the firm’s portfolio from the team you would hire. A case may feature a senior strategist or design lead who will not be assigned to a new engagement. Ask who performed the work shown, who would lead your project, how much time the proposed team would spend, and how decisions move between agency and client. The public case can show the firm’s capability; the proposal should explain the people and working arrangement available to you.

- What did the agency do directly?
- Which work belonged to the client or another partner?
- Was the agency involved in research, implementation, or both?
- Can a client-side source confirm the role when that distinction matters to your selection?

## Look for a traceable research and design process

A strong account connects user understanding to design decisions. Useful details include the question the team needed to answer, the people it learned from, the method chosen, what changed after the findings, and how the proposed experience was evaluated.

Method names alone do not show rigor. “We conducted interviews” leaves questions about who took part, how they relate to the product’s users, and how the findings influenced the work. A small usability study may reveal specific interaction problems; it should be interpreted within the tasks, participants, and conditions described for that study. A qualitative sample can help a team understand behaviors and uncover issues; it does not automatically estimate how common those behaviors are across a larger population.

The GOV.UK Service Manual recommends grounding service decisions in user needs, testing prototypes, and continuing research as a service develops. The U.S. Digital Services Playbook similarly calls for research with real users, documented findings, prototype testing, and regular evaluation as a service is built. The Centers for Medicare & Medicaid Services describes usability research as observing people attempt tasks and asking follow-up questions. These are practical public-sector references, not a universal certification checklist. Use them to ask how a case learned and adapted. [5](#source-5) [6](#source-6) [7](#source-7) [1](#source-1) [2](#source-2)

- What question was the research meant to answer?
- Who participated, and how were they recruited?
- What tasks or concepts were tested?
- What did the team change after observing or hearing from users?
- Was the work tested again after the change?

## Separate deliverables from outcomes

A prototype, redesigned flow, research report, or design system is a deliverable. These artifacts can show what the team produced and help you judge whether the work resembles your needs.

An outcome describes a change after the work reached its intended setting: for example, task completion, errors, onboarding progress, support requests, adoption, or conversion. The case should identify who observed or measured the change and how it relates to the agency’s contribution. A finished interface by itself does not show that users adopted it or that a business metric improved.

Delivery status matters. A concept, clickable prototype, pilot, limited release, and generally available product are different stages. A case study can show valuable strategic or exploratory work even when a concept did not ship, but the page should make that stage clear. If a case uses present tense or says a product “launched,” look for a current product page, app, client statement, or other source that supports the status. [1](#source-1)

## Read every metric with its context

Before using a percentage or testimonial in a decision, look for the baseline, measurement period, population, data source, and definition of the metric. Ask whether the figure covers all users or a selected group, and whether other changes happened during the same period. [3](#source-3)

A result reported by an agency should be labeled as agency-reported unless a client or independent source confirms it. A client quotation can support that person’s stated experience. It does not replace a measurement method or establish that the same result will occur in your product.

If the source leaves out context, keep the useful part and record the gap: “The agency reports an increase in task completion; the public case does not state the baseline or measurement period.” That is more informative than repeating the number without qualification or discarding the case entirely.

- Baseline: what was measured before the work?
- Population: whose behavior or experience is represented?
- Period: when and for how long was the result measured?
- Method: where did the data come from, and how was the metric defined?
- Attribution: who reports the result, and what supports a link to this work?

## Match the source to the question you are asking

Different sources answer different questions. An agency portfolio is usually the best starting point for the agency’s own description of its work. A client announcement or product record can add independent confirmation of a relationship, launch, or role when it names those details. A review can help you understand one client’s reported experience with a firm. An award record can establish that a named project received a particular award.

Do not let a source answer a broader question than it supports. A business directory may show a company’s listed location or services. It does not establish that a specific product was designed by that company. A social portfolio can show visuals posted by an account. It does not, by itself, confirm the client, the agency’s role, or whether the work shipped. A review platform’s category placement is a fact about that platform’s listing; it is not a finding from your research.

For a high-stakes project, a second source may matter more than a longer portfolio story. For an early concept sprint, the agency’s method and fit with your team may matter most. Decide which fact would change your selection, then seek a source capable of establishing that fact.

- Agency portfolio: what the firm says it did and what it chose to publish.
- Client or product source: independently stated relationship, role, launch, or result, if the source names it.
- Review platform: an attributed client account and the platform’s review fields.
- Award body: a specific award, category, and project named by the issuer.
- Directory or social portfolio: listed company details or publicly posted visuals, with the limits of that source kept visible.

## Check the use environment when the work is complex

Complex products are shaped by where, when, and under what conditions people use them. A case about clinical software, an industrial control panel, a field-service tool, or a multi-role data platform should describe enough of that context to explain the design decisions. Consider interruptions, time pressure, device constraints, permissions, accessibility needs, and what happens when a user makes a mistake.

This is especially important in healthcare and other safety-sensitive work. FDA human-factors guidance for medical devices focuses on intended users, use environments, and the user interface, with attention to use-related risk. That guidance applies to medical devices and regulated contexts; it is not a general UX scorecard. For a health product case, it can help frame questions about user groups, critical tasks, foreseeable errors, and how evaluation related to risk. [4](#source-4)

For enterprise software, ask similar contextual questions without treating the FDA guidance as applicable regulation: which roles use the product, how work passes between them, what information is sensitive, which systems are connected, and whether the design was evaluated under realistic tasks. A simple dashboard screenshot may not reveal whether those workflows were understood.

## Read participant details without demanding false precision

Participant information helps you judge relevance. Look for the characteristics that affected the project: job role, experience, access needs, language, device, environment, or familiarity with the task. The right details depend on the research question. A case study should preserve privacy, so it may not publish a precise participant profile or every recruitment criterion.

Treat sample size in context. In moderated qualitative research, the purpose may be to observe task behavior, identify usability problems, or explore needs. A count alone does not tell you whether the sessions addressed the right users or tasks. Conversely, a small sample should not be presented as a population-wide estimate unless the study design supports that inference.

Ask whether participants were current or likely users, who recruited them, what exclusions applied, and which findings were directional versus measured. For research intended to compare rates or estimate prevalence, ask what quantitative method and sampling approach support those conclusions. The case should connect the method to the question instead of treating a familiar method label as proof.

## Look for what the case leaves out

Published cases are selected stories. They can omit unsuccessful directions, tradeoffs, handoff issues, later product changes, or evidence that did not support the preferred solution. The omission does not prove that the agency handled the project poorly; it tells you what remains unknown from the public record.

Notice whether the story explains constraints and decisions as well as polished outcomes. Did the team test an assumption that proved wrong? Which options were considered and set aside? What depended on client decisions or engineering capacity? Did a later release change the original design? A case that describes the path can help you assess how the team responds to uncertainty, not just how it presents a final screen.

You can ask an agency to discuss a challenge without asking it to disclose confidential client information. A responsible answer may describe the tradeoff, the decision process, and what the team learned while protecting identities and proprietary details.

## Use a small evidence ledger

A consistent note-taking format makes agency comparisons easier to revisit. Keep claims separate from the evidence and from your own interpretation. This also prevents a persuasive case study from receiving more weight simply because it is longer or better produced.

- Claim: the exact point you want to rely on.
- Source: direct URL, publisher, source type, and date checked.
- What it supports: the specific project, role, activity, launch, review, or result stated there.
- Confidence: agency-reported, client-confirmed, independently corroborated, or still unresolved.
- Open question: what you need to ask the agency or verify elsewhere.

## Questions to take into an agency conversation

Public case studies rarely answer every procurement question. Use the gaps to guide a conversation with the people who would do the work. Ask each shortlisted agency the same questions so you can compare the answers against your brief.

- Who from this case would be on our project, and what would each person own?
- Which part of the case is most comparable to our users and constraints?
- What did you learn that changed the direction of the work?
- Which results came from client measurement, and which are your team’s account?
- What happened after handoff or launch, and what evidence can you share?
- Which assumptions in our brief would you validate first?

## Make a decision with the evidence you have

A case study is one input in a wider selection process. Compare relevant work with the proposed team, the delivery plan, references, and the constraints in your brief. Record uncertainty explicitly and ask whether it matters to the project you are commissioning.

The goal is not to award a universal score to every portfolio. It is to understand why a particular agency may fit a particular job, what supports that view, and what you still need to learn before signing a contract. Related: [our source standards](https://uxcomparation.com/sources/) · [research methodology](https://uxcomparation.com/methodology/).

## Sources and further reading

### <a id="source-1"></a>[1] [The Digital Services Playbook](https://playbook.usds.gov/)

Publisher: U.S. Digital Service. A U.S. government playbook calling for research with real users, documented findings, prototype testing, and continued evaluation while a service is built.

### <a id="source-2"></a>[2] [Usability Research at the Digital Service at CMS](https://www.cms.gov/digital-service/usability-research)

Publisher: Centers for Medicare & Medicaid Services. A U.S. federal example of task-based usability sessions, observing users, and asking follow-up questions.

### <a id="source-3"></a>[3] [Usability Testing](https://digital.gov/topics/usability-testing/)

Publisher: Digital.gov. Federal guidance on usability testing, goal completion, and evaluating digital services.

### <a id="source-4"></a>[4] [Applying Human Factors and Usability Engineering to Medical Devices](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/applying-human-factors-and-usability-engineering-medical-devices)

Publisher: U.S. Food and Drug Administration. Device-specific guidance on intended users, use environments, interface design, and reducing use-related risks. Apply only where the project involves medical devices or relevant regulated work.

### <a id="source-5"></a>[5] [Understand users and their needs](https://www.gov.uk/service-manual/service-standard/point-1-understand-user-needs)

Publisher: GOV.UK Service Manual. Guidance on grounding service decisions in research about actual user needs and the context around a problem.

### <a id="source-6"></a>[6] [Making prototypes](https://www.gov.uk/service-manual/design/making-prototypes)

Publisher: GOV.UK Service Manual. Guidance on using prototypes to explore and test design approaches before committing to a build.

### <a id="source-7"></a>[7] [Using moderated usability testing](https://www.gov.uk/service-manual/user-research/using-moderated-usability-testing)

Publisher: GOV.UK Service Manual. Guidance on research questions, relevant participants, tasks, and observing people use a service.

HTML article: https://uxcomparation.com/articles/what-good-agency-evidence-looks-like/