Skip to main content

Table 3 Prioritization and categorization of usability problems

From: The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability

Severity rating

Complexity

Scope

Abbreviated UP

Usability problem

Problem types

1.33

High

2

Focus on barriers detracts from case presentation

During initial case presentations, clinicians tend to focus on barriers to actually applying MBC, potentially detracting from other important topics of discussion and decreasing motivation to implement MBC (inferred).

U

H

ST

F

CS

1.67

Medium

5

Unprepared to identify solutions to barriers

When generating solutions to perceived barriers to using MBC during late-stage consultation calls, clinicians don't feel prepared to identify appropriate/insightful solutions in the moment, leaving them unsure how to proceed (stated), and discouraged or unmotivated to use MBC (inferred).

U

H

ST

F

CS

1.67

Medium

7

Inadequate on-site technology

Consultation calls employ videoconference technologies and equipment, but some clinicians do not have necessary hardware or technological supports, which might detract from the level of engagement or ability to participate during the calls (inferred).

U

H

ST

F

CS

2.00

Medium

5

Rapid assessment misaligned with available time

The consultation protocol assumes a rapid assessment and feedback process between meetings to identify treatment goals (4 weeks), which clinicians experienced as shorter than amount of time often allotted, creating a barrier to implementing MBC (stated) and/or decreased engagement with consultation (inferred).

U

H

ST

F

CS

2.00

High

5

Digressions derail barrier problem solving and engagement

When clinicians are asked to articulate and prioritize perceived barriers to applying MBC, they frequently digress, resulting in other clinicians disengaging from the call (stated), worries about describing contextual constraints of their roles (stated), and uncertainty about quality of feedback that is contingent on their ability to adequately present information (stated).

U

H

ST

F

CS

2.00

Low

7

Unprepared to articulate updated monitoring targets

When prompted to articulate their plan to present updated monitoring targets to the student, clinicians feel put on the spot and question the quality of the feedback they are receiving, resulting in less confidence (stated) and unwillingness to participate in the call (inferred).

U

H

ST

F

CS

2.33

Low

2

No storage for barrier solutions

When articulating possible solutions to anticipated barriers, clinicians had no clear place to store their recorded solutions, decreasing the likelihood that they would be able to access the solutions at a later time (stated).

U

H

ST

F

CS

2.33

High

2

Regular calls incompatible with time/availability

The consultation call model expects clinicians to attend regular/scheduled calls, which clinicians find incompatible with their time and availability, which might lead to lower participation (inferred).

U

H

ST

F

CS

2.67

Medium

1

Case presentations exceed time allotted

During initial case presentations, stated call expectations that presentations are brief (i.e., 1–3 min) results in clinicians potentially exceeding the time allotted (stated), which might detract from other important topics of discussion (inferred).

U

H

ST

F

CS

3.00

Medium

5

Unfamiliar case update structure

When providing case updates on subsequent calls, the case presentation structure (i.e., providing rationale, positive intervention response, and next steps) may be unfamiliar and a deviation from the case presentation approaches clinicians are used to, resulting in wariness and a lack of confidence (stated).

U

H

ST

F

CS

3.00

Medium

4

Duration misaligned with preferences

The consultation call model may be too brief to align with clinicians' stated preferences needing longer overall duration of consultation, potentially leading to a sense of lack of confidence and support (inferred) to effectively implement MBC.

U

H

ST

F

CS

3.00

Low

3

No continued access to resources

Upon concluding live consultation, clinicians experienced concerns over the absence of continued access to resources (guidance, training, etc.) and peer discourse, which might result in feeling a lack of support or uncertainty in how to proceed with MBC (inferred).

U

H

ST

F

CS

3.33

Low

2

Discomfort with assessments in case presentations

During initial case presentations, clinicians experience potential discomfort presenting information from MBC assessments that they have not yet mastered, leading to less confidence in implementing MBC (inferred).

U

H

ST

F

CS

3.33

Low

4

Confusion over MBC terminology

When presenting the results of standardized assessments during initial case presentations, clinicians experience confusion over the terminology (established norms , clinically significant) that is foundation for MBC which could lead to less confidence in using MBC practices (inferred), disengagement from the calls (inferred), and interfere with accurate score interpretation for students on their caseloads (stated).

U

H

ST

F

CS

3.33

Medium

4

Confidentiality concerns when reporting results

When presenting the results of standardized assessments during initial case presentations, clinicians are concerned over privacy and confidentiality (stated), which may have a negative impact on their confidence and interest in participating in group calls (inferred).

U

H

ST

F

CS

3.33

Medium

1

Difficulty articulating what is being measured

When reporting on individualized goals (and not on standardized measures), clinicians struggle to articulate what they are measuring, which results in hesitation (inferred) and fear of reporting incorrectly (inferred).

U

H

ST

F

CS

3.33

High

1

Constraints on access to school buildings/students

When discussing solutions for addressing perceived barriers during late-stage consultation calls, clinicians from outside community mental health agencies have more constraints surrounding their access to school buildings and students, which results in more limitations (and less control) surrounding the execution of their identified solutions.

U

H

ST

F

CS

3.33

Medium

5

Distraction from multi-tasking online during calls

When navigating online training resources (i.e. the online message board) during the call, clinicians might get distracted or struggle to follow the call discussion, which might negatively impact group discourse or engagement (inferred).

U

H

ST

F

CS

3.33

Low

2

Unaware of available follow-up supports

When identifying the most potentially impactful barriers to MBC during late-stage calls, clinicians are unaware what follow-up/feedback is available after the call is finished, resulting in a sense of lack of support (inferred).

U

H

ST

F

CS

4.00

Medium

2

Technological difficulties are disengaging

When having technological difficulties (login, access to resources, etc.) on the consultation call, clinicians might feel distracted and disengaged them from the call discussion and be prevented from accessing necessary resources (inferred).

U

H

ST

F

CS

4.00

Low

3

Unfamiliar language in consultation model

Overall the consultation model uses language clinicians might experience as unfamiliar, confusing, and difficult to understand, which might 'alienate' clinicians (stated) or disengage them from participation (inferred).

U

H

ST

F

CS

  1. Complexity: refers to how straightforward (or not) it is to address an issue
  2. U: User access to knowledge/experience problem
  3. H: Hidden problem
  4. ST: Sequence and timing problem
  5. F: Feedback problem
  6. CS: Cognitive or social demands problem