Table 1 Rapid-cycle Systems Modeling: illustrative case study

RCSM Step 1: Identify the relevant stakeholder questions

Identify relevant stakeholders and the questions and priorities involved in the decisions they confront.

• Interviews

• Focus groups

• Surveys

• Observations

Sampling framework and documentation of stakeholder questions and priorities

• Goal: To identify questions and decisions confronted by mid-level managers when developing universal trauma screening and/or assessment protocols for youth entering foster care.

• Method: Sampling framework sought key informants who were engaged in decision-making processes regarding the provision of trauma-informed services for children in foster care. Approximately hour-long semi-structured interviews were conducted by telephone with 31 key informants across 12 states. Key informants were mid-level administrators from Medicaid, child welfare, and mental health agencies with roles developing policy for the provision of trauma-informed services for children in foster care. The interview guide was based on a decision sampling framework, with questions grounded in core domains of decision analysis. Respondents were sampled until no new themes in the core domains emerged (i.e., thematic saturation). Trained qualitative researchers (TM, AS, BF, ER) conducted interviews and analysis at their respective research institute. Research team engaged a modified framework analysis in DeDoose^TM, consisting of seven steps to identify and index the specific decisions relevant to the implementation of trauma-informed screening. The research protocol for the illustrative case study was reviewed and approved by the Institutional Review Board at [withheld to preserve anonymity]. Additional details of the methodological approach are previously published [citation with to preserve anonymity].

• Product: Decision set of five decision points (see Results).

RCSM Step 2: Develop Simulation Model

Develop and/or refine a simple simulation model to address the questions identified in Step 1.

• Monte-Carlo model

• Discrete-event model

• System dynamics

• Agent-based model

• Simulation model

• Evidence synthesis

• Goal: Refine a simulation model and conduct virtual experiments that address questions relevant to statewide implementation of trauma-informed screening

• Method: Adapted a Monte-Carlo model of a typical screening process (hereafter, the “baseline model”; see Fig. 1) from previous research [12]. Virtual experiments focused on the sensitivity of the overall process for moving children with trauma to treatment, the false positive rate, influence of screening on demand for services, workforce capacity to provide treatment, and the potential for waitlists if demand exceeds supply. To address stakeholders’ questions about merits of altering screening thresholds, sensitivity analyses focused on an increase in screening thresholds, which increases specificity but lowers sensitivity.

• Product: Simulation model synthesized evidence from systematic reviews; a slide deck and presentation detailed the Monte-Carlo model, analytic results, and results relevant to each question from step 1.

RCSM Step 3. Stakeholder engagement with iterated simulation model

• Assess relevance of model to stakeholder decisions

• Seek insight into question

• Discuss utility for informed decision-making in support of implementation

• Identify relevant stakeholders for iterative model development.

• Validation of qualitative data to search for disconfirming evidence, probe underlying assumptions

• Group dialog in service of inquiry into evidence and its application, critical discussion of competing hypotheses, and deliberation regarding best course of action

• Identification of alternative strategies

• Identification of potential barriers & mitigation plans

• Articulate hypotheses regarding key causal mechanisms

• Goal: To assess model relevance, seek insight regarding systemic factors likely to drive success, and discuss model utility to support implementation

• Method: To provide input, we sampled both intermediaries and a subset of key informants initially interviewed in Step 1. Trained qualitative researchers [TM, AS] convened four member-checking group interviews through an online platform with key informants (n = 8) of the 31 key informants engaged in Step 1 semi-structured interviews and two additional group interviews with “intermediaries” (n = 8) who developed, evaluated and/or provided technical assistance for mental health screening and trauma-specific interventions. The study team presented a standardized slide deck. Respondents were provided findings and asked questions after each section. Following the presentation of the simulation model, respondents were asked “Does this model seem applicable to your delivery system? If so, how?”, “How, if at all, would you want to change the model to accommodate your delivery system?” and “What would be the strengths or limitations of this model when applied to your delivery system?” Each member-checking group interview transcript was analyzed following completion. We used an immersion-crystallization approach in which two study team members listened to and read each group interview to identify important concepts and engaged open coding and memos to identify themes and disconfirming evidence. Additional details of the methodological approach are previously published [citation with to preserve anonymity].

• Product. Summary of utility and potential modifications to customize to decision-maker needs.