Table 2 Five-step methodology with FFPSA illustrative case study

1

Primary: Determine which policy implementation phase (e.g., Exploration, Preparation, Implementation, Sustainment) is of interest and at which level (e.g., state, federal).

Case study definition: The current project aimed to support EBP adoption decisions at the state level in response to a federal policy, FFPSA.

Secondary: identify sites by their likely policy implementation stage, including whether potential sites will be adopting a policy.

Data sources/search techniques: Sites were identified by consulting a federal agency website (Children’s Bureau, an Office of the Administration for Children and Families) that reported whether states had submitted, received approval, or not yet submitted plans for federal approval to implement FFPSA. (Accessed 29 August 2022). Considering that there might be a lag in reporting on this federal website and that some states might be actively preparing their plans, state-specific websites were also searched by entering the policy and state name in a web search, or by searching the official website for the state agency likely to be responsible for implementing the policy (e.g., department of social services).

2

Identify who is involved in policy decisions during the focal implementation phase, as these individuals can become key actors or targets of implementation strategies.

Case study definition: The group of individuals involved in shaping the State Plan comprised “the room.” States were categorized as having enough information to identify the room and included in step 3, having insufficient information, or as being non-adopters of FFPSA.

Data sources/search techniques: Manually searched the official website for the state agency likely to be responsible for implementing the policy (e.g., department of social services) and reviewed any available documentation such as the submitted or approved State Plan and meeting minutes or presentation documents (e.g., Power Point).

Results: States were considered as being in Exploration if they had not submitted a plan but sufficient information was available to determine the state’s intention to submit (n = 1) or if they had a submitted (n = 15) or federally approved plan (n = 21). Some states had a singular “room” where all FFPSA decisions were made, while others had multiple “sub-rooms” (i.e., workgroups). For example, some states had workgroups focused on specific implementation activities such as identifying which evidence-based practices to adopt (e.g., “Prevention Subcommittee” with In-home parenting, Mental Health, and Substance Use Disorder Workgroups-Ohio; “Services Continuum”-Colorado) or “sub-rooms” comprised specific actors (e.g., “FFPSA Leadership Team, FFPSA Stakeholder Workgroups”-Maine; “Leadership Advisory Team”-North Carolina).

3

Characterize the organizations in the room to help describe actors’ roles and perspectives.

Results: Ten states detailed organizations involved in state prevention planning for FFPSA, with a range of 1–41 specified organizations per state. Two additional states broadly referenced organizations or types of organizations that were involved in decision-making but did not specify all organization names (for example, simply provided a descriptor such as “community providers, judiciary, other state agencies, private sector businesses”) or individuals associated with decision-making. Organization types included state and local government, non-governmental service agencies, advocacy or lived-experience, and research-oriented. Individuals with lived experience represented their own “organization” type and thus were counted as unique organizations.

The state’s child welfare, social service, or other human services agency responsible for that state’s child welfare cases was consistently present, whereas leadership from smaller jurisdictions from the relevant agency were inconsistently or incompletely present (e.g., some county or regional directors but not all).

Data sources/search/analysis techniques: Conducted web-searches for organization-specific websites to extract their mission statements or, in the absence of a mission statement, review what services are provided (often under a site tab such as “About us” or “What we do”). When available, mission statements were directly pasted into the codebook to facilitate reflection on why specific codes were created and/or applied. Often gathered organization charts at this point.

Potential analyses and alternative considerations: A range of qualitative methods could be applied, such as rapid qualitative analysis and content analysis, as well as more descriptive analyses (e.g., matrices with binary indicators by stated role type).

4

Identify the extent to which policy actors from step 3 might influence decision-making in subsequent policy implementation phases by characterizing roles, responsibilities, and actions.

Results: Eight states listed specific individuals involved in FFPSA planning and/or active implementation, ranging from 2 to 87 individuals per state. Individuals were coded into 13 categories distinguished by leadership level through their organizational role and their organizational affiliation. Roles and organizations could not be identified for 15 individuals. The majority of individuals (67%) were in leadership roles at their institution. These policy actors, in particular, could be important partners in the policy implementation phase. A relatively small group (4%) of individuals brought a lived experience perspective. All but four of these individuals seemed to be invited to the room through their organization, such as employment or volunteering through a community advocacy organization. These individuals were from the same state. In reality, input from those with lived experience might have been much higher, as most states described hosting public listening sessions such as town halls. Because these individuals were not listed as part of the room and did not have decision-making authority regarding policy implementation, they were excluded from frequency counts (step 5).

Data sources/search techniques: Similar to step 3, organization-specific websites were searched, as were professional networking sites such as LinkedIn. Individual roles and/or organizational affiliations were sometimes most easily identified through reports released by their respective agency in which the report was prepared or signed by the individual. Local news articles or organization newsletters also provided important details, particularly when an individual was no longer in the role that they assumed when the policy room was originally formed.

Potential analyses and alternative considerations: Crable et al. (2022) [12] provide further recommendations for specifying and reporting how policy actors’ roles, responsibilities, and actions might vary throughout policy implementation. Presseau et al. (2019) [45] provide a framework for specifying actions. Systems science methods such as agent-based modeling and causal loop diagramming could aid hypothesis generation and simulation-based testing of these hypotheses, such as potential engagement and implementation or dissemination strategy specification choice impacts on implementation and policy outcomes.

5

Identify which perspectives or interests might be more saturated than others.

Results: There was extensive heterogeneity in the size, composition, and structural processes of each state’s room. Some states including several branches of state government, representing a potentially broader, more comprehensive implementation compare to states that only included the lead organization.

Data sources/search techniques: This stage did not require new data sources. However, it did often require revisiting sources from other stages to interpret organization and professional roles within the context of one another.

Potential analyses and alternative considerations: Descriptive statistics (e.g., tallies), rapid qualitative analysis, landscape analysis, audience segmentation, social network analysis.