Phase | Step | Method | Purpose/adoption criteria |
---|---|---|---|
Phase 1: Item development | Step 1: Identification of domain and item | Choose an existing framework that has been validated. | Because CFIR (five domains, 38 constructs) is a meta-theoretical framework that integrates about 20 theories and models and has already been used and validated in many studies (over 2600 citations). |
Step 2: Content validity | Forward translation and reconciliation: 4 individuals translated, consulted, and integrated. Modify the contents to fit the context of Japanese health programs: 7 experts consulted. | To ensure the quality of the translation by adhering to the forward translation procedure by multiple people. To ensure the quality of item selection and regeneration by Japanese experts with experience of health program implementation. | |
Phase 2: Scale development | Step 3: Pre-testing | Pre-test with the target group | To ensure the questions and answers are meaningful. Adopt those with at least 70% agreement for each question of "easy to understand" and "think it is important”. |
Step 4: Survey administration and sample size | National survey of the target population | To ensure the quality of reliability and validity by calculating a sufficient sample size for the survey and distributing it evenly throughout the country. | |
Step 5: Item analysis | Ceiling/floor effect | To examine whether there is a scale attenuation effect depending on the degree of asymmetrically of the frequency distribution. Items with mean + 1SD < 5 or mean − 1SD > 0 are deleted. | |
Item-total correlation analysis | To check if any item in the set of tests is inconsistent with the averaged behavior of the others, and thus can be discarded. Item-total correlation coefficient ≥ 0.5 | ||
Phase 3: Scale evaluation | Step 6: Tests of reliability | Cronbach’s coefficient alpha | To measure the internal consistency of IDAS. Adopted at 0.8 or higher. |
Split-half correlations (odd/even) method | To check the verification of stability of IDAS. Adopted at the Spearman–Brown reliability coefficient 0.8 or higher. | ||
Step 7: Tests of validity | Construct validity | To verify the model fitness of construct validity with a confirmatory factor analysis. Check the goodness-of-fit index for fit in the same five domains as CFIR GFI, CFI ≥ 0.9 RMSEA < 0.08 | |
Criterion validity | To estimate the extent to which a test correlates with an established standard of comparison (RUC). Correlation coefficient ≥ 0.5 | ||
Differentiation by “Known Groups” | To examine if the concept measured behaves as expected in relation to “known groups”. (Four groups depending on the years of experience and whether the respondent was a supervising PHN or not) Significant difference between the groups (P < 0.05) |