Integrated care is increasingly recognized as a crucial approach to delivering effective and patient-centered healthcare services. To support the development and implementation of integrated care models, reliable and valid assessment tools are essential. The SCIROCCO tool has emerged as a valuable instrument in this domain, designed to evaluate the maturity of integrated care within healthcare systems. This article delves into the measurement properties of the SCIROCCO tool, exploring its structural validity, internal consistency, and convergent validity. By understanding these properties, healthcare professionals and policymakers can confidently utilize the SCIROCCO tool to guide and improve their integrated care initiatives.
Understanding the Measurement Properties of the SCIROCCO Tool
When evaluating any assessment tool, it is critical to examine its measurement properties to ensure it accurately and reliably measures the intended construct. In the case of the SCIROCCO tool, this means determining how well it assesses the maturity of integrated care. The key measurement properties investigated in this study are structural validity, internal consistency, and convergent validity, all of which contribute to the overall construct validity of the tool. Construct validity is fundamental, particularly when a gold standard for measuring integrated care maturity is not available.
Structural Validity
Structural validity refers to the extent to which the scores obtained from the SCIROCCO tool accurately reflect the different dimensions or components of integrated care maturity. In simpler terms, it examines whether the tool is measuring what it is supposed to measure in terms of the underlying structure of integrated care. To assess structural validity, factor analysis is typically employed. This statistical technique helps to identify the underlying factors or dimensions that are being measured by the tool’s items. By examining the factor structure, we can gain insights into whether the SCIROCCO tool effectively captures the multifaceted nature of integrated care maturity.
Internal Consistency
Internal consistency is a measure of the homogeneity of the SCIROCCO tool. It assesses the degree to which the items within the tool are inter-related and measuring the same underlying construct. A high level of internal consistency indicates that the items in the SCIROCCO tool are consistently measuring different aspects of integrated care maturity and are not measuring unrelated concepts. This is an important aspect of reliability, ensuring that the tool provides stable and consistent measurements.
Convergent Validity
Convergent validity examines the extent to which the SCIROCCO tool correlates with other tools that are designed to measure similar constructs. In this study, convergent validity is assessed by comparing the SCIROCCO tool with the DMIC Quickscan, another instrument focused on integrated care development. If the SCIROCCO tool demonstrates good convergent validity, it means that it aligns with other established measures in the field, further supporting its validity as a tool for assessing integrated care maturity.
Methodology: Validating the SCIROCCO Tool
To rigorously evaluate the measurement properties of the SCIROCCO tool, a comprehensive study was conducted using a multi-stage data collection process and robust statistical analysis techniques.
Sample and Data Collection
Data for assessing structural validity and internal consistency were collected through an online survey using the SCIROCCO tool across three rounds between June 2017 and February 2018. Participants were carefully selected from European regions actively involved in integrated care initiatives. The recruitment criteria ensured a diverse sample, including individuals from various disciplines (decision-makers, healthcare professionals, IT specialists, regulators, payers, user groups, innovation agencies), sectors (healthcare, social care, housing, voluntary sector), and positions (senior management, front-line staff, back-office).
The recruitment process was conducted in three rounds to maximize participation and broaden the geographical representation. The first round targeted individuals from the five regions initially involved in the SCIROCCO project (Basque Country, Norrbotten, Puglia, Olomouc, and Scotland). The second round expanded to include participants from other relevant EU projects, leveraging dissemination activities within the SCIROCCO project network. The final round involved recruitment by researchers from Vrije Universiteit Brussel, reaching out to contacts in other European regions (Denmark, France, Germany, Netherlands, and United Kingdom) through convenience sampling. All potential participants received a detailed invitation email explaining the study’s purpose and procedure, along with an overview of the SCIROCCO tool and links to illustrative videos and demonstrations.
The DMIC Quickscan as a Comparator
To assess convergent validity, participants from the first round were also invited to complete the DMIC Quickscan within 6–24 weeks after completing the SCIROCCO tool. The DMIC Quickscan, derived from the comprehensive Development Model of Integrated Care (DMIC) questionnaire, is a shorter, 22-item instrument designed to assess the development of integrated care. The DMIC Quickscan was chosen as a comparator because it aligns with the dimensions of the B3-MM (another integrated care assessment framework) and covers a wide range of activities relevant to integrated care implementation, grouped into nine clusters: patient-centeredness, delivery system, performance management, quality of care, result-focused learning, interprofessional teamwork, roles and tasks, commitment, and transparent entrepreneurship.
The DMIC Quickscan is designed for healthcare professionals, managers, and integrated care coordinators to support improvement activities. It has a strong level of evidence for content validity and has been empirically validated in various healthcare settings. The DMIC Quickscan was deemed an appropriate comparator for the SCIROCCO tool because, while no identical instruments existed, it measures related constructs in integrated care. To ensure a high response rate and minimize participant burden, the DMIC Quickscan was preferred over the full DMIC questionnaire due to its shorter completion time (10 minutes versus 45 minutes). Participants rated the statements in the Quickscan on a 5-point scale, indicating their agreement with how well the statements matched their current integrated services/network situation.
Data Analysis Techniques
Quantitative data analysis was performed using IBM SPSS Statistics software to assess structural validity, internal consistency, and convergent validity.
Structural Validity Analysis
Exploratory Factor Analysis (EFA) was employed to examine the structural validity of the SCIROCCO tool. Given that the data did not meet the assumptions of conventional EFA methods (normality and equal interval scales), a robust approach was used. Specifically, the polychoric correlation matrix, suitable for ordinal data, was analyzed using the Minimum Residual (MINRES) method. MINRES is a robust factor extraction method that does not require distributional assumptions and is appropriate for smaller sample sizes.
To determine the optimal number of factors to extract, two established techniques were used in combination: Parallel Analysis (PA) and Comparative Data (CD). PA and CD are known for their accuracy in factor retention, even with ordinal data and smaller samples. PA was conducted using random column permutations, polychoric correlation, and the mean eigenvalue criterion, simulating 1000 datasets. CD utilized the Spearman rank order correlation matrix. Oblique rotation was selected as the rotation technique, assuming that the factors related to integrated care maturity would be correlated. A factor loading threshold of > 0.35 was applied. Prior to EFA, data suitability was assessed using Bartlett’s test for sphericity and the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy. Data were also screened for invalid patterns, skewness, and missing values. Items with extreme skewness (> 90% in one category) or high non-response (> 5% missing values) were considered for exclusion.
Internal Consistency Analysis
Internal consistency was assessed using both Cronbach’s alpha and ordinal alpha coefficients. While Cronbach’s alpha is traditionally used for continuous variables, it is often applied in practice with ordinal data, although it can be negatively biased. Ordinal alpha is recommended as a more appropriate measure for ordinal variables when normality assumptions are violated. Therefore, both measures were calculated to provide a comprehensive assessment of internal consistency for each identified factor.
Convergent Validity Analysis
Convergent validity was evaluated by examining the correlations between items of the SCIROCCO tool and corresponding items of the DMIC Quickscan. It was hypothesized that moderate positive correlations would be observed, reflecting the conceptual overlap between the two instruments, while acknowledging that they measure related but not identical constructs. Spearman’s ρ correlation coefficients were used to assess the agreement between the instruments, given the skewed data distribution. To provide a more robust indication of the statistical effect, bias-corrected accelerated (BCa) confidence intervals (CI, 95%) were computed using bootstrapping (1000 samples) for all correlations. This bootstrapping technique is recommended when parametric assumptions are not met. Correlation coefficients were interpreted as low (0.30–0.50), moderate (0.50–0.70), or high (0.70–0.90).
Expected Outcomes and Significance
This rigorous validation study is expected to provide evidence for the sound measurement properties of the Scirocco Tool For Integrated Care. Demonstrating strong structural validity, internal consistency, and convergent validity will solidify the SCIROCCO tool as a robust and trustworthy instrument for assessing integrated care maturity across diverse healthcare settings. The findings will have significant implications for researchers, practitioners, and policymakers seeking to effectively evaluate and advance integrated care initiatives. By utilizing a validated tool like SCIROCCO, stakeholders can gain valuable insights into the strengths and areas for improvement within their integrated care systems, ultimately leading to better patient outcomes and more efficient healthcare delivery.
References
[4] de Vet HCW, Terwee CB, Mokkink LB, Knol DJ. Measurement in Medicine: A Practical Guide. New York: Cambridge University Press; 2011.
[6] Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL, et al. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content. BMC Med Res Methodol. 2010;10:22.
[7] Carlson KD, Herdman AO. Understanding the impact of convergent validity on research results. Organ Res Methods. 2012;15:17–32.
[8] Minkman M, Ahaus K, Fabbricotti I, Nabitz U, Huijsman R. A quality management model for integrated care: results of a Delphi and concept mapping study. Int J Qual Heal Care. 2009;21:66–75.
[5] Grooten L, Vrijhoef HJM, Borgermans L. An instrument to measure maturity of integrated care: a first validation study. Int J Integr Care. 2018;18(1):10.
[9] Minkman M. The Development Model for Integrated Care: a validated tool for evaluation and development. J Integr Care. 2016;24:38–52 Available from: https://doi.org/10.1108/JICA-01-2016-0005 .
[10] Minkman MMN, Vermeulen RP, Ahaus KTB, Huijsman R. The implementation of integrated care: the empirical validation of the development model for integrated care. BMC Health Serv Res. 2011;11:177.
[11] Zonneveld N, Vat LE, Vlek H, Minkman MMN. The development of integrated diabetes care in the Netherlands: a multiplayer self-assessment analysis. BMC Health Serv Res. BMC Health Serv Res. 2017;17:1–9.
[12] Vat LE, Middelkoop I, Buijck BI, Minkman MMN. The development of integrated stroke Care in the Netherlands a benchmark study. Int J Integr Care. 2016;16:12.
[13] Longpré C, Dubois C-A. Implementation of integrated services networks in Quebec and nursing practice transformation: convergence or divergence? BMC Health Serv Res. 2015;15:84.
[14] Courtney MGR. Determining the number of factors to retain in EFA : using the SPSS R-menu v2 . 0 to make more judicious estimations. Pract Assessment Res Eval. 2013;18:1–14.
[15] Holgado-Tello FP, Chacón-Moscoso S, Barbero-García I, Vila-Abad E. Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables. Qual Quant. 2009;44:153–66.
[16] Jöreskog KG. Factor analysis by MINRES. Chicago Sci Softw Retrieved June. 2003;1:1–13.
[17] Horn JL. A rationale and test for the number of factors in factor analysis. Psychometrika. 1965;30:179–85.
[18] Ruscio J, Roche B. Determining the number of factors to retain in an exploratory factor analysis using comparison data of known factorial structure. Psychol Assess. 2012;24:282–92.
[19] Garrido LE, Abad FJ, Ponsoda V. A new look at horn’s parallel analysis with ordinal variables. Psychol Methods. 2013;18:454–74.
[20] Costello AB, Osborne JW. Best practices in exploratory factor analysis : four recommendations for getting the Most from your analysis. Pract Assessment Res Educ. 2005;10:1–9.
[21] Gaskin CJ, Happell B. On exploratory factor analysis: a review of recent evidence, an assessment of current practice, and recommendations for future use. Int J Nurs Stud. 2014;51:511–21.
[22] Basto M, Pereira JM. An SPSS R -menu for ordinal factor analysis. J Stat Softw. 2012;46:1–29.
[23] Thompson B. Exploratory and confirmatory factor analysis: understanding concepts and applications. Washington, DC: American Psychological Association; 2004. p. 2004.
[24] Zumbo BD, Gadermann AM, Zeisser C. Ordinal versions of coefficients alpha and Theta for Likert rating scales. J Mod Appl Stat Methods. 2007;6:21–9.
[25] Chalmers RP. On misconceptions and the limited usefulness of ordinal alpha. Educ Psychol Meas. 2017;78(6):1056–71.
[26] Hinkle DE, Wiersma W. Applied Statistics for the behavioral sciences. Boston: Houghton Mifflin; 2002.
[27] MacKinnon DP, Lockwood CM, Williams J. Confidence limits for the indirect effect: distribution of the product and resampling methods. Multivariate Behav Res. 2004;39:99–128.
[28] Chan W, Chan DW-L. Bootstrap standard error and confidence intervals for the correlation corrected for range restriction: a simulation study. Psychol Methods. 2004;9:369–85.
[29] Efron B, Tibshirani RJ. An Introduction to the Bootstrap. New York, NY Chapman & Hall. 1993.
