Towards the clinical integration of digital health metrics: a data-driven framework enabling their automated selection and validation

Like Comment
Read the paper

The rating of patients’ impairments, for example in the arm and hand after a neurological injury, through experienced clinical personnel is a fundamental pillar of the global healthcare system. Such information is essential to guide the definition of a therapy plan and evaluate its efficacy. However, the established rating process leaves room for improvement, as its subjectivity implies that different assessors can reach varying rating results regarding the type and severity of the impairments. In addition, the employed rating scales are often coarse, provide only few rating options with a lower and upper limit (e.g., severe, moderate, mild impairment), thereby challenging the detection of subtle changes.

Digital health metrics promise to address these limitations by providing an objective and sensitive characterization of impairments as complementary information for clinical personnel to support evidence-based decision making. Such metrics offer high resolution, enabling the evaluation of subtle changes in impairment severity. They can be extracted by applying multiple mathematical processing steps to health-related sensor data, for example to describe hand movement and grip force control.

Despite the great potential of digital health metrics, their clinical integration is still scarce as a result of two key factors: first, researchers who intend to develop digital health metrics are confronted with the demanding task of selecting the metrics to extract from their health-related sensor data. This is challenging, as a heterogeneous landscape of metrics with varying physiological interpretation and statistical properties (e.g., reliability) exists. In the example of analyzing movement data in neurological injuries, over 150 metrics have been proposed [1]. Second, clinical personnel expect that the rating scales that guide their clinical decisions have been thoroughly validated from a statistical point of view. This is often not the case for digital health metrics, as typically no accurate ground truth about the targeted impairment exists, which challenges the methodological implementation of the validation process.

Figure 1 Data-driven framework for the selection and validation of digital healh metrics. The framework relies on an initial set of digital health metrics collected with a digital assessment task for a reference (e.g., unaffected controls) and a target (e.g., neurological subjects) population. Five validation and selection steps are implemented that evaluate the metrics based on clinically-relevant statistical concepts, including their clinimetric properties. Throughout these steps, metrics of low statistical quality are discarded, leading to a final set of validated core metrics.

In our work presented in npj Digital Medicine [2], we address these two challenges by proposing a data-driven framework that automatizes the selection and validation of digital health metrics based on clinically-accepted statistical properties. The approach starts with clear physiological hypotheses motivating an initial set of digital health metrics. Through multiple validation steps, metrics that do not fulfill all relevant statistical properties according to clinical standards are discarded, yielding a reduced set of validated application-specific metrics (Figure 1).

Figure 2 Virtual Peg Insertion Test (VPIT). The technology-aided assessment of upper limb sensorimotor impairments was implemented with the VPIT. This approach features a virtual-reality based task requiring goal-directed object manipulations. It allows recording movement and grip force data through a haptic device and an instrumented handle, respectively.

As an exemplary use-case, this framework was applied to 77 digital health metrics gathered in 120 neurologically intact and 89 affected individuals with a technology-aided assessment of upper limb sensorimotor impairments (Figure 2). This allowed establishing a set of 10 validated digital health metrics sensitively characterizing sensorimotor impairments in arm and hand (Figure 3). These metrics provided additional clinical value by detecting impairments in neurological subjects that did not show any deficits according to conventional scales.

Figure 3 Exemplary VPIT data for a control and a neurological subject. On the left side, position and grip force (color coded: increasing brightness indicates increasing grip force) data from the VPIT is shown for one representative control (top) and neurological subject (bottom). The application of the data-driven metric selection framework allowed identifying a core set of 10 validated metrics. On the right side, these metrics were used to characterize the sensorimotor impairments of the two exemplary individuals. Each pie segment represents one of the metrics, and increasing segment sizes indicate decreasing task performance. This underlines that the VPIT allows capturing abnormalities in movement and grip force patterns, and the metrics enable an objective and fine-grained representation of sensorimotor impairments in arm and hand.

Given that the proposed framework (source code) is based on a data-driven, application-dependent evaluation of clinically-accepted statistical criteria, it overcomes the caveats of expert groups providing recommendations about the choice of metrics, as these typically cannot account for dependency of the metrics’ statistical properties to each application. Moreover, the transparent and clinically-motivated framework complements existing data-driven machine learning algorithms, which are often black-box approaches that do not consider all clinically-accepted statistical criteria.

Hence, this work makes an important contribution to the implementation of digital health metrics as complementary endpoints for clinical trials and routine. We urge researchers and clinicians to capitalize on the promising properties of digital health metrics and contribute to their validation and clinical integration, which in the long-term will lead to a more thorough understanding of disease mechanisms and enable novel therapeutic approaches with the potential to improve healthcare quality.

[1] Schwarz, A., Kanzler, C.M., Lambercy, O., Luft, A.R., & Veerbeek, J. M. (2019). Systematic Review on Kinematic Assessments of Upper Limb Movements After Stroke. Stroke, 50(3), 718–727. https://doi.org/10.1161/STROKEAHA.118.023531

[2] Kanzler, C.M., Rinderknecht, M.D., Schwarz, A., Lamers, I., Cynthia, G., Held, J.P.O., Feys, P., Luft, A.R., Gassert, R., & Lambercy, O (2020). A data-driven framework for selecting and validating digital health metrics: use-case in neurological sensorimotor impairments. npj Digital Medicine, 3, 80. https://doi.org/10.1038/s41746-020-0286-7

Acknowledgements:

This project received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. 688857 (SoftPro). The authors would like to thank Stefan Schneller for support in the graphics design.

Christoph M. Kanzler

PhD Candidate, ETH Zurich

No comments yet.