ASSIST – Assessment and evaluation tools for Telemedicine

The holistic goal of this study is the development and validation of a comprehensive methodology to evaluate outcomes and impacts generated by ESA-sponsored projects related to the provision of sustainable telemedicine services. The ultimate goal is to provide clear, relevant and impartial information to be used for evidence-based decision making concerning the migration of pilot services into real, operational undertakings.

The evaluation of health information services has to deal with a multitude of heterogeneous variables, such as actors (the people), artefacts (the technology), and the environment in which they are implemented as well as with their interactions and most importantly, their outputs and outcomes. Telemedicine has medical, technical, psycho-social, organisational, business and societal aspects. Assessing telemedicine means to incorporate all these aspects. Research shows that there is a huge bandwidth of evaluation methods, but many - if not most - of them are not very suitable to deal with a comprehensive, holistic evaluation of a telemedicine service. It should be emphasised that a telemedicine system is only part of this solution, whereas the telemedicine service also contains the whole delivery chain and all stakeholders. In addition, it is different from project evaluation which is measuresing the achievement of project objectives, whereas ASSIST is focusing on the impact achieved or achievable.

Selection criteria and critical factors to be considered in this context are the extent to which methods allow for:

• Benefits and costs estimation
• Provisions for testing rigour
• Dealing with objectivity issues and bias in measurements and estimations
• Sustainability analysis: ex-ante and formative assessment, business modelling, affordability considerations, risk assessment
• Multiple stakeholders to be considered
• Health system and operational environment variables to be taken into account
• Time dimensions (from planning to routine services)
• Validity, reliability and comparability of measures and options Conciseness of model
• Applicability to telemedicine / space technology

None of the methods available as of today appears to be readily suitable for telemedicine assessment. All lack some elements, sub-methods or techniques required for the purpose at hand. The starting point for the tools developed in the ASSIST project is the EHR Impact cost-benefit analysis approach and method, which provides the most advanced, and differentiated tools and techniques for data gathering and analysis. ASSIST will further develop and extend it with additional techniques to fit the project goals. ASSIST tools will allow a telemedicine project to self-evaluate their project impacts in a multi-dimensional way. It will support the evaluator in finding the data needed and analysing them. A small number of outcome measures allow to easily understanding the relation of benefits and cost. Pie charts allow assessing the distribution of cost and benefits over different stakeholder groups, identifying benefits shifts and the distribution between financial and non-financial benefits. Project leads will receive projections of results which allow them to sketch out scenarios of future use.
The project team will always take the perspective of stakeholders and decision makers, and look at the added value they may receive.

The study plan includes the following parts:

1. A critical review of the existing, consolidated and comprehensive methodologies established so far for evaluating telemedicine projects.
2. The elaboration of a set of adequate methods that can be adopted to carry out a systematic evaluation of the results of such projects within a multi-stakeholder perspective.
3. The development of an evaluation model for the results of such projects, based on the selected assessment framework and its methods, and composed of a set of guidelines, procedures and an associated supporting toolset.
4. The application of the evaluation model to a number of space-based telemedicine projects for validating the model, with a subsequent critical revision of the model.
5. The delivery, presentation and promotion of the final evaluation model, and dissemination of the associated results.