definition of evaluation by different authors

It has been suggested that a major problem in arriving at a definition of evaluation is confusion with related terms such as measurement, One notable definition is provided by Scriven (1991) and later adopted by the American Evaluation Association (): "Evaluation is the systematic process to determine merit, worth, value, or . It is possible to incorporate both metrics and narratives within systems, for example, within the Research Outcomes System and Researchfish, currently used by several of the UK research councils to allow impacts to be recorded; although recording narratives has the advantage of allowing some context to be documented, it may make the evidence less flexible for use by different stakeholder groups (which include government, funding bodies, research assessment agencies, research providers, and user communities) for whom the purpose of analysis may vary (Davies et al. It is now possible to use data-mining tools to extract specific data from narratives or unstructured data (Mugabushaka and Papazoglou 2012). If metrics are available as impact evidence, they should, where possible, also capture any baseline or control data. 0000002318 00000 n Describe and use several methods for finding previous research on a particular research idea or question. Any information on the context of the data will be valuable to understanding the degree to which impact has taken place. 0000002109 00000 n Organizations may be interested in reviewing and assessing research impact for one or more of the aforementioned purposes and this will influence the way in which evaluation is approached. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide, This PDF is available to Subscribers Only. Concerns over how to attribute impacts have been raised many times (The Allen Consulting Group 2005; Duryea et al. 0000328114 00000 n Assessment is the process of gathering and discussing information from multiple and diverse sources in order to develop a deep understanding of what students know, understand, and can do with their knowledge as a result of their educational experiences; the process culminates when assessment results are used to improve subsequent learning. Also called evaluative writing, evaluative essay or report, and critical evaluation essay . Evaluation is a process which is continuous as well as comprehensive and involves all the tasks of education and not merely tests, measurements, and examination. The point at which assessment takes place will therefore influence the degree and significance of that impact. 2008), developed during the mid-1990s by Buxton and Hanney, working at Brunel University. To adequately capture interactions taking place between researchers, institutions, and stakeholders, the introduction of tools to enable this would be very valuable. %PDF-1.4 % In designing systems and tools for collating data related to impact, it is important to consider who will populate the database and ensure that the time and capability required for capture of information is considered. What are the methodologies and frameworks that have been employed globally to assess research impact and how do these compare? The reasoning behind the move towards assessing research impact is undoubtedly complex, involving both political and socio-economic factors, but, nevertheless, we can differentiate between four primary purposes. The Social Return on Investment (SROI) guide (The SROI Network 2012) suggests that The language varies impact, returns, benefits, value but the questions around what sort of difference and how much of a difference we are making are the same. We will focus attention towards generating results that enable boxes to be ticked rather than delivering real value for money and innovative research. As Donovan (2011) comments, Impact is a strong weapon for making an evidence based case to governments for enhanced research support. Cb)5. This is being done for collation of academic impact and outputs, for example, Research Portfolio Online Reporting Tools, which uses PubMed and text mining to cluster research projects, and STAR Metrics in the US, which uses administrative records and research outputs and is also being implemented by the ERC using data in the public domain (Mugabushaka and Papazoglou 2012). Dennis Atsu Dake. Case studies are ideal for showcasing impact, but should they be used to critically evaluate impact? Clearly the impact of thalidomide would have been viewed very differently in the 1950s compared with the 1960s or today. Collating the evidence and indicators of impact is a significant task that is being undertaken within universities and institutions globally. Assessment refers to a related series of measures used to determine a complex attribute of an individual or group of individuals. % A discussion on the benefits and drawbacks of a range of evaluation tools (bibliometrics, economic rate of return, peer review, case study, logic modelling, and benchmarking) can be found in the article by Grant (2006). Incorporating assessment of the wider socio-economic impact began using metrics-based indicators such as Intellectual Property registered and commercial income generated (Australian Research Council 2008). 1.3. 2007; Nason et al. Attempting to evaluate impact to justify expenditure, showcase our work, and inform future funding decisions will only prove to be a valuable use of time and resources if we can take measures to ensure that assessment attempts will not ultimately have a negative influence on the impact of our research. What is the Difference between Formative and Summative Evaluation through Example? Authors from Asia, Europe, and Latin America provide a series of in-depth investigations into how concepts of . One of the advantages of this method is that less input is required compared with capturing the full route from research to impact. It is therefore in an institutions interest to have a process by which all the necessary information is captured to enable a story to be developed in the absence of a researcher who may have left the employment of the institution. Enhancing Impact. Where narratives are used in conjunction with metrics, a complete picture of impact can be developed, again from a particular perspective but with the evidence available to corroborate the claims made. Productive interactions, which can perhaps be viewed as instances of knowledge exchange, are widely valued and supported internationally as mechanisms for enabling impact and are often supported financially for example by Canadas Social Sciences and Humanities Research Council, which aims to support knowledge exchange (financially) with a view to enabling long-term impact. 2009; Russell Group 2009). Two areas of research impact health and biomedical sciences and the social sciences have received particular attention in the literature by comparison with, for example, the arts. 0000001883 00000 n Inform funding. Table 1 summarizes some of the advantages and disadvantages of the case study approach. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. 0000006922 00000 n In many instances, controls are not feasible as we cannot look at what impact would have occurred if a piece of research had not taken place; however, indications of the picture before and after impact are valuable and worth collecting for impact that can be predicted. working paper). Differences between these two assessments include the removal of indicators of esteem and the addition of assessment of socio-economic research impact. 2010). Oxford University Press is a department of the University of Oxford. It can be seen from the panel guidance produced by HEFCE to illustrate impacts and evidence that it is expected that impact and evidence will vary according to discipline (REF2014 2012). Capturing data, interactions, and indicators as they emerge increases the chance of capturing all relevant information and tools to enable researchers to capture much of this would be valuable. To be considered for inclusion within the REF, impact must be underpinned by research that took place between 1 January 1993 and 31 December 2013, with impact occurring during an assessment window from 1 January 2008 to 31 July 2013. This might include the citation of a piece of research in policy documents or reference to a piece of research being cited within the media. It is worth considering the degree to which indicators are defined and provide broader definitions with greater flexibility. Every piece of research results in a unique tapestry of impact and despite the MICE taxonomy having more than 100 indicators, it was found that these did not suffice. From an international perspective, this represents a step change in the comprehensive nature to which impact will be assessed within universities and research institutes, incorporating impact from across all research disciplines. We suggest that developing systems that focus on recording impact information alone will not provide all that is required to link research to ensuing events and impacts, systems require the capacity to capture any interactions between researchers, the institution, and external stakeholders and link these with research findings and outputs or interim impacts to provide a network of data. 0000346296 00000 n The case study does present evidence from a particular perspective and may need to be adapted for use with different stakeholders. 5. HEFCE indicated that impact should merit a 25% weighting within the REF (REF2014 2011b); however, this has been reduced for the 2014 REF to 20%, perhaps as a result of feedback and lobbying, for example, from the Russell Group and Million + group of Universities who called for impact to count for 15% (Russell Group 2009; Jump 2011) and following guidance from the expert panels undertaking the pilot exercise who suggested that during the 2014 REF, impact assessment would be in a developmental phase and that a lower weighting for impact would be appropriate with the expectation that this would be increased in subsequent assessments (REF2014 2010). A comprehensive assessment of impact itself is not undertaken with SIAMPI, which make it a less-suitable method where showcasing the benefits of research is desirable or where this justification of funding based on impact is required. The book also explores how different aspects of citizenship, such as attitudes towards diverse population groups and concerns for social issues, relate to classical definitions of norm-based citizenship from the political sciences. 0000001862 00000 n It is perhaps worth noting that the expert panels, who assessed the pilot exercise for the REF, commented that the evidence provided by research institutes to demonstrate impact were a unique collection. This is particularly recognized in the development of new government policy where findings can influence policy debate and policy change, without recognition of the contributing research (Davies et al. << /Length 5 0 R /Filter /FlateDecode >> Published by Oxford University Press. In the UK, evidence and research impacts will be assessed for the REF within research disciplines. Accountability. Standard approaches actively used in programme evaluation such as surveys, case studies, bibliometrics, econometrics and statistical analyses, content analysis, and expert judgment are each considered by some (Vonortas and Link, 2012) to have shortcomings when used to measure impacts. 0000002868 00000 n The definition of health is not just a theoretical issue, because it has many implications for practice, policy, and health services. Findings from a Research Impact Pilot, Institutional Strategies for Capturing Socio-Economic Impact of Research, Journal of Higher Education Policy and Management, Introducing Productive Interactions in Social Impact Assessment, Measuring the Impact of Publicly Funded Research, Department of Education, Science and Training, Statement on the Research Excellence Framework Proposals, Handbook on the Theory and Practice of Program Evaluation, Policy and Practice Impacts of Research Funded by the Economic Social Research Council. The Goldsmith report concluded that general categories of evidence would be more useful such that indicators could encompass dissemination and circulation, re-use and influence, collaboration and boundary work, and innovation and invention. 4 0 obj The development of tools and systems for assisting with impact evaluation would be very valuable. In line with its mandate to support better evaluation, EvalNet is committed to working with partners in the global evaluation community to address these concerns, and is currently exploring options for additional work. (2005), Wooding et al. Clearly there is the possibility that the potential new drug will fail at any one of these phases but each phase can be classed as an interim impact of the original discovery work on route to the delivery of health benefits, but the time at which an impact assessment takes place will influence the degree of impact that has taken place. They are often written with a reader from a particular stakeholder group in mind and will present a view of impact from a particular perspective.

Trustee Delegation Of Authority Form, Matamoros, Mexico Crime, Articles D