Evaluation
From Digipedia
| How useful is this article to you? 1 (not useful) - 5 (very useful) |
| Current average rating: 0 |
| Please login or create an account to rate this article |
Contents |
OVERVIEW
The assessment of how well used your digital content is and how satisfied your users feel are essential steps in the digital content life cycle. Evaluation is an activity which informs development of a service and helps its managers to understand the value and impact the service is having on its users, answering the "What difference has it made?" question.
Introduction
Evaluation helps with quality assessment, benchmarking, business planning and understanding users' needs. It should be an activity that is built in to all phases of a project and is a regular activity for established services. It is an essential component of audience research The Evaluation Trust defines evaluation as "the assessing and judging the value of a piece of work, an organisation or a service. Its main purpose is to help an organisation reflect on what it is trying to achieve, assessing how far it is succeeding, and identify required changes"
There are two main evaluation activities:
- Formative evaluation which is used throughout the life of a project to assess progress at key points so that changes can be made to next steps in the light of the findings, if necessary
- Summative evaluation which looks at the impact of the entirety of the project and informs future developments.
Both use two main methods:
- Quantitative evaluation which measures the use of the content in various ways
- Qualitative evaluation which looks at the impact on users of the service
Quantitative evaluation
Getting a feel for the amount of use your content is getting, which parts of the service are most used and where users are coming from will help you build an overall picture of the use of the service. Figures cannot help you understand whether what you are doing is the right thing, but if use is as anticipated, repeat use is taking place regularly and users are coming from the kind of starting points you expect, then you are probably on the right track. Measuring the use of your digital content will involve using a number of techniques including:
- Web metrics are intended to measure activity on web sites. Activity might include measurements of where users come from (i.e. the domain and country), how often they visit, and the use and effectiveness of advertising on the website.
- Web analytics, defined by the Web Analytics Association as "the measurement, collection, analysis and reporting of Internet data for the purposes of understanding and optimizing Web usage".
- Log file analysis of a web server log which maintains a history of page requests. Information about the page request, including client IP address, request date/time, page requested, HTTP code, bytes served, user agent, and referer are typically added. These data can be combined into a single file, or separated into distinct logs, such as an access log, error log, or referer log. However, server logs typically do not collect user-specific information. The W3C maintains a standard format for web server log files, but other proprietary formats exist. Statistical analysis of the server log can be used to look at traffic patterns.
- Bibliometrics, used in the academic sector to analyse the patterns of URL citations in scholarly publications.
Measurements taken by these techniques are usually of two main types:
- User sessions: The number of distinct user sessions chalked up over the reporting period. User sessions are determined by grouping together all requests that come from the same IP address within a time interval of no less than 30 minutes between each request. A figure of 30 minutes is widely used and is the default in many Web analysis packages. Note that this does not give you the number of unique users to the site as it will include repeat visits from the same IP address.
- Page impressions: the total number of requests for files that are defined as pages. Generally files that have extensions .htm, .html, .shtml, .php, .asp, .pl, .cgi and so forth. The exact set may differ amongst projects, projects will be expected to set up their analysis packages so that all page-type files are measured. Note that this will not include images, graphics, stylesheets, external script files or other "component" files that together comprise one page.
There are many software tools such as Webtrends or Google analytics which can be installed to carry out this kind of statistical analysis.
Qualitative evaluation
Understanding the impact of your content on its users is essential to planning for the future. Qualitative evaluation as you build your content will help fine tune the end product and continuing qualitative evaluation exercises regularly will make sure that you have a good range of feedback which will inform service development. A number of techniques can be used:
- Structured interviews are a useful way of understanding impact across a wide range of people including users, funders, project staff, subject specialists and other stakeholders. A structured interview means asking everyone the same questions so that responses can be compared on a like for like basis and a benchmark derived.
- Focus groups are a mechanism for establishing the views of a controlled group of people and can often provide valuable ideas on different ways of doing things. These groups will need time to work with, but as a captive think-tank, they are a useful sounding board.
- User feedback to encourage people to tell you what they think of the service or to make suggestions for improvements. An email facility on the website is usually the way to do this. Feedback may be broad in its scope, and each response will need to be considered on its own merits, but if the same feedback is received from a large number of users, then that would suggest some action should be taken promptly.
- Online surveys are able to establish factual information about users such as location, gender, age, and, with careful multiple choice structuring, can provide information about the use and usefulness of the content to the respondant.
In order to make sure results are reliable, it is always good to use a range of measures and check each against the other. This process is called triangulation.
Results should then give outcomes and impact of service.
Case studies
Strategic Content Alliance Audience Research Case Studies
Oxford Internet Institute
the Oxford Internet Institute Digitised Resources: A Usage and Impact Study. Funded by JISC, this project is developing a best practice toolkit for the assessment of the impact of digitisation projects.
Tavistock Institute
Books and bytes. This evaluation of the £120M Lottery-funded People's Network ICT learning centres in UK public libraries and associated ICT training programme for public library staff was undertaken by the Tavistock Institute. It demosntrates the use of a range of evaluation techniques to assess impact and suggest future developments.
Related Digipedia articles
Audience analysis and modelling
Further information
eVALUEd toolkit. eVALUEd was set up to develop a transferable model for e-library evaluation in higher education and to provide dissemination and training in e-library evaluation.
UK Evaluation Society. A professional membership organisation that exists to promote and improve the theory, practice, understanding and use of evaluation. Its website offers advice and guidance and onward links to other resources.
The Evaluation Trust. The Trust is a community development agency with skills in capacity building. It Works closely with voluntary and community organisations to provide advice, information and ideas on ways to improve performance and best meet the needs of the people who use the services provided.
Usage statistics for websites This document was produced by the JISC-funded QA Focus project provided by UKOLN and AHDS
UKOLN Good practice guide section on performance indicators
[[role::newcomer]] [[role::strategy manager]] [[role::policy maker]] [[role::project manager]] [[goal::evaluating]] [[goal::audience]] [[level::basic]] [[level::medium]]
