Self-Evaluation Process

How good is our public library service has a number of important elements:

(i) emphasis on the self-evaluation process, with the main aim being a mechanism for improvement in public library services;

(ii) staff work together to plan the process and gather evidence and data, ensuring all staff and stakeholders are successfully engaged;

(iii) external peer reviewers, acting as ‘critical friends’, visit to review the self-evaluation report, and discuss it with staff. 

Reviews using the framework of How good is our public library service ensure consistency in self-evaluation across authorities. This peer review process provides a mechanism for identifying and sharing good practice between libraries, while also maintaining and updating the benchmark statements.

In planning for the self-evaluation process, library services can choose which of the Quality Indicators to address, fitting in with individual circumstances, plans and priorities within each library service. Managers should aim for participation by different levels of staff across the service for the self-evaluation process. The team involved with each Quality Indicator should ideally represent a cross section of management and library staff to help to ensure a shared understanding and ownership of strengths, areas for improvement and impact of the library service. 

It is useful for the team to ask the “so what?” question regularly as they gather evidence and draft the report. This will strengthen the self-evaluation and assist identifying sources of evidence to demonstrate value and impact. Self-evaluation should be viewed as a positive diagnostic tool for a service.

RADAR© can be particularly helpful here. At its highest level, the RADAR logic states that an organisation needs to:

• Determine the Results it is aiming to achieve as part of its strategy;

• Have in place an Approach that will deliver the required result, both now and in the future;

• Deploy this approach appropriately;

• Assess and Refine the deployed approach to learn and improve.

*RADAR tool from the EQFM Model, Revised 2^nd Edition (www.efqm.org).

The Self-Evaluation Report is the document that library services will use to bring together all the information and outcomes of the self-evaluation process on the quality indicator in question. It is the document which will be sent to the peer reviewers ahead of the visit. 

The Self-Evaluation Report should be a reflective and evaluative document. It should be concise and succinct, not being longer than eight pages. It should also be capable of acting as a working document for library services to record planning, progress and outcomes. The ultimate aim being that this allows for a more complete picture of the process undertaken and outcomes achieved when the peer-review panel add their final report to produce the overall assessment. The use of hyperlinks within the text is encouraged to enable other supporting documents to be viewed remotely; these can be appended in an additional list of references, not exceeding two pages in length. 

Self-evaluation needs to be based on the collection and review of the evidence of outcomes and impacts. The Quality Indicators provide a number of key evaluation themes, and public library services are asked to identify ways in which they know about and can demonstrate how they are performing. Assessments should be made about the public library service’s performance based on the evidence.  

Evidence helps to support self-evaluation processes by demonstrating proof of activity and impact. It can help in establishing how well the service is meeting users’ needs. Evidence can be quantitative or qualitative. Quantitative evidence records what can be measured, for example numbers of registered learners. Qualitative evidence will seek to draw out the value which users may put on services, and this is often unstructured in format, for example feedback from users, or evidence of successful learners moving on to other education or employment opportunities.

These sources of evidence are complementary. For example, a policy statement seeking to “promote learning for all” can be observed across the range of learning opportunities offered, and backed up by feedback from learners. Examples of sources of evidence are suggested in the Quality Indicators for each theme, and some more general examples are included here (they need not all be used and equally the list is not exhaustive):

Consult users, non-users and staff

• Feedback, oral or written, from discussions with individuals and staff about specific activities;
• Formal discussion with individuals, non-users, elected members, other council officers and library staff, possibly as part of community engagement activities forming part of the local Community Planning process;
• Surveys and questionnaires run nationally or locally. 

Observe activities

• Examine the use of the library;
• Review the range of resources and services available;
• Observe individuals and staff across a range of activities;
• Review the library environment. 

Consider measurable outcomes

• Analyse quantitative evidence from sources such as statutory performance indicators, CIPFA, surveys and the library management system. 

Examine documentation and resources

• Local authority strategic/corporate plan;
• Community Learning and Development Strategy;
• The Community Plan and other Community Planning information relevant to library services;
• Public library service objectives and improvement plan;
• Event programmes for range of individuals;
• Lifelong learning programmes and materials;
• Range of resources in relation to community diversity profiles;
• Public information documentation.
• Marketing and publicity strategies and documentation;
• Press coverage;
• Staff training programme;
• Evidence of library staff involvement in internal and external partnerships;
• Evaluation of externally-funded programmes;

Photographs, audio and video clips of library activities are also useful to demonstrate particular strengths.

In looking at evidence, all of the RADAR© Elements should be considered to understand what aspects of performance are working well and where there are opportunities for improvement.

How will you know if what you are doing, or planning to do, in the public library service will make a difference for individuals and groups in the community? We can describe, using the various ‘inputs’ and ‘outputs’, what the library does: the size of the adult fiction collection, the number of hours open, how many borrowers it has, the number of books issued to children, the various outreach programmes it is involved with and so on, but then we have to ask the question ‘so what?’. How do you demonstrate the value that the various activities and services provided by the library service have for individuals and communities served? This is where impact comes in, and involves the planning and gathering of evidence to demonstrate the value and impact of library services.

In gathering evidence, library services need to act strategically and the evidence may be both qualitative and quantitative. Sophisticated quantitative data and good quality impact information should complement each other as they have different roles and purposes; the quantitative data can reassure stakeholders, but this, presented alongside the qualitative evidence, can presented a broader picture and demonstrate value and impact very effectively. It is recognised that gathering evidence to demonstrate impact is neither an easy nor simple task.

One significant aspect of undertaking a self-evaluation process such as How good is our public library service is to identify areas for improvement, and also to share good practice with colleagues and other library services. This process can also assist staff when considering difficult decisions such as ‘should we really be doing this piece of work with this partner’ or ‘will this activity really make a difference’. Impact and value are not necessarily monetary (indeed they frequently are not) but rather about the difference they make to individuals or to communities. 

When the evidence gathering stage is completed (this may however be ongoing if the self-evaluation process is truly embedded in library service planning) the drafting of the Self-Evaluation Report should take place. The Self-Evaluation Report is a free text document of no longer than eight pages per Quality Indicator that seeks to draw everything together. 

The report should aim to document current performance, what difference it makes (the impact), and identify where improvements can be made. The report will recognise key strengths and identify ways in which improvements can be made. From this, an overall and realistic grade should be given by the library service. 

One of the key benefits of, and important outcomes from How good is our public library service is the sharing of good practice about what works well in public library services. As part of the self-evaluation process, one way of ensuring good practice is captured is for library services to offer up examples of good practice, a snapshot or small case study, as part of the self-evaluation process. Such examples of good practice can be shared and disseminated later. 

Encouragement to also share these experiences at events such as conference and other meetings would take this a step further, giving the teams involved more opportunities to showcase what works well in their library service, while also providing valuable opportunities for staff development.

How good is our public library service is primarily a self-evaluation process supported by a system of peer review. The combination of these two elements is widely considered to be appropriate and robust for an effective quality standards mechanism; the approach is also embedded in a range of other public sector quality improvement mechanisms.

Peer-review is the solid basis for driving continuous improvement and enhancement. Peer-review within the How good is our public library service process exists to ensure consistency, credibility, transparency and externalisation. Those involved in peer reviewing bring with them the benefits of a wider perspective, but with enough pragmatism to understand situations ‘on-the-ground’.

The role of the peer reviewers is to consider the self-evaluation report and the levels allocated by the library service. Three peer reviewers consider the self-evaluation report, visit the library service to meet with staff and users, examine the evidence, and ensure the levels allocated are an accurate reflection of service delivery in relation to characteristics and benchmarks of other Scottish public library services.  Peer review is an integral aspect of the How good is our public library service process.

By inviting knowledgeable library professionals to review their self-evaluation process, library services benefit from hearing an informed and impartial assessment from an external and knowledgeable perspective.  The peer reviewers may best be described as acting as ‘critical friends’ who examine the library service’s self-evaluation thoroughly, reviews the evidence impartially, engage with staff and stakeholders, and comes to an independent assessment that they share in a frank and friendly manner with the library service.

The peer review panel consists of three members with one acting as convener; the members will be drawn from staff in other library services across Scotland. The peer review visit will normally be a one day event. 

One of the members will act as convener and have responsibility for compiling the summative report of the panel regarding the self-evaluation, the ratings and the experiences on the day of the visit. The final report from the peer review panel will be summative in nature, normally around two or three pages in length and will be appended to the self-evaluation report.