The Welsh Education Research Network 2007-8

An Evaluation – Final Report

Executive Summary

S1. In terms of its ‘… primary intention … to trial a funding and support structure for educational researchers in Wales that harnesses collaboration between institutions to build research capacity in an All Wales Educational Research Network’ (WERN Proposal, p1) the WERN initiative has been highly successful.

S2. The WERN Executive Committee worked effectively as a governance structure for the initiative, with its success gained largely as a result of having representatives from all of the institutions involved. They promoted and managed the award of bursaries to eight inter-institutional collaborative groups. Initially this involved engaging 93 academics from the ten higher education institutions (HEIs) involved in the 24 bursary applications. The final eight bursaries involved 51 academics from nine institutions.

S3. The eight bursary application topics were:

Working lives: narratives of occupational change from further and higher education in post-devolution Wales;

Rural education;

Polish migrants’ children and education for sustainable development and global citizenship;

Exploring early years practitioners’ use of ‘effective’ verbal interactions in outdoor environments in Wales;

Perceptions of play and playfulness: implications for the implementation of the Foundation Phase in Wales;

An investigation of the affordances of ICT for the development of effective pedagogy in mathematics and science classrooms;

Learning Welsh as a second language in Key Stage 4;

Early years bilingualism: Welsh as a second language in the Foundation Phase.

S4. The intention to bring on new and inexperienced researchers was successfully accomplished with 15 academics with no research experience and 12 with 1-5 years of experience joining in bursary projects with 15 researchers of 11+ years of experience.

S5. The timescale for the initiative (October 2007 to June 2008) was a problem for all concerned and made the requirement for bursary groups to draw up proposals for a major funded research project between the beginning of January and the final report deadline of mid-May 2008 to be more or less impractical. Only one group had a well-advanced proposal by the due date and most others had only managed to do some of the groundwork. However, 11 grant proposals were underway, one grant had been won and nine paper presentations covering research from the bursary activities had been submitted to major conferences. It should be emphasized, too, that all of the bursary groups were continuing in their proposal development work after the final report deadline of mid-May.

S6. Most of the 37 interviewees who participated in the evaluation strongly endorsed the initiative. They responded to a specific question on their general impression of the worth of the initiative with comments such as ‘excellent’, ‘absolutely brilliant’ and ‘profound experience’. These positive comments were variously linked to the benefits afforded by collaboration, at the level of institution and at an inter-disciplinary and individual level, the personal professional learning achieved, the sense of purpose generated, the focus on Welsh issues and the high degree of commitment and support from institutions.

S7. A small minority of ambiguous or negative views were recorded from five interviewees with comments that included: ‘Noble idea but too little, too late’, ‘a curate’s egg – potentially good but too ambitious’, ‘right idea but too rushed’ and ‘pleasing but may be too late’.

S8. The collapse in Welsh educational research capacity is well-recognized and WERN was never formulated as a transforming ‘silver bullet’ initiative. It was planned as a pilot initiative to explore how the situation might be transformed, using a collaborative network model based around competitive small-scale bursaries. The evaluation has nevertheless shown that a degree of transformation has been inspired by WERN. For example, at least three groups were expressly intending to continue the research collaborations they had begun, regardless of whether there was a WERN II follow-on, and at least one group had almost completed a proposal for submission to the ESRC despite the short period in which they had to develop it.

S9. Institutions contributed to an average of six applications (range 1-11) and were involved with an average of five (1-7) collaborating institutions overall. The average number of staff making applications was 9 (2-16) while the average number engaged in funded groups was 5 (0-9). The average number of funded awards, in which institutions were involved, was 2 (0-5).

S10. Thirty-one responses were made in relation to aspects of leadership, management and administration of the overall initiative (16) and the individual projects (15). Nine interviewees enthusiastically endorsed the high quality of management and administration of the initiative overall. Comments such as ‘fantastic job’, ‘amazing’ and ‘excellent’ were used to describe the efforts of the coordinator and administrator of the initiative. A further seven interviewees described the management and administration as good, sometimes with qualifications about specific aspects such as financing.

S11. At project level, the views expressed on bursary group leadership were of a similar kind. Leadership of the groups was considered by 10 of the 15 responding interviewees to be very good or better; two comments being ‘fab!’ and ‘excellent’. Three felt that their projects were well managed and two felt the management of their projects was average – one person claiming that they ‘felt they haven’t got anywhere’.

S12. The role of mentors in the bursary groups was almost universally hailed as a success, with 16 of the 20 respondents on the issue paying tribute to the mentors and only four voicing a less satisfied view, e.g. ‘problematic’, ‘average’ and ‘too early to say’. The mentors were all very experienced researchers and the 16 interviewees variously considered them to have given of their time and expertise willingly and in a highly professional but unassuming and non-patronizing way. Mentors were described as ‘inspirational’, ‘insightful’, ‘fantastic’ and ‘extremely effective’. One made ‘everyone feel valued’, another ‘guided us away from blind alleys’ and another ‘kept us grounded’. It seems reasonable to conclude that the mentoring feature of WERN was a highly successful aspect of the research capacity building.

S13. The use of the VRE (virtual research environment) was limited to more or less two groups. Fourteen of the 21 bursary-related interviewees who responded said they did not use it at all with the remaining seven indicating that they did not use it much. It should be emphasized, however, that the lack of use of the VRE did not have any manifestly detrimental impact on the progress of the various bursary award activities, while the modest use made in a couple of instances did demonstrate that it could add value.

Recommendations

S14. In order to foster further collaboration and research capacity building, a strategic research agenda, for example identifying educational research needs over the next five years, should be drawn up by the Welsh Assembly Government (WAG) and the Higher Education Funding Council for Wales (HEFCW) in consultation with the HEIs and other relevant stakeholders.

S15. Funds should be set aside for commissioning research into these topics and also for non-predetermined areas to ensure there is potential for creative, in addition to responsive, proposals from institutions. The commissioning criteria should include clear requirements for user engagement, inter-institutional collaboration, inter-disciplinarity and research capacity building. Commissioning models should include the successful bursary scheme as well as larger-scale awards. The former should be engaged for at least one more tranche to capitalize on the existing good proposals that could not be funded in the pilot initiative. Whether there are benefits to be gained from future tranches will not be determined until the final outcomes are established for the current bursaries.

S16. Building on its success in seeding research activity and capacity within a competitive yet highly collaborative framework, the WERN Executive Committee should act as the commissioning body. Its performance should be reviewed on an annual basis through a comprehensive progress report and key performance indicators of activities and outputs that can be linked to WERN. A tentative list might involve dimensions such as:

Collaborative grant applications involving two or more institutions (with a view to geographic location, perhaps);

Numbers and type of staff new-to-research who have become involved;

Numbers and types of training events and the staff attending them;

Numbers of published papers, conference presentations (particularly international conferences).

S17. Given the potential benefits of increasing institutional and national research activity, the committee should be funded by a modest subscription from each institution to facilitate meetings (and travel costs) in the three main regions: south, west and north. In addition to its commissioning and monitoring role, it should seek to provide a voice for the promotion of research capacity building and collaboration, to lobby for increased funding to support educational research, to identify strategic goals and topics for educational research in Wales, and to organize pan-Wales training events and an annual showcase event for Welsh educational research.

The Welsh Education Research Network 2008-9

An Evaluation – Final Report

Executive Summary

S1. WERN II was designed to extend WERN I by introducing a variety of research capacity building approaches. In addition to the WERN I-type bursary for groups planning a grant application, WERN II offered competitive awards for small-scale studies, collaborative fellowships (supported by mentors) and a local authority collaborative project (see Table 4). WERN II also hosted a review of the implications for Wales of outcomes from the ESRC’s Teaching and Learning Research Programme (TLRP) projects.

S2. WERN II was considered by the interviewees to be a success though there was a variety in the conceptions of this success. For example, success was perceived as building on WERN I by ‘mopping’ up proposals that had not been supported in WERN I, extending WERN I activities, consolidating the educational research community that WERN I had fostered and diversifying into new activities. Concerns expressed included the perception that an emphasis on school-phase education had marginalized other aspects of education and that the importance of capacity building (in terms of training courses etc) had been lost.

S3. The range of content for the various activities in WERN II confirms that a variety of important areas was addressed and that a significant research effort had clustered around early years education (see Table 5).

S4. Multi-institutional collaborative groups were less in evidence in WERN II than in WERN I with institutions collaborating on average with 2.2 other institutions compared with 2.9 for WERN I (see tables 6 and 7). The majority of collaborations (6/9) featured pairs of institutions.

S5. The collaborative fellowships were particularly successful in promoting the professional learning of the individuals involved. The quality of the support from the mentors was unanimously appreciated and praised, with a variety of outputs and opportunities (off-campus meetings, conference attendance, collaborative writing, 1-to-1 training and so on) spinning off what was a relatively modest amount of funding (£2,250 per fellowship).

S6. A wide range of professional development was recorded by participants in WERN II. Skills featured frequently in these responses and fell into four main categories: research methods, communication, organizational and personal/interpersonal.

S7. WERN II was roundly commended for continuing to raise the profile of educational research and capacity building, and the institutional links and collaborations that had been promoted. There were reports of raised awareness of the importance and breadth of educational research at institutional level, with various indicators such as staff taking up doctoral study, the initiation of a ‘research day’, the impact of dissemination of WERN-based research findings on teaching, and discussions on the interface between research and practice at management level. Other reports, however, indicated that there was little or no evidence of institutional impact.

S8. Vice-chancellors and the senior management of higher education institutions were viewed as the key influencers in achieving a better research-teaching interface in institutions, particularly in facilitating existing or nascent research cultures through the provision of time and staff resources.

S9. Respondents variously distinguished between engagement in research and engagement with research. The requirement to use research to support reflection on one’s own practice (in its ‘keeping up with the literature’ rather than ‘carrying out a project’ meaning) was mentioned by several respondents. However, a case was explicitly made for active engagement by staff in research.

S10. Some respondents argued the need for institutions dominated by teacher education programmes to recognize the benefits of research-informed teacher education and to ring changes in their workload policies and general disposition to research activity. Two of these people, from different teaching-intensive institutions, qualified their views by arguing that it was also important to ensure that time for teaching and administrative activities did not expand inefficiently, inadvertently curtailing time for scholarship and research.

S11. The two phases of WERN had engaged some 150 academic staff in funded research activities (not counting the considerable voluntary inputs of Executive and Advisory group members, the unfunded mentors and voluntary TLRP reviewers). Almost every institution had members of staff participating in one or more of the funded activities in WERN II (see Table 8).

S12. In the main, the bursary group activities (see Table 5) built on work that had been initiated in WERN I, though it should be emphasized that some of these activities had been prompted by existing research strengths. For example the rural education, ‘working lives’, ICT and Foundation Stage topics had strong links to previous work and, indeed, were proposed by members of WERN I groups. Two of the WERN II awards – relating to music education and bullying in the early years – had been successfully re-worked from unfunded WERN I applications.

S13. A positive community dimension was mentioned by participants, with the WERN II Colloquium as a particular indicator of this success. The outcomes of this community development included a perceived reduction in ‘suspicion’ and improved relationships brought about by the collaboration of research-intensive and non-research-intensive institutions. However, one interviewee reported feeling that their personal presence in WERN activities often seemed to be over-shadowed by other participants’ perceptions of their institution.

S14. As a collaborative forum comprising representatives from every institution, the Executive Group was widely acknowledged to have played a major role in ensuring the success and cohesion of WERN II. There are certainly grounds for recommending that future capacity building efforts should retain the same type of forum and cooperative processes. A small number of counter-views on the efficacy of the Executive Group tended to relate to the working processes and functions of the Group rather than the principle of a collaborative forum or network.

S15. The Advisory Group drew widespread praise for its expertise and selfless contributions of time and effort to the cause of reviving educational research in Wales. It was widely regarded as fair in its evaluations of proposals but there was a small number of counter-views.

S16. When asked how WERN II might have been altered to improve the benefits for individuals and institutions the most frequent comment related to the need to have increased time built into the programme. Opinion was divided in the e-survey (see Table 9) on whether there was sufficient time for making applications. Most of the respondents felt that the WERN II application process was clearly outlined but almost equal numbers contested the view that WERN II was sufficiently funded.

S17. There was a degree of confidence that individuals would maintain their research interests and skills and that contacts initiated by both phases of WERN would endure. However, counter-views perceived little sustained impact on institutions owing to the limited period of operation of WERN and a lack of institutional commitment. It was also felt by some that in a post-WERN situation the inter-institutional networks created would be less likely to survive than single institution groupings.

S18. A number of interviewees looked to HEFCW and WAG investment to sustain the momentum and growth of an educational research community, and specifically the networking and collaborative ventures. There was ambivalence in the e-survey about the value of WERN as a “… solid platform of experience of the bidding process for institutional teams and inter-institutional networks to create good proposals in the future” with as many respondents agreeing as expressing uncertainty. However, only a small minority disagreed outright with the proposition.

S19. Relatively high majorities (70%+) of the e-survey respondents (see Table 10) felt that in any future development:

• Funds for researchers and doctoral students should be distributed across institutions;
• Research should not simply be focused on Wales-related issues;
• Institutional benefits must be clear and tangible to ensure the full commitment of institutional managers;
• High quality research will be the best foundation for capacity building in Wales;
• Opportunities must be open for all institutions;
• Funding sources from across Wales and beyond must be actively pursued.

Conclusions

S20. All things considered I feel confident that the various sources of data and information in this evaluation have provided me with a sufficient grasp of the issues to enable me tentatively to propose a set of conclusions.

S21. It is clear to me, therefore, that it would be a major mistake to allow the gains in collaboration, capacity building and awareness of the importance of research to improving education and teacher education, made through WERN activities, simply to fade away.

S22. Responsibility for developing and improving the educational research base in Wales, however, cannot simply be laid at any one group’s door.

S23. There is general consensus within the WERN community on the following points:

• Educational research is a principal means for promoting improvement and positive change in education in all sectors.
• Without an appropriate level of research activity, designed to increase understanding of how education can be improved at the system, social and individual levels within its own context, any nation risks an unhealthy stasis in the quality of educational experience and outcomes for all of its citizens. 
• Educational research is valuable across a spectrum of levels.

At one end, for example, existing research-based knowledge and literature can be used continuously to inform and promote improvements in practice at the level of practitioners and their institutions, schools or workplaces.

At the other end, research of the highest quality carried out locally can contribute both to significant changes in national policy and practice and to global knowledge building.

• Wales has the potential to increase its contribution to UK and international research excellence and to this end it needs a highly skilled community of educational researchers.

S24. The evaluation of WERN has shown that:

• It is possible, at a relatively modest level of funding, to generate increased awareness of the importance of educational research and to promote research capacity building and collaborative research activity across institutions that may differ significantly in mission (e.g. institutions that focus on different aspects or sectors of education, or which are either predominantly teaching or research-intensive).
• There is a viable and growing core of research expertise in a number of specific policy and practice areas, for example post-compulsory and adult education, rural education, ICT in education and early years education.
• Despite involvement in WERN, some teaching and/or teacher education-intensive institutions appear not yet to have espoused and facilitated, at an institutional level, the development of a research-informed or research-active culture for improvement.
• WERN, as a pan-Wales forum for inter-institutional representation and engagement on educational research matters, has brought considerable benefits to institutions and individual researchers, and the overall well-being of educational research in Wales.

S25. WERN II has in my view brought increased focus to the implications for all stakeholders in educational research. The statements that follow are generalized to all members of each stakeholder community. However, I should like to emphasize that I do recognize there are those who need to consider the advice and those who already espouse and act upon the ideas or good practices involved.

• For government there is a need to provide a strategic lead in defining the nation’s educational research requirements, to sponsor competitive research funding programmes and to challenge institutions to develop their use of and engagement in research.

Action in these areas can be considerably strengthened by a purposeful liaison with institutional representatives, e.g. in some form of WERN-like forum, in the planning and outworking of any strategy or programme.

• For managers in higher education institutions there is a need to evaluate their institutional investment in educational research and its potential to improve their programmes.

They need to consolidate any strengths and address any weaknesses in their educational research activities. Where appropriate, they must consider how best to address the workload implications of improving the teaching-research interface.
There is also a need to be pro-active in liaising with government and other potential research users to identify strategic developments and needs in educational research.

• For individual academics there is the need to recognize that they must engage with or in educational research if its potential to improve their practice and their students’ learning outcomes is to be realized.

In teaching intensive contexts specifically, including teacher education, there is a need to ensure that the personal management of teaching and administration is optimized in order to facilitate engagement with or in educational research.