Six inspectorates, from France, Scotland, England, Ireland, Sweden and The Netherlands decided to work together to evaluate each other’s practices and instruments. We formulated two goals for our peer reviews.

The first was to identify strengths and weaknesses in the following areas:

• The context in which inspectorates operate. Including their mission, their mandate and their position.
• The instruments that are used to evaluate ICT. Looking at the way the instruments are used, the quality of indicators, frequency and impact etc.
• The expertise and effectiveness of inspectors. Including professional development and the way inspectors operate during school visits but also looking at the interaction between school staff and inspectors.

The second goal of the project was to see whether we could come not only to a shared and improved understanding of the impact of ICT on education, but also whether we could go as far as to actually build a framework with common criteria and indicators.

We started with a meeting in Haarlem, The Netherlands. Each inspectorate made a short presentation of the state of affairs with regards to education policy, ICT policy and evaluation. Also during this first meeting, triangles were formed. The first triangle consisted of France, The Netherlands and Scotland. The other contained England, Ireland and Sweden. These triangles were formed quite randomly – taking the view that there was something to learn from each of the participants. At the same time we tried to make good mixes in terms of previous collaborations, country contexts and of course language ability.

A visit schedule
An important question was what a typical review should consist of. The guests need a good idea of the school system, the education policy, the position of inspectorates and the instruments that are used. To achieve this, we exchanged information beforehand and organised a briefing session as the start of each peer review. During the briefing sessions, more background information was provided, often at a high pace as the schedule for the visit was very demanding. It meant the inspectors had to take in a massive amount of information, often in a foreign language. All this was not only necessary to answer the central questions of p2p-i about the context, but moreover prepared the visitors for the school visits that were the pivotal part of the peer review. This also provided the necessary focus to the briefing session, as the hosts asked the question: what information do we deem absolutely necessary for the guests to be able to understand what they are going to see? The visits ranged from half a day to a whole day. This meant that they would sometimes reflect a complete visit as it would take place, but in other cases only gave a glimpse of a more complex and longer visit. Going on school visits could entail extensive travel. Although the hosts tried to limit this, they had to balance this requirement with the inspecting schemes that were already planned and the need for an ICT focus. In some cases, travel time was high. As an alternative, parts of the inspections were planned especially for p2p. In the majority of cases however this did not endanger the reliability of what we observed in terms of how realistic the visit was in relation to the everyday work of our colleagues.

In the evening or sometimes on the way back from the last school visits, we had time to reflect on what we had seen. On the second or third day of the visit a debriefing session was organised. This gave us opportunity to look back, to ask questions, to appreciate what went well and last but not least to be frank about what could be improved. Each colleague briefly described the visits they had observed and what struck them. The hosts replied with explanations, sometimes recognising strengths and weaknesses, sometimes defending them. In all cases there seemed to be a strong commitment to make the feedback session as useful as possible. This session was very useful and it acted as a primer for the country reports that were written about each of the six review visits. For each visit a rapporteur was appointed, who had the task to bring all the individual written reports and the outcomes of discussions together.

The project had a final meeting in Brussels. All participants had had some time to reflect during the summer holidays. We focused on the discussion of essential questions that arose from the country reports. In addition, we discussed the value of peer reviewing, what could be improved and how we could build on the experiences – both in terms of sharing what we had learned with others in our organisation and in terms of possible follow-ups. Finally, we worked on the definition of our framework. To this aim we had gathered all categories, indicators, descriptions and so on from all the existing instruments. By the end of this two day session, this had led to the outlines of a shared framework which we are still working on through revision sessions through e-mail.

What did it bring?
Participants felt the project was a success and a number of key wins were identified:

• Recognition. The recognition that you are on the right track, recognition of the problems you face and the solutions that are there. Sometimes this strengthened the resolve to go on in a certain direction, although there may be forces within one’s own country that resist this.
• Inspiration. Seeing other professionals do their work is very inspiring. It is also inspiring to see other solutions, things that you had not thought about yourself. The visits also inspired change, the conviction that it is time to change certain aspects in one’s work or organisation or sometimes maybe even education policy. These processes can take up a lot of energy and time and it is good to be inspired to pursue what you are trying to achieve.
• Professional dialogue with peers. It is not on a daily basis that inspectors get the chance to discuss with peers. Although there are many people with whom one can have inspiring, stimulating and professional dialogue, it is very fruitful to have discussions with real peers about external evaluation. There are many things that require neither introduction nor explanation when talking to fellow inspectors so discussions almost immediately reach a higher level.

Finally, participants felt that the process had increased their understanding of ICT’s place in learning.

In terms of innovations, what types of improvements of existing practices could come out of this process? In the case of p2p-I there are three areas in which innovations can take place as a result of our reviews:

• Improving national practice, based on feedback from visitors. One can see peer reviewing as part of the continuing professional development of inspectorates, where the reflection of peers can speed up or renew existing actions and initiatives. Sometimes the hosts specifically asked the guests to focus on and provide feedback about aspects, such as a new framework or methodology;
• Introducing new ways of working, based on what was observed abroad, e.g. colleagues who translated instruments. In terms of complexity, this is one step up form the previous improvement. It involves much more long-term resilience and the conviction that what was observed elsewhere could work in one’s own country. In addition, tools, instruments or methodologies need to be contextualised to fit local circumstances. Finally, it involves trying to convince others that were not part of the review process
  of the need to replace existing practice with something that worked elsewhere;
• Using common criteria, as a replacement or in addition to existing national ones. This was one of the aims of the project. Judging on the effort that participating inspectors are currently putting into the revision of the framework, it seems that there is a willingness to implement (parts) of the common framework. However, the more refined and final the common framework becomes, the more far reaching the implication: is this really applicable in our own work, does it fit our vision on ICT and education?

Factors of success
If we want peer reviews to contribute to improvements and innovations, we need to ensure that the conditions under which the peer review takes place are optimized. As said previously in this article, we had no rock solid ideas on how to shape this peer review project and some of the methodology developed along the way was based on the experience of the first reviews. However, in hindsight we can say that the following factors contributed to its effectiveness:

• Respect for each others’ knowledge, experience and background. Without this respect, it would have been difficult to make value free judgments. This also implies leaving behind any preconceptions about local (education) culture and policy;
• Common tasks. Regardless of the many differences that can be identified in the work of inspectorates, we all share the task of evaluating and improving the quality of education. This brings with it an almost automatic understanding of each other’s problems and challenges and a true and deep interest in solutions that were chosen elsewhere;
• A balance between similarities and differences. In peer reviewing there is a risk of trying to create identical twins as pairs. Although this would lead to much recognition, there is a risk that little comes out of it. At the other extreme, total disagreement is also not idealas most of the time would be spent explaining the uniqueness and strengths of local solutions. We need pairs with enough differences to question and to be surprised but with sufficient common ground to ensure understanding and respect.
• Small teams. The size of the teams meant that an informal atmosphere was created enabling confidentiality. It also meant that each visitor could see several examples of local work.
• More then two countries involved. This proved to be a way to balance the differences and similarities – there was always someone bringing in an alternative vision on things.
• Participative work. It was a great experience to actually join colleagues in their day-to-day work. Our understanding was really deepened and what’s more, we could test our assumptions of the system and the claims that were made in policy papers and during the briefing session.
• Structure. There needs to be some kind of structure in the form of a hypothesis, a central theme or, as in this case, main questions.
• A clear deliverable. The fact that one of the aims was to define a shared framework, meant that work and contacts continue also after the actual peer reviews were carried out. It also provided focus during visits as we were looking for useful instruments and indicators.

Room for improvement
Being inspectors, of course we also found a number of weaknesses of the project. The first problem was that we sometimes felt we had too little time to thoroughly understand and go through the inspecting process as it is actually taking place under normal circumstances. It is hard to see how this weakness could be remedied as it is primarily a logistical and financial problem: no one could free up the time and funds to spend maybe two weeks to join an inspecting team from start to finish. However, through e-mail contact before and after, even better preparation and involvement of the visitor by the host and in some case maybe one more day, the review process would certainly be enhanced.

The second problem was dissemination to others. This is probably a common problem to all groups that do reviews, exchange information and the like: how do you ensure that what you have learned actually reaches others and that it has long term effects on your work? We do know that the review affected our personal views and that through this
our organisations will change. But are there not more explicit ways of change possible and how can these be applied throughout the whole process?

A final problem was that of timing. If we expect concrete and direct innovation, improvements and change to come out of peer reviews - if we want to implement a shared framework, the timing needs to be perfect. There has to be a strong conviction that change is needed at many levels; a readiness to accept solutions. In some cases it will be hard to pick the fruits of peer reviews to the full if there has just been a major revision in one’s country in terms of policy or practice, or when new frameworks have just been introduced. The extent to which different timings and stages of policy and practice reforms in the participating countries have a negative or positive effect on the outcomes of the review will depend on the expectations and openness and the mandate of the participants and the goals of the peer review. In a more open review, where there is no demand for immediate change as one of the outcomes, the issue of timing would to be less problematic.

Next steps
The expectation is that the shared framework will be available April 2006. Only then can we assess whether we can find the funds and the partners to take a next step: the testing and use of the shared framework in inspecting practice.

More information on p2p-i is available at http://p2p.eun.org or by contacting Bert Jaap van Oel at b.vanoel@owinsp.nl