Evaluation approach

Major goal of European Initiative Analysis was to collect examples for the database that really reveal the good practice and not some random choice of examples. After the data was gathered, evaluation followed. Each case was analysed on the basis of seven criteria which derive from the theory of quality standards for policy making, impact assessment practices and similar evaluation procedures performed for the projects in postmining regions. The framework for the evaluation was agreed upon by all scientific partners and consists of the following criteria:

• EFFICIENCY: Project/programme/policy (instrument) creates maximum benefits by using least sources regarding the distributional effects in the society, economic and social effects, there is an on-going control of costs, benefits, distributional effects, effects on competition and market openness
• FEASIBILITY: Project/programme/policy (instrument) considers capacity of the system (location, financial and human resources, and administrative framework)
• SUSTAINABILITY: Project/programme/policy (instrument) has planned solutions in a way that they do not hamper the future use of these potentials; these are local solutions which fit local circumstances; they protect environment, enhance economy and human resources potentials, and finally improve living conditions for the local inhabitants
• TRANSPARENCY, OPENNESS: Community has been sufficiently involved at all stages of preparation and implementation, all the information, needed for decision making, was available, an on-going dialog has been created and several different techniques used to engage community which has resulted in an agreement about the future development and shared responsibility for it
• LEGITIMACY: Solution presents the compromise of all the stakeholders and actors possibly affected by it, there is a full government, local commitment to restructuring of the area
• INNOVATIVENESS: Project/programme/policy (instrument) , a concept is unique, new/upgraded, and has generated good design, has reinvented regional image, has an interesting story to tell and attracts with its originality
• TRANSFERABILITY: An idea, a concept can be re-used in similar conditions (mining area) on different location and in different governance frameworks

Evaluation was based on an expert opinion whether the project fulfilled an individual criterion. If so, this criterion was ticked by an evaluator. Such a simple correlation was chosen regarding the scope and nature of the collected data. The examples were not centrally collected; so it is very difficult to guarantee the same quality and dimension for all projects. To prevent subjectivity of expert opinions evaluation was done in two steps and then the final sum up procedure:

• First step: To minimize subjectivity each partner formed a group of three evaluators who were familiar with the project and collected cases. Each member of evaluation group evaluated each project according to the criteria. After individual evaluations the sum-up check was done. If one criterion has got at least 2 of 3 ticks, the project fulfils the criterion which is also ticked in the summed up column.
• Second step was a cross evaluation of the projects. The whole evaluation process was repeated by another partner's evaluation group on a different set of cases.
• evaluation to another partner's evaluation group which repeated the whole process of evaluation, this time on examples from other countries. The matrix for cross evaluation was following:
• Third step was a final step in evaluation. The evaluation data from step one and step 2 were gathered and summed up. Regarding the sum of the ticks - project's achievement each project was assigned "a final mark" regarding their fulfilment of the good practice criteria:
  • 0 - 1 tick: does not achieve GP criteria
  • 2 - 3 ticks: only partially achieves GP criteria
  • 4 - 5 ticks: mostly achieves GP criteria
  • 6 - 7 ticks: fully achieves GP criteria

Evaluation data serves as orientation information to those browsing the good practice database on "how good and useful" project really is.


Results

Two projects were not included into evaluation because at the time of collection they were still in pre-investment phase. On average projects were evaluated as "mostly achieves the good practice criteria". Altogether only three projects do not achieve GP criteria, nine of them reached 2 or 3 criteria, 26 apply to 4 or 5 criteria and 10 projects fully achieve the criteria. All the criteria is achieved only by two projects, one engaged with Natura 2000 area, the other is a comprehensive programme of measures', adopted to tackle the post-mining consequences in a former mining region. The other "successful" projects focus on tourist use of mines or mining heritage, one project includes production of biomass, the other use of geothermal energy and the last one is centre of renewable energies.


Percentage of projects achieving criteria by the type of project

Projects based on natural potentials achieved the highest average mark - 4.4 , so they mostly achieve the good practice criteria. There is no prevailing use of post mining potentials that can be claimed as the most successful, while on the other hand the lowest marks were given to geothermal projects. These marks do not necessary reflect project quality since not enough data is available, thus it was difficult to evaluate the value of the project. Among projects which mostly achieve the criteria are four times aqua parks, one biomass project, two golf courses, one geological study area and two landscape-recreational parks. Among the most successful are projects using renewables: biomass and geothermal energy.

Evaluation of projects based on cultural potentials sums up into an average mark of 4.1; hence good practice criteria are again mostly achieved. Cultural potentials' projects especially scored high the criterion of feasibility, legitimacy, sustainability and transferability. These projects are far better in reaching transparency than other projects but mostly they are not innovative. Best marks were given to tourist use projects such as Adventure Erzberg, Slazkammergut (historical salt region), Anthony's shaft and Peca underground. Good practice level is also reached at UNESCO sites such Erzgebirge and Rammelsberg, in Oxygen Adrenaline Park, underground mines, Erzrod, and two educational trails. Four projects were not recognized as good practice. Evaluators explained this happened mostly because of the lack of data.

Integrative approaches are diversified regarding uses and approaches. Some of them are umbrella projects such as IBA, the others are more specific. On average these projects achieved average mark of "mostly achieves GP criteria". None of the projects got the lowest evaluation mark; four projects only partially achieve criteria and have no common characteristics. In comparison with other projects the integrated approached scored the best only for sustainability and efficiency criteria. Thus, we can conclude comprehensive approaches result in higher efficiency than others. Projects of integrated approaches which engage with tourist, economic, educational, cultural and social activities are also good in feasibility, legitimacy and transferability. However, only half of the projects were prepared transparently, and six out of sixteen were considered innovative.

Sum-up table with detailed information on each project’s evaluation




Evaluation validity

Evaluation, although carefully thought through, has some limitations. For some project evaluation was hampered due to data unavailability and lack of information. For some evaluators some projects did not relate sufficiently to mining. Furthermore, due to rather loose criteria (i.e. without exact indicators) the evaluation is supported by subjective expert opinion. Subjectivity was compensated by 2-step group evaluation in which every project was evaluated at least with six individual estimations and two sum-up marks. Evaluation results thus give orientation information about the project quality.