The main idea of ​​forecasting based on expert assessments is to build rational procedure for human intuitive-logical thinking in combination with quantitative methods for assessing and processing the results obtained.

The essence of expert assessment methods is that the forecast is based on opinion specialist or team of specialists, based on professional, scientific and practical experience.

Individual expert assessments- are based on the use of the opinions of expert specialists in the relevant profile.

1. Interview method involves a conversation between a forecaster and an expert using a question-and-answer scheme, during which the forecaster, in accordance with a pre-developed program, poses questions to the expert regarding the prospects for the development of the forecasted object. The success of such an assessment depends to a large extent on the ability of the expert to give impromptu opinions on a wide variety of issues.

2. Questionnaire method consists in the fact that the expert is asked to fill out a questionnaire (questionnaire) containing a list of questions, each of which is logically related to the research task.

The following types of questions can be used in the questionnaire:

· open – answers to these questions can be formulated in any form;

· closed type – answer options are offered, one of which must be chosen by the expert.

The use of closed-ended questions in the questionnaire is preferable, as it simplifies the statistical processing of the answer results and facilitates the work of the expert when filling out the questionnaire. On the other hand, the list of answers to the question may not contain the expert’s opinion. Therefore, when creating a list of answer options for some questions, it should be possible for the expert to put forward his own answer option or avoid answering.

3. Analytical method(analytical notes) provides for careful independent work by the expert to analyze trends, assess the state and development paths of the predicted object. The expert can use all the information he needs about the forecast object. He draws up his conclusions in the form of a memorandum. The main advantage of this method is the ability to make maximum use of the expert’s individual abilities. However, it is of little use for predicting complex systems and developing strategies due to the limited knowledge of one specialist expert in related fields of knowledge.

The main advantage of individual expert assessment methods is the possibility of maximizing the use of the individual abilities of experts. However, these methods are of little use for predicting the most general strategies due to the limited knowledge of one expert about the development of related areas of science and practice.

An example of the use of expert assessments in planning the development of socio-economic systems is the multicriteria task of choosing a solution option, which is currently relevant in many areas of human activity.

The multicriteria selection procedure includes the following steps:

1. Identification of the most significant indicators (criteria) characterizing the object under study;

2. Determination of a method for quantitative assessment of indicators;

3. Determination of acceptable limits for changes in indicators;

4. Choosing a method for finding the best option;

5. Solving the problem and analyzing the results.

As an objective function for evaluating solution options, additive convolution of criteria is most often used:

Or , (2.18)

where are the weighting coefficients characterizing the significance of the criterion. Numerical values ​​are determined by experts, and it is desirable to comply with the following condition:

If the criteria have different units of measurement, then they must be brought to a single dimensionless scale so that the following inequalities are satisfied:

Example . According to experts, the main indicators of economic and social development of the region are:

Gross domestic (regional) product;

Employment level;

Average monthly salary.

An expert assessment of the significance of the criteria on a ten-point scale is presented in Table. 2.2.

The regional leadership has been proposed four targeted regional development programs aimed at priority financing:

1. Agro-industrial complex;

2. Food industry enterprises;

3. Sectors of the socio-cultural sphere;

4. Housing construction.

The expected values ​​of the main indicators obtained during the implementation of the target programs under consideration are given in Table. 2.3.

Table 2.2

Expert assessment results

Table 2.3

Expected values ​​of the main socio-economic indicators of the region's development

It is necessary to determine the most appropriate development program for the region.

Solution:

Let's determine the values ​​of the weighting coefficients:

; ; .

Thus, as a result of processing expert assessments, the objective function has the following form:

Considering that target program No. 3 is obviously ineffective compared to program No. 2 (1500<2000; 80=80; 1000<2000), удалим её из матрицы возможных решений:

Since the values ​​of the indicators have different dimensions, they must be brought to a single dimensionless scale. This is achieved by dividing the elements of each column by the maximum value in the column:

At the final stage, we determine the value of the objective function for the proposed programs:

The maximum value of the objective function corresponds to program No. 1. Therefore, the implementation of this program is most appropriate.

The most reliable are collective expert assessments - involve determining the degree of consistency of expert opinions on promising areas of development of the forecast object, formulated by individual specialists.

To organize expert assessments, working groups are created whose functions include conducting a survey, processing materials and analyzing the results of a collective expert assessment. The working group appoints experts who provide answers to the questions raised regarding the prospects for the development of this facility.

1. The essence method of collective idea generation (brainstorming) consists of using the creative potential of specialists in brainstorming a problem situation, which first involves the generation of ideas, and then their structuring, analysis and criticism, putting forward counter-ideas and developing a coherent point of view.

The method of collective idea generation involves the implementation of the following stages:

1. forming a group of brainstorming participants to solve a specific problem. The optimal group size is found empirically. Groups consisting of 10-15 people are considered the most productive.

2. The analysis group draws up a problem note, which formulates the problem situation and contains a description of the method and the problem situation.

3. Idea generation stage. Each participant has the right to perform multiple times. Criticism of previous performances and skeptical remarks are not allowed. The facilitator adjusts the process, welcomes improvements or combinations of ideas, and provides support, freeing participants from constraint. The duration of the brainstorming session is no less than 20 minutes and no more than 1 hour, depending on the activity of the participants.

4. Systematization of ideas expressed at the generation stage. A list of ideas is formed, characteristics by which ideas can be combined are identified, ideas are combined into groups according to the identified characteristics.

5. At the fifth stage, the destructuring (destruction) of systematized ideas is carried out. Each idea is subject to comprehensive criticism by a group of highly qualified specialists consisting of 20-25 people.

6. At the sixth stage, critical comments are assessed and a list of practically implementable ideas is compiled.

Method "635"- one of the varieties of “brainstorming”. The numbers b, 3, 5 indicate 6 participants, each of whom must write down 3 ideas within 5 minutes. The leaf goes around in a circle. Thus, in half an hour, everyone will write down 18 ideas, and all together - 108. The structure of ideas is clearly defined. Method modifications are possible. This method is widely used in foreign countries (especially in Japan) to select from a variety of ideas the most original and progressive solutions to certain problems.

2. Delphi method. The purpose of the method is to develop a program of sequential multi-round individual surveys. Individual surveys of experts are usually carried out in the form of questionnaires. Then they are statistically processed on a computer and a collective opinion of the group is formed, arguments in favor of various judgments are identified and summarized. The computer-processed information is communicated to experts, who can adjust the assessments, while explaining the reasons for their disagreement with the collective judgment. This procedure can be repeated up to 3-4 times. As a result, the range of assessments is narrowed and a consistent judgment is developed regarding the prospects for the development of the object.

Features of the Delphi method:

a) anonymity of experts - interaction between group members when filling out questionnaires is completely excluded;
b) the possibility of using the results of the previous round of the survey;

c) statistical characteristics of group opinion.

3. "Commission" method- based on the work of special commissions. Groups of experts at a round table discuss a particular problem in order to harmonize points of view and develop a common opinion. The disadvantage of this method is that the group of experts in their judgments is guided mainly by the logic of compromise.

The method of expert commissions can be organized in one of the following forms:

As practice has shown, the “commission” method has significant disadvantages:

The great influence of such a psychological factor as the opinion of authoritative experts, to which other experts join without expressing their point of view;

Reluctance of experts to publicly renounce their previously expressed opinions;

When commissions work, most often there is a dispute between two or three of the most authoritative experts, as a result of which other experts either do not take part in the discussion or their opinions are not taken into account.

4. Method of trial – is based on organizing the work of a team of experts in the form of conducting a trial. The use of this method is advisable when there are several groups of experts, each of which defends its own point of view. In this case, the object of forecasting acts as the “defendant”. Leaders of groups expressing alternative points of view act as prosecution and defense (prosecutor, lawyer). Individual experts play the role of witnesses, providing the court with the information necessary to make a decision. The role of the judge is played by an interested person (group of persons). For example, in the television program “The Trial,” based on the use of the court method to analyze and predict the development of various socio-economic processes, the role of the judge was played by the audience, voting during the program by telephone calls for the point of view that they supported.

Method of morphological analysis involves choosing the most acceptable solution to a problem from among the possible ones. It is advisable to use it when forecasting fundamental research. The method includes a number of techniques that involve a systematic consideration of the characteristics of an object. The study is carried out using the “morphological box” method, which is built in the form of a tree of goals or a matrix in which the corresponding parameters are entered in the cells. Connecting a parameter of the first level in series with one of the parameters of subsequent levels is a possible solution to the problem. The total number of possible solutions is equal to the product of the number of all parameters presented in the "box", taken by row. Through permutations and various combinations, it is possible to develop probabilistic characteristics of objects.

Script writing method- is based on determining the logic of a process or phenomenon over time under various conditions. It involves establishing a sequence of events that develop during the transition from the existing situation to the future state of the object. The forecast scenario determines the development strategy of the forecasted object. It should reflect the general goal of the development of the object, the criteria for assessing the upper levels of the “goal tree”, the priorities of problems and resources to achieve the main goals. The scenario displays a sequential solution to the problem and possible obstacles. In this case, the necessary materials for the development of the forecast object are used.

A forecast graph is a figure consisting of vertex points connected by edge segments. A “goal tree” is a tree graph expressing the relationship between vertices-stages or problems of achieving a goal. Each vertex represents a target for all branches emanating from it. The “tree of goals” involves identifying several structural or hierarchical levels.

Building a “tree of goals” requires solving many problems: forecasting the development of the object as a whole; formulating a scenario for the predicted goal, determining the levels and vertices of the “tree”, criteria and their weights in ranking the vertices. These tasks can be solved, if necessary, using expert assessment methods. It should be noted that this goal as a forecast object can correspond to many different scenarios.

The scenario is usually multivariate in nature and highlights three lines of behavior: optimistic - the development of the system in the most favorable situation; pessimistic - development of the system in the least favorable situation; working - development of the system taking into account counteraction to negative factors, the occurrence of which is most likely. As part of the forecast scenario, it is advisable to develop a backup strategy in case of unforeseen situations.

The finished script must be analyzed. Based on the analysis of information found to be suitable for the upcoming forecast, goals are formulated, criteria are determined, and alternative solutions are considered.

Expertise is an assessment obtained by seeking the opinions of experts. An expert (from the Latin e x p e r t u s - experienced) is a person knowledgeable in a certain field of activity, invited to solve an issue that requires special knowledge. Expertise can be individual (when one specialist is involved in solving a problem) or group. Experts can express their opinion orally or fill out a special form. The opinion of specialists is sought whenever it is impossible or very difficult to carry out measurements using more accurate methods (V.M. Zatsiorsky, 1982; B.G. Litvak, 1996; V.S. Rubin, 2006; E.R. Yakhontov, 2006) .

For example, expertise is resorted to in the following cases: a) when predicting a situation; b) when analyzing events for which there are no other measurement methods; c) when justifying the adoption of a particular management decision under conditions of uncertainty (V.V. Muzychenko, 2003; N.N. Pilipenko, E.L. Tatarsky, 2007).

Carrying out an examination includes the following main stages: forming its purpose, selecting experts, choosing a methodology, conducting a survey of experts, processing the information received, including assessing the consistency of individual expert assessments. Expert assessments are divided into quantitative and qualitative (B.G. Litvak, 2002).

Selection of experts is the most important stage of the examination. The main requirements for experts are: competence, interest in the work of the expert commission, efficiency, open-mindedness, objectivity and independence of judgment. The accuracy of expert assessment largely depends on the number of experts. As practice shows, the optimal number of experts is 7-12 people (E.M. Korotkov, 2003). Expert assessment can be carried out by using the following approaches: a) closed discussion followed by closed voting or filling out special expert forms; b) free expression without discussion or voting; c) open discussion of the issues raised, followed by open or closed voting.

There are many different ways to conduct a quality assessment. The simplest of them is called the method of preference (or ranking of alternative options). Using this method, experts arrange the objects being assessed in order of deterioration in their quality (Table 1).

Table 1. Form of the expert form compiled during the examination using the preference method

TYPE OF TOURISM	Ranking results
Expert number	Total points
Rustic
Wellness
Pilgrimage
Informative
Adventure
Entertaining
Recreational
Sports
Exotic
Ecological

The place occupied by each object is determined by the number of points scored: the lower the sum of points, the higher the place occupied. As an example, Table 1 presents the results of ranking ten types of tourism by six specialists (experts) according to the level of their attractiveness for residents of Russia.

Another method of conducting an examination is often used - the paired comparison method. In this case, the expert fills out a table in which all compared objects are indicated both horizontally and vertically (Table 2).

Table 2. An example of an expert form filled out by each expert when conducting an examination using the paired comparison method

									Sum of points
Bulgaria
Germany

For example, it is necessary to determine the most popular country out of the eight presented, to which Russians would like to make a tourist trip. In Table 2, each cell refers to two objects being compared, and a unit is assigned to the one that, in the expert’s opinion, is of higher quality or more important. The other of these compared objects is assigned a zero. Then the total number of points scored is calculated and the place (rank) of the object of examination is determined.

Evaluation of the results of the examination can also take a more complex form. Next, as an example, we will give a step-by-step examination aimed at “Selecting the head of a travel company.”

Stage No. 1. Filling out the expert form. As we have already established, each of the experts must fill out an expert form, which, in essence, is a matrix of preferences, and the columns and rows of this matrix are called selected qualities (Table 3). The meaning of this filling is to compare all the qualities one by one with each other. In this case (i.e., in a mutual comparison), the more preferable quality is given 2 points, and the less preferable quality is given 0 points. If it is impossible to give preference to any of the two compared qualities, each of them is given 1 point.

Table 3. Matrix of preferences

So, in our example, in the first step we begin to compare qualities in the first line, i.e. First of all, we compare the “sociability” of the 1st line and the “commitment” of the 2nd line. Let, in the opinion of one of the experts, for the director of a travel company, whose qualities are compared with each other, “sociability” is preferable to “commitment.” Then in “cell 1.2” the value 2 is entered, accordingly “mandatory” is assigned 0 points and this value is entered in “cell 2.1”. Thus, at the first step, only the first row and, accordingly, the first column are filled in, “automatically” depending on the selected preferences for the first row. Next, the expert compares the qualities in the second line, i.e. compares “commitment” with all other qualities. Moreover, there is no longer any need to compare “sociability” and “commitment”, since this comparison has already been made at the first step. All remaining qualities are compared in a similar way. In our example, the completed preference matrix will have the following form (Table 3).

To check the correct filling of the preference matrix, you should pay attention to the fact that all elements relative to the main diagonal have mutual correspondence, that is, if the value 2 is written in “cell 1.2”, then the value 0 should be written in “cell 2.1”, respectively, if in “cell 1.3” the value 1 is written, then in “cell 3.1”, accordingly, the value 1 should also be written, etc.

Stage No. 2. Processing the preference matrix. At this stage, each expert begins processing the preference matrix. First of all, all the values ​​of the matrix cells are summed by row. In this way, the total score obtained by each alternative quality can be calculated, i.e. essentially find out the absolute weight of each individual quality (V). Note that the maximum absolute weight of each quality () is equal to:

where N is the number of qualities being compared.

In our example, this value is 14. After processing the preference matrix, it is clear that a quality such as “sociability” has an absolute weight of 5, “commitment” - 9, “punctuality” - 4, etc.

Then it is necessary to determine the total absolute weight of all qualities of the preference matrix using the formula:

In our example = 8 (8 - 1) = 56.

If several experts take part in the examination, then the average weight of each quality should be calculated using the formula:

Where? = ? + ? + … + ?; k - number of experts; and 1, 2…n is the current line number (quality sequence number).

And finally, it is easy to calculate the relative weight of each quality using the formula:

= [ / N (N - 1)] ?100%, if several experts participate in the examination, and

100% if one expert participates in the examination.

In our case, we have only one expert and, therefore, the value k=1. Hence, the relative weight corresponding to the quality “sociability” is equal to (5/56) ? 100% = 8.9%; quality “commitment” - (9/56)? 100% = 16.2%; quality “punctuality” - (4/56) ? 100% = 7.1%; quality “balance” - (8/56) ? 100% = 14.3%; quality “work experience” - (5/56) ? 100% = 8.9%; quality “fairness” - (11/56) ? 100% = 19.6%; quality “competence” - (14/56) ? 100% = 25.0%.

Stage No. 3. Analysis of the examination results. Knowing the relative weight of each quality, you can rank them, placing them in order of increasing importance. Another important question, which should also be answered using the results obtained, what are the qualities without which a candidate cannot occupy the proposed position under any circumstances? To find the boundary between necessary and sufficient qualities, it is recommended to use a boundary coefficient equal to 4/3n (V.V. Muzychenko, 2003). So, if n qualities, then the boundary will pass through a weight equal to (4/3) n. In our case, this value is 11 and, therefore, qualities such as “competence” and “fairness” are necessary for the head of a travel company.

Among the more complex examination methods, the “Delphi Method”, “Brainstorming Method”, “Method 6.3.5” have become widespread. and some other techniques.

Delphi method. Its name comes from the ancient Greek city of Delphi, where, according to legend, at the temple of Apollo in the period from the 9th century. BC. according to IV century AD there was a council of wise men (“the Delphic oracle”), famous for its predictions. The essence of the method is to develop consensus opinions by repeatedly repeating an individual written survey of the same experts. After the first round of the survey, all answers are analyzed and summarized to each expert. Then, after each round, the survey data is processed again, and the results are reported to the experts, indicating the location of the ratings. The first round of the survey is conducted without argumentation. In the second, answers that differ from others are subject to argumentation, or the expert can change the assessment. After the assessments have stabilized, the survey stops and the decision proposed by the experts is adopted (A. Durovich, L. Anastasova, 2002).

“Brainstorming” is one of the main principles in organizing and conducting an examination (B.G. Litvak, 1996; E.M. Korotkov, 2003; E.R. Yakhontov, 2006). Brainstorming usually consists of two rounds. In the first round, ideas are generated, and in the second round, the identified ideas are discussed, evaluated and a collective point of view is developed.

The first round is conducted so that each of the experts can freely express their opinion. Any point of view or idea expressed must be discussed and cannot be declared false. The main task of the first round is to obtain, perhaps, a more complete picture of the factors that may influence the development of the situation. In the second round, of the factors identified in the first round, only the most significant ones must be retained. To do this, they need to be critically assessed, so the experts participating in the second stage are divided into supporters and opponents of the expressed opinion. Supporters try to provide the necessary evidence in favor of the expressed point of view, and opponents try to refute them. Then, based on the results of the discussion, a final decision is made.

The method of qualitative expert assessments, as we see, has many options, one of which is the “6. 3.5". Its essence is that 6 experts offer 3 options for solving the problem under study in 5 minutes. The experts write their answer on special forms (Table 4). Five minutes later, the next six experts are invited for the same procedure. Thus, in half an hour you can get 108 new offers.

Table 4. An example of an expert form filled out when conducting an examination using the “6. 3.5"

Problem under investigation
Option No.	Suggestions for solving the problem by an expert group
First expert of the group	Second expert		Fourth		group expert
Second option
Third option

Numerous proposals accumulated in a short time are then carefully analyzed, and specialist experts draw conclusions and make specific proposals on the issue raised.

Sometimes the specificity of the objects of expert assessment is such that experts can give a quantitative assessment of individual indicators. In these cases, the use of methods for quantitative assessment of objects of examination is more justified. Among the methods for obtaining quantitative expert estimates, the most commonly used are “Direct Quantitative Assessment”, “Midpoint Method” and “Churchman-Akoff Method”.

Direct quantitative assessment is used when it is necessary to determine the value of an indicator that is measured quantitatively. In this case, each of the experts directly indicates the value of the indicator for the object being assessed. For example, the estimated capacity of the tourism market is assessed; unit price at which it will have competitive demand; optimal production volume; company value, etc.

Midpoint method. The method is used when there are quite a lot of alternative options that can be assessed quantitatively. In this case, if by f () we denote the assessment of the first alternative option, and by f () - the assessment of the second alternative, then the expert is then asked to determine the third alternative, the assessment of which f () is located in the middle between the values ​​of f () and f () and equals f () + f ()/2. Next, the expert indicates an alternative option, the value of which is located in the middle between f () and f (), then the option, the value of which f () is located in the middle between the values ​​of f (and f (). The procedure ends when the comparative preference of all alternatives participating in the examination is determined options.

Churchman-Akoff method. This method is used to quantify the comparative preference of alternative options and allows for adjustments to the assessments given by experts. All alternative options are ranked by preference, and the expert assigns quantitative ratings to each of them, usually in fractions of one, while the total sum of ratings for alternative options should be equal to 1 (or 100%). Next, the expert compares, in terms of preference, the assessment of the first alternative option f () and the sum of the remaining alternative options. If the first option is preferable to the sum of the remaining alternative options, then it is excluded from further consideration. When less preferable than the sum of the remaining alternatives, it is compared with the sum of the alternatives, excluding the last one. If an alternative option at some step turns out to be preferable to the sum of the remaining alternative options, then it is excluded from further consideration. This process continues until all alternatives have been reviewed sequentially, and estimates can be adjusted accordingly.

As we see, tourism covers many areas of human life, and therefore the study of tourism is closely related to human psychology. In our opinion, the need to use psychodiagnostic methods in tourism was most fully substantiated in his work by M.B. Birzhakov (1999, p. 181): “What travel destination will the tourist choose, which country will be preferable this season, what type of tourism will be the most popular? What is profitable to offer on the tourism market, where should you direct your activity in promoting your tourism product? How to structure a tour to best satisfy the client’s wishes? Many of these questions cannot be answered without studying human psychology and the preconditions for motivating actions and making decisions.”

Expert assessment methods is a way of predicting and assessing future results of actions based on specialist forecasts.

When applying the expert assessment method, a survey of a special group of experts (5-7 people) is carried out in order to determine certain variables necessary to assess the issue under study. Experts should include people with different types of thinking - figurative and verbal-logical, which contributes to the successful solution of the problem.

Involved experts can express their opinions on the best ways to mobilize reserves, attract investments, deadlines for achieving set goals, criteria for selecting optimal solution options, and the like.

A necessary condition for the effective use of expert assessment methods is the expert’s sufficient knowledge of the problem under study, a high level of erudition, and his ability to give clear, comprehensive answers, and also impromptu. In addition, the expert should not be interested in one or another solution to the problem posed to him. Experts are selected on the basis of their formal professional status - position, academic degree, work experience, etc. This selection contributes to the fact that the number of experts includes highly professional specialists with extensive practical experience in this field.

Thus, expert assessment methods require careful training of experts, whose work contains:

1) a clear definition of goals and objectives, and in some cases, combining and systematizing conclusions;

2) recruitment of sufficiently competent independent experts in the field of relevant objects;

3) discussing the issue in a group of experts or excluding direct communication between them;

4) providing the participants of the examination at each next stage with the results and conclusions of the previous stage. This allows us to draw certain conclusions that are shared by most experts;

5) selection of optimal suitable methods for processing expert conclusions;

6) precise formulation of the final conclusions in the expert work.

The expert assessment method is actually a forecasting method, the fundamental criterion of which is to achieve agreement among all members of the expert group. Organizationally it looks like this. Experts familiar in interrelated fields of activity answer in detail the questions of the questionnaire related to the problem being studied. Each of them records their opinion about the problem and then reports the answer to their colleagues. If his forecast differs from the opinions of others, the expert is obliged to explain the reason for such discrepancy. The procedure is then repeated until the experts’ opinions coincide. In this case, it is necessary to maintain anonymity, which helps to avoid the possibility of group reflection on a problematic situation.

Thanks to the use of expert assessments, two types of information are obtained, on the basis of which two types of problems of varying significance and at different levels of management are solved:

1. Information about single cause-and-effect relationships in specific conditions of place and time. Basically, this information is obtained as a result of a survey of the heads of production departments of the enterprise (foremen, department head, shop manager) and workers. It is intended to find ways to improve the efficiency of production and sales of products by identifying the causes of unproductive use of resources and developing effective measures to eliminate them.

2. Information about the typical relationships of the studied economic phenomena and processes. Such information can only be provided by high-class experts, professionals who deeply know the essence and patterns of manifestation of these phenomena in various economic conditions.

The main tasks that are most often solved in practice on the basis of information received from experts are:

Ranking (ordering, placed in ascending or descending order) of factors and corresponding indicators that characterize, according to their significance in the development of the phenomenon or process under study;

Ranking of enterprises or their structural production units (teams, workshops, sections) according to a rating based on a set of various indicators characterizing the results of financial and economic activity or its individual types (financial condition, profitability, solvency, etc.);

Preliminary assessment of plan implementation for a certain indicator.

Targeted analysis, based on the results of expert assessments, is carried out in several stages:

1. Determining the purpose of the study.

2. Determination of the necessary quantitative and qualitative composition of the group.

3. Create a group.

4. Determination of the survey method.

5. Drawing up a survey program and questionnaire (sheet).

6. Conducting a survey.

7. Information, grouping and analysis of information received from experts.

8. Summarizing the results of the examination and developing possible solutions to achieve the goal.

All expert methods are divided into two groups - individual and collective - and subgroups (Fig. 14.3).

Individual expert methods- this is the use of expert opinions, which are formulated personally by each of them independently, without taking into account the opinions of other experts. Individual expert methods include: interviews and questionnaires.

The essence of the interview method is to organize an interview between an analyst and an expert, during which the expert answers the analyst’s question about the factors influencing the object under study, expected business results, unused reserves, ways out of the crisis, areas for increasing production efficiency, and the like.

The survey method (analytical expert assessment) involves the expert providing written answers to the questions in the questionnaire. However, this method has certain disadvantages, in particular, the expert may not understand the questions of the questionnaire, show subjectivity, reluctance to criticize the management and leave his written answer, etc.

Rice. 14.3. Based on types of expert assessment methods

The main advantages of individual methods of expert assessments are the simplicity of organizing the survey, clarity, taking into account and using the acquired knowledge and experience of each expert. The limitation of the use of these methods is the limited knowledge and information of experts from related fields of activity. Based on this, collective expert methods have become more widespread in practice.

Collective expert methods- these are methods that ensure the formation of a single common opinion as a result of the interaction of involved expert specialists.

Among the collective methods of expert assessment there are: the commission method (including holding production meetings, conferences, seminars, round table discussions), Delphi methods, detached assessment, conference of ideas, etc.

The commission’s method is for experts to develop the best option for achieving the goal, taking into account all the proposals and ideas expressed at the meeting.

A positive feature of this method is the ability to attract specialists with a wide range of knowledge from related fields of science and practice for examination. Subjectivism, existing stereotypes of thinking that experts have developed, and their tendency to compromise can be negative.

The detached evaluation method consists in choosing the optimal independent solution from among those expressed by experts at the meeting. The work of the meeting is divided into two parts: putting forward ideas and their critical analysis.

Delphi method- one of the methods of collective expert assessment, which involves conducting an expert survey among a group of specialists in several rounds (usually 3-4 rounds) to select the best solution. The Delphi method, or as it is also called the Delphic method, the method of the Delphic oracle, got its name from the name of the town of Delphi in Ancient Greece, where the seer oracles lived at the temple of the god Apollo. The word of the main oracle was not subject to doubt and was accepted as truth.

The purpose of using the Delphi method is to improve the group approach to solving the problem of developing a forecast and assessment through mutual criticism of the views of individual specialists, expressed without direct contact between them and while maintaining the anonymity of opinions or arguments in their defense.

In one variation of this method, direct discussion is replaced by the exchange of information using specially designed questionnaires. It is also possible to use special survey techniques via a computer.

According to the Delphi method, participants are asked to express their thoughts, justify them, and in each subsequent round of the survey they are given new, updated information about the thoughts expressed, which is obtained as a result of calculating the coincidence of opinions on previously completed stages of work. This process continues until there is almost complete agreement of opinions. After this, thoughts that do not coincide are recorded.

This method is successfully used in marketing. It is used to make expert forecasting by organizing a system for collecting and mathematically processing expert assessments.

An idea conference is similar to a brainstorming session, but differs from it in the pace of the meetings and in allowing short, friendly criticism of ideas in the form of remarks and comments. At the same time, the combination of several proposals and imagination are stimulated, which helps to improve the quality of ideas.

All ideas put forward are recorded in a protocol without indicating their authors. The participants in the conference of ideas include not only highly qualified specialists, but also newcomers, non-specialists - unbiased and capable of putting forward fresh, new, extraordinary approaches.

Thus, expert assessment methods play an important role in economic research, especially in conducting strategic and functional cost analysis. The use of these methods makes it possible to determine, for example, the volume and structure of consumption of food, goods or services by the population for a significant range of indicators, while the use of other methods of analysis is difficult due to the lack of necessary information.

In practical marketing research, the method of expert assessments can be used to develop medium- and long-term forecasts of the structure of demand for consumer goods; forecasting the specified structure for the next year; identification of groups of potential consumers; as well as to assess the volume of unmet demand by groups and types of goods. For example, the method of expert assessment of the consumer value of a product and its price is a method from the group of normative-parametric pricing methods. It is based on the results of a survey or the results of judgments of a team of experts about the possible value of a product on the market, demand for it and proposals for its price.

There are also many methods of expert assessment related to forecasting economic and social phenomena and processes.

3. METHODS OF DECISION MAKING

3.4. EXPERT DECISION MAKING METHODS

3.4.1. Basic ideas of expert assessment methods

Examples of expert assessment methods. How will the economic environment change over time? What will happen to the natural environment in ten years? How will the environmental situation change? Will the environmental safety of industrial production be ensured, or will a man-made desert begin to spread around? It is enough to think about these formulations of natural questions, to analyze how ten or even more so twenty years ago we imagined today to understand that there simply cannot be one hundred percent reliable forecasts. Instead of statements with specific numbers, you can only expect qualitative estimates. Nevertheless, we, managers, economists, engineers, must make decisions, for example, about environmental and other projects and investments, the consequences of which will be felt in ten, twenty, etc. years. What should I do? It remains to turn to the methods of expert assessments. What are these methods?

It is absolutely indisputable that in order to make informed decisions it is necessary to rely on the experience, knowledge and intuition of specialists. After the Second World War, within the framework of cybernetics, control theory, management and operations research, an independent discipline began to develop - the theory and practice of expert assessments.

Methods of expert assessments are methods of organizing work with expert specialists and processing expert opinions. These opinions are usually expressed partly in quantitative and partly in qualitative form. Expert research is carried out with the aim of preparing information for decision-making by the decision maker (remember, the decision maker is the decision maker). To carry out work using the method of expert assessments, a Working Group (abbreviated as WP) is created, which organizes, on behalf of the decision-maker, the activities of experts united (formally or in essence) into an expert commission (EC).

Expert assessments are individual And collective. Individual assessments- These are the assessments of one specialist. For example, a teacher single-handedly gives a mark to a student, and a doctor gives a diagnosis to a patient. But in complex cases of illness or the threat of expulsion of a student for poor studies, they turn to collective opinion - a symposium of doctors or a commission of teachers. The situation is similar in the army. Usually the commander makes the decision alone. But in difficult and responsible situations, a military council is held. One of the most famous examples of this kind is the military council of 1812 in Fili, at which, under the chairmanship of M.I. Kutuzov, the question was decided: “Should we or should we not give the French a battle near Moscow?”

Another simple example of expert assessments is the assessment of numbers in KVN. Each of the jury members raises the plywood with their score, and the technical worker calculates the arithmetic average score, which is announced as the collective opinion of the jury (we will see below that this approach is incorrect from the point of view of measurement theory).

In figure skating, the procedure becomes more complicated - before averaging the largest and smallest scores are discarded. This is done so that there is no temptation to overestimate one athlete (for example, a compatriot) or underestimate another. Such assessments that stand out sharply from the general series will be immediately discarded.

Expert judgments are often used in selection, for example:

One version of a technical device to be launched into a series of several samples,

Groups of astronauts from many applicants,

A set of research projects for funding from a mass of applications,

Recipients of environmental loans from among many who wish,

When choosing investment projects for implementation among those presented, etc.

There are many methods for obtaining expert assessments. In some, they work with each expert separately; he does not even know who else is an expert, and therefore expresses his opinion regardless of authorities. In others, experts are brought together to prepare materials for decision-makers, and the experts discuss the problem with each other, learn from each other, and incorrect opinions are discarded. In some methods, the number of experts is fixed and such that statistical methods of checking the consistency of opinions and then averaging them make it possible to make informed decisions. In others, the number of experts grows during the examination process, for example, when using the “snowball” method (more on this later).

There are no less methods for processing expert responses, including those that are very mathematical and computerized. Many of them are based on the achievements of statistics of objects of non-numerical nature and other modern methods of applied statistics.

One of the most well-known methods of expert assessments is Delphi method. The name is given by association with the ancient custom of turning to the Delphic Temple to receive support when making decisions. It was located at the outlet of poisonous volcanic gases. The priestesses of the temple, having inhaled the poison, began to prophesy, uttering incomprehensible words. Special “translators” - the priests of the temple - interpreted these words and responded to the questions of the pilgrims who came with their problems. According to tradition, they say that the Delphic Temple was located in Greece. But there are no volcanoes there. Apparently, he was in Italy - near Vesuvius or Etna, and the described predictions themselves took place in the 12th-14th centuries. This follows from the highest achievement of modern historical science - the new statistical chronology.

In the USA in the 1960s, the Delphi method was called an expert procedure for forecasting scientific and technological development. In the first round, experts named probable dates for certain future achievements. In the second round, each expert got acquainted with the forecasts of all the others. If his forecast was very different from the forecasts of the general population, he was asked to explain his position, and often he changed his estimates, moving closer to the average values. These average values ​​were given to the customer as a group opinion. It must be said that the actual results of the study turned out to be quite modest - although the date of the American landing on the moon was predicted with an accuracy of up to a month, all other forecasts failed - cold thermonuclear fusion and a cure for cancer in the twentieth century. humanity did not wait.

However, the technique itself turned out to be popular - over the following years it was used at least 40 thousand times. The average cost of an expert study using the Delphi method is 5 thousand US dollars, but in some cases it was necessary to spend larger sums - up to 130 thousand dollars.

Somewhat outside the mainstream of expert assessments lies script method, used primarily for expert forecasting. Let's consider the main ideas of the technology of scenario expert forecasts. Environmental or socio-economic forecasting, like any forecasting in general, can be successful only under some stability of conditions. However, decisions of authorities, individuals, and other events change conditions, and events develop differently than previously expected. It is quite obvious that after the first round of the 1996 presidential elections, further developments of events could only be discussed in terms of scenarios: if B.N. wins in the second round. Yeltsin, then this and that will happen if G.A. wins. Zyuganov, then events will go this way and that way.

The scenario method is necessary not only in the socio-economic or environmental field. For example, when developing methodological, software and information support risk analysis chemical technology projects, it is necessary to compile a detailed catalog of accident scenarios associated with leaks of toxic chemicals. Each of these scenarios describes an accident of its own type, with its individual origin, development, consequences, and prevention capabilities.

Thus, the scenario method is a method of decomposing the forecasting problem, which involves identifying a set of individual options for the development of events (scenarios), collectively covering all possible development options. Moreover, each individual scenario must allow for the possibility of fairly accurate forecasting, and the total number of scenarios must be foreseeable.

The possibility of such a decomposition is not obvious. When applying the scenario method, it is necessary to carry out two stages of research:

Construction of a comprehensive but manageable set of scenarios;

Forecasting within each specific scenario in order to obtain answers to questions of interest to the researcher.

Each of these stages is only partially formalized. A significant part of the reasoning is carried out at a qualitative level, as is customary in the socio-economic and human sciences. One of the reasons is that the desire for excessive formalization and mathematization leads to artificial introducing certainty where there is essentially none, or using cumbersome mathematical apparatus. Thus, reasoning at the verbal level is considered evidential in most situations, while an attempt to clarify the meaning of the words used using, for example, the theory of fuzzy sets leads to very cumbersome mathematical models.

The set of scenarios should be visible. We have to exclude various unlikely events - the arrival of aliens, the fall of an asteroid, mass epidemics of previously unknown diseases, etc. The creation of a set of scenarios in itself is a subject of expert research. In addition, experts can assess the likelihood of a particular scenario occurring.

Forecasting within each specific scenario in order to obtain answers to questions of interest to the researcher is also carried out in accordance with the forecasting methodology described above. Under stable conditions, statistical methods for time series forecasting can be applied. However, this is preceded by analysis with the help of experts, and often forecasting at the verbal level is sufficient (to obtain conclusions of interest to the researcher and decision maker) and does not require quantitative clarification.

As is known, when making decisions based on situation analysis(as they say, when situational analysis), including analysis of the results of predictive studies, can be based on various criteria. So, you can focus on the fact that the situation will turn out in the worst, or best, or average (in some sense) way. You can try to outline measures that provide the minimum acceptable useful results in any scenario, etc.

Another option for expert assessment is brainstorm. It is organized as a meeting of experts, whose speeches are subject to one, but very significant restriction - you cannot criticize the proposals of others. You can develop them, you can express your ideas, but you can’t criticize them! During the meeting, experts, “infecting” each other, express more and more extravagant ideas. About two hours later, the meeting recorded on a tape recorder or video camera ends, and the second stage of brainstorming begins - analysis of the ideas expressed. Typically, out of 100 ideas, 30 deserve further development, out of 5-6 they make it possible to formulate applied projects, and 2-3 ultimately bring a useful effect - profit, increased environmental safety, improvement of the natural environment, etc. Moreover, the interpretation of ideas is a creative process. For example, when discussing the possibilities of protecting ships from a torpedo attack, the idea was put forward: “Line the sailors along the side and blow on the torpedo to change its course.” After development, this idea led to the creation of special devices that create waves that knock the torpedo off course.

The main stages of the expert survey. Let's look at the individual stages of expert research in more detail. As experience shows, from the point of view of the manager - organizer of such a study, it is advisable to distinguish the following stages of conducting an expert survey.

1) Making a decision on the need to conduct an expert survey and formulating its goals by the Decision Maker (DM). Thus, the initiative must come from management, which will further ensure the successful solution of organizational and financial problems. Obviously, the initial impetus can be given by a memo from one of the employees or a discussion at a meeting, but the real start of the work is the decision of the decision-maker.

2) Selection and appointment of decision-makers of the main composition of the Working Group, abbreviated as RG (usually scientific supervisor and secretary). In this case, the scientific supervisor is responsible for organizing and conducting the expert research as a whole, as well as for analyzing the collected materials and formulating the conclusion of the expert commission. He participates in forming a team of experts and issuing tasks to each expert (together with the decision-maker or his representative). He himself is a highly qualified expert and recognized by other experts as the formal and informal leader of the expert commission. The secretary's job is to maintain documentation of the expert survey and solve organizational problems.

3) Development RG(more precisely, its main staff, primarily the scientific director and secretary) and approval by the decision maker of the technical specifications for conducting an expert survey. At this stage, the decision to conduct an expert survey becomes clear in terms of time, financial, personnel, material and organizational support. In particular, a Working Group is being formed; various groups of specialists are allocated in the WG - analytical, econometric (method specialists), computer, working with experts (for example, interviewers), organizational. It is very important for success that all these positions are approved by the decision maker.

4) Development by the WG analytical group of a detailed scenario (i.e. regulations) for collecting and analyzing expert opinions (assessments). The scenario includes, first of all, a specific type of information that will be received from experts (for example, words, conditional gradations, numbers, rankings, partitions or other types of objects of a non-numerical nature). For example, quite often experts are asked to speak freely, while answering a number of pre-formulated questions. In addition, they are asked to fill out a formal card, selecting one of several gradations at each point. The script must also contain specific methods for analyzing the collected information. For example, calculating the Kemeny median, statistical analysis of Lucians, the use of other methods of statistics of objects of non-numerical nature and other sections of applied statistics (some of these methods will be discussed below). This work falls on the econometric and computer group of the RG. A traditional mistake is to first collect information and then think about what to do with it. As a result, as sad experience shows, no more than 1-2% of information is used.

5) Selection of experts according to their competence. At this stage, the WG compiles a list of possible experts and assesses their suitability for the planned study.

6) Formation of an expert commission. At this stage, the WG negotiates with experts and obtains their consent to work in the expert commission (abbreviated as EC). It is possible that some of the experts identified by the WG cannot join the expert commission (illness, vacation, business trip, etc.) or refuse for one reason or another (employment, contract conditions, etc.). The decision maker approves the composition of the expert commission, possibly deleting or adding some experts to the WG’s proposals. Agreements are concluded with experts on the conditions of their work and their payment.

7) Conducting collection of expert information. Often, this is preceded by the recruitment and training of interviewers - one of the groups included in the RG.

8) Computer analysis of expert information using the methods included in the script. It is usually preceded by entering information into computers.

9) When applying an expert procedure from several rounds according to the scenario - repetition two previous stages.

10) Final analysis of expert opinions, interpretation of the results obtained analytical group of the WG and preparation of the final document EC for decision makers.

11) Official ending activities of the RG, including approval by the decision maker of the final document of the EC, preparation and approval of scientific and financial reports of the Working Group on the conduct of expert research, remuneration of experts and employees of the Working Group, official termination of the activities (dissolution) of the EC and the Working Group.

Let us examine in more detail the individual stages of expert research. Let's start with the selection of experts: personnel is everything! The quality of the expert commission's conclusion is the same as the experts.

Selection of experts. The problem of selecting experts is one of the most difficult in the theory and practice of expert research. Obviously, it is necessary to use as experts those people whose judgments will most help in making an adequate decision. But how to identify, find, select such people? It must be said frankly that There are no methods for selecting experts that will surely ensure the success of the examination. Now we will not discuss the problem of the existence of various “parties” among experts and will pay attention to various other aspects of the procedures for selecting experts.

The problem of selecting experts can be divided into two components: compiling a list of possible experts and selecting an expert commission from them in accordance with the competence of the candidates.

Compiling a list of possible experts is easier when the type of examination in question is carried out repeatedly. In such situations it is usually carried out registry possible experts, for example, in the field of state environmental assessment or figure skating judging, from which you can select according to various criteria or using a pseudo-random number sensor (or table).

What to do if the examination is carried out for the first time and there are no established lists of possible experts? However, even in this case, each specific specialist has some idea of ​​what is required from an expert in such a situation. There is a useful method for generating a list "snowball" in which from each specialist involved as an expert, they receive a certain number (usually 5 - 10) of names of those who may be an expert on the topic under consideration. Obviously, some of these names were encountered earlier in the activities of the RG, and some are new. Each new person is interviewed according to the same scheme. The process of expanding the list stops when new names practically cease to appear. The result is a fairly extensive list of possible experts. Method "snowball" It also has disadvantages. The number of rounds before the coma growth process stops cannot be predicted in advance. In addition, it is clear that if at the first stage all the experts were from the same “clan”, held somewhat similar views or were engaged in similar activities, then the “snowball” method will most likely yield people from the same “clan” . The opinions and arguments of other "clans" will be missed. (Here we are talking about the fact that the community of specialists is actually divided into groups, called above “clans”, and communication takes place mainly within the “clans”. The informal structure of science, to which “clans” belong, is quite difficult to study. Let us note here that that “clans” are usually formed on the basis of large formal centers (universities, scientific institutes, scientific schools.)

The issue of assessing the competence of experts is no less complex. It is clear that successful participation in previous examinations is a good criterion for the activities of a taster, a doctor, a judge in sports competitions, i.e. such experts who participate in long series of similar examinations. However, alas, the most interesting and important are the unique examinations of large projects that have no analogues. The use of formal indicators of experts (position, academic degree and title, length of service, number of publications...), obviously, in modern rapidly changing conditions can only be of an auxiliary nature, although such indicators are the easiest to use.

It is often proposed to use methods of self-assessment and mutual assessment of the competence of experts. Let's discuss them, starting with the self-assessment method, in which the expert himself provides information about in which areas he is competent and in which he is not. On the one hand, who can know the capabilities of an expert better than himself? On the other hand, when self-assessing competence, the degree of self-confidence of the expert is assessed rather than his actual competence. Moreover, the very concept "competence" not strictly defined. It can be clarified by highlighting its components, but this complicates the preliminary part of the expert commission’s activities. Quite often, an expert exaggerates his actual competence. For example, most people believe that they are well versed in politics, economics, problems of education and upbringing, family and medicine. In fact, there are very few experts (and even knowledgeable people) in these areas. There are also deviations in the other direction, an overly critical attitude towards one’s capabilities.

When using the mutual assessment method, in addition to the possibility of showing personal and group likes and dislikes, the experts’ low awareness of each other’s capabilities plays a role. In modern conditions, only specialists who have been working together for many years (at least 3-4), in the same room, on the same topic, can have a fairly good acquaintance with each other’s work and capabilities. It is about such couples that we can say that they " together we ate a pound of salt"However, attracting such pairs of specialists is not very advisable, since their views, due to the similarity of their life paths, are too similar to each other.

If the expert survey procedure involves direct communication between experts, a number of other circumstances must be taken into account. Their personal (socio-psychological) qualities are of great importance. Yes, the only one" talker"can paralyze the activities of the entire commission at a joint meeting. Hostile relations between the members of the commission and the greatly different scientific and official status of the members of the commission can lead to a breakdown. In such cases, it is important to comply with the work regulations developed by the WG.

It must be emphasized that the selection of experts is one of the main functions of the Working Group, and no selection methods relieve it of responsibility. In other words, it is the Working Group that is responsible for the competence of the experts, for their fundamental ability to solve the task. An important requirement is for the decision maker to approve the list of experts. At the same time, the decision maker can either add individual experts to the commission or delete some of them - for his own reasons, which the members of the Working Group and the Executive Committee do not need to get acquainted with.

There are a number of normative documents regulating the activities of expert commissions in certain areas. An example is the Law of the Russian Federation “On Environmental Expertise” of November 23, 1995, which regulates the procedure for the examination of “planned economic or other activities” in order to identify possible harm that the activity in question may cause to the natural environment.

On the development of regulations for the collection and analysis of expert opinions. There are many methods for obtaining expert assessments. In some, they work with each expert separately; he does not even know who else is an expert, and therefore expresses his opinion regardless of authorities, “clans” and individual colleagues. In others, experts are brought together to prepare materials for decision-makers, with experts discussing the problem with each other, accepting or rejecting each other's arguments, learning from each other, and incorrect or insufficiently substantiated opinions being discarded. In some methods, the number of experts is fixed and such that statistical methods of checking the consistency of opinions and then (in the case of a sufficiently good agreement of opinions) averaging them allow making informed decisions from an econometric point of view. In others, the number of experts grows during the examination process, for example, when using the “snowball” method to form a team of experts.

Currently does not exist a generally accepted scientifically based classification of expert assessment methods and, even more so, clear recommendations for their use. An attempt to forcefully approve one of the possible points of view on the classification of expert assessment methods can only bring harm.

However, to talk about the diversity of expert assessments, some kind of working classification of methods is necessary. We give one of these possible classifications below, listing the grounds on which we divide expert assessments.

One of the main questions is what exactly should the expert commission present as a result of its work - information for the decision-maker to make a decision or a draft decision itself? The organization of the work of the expert commission depends on the answer to this methodological question, and it serves as the first basis for dividing the methods.

GOAL - COLLECTION OF INFORMATION FOR DECISION MANAGERS. Then the Working Group should collect as much relevant information as possible, arguments for and against certain decision options. The following method of gradually increasing the number of experts is useful. First, the first expert gives his thoughts on the issue under consideration. The material compiled by him is transferred to the second expert, who adds his arguments. The accumulated material goes to the next - third - expert... The procedure ends when the flow of new considerations dries up.

Let us note that in the method under consideration, experts only provide information and arguments for and against, but do not develop an agreed draft decision. There is no need to strive to ensure that expert opinions are consistent with each other. Moreover, the greatest benefit comes from experts with a mindset that deviates from the mainstream. It is from them that one should expect the most original arguments.

GOAL - PREPARATION OF A DRAFT SOLUTION FOR DECISION MANAGER. Mathematical methods in expert assessments are usually used specifically to solve problems related to the preparation of a draft solution. At the same time, the dogmas of consistency and one-dimensionality are often uncritically accepted. These dogmas “wander” from one publication to another, so it is advisable to discuss them.

DOGMA OF CONSISTENCY. It is often believed, without any justification, that a decision can only be made on the basis of consensus expert opinions. Therefore, those whose opinion differs from the majority opinion are excluded from the expert group. At the same time, both unqualified persons who were included in the expert commission due to a misunderstanding or for reasons not related to their professional level, as well as the most original thinkers who penetrated deeper into the problem than the majority, are eliminated. Their arguments should be clarified and they should be given the opportunity to substantiate their points of view. Instead, their opinions are ignored.

It also happens that experts are divided into two or more groups that have common group points of view. Thus, there is a well-known example of dividing specialists when assessing the results of scientific research into two groups: “theorists” who clearly prefer research work in which theoretical results were obtained, and “practitioners” who choose those research projects that allow obtaining direct applied results (we are talking about research competition at the Academic Institute of Control Problems (automation and telemechanics)).

It is sometimes claimed that if two or more groups of experts are found (instead of one that agrees in their opinions), the survey has failed to achieve its goal. This is wrong! The goal has been achieved - it has been established that there is no consensus. This is quite important. And the decision maker must take this into account when making decisions. The desire to ensure consistency in the opinions of experts of any kind can lead to a deliberate one-sided selection of experts, ignoring all points of view except one, the one most beloved by the Working Group (or even “prompted” by the decision-maker).

Another purely econometric circumstance is often not taken into account. Since the number of experts usually does not exceed 20-30, the formal statistical consistency of expert opinions (established using certain criteria for testing statistical hypotheses) can be combined with the actual division of experts into groups, which makes further calculations irrelevant to reality. As an example, let's turn to specific calculation methods using concordance coefficients (i.e., translated as agreement) based on Kendall's or Spearman's rank correlation coefficients. It must be recalled that, according to econometric theory, a positive result of testing consistency in this way means nothing more or less than a rejection of the hypothesis of independence and uniform distribution of expert opinions on the set of all rankings. Thus, the null hypothesis is tested, according to which the rankings describing the opinions of experts are independent random binary relations, uniformly distributed over the set of all rankings. Rejection of this null hypothesis is traditionally interpreted as agreement between the experts' answers. In other words, we fall victim to misconceptions arising from a peculiar interpretation of words: checking consistency in the specified mathematical-statistical sense is not at all a checking of consistency in the sense of the practice of expert assessments. (It was precisely the inadequacy of the considered mathematical and statistical methods for analyzing rankings that led a group of specialists to develop a new econometric apparatus for checking consistency - nonparametric methods based on the so-called Lucians and included in the modern section of econometrics - statistics of non-numeric data). Groups of experts with similar methods can be identified using econometric methods of cluster analysis.

OPINIONS OF DISSIDENTS. In order to artificially achieve consistency, they try to reduce the influence of expert opinions - dissidents, i.e. dissenters compared to the majority. Hard The way to deal with dissidents is to ignore their opinions, i.e. in fact, in their exclusion from the expert commission. Rejection of experts, as well as rejection of outliers, results in procedures that have poor or unknown statistical properties. Yes, it is known extreme instability classical methods for rejecting outliers in relation to deviations from the model assumptions (see, for example, the tutorial).

Soft The way to deal with dissidents is to use robust (stable) statistical procedures. The simplest example: if the expert's answer is a real number, then the dissident's opinion that stands out strongly affects the arithmetic mean of the experts' answers and does not affect their median. Therefore, it is reasonable to consider the median as the consensus opinion. However, this ignores (does not reach the decision maker) the arguments of dissidents.

In any of the two methods of dealing with dissidents, the decision maker is deprived of information coming from dissidents, and therefore can make an unfounded decision, which will subsequently lead to negative consequences. On the other hand, presenting the entire set of opinions to the decision maker removes some of the responsibility and labor for preparing the final decision from the commission of experts and the working group for conducting the expert survey and shifts this responsibility and labor onto the shoulders of the decision maker.

DOGMA OF ONE-DIMENSIONALITY. In outdated, and sometimes in modern scientific and technical literature, a rather controversial approach of the so-called “qualimetry” is common, according to which the object of examination can always be assessed one number. Strange idea! It only occurred to people in slave markets to evaluate a person using one number.. It is unlikely that even the most zealous qualimeterists consider a book or a painting as the equivalent of a number - its “market value”. Almost all real objects are quite complex, and therefore they can be described with any accuracy only with the help of many, many numbers, as well as mathematical objects of a non-numerical nature.

At the same time, one cannot completely deny the very idea of ​​​​searching for generalized indicators of quality, technical level and similar ones. Thus, each object can be assessed according to many quality indicators. For example, a passenger car can be assessed according to the following indicators:

gasoline consumption per 100 km (on average);

reliability (including the average cost of repairs per year);

environmental safety, assessed by the content of harmful substances in exhaust gases;

maneuverability (including turning radius);

speed of 100 km/h after starting to move; maximum achievable speed;

the duration of maintaining a positive temperature in the cabin at a low outside temperature (for example, minus fifty degrees Celsius) and the engine is turned off;

design (attractiveness and “fashionability” of appearance and interior decoration);

weight, etc.

Is it possible to combine the scores for these indicators together? It is clear that the specific situation for which the car is selected is decisive. The maximum speed achieved is important for the driver, but, in our opinion, is of little practical importance for the driver of an ordinary private car, especially in a city with a severe maximum speed limit. For such a driver, gas mileage, maneuverability and reliability are more important. For cars of various public administration services, reliability is apparently more important than for a private owner, and gasoline consumption is the opposite. For regions of the Far North, thermal insulation of the interior is important, but for southern regions it is not. Etc.

Thus, a specific (narrow) formulation of the task before experts is important. But such a setting often does not exist. And then “games” to develop a generalized quality indicator - for example, in the form of a linear function of the listed variables - cannot give objective conclusions. An alternative to the only generalized indicator is a mathematical apparatus like multicriteria optimization- Pareto sets, etc.

In some cases, it is still possible to compare objects globally - for example, with the help of the same experts, obtain an ordering of the objects in question - products or projects. Then you can SELECT coefficients for individual indicators so that ordering using a linear function may have more closely matched the global ordering(for example, find these coefficients using the least squares method). On the contrary, in such cases, the specified coefficients SHOULD NOT be estimated with the help of experts. This simple idea has not yet become obvious to some compilers of methods for conducting expert surveys and analyzing their results. They try hard to get experts to do what they do unable- indicate the weights with which individual quality indicators should be included in the final generalized indicator.

Experts can usually compare objects or projects as a whole, but cannot isolate the contribution of individual factors . Since the survey organizers ask, the experts answer, but these answers do not carry reliable information about reality...

THE SECOND BASIS FOR THE CLASSIFICATION OF EXPERT PROCEDURES IS THE NUMBER OF ROUND. Examinations may include one round, a certain fixed number of rounds (two, three,...) or an indefinite number of rounds. The more rounds, the more thorough the analysis of the situation is, since experts usually return to consider the subject of their examination many times. But at the same time, the total time for the examination increases and its cost increases. You can reduce costs by introducing not all experts into the examination at once, but gradually. So, for example, if the goal is to collect arguments for and against, then the initial list of arguments can be compiled by one expert. The second will add his own arguments to it. The summary material will go to the first and third, who will contribute their arguments and counterarguments. And so on - one expert is added for each new round.

The greatest difficulties are caused by procedures with a predetermined number of rounds, for example, “snowball”. Often the maximum possible number of rounds is specified, and then the uncertainty comes down to whether this maximum number of rounds will have to be carried out or whether it will be possible to limit oneself to a smaller number.

THE THIRD BASIS FOR THE CLASSIFICATION OF EXPERT PROCEDURES IS THE ORGANIZATION OF COMMUNICATION OF EXPERTS. Let's consider the advantages and disadvantages of each of the elements of the scale: lack of communication - correspondence anonymous communication - correspondence communication without anonymity - face-to-face communication with restrictions - face-to-face communication without restrictions. In the absence of communication the expert expresses his opinion without knowing anything about other experts and their opinions. He is completely independent, which is both good and bad. Typically this situation corresponds to a one-round examination . Absentee anonymous communication, for example, as in the Delphi method, means that the expert gets acquainted with the opinions and arguments of other experts, but does not know who exactly expressed this or that position. Therefore, the examination must include at least two rounds. Correspondence communication without anonymity corresponds, for example, to communication over the Internet. All options for correspondence examination are good because there is no need to gather experts together, therefore, to find a convenient time and place for this.

During face-to-face examinations, experts speak rather than write, as during absentee examinations, and therefore manage to say significantly more in the same time. In-person examination with restrictions very common. This is a meeting that follows a fixed schedule. An example is the military council in the imperial Russian army, when experts (officers and generals) spoke in order from junior (by rank and position) to senior. Finally, face-to-face examination without restrictions- this is a free discussion. All face-to-face examinations have disadvantages associated with the possibility of a negative influence on their conduct by the socio-psychological properties and clan (party) predilections of the participants, as well as the inequality of their professional, official, and scientific status. Imagine that 5 lieutenants and 3 generals get together. Regardless of what information one or another participant in the meeting has, its course is not difficult to predict: the generals will talk, and the lieutenants will remain silent. At the same time, it is quite obvious that the lieutenants received their education later than the generals, and therefore have useful information that the generals do not have.

COMBINATION OF DIFFERENT TYPES OF EXAMINATION. Actual examinations are often combinations of the various types of examinations described above. As an example, consider a student’s defense of a diploma project. First, there is a multi-round face-to-face examination conducted by the supervisor and consultants, as a result the student prepares the project for defense. Then two experts work in absentia - the author of the review from a third-party organization and the head of the department who allows the work to be defended. Pay attention to the difference in the tasks of these experts and the amount of work they perform - one writes a detailed review, the second authorizes its defense by signing on the title page of the project. Finally - a face-to-face examination without restrictions (for members of the State Attestation Commission - the State Attestation Commission). The diploma project is assessed collegiately, by majority vote, with one of the experts (scientific supervisor) knowing the work in detail, and the rest - mainly only from the report. Note that the opinions of experts are taken into account with weights, namely, the opinions of the members of the SAC - with a weight of 1, the opinions of all others - with a weight of 0 (advisory vote). Thus, we have a combination of multi-round and single-round, correspondence and face-to-face examinations. Such combinations are typical for many actual examinations.

Previous

In qualimetry, the expert method is used:

1) to measure quality indicators;

2) to determine the values ​​of the weighting coefficients.

However, it does not belong only to qualimetry. The expert method is also used in measuring physical quantities, in medicine (consiliums), in art (juries), in the socio-political sphere (referendums), in state and economic management (collegiality). But it was precisely the needs of qualimetry that put this measurement method on a strict scientific basis.

Regardless of the goals and objectives, the use of the expert method requires compliance with the following conditions:

expert assessment should be carried out only in cases where it is impossible to use more objective methods to resolve the issue;

should not be present in the work of the expert commission. Factors that could influence the sincerity of expert judgments; expert opinions must be independent;

questions posed to experts should not allow for different interpretations;

experts must be competent in the issues being resolved;

the number of experts should be optimal;

Experts' answers must be unambiguous and provide the possibility of their mathematical processing.

The qualitative composition of the expert commission is an important condition for the effectiveness of the expert method. It is quite obvious that in all cases, without exception, the examination must be carried out by competent, highly qualified, fully competent in the issues under consideration and sufficiently experienced specialists. Their special preliminary training is very useful and instruction is absolutely necessary. At the final stage of forming an expert group, it is advisable to conduct testing, self-assessment, mutual assessment of experts, analysis of their reliability and checking the consistency of opinions.

Testing consists in solving by experts problems similar to real ones, with known (but not to experts) answers. Based on the test results, the competence and professional suitability of experts is established.

Self-esteem experts consists of each of them answering the questions of a specially compiled questionnaire in a strictly limited time, as a result of which their professional knowledge and business qualities are quickly and easily tested by themselves. Their assessment is given by each expert using a point system. Despite the subjectivity of such an assessment, experience shows that expert groups with high levels of self-assessment of experts are less mistaken.

It is very indicative mutual assessment by experts of each other(also based on a point system). To do this, they must, of course, have experience working together.

If there is information about the results of the expert’s work in other expert groups, the criterion for his qualification may be index or degree of reliability- the ratio of the number of cases when the expert’s opinion coincided with the results of the examination to the total number of examinations in which he participated. Using this approach to selecting experts requires the accumulation and analysis of a large amount of information, but opens up the possibility of continuous improvement of the qualitative composition of expert groups.

Each expert gives one of the reading values, which, according to the basic postulate of metrology, is a random number. The procedure and rules for further actions are discussed in Chapter. 2. In particular, a single measurement using the expert method requires the use of a large amount of a priori information. When visual topographic surveying, for example, the expert’s eye is of great importance, when measuring aesthetic quality indicators, his artistic taste, etc. Repeated measurements of the same physical (or other) quantity of constant size or quality indicator can be organized with subsequent averaging experimental data over time (if the measurement is performed by one expert) or over a set (if the measurement is performed simultaneously by several experts). The first method is rarely used, since the subjective characteristics of the expert act in this case as permanent factors that are difficult to exclude, compensate or take into account. In the second method, they act as random and are leveled out when averaging over the set. The reading obtained by a group of experts is represented by a set of its individual values ​​or by the law of probability distribution. With a large number of individual reading values, according to the “three sigma” rule, erroneous ones are easily detected and eliminated. If the count obeys the normal probability distribution law, then its arithmetic mean with the number of experts p> 30 ... 40 also obeys the normal law, and with a smaller number - the Student probability distribution law. The interval of possible values ​​of the measured quantity or quality indicator in the vicinity of the arithmetic mean value with the selected confidence probability is established according to the graphs shown in Fig. 38.

When selecting experts, much attention is paid consistency their opinions, which is characterized by a biased or unbiased estimate of the sample dispersion. For this purpose, at the stage of forming an expert group, control measurements are carried out with mathematical processing of their results. Often, not one, but several measurement objects are used at once, which, depending on their value or quality, must be arranged on a scale of order, i.e. determine their rank, because measuring on a scale of order is called ranking. In this case, the so-called concordance coefficient.

Where S- the sum of squared deviations of the sum of ranks of each object of examination from the arithmetic mean of ranks; P- number of experts; m- number of objects of examination. Depending on the degree of agreement between experts’ opinions, the concordance coefficient can take values ​​from 0 (in the absence of agreement) to 1 (in complete unanimity).

Example 75. Determine the degree of consistency of opinions of 5 experts, the results of ranking of 7 objects of examination are given in Table. 45.

Solution.1. Arithmetic mean of ranks

2. Using the results of intermediate calculations given in Table 45, we obtain S= 630.

3. Concordance coefficient

The degree of agreement between expert opinions can be considered satisfactory.

If the degree of agreement between experts' opinions turns out to be unsatisfactory, special measures are taken to increase it. They come down mainly to conducting training sessions with discussing the results and analyzing mistakes. If there is no possibility for preliminary training of experts, measurement using the expert method is carried out according to Delphi method*. The characteristic features of this method are:

anonymity; experts do not meet with each other to avoid the influence of the authority and eloquence of any of them;

multi-stage; After each round of the survey, all experts get acquainted with each other’s opinions and, if necessary, provide written reasons for their points of view. By agreeing or disagreeing with the opinions of their colleagues, they can revise their point of view;

control; After each round, the consistency of expert opinions is checked until the spread of individual opinions is reduced to a pre-selected value.

For particularly important measurements using the expert method, the weighting coefficients of the experts’ qualifications can be taken into account.

* This method was first proposed in the early 1950s by American scientists T. J. Gordon and O. Helmer to solve military problems. Its name comes from the ancient Greek city of Delphi, where, according to legend, at the Temple of Apollo from the 9th century. BC e. to the 4th century n. e. There was a council of wise men (the “Delphic oracle”), famous for its predictions.

The number of experts also plays an important role. As the number of experts in the group increases, the accuracy of the measurement increases. This fundamental property of any multiple measurement is defined by expression (11). To use it to determine the size of the expert group n, ensuring a given measurement accuracy, it is again necessary during the preparatory period to establish the law of distribution of the probability of the reading obtained by the expert method, or at least its standard deviation, independent of n. Then according to the graph in Fig. 159, reflecting dependence (11), we can find the number of experts n, at which the standard deviation of the arithmetic mean will correspond to the required one. The initial size of the expert group is usually at least 7 people. In some cases, it reaches 15 ... 20 experts (mass surveys are carried out, as a rule, only in sociological research). If it is not determined during the preparatory period, then achieving the required accuracy by expanding the expert group is achieved already in the process of measurement using the expert method, as shown in Fig. 39.

In some cases, it is necessary to ensure the highest possible measurement accuracy using an expert method. In these cases, it is advisable to limit the composition of the expert group to this number of experts P, in which the differences between the arithmetic means and estimates of the variances of the measurement results at n And n+ 1 experts cease to be significant. These conditions are checked using the algorithms shown in Fig. 41 and 43.

According to the form in which experts express their opinions, i.e. According to the method of conducting the examination, they are distinguished:

direct measurement;

ranging;

comparison.

At direct measurements Using the expert method, the values ​​of physical quantities or quality indicators are determined immediately in established units (either in SI units, or in points, standard hours, rubles, standard fuel units, etc.). Such measurements can be carried out either on a ratio scale or on an interval or order scale. Ratio scale measurements require standards. These include organoleptic methods for measuring length, mass, light intensity and many others. Direct measurement of weight coefficients, the sum of which must be equal to unity, is carried out on an order scale. The values ​​of these coefficients are calculated using the formula

Where P - number of experts; m- number of “weighted” indicators; - weight coefficient of the j-th indicator in points, given by the i-th expert.

Using reference order scales, the strength of sea waves, the strength of earthquakes, etc. are measured in points. Directly by assigning points (usually from 1 to 10), properties for which there are neither standards nor objective criteria can be measured on an order scale. In the latter case, no quantitative conclusions can be drawn from the score ratio.

Direct measurement by the expert method is the most complex and places the highest demands on experts.

Ranging consists of arranging measurement objects or indicators in order of their preference, importance or weight. The place occupied in this arrangement is called rank. The higher the rank, the more preferable the object, the more weighty, the more important the indicator.

An example of ranking by five experts of seven objects of examination is given in Table. 45. If these are, say, works of art, then the result of measuring their quality on an order scale is as follows:

the best is the seventh, the second best is the fourth, then the sixth, first, second, third and fifth. If the ranking was carried out to determine the weight coefficients g i for seven quality indicators, they are calculated using formula (53), in which - rank j- th indicator established i-th expert, In example 75

The comparison can be sequential or pairwise. Sequential matching everyone. The object of examination with the totality of all those that are lower in rank allows you to adjust the ranked series, clarify the positions of the objects included in it, taking into account their importance. It makes sense when several objects of examination can be considered as one composite object of the same nature. The order of sequential comparison is as follows.

1. Objects of examination are arranged in order of their preference (ranking).

2. The most important object is assigned a score or weight equal to 1; all others, in order of decreasing relative importance, are given points or weights from 1 to 0.

3. The first object is compared with the totality of all the others. If, in the expert’s opinion, it is preferable to the sum of all the others taken together, then the result of its measurement in points or weight coefficient is adjusted upward so that it becomes greater (sometimes it is determined how much more) the sum of points or weight coefficients all other objects of examination that are of lower rank. Otherwise, the measurement result or weight coefficient of the first object is adjusted downwards so that it is less than the sum of the points or weight coefficients of the remaining objects.

4. The second object is compared with the totality of all the others that are of lower rank. According to the rule established above, the result of its measurement or the value of the weighting coefficient is adjusted (in this case, care must be taken not to violate the preference of the first object over the totality of all others, if it was established at the previous stage). This procedure of comparisons and adjustments continues until the penultimate object.

5. The obtained measurement results or weighting coefficients are normalized, i.e. divided by the total score or weighting coefficients. After that, they take values ​​ranging from 0 to 1, and their sum becomes equal to 1.

Pairwise matching the simplest and most justified from a psychological point of view is considered in examples 21 and 22. As you can see, table. 17 and 18 are redundant. For pairwise comparison, the data given in the tables on one side of the diagonal is sufficient. In this case, preference is expressed by indicating the number of the preferred object as shown in Table 46.

Point j- th object or weight j- th indicator are calculated using formula (53). In this case

where is the frequency of preference by the i -th expert of the j -th object of examination; C - the total number of judgments of one expert, associated with the number of objects of examination m(number of measured indicators or weighting factors) by the ratio

Example 76. Let us assume for simplicity that five experts expressed their opinion about the six objects of examination in the same way: since this is presented in table. 46. ​​Determine the weight of each object and 1 construct a ranked series.

Solution 1: Preference Frequencies

Therefore, the values ​​​​obtained in paragraph 3 Gj can already be considered as normalized and, in particular, used as weighting coefficients.

5. The ranked number of objects of examination has the form: No. 3; № 1;№2; №6; №5; №4.

Experience of pairwise comparison according to table. 46 shows that due to the peculiarities of the human psyche, experts sometimes unconsciously give preference not to the object in the next pair under consideration, which is more important, but to the one that comes first on the list. To avoid this, use the free part of the table and perform pairwise matching twice (for example, first the first object with the second, third, fourth, etc., then the second with the first, third, fourth, ... and so on until the last, and then in reverse order: last with the penultimate, and to the first; penultimate with the last, previous... and again to the first). Thus, each pair of objects is compared twice, in a different order and after some time has passed. With this comparison, called complete or double, It is sometimes possible to avoid accidental errors and, in addition, to identify experts who are negligent in their duties or do not have a definite point of view. In other words, double pairwise comparison has higher reliability than single pairwise comparison. The calculation procedure remains the same, except that C = t(t-1).

The measurement results or the values ​​of the weighting coefficients obtained by pairwise comparison can be clarified using the method of successive approximation. The initial results (see paragraph 3 of Example 76) are considered in this case as a first approximation. As a second approximation, they are used as weighting coefficients Gj(1) expert judgments. The new results obtained taking into account these weighting coefficients are considered in the third approximation again as weighting coefficients Gj(2) the same expert opinions, etc. According to the Perron-Frobenius theorem, under certain conditions, which are always met in practice, this process converges, i.e. standardized measurement results g j or the weighting coefficients tend to certain constant values ​​that strictly reflect the relationships between the objects of examination with the initial data established by the experts.

Example 77. The results of a complete pairwise comparison by one expert of five objects of examination are presented in Table. 47, similar to table. 18, with the only difference that in order to exclude negative numbers from consideration, preference j-th object in front of i-m is indicated by the number 2, equivalence is indicated by the number 1, and preference i- th object in front j- m - number 0.

What can be said about the measurement result in the third approximation? Solution.

1. As a first approximation

G 1 (1) = 1+2+2+1+2= 8;

G 2 (1) = 0+1+2+2+2= 7;

G 3 (1) = 0+0+1+0+0= 1;

G 4 (1) = 1+0+2+1+2= 6;

G 5 (1) = 0+0+2+0+1= 3.

2. In the second approximation

G 1 (2) = 8 * 1+7 * 2+1 * 2+6 * 1+3 * 2= 36;

G 2 (2) = 8 * 0+7 * 1+1 * 2+6 * 2+3 * 2= 27;

G 3 (2) = 8 * 0+7 * 0+1 * 1+6 * 0+3 * 0= 1;

G 4 (2) = 8 * 1+7 * 0+1 * 2+6 * 1+3 * 2= 22;

G 5 (2) = 8 * 0+7 * 0+1 * 2+6 * 0+3 * 1= 5.

3. In the third approximation

G 1 (3) = 36 * 1+27 * 2+1 * 2+22 * 1+5 * 2= 124;

G 2 (3) = 36 * 0+27 * 1+1 * 2+22 * 2 +5 * 2 = 83;

G 3 (3) = 36 * 0+27 * 0+1 *1+22 * 0+5 * 0 = 1;

G 4 (3) = 36 * 1+27 *0+1 * 2+22 * 1+5 * 2 = 70;

G 5 (3) = 36* 0+27* 0+1* 2+22 * 0+5 *1 = 7.

4. Values gj, given in table. 47, differ noticeably in the first and third approximations. With each subsequent approach they will be refined. In the course of clarification, the preference of the first object of examination and the low importance of the third (to a lesser extent the fifth) are increasingly emphasized.

5. If there are several experts, then the final results should be obtained by averaging their data.

The method of successive approximation allows one to obtain strict quantitative measurement results on a ratio scale if it is known (or determined by an expert method) how many times the weight or indicator of the best object of examination exceeds the weight or the same indicator of the worst object. In this case, through this relationship, and preference j- th object of examination before i- m is expressed by the number 1 +, equivalence is expressed by one, and preference i- th object in front j- m - number 1 - , where

After this, pairwise comparison is performed using the successive approximation method. Process of clarifying values g j continues until the accuracy reaches the specified value. Since with each approach there is a change g j becomes smaller and smaller, this condition can be written in the form where it is usually taken = 0.001 if 1< <=1,5, и =0,01, если >5. For intermediate values, intermediate values ​​are also selected.

After completing the calculations, the actual ratio of the values ​​of the extreme terms of the ranked series Ф is compared with the original one . If the ratio is close to one, the problem is considered solved. Otherwise it is adjusted and the calculation is repeated.

Example 78. The best object out of six outperforms the worst one by 2.4 times in terms of the compared indicator. Hence,

The expert's opinions about the objects are presented in table. 48.

Go to the source data to calculate weighting coefficients with an accuracy of at least 0.5%.

6. Thus, the initial data for calculating weight coefficients with the required accuracy have the form presented in Table. 49.

The survey of experts can be in person and in absentia, group and individual, personalized and anonymous. Experts can express their opinions in writing (by filling out tables, questionnaires) or orally (giving interviews, participating in discussions). All of these and any other options for expert surveys have their advantages and disadvantages, so the choice of one or another of them is made depending on specific conditions and circumstances.