Denotatum-Based Models of Knowledge Creation for Monitoring and Evaluating R&D Program Implementation

Igor Zatsman
Institute of Informatics Problems The Russian Academy of Sciences (RAS) Moscow, Russia IZatsman@yandex.ru
Abstract The paper presents two semiotic models for a description of development stages of indicators, including generation microprocesses of expert knowledge about developed indicators. For the description of stages of these microprocesses, a new notion of Freges space is introduced. A related example is an application of two semiotic models to knowledge acquisition for an expert knowledge base named the proactive dictionary, which is a component of an evaluation system. This dictionary enables experts to fix stages of indicators development, to present the results of developing different variants of indicators in graphic form, to compare and evaluate these variants. Knowledge based systems; knowledge acquisition; knowledge representation; cognitive informatics; research and development management; semiotics.

The model for describing of a frozen state of the process of indicator development, named the frozenstate model [4]. The model for identifying of a dynamics of the process of indicator development, named the time-dependent model [5]. II. TWO MODELS OF INDICATOR DEVELOPMENT

I.

INTRODUCTION

The paper presents a semiotic approach to the study of knowledge creation microprocesses. These microprocesses are aimed at generating expert knowledge bases in different subject areas. Now we are studying this generation in scientometric and linguistic investigations. Our prime application aim is to create an expert knowledge base about new developed indicators for monitoring and evaluating implementation of R&D programs [1, 2, 3]. The indicator development was carried out under the First Russian Academic (FRA) Program, which was adopted by the Government of the Russian Federation in February 2008 as a tool of public intervention in the area of science. The Russian Academy of Sciences (RAS) has decided to create an evaluation system for providing assessment of the FRA Program as a whole and of its thematic subprograms. In order to develop new program-oriented indicators, we are designing the proactive dictionary as a tool for indicator development [2, 3]. The dictionary is a part of the evaluation system for monitoring and evaluating implementation of the FRA Program. Our design of the dictionary is based on two semiotic models for a description of development stages of indicators, including generation microprocesses of expert knowledge about developed indicators:

The idea of the frozen-state and time-dependent models has arisen in the course of our study of generation microprocesses of knowledge [6-8] and the creative space introduced by Wierzbicki and Nakamori [9-11]. These models are based on the Freges triangle, which consists of concept, name and denotatum vertices. For each state of new developed indicator these three vertices of the triangle are the sign-meaning (the indicator concept), the sign-form (the indicator name), and the denotatum of the sign (both an indicator computer program and source data). We use three following media for the study of the process of indicators development: the knowledge medium, the social information medium and the digital medium (Fig. 1). The knowledge medium contains indicator concepts, the social information medium - indicator names, the digital medium computer codes of indicator programs and source data. On Fig. 1, the digital medium is divided into two parts. The top digital medium has an indicator denotatum. The bottom digital medium contains three codes for the indicator denotatum, its concept and its name, as well as the digital semiotic triangle, which comprises these three computer codes. This indicator is evaluated by a computer, which uses both source data and a computer program for its evaluation. A. Frozen-state model of a state of new developed indicator At a discrete point in time, the frozen-state semiotic model of any state of new developed indicator consists of (see Fig. 1): Knowledge, social information and digital media. The Freges triangle of one state of new developed indicator.

This research is funded by RFH grant No. 12-02-00407a.

The digital semiotic triangle of the state.

Experts make a description of each state of new developed indicator as a descriptor of the proactive dictionary. In the model, three vertices of the Freges triangle are encoded by the evaluation system into three following computer codes, recorded into the proactive dictionary at the discrete point in time: A semantic code for the indicator concept. An information code for the indicator name. An object code for the indicator denotatum.

developed indicator is analyzed and described by experts in the proactive dictionary from three points of view: As an emerging indicator concept. As a variable indicator name. As a changeable indicator denotatum.

Each changeable indicator denotatum is a computer program of indicator computation together with corresponding source data, which are information resources of the evaluation system. The indicator names are given and changed by the experts, developing indicators.

The main design of computer coding of each new emerging indicator concept, its name and its denotatum is that each

Indicator denotatum

1. Source data for indicator evaluation 2. Computer program

Digital medium for denotata

Indicator concept Indicator name

Knowledge medium Social information medium

Object code Semantic code Information code

Digital medium for

codes

System proactive dictionary

Figure 1. The frozen-state semiotic model of a state of new developed indicator

B. Time-dependent model of new developed indicator During the development process of any indicator, its emerging concept description, its name, and its denotatum may be changed by the experts in great extent. Moreover, key stages of the development process of each indicator are the coordination of different personal concept descriptions between

the experts to form a collective (group) concept description of its indicator. The time-dependent semiotic model takes into account the variability of emerging indicator concepts, their names, and their denotata. The model of new developed indicator consists of (see Fig. 2):

Knowledge, social information and digital media. Freges triangles of all states of new developed indicator at the discrete points in time (ti, i = 1,2, ...). Digital semiotic triangles of states at ti, i = 1,2, ... .

The time-dependent model is based on the frozen-state model. Fig. 2 shows two stages of new indicator generation.

The first stage of the generation of the new indicator is on the left. At this stage, it is the first state of the indicator at the discrete point in time t1. At the first stage, the denotatum of the new indicator has been created and interpreted by the expert, and its concept and its name have been created by the same expert. The first digital semiotic triangle comprises three codes (semantic, information, and object codes) generated by the evaluation system at t1.

Indicator concept (t1)

Indicator concept (t2)

Knowledge medium

Stage 1 at t1
Indicator name (t1)

Stage 2 at t2
Indicator name (t2)

Social information medium Digital medium for denotata and codes System proactive dictionary

Indicator denotatum (t1)

Object code (t1) Semantic code (t1)

Indicator denotatum (t2)

Object code (t2)

t1

Information code (t1)

Semantic code (t2)

t2

Information code (t2)

Stage 1 at t1
The second stage of the generation of the developed indicator is on the right. At this stage, its the second state of the indicator at the discrete point in time t2. At the second stage, the denotatum of the new indicator has been changed and interpreted by the expert. During interpretation both its concept and its name have been changed by the same expert. The second digital semiotic triangle also comprises three codes generated by the evaluation system at t2. Using the time-dependent model we define a set of points named the Freges space as a trace space for the process of indicators development [5].

Stage 2 at t2
C. The Freges space This space is built to display development stages of indicators, including generation microprocesses of expert knowledge about developed indicators. The purpose of the design of the Frege's space is to represent how all developed indicators change in time, using sequences of semantic, information, and object codes. The Freges space has three axes of reference: semantic, information, and object code axes, respectively (see Fig. 3), as well as the fourth one the time axis. The time-dependent model describes all development stages of each indicator as one generation trace in the Freges space at discrete points in time (ti, i = 1,2,...), where ti is the i-th

Figure 2. The time-dependent model of new developed indicator for two stages

stage of the indicator development. It is supposed, that at each ti for each developed indicator, which changed at the ti , experts describe a frozen-state of the developed indicator and the evaluation system generates three computer codes: semantic, information, and object ones. The Freges space is a set of all

points of generation traces of all developed indicators. The space is built according the following requirement: any of these three computer codes must correspond to just one frozen-state and one point of the space.

Semantic codes

Information codes (for indicator names)

t2

Semantic code (t2) Semantic code (t1)

Name code (t2) Name code (t1)

t1

Object

codes
Object Object

code (t1)

code (t2)

Figure 3. Two points in the Freges space (the time axis is not on the figure)

III.

PROACTIVE DICTIONARY AS A PART OF THE

EVALUATION SYSTEM

The users of the proactive dictionary may be only the experts who are able to develop new indicators. Experts use the proactive dictionary as a tool for indicator development to describe the dynamics of generation microprocesses of their own knowledge about developed indicators. Each state of any developed indicator is described by experts as a descriptor of the proactive dictionary. At discrete points in time (ti, i = 1,2,...), each expert may create descriptors of the proactive dictionary. Experts can access existing descriptors of the proactive dictionary through the expert console software of the evaluation system and receive information about any frozenstate of each developed indicator. In the proactive dictionary the following events of indicator development are fixed:

Creation of a descriptor for each uninterpreted or interpreted frozen-state of any developed indicator, including its definition and its name (with indication of their authors from a group of experts). Establishment (change) of links between descriptors. Establishment (change) of descriptor references to the programs that calculate values of developed indicators, and of descriptor references to their source data.

Thus, each descriptor of the proactive dictionary contains a frozen-state indicator definition, its name, three computer codes, and links to other descriptors, as well as references to source data, which are information resources of the evaluation system (see references to information resources at t1 and tn on Fig. 4), and computer programs1, which evaluate values of

Computer programs were developed by V. Kosarik.

developed indicators (see references to program resources at t1 and tn). The development of any indicator includes two main phases: personal (one expert creates an indicator) and collective (an indicator is coordinated by several experts of the group). All the experts may refuse an indicator; then it is considered to be a non-actual, but still resides in the proactive dictionary. Besides the proactive dictionary, which is periodically enriched by new frozen-states of developed indicators, linguistic resources of the evaluation system include a semantic dictionary, which is accessed by all types of users of the evaluation system [1]. All descriptors of the semantic dictionary do not change in time. While the proactive dictionary deals with time-dependent indicators, the semantic dictionary describes the stable

indicators and has stable links and references (see Fig. 4). Each time-dependent indicator can potentially turn into a stable one and thus can be incorporated into the semantic dictionary, if this indicator is coordinated between the experts and is confirmed by decision-makers. The proactive dictionary supports the possibility to show any frozen-state of developed indicators, because it keeps all their frozen-states, which have ever existed. For the proactive dictionary, operations of modifying or deletion of descriptors are invalid: the modification of a descriptor leads just to the generation of a new descriptor, while maintaining the old one. So, it is more convenient to consider a new descriptor as the successor of the original. In the proactive dictionary, such inheritance can be multiple, because a new descriptor may be the heir to multiple source descriptors.

Evaluation system for assessment of R&D programs

Semantic dictionary - X indicator descriptor - Y indicator descriptor

Linguistic resources Authority resources

Proactive dictionary (new indicator descriptors) - U indicator descriptor (t1) - V indicator descriptor (t1) - U indicator descriptor (t n) - V indicator descriptor (t n)

Evaluation legislation Information resources Program resources Resources for stable indicators

Evaluation Information legislation resources (t1) (t1) Resources for developed indicators Evaluation legislation (t n) Information resources (t n)

Program resources (t1)

Program resources (t n)

Figure 4. Semantic and proactive dictionaries of the evaluation system

IV. DEVELOPMENT OF A SET OF INDICATORS We are performing an experiment on the development of variants of an indicator of the authors age distribution of the articles of participants of the R&D thematic subprogram of the FRA Program named Basic research for medicine. One of the goals of this subprogram is to attract young scientists to participate in its projects. In total, the subprogram holds about

140 projects. The total number of its participants is about 1,500. One of the approved indicators of all FRA thematic subprograms represents a share of participants under the age of 39 years. But this indicator says nothing about the productivity of young scientists. Therefore, we proposed to develop a set of indicators to calculate the authors age distribution of the articles, dividing the participants into 14 age groups (20 24, 25 29 and up to 85 89 years).

The results of calculation of the set of variants of the developed indicator are given in the form of graphs (see Fig. 5). Here we describe the first five stages of the experiment. Five experts take part in the indicator development: A, B, C, D and E. First stage. Expert A creates the first variant of the developed indicator having the following characteristics: To calculate the values of the developed indicator, expert A uses articles of participants of the R&D thematic subprogram Basic research for medicine from journals, issued in 2009 and entered into the database of the evaluation system. If an article has been written by co-authors, corresponding age groups receive 1 point for each author. The normalization procedure using the size of age groups is not applied.

At the same time (at the first stage) expert B creates the second variant of the developed indicator with same the first and the third characteristics, but the second one has a different value: corresponding age groups receive 1/N point for an article having N co-authors; the expert B also coordinates the second variant with expert C. Second stage . The expert C changes his mind and agrees with the expert A. In other words, the expert C refuses to coordinate the second variant of the indicator, thinking it is right to add just 1 point to corresponding age groups, thus coordinating the first variant of the indicator. So, the first variant becomes a collective one, while the second variant becomes personal. The variants themselves are identical to those of the first stage (these variants are not on Fig. 5). Third stage. Experts A, B and C decide to take into account, when calculating values of their two variants of the developed indicator, the number of people in each age group. It is reflected in changes of corresponding algorithms the normalization procedure is used in the calculations.

First variant of developed indicator

% for age group

14 12 10 8 6 4 2 0

16 14 12 10 8 6 4 2 0

%
14 12 10 8 6 4 2 0

20 30 40 50 60 70 80 24 34 44 54 64 74 84

First stage

20 30 24 34

Third stage

40 44

50 60 70 54 64 74

80 84

20 30 40 50 60 70 80 years 24 34 44 54 64 74 84 years

Fourth stage

18 16 14 12 10 8 6 4 2 0

Second variant of developed indicator 20

16 14 12 10 8 6 4 2 0 18 16 14 12 10 8 6 4 2 0

First stage

Third stage

Fourth stage

Figure 5. Results of calculation of values of the developed indicator

Fourth stage. Experts A, B and C decide to take into account only articles, printed in the journals from the rating list of the Nation Certification Commission of Russia, issued in 2009 and entered into the database of the evaluation system.

Fifth stage. Now the experts remain unchanged states of their variants of the developed indicator. Yet the points of view of some of them change. The expert C refuses the variant of the expert A, so this concept becomes a personal concept of the expert A. Two other experts, D and E, join the second variant

of the indicator (these variants are not on Fig. 5). This changes its level of coordination. In the course of the experiment, the coordination level function Fcoor was determined. We have constructed the coordination level function for first five stages of the

experiment (see Fig. 6). This function is useful for comparing two coordination levels current and desirable. If a desirable coordination level equals N, then we compare a current coordination level with N. The process of knowledge acquisition is terminated when the current level is above or equal to N.

Fcoor
3

Experts B, D and E

Experts B and C
2

Experts A and C

Experts A and C

Experts A and C

Expert A

Expert B

Expert B

Expert B

Expert A

t1

t2

t3

t4

t5

Figure 6. Coordination level function for first five stages of the experiment

V. CONCLUDING REMARKS Aiming to create a tool for developing sets of indicators for monitoring and evaluating R&D activities, we have introduced and applied the proactive dictionary as a part of the evaluation system. The new notion of Freges space has been introduced for quantitative representation of changeable indicator denotata, their emerging concepts and variable names during development of new indicators. The obtained results, considered in the paper, consist of: Two semiotic models of the processes of indicators development. The proactive dictionary, created on the base of these models. Results of the experiment on the development of variants of the indicator of the authors age distribution of the articles by a group of experts. The definition of the coordination level function Fcoor .

indicators for evaluating the FRA Program, specifying the states of developed indicators as descriptors. Development of indicators is not the only goal of the proactive dictionary design. Its implementation provides the solution of other important tasks. First, descriptors of the proactive dictionary set a one-to-one correspondence between indicator denotata, including their computer programs, and indicator names. In other words, each name of an indicator corresponds to a unique program calculating its values. Second, each descriptor fixes a rule of selecting source data, which are used in calculating values of the indicator. That is why the process of confirmation of any new indicator is a simultaneous approval of the following entities: its name, its rubric(s) in a classification system, its definition, the computer program of calculating its values, and the rule of selecting source data used in calculating its values. REFERENCES
[1] I. Zatsman and O. Kozhunova, Evaluation system for the Russian Academy of Sciences: objectives-resources-results approach and R&D indicators, in E-print Proceedings of the International Conference ATLC2009 (Atlanta Conference on Science and Innovation Policy), 2009, from: http://smartech.gatech.edu/bitstream/1853/32300/1/104674-1-PB.pdf.

These denotatum-based models of knowledge creation and the proactive dictionary show new possibilities for developing

[2]

[3]

[4]

[5]

I. Zatsman and A. Durnovo, Incompleteness problem for indicators system of research program, in Abstracts book of the 11th International Conference on Science and Technology Indicators (STI2010), Leiden: Universiteit Leiden, 2010, pp. 309311. I. Zatsman and A. Durnovo, Modelling of processes for creation of expert knowledge for monitoring of goal-oriented program activities, Informatics and its Applications, vol. 5(4), pp. 8498, 2011 (in Russian; abstract in English from: http://www.ipiran.ru/journal/issues/ 2011_04_eng/annot.asp). I. Zatsman, A semiotic model of correlations between concepts, information objects and computer codes, Informatics and its Applications, vol. 3(2), pp. 6581, 2009 (in Russian; abstract in English from: http://www.ipiran.ru/journal/issues/2009_02_eng/annot.asp). I. Zatsman, Time-dependent semiotic model of computer coding of concepts, information objects and denotata, Informatics and its Applications, vol. 3(4), pp. 87101, 2009 (in Russian; abstract in English from: http://www.ipiran.ru/journal/issues/2009_04_eng/ annot.asp).

[6]

I. Nonaka, The knowledge-creating company, Harvard Business Review, vol. 69(6), pp. 96104, 1991. [7] I. Nonaka and H. Takeuchi, The Knowledge-Creating Company. New York: Oxford University Press, 1995. [8] Knowledge emergence. I. Nonaka and T. Nishiguchi, Eds. New York: Oxford University Press, 2001. [9] A. Wierzbicki and Y. Nakamori, Basic dimensions of creative space, in Creative Space: Models of Creative Processes for Knowledge Civilization Age, A. Wierzbicki and Y. Nakamori, Eds. Heidelberg: Springer, 2006, pp. 59-90. [10] A. Wierzbicki and Y. Nakamori, Knowledge sciences: some new developments, Zeitschrift fr Betriebswirtschaft, vol. 77(3), pp. 271295, 2007. [11] H. Ren, J. Tian, Y. Nakamori, and A. Wierzbicki, Electronic support for knowledge creation in a research institute, Journal of Systems Science and Systems Engineering, vol. 16(2), pp. 235253, 2007.