Data capturing strategies used in Istat to improve quality

Conference of European Statisticians

Work session on statistical data editing (Bonn, 25-27 September 2006) Editing nearer the source session

Rossana Balestrino, Stefania Macchia, Manuela Murgia ISTAT Italian National Statistics Bureau Rome, Italy balestri@istat.it, macchia@istat.it, murgia@istat.it
1

CASIC techniques have been introduced at Istat in the 1980s

CATI and CAPI were adopted first nearly one decade later, CASI was taken into consideration

CATI/CAPI offer already mature and well tested solutions so have a higher rate of consolidation CASI techniques are younger and more depending on the continuously evolving of IT solutions and network tools 2

In Istat, for all the techniques :

the internal demand shows an increasing trend
the experience has taught that it is important that Istat plays a very active role and keeps at least the design and the monitoring phases of the process inside the Institute, in order to get standard solutions driven by quality requirements and enriched with suggestions coming from previous results
3

Strategies for CATI and CAPI surveys

Strategies for CASI

CATI and CAPI advantages

reduction of costs and time necessary to have data ready to be processed (Groves et al. 2001) help in preventing from non sampling errors, through the management of vast consistency plans during the interviewing phase

(CAPI is not so widely used as CATI in Istat, because is more expensive)

Organisation for CATI surveys

the content of the survey, made clear in the questionnaire, is designed in Istat, while private companies are charged with the entire data collection procedure.

Frequent problems encountered with this organisation

Private companies
had never faced in advance the development of electronic questionnaires so complicated in terms of skipping and consistency rules between variables had never put in practice strategies to prevent and reduce non response errors had not at their disposal a robust set of indicators to monitor the interviewing phase.
7

New organisation for CATI surveys:

in-house strategy
It consists in relying on a private company for the call centre, the selection of interviewers and to carry out the interviews, but in giving it all the software procedure, developed in Istat, to manage the data capturing phase: calls scheduler electronic questionnaire set of indicators to monitor the interviewing phase
8

In-house strategy: the software procedure

It integrates different software packages, but the core is developed with the Blaise system (produced by Statistics Netherlands and already used by a lot of National Statistics Administrations for data capturing carried out with different techniques)
9

Quality oriented procedure planning

Quality standards have been defined for: the data capturing phase the monitoring phase the secure transmission of data

10

Standards for the data capturing phase

the layout of the electronic questionnaire to reduce the segmentation effect the customisation of questions wording to make the interview more friendly and questions easy to be answered the management of errors to prevent from all the possible type of errors without increasing the respondent burden and making the interviewers job easier
11

Standards for the data capturing phase

the control of data with information from previous surveys or administrative archives to improve the quality of the collected data the assisted coding of textual answers to improve the coding results and to speed up the coding process the scheduling of contacts to enhance the interviewers productivity and to avoid distortion on the probability of respondents to be contacted.
12

Standards for the monitoring phase

A limited but exhaustive set of indicators to monitor the trend of contact results Ad hoc instruments to monitor particular aspects of the survey

13

Set of indicators to monitor the trend of contact results

n-ways contingency tables useful to keep under control the interviewers productivity and the presence of odd behaviours in assigning contact results Visual Basic, based on an Access database, which produces Excel files

Ad hoc instruments to monitor particular aspects of the survey

for example, control charts to monitor the assisted coding of textual variables (if used), like the Occupation SAS QC procedure which produces control charts for particular variables

14

Standards for the secure transmission of data

The aim is to assure both the secure transfer of survey data from the private company to Istat and vice versa, and the timeliness of
the delivery

The daily transmission is based on a secure protocol (HTTPS) and puts data on an Istat server, INDATA, placed outside the
firewall and devoted to data collection

15

Surveys which used the in-house strategy

Surveys Sample births survey 2001 Sample births survey 2004 University-to-work transition survey and perspectives 2004 Upper secondary school graduates survey 2004 Water System Surveys (preliminary survey) 2006 Violence against women survey (in progress) Nr of interviews
Long Short Long Short 16,597 33,838 15,642 33,515 25,510 20,408 1,320 25,000

Interviews length
1200 500 1348 543 10 56 13 20 903 2654

Response rates
92.6% 93.2% 94.7% 96.8% 95.8% 94.7% 99.8% 72.4%

Refusal rates
5.4% 4.9% 3.9% 2.2% 3.6% 4.8% 0.1% 16.0%
16

Surveys which used the in-house strategy

Characteristics of the questionnaires
Surveys Nr of variables of the electronic questionnaire
Long Long 677 707 218 315 30,000

Nr of checking rules
195 205 324 122 52

Sample births survey 2001 Sample births survey 2004 University-to-work transition survey and perspectives 2004 Upper secondary school graduates survey 2004 Water System Surveys (preliminary survey) 2006

Violence against women survey (in progress)

2,774

280
17

Checking rules in the data capturing phase with the in-house strategy

The number checking rules included in the data capturing phase (together with the number of variables) are surely significant indicators of the complexity of the survey questionnaire

This complexity has not negatively affected the response and refusal rates

because
18

the trade-off between the quality of data and the fluency of the interview has been taken into consideration
different treatments of the rules to detect errors have been implemented

19

The trade-off between the quality of data and the fluency of the interview
The consistency plans included in the electronic questionnaires comprised a great part, even if not all, of the rules proper of the edit and imputation plans avoiding, during the interview, a too frequent display on the pc-screen of a dialog window asking for the confirmation of the given answer
(including the complete edit plan in the data capturing phase would have guaranteed a high quality of the answer but would have definitely burdened the respondent and the interviewer, thus increasing the interruption rate)
20

Different treatments of the rules to detect errors

hard mode it is not possible to go on with the interview without solving the error soft mode the respondent can confirm his inconsistent response, without compromising the completion of the interview
21

Performance of the in-house strategy in terms of quality

Case study two surveys
Upper secondary school graduates survey University-to-work transition survey and perspectives

Carried out in: 2001 old strategy 2004 in house strategy

22

2004 and 2001 response and refusal rates

Upper secondary school graduates survey
University-to-work transition survey and perspectives

2004

2001

2004

2001

Response rate Refusal rate

94.7% 4.8%

85.4% 10.8%

95.8% 3.6%

94.0% 3.9%

23

Prevention from non sampling errors

Upper secondary school graduates survey
Errors per record
Errors per record 2004 survey (conducted with the in-house strategy)
Abs % Cumulate %

2001 survey (conducted with the external company strategy)

Abs % Cumulate %

No errors
From 1 to 2 errors From 3 to 4 errors 5 and more errors

13,013
5,742 1,183 470

63.8
28.1 5.8 2.3

63.8
91.9 97.7 100

12,245
9,029 1,582 406

52.6
38.8 6.8 1.8

52.6
91.4 98.2 100

Total

20,408

23,262

Prevention from non sampling errors

Upper secondary school graduates survey
Incidence of errors on the variables Most positive result Occupation
in-house strategy - coded during the interview with an assisted coding function external company strategy - manually coded after the interview

- 2001: 4.92% of raw data had to be corrected, during the edit and imputation phase - 2004: 0.81% (with the new strategy) had to be corrected, during the edit and imputation phase
25

Strategies for CATI and CAPI surveys

Strategies for CASI

26

CASI
prototypal experiences realised in the late 1990s current situation comprises several Web sites, located at Istat side and dedicated to the capture of surveys data for approximately 30 surveys

The need of designing a new environment and new rules aimed at introducing more standard solutions and effective security measures came out.
27

Strategy for CASI surveys

To set up a cross data capturing Web site to be used as a unique front-end for respondents to any survey
INDATA (https://indata.istat.it)

This new policy, already launched, is still in progress

28

INDATA web site: aims

To present the Institute outside with a homogeneous and stable public image and identity; To guarantee the mutual identity of data sender and receiver; To guarantee data confidentiality in the data collection phase and comprehensive security of the production environment; To minimize the impact on the technical environment of the respondent (it is not necessary to install SW on the client workstation). 29

INDATA web site: aims

To reply to the user about the action carried out by him (confirmation e-mail); To facilitate monitoring of collection activities; To favour the internal management and contain cost of the operational environment dedicated to data capturing.
30

31

Main functions offered to users

To be informed about the survey; To get and print forms and instructions;

To fill in electronic forms online;

To download electronic forms; To upload forms completed offline; To transfer any dataset in a safe way.
32

In synthesis
Both primary (single questionnaire, CSAQ = Computer Self Administrated Questionnaire ) and secondary data collection (collection of data) are dealt with.

Primary data collection is dealt in online and offline mode.

33

The INDATA web platform

The platform was initiated in the late 90s with prototype applications. Present Technological Features:
Operation system LINUX Red Hat 2.6.9; Web server APACHE 2.0.52; DBMS MYSQL and ORACLE 10; Application language PHP 5.1.2; Authenticity Certificate by Postecert; Secure HTTP.

34

INDATA architecture: requirements and constraints

Three level architecture ( WEB, APPLICATION, DB)
Secure system, safe back-end intranet Balanced load High level of reliability
35

System Architecture
Firewall Load Balancer Load Balancer

Web server

Web server

Front End

Firewall Web application server

Web application server

Back End

DB server

DB server

36

Web Surveys and Directorates

Central Directorate for Structural Surveys on Businesses Central Directorate for Short Term Surveys on Businesses Central Directorate for Surveys on Institutions

13
6 2

TOTAL

21

37

Electronic Questionnaire Type

Generation mode
PHP language - PDF questionnaire via TELEFORM - online compilation

N. of treated surveys 10 8
1

PHP language - EXCEL questionnaire offline compilation PHP language - BLAISE questionnaire offline compilation

38

CSAQ and Editing Rules

PDF questionnaire: editing rules are implemented in javascript language and comprise both range and consistency rules; the outcome of the editing activity is presented to the respondent globally, as a sequence of error messages, at the end of the compilation after pressing the submit button; EXCEL questionnaire: no editing macro is implemented in order not to discourage the respondent with alarm messages; all the cells are blocked apart from the input ones; data validation in single cells and default formulas in calculated variables are available; no or minimum consistency checking is performed. 39

E-response rates for Structural Business Statistics

Survey 10. Yearly Survey on Business Accounts Year 2003 2004 2005 11. Yearly Survey on Provisional Estimate of Value Added 12. Yearly Industrial Production Survey 2004 Observed users 10,000 10,000 10,000 10,000 Form Pages 10 10 10 1 E-response rate 36% 60% ... 32%

2005
2004 2005

10,000
45,000 68,000

1
2 2

75%
23% ...

13. Yearly Survey on the structure of Labour Cost

14. Yearly Survey on Telecommunications

2004
2004 2005

15,000
250 250

15
3 3

30%
100% ...
40

Surveys and data capture mode

1 Survey on book production Works published in 2005 PHP language - EXCEL questionnaire - offline compilation

2
3 4 5 6 7 8

Quarterly survey on turnover and orders

Quarterly Business Survey on job vacancies Periodic Survey on Hotel Activity Monthly Survey on employment, working hours and wages Monthly Survey on retail sales Yearly Survey on transports by rail Yearly Survey on Information Technology in financial businesses

PHP language - PDF questionnaire via TELEFORM - online compilation

PHP language - PDF questionnaire via TELEFORM - online compilation PHP language - PDF questionnaire via TELEFORM - online compilation PHP language - PDF questionnaire via TELEFORM - online compilation PHP language - PDF questionnaire via TELEFORM - online compilation PHP language - PDF questionnaire via TELEFORM - online compilation PHP language - PDF questionnaire via TELEFORM - online compilation

Yearly Survey on Information Technology in non-financial businesses

PHP language - PDF questionnaire via 41 TELEFORM - online compilation

Surveys and data capture mode

10
11 12 13 14 15 16 17 18

Yearly Survey on business accounts

Yearly Survey on Provisional Estimation of the Value Added Yearly Industrial Production Survey (PRODCOM)

PHP language - EXCEL questionnaire - offline compilation

PHP language - EXCEL questionnaire - offline compilation PHP language - EXCEL questionnaire - offline compilation

Yearly Survey on the Structure of Labour PHP language - EXCEL questionnaire - offline Cost compilation Yearly Survey on Telecommunication Enterprises Yearly Survey on structure and production of farms Quick Survey on certificates of balance accounts of Municipalities Quick Survey on certificates of balance accounts of Provincial Administrations Three-year survey on graduates (survey addressed to Universities) PHP language - EXCEL questionnaire - offline compilation PHP language BLAISE executable questionnaire - offline compilation Documentation and instructions for sending a file Documentation and instructions for sending a file PHP language - EXCEL questionnaire -42 offline compilation

Surveys and data capture mode

19 20 21 Six-month estimative survey on the consistency of livestock Yearly Survey on fishery in lakes and artificial docks Yearly Survey on economical results of farms PHP language - PDF questionnaire via TELEFORM - online compilation PHP language - PDF questionnaire via TELEFORM - online compilation PHP language - EXCEL questionnaire - offline compilation

43

Thanks

44