New IQ Paper ample - Semantic Scholar
DATA QUALITY IN ENGINEERING ASSET MANAGEMENT ORGANISATIONS – CURRENT PICTURE IN AUSTRALIA (Completed Paper) Jing Gao University of South Australia [email protected]
Shien Lin University of South Australia [email protected]
Andy Koronios University of South Australia [email protected] Abstract: Data Quality (DQ) has been an acknowledged issue for a long time. Researches have indicated that maintaining the quality of data is often acknowledged as problematic, but is also seen as critical to effective decision-making in engineering asset management (AM). The paper reports exploratory research on how engineering asset organizations in Australia are addressing DQ issues, and is drawn from a large scale national-wide DQ survey. Preliminary findings suggest that while the organizations are concerning the quality of data, there is a disconnection between data custodians and data producers and high level data owners. The majority of AM organizations still adopt a reactive approach on DQ management. This paper reports on a national survey as well as structured interviews with a number of stakeholders from engineering asset management organizations. Key Words: Data Quality, Engineering Asset Management
INTRODUCTION Almost every process and task in organisations involves data. Levitin and Redman [18] suggest that data provides the foundation for operational, tactical, and strategic decisions; further, they identify that, through the judicious use of data, managers are able to plan, organize and control an organisation’s resources to seek out new business opportunities, improvements to processes and the development of innovative products and services. As data becomes increasingly important in supporting organizational activities, the quality of the data that managers use becomes critical. Poor-quality data, if not identified and corrected, can have disastrous economic and social impacts on the health of the company [41]. Industry has recently put a strong emphasis on to the area of asset management (AM). In order for organizations to generate revenue they need to utilize assets in an effective and efficient way. Often the success of an enterprise depends largely on its ability to utilize assets efficiently. In other words, asset management is regarded as a set of business-capability building-processes within organisations. Thus, asset management has been regarded as an essential business process in many organizations, and is moving to the forefront of contributing to an organization's financial objectives. As an important initiative proposed by the Australian federal government and the industry sector, studies were commenced in 2003 into the impact of the quality of data on AM organisations including the Australian Navy, state utilities, transportation & mining companies, and local governments. A number of research findings [25,32,17,19,16] were published in various conferences and journals. In 2006, a large scale national-wide survey was conducted into data quality issues in asset management, with a sample
size of 2000 and a response rate of over 20%. This is one of the largest nation-wide surveys of its kind, aimed as directly addressing data quality issues in engineering asset management organisations in Australia. This paper discusses the development of this survey and presents some of its findings.
DEFINITION Data Quality Managers intuitively differentiate information from data, and describe information as data that has been processed. However, data and information are often used synonymously in practice, particularly when addressing quality issues. Therefore, this paper (and the survey) uses “data” interchangeably with “information”, as well as using “data quality” (DQ) interchangeably with “information quality” (IQ). There are a number of theoretical frameworks for understanding data quality. These DQ frameworks [39] [41,40,33,29,15,12,5,9,22,14,10] have been proposed to organize and structure important issues in information quality, albeit from different points of view. This paper (and the survey) follows Wang & Strong’s [41] data quality definition and regards the quality of data as being multi-dimensional, including as it does accuracy, reliability, importance, consistency, precision, timeliness, fineness, understandability, conciseness, and usefulness. In addition, it is also considered that the quality of data is dependent on how the data will be used [3,24,37,8,31,27]. This fitness for use can be defined as the intersection of the quality dimension being considered, the proposed use of the data (purpose), and the data fields which are identified for use in order to fulfil the purpose [23].
Asset Management According to British Standards Institute [4], asset management encompasses activities that are aimed at establishing the optimum way of managing assets to achieve a desired and sustained outcome. The objective of asset management is to optimize the lifecycle value of the physical assets by minimizing the long term cost of owning, operating, maintaining, and replacing the asset, while ensuring the required level of reliable and uninterrupted delivery of quality service [7,35,13]. At its core, asset management seeks to manage the facility’s asset from before it is operationally activated until long after it has been deactivated. This is because, in addition to managing the present and active asset, asset management also addresses planning and historical requirements. Asset management is process-oriented. The AM process itself is quite sophisticated and involves the whole asset lifecycle that can span a long period of time [36]. The lifecycle for a typical asset involves several interdependent stages including design, plan, acquisition, installation, operation, maintenance, rehabilitation and disposal. An overview is shown in Figure 1. At every stage of the process, AM also needs to collaborate and synchronize with other business processes, which is vital to the effective management of engineering assets (as shown in Figure 2). The cost and complexity of engineering assets demands considerable planning to identify appropriate solutions and evaluate investment opportunities. These same characteristics are reflected in the need for an extended acquisition process, a comprehensive request for proposal, and an equally comprehensive purchase agreement that addresses guarantees and warranties. Installation and placing in service of engineering assets is also complex and requires a proper set of processes to manage contractors. Once the asset is acquired, it must be tracked throughout its useful life. Finally, records must be made of its eventual disposition.
Figure 1- Asset Lifecycle Stages (Source: [34])
Figure 2- Collaborative Asset Lifecycle Management (Source: Adopted and modified from [34]) The sophistication of the engineering asset management process requires substantial information to be collected throughout all stages of a typical asset’s lifecycle. This information needs to be maintained for a very long time, often dozens of years in order to identify long-term trends. This kind of process also uses this information to plan and schedule asset maintenance, rehabilitation, and replacement activities. In
order to manage and support the complicated AM process and its data requirements, a variety of specialized technical, operational and administrative systems exist in asset management. These not only manage, control and track the asset through its entire lifecycle, but also provide maintenance support throughout the lifecycle of the asset. Table 1 shows some of these technical systems. Considering the complexity and importance of asset management, these systems are normally bought from multiple vendors and each is specialized to accomplish its task. Unfortunately, this leads to an extremely difficult integration job for the end-user. Asset Lifecycle Stage
Asset Management Technical System
Asset Creation
• • • • • • • • • • • • • • • • • • •
Asset Operation & Maintenance
Computer Aided Design (CAD) Document Management System (DMS) Supervisory Control and Data Acquisition (SCADA) systems Enterprise Asset Management (EAM) Plant Asset Management (PAM) Computerized Maintenance Management Systems (CMMS) Enterprise Asset Maintenance Systems Reliability Centred Maintenance (RCM) Total Productive Maintenance (TPM) Geographic Information Systems (GIS) Product Service Management (PSM) Reliability Assessment Systems Turbo-machinery Safety Systems Rotating Machine Vibration Condition Monitoring Systems Electrical Motor Testing Systems Root Cause Analysis Systems Asset Capacity Forecasting Systems Operational Data Historians Physical Asset Data Warehouse Systems
Table 1: Asset Management Technical Systems (Source: Developed by the authors) Engineering processes rely heavily on input of data and also produce a large amount data. Engineering data itself is quite different to typical business-oriented data as illustrated in Table 2. It has unique data characteristics and complex data capture processes from a large variety of data sources. This large amount of data therefore can suffer from data quality problems. The nature of such data quality problems has not previously been investigated in Australian engineering-oriented organizations. Element Data Environment Data Characteristics
Typical Business Environment Transaction-driven, product-centric business data environments o o o o o
Self-descriptive Static Intrinsic quality Discrete value with fewer or no constraints Current
Engineering Asset Management Continuous data, process-centric open control system and manufacturing data environments o o o o o
Non self-descriptive Dynamic Intrinsic / extrinsic quality Continuous value with constraints (e.g. within a range), precision value Temporal
Transactional data Often structured Easy to audit Can be cleansed using existing tools o Similar data types Inventory data, customer data, financial data, supplier data, transaction data etc
o o o o o
Data Sources
Mainly transaction-based textual records from business activities
Data Capture
o
Disparate data sources o Spatial data – plans/maps, drawings, photo o Textual records – inspection sheets, payment schedules o Attribute records – separate databases, maintenance/renewal records, fault/failure records, field books o Real-time CMS/SCADA o Other sources – existing/previous staff and contractors, photos o Electronically, involving sensors, technical systems such as SCADA systems, condition monitoring systems o Manually, involving field devices, field force, contractors, business rules o Data collected in a variety of formats o Requires to collect substantial data from many different parts of the organization o Data often entered by less/un trained, less dedicated personnel without proper relevant knowledge o Data entry environment can be unstable, harsh, less preorganized o Data entry point can be far from the organization site o Very large amount of data to be maintained for extended time for AM engineering and planning process o Data stored on various operational and administrative systems o Not comprehensive o Process dependent o Complex to integrate data, need both vertical and horizontal data integration o Data to be shared among various technical (e.g. design, operations, maintenance) and business systems o Data to be communicated to an array of stakeholders, business partners and contractors, subcontractors o Need experts with professional knowledge to interpret data o Difficult to translate asset data into meaningful management information
o o o o
Data Category
o
o o
Data Storage
o o
Data Processing
o o o
Data Usage & Analysis
o o o o
Often manually by data providers in fixed format Data often entered by reasonably trained, dedicated personnel with proper relevant knowledge Data collection environment is stable, well pre-organized Data entry point is within the business
Data to be kept in accordance with appropriate compliance requirements Data stored on functional information systems Comprehensive Process independent Easy for data integration Data to be shared only among relevant business systems Data to be communicated to internal stakeholders Use general, common knowledge to interpret data Easy for management use
Time-series streaming data Often unstructured Difficult to be audited Difficult to be cleansed using existing tools Diversity of data types
Inventory data, condition data, performance data, criticality data, lifecycle data, valuation data, financial data, risk data, reliability data, technical data, physical data, GPS data etc
Table 2: Differences between engineering asset data and typical business data (Source: Developed by the authors)
SURVEY DEVELOPMENT – RESEARCH DESIGN
This research focuses on a pluralist methodological approach to ensure that evidence is collected from a great variety of engineering organizations. A national research survey was undertaken in order to provide an overview of data quality issues in Australian engineering asset managing organizations. Case-based research was then undertaken with a selected number of organizations to provide richer data about the sources of data quality problems and to seek reasons for these problems. Pinsonneault and Kraemer [28] propose that survey research is believed to be well understood and applied by management information systems (MIS) scholars. The survey-based research requires “standardized information from and/or about the subjects being studied” and usually is in the form of questionnaires. Zikmund [43] and Davis and Cosenza [6] claim that surveys can provide quick, inexpensive, efficient and accurate means of assessing information about the population. In contrast to case studies, survey questionnaires have certain advantages: • • • •
they reach a geographically-dispersed sample simultaneously and at a relatively low cost; standardised questions make the responses easy to compare; they capture responses people may not be willing to reveal in a personal interview; results are not open to different interpretations by the researcher.
Though survey research is among the more popular methods used by the information systems research community, survey research has its own weaknesses. The analysis by Pinsonneault and Kraemer [28] indicates that survey research is often misapplied or ineffective due to several reasons: • • •
Single-method designs where multiple methods are needed; Unsystematic and often inadequate sampling procedures; Low response rates.
The researchers were fully aware of these issues and important steps were taken during the survey development. For example, the survey study is one part of a series of research approaches (including interviews, field observations and other research forms). Understandings of the data quality issues in the given domain (asset management organisations) were accumulated to avoid the limitation of single research method design.
Sampling Procedure As the sample was part of a nation-wide survey of asset management organisations, precautions were taken during the sampling stage in order to obtain a representative sample. Overall, 2000 participants were identified, using statistical sampling methods, through several Australian Business Directories include Kompass and “Business who is who” (Australian Business Directory). The survey population for the questionnaire was chosen from engineering asset management organizations based in Australia. These organizations represent a variety of industries: • • • • •
Utilities (water, electricity, gas, oil); Mining & resources; Transport (rail, airline, ship, automobile); Defence; and Local government.
Considering the uneven population distribution in different Australian states, different numbers of survey participants were accordingly selected, as shown in Table 3 & 4: Unit: 1,000
Australian State Total Population %
NSW & ACT
VIC
QLD
SA
WA
TAS
NT
Total
7,055
4,973
3,882
1,534
1,982
482
200
20,111
35%
25%
19%
8%
10%
2%
1%
100%
Table 3: Australian State Population Distribution Year 2005 (Source: Australian Bureau of Statistics [2]) (Note: The Australian States: NSW – New South Wales; ACT – Australian Capital Territory; VIC – Victoria; QLD – Queensland; SA – South Australia; WA – Western Australia; TAS – Tasmania; NT – Northern Territory)
Australian State Defense Local Government Mining & Resources Transport (rail, airline, ship, automobile) Utility (electricity, gas, oil) Water utility Total %
NSW & ACT 21
VIC
QLD
SA
WA
TAS
NT
Total
15
3
43
82
113
143
134
126
170
71
62
20
18
160
132
86
33
21
36
2
310
81
34
53
21
48
10
247
58 476 24%
64 404 20%
68 311 16%
7 236 12%
17 431 22%
7 70 4%
51
64
801
2
333
6 72 4%
227 2,000 100%
Table 4: Number of Survey Samples by the State (Note: Western Australia (WA) is the center of Australian mining and manufacturing industry where many engineering asset management organisations are located.) The size of organisations was also considered, while the focus has been placed on the middle-sized organizations (Figure 3). Thus, it is believed that the chosen sample is representative.
Size of The Organisations 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
40%
16%
14%
13%
8% 4%
< 10
2% 10 - 19
20 - 49
50 - 99
100 - 499 500 - 999
1,000 4,999
1%
2%
5,000 9,999
10,000+
Figure 3: Size of Organisations
Increasing Response Rate This research uses traditional, physical-mail surveys, instead of online or email surveys. This reflected the outcomes of research, showing that mail surveys usually have relatively higher response rates. All the survey questionnaires were posted with reply-paid return envelopes. More importantly, substantial efforts were undertaken to find the names of participants in each chosen organisation (e.g. CEO and managers’ names), which enabled a personalised envelope and cover letter to be delivered to the participants. Moreover, a reminder letter was posted to all participants after the deadline. Overall, this survey achieved a response rate of 23.2% which the authors consider to be satisfactory for this type of national survey.
Research Objectives and Questions This study is aimed at obtaining an overall understanding of how asset management organisations in Australia address concerns related to various data quality issues. In particular, this survey attempts to answer the following questions: 1) To what extent, are Australian asset management organisations aware of the importance of data quality? 2) What are the current data quality problems and issues that these organisations are facing; and how do they manage data quality? 3) What strategies are currently employed in addressing data quality problems in engineering asset management organisations? Through the development of the survey questions, a number of previous data quality studies were considered (as discussed in the next section). It is thought that comparisons can also go some way in addressing the following questions: 4) To what extent do data quality issues in engineering asset management organisations differ from other types of enterprises and organisations? 5) To what extent do data quality issues in Australia differ from organisations in other countries? The results of this survey are thought to be of value for developing a DQ framework specific to engineering asset management organisations. These results will form the foundation for further research in order to:
a. b. c. d. e. f.
Perform data audits to identify the nature and volume of DQ problems; Categorise the various DQ problems; Identify DQ dimensions to be assessed; Prioritise the dimensions; Identify metrics that specify the criteria to be used to measure DQ; Develop specific assessment metrics for each dimension.
Limitations of Previous Data Quality Surveys International consulting companies have conducted global data quality surveys. The PriceWaterHouseCoopers and Advance Information Management (AIM) groups are two of the most wellknown examples. PriceWaterHouseCoopers’s Global Data Management Survey [30], which built on a 2001 survey, selected 450 companies across the US, UK and Australia, targeting senior management, or equivalent executives at the same level (89% of respondents were senior IT staff: CIOs, IT directors, data controllers, network administrators, etc; and the remaining 11% of respondents were senior executives from outside the IT organisation). The survey results reveal that the top six data quality initiatives among senior management included: • • • • • •
Improving data accuracy (26%); More rigorous data management (14%); System upgrades (13%); Improving security (11%); Data standardisation (10%) and Improving usage/analysis of data (10%).
Given the topic of the survey questions, and the level of respondents, it appears that the PriceWaterHouseCoopers’ data quality survey primarily focused on the IT industry and was concerned with the strategic development of data management within organisations. Although the overall response rate was not published, it is felt that the sample size of 450 (spread over three countries) may not be able to provide a comprehensive understanding of data quality issues. In particular, the focus on IT companies excludes the government sector and engineering asset management organisations, which cannot provide answers to the research questions proposed in this study. AIM Software [1] in conjunction with the Vienna University of Economics and Business Administration, has also attempted a series of international data surveys. The Global Data and Risk Management survey was originally conducted in 2004, with a sample size of 1,736 financial institutions in 63 countries from all of the regions (North America, Central & South America, Western Europe, Central and Eastern Europe, the Middle-East, Asia, Australia and South Africa). The responding companies were requested, between January and August 2004, to answer a 14-question questionnaire on data management. This survey was then modified and replicated in 2005 and 2006 (the 2006 survey has not yet been completed). In 2005, 1070 companies from 88 countries were selected (62% banks, 22% asset management, 10% broker/dealer and 6% insurance) to answer 9 questions. The findings were published on August 2005 and can be accessed online (from AIM official website - http://www.aim-sw.com). AIM’s survey provides a comprehensive picture of how companies expressed concerns regarding data management issues, across different countries. However, as the survey has a particular interest in financial institutions, the findings may not be applicable in engineering asset management organisations Woodhouse [42] and GAO [11] have argued the uniqueness of data management in engineering asset
management organisations. Moreover, when looking at the actual questions, the AIM survey aims to focus on the current planning methods and future strategy developments for data management. It is difficult to anticipate that a 9-question survey is adequate to explore the actual data quality problems in practice, at both business and operational levels within organisations. OMNILINK [26] conducted the first industry-wide data management survey in Australia in 2004. The sample was drawn from a range of sectors and industries in Australia and New Zealand, from small to large organisations. This survey was replicated in 2005 (the findings are yet to be published). OMNILINK’s survey addressed several data quality issues including data accuracy, data security and data currency. The report provides an analysis of current data management practices, in both the public and private sectors, and an insight into the role of Geographic Information Systems (GIS) in data management. After examining this report, it appears that OMNILINK primarily emphasized the use of IT in data management. The issues associated with human factors were not discussed in detail.
Questionnaire Development A number of issues were considered during the development of the current questionnaire, in order to better address the uniqueness and comprehensiveness of the survey finding. Firstly, unlike the above questionnaire surveys, which only targeted participants at a specific organisational level, four types of stakeholders have been identified intuitively for the current survey: data producers, data custodians, data consumers, and data managers [38], as defined below: 1. 2. 3. 4.
Data producers are those who create or collect asset data e.g. asset managers; Data custodians are those who design, develop, manage, and operate the asset management systems e.g. IT managers; Data consumers are those who use the asset information in their work activities e.g. maintenance engineers, senior managers; Data managers are those responsible for managing the entire data quality in asset management systems.
Based on these four distinct data roles, the survey consists of five sections. The first section collects demographic data of the four data roles, and the rest four sections correspond to the four different data roles. Respondents choose one section from the rest four sections according to their particular data role. This ensures that the results can present valuable and insightful issues at all organisational levels. In total, an average 20 questions are to be answered by each individual respondent from the almost 80 questions which were proposed, are being derived from a well-known TOP framework. Mitroff and Linstone [21] argue that any phenomenon, subsystem or system can be analysed from what they call a Multiple Perspective method – employing different ways of seeing, to seek perspectives on the problem. These different ways of seeing are demonstrated in the TOP model of Linstone [20] and Mitroff and Linstone [21]. The TOP model allows analysts to look at the problem context from either Technical or Organizational or Personal points of view: • • •
The technical perspective (T) sees organizations as hierarchical structures or networks of interrelationships between individuals, groups, organizations and systems. The organisational perspective (O) considers an organization’s performance in terms of effectiveness and efficiencies. For example, leadership is one of the concerns. The personal perspective (P) focuses on individual concerns. For example, the issues of job description and job security are some of the main concerns in this perspective.
This framework has been used in previous studies (Koronios, Lin & Gao 2005; Lin, Gao & Koronios
2006) using interview case studies. By employing this approach and using experiences/findings obtained from previous studies, the survey questionnaire design is believed to cover a wide area of data quality issues (please email us for a copy of the survey).
Data Validity The multi-section questionnaires were mailed to a large, random sample of 2000 asset managers, data producers, data custodians and data users, in 1100 geographically dispersed engineering asset management organizations in Australia (including 572 organizations in the public sector). The same surveys were posted to different participants within same organisation (to address the issues of those with different data roles). Three types of questions were classified during the design stage: 1. Questions covering all data roles E.g. How important is data quality to the success of your organisation? 2. Questions unique to individual data role E.g. Do you have the following difficulties for any data entry task? – Data producers only 3. Different questions for different role, but can be used for cross-checking E.g. How often do you monitor the quality of your data? – Data / Asset managers How often do you have any form of management reviews done in relation to your data collection performance? – Data producers In addition, the questionnaire was pre-tested by initially mailing it to 15 companies. Changes were then incorporated based on feedback received from the pilot survey, before the final questionnaires were mailed to the remaining companies. A number of changes and additions were made to the instrument after pre-testing. These included: • • •
Refining of some of the questions to increase clarity and remove ambiguities; Adding of some additional items to achieve greater integrity; and Changes to some of the measurement scales.
The information obtained from the pre-tests was used to alter the questionnaire to achieve higher validity. Pre-testing also involved a preliminary analysis of the data collected, ensuring that the data collected was appropriate for testing the research model and therefore addressing the problem [43]. The overall approach is therefore considered to be adequate in addressing validity issues of the survey.
METHOD OF ANALYSIS Statistical methods were used for the analysis of the responses. Software package SPSS were used. A three-stage analysis will be performed including: 1. Basic descriptive statistics analysis including frequencies, means, standard deviation; 2. Multivariate data analysis techniques including factor analysis, multiple regression analysis; 3. Qualitative analysis on participants comments As the analysis of the data has not been yet completed, this paper provides preliminary and ‘first impressions’ results of this research.
FINDINGS AND DISCUSSION This survey is the first national data quality survey performed in Australia, focusing specifically on the data quality issues of engineering asset management organisations. The following findings show the different attitudes and perceptions towards data quality. More importantly, it shows that the current strategy, policy and tools that the organisations employ for their data management solution.
Current Attitude & Awareness towards Data Quality
How important is data quality to the success of your organisation? 60% 50% 40% 30% 20% 10% 0%
Data Owner
po rta nt Im po rta nt Do n’ tk No no tv w er y im po No r ta ti m nt po rta nt at al l
Data Custodian Data Consumer
Ve ry
im
Cr itic
al
Data Producer
The above result shows that the majority of participants felt that data quality is an important success determinant for their organizations. However, it also suggests that the data consumers have a much higher needs of quality data.
Current Perception of Organisational Data Quality Management From Data Owners’ point of view Assuming you get sufficient data, how often do you rely on your data to make critical decisions?
How often do you get sufficient data fromthe systems in order to make decision for daily work?
40%
48%
50%
40%
40% 30%
47%
50%
60%
30%
26%
20%
20%
14%
10%
10% 1%
1%
0%
7%
10%
5%
1%
0%
0% All the time
Sometimes
Occasionally
Don’t know
Rarely
Never
All the time
Sometimes
Occasionally
Don’t know
Rarely
Never
The above left figure shows that the majority of data owners (Asset managers) are happy with the amount of the data that they can access. However, there is a relatively small group (about 7%), who is not satisfied with the quality of data for decision making as shown in the figure on the right. From Data Consumers’ point of view How often do you get sufficient data fromthe systems in order to make decision for daily work? 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
37%
Assuming you get sufficient data, how often do you rely on your data to make critical decisions?
40%
60% 50%
48% 41%
40% 30%
17%
20% 5% 1%
0%
5%
10%
5%
1%
0%
0% All the time
Sometimes
Occasionally
Don’t know
Rarely
Never
All the time
Sometimes
Occasionally
Don’t know
Rarely
Never
A consistent opinion is suggested by the data consumers’ group. However, it does necessarily show that the data quality problem is not a major concern within the participating organisations. From Data Custodians’ point of view Is the data in your system(s) up-to-date to the required level?
Do you think that the data in your system(s) is accurate to the required level? 55%
60% 50% 40%
34%
30% 20%
11%
10%
0%
0% I have total confidence
It is not a critical problem
There are problems
I don’t rely on them at all
50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
47% 35%
7%
Strongly agree
7%
Agree
Don’t know
4%
Disagree
Strongly disagree
From Data Producers point of view The data in the systemis error free, do you agree?
How often do you have confidence (accuracy) with the data captured by the electronic sensors or hand-held devices?
37%
15%
10%
2%
0%
0% Strongly agree
Agree
Neutral
Disagree
Strongly disagree
Al l th e
5%
10%
tim e
10%
18% 2%
2%
ap pl ica ble
20%
31%
No t
25%
35%
40% 35% 30% 25% 20% 15% 10% 5% 0%
Ne ve r
27%
Ra rel y
27%
Do n’t kn ow
30%
Oc ca sio na ly
35%
So me tim es
40%
The above figures shows that the data producers and data custodians acknowledged that there were data quality problems. Especially, data producers do not have much confidence on their data quality. Perhaps, the data owners and data consumers may have higher levels tolerance of poor quality data. Nevertheless data quality problem is still facing critical challenges in most organisations. Also, the different attitudes found between groups may ring the alert bell that there may be a disconnection between the operational level personnel and the strategic level managers. Some problems are suggested by the data custodians. For example, it suggests that moving/migrating data may generate many quality related problems, as shown below.
Fromyour perspective, what are the major problems when moving data? 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
42% 30% 19% 9%
I can move data from I have experienced The moving process different sources some problems in the can be done, but there without any problems past; such problems are unresolved will be easily resolved problems
I have experienced some significant problems
Current Strategies, Policies & Tools Employed for Achieving High Quality of Data According to the 60% of data owners, there is data management strategy for data quality in place in their organisations, as shown in below. Do you already have a data management strategy in place in your organisation? 70%
60%
60% 50% 34%
40% 30% 20%
6%
10% 0% No
Yes
Don't know
Down to the data capture level, data producers listed different ways of data collection and entry. How do you collect asset data e.g. asset health data? 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
How do you enter the data into the system?
39% 27%
26%
8%
I physically inspect the assets and write down the details
I use hand-held devices to collect asset data
I use electronic sensors which will read and record the data automatically
Other
46%
50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
25%
19% 4% It is a manual process; I need to type into the system from the paper records
I give my paper records to office clerks who are responsible for data entry
I can enter them I upload the data from where I am; I electronically have a networked (e.g. from a mobile system for docking station) data entry when returning to the office.
The majority of data producers still adopt a manual data entry process, which primarily rely on paper records. Especially, the figure below suggests that these data may not be entered immediately on site. Approximately 49% of these data were entered at the office after several days. Thus, the accuracy and completeness of these data may not be satisfactorily achieved. When and where do you make data entries to the systems? 60%
49%
50% 40%
32%
30% 20%
12%
7%
10% 0% At the location where I At my office – When I captured the data – go back from field Immediately
At my office – After several days
I give my data to someone else, so I don’t know
6% Other
Towards Future It is “interesting” to find that 39% of asset owners have no plans to implement any data management solutions in near future. While the answer from Data custodians to the question “Has implementing a data quality system had a positive impact on the success of any major IT implementations which your organization has put in place (e.g. Enterprise Resource Planning)?” shows that no data quality systems (e.g. data profiling and cleansing systems) were implemented or planned to be implemented. positive impact?
Do you plan to implement a data management solution in the following way? 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
39% 24% 17%
13% 6% 2%
45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
39% 28% 20% 11% 2% A major impact
Proprietary Buy a solution Buy a solution Outsource the Outsource the Not planned development and extend development processing
A slight impact
Don't know
No impact at all
No data quality system implementation
CONCLUSION AND FUTURE RESEARCH This paper only included a small proportion of survey findings. Nevertheless, these results suggest that while the organizations are concerned about the quality of data, there is lack of scrutinized discussion on the various issues associated with data quality problems. More importantly, there is a disconnection between data custodians and data producers and high level data owners. The majority of engineering asset organisations in Australia has no plans to neither implement any data quality management solutions nor develop any strategic plans. This finding is very different from AIM and PriceWaterHouse findings. Perhaps, the engineering asset management organisations in Australia still adopt a reactive approach and only focus on the daily operations. A more comprehensive analysis will show the different attitudes and management strategies in relation to sizes of the organisations and the industries that they operate within. Further, these findings will be compared with the general data quality surveys. The result will be published in the research report or upcoming journal paper. This paper provided a better understanding of data quality issues for asset management and is assisting in identifying elements which will contribute towards the development of a data quality framework specific to engineering asset management. This in turn will assist in providing useful advice for improving data quality in this area. It is thought that more and more organisations are putting efforts and financial resources in data management solutions, there is an increasing need of guidelines for helping them develop appropriate strategies and employ the right tool. Perhaps this is why data quality research becomes more critical.
REFERENCES [1] [2]
[3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]
[15] [16]
[17]
[18] [19]
[20] [21] [22]
[23] [24] [25] [26]
AIM Software, “Global Data and Risk Management Survey 2005”, Research Report, 2005. Australian Bureau of Statistics, “Australian Social Trends 2005”, online accessed on 15 June 2006, http://www.abs.gov.au/ausstats/[email protected]/94713ad445ff1425ca25682000192af2/8ef24f9fe10ee078ca25703b 0080ccaf!OpenDocument Ballou, D.P., & Pazer, H.L. (1995). Designing Information Systems to Optimize the Accuracy-timeliness Tradeoff. Information Systems Research. 6(1), 51-72. British Standards Institution, PAS 55: Asset Management. Part 1: Specification for the optimised management of physical infrastructure asset, 2004. Caballero, I., & Piattini, M. (2003). Data Quality Management Improvement. ACS/IEEE International Conference on Computer Systems and Applications (AICCSA’03), July 14-18 2003, Tunisia. 56-65. Davis, D. & Cosenza, R.M., Business Research for Decision Making, 2nd Edition, Boston, PWS-Kent, 1988. Eerens, E., Business Driven Asset Management for Industrial & Infrastructure Assets, Le Clochard, Australia, 2003. English, L.P., Improving Data Warehouse and Business Information Quality: Methods for reducing costs and increasing Profits, Willey & Sons, 1999. Eppler, M.J. (2001). The Concept of Information Quality: An Interdisciplinary Evaluation of Recent Information Quality Frameworks. Studies in Communication Sciences. 1, 167-182. Firth, C. (1996). Data Quality in Practice: Experience from the Frontline. The Proceedings of 1996 Conference of Information Quality. 25-26 October 1996, Cambridge, USA. GAO (U.S. General Accounting Office), “Water Infrastructure: Water Utility Asset Management”, GAO-04461, Washington, D.C., 2004. Giannoccaro, A., Shanks, G., & Darke, P. (1999). Stakeholder perceptions of data quality in a data warehouse environment. Australian Computer Journal. 31(4), 110-117. IPWEA, International Infrastructure Management Manual, Australia/New Zealand Edition, 2002. Jarke, M., Jeusfeld, M.A., Quix, C., & Vassiliadis, P. (1998). Architecture and Quality in Data Warehouses. Proceedings of Tenth International Conference CAiSE’98. 8-9 June 1998, Pisa, Italy. Springer-Verlag. 93113. Kahn, B.K., Strong, D.M., and Wang, R.Y., “Information Quality Benchmarks: Product and Services Performance”, Communications of the ACM, 45(4), 2002, pp. 184-192. Koronios, A., and Haider, A., “Managing Engineering Assets: A Knowledge Based Asset Management Methodology Through Information Quality”, E-Business and Organisations in the 21th Century, 2003, pp. 443-452. Koronios, A., Lin, S., & Gao, J. (2005). A Data Quality Model for Asset Management in Engineering Organisations. Proceedings of the 10th International Conference on Information Quality (ICIQ 2005). 4-6 November 2005, Cambridge, MA. USA. 27-51. Levitin, A.V. and Redman, T.C., “Data as a resource: properties, implications and prescriptions”, Sloan Management Review, 40(1), 1998, pp. 89-101. Lin, S., Gao, J., & Koronios, A. (2006). Key Data Quality Issues for Enterprise Asset Management in Engineering Organisations. International Journal of Electronic Business Management (IJEBM). 4(1), 96110. Linstone, H. A., Decision Making for Technology Executives: Using Multiple Perspectives to Improve Performance, Artech House Publisher, 1999. Mitroff, I. I. and H. A. Linstone, The Unbounded Mind: Breaking the Chains of Traditional Business Thinking, New York, OxFord University Press, 1993. Nauman, F., & Roth, M. (2004). Information quality: How good are off-the-shelf DBMS? Proceedings of the 9th International Conference on Information Quality (ICIQ-04). 5-7 November 2004, Cambridge, MA. USA. 260-274. Neely, P., & Pardo, T., “Teaching Data Quality Concepts Through Case Studies”, Center for Technology in Government, Albany, 2002. Neely, P., “A Framework for the Analysis of Source Data Revised”, Proceedings of AMCIS 2001. Neely, P., Gao, J., Lin, S., & Koronios, A., “The Deficiencies of Current Data Quality Tools in the Realm of Engineering Asset Management”, Proceedings of AMCIS, Acapulco, México, 2006. OMNILINK, “2004 Data Management Report”, Research Report, Sydney, 2004.
[27] [28] [29]
[30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43]
Orr, K., “Data Quality and System Theory”, Communications of the ACM, 41(2), 1998, pp. 66-71. Pinsonneault, A. and Kraemer, K. L., “Survey research methodology in management information systems: An assessment”, Journal of Management Information Systems, 10(2), 1993, pp. 75-105. Price, R.J., & Shanks, G. (2004). A Semiotic Information Quality Framework. Decision Support in an Uncertain and Complex World: The IFIP TC8/WG8.3 International Conference 2004. 1-3 July 2004, Prato, Italy. 658-672. PriceWaterHouseCoopers, “Global Data Management Survey”, Research Report, 2004. Salaun, Y., and Flores, K., “Information Quality: Meeting the Needs of the Consumer”, International Journal of Information Management, 21(1), 2001, pp. 21-37. Saunders, D., “Innovation in Asset Management - Achieving a Nexus of Cost and Capability”, UDT Pacific 2004, Hawaii, USA, 2004. Shanks, G., and Darke, P., “Understanding data quality in a data warehouse”, Australian Computer Journal, 30(4), 1998, pp. 122-128. Snitkin, S. (2003). Collaborative Asset Lifecycle Management Vision and Strategies. Research Report. Dedham, USA: ARC Advisory Group. Spires, C., “Asset and maintenance management – becoming a boardroom issue”, Managing Service Quality, 6(3), 1996, pp. 13-15. Steed, J. C., “Aspects of how asset management can be influenced by modern condition monitoring and information management systems”, IEE, 1988. Strong, D.M., “IT Prosess designs for Improving Information Quality and reducing Exception Handling: A Simulation Experiment”, Information and Management, (31), 1997, pp. 251-263. Strong, D.M., Lee, Y.W. & Wang R.Y., “Data Quality in context”, Communications of the ACM, May 1997, pp. 103- 110. Wand, Y., and Wang, R.Y., “Anchoring Data Quality Dimensions in Ontological Foundations”, Communications of the ACM, 39(11), 1996, pp. 86-95. Wang, R., “A product perspective on data quality management”, Communications of the ACM, 41(2), 1998, pp. 58-65. Wang, R.Y., and Strong, D.M., “Beyond Accuracy: What Data Quality Means to Data Consumers”, Journal of Management Information Systems, 12(4), 1996, pp. 5-33. Woodhouse, J., Asset Management: concepts & practices. Research Article. Kingsclere, UK: The Woodhouse Partnership, 2003. Zikmund, W.G., Business Research Methods, The Dryden Press, 1997.
Shien Lin University of South Australia [email protected]
Andy Koronios University of South Australia [email protected] Abstract: Data Quality (DQ) has been an acknowledged issue for a long time. Researches have indicated that maintaining the quality of data is often acknowledged as problematic, but is also seen as critical to effective decision-making in engineering asset management (AM). The paper reports exploratory research on how engineering asset organizations in Australia are addressing DQ issues, and is drawn from a large scale national-wide DQ survey. Preliminary findings suggest that while the organizations are concerning the quality of data, there is a disconnection between data custodians and data producers and high level data owners. The majority of AM organizations still adopt a reactive approach on DQ management. This paper reports on a national survey as well as structured interviews with a number of stakeholders from engineering asset management organizations. Key Words: Data Quality, Engineering Asset Management
INTRODUCTION Almost every process and task in organisations involves data. Levitin and Redman [18] suggest that data provides the foundation for operational, tactical, and strategic decisions; further, they identify that, through the judicious use of data, managers are able to plan, organize and control an organisation’s resources to seek out new business opportunities, improvements to processes and the development of innovative products and services. As data becomes increasingly important in supporting organizational activities, the quality of the data that managers use becomes critical. Poor-quality data, if not identified and corrected, can have disastrous economic and social impacts on the health of the company [41]. Industry has recently put a strong emphasis on to the area of asset management (AM). In order for organizations to generate revenue they need to utilize assets in an effective and efficient way. Often the success of an enterprise depends largely on its ability to utilize assets efficiently. In other words, asset management is regarded as a set of business-capability building-processes within organisations. Thus, asset management has been regarded as an essential business process in many organizations, and is moving to the forefront of contributing to an organization's financial objectives. As an important initiative proposed by the Australian federal government and the industry sector, studies were commenced in 2003 into the impact of the quality of data on AM organisations including the Australian Navy, state utilities, transportation & mining companies, and local governments. A number of research findings [25,32,17,19,16] were published in various conferences and journals. In 2006, a large scale national-wide survey was conducted into data quality issues in asset management, with a sample
size of 2000 and a response rate of over 20%. This is one of the largest nation-wide surveys of its kind, aimed as directly addressing data quality issues in engineering asset management organisations in Australia. This paper discusses the development of this survey and presents some of its findings.
DEFINITION Data Quality Managers intuitively differentiate information from data, and describe information as data that has been processed. However, data and information are often used synonymously in practice, particularly when addressing quality issues. Therefore, this paper (and the survey) uses “data” interchangeably with “information”, as well as using “data quality” (DQ) interchangeably with “information quality” (IQ). There are a number of theoretical frameworks for understanding data quality. These DQ frameworks [39] [41,40,33,29,15,12,5,9,22,14,10] have been proposed to organize and structure important issues in information quality, albeit from different points of view. This paper (and the survey) follows Wang & Strong’s [41] data quality definition and regards the quality of data as being multi-dimensional, including as it does accuracy, reliability, importance, consistency, precision, timeliness, fineness, understandability, conciseness, and usefulness. In addition, it is also considered that the quality of data is dependent on how the data will be used [3,24,37,8,31,27]. This fitness for use can be defined as the intersection of the quality dimension being considered, the proposed use of the data (purpose), and the data fields which are identified for use in order to fulfil the purpose [23].
Asset Management According to British Standards Institute [4], asset management encompasses activities that are aimed at establishing the optimum way of managing assets to achieve a desired and sustained outcome. The objective of asset management is to optimize the lifecycle value of the physical assets by minimizing the long term cost of owning, operating, maintaining, and replacing the asset, while ensuring the required level of reliable and uninterrupted delivery of quality service [7,35,13]. At its core, asset management seeks to manage the facility’s asset from before it is operationally activated until long after it has been deactivated. This is because, in addition to managing the present and active asset, asset management also addresses planning and historical requirements. Asset management is process-oriented. The AM process itself is quite sophisticated and involves the whole asset lifecycle that can span a long period of time [36]. The lifecycle for a typical asset involves several interdependent stages including design, plan, acquisition, installation, operation, maintenance, rehabilitation and disposal. An overview is shown in Figure 1. At every stage of the process, AM also needs to collaborate and synchronize with other business processes, which is vital to the effective management of engineering assets (as shown in Figure 2). The cost and complexity of engineering assets demands considerable planning to identify appropriate solutions and evaluate investment opportunities. These same characteristics are reflected in the need for an extended acquisition process, a comprehensive request for proposal, and an equally comprehensive purchase agreement that addresses guarantees and warranties. Installation and placing in service of engineering assets is also complex and requires a proper set of processes to manage contractors. Once the asset is acquired, it must be tracked throughout its useful life. Finally, records must be made of its eventual disposition.
Figure 1- Asset Lifecycle Stages (Source: [34])
Figure 2- Collaborative Asset Lifecycle Management (Source: Adopted and modified from [34]) The sophistication of the engineering asset management process requires substantial information to be collected throughout all stages of a typical asset’s lifecycle. This information needs to be maintained for a very long time, often dozens of years in order to identify long-term trends. This kind of process also uses this information to plan and schedule asset maintenance, rehabilitation, and replacement activities. In
order to manage and support the complicated AM process and its data requirements, a variety of specialized technical, operational and administrative systems exist in asset management. These not only manage, control and track the asset through its entire lifecycle, but also provide maintenance support throughout the lifecycle of the asset. Table 1 shows some of these technical systems. Considering the complexity and importance of asset management, these systems are normally bought from multiple vendors and each is specialized to accomplish its task. Unfortunately, this leads to an extremely difficult integration job for the end-user. Asset Lifecycle Stage
Asset Management Technical System
Asset Creation
• • • • • • • • • • • • • • • • • • •
Asset Operation & Maintenance
Computer Aided Design (CAD) Document Management System (DMS) Supervisory Control and Data Acquisition (SCADA) systems Enterprise Asset Management (EAM) Plant Asset Management (PAM) Computerized Maintenance Management Systems (CMMS) Enterprise Asset Maintenance Systems Reliability Centred Maintenance (RCM) Total Productive Maintenance (TPM) Geographic Information Systems (GIS) Product Service Management (PSM) Reliability Assessment Systems Turbo-machinery Safety Systems Rotating Machine Vibration Condition Monitoring Systems Electrical Motor Testing Systems Root Cause Analysis Systems Asset Capacity Forecasting Systems Operational Data Historians Physical Asset Data Warehouse Systems
Table 1: Asset Management Technical Systems (Source: Developed by the authors) Engineering processes rely heavily on input of data and also produce a large amount data. Engineering data itself is quite different to typical business-oriented data as illustrated in Table 2. It has unique data characteristics and complex data capture processes from a large variety of data sources. This large amount of data therefore can suffer from data quality problems. The nature of such data quality problems has not previously been investigated in Australian engineering-oriented organizations. Element Data Environment Data Characteristics
Typical Business Environment Transaction-driven, product-centric business data environments o o o o o
Self-descriptive Static Intrinsic quality Discrete value with fewer or no constraints Current
Engineering Asset Management Continuous data, process-centric open control system and manufacturing data environments o o o o o
Non self-descriptive Dynamic Intrinsic / extrinsic quality Continuous value with constraints (e.g. within a range), precision value Temporal
Transactional data Often structured Easy to audit Can be cleansed using existing tools o Similar data types Inventory data, customer data, financial data, supplier data, transaction data etc
o o o o o
Data Sources
Mainly transaction-based textual records from business activities
Data Capture
o
Disparate data sources o Spatial data – plans/maps, drawings, photo o Textual records – inspection sheets, payment schedules o Attribute records – separate databases, maintenance/renewal records, fault/failure records, field books o Real-time CMS/SCADA o Other sources – existing/previous staff and contractors, photos o Electronically, involving sensors, technical systems such as SCADA systems, condition monitoring systems o Manually, involving field devices, field force, contractors, business rules o Data collected in a variety of formats o Requires to collect substantial data from many different parts of the organization o Data often entered by less/un trained, less dedicated personnel without proper relevant knowledge o Data entry environment can be unstable, harsh, less preorganized o Data entry point can be far from the organization site o Very large amount of data to be maintained for extended time for AM engineering and planning process o Data stored on various operational and administrative systems o Not comprehensive o Process dependent o Complex to integrate data, need both vertical and horizontal data integration o Data to be shared among various technical (e.g. design, operations, maintenance) and business systems o Data to be communicated to an array of stakeholders, business partners and contractors, subcontractors o Need experts with professional knowledge to interpret data o Difficult to translate asset data into meaningful management information
o o o o
Data Category
o
o o
Data Storage
o o
Data Processing
o o o
Data Usage & Analysis
o o o o
Often manually by data providers in fixed format Data often entered by reasonably trained, dedicated personnel with proper relevant knowledge Data collection environment is stable, well pre-organized Data entry point is within the business
Data to be kept in accordance with appropriate compliance requirements Data stored on functional information systems Comprehensive Process independent Easy for data integration Data to be shared only among relevant business systems Data to be communicated to internal stakeholders Use general, common knowledge to interpret data Easy for management use
Time-series streaming data Often unstructured Difficult to be audited Difficult to be cleansed using existing tools Diversity of data types
Inventory data, condition data, performance data, criticality data, lifecycle data, valuation data, financial data, risk data, reliability data, technical data, physical data, GPS data etc
Table 2: Differences between engineering asset data and typical business data (Source: Developed by the authors)
SURVEY DEVELOPMENT – RESEARCH DESIGN
This research focuses on a pluralist methodological approach to ensure that evidence is collected from a great variety of engineering organizations. A national research survey was undertaken in order to provide an overview of data quality issues in Australian engineering asset managing organizations. Case-based research was then undertaken with a selected number of organizations to provide richer data about the sources of data quality problems and to seek reasons for these problems. Pinsonneault and Kraemer [28] propose that survey research is believed to be well understood and applied by management information systems (MIS) scholars. The survey-based research requires “standardized information from and/or about the subjects being studied” and usually is in the form of questionnaires. Zikmund [43] and Davis and Cosenza [6] claim that surveys can provide quick, inexpensive, efficient and accurate means of assessing information about the population. In contrast to case studies, survey questionnaires have certain advantages: • • • •
they reach a geographically-dispersed sample simultaneously and at a relatively low cost; standardised questions make the responses easy to compare; they capture responses people may not be willing to reveal in a personal interview; results are not open to different interpretations by the researcher.
Though survey research is among the more popular methods used by the information systems research community, survey research has its own weaknesses. The analysis by Pinsonneault and Kraemer [28] indicates that survey research is often misapplied or ineffective due to several reasons: • • •
Single-method designs where multiple methods are needed; Unsystematic and often inadequate sampling procedures; Low response rates.
The researchers were fully aware of these issues and important steps were taken during the survey development. For example, the survey study is one part of a series of research approaches (including interviews, field observations and other research forms). Understandings of the data quality issues in the given domain (asset management organisations) were accumulated to avoid the limitation of single research method design.
Sampling Procedure As the sample was part of a nation-wide survey of asset management organisations, precautions were taken during the sampling stage in order to obtain a representative sample. Overall, 2000 participants were identified, using statistical sampling methods, through several Australian Business Directories include Kompass and “Business who is who” (Australian Business Directory). The survey population for the questionnaire was chosen from engineering asset management organizations based in Australia. These organizations represent a variety of industries: • • • • •
Utilities (water, electricity, gas, oil); Mining & resources; Transport (rail, airline, ship, automobile); Defence; and Local government.
Considering the uneven population distribution in different Australian states, different numbers of survey participants were accordingly selected, as shown in Table 3 & 4: Unit: 1,000
Australian State Total Population %
NSW & ACT
VIC
QLD
SA
WA
TAS
NT
Total
7,055
4,973
3,882
1,534
1,982
482
200
20,111
35%
25%
19%
8%
10%
2%
1%
100%
Table 3: Australian State Population Distribution Year 2005 (Source: Australian Bureau of Statistics [2]) (Note: The Australian States: NSW – New South Wales; ACT – Australian Capital Territory; VIC – Victoria; QLD – Queensland; SA – South Australia; WA – Western Australia; TAS – Tasmania; NT – Northern Territory)
Australian State Defense Local Government Mining & Resources Transport (rail, airline, ship, automobile) Utility (electricity, gas, oil) Water utility Total %
NSW & ACT 21
VIC
QLD
SA
WA
TAS
NT
Total
15
3
43
82
113
143
134
126
170
71
62
20
18
160
132
86
33
21
36
2
310
81
34
53
21
48
10
247
58 476 24%
64 404 20%
68 311 16%
7 236 12%
17 431 22%
7 70 4%
51
64
801
2
333
6 72 4%
227 2,000 100%
Table 4: Number of Survey Samples by the State (Note: Western Australia (WA) is the center of Australian mining and manufacturing industry where many engineering asset management organisations are located.) The size of organisations was also considered, while the focus has been placed on the middle-sized organizations (Figure 3). Thus, it is believed that the chosen sample is representative.
Size of The Organisations 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
40%
16%
14%
13%
8% 4%
< 10
2% 10 - 19
20 - 49
50 - 99
100 - 499 500 - 999
1,000 4,999
1%
2%
5,000 9,999
10,000+
Figure 3: Size of Organisations
Increasing Response Rate This research uses traditional, physical-mail surveys, instead of online or email surveys. This reflected the outcomes of research, showing that mail surveys usually have relatively higher response rates. All the survey questionnaires were posted with reply-paid return envelopes. More importantly, substantial efforts were undertaken to find the names of participants in each chosen organisation (e.g. CEO and managers’ names), which enabled a personalised envelope and cover letter to be delivered to the participants. Moreover, a reminder letter was posted to all participants after the deadline. Overall, this survey achieved a response rate of 23.2% which the authors consider to be satisfactory for this type of national survey.
Research Objectives and Questions This study is aimed at obtaining an overall understanding of how asset management organisations in Australia address concerns related to various data quality issues. In particular, this survey attempts to answer the following questions: 1) To what extent, are Australian asset management organisations aware of the importance of data quality? 2) What are the current data quality problems and issues that these organisations are facing; and how do they manage data quality? 3) What strategies are currently employed in addressing data quality problems in engineering asset management organisations? Through the development of the survey questions, a number of previous data quality studies were considered (as discussed in the next section). It is thought that comparisons can also go some way in addressing the following questions: 4) To what extent do data quality issues in engineering asset management organisations differ from other types of enterprises and organisations? 5) To what extent do data quality issues in Australia differ from organisations in other countries? The results of this survey are thought to be of value for developing a DQ framework specific to engineering asset management organisations. These results will form the foundation for further research in order to:
a. b. c. d. e. f.
Perform data audits to identify the nature and volume of DQ problems; Categorise the various DQ problems; Identify DQ dimensions to be assessed; Prioritise the dimensions; Identify metrics that specify the criteria to be used to measure DQ; Develop specific assessment metrics for each dimension.
Limitations of Previous Data Quality Surveys International consulting companies have conducted global data quality surveys. The PriceWaterHouseCoopers and Advance Information Management (AIM) groups are two of the most wellknown examples. PriceWaterHouseCoopers’s Global Data Management Survey [30], which built on a 2001 survey, selected 450 companies across the US, UK and Australia, targeting senior management, or equivalent executives at the same level (89% of respondents were senior IT staff: CIOs, IT directors, data controllers, network administrators, etc; and the remaining 11% of respondents were senior executives from outside the IT organisation). The survey results reveal that the top six data quality initiatives among senior management included: • • • • • •
Improving data accuracy (26%); More rigorous data management (14%); System upgrades (13%); Improving security (11%); Data standardisation (10%) and Improving usage/analysis of data (10%).
Given the topic of the survey questions, and the level of respondents, it appears that the PriceWaterHouseCoopers’ data quality survey primarily focused on the IT industry and was concerned with the strategic development of data management within organisations. Although the overall response rate was not published, it is felt that the sample size of 450 (spread over three countries) may not be able to provide a comprehensive understanding of data quality issues. In particular, the focus on IT companies excludes the government sector and engineering asset management organisations, which cannot provide answers to the research questions proposed in this study. AIM Software [1] in conjunction with the Vienna University of Economics and Business Administration, has also attempted a series of international data surveys. The Global Data and Risk Management survey was originally conducted in 2004, with a sample size of 1,736 financial institutions in 63 countries from all of the regions (North America, Central & South America, Western Europe, Central and Eastern Europe, the Middle-East, Asia, Australia and South Africa). The responding companies were requested, between January and August 2004, to answer a 14-question questionnaire on data management. This survey was then modified and replicated in 2005 and 2006 (the 2006 survey has not yet been completed). In 2005, 1070 companies from 88 countries were selected (62% banks, 22% asset management, 10% broker/dealer and 6% insurance) to answer 9 questions. The findings were published on August 2005 and can be accessed online (from AIM official website - http://www.aim-sw.com). AIM’s survey provides a comprehensive picture of how companies expressed concerns regarding data management issues, across different countries. However, as the survey has a particular interest in financial institutions, the findings may not be applicable in engineering asset management organisations Woodhouse [42] and GAO [11] have argued the uniqueness of data management in engineering asset
management organisations. Moreover, when looking at the actual questions, the AIM survey aims to focus on the current planning methods and future strategy developments for data management. It is difficult to anticipate that a 9-question survey is adequate to explore the actual data quality problems in practice, at both business and operational levels within organisations. OMNILINK [26] conducted the first industry-wide data management survey in Australia in 2004. The sample was drawn from a range of sectors and industries in Australia and New Zealand, from small to large organisations. This survey was replicated in 2005 (the findings are yet to be published). OMNILINK’s survey addressed several data quality issues including data accuracy, data security and data currency. The report provides an analysis of current data management practices, in both the public and private sectors, and an insight into the role of Geographic Information Systems (GIS) in data management. After examining this report, it appears that OMNILINK primarily emphasized the use of IT in data management. The issues associated with human factors were not discussed in detail.
Questionnaire Development A number of issues were considered during the development of the current questionnaire, in order to better address the uniqueness and comprehensiveness of the survey finding. Firstly, unlike the above questionnaire surveys, which only targeted participants at a specific organisational level, four types of stakeholders have been identified intuitively for the current survey: data producers, data custodians, data consumers, and data managers [38], as defined below: 1. 2. 3. 4.
Data producers are those who create or collect asset data e.g. asset managers; Data custodians are those who design, develop, manage, and operate the asset management systems e.g. IT managers; Data consumers are those who use the asset information in their work activities e.g. maintenance engineers, senior managers; Data managers are those responsible for managing the entire data quality in asset management systems.
Based on these four distinct data roles, the survey consists of five sections. The first section collects demographic data of the four data roles, and the rest four sections correspond to the four different data roles. Respondents choose one section from the rest four sections according to their particular data role. This ensures that the results can present valuable and insightful issues at all organisational levels. In total, an average 20 questions are to be answered by each individual respondent from the almost 80 questions which were proposed, are being derived from a well-known TOP framework. Mitroff and Linstone [21] argue that any phenomenon, subsystem or system can be analysed from what they call a Multiple Perspective method – employing different ways of seeing, to seek perspectives on the problem. These different ways of seeing are demonstrated in the TOP model of Linstone [20] and Mitroff and Linstone [21]. The TOP model allows analysts to look at the problem context from either Technical or Organizational or Personal points of view: • • •
The technical perspective (T) sees organizations as hierarchical structures or networks of interrelationships between individuals, groups, organizations and systems. The organisational perspective (O) considers an organization’s performance in terms of effectiveness and efficiencies. For example, leadership is one of the concerns. The personal perspective (P) focuses on individual concerns. For example, the issues of job description and job security are some of the main concerns in this perspective.
This framework has been used in previous studies (Koronios, Lin & Gao 2005; Lin, Gao & Koronios
2006) using interview case studies. By employing this approach and using experiences/findings obtained from previous studies, the survey questionnaire design is believed to cover a wide area of data quality issues (please email us for a copy of the survey).
Data Validity The multi-section questionnaires were mailed to a large, random sample of 2000 asset managers, data producers, data custodians and data users, in 1100 geographically dispersed engineering asset management organizations in Australia (including 572 organizations in the public sector). The same surveys were posted to different participants within same organisation (to address the issues of those with different data roles). Three types of questions were classified during the design stage: 1. Questions covering all data roles E.g. How important is data quality to the success of your organisation? 2. Questions unique to individual data role E.g. Do you have the following difficulties for any data entry task? – Data producers only 3. Different questions for different role, but can be used for cross-checking E.g. How often do you monitor the quality of your data? – Data / Asset managers How often do you have any form of management reviews done in relation to your data collection performance? – Data producers In addition, the questionnaire was pre-tested by initially mailing it to 15 companies. Changes were then incorporated based on feedback received from the pilot survey, before the final questionnaires were mailed to the remaining companies. A number of changes and additions were made to the instrument after pre-testing. These included: • • •
Refining of some of the questions to increase clarity and remove ambiguities; Adding of some additional items to achieve greater integrity; and Changes to some of the measurement scales.
The information obtained from the pre-tests was used to alter the questionnaire to achieve higher validity. Pre-testing also involved a preliminary analysis of the data collected, ensuring that the data collected was appropriate for testing the research model and therefore addressing the problem [43]. The overall approach is therefore considered to be adequate in addressing validity issues of the survey.
METHOD OF ANALYSIS Statistical methods were used for the analysis of the responses. Software package SPSS were used. A three-stage analysis will be performed including: 1. Basic descriptive statistics analysis including frequencies, means, standard deviation; 2. Multivariate data analysis techniques including factor analysis, multiple regression analysis; 3. Qualitative analysis on participants comments As the analysis of the data has not been yet completed, this paper provides preliminary and ‘first impressions’ results of this research.
FINDINGS AND DISCUSSION This survey is the first national data quality survey performed in Australia, focusing specifically on the data quality issues of engineering asset management organisations. The following findings show the different attitudes and perceptions towards data quality. More importantly, it shows that the current strategy, policy and tools that the organisations employ for their data management solution.
Current Attitude & Awareness towards Data Quality
How important is data quality to the success of your organisation? 60% 50% 40% 30% 20% 10% 0%
Data Owner
po rta nt Im po rta nt Do n’ tk No no tv w er y im po No r ta ti m nt po rta nt at al l
Data Custodian Data Consumer
Ve ry
im
Cr itic
al
Data Producer
The above result shows that the majority of participants felt that data quality is an important success determinant for their organizations. However, it also suggests that the data consumers have a much higher needs of quality data.
Current Perception of Organisational Data Quality Management From Data Owners’ point of view Assuming you get sufficient data, how often do you rely on your data to make critical decisions?
How often do you get sufficient data fromthe systems in order to make decision for daily work?
40%
48%
50%
40%
40% 30%
47%
50%
60%
30%
26%
20%
20%
14%
10%
10% 1%
1%
0%
7%
10%
5%
1%
0%
0% All the time
Sometimes
Occasionally
Don’t know
Rarely
Never
All the time
Sometimes
Occasionally
Don’t know
Rarely
Never
The above left figure shows that the majority of data owners (Asset managers) are happy with the amount of the data that they can access. However, there is a relatively small group (about 7%), who is not satisfied with the quality of data for decision making as shown in the figure on the right. From Data Consumers’ point of view How often do you get sufficient data fromthe systems in order to make decision for daily work? 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
37%
Assuming you get sufficient data, how often do you rely on your data to make critical decisions?
40%
60% 50%
48% 41%
40% 30%
17%
20% 5% 1%
0%
5%
10%
5%
1%
0%
0% All the time
Sometimes
Occasionally
Don’t know
Rarely
Never
All the time
Sometimes
Occasionally
Don’t know
Rarely
Never
A consistent opinion is suggested by the data consumers’ group. However, it does necessarily show that the data quality problem is not a major concern within the participating organisations. From Data Custodians’ point of view Is the data in your system(s) up-to-date to the required level?
Do you think that the data in your system(s) is accurate to the required level? 55%
60% 50% 40%
34%
30% 20%
11%
10%
0%
0% I have total confidence
It is not a critical problem
There are problems
I don’t rely on them at all
50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
47% 35%
7%
Strongly agree
7%
Agree
Don’t know
4%
Disagree
Strongly disagree
From Data Producers point of view The data in the systemis error free, do you agree?
How often do you have confidence (accuracy) with the data captured by the electronic sensors or hand-held devices?
37%
15%
10%
2%
0%
0% Strongly agree
Agree
Neutral
Disagree
Strongly disagree
Al l th e
5%
10%
tim e
10%
18% 2%
2%
ap pl ica ble
20%
31%
No t
25%
35%
40% 35% 30% 25% 20% 15% 10% 5% 0%
Ne ve r
27%
Ra rel y
27%
Do n’t kn ow
30%
Oc ca sio na ly
35%
So me tim es
40%
The above figures shows that the data producers and data custodians acknowledged that there were data quality problems. Especially, data producers do not have much confidence on their data quality. Perhaps, the data owners and data consumers may have higher levels tolerance of poor quality data. Nevertheless data quality problem is still facing critical challenges in most organisations. Also, the different attitudes found between groups may ring the alert bell that there may be a disconnection between the operational level personnel and the strategic level managers. Some problems are suggested by the data custodians. For example, it suggests that moving/migrating data may generate many quality related problems, as shown below.
Fromyour perspective, what are the major problems when moving data? 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
42% 30% 19% 9%
I can move data from I have experienced The moving process different sources some problems in the can be done, but there without any problems past; such problems are unresolved will be easily resolved problems
I have experienced some significant problems
Current Strategies, Policies & Tools Employed for Achieving High Quality of Data According to the 60% of data owners, there is data management strategy for data quality in place in their organisations, as shown in below. Do you already have a data management strategy in place in your organisation? 70%
60%
60% 50% 34%
40% 30% 20%
6%
10% 0% No
Yes
Don't know
Down to the data capture level, data producers listed different ways of data collection and entry. How do you collect asset data e.g. asset health data? 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
How do you enter the data into the system?
39% 27%
26%
8%
I physically inspect the assets and write down the details
I use hand-held devices to collect asset data
I use electronic sensors which will read and record the data automatically
Other
46%
50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
25%
19% 4% It is a manual process; I need to type into the system from the paper records
I give my paper records to office clerks who are responsible for data entry
I can enter them I upload the data from where I am; I electronically have a networked (e.g. from a mobile system for docking station) data entry when returning to the office.
The majority of data producers still adopt a manual data entry process, which primarily rely on paper records. Especially, the figure below suggests that these data may not be entered immediately on site. Approximately 49% of these data were entered at the office after several days. Thus, the accuracy and completeness of these data may not be satisfactorily achieved. When and where do you make data entries to the systems? 60%
49%
50% 40%
32%
30% 20%
12%
7%
10% 0% At the location where I At my office – When I captured the data – go back from field Immediately
At my office – After several days
I give my data to someone else, so I don’t know
6% Other
Towards Future It is “interesting” to find that 39% of asset owners have no plans to implement any data management solutions in near future. While the answer from Data custodians to the question “Has implementing a data quality system had a positive impact on the success of any major IT implementations which your organization has put in place (e.g. Enterprise Resource Planning)?” shows that no data quality systems (e.g. data profiling and cleansing systems) were implemented or planned to be implemented. positive impact?
Do you plan to implement a data management solution in the following way? 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
39% 24% 17%
13% 6% 2%
45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
39% 28% 20% 11% 2% A major impact
Proprietary Buy a solution Buy a solution Outsource the Outsource the Not planned development and extend development processing
A slight impact
Don't know
No impact at all
No data quality system implementation
CONCLUSION AND FUTURE RESEARCH This paper only included a small proportion of survey findings. Nevertheless, these results suggest that while the organizations are concerned about the quality of data, there is lack of scrutinized discussion on the various issues associated with data quality problems. More importantly, there is a disconnection between data custodians and data producers and high level data owners. The majority of engineering asset organisations in Australia has no plans to neither implement any data quality management solutions nor develop any strategic plans. This finding is very different from AIM and PriceWaterHouse findings. Perhaps, the engineering asset management organisations in Australia still adopt a reactive approach and only focus on the daily operations. A more comprehensive analysis will show the different attitudes and management strategies in relation to sizes of the organisations and the industries that they operate within. Further, these findings will be compared with the general data quality surveys. The result will be published in the research report or upcoming journal paper. This paper provided a better understanding of data quality issues for asset management and is assisting in identifying elements which will contribute towards the development of a data quality framework specific to engineering asset management. This in turn will assist in providing useful advice for improving data quality in this area. It is thought that more and more organisations are putting efforts and financial resources in data management solutions, there is an increasing need of guidelines for helping them develop appropriate strategies and employ the right tool. Perhaps this is why data quality research becomes more critical.
REFERENCES [1] [2]
[3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]
[15] [16]
[17]
[18] [19]
[20] [21] [22]
[23] [24] [25] [26]
AIM Software, “Global Data and Risk Management Survey 2005”, Research Report, 2005. Australian Bureau of Statistics, “Australian Social Trends 2005”, online accessed on 15 June 2006, http://www.abs.gov.au/ausstats/[email protected]/94713ad445ff1425ca25682000192af2/8ef24f9fe10ee078ca25703b 0080ccaf!OpenDocument Ballou, D.P., & Pazer, H.L. (1995). Designing Information Systems to Optimize the Accuracy-timeliness Tradeoff. Information Systems Research. 6(1), 51-72. British Standards Institution, PAS 55: Asset Management. Part 1: Specification for the optimised management of physical infrastructure asset, 2004. Caballero, I., & Piattini, M. (2003). Data Quality Management Improvement. ACS/IEEE International Conference on Computer Systems and Applications (AICCSA’03), July 14-18 2003, Tunisia. 56-65. Davis, D. & Cosenza, R.M., Business Research for Decision Making, 2nd Edition, Boston, PWS-Kent, 1988. Eerens, E., Business Driven Asset Management for Industrial & Infrastructure Assets, Le Clochard, Australia, 2003. English, L.P., Improving Data Warehouse and Business Information Quality: Methods for reducing costs and increasing Profits, Willey & Sons, 1999. Eppler, M.J. (2001). The Concept of Information Quality: An Interdisciplinary Evaluation of Recent Information Quality Frameworks. Studies in Communication Sciences. 1, 167-182. Firth, C. (1996). Data Quality in Practice: Experience from the Frontline. The Proceedings of 1996 Conference of Information Quality. 25-26 October 1996, Cambridge, USA. GAO (U.S. General Accounting Office), “Water Infrastructure: Water Utility Asset Management”, GAO-04461, Washington, D.C., 2004. Giannoccaro, A., Shanks, G., & Darke, P. (1999). Stakeholder perceptions of data quality in a data warehouse environment. Australian Computer Journal. 31(4), 110-117. IPWEA, International Infrastructure Management Manual, Australia/New Zealand Edition, 2002. Jarke, M., Jeusfeld, M.A., Quix, C., & Vassiliadis, P. (1998). Architecture and Quality in Data Warehouses. Proceedings of Tenth International Conference CAiSE’98. 8-9 June 1998, Pisa, Italy. Springer-Verlag. 93113. Kahn, B.K., Strong, D.M., and Wang, R.Y., “Information Quality Benchmarks: Product and Services Performance”, Communications of the ACM, 45(4), 2002, pp. 184-192. Koronios, A., and Haider, A., “Managing Engineering Assets: A Knowledge Based Asset Management Methodology Through Information Quality”, E-Business and Organisations in the 21th Century, 2003, pp. 443-452. Koronios, A., Lin, S., & Gao, J. (2005). A Data Quality Model for Asset Management in Engineering Organisations. Proceedings of the 10th International Conference on Information Quality (ICIQ 2005). 4-6 November 2005, Cambridge, MA. USA. 27-51. Levitin, A.V. and Redman, T.C., “Data as a resource: properties, implications and prescriptions”, Sloan Management Review, 40(1), 1998, pp. 89-101. Lin, S., Gao, J., & Koronios, A. (2006). Key Data Quality Issues for Enterprise Asset Management in Engineering Organisations. International Journal of Electronic Business Management (IJEBM). 4(1), 96110. Linstone, H. A., Decision Making for Technology Executives: Using Multiple Perspectives to Improve Performance, Artech House Publisher, 1999. Mitroff, I. I. and H. A. Linstone, The Unbounded Mind: Breaking the Chains of Traditional Business Thinking, New York, OxFord University Press, 1993. Nauman, F., & Roth, M. (2004). Information quality: How good are off-the-shelf DBMS? Proceedings of the 9th International Conference on Information Quality (ICIQ-04). 5-7 November 2004, Cambridge, MA. USA. 260-274. Neely, P., & Pardo, T., “Teaching Data Quality Concepts Through Case Studies”, Center for Technology in Government, Albany, 2002. Neely, P., “A Framework for the Analysis of Source Data Revised”, Proceedings of AMCIS 2001. Neely, P., Gao, J., Lin, S., & Koronios, A., “The Deficiencies of Current Data Quality Tools in the Realm of Engineering Asset Management”, Proceedings of AMCIS, Acapulco, México, 2006. OMNILINK, “2004 Data Management Report”, Research Report, Sydney, 2004.
[27] [28] [29]
[30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43]
Orr, K., “Data Quality and System Theory”, Communications of the ACM, 41(2), 1998, pp. 66-71. Pinsonneault, A. and Kraemer, K. L., “Survey research methodology in management information systems: An assessment”, Journal of Management Information Systems, 10(2), 1993, pp. 75-105. Price, R.J., & Shanks, G. (2004). A Semiotic Information Quality Framework. Decision Support in an Uncertain and Complex World: The IFIP TC8/WG8.3 International Conference 2004. 1-3 July 2004, Prato, Italy. 658-672. PriceWaterHouseCoopers, “Global Data Management Survey”, Research Report, 2004. Salaun, Y., and Flores, K., “Information Quality: Meeting the Needs of the Consumer”, International Journal of Information Management, 21(1), 2001, pp. 21-37. Saunders, D., “Innovation in Asset Management - Achieving a Nexus of Cost and Capability”, UDT Pacific 2004, Hawaii, USA, 2004. Shanks, G., and Darke, P., “Understanding data quality in a data warehouse”, Australian Computer Journal, 30(4), 1998, pp. 122-128. Snitkin, S. (2003). Collaborative Asset Lifecycle Management Vision and Strategies. Research Report. Dedham, USA: ARC Advisory Group. Spires, C., “Asset and maintenance management – becoming a boardroom issue”, Managing Service Quality, 6(3), 1996, pp. 13-15. Steed, J. C., “Aspects of how asset management can be influenced by modern condition monitoring and information management systems”, IEE, 1988. Strong, D.M., “IT Prosess designs for Improving Information Quality and reducing Exception Handling: A Simulation Experiment”, Information and Management, (31), 1997, pp. 251-263. Strong, D.M., Lee, Y.W. & Wang R.Y., “Data Quality in context”, Communications of the ACM, May 1997, pp. 103- 110. Wand, Y., and Wang, R.Y., “Anchoring Data Quality Dimensions in Ontological Foundations”, Communications of the ACM, 39(11), 1996, pp. 86-95. Wang, R., “A product perspective on data quality management”, Communications of the ACM, 41(2), 1998, pp. 58-65. Wang, R.Y., and Strong, D.M., “Beyond Accuracy: What Data Quality Means to Data Consumers”, Journal of Management Information Systems, 12(4), 1996, pp. 5-33. Woodhouse, J., Asset Management: concepts & practices. Research Article. Kingsclere, UK: The Woodhouse Partnership, 2003. Zikmund, W.G., Business Research Methods, The Dryden Press, 1997.
