Data Warehouse

The paper proposes an overview of decision support systems in order to define the role of a system to assist decision in university management. The authors present new technologies and the basic concepts of multidimensional data analysis using models of business processes within the universities. Based on information provided by scientific literature and on the authors’ experience, the study aims to define selection criteria in choosing a development environment for designing a support system dedicated to university management. The contributions consist in designing a data warehouse model and models of OLAP analysis to assist decision in university management.

The paper proposes an overview of decision support systems in order to define the role of a system to assist decision in university management. The authors present new technologies and the basic concepts of multidimensional data analysis using models of business processes within the universities. Based on information provided by scientific literature and on the authors' experience, the study aims to define selection criteria in choosing a development environment for designing a support system dedicated to university management. The contributions consist in designing a data warehouse model and models of OLAP analysis to assist decision in university management.

Data warehouses can be defined as 'subject-oriented ', integrated, time-varying, non-volatile collections of data that is used primarily in organizational decision making. Nowadays, data warehousing became an important strategy to integrate heterogeneous information sources in organizations, and to enable On-Line Analytic Processing (OLAP). Unfortunately, neither the accumulation, nor the storage process, seem to be completely credible. For example, it has been suggested in the literature that more than $2 billion of U.S. federal loan money have been lost because of poor data quality at a single agency, that manufacturing companies spent over 25% of their sales on wasteful practices, a number which came up to 40% for service companies. The question that arises, then, is how to organize the design, administration and evolution choices in such a way that all the different, and sometimes opposing, quality user requirements can be simultaneously satisfied. To tackle this problem, this thesis contributes as follows:

Time is one of the dimensions we frequently find in data warehouses allowing comparisons of data in different periods. In current multi-dimensional data warehouse technology changes of dimension data cannot be represented adequately since all dimensions are (implicitly) considered as orthogonal. We propose an extension of the multi-dimensional data model employed in data warehouses allowing to cope correctly with changes in dimension data: a temporal multi-dimensional data model allows the registration of temporal versions of dimension data. Mappings are provided to transfer data between different temporal versions of the instances of dimensions and enable the system to correctly answer queries spanning multiple periods and thus different versions of dimension data.

Data are considered to be important organizational assets because of their assumed value, including their potential to improve the organizational decision-making processes. Such potential value, however, comes with various costs, including those of acquiring, storing, securing and maintaining the given assets at appropriate quality levels. Clearly, if these costs outweigh the value that results from using the data, it would be counterproductive to acquire, store, secure and maintain the data. Thus cost-benefit assessment is particularly important in data warehouse (DW) development; yet very few techniques are available for determining the value that the organization will derive from storing a particular data table and hence determining which data set should be loaded in the DW. This research seeks to address the issue of identifying the set of data with the potential for producing the greatest net value for the organization by offering a model that can be used to perform a cost-benefit analysis on the decision support views that the warehouse can support and by providing techniques for estimating the parameters necessary for this model.

The data warehouse design task needs to consider both the end-user requirements and the organization data sources. For this reason, the data warehouse design has been traditionally considered a reengineering process, guided by requirements, from the data sources.Most current design methods available demand highly-expressive end-user requirements as input, in order to carry out the exploration and analysis of the data

The Data Warehouse techniques for powered strategic decision-making information are dominating the companies attention since the 1990s. The healthcare organizations, including the public healthcare organizations, are adopting this technology to achieve the better efficiency in the health management. The objective of this work is to present a Data Warehouse project for the management of the public health. The Data Warehouse will supply the Health Secretary of São Paulo with strategic information through the integration from several isolated data sources. This Paper details the methodology and strategy adopted for the project, the used tools, the challenges and the solutions. The conclusion show the success in the implementation of the first part of project and stimulate the continuity of the project. The experience obtained with the implementation of the data warehouse showed by this work, can be used to similar projects in the health care organizations.

Enhancing on line analytical processing through efficient cube computation plays a key role in Data Warehouse management. Hashing, grouping and mining techniques are commonly used to improve cube pre-computation. BitCube, a fast cubing method which uses bitmaps as inverted indexes for grouping, is presented. It horizontally partitions data according to the values of one dimension and for each resulting fragment it performs grouping following bottom-up criteria. BitCube allows also partial materialization based on iceberg conditions to treat large datasets for which a full cube pre-computation is too expensive. Space requirement of bitmaps is optimized by applying an adaption of the WAH compression technique. Experimental analysis, on both synthetic and real datasets, shows that BitCube outperforms previous algorithms for full cube computation and results comparable on iceberg cubing.

Poor performance can turn a successful data warehousing project into a failure. Consequently, several attempts have been made by various researchers to deal with the problem of scheduling the Extract-Transform-Load (ETL) process. In this paper we therefore present several approaches in the context of enhancing the data warehousing Extract, Transform and loading stages. We focus on enhancing the performance of extract and transform phases by proposing two algorithms that reduce the time needed in each phase through employing the hidden semantic information in the data. Using the semantic information, a large volume of useless data can be pruned in early design stage. We also focus on the problem of scheduling the execution of the ETL activities, with the goal of minimizing ETL execution time. We explore and invest in this area by choosing three scheduling techniques for ETL. Finally, we experimentally show their behavior in terms of execution time in the sales domain to understand the impact of implementing any of them and choosing the one leading to maximum performance enhancement.

Data warehousing and Online Analytical Processing (OLAP) technologies are now moving onto handling complex data that mostly originate from the web. However, integrating such data into a decision-support process requires their representation in a form processable by OLAP and/or data mining techniques.

Abstract: in this paper the implementation of Data Warehouse SAP BW in the Production Company is presented. A data warehouse is designed for efficient query processing. The technical environment and data structures are optimized for answering business questions. Data warehouse is stored persistently by time over a particular time period. Users can use comprehensive analysis tools to access data. We will try to answer the question how to implement the SAP Business Warehouse in a company

Advances in computer and networking technology have led to the introduction of very powerful hardware and software platforms that can collect, manage and distribute large amount of pertinent data. Data Warehousing is one of the most interesting and dynamic among the new technological transitions. It is a repository of subjectively selected and adapted operational data, which can successfully answer any ad hoc, complex, statistical or analytical queries and it provides means for implementing effective decision support system by utilizing data scattered all over the organization. The aim of the paper is to generate interest and understanding among the library and information professionals regarding the concept of data warehousing. Major characteristics, goal, structure, application and advantages of data warehousing are also described in detail.

Data warehouse is used to provide effective results from multidimensional data analysis. The accuracy and correctness of these results depend on the quality of the data. To improve data quality, data must be properly extracted, transformed and loaded into the data warehouse. This ETL process is the key to the success of a data warehouse. In this paper we propose a conceptual ETL framework for an object oriented data warehouse design, the framework is called UCLEAN. This framework takes into account the concept of requirements of the users .The data is extracted from different UML sources and is converted into a multidimensional model. It is then cleaned and loaded in the data warehouse. We validate the effectiveness of the framework through a case study.

ABSTRACT A data warehouse provides information for analytical processing, decision making and data mining tools. As the concept of real-time enterprise evolves, the synchronism between transactional data and data warehouses, statically implemented, has been redefined. ...

Plant-specific data is managed in heterogeneous formats and is dispersed geographically. Based on this data, efficient analyses require a materialised integration, often realised with data warehouse technology today. We describe the requirements, problems and solution strategies for domain-crossing integration as the fundament for analysing plant biological data based on three current case studies. First, we introduce a system for retrieval of markers and mapping positions based on clustering of ESTs. The second case study illustrates the steps for diversity studies after genotyping a collection of about 3,000 ryegrass accessions (Lolium spp.), whereas in the third example data of approximately 250 barley cultivars (Hordeum vulgare) were used for associating haplotype-and SNP-patterns with malting parameters. For all case studies, we integrate data from different domains-sequence and marker data as well as IPK Genebank data including passport and phenotypic information. Specific problems associated with plant biological data and possible solution strategies are shown.

Decision makers in the tourism sector face real challenges. There is no chance to access a unified data source that can supply information to meet their inquiries and expectations. In this paper, we propose a data warehouse system for supporting the decision makers in the Egyptian tourism sector. The methodology followed to build this system is descried in details to guide the other developing countries who are trying to support their decision makers in the tourism sector by building a similar data warehouse system. In addition, we evaluated the tourism data warehouse system. The results of evaluation indicated that it can meet the needs and requirements of the decision makers in the tourism sector.

RESUMO. O conhecimento armazenado ao longo dos anos nas bases de dados organizacionais é um bem vital para a sobrevivência de uma empresa neste mercado competitivo. Para a sua efetiva utilização como apoio ao processo decisório de uma organização, os dados precisam estar organizados de uma forma que possibilite o seu acesso rápido e fácil entendimento. Este trabalho apresenta a modelagem dimensional como alternativa para a construção de um data warehouse, mostrando os seus elementos, algumas técnicas de modelagem, assim como os passos para o seu desenvolvimento, atendendo assim aos requisitos citados. Também apresenta os resultados e considerações, de um estudo de caso, com a aplicação do modelo dimensional para a criação de um data mart, para um portal jurídico da internet Palavras-chave: armazém de dados, modelagem dimensional, suporte a decisão. ABSTRACT. Developing data warehouse using dimensional model. Stored knowledge in organizational databases during the years is vital for the survival of a company in a competitive business world. For its effective use as support in a decision-making process of a organization, data must be organized for fast access and easy comprehension. A dimensional modeling is provide as an alternative for the construction of a data warehouse, with its elements, modeling techniques, and steps for its development to meet the mentioned requirements. Results and considerations about a case study are given, together with the application of a dimensional model for the creation of a data mart, for a juridical internet portal.

Since many years, theoretical concepts of Data Mining have been developed and improved. Data Mining has become applied to many academic and industrial situations, and recently, soundings of public opinion about privacy have been carried out. However, a consistent and standardized definition is still missing, and the initial explanation given by Frawley et al. has pragmatically often changed over the years. Furthermore, alternative terms like Knowledge Discovery have been conjured and forged, and a necessity of a Data Warehouse has been endeavoured to persuade the users. In this work, we pick up current definitions and introduce an unified definition that covers existing attempted explanations. For this, we appeal to the natural original of chemical states of aggregation.

This paper presents a new approach to develop the strategy of Model–View–Controller architecture pattern in Business Intelligence (BI) architecture. The Business Intelligence architecture consists of the ETL tools and processes, the data warehouse, the technical infrastructure, and the user-interface tools. The major issue related to Business Intelligence architecture is to store the data from heterogeneous sources. The heterogeneous sources might lead to inconsistencies in storage and retrieval of data. Another issue related to Business Intelligence architecture is to provide multiple and synchronized views. The Business information must be presentable in various formats with ‘look and feel ’ approach. Model-View-Controller provides independent approach to the components of Business Intelligence architecture. Model provides guideline for storing the heterogeneous data in one format and implements core functionality of the system. View provides multiple and synchronized views so tha...

Revenue Management’s most cited definitions is probably “to sell the right accommodation to the right customer, at the right time and the right price, with optimal satisfaction for customers and hoteliers”. Smart Revenue Management (SRM) is a project, which aims the development of smart automatic techniques for an efficient optimization of occupancy and rates of hotel accommodations, commonly referred to, as revenue management. One of the objectives of this project is to demonstrate that the collection of Big Data, followed by an appropriate assembly of functionalities, will make possible to generate a Data Warehouse necessary to produce high quality business intelligence and analytics. This will be achieved through the collection of data extracted from a variety of sources, including from the web. This paper proposes a three stage framework to develop the Big Data Warehouse for the SRM. Namely, the compilation of all available information, in the present case, it was focus only the...

A data-parallel framework is very attractive for large-scale data processing since it enables such an application to easily process a huge amount of data on commodity machines. MapReduce, a popular data-parallel framework, is used in various fields such as web search, data mining and data warehouses; it is proven to be very practical for such a data-parallel application. A star-join query is a popular query in data warehouses that are a current target domain of data-parallel frameworks. This article proposes a new algorithm that efficiently processes star-join queries in data-parallel frameworks such as MapReduce and Dryad. Our star-join algorithm for general data-parallel frameworks is called Scatter-Gather-Merge, and it processes star-join queries in a constant number of computation steps, although the number of participating dimension tables increases. By adopting bloom filters, Scatter-Gather-Merge reduces a nontrivial amount of IO. We also show that Scatter-Gather-Merge can be easily applied to MapReduce. Our experimental results in both cluster and cloud environments show that Scatter-Gather-Merge outperforms existing approaches.

Monitoring business activities using Business Intel- ligence (BI) tools is a well-established concept. How- ever, online process monitoring is an emerging area which helps organizations not only plan for future im- provements but also change and alter their current ongoing processes before problems happen. In this paper, we explore how monitoring process perfor- mance can help evolve process goals and require- ments. We elaborate an approach that uses the User Requirements Notation (URN) to model the goals and processes of the organization, and to monitor and align processes against their goals. A BI tool exploit- ing an underlying data warehouse provides the Key Performance Indicators (KPI) used to measure the satisfaction of goals and process requirements. Feed- ing this information into the URN modeling tool, we can analyze the consequences of current business ac- tivities on desired business goals, which can be used for process and business activity alignment thereafter. We illu...

Documenting business rules is a complex task in tenns of rule extraction, reuse and integration into current business documentation, where most of business rules (and laws) are buried in data warehouses as raw data. In this paper, we propose a method to simplify the business rules documentation processes using XML technologies. These generated documents can be used as a specification requirement for business development. Our proposed method will enable and help businesses to extract, document and reuse business rules from any data warehouses or rules repositories to generate the business rules documents. We also incorporate filtration and validation methods into the rules formation process to generate business rules documents.

A data warehouse (DW) provides an information for analytical processing, decision making, and data mining tools. On the one hand, the structure and content of a data warehouse reflects a real world, i.e. data stored in a DW come from real production systems. On the other hand, a DW and its tools may be used for predicting trends and simulating a virtual business scenarios. This activity is often called the what-if analysis. Traditional DW systems have static structure of their schemas and relationships between data, and therefore they are not able to support any dynamics in their structure and content. For these purposes, multiversion data warehouses seem to be very promising. In this paper we present a concept and an ongoing implementation of a multiversion data warehouse that is capable of handling changes in the structure of its schema as well as simulating alternative business scenarios.

Extraction-Transformation-Loading (ETL) tools are pieces of software responsible for the extraction of data from several sources, their cleansing, customization, and insertion into a data warehouse. In this paper, we delve into the logical optimization of ETL processes, modeling it as a state-space search problem. We consider each ETL workflow as a state and fabricate the state space through a set of correct state transitions. Moreover, we provide an exhaustive and two heuristic algorithms toward the minimization of the execution cost of an ETL workflow. The heuristic algorithm with greedy characteristics significantly outperforms the other two algorithms for a large set of experimental cases.

Data reconciliation is the process of matching records across dierent databases. Data reconciliation requires``joining'' on ®elds that have traditionally been non-key ®elds. Generally, the operational databases are of sucient quality for the purposes for which they were initially designed but since the data in the dierent databases do not have a canonical structure and may have errors, approximate matching algorithms are required.

In Data Warehouse (DW) scenarios, ETL (Extraction, Transformation, Loading) processes are responsible for the extraction of data from heterogeneous operational data sources, their transformation (conversion, cleaning, normalization, etc.) and their loading into the DW. In this paper, we present a framework for the design of the DW back-stage (and the respective ETL processes) based on the key observation that this task fundamentally involves dealing with the specificities of information at very low levels of granularity including transformation rules at the attribute level. Specifically, we present a disciplined framework for the modeling of the relationships between sources and targets in different levels of granularity (including coarse mappings at the database and table levels to detailed inter-attribute mappings at the attribute level). In order to accomplish this goal, we extend UML (Unified Modeling Language) to model attributes as first-class citizens. In our attempt to provide complementary views of the design artifacts in different levels of detail, our framework is based on a principled approach in the usage of UML packages, to allow zooming in and out the design of a scenario.

Forged documents specifically passport, driving licence and VISA stickers are used for fraud purposes including robbery, theft and many more. So detecting forged characters from documents is a significantly important and challenging task in digital forensic imaging. Forged characters detection has two big challenges. First challenge is, data for forged characters detection is extremely difficult to get due to several reasons including limited access of data, unlabeled data or work is done on private data. Second challenge is, deep learning (DL) algorithms require labeled data, which poses a further challenge as getting labeled is tedious, time-consuming, expensive and requires domain expertise. To end these issues, in this paper we propose a novel algorithm, which generates the three datasets namely forged characters detection for passport (FCD-P), forged characters detection for driving licence (FCD-D) and forged characters detection for VISA stickers (FCD-V). To the best of our kn...

We present an efficiency model for data warehouse operations and analyze a data set to evaluate the model. The model contains salient variables to evaluate the efficiency of an organization's data warehouse operations for refresh processing and query production. The variables in the model include resource consumption (labor usage and computing budgets), system usage measures (users and queries), quality measures (data age and availability), and a size measure (change data amount). We report on the evaluation of a data set collected from USA and non USA-based (mostly Korean) organizations using input-oriented Data Envelopment Analysis. The analysis indicates wide dispersions in reported data, large differences in labor budgets between efficient and inefficient firms, few organizations efficient in both refresh processing and query production, and the difficulty of providing some data (computing budgets and flexibility ratio). Follow-up interviews with selected organizations provided insights about the value of efficiency comparisons of information technology organizations and suggestions to improve the model.

Data warehousing has evolved with every passing decade and it has come a long way from its inception and the modern era has made it an adequate part of pre-existing analytical methodologies. In the present status, data warehousing has evolved into a system which is capable of furnishing key performance metrics to high-level management, ensuring capability of analytical strength to middle-level management and aligning to the ability of providing corrective data to-and-fro back to low-level based on the basis of information derived from the analytical system.

We show that the e-commerce domain can provide all the right ingredients for successful data mining and claim that it is a killer domain for data mining. We describe an integrated architecture, based on our experience at Blue Martini Software, for supporting this integration. The architecture can dramatically reduce the pre-processing, cleaning, and data understanding effort often documented to take 80% of the time in knowledge discovery projects. We emphasize the need for data collection at the application server layer (not the web server) in order to support logging of data and metadata that is essential to the discovery process. We describe the data transformation bridges required from the transaction processing systems and customer event streams (e.g., clickstreams) to the data warehouse. We detail the mining workbench, which needs to provide multiple views of the data through reporting, data mining algorithms, visualization, and OLAP. We conclude with a set of challenges.

Data warehouses are the core of decision support systems, which nowadays are used by all kind of enterprises in the entire world. Although many studies have been conducted on the need of decision support systems (DSSs) for small businesses, most of them adopt existing solutions and approaches, which are appropriate for large-scaled enterprises, but are inadequate for small and middle-sized enterprises.

In modern manufacturing environments, vast amounts of data are collected in database management systems and data warehouses from all involved areas, such as product and process design, assembly, materials planning, quality control, scheduling, maintenance, fault detection and so on. Data mining has emerged as an important tool for knowledge acquisition in manufacturing databases. This paper reviews the literature dealing with knowledge discovery and data mining applications in the broad domain of manufacturing with an special emphasis on the type of functions to be performed on data. The major data mining functions to be performed include characterization and description, association, classification, prediction, clustering and evolution analysis. The papers reviewed have therefore been categorized in these five categories. It has been shown that there is a rapid growth in the application of data mining in the context of manufacturing processes and enterprises in the last 3 years. This review reveals the progressive applications and existing gaps identified in the context of data mining in manufacturing. A novel text mining approach has also been applied to the abstracts and keywords of 150 identified literatures to identify the research gaps and find the linkages between knowledge area, knowledge type and data mining tools and techniques applied.

The clinical data repository (CDR) is a frequently updated relational data warehouse that provides users with direct access to detailed, flexible, and rapid retrospective views of clinical, administrative, and financial patient data for the University of Virginia Health System. This article presents a case study of the CDR, detailing its five-year history and focusing on the unique role of data warehousing in an academic medical center. Specifically, the CDR must support multiple missions, including research and education, in addition to administration and management. Users include not only analysts and administrators but clinicians, researchers, and students. KEYWORDS • Data warehousing • Database • Academic medical center • Clinical research • Case study • Evaluation Large organizations build data warehouses to "analyze what has occurred within the business across time" in order to obtain "a competitive edge in the marketplace." 1 Many healthcare organizations see data warehousing as a way to facilitate operational efficiency and informed administrative decision making. In the 2000 HIMSS Leadership Survey of more than eleven hundred healthcare professionals, 58 percent of respondents indicated that their organizations were currently creating data warehouses or planned to do so within the next two years. 2 However, the missions of academic medical centers extend beyond administrative analysis and decision making. The mission statement for the University of Virginia Health System, for example, refers to 165

This paper describes our work in using mathematical optimization to assist air cargo load planning for an air cargo company, one of the world's top 10 air cargo carriers. The airline handles roughly 80 to 90 thousand tones of cargo each month. With this volume of cargo, even the slightest improvement in the load planning process will have a major impact to overall performance and efficiency. In this project, we developed a Web-based application to firstly perform long-term forecasting based on an analysis of historical data and then secondly operational load planning with mathematical optimization. Forecasting is important as the data will assist the airline in drafting its long-term contracts and determination of allotments. Load planning, on the other hand, is for operational term planning of how ULDs (Unit Load Devices - air cargo containers/pallets) are loaded into aircrafts. This process is usually done roughly 2 hours before departure when all the details of the cargo are ...

Abstract. The high cost of data consolidation is the key market inhibitor to the adoption of traditional information integration and data warehousing solutions. In this paper, we outline a next-generation integrated database management system that takes traditional information integration, content management, and data warehouse techniques to the next level: the system will be able to integrate a very large number of information sources and automatically construct a global business view in terms of “Universal Business Objects”. We describe techniques for discovering, unifying, and aggregating data from a large number of disparate data sources. Enabling technologies for our solution are XML, web services, caching, messaging, and portals for real-time dashboarding and reporting. 1

Extraction-Transformation-Loading (ETL) tools are pieces of software responsible for the extraction of data from several sources, their cleansing, customization and insertion into a data warehouse. In this paper, we propose a methodology for the earliest stages of the data warehouse design, with the goal of tracing the analysis of the structure and content of the existing data sources and their intentional mapping to the common conceptual data warehouse model. The methodology comprises a set of steps that can be summarized as follows: (a) identification of the proper data stores; (b) candidates and active candidates for the involved data stores; (c) attribute mapping between the providers and the consumers, and (d) annotation of the diagram with runtime constraints.

In the last few days, data and the internet have become increasingly growing, occurring in big data. For these problems, there are many software frameworks used to increase the performance of the distributed system. This software is used for available ample data storage. One of the most beneficial software frameworks used to utilize data in distributed systems is Hadoop. This software creates machine clustering and formatting the work between them. Hadoop consists of two major components: Hadoop Distributed File System (HDFS) and Map Reduce (MR). By Hadoop, we can process, count, and distribute each word in a large file and know the number of affecting for each of them. The HDFS is designed to effectively store and transmit colossal data sets to high-bandwidth user applications. The differences between this and other file systems provided are relevant. HDFS is intended for low-cost hardware and is exceptionally tolerant to defects. Thousands of computers in a vast cluster both have directly associated storage functions and user programmers. The resource scales with demand while being cost-effective in all sizes by distributing storage and calculation through numerous servers. Depending on the above characteristics of the HDFS, many researchers worked in this field trying to enhance the performance and efficiency of the addressed file system to be one of the most active cloud systems. This paper offers an adequate study to review the essential investigations as a trend beneficial for researchers wishing to operate