Data Grid

In this paper medical applications on a Grid infrastructure, the MAGIC-5 Project, are presented and discussed. MAGIC-5 aims at developing Computer Aided Detection (CADe) software for the analysis of medical images on distributed databases by means of GRID Services. The use of automated systems for analyzing medical images improves radiologists' performance; in addition, it could be of paramount importance in screening programs, due to the huge amount of data to check and the cost of related manpower. The need for acquiring and analyzing data stored in different locations requires the use of Grid Services for the management of distributed computing resources and data. Grid technologies allow remote image analysis and interactive online diagnosis, with a relevant reduction of the delays presently associated with the diagnosis in the screening programs. The MAGIC-5 project develops algorithms for the analysis of mammographies for breast cancer detection, Computed-Tomography (CT) images for lung cancer detection and Positron Emission Tomography (PET) images for the early diagnosis of Alzheimer Disease (AD). A Virtual Organization (VO) has been deployed, so that authorized users can share data and resources and implement the following use cases: screening, tele-training and tele-diagnosis for mammograms and lung CT scans, statistical diagnosis by comparison of candidates to a distributed data-set of negative PET scans for the diagnosis of the AD. A small-scale prototype of the required Grid functionality was already implemented for the analysis of digitized mammograms.

A high throughput Basic Local Alignment Search Tool (BLAST) system based on Web services is implemented. It provides an alternative BLAST service and allows users to perform multiple BLAST queries at one run in a distributed, parallel environment through the Internet.

The Network Common Data Form (netCDF) is one of the primary methods of self-documenting community and recent evolution toward tion via messages in the defacto standard XML language. XML is a text-based language while netCDF is based on a binary file storage mechanism; thus NcML is a natural augmentation of netCDF with bjects in an XML tation of internal concept. In fact, the netCDF Markup Language (NcML) and NcML-G (NcML-Geography) extensions described in this article have applications beyond merely representing the internal netCDF documentation in the XML language. The NcML coordinate system makes it possible to describe the coordinate system used to represent the netCDF dataset. Furthermore the NcML dataset is a tool for nd implementation o Language (NcML) and

Data replication is the creation and maintenance of multiple copies of the same data. Replication is used in Data Grid to enhance data availability and fault tolerance. One of the main objectives of replication strategies is reducing response time and bandwidth consumption. In this paper, a dynamic replication strategy that is based on Fast Spread but superior to it in terms of total response time and total bandwidth consumption is proposed. This is achieved by storing only the important replicas on the storage of the node. The main idea of this strategy is using a threshold to determine if the requested replica needs to be copied to the node. The simulation results show that the proposed strategy achieved better performance compared with Fast Spread with Least Recently Used (LRU), and Fast Spread with Least Frequently Used (LFU).

Abstract. The Social Informatics Data Grid is a new infrastructure designed to transform how social and behavioral scientists collect and annotate data, collaborate and share data, and analyze and mine large data repositories. An important goal of the project is to be compatible with existing databases and tools that support the sharing, storage and retrieval of archival data sets. It is built on web and grid services to enable transparent access to data and analysis resources from anywhere and to leverage new and emerging web-based ...

Large distributed systems such as Computational/Data Grids require large amounts of data to be colocated with the computing facilities for processing. Ensuring that the data is there in time for the computation in today's Internet is a massive problem. From our work developing a scalable distributed network cache, we have gained experience with techniques necessary to achieve high data throughput over high bandwidth Wide Area Networks (WAN). In this paper, we discuss several hardware and software design techniques and issues, and then describe their application to an implementation of an enhanced FTP protocol called GridFTP. We also describe results from two applications using these techniques, which were obtained at the Supercomputing 2000 conference.

Selecting optimal resources for submitting jobs on a computational Grid or accessing data from a data grid is one of the most important tasks of any Grid middleware. Most modern Grid software today satisfies this responsibility and gives a best-effort performance to solve this problem. Almost all decisions regarding scheduling and data access are made by the software automatically, giving users little or no control over the entire process. To solve this problem, a more interactive set of services and middleware is desired that provides users more information about Grid weather, and gives them more control over the decision making process. This paper presents a set of services that have been developed to provide more interactive resource management capabilities within the Grid Analysis Environment (GAE) being developed collaboratively by Caltech, NUST and several other institutes. These include a steering service, a job monitoring service and an estimator service that have been designed and written using a common Grid-enabled Web Services framework named Clarens. The paper also presents a performance analysis of the developed services to show that they have indeed resulted in a more interactive and powerful system for user-centric Gridenabled physics analysis.

Selecting optimal resources for submitting jobs on a computational Grid or accessing data from a data grid is one of the most important tasks of any Grid middleware. Most modern Grid software today satisfies this responsibility and gives a best-effort performance to solve this problem. Almost all decisions regarding scheduling and data access are made by the software automatically, giving users little or no control over the entire process. To solve this problem, a more interactive set of services and middleware is desired that provides users more information about Grid weather, and gives them more control over the decision making process. This paper presents a set of services that have been developed to provide more interactive resource management capabilities within the Grid Analysis Environment (GAE) being developed collaboratively by Caltech, NUST and several other institutes. These include a steering service, a job monitoring service and an estimator service that have been designed and written using a common Grid-enabled Web Services framework named Clarens. The paper also presents a performance analysis of the developed services to show that they have indeed resulted in a more interactive and powerful system for user-centric Gridenabled physics analysis.

Grid computing has recently gained in popularity. Grid applications can be very demanding of the data storage facilities in the Grid. The existing data grid services are often insufficient and additional optimization of the data access is necessary. This paper presents research concerning the optimization aspects of data management for Hierarchical Storage Management (HSM) systems in the Grid.

The Sloan Digitial Sky Surveys (SDSS) have been collecting imaging and spectoscopic data since 1998. These data as well as their derived data products are made publicly available through regular data releases, of which the 13th took place summer 2016. Although public data releases can be challenging to manage, they signficantly increase the impact of a survey, both scientifically and educationally.

In data grids, data replication on variant nodes can change some problems such as response time and availability. Also, in data replication, there are some challenges to finding the best replica efficiently in relation to performance and location of physical storage systems. In this paper, various replica placement strategies are discussed. These replica placement strategies are available in the works. Replica placement contains recognizing the best possible node to duplicate data based on network latency and user request. These strategies measure and analyze different parameters such as access cost, bandwidth consumption, scalability, execution time and storage consumption. This paper also analyses the performance of various strategies with respect to the parameters mentioned above in data grid.

We propose a novel approach for joint denoising and interpolation of noisy Bayer-patterned data acquired from a digital imaging sensor (e.g., CMOS, CCD). The aim is to obtain a full-resolution RGB noiseless image. The proposed technique is speciÞcally targeted to Þlter signal-dependant, e.g. Poissonian, or heteroscedastic noise, and effectively exploits the correlation between the different color channels. The joint technique for denoising and interpolation is based on the concept of local polynomial approximation (LPA) and intersection of conÞdence intervals (ICI). These directional Þlters utilize simultaneously the green, red, and blue color channels. This is achieved by a linear combination of complementary-supported smoothing and derivative kernels designed for the Bayer data grid. With these Þlters, the denoised and the interpolated estimates are obtained by convolutions over the Bayer data. The ICI rule is used for data-adaptive selection of the length of the designed cross-color directional Þlter. Fusing estimates from multiple directions provides the Þnal anisotropic denoised and interpolated values. The full-size RGB image is obtained by placing these values into the corresponding positions in the image grid. The efficiency of the proposed approach is demonstrated by experimental results with simulated and real camera data.

Abstract: Recently, there have been many efforts to develop middlewares supporting applications in managing distributed data in Grid computing environment. Although these research activities address various requirements in Grid data management, none of them provides a general ...

In different scientific disciplines, large-scale data are generated with enormous storage requirements. Therefore, effective data management is a critical issue in distributed systems such as the cloud. As tasks can access a nearby site to access the required file, replicating the desired file to an appropriate location improves access time and reliability. Replicating the popular file to an appropriate site is a good choice, as tasks can get the necessary file from a nearby site. In this research, a novel data replication algorithm is proposed that is consisted of four main phases: 1-determining 20% of commonly used files, 2-computing five conflicting objectives (i.e., average service time, load variance, energy consumption, average response time and cost) 3-finding the near-optimal solution (i.e., suitable locations for new replica) by the PSO technique to acquire a trade-off among the desired objectives. 4-replica replacement considering a fuzzy system with three inputs (i.e., Number of accesses, size of replica and the last access time). The experimental results denote that the proposed replication algorithm outperforms the Profit oriented Data Replication (PDR) and Bee colony-based approach for Data Replication (BCDR) strategies in terms of energy consumption, average response time, load variance, number of connections, Hit ratio, Storage usage, and cost.

GriPhyN (Grid Physics Network) is a large US collaboration to build grid services for large physics experiments, one of which is LIGO, a gravitational-wave observatory. This paper explains the physics and computing challenges of LIGO, and the tools that GriPhyN will build to address them. A key component needed to implement the data pipeline is a virtual data service; a system to dynamically create data products requested during the various stages. The data could possibly be already processed in a certain way, it may be in a file on a storage system, it may be cached, or it may need to be created through computation. The full elaboration of this system will allow complex data pipelines to be set up as virtual data objects, with existing data being transformed in diverse ways. 1 Physics LIGO (Laser Interferometer Gravitational-Wave Observatory) [1] is a joint Caltech/ MIT project to directly detect the gravitational waves predicted by Einstein's theory of relativity: oscillations in the fabric of space-time. Currently, there are two observatories in the United States, (Washington state and Louisiana), one of which recently went through the "first lock" phase, in which initial calibration data was collected. Theoretically, gravitational waves are generated by accelerating masses, however they are so weak that so far they have not been directly detected (although their indirect influence

The success of grid computing depends on the existence of grid middleware that provides core services such as security, data management, resource information, and resource brokering and scheduling. Current general-purpose grid resource brokers deal only with computation requirements of applications, which is a limitation for data grids that enable processing of large scientific data sets. In this paper, a new dataaware resource brokering scheme, which factors both computational and data transfer requirements into its cost models, has been implemented and tested. The experiments reported in this paper clearly demonstrate that both factors should be considered in order to efficiently schedule data intensive tasks.

The Grid provides mechanisms to share dynamic, heterogeneous, distributed resources spanned across multiple administrative domains. Resources required to execute a job are identified from the resource pool based on the desired set of attributes of the resources that exhibit various degrees of dynamism. A Resource Discovery service provides mechanisms to identify the set of resources that are capable of satisfying the job requirements. Current grid systems (Condor, European Data Grid) employ centralized matchmakers for resource discovery. Centralized matchmakers become a performance bottle neck as the grid grows. To alleviate this problem and to effectively exploit the grid, a distributed matchmaking approach is desirable not only for performance reasons, but also to surmount single point failures for grid resource discovery. In this paper, a novel distributed matchmaking approach for resource discovery in large, dynamically varying collections of grid resources is proposed.

Current trend of network-based multimedia storage, distributed scientific simulations and distributed geographic information system applications are require both compute intensive and data intensive. This is unavoidable since grid and cloud computing technologies are more promising and demanding because of its scalability and performance. The greatest challenge to develop infrastructure services for grid and cloud computing is heterogenity. This heterogenity includes machine architecture, operating system and network resources. Most common solution to increase processing power and storage is by adding more node and storage units to the cluster and also adding more network interface and connection to increase the total communication bandwidth among the cluster. In this paper, we reviewed available solutions on concurrent data transfer in Grid Computing in order to support data intensive application in grid computing by utilizing available multiple network interfaces. We organized our review based on solutions based on three different layers: Data-Link, Transport and Middleware.

The SHAMAN project targets a framework integrating advances in the data grid, digital library, and persistent archives communities in order to archive a longterm preservation environment. Within the project we identified several challenges for digital preservation in the area of memory institutions, where already existing systems start to struggle with e.g. complex or many small objects. In order to overcome these, we propose a grid based framework for digital preservation. In this paper we describe the main objectives of the project SHAMAN and the identified challenges for a heterogeneous and distributed environment. We on the one hand assess in a bottom-up approach the capabilities and interfaces of legacy systems and on the other hand derive requirements based on project objectives. The focus points to the integration of storage infrastructures and distributed data management. In the end we derive a service-oriented architecture with an grid-based integration layer as approach to manage the challenges.

Data warehouse view maintenance is an important issue due to the growing use of warehouse technology for information integration and data analysis. Given the dynamic nature of modern distributed environments, both data updates and schema changes are likely to occur in different data sources. In applications for which the real-time refresh of the data warehouse extent under source change is not critical, the source updates are usually maintained in a batch fashion to reduce the maintenance overhead. However, most prior work can only deal with the batching source data updates. In this paper, we now provide a solution strategy that is capable of batching both source data updates and schema changes. We propose techniques to first preprocess the initial source updates to construct summarized delta changes for each source. We then design a view adaptation algorithm to adapt the warehouse view under these delta changes. We have implemented our proposed batching solution and incorporated it into an existing data warehouse prototype system. The experimental studies demonstrate the excellent performance achievable by our batch technique.

Some of the most recently proposed algorithms for the incremental maintenance of materialized data warehouses (DW), such as SWEEP and PSWEEP, offer several significant advantages over previous solutions, such as high-performance, no potential for infinite waits and reduced remote queries and thus reduced network and information source (IS) loads. However, similar to many other algorithms, they still have the restricting assumption that each IS can be composed of just one single relation. This is unrealistic in practice. In this paper, we hence propose a solution to overcome this restriction. The Multi-Relation Encapsulation (MRE) Wrapper supports multiple relations in ISs in a manner transparent to the rest of the environment. The MRE Wrapper treats one IS composed of multiple relations as if it were a single relation from the DW point of view; thus any existing incremental view maintenance algorithms can now be applied even to such complex ISs without any changes. Hence, our method maintains all advantages offered by existing algorithms in particular SWEEP and PSWEEP, while also achieving the additional desired features of being nonintrusive, efficient, flexible and well-behaved.

We describe a Grid market for exchanging data mining services based on the Catallactic market mechanism proposed by von Hayek. This market mechanism allows selection between multiple instances of services based on operations required in a data mining task (such as data migration, data pre-processing and subsequently data analysis). Catallaxy is a decentralized approach, based on a "free market" mechanism, and particularly useful when the number of market participants is large, or conditions within the market change often. It is therefore particularly suitable in Grid and Peer-2-Peer systems. The approach assumes that the service provider and user are not co-located, and require multiple message exchanges to carry out a data mining task. A market of J48-based decision tree algorithm instances, each implemented as a Web Service, is used to demonstrate our approach. We have validated the feasibility of building catallactic data mining grid applications, and implemented a proof-of-concept application (Cat-COVITE) mapped to a Catallactic Grid Middleware.

Nowadays, the process of data mining is one of the most important topics in scientific and business problems. There is a huge amount of data that can help to solve many of these problems. However, data is geographically distributed in various locations and belongs to several organizations. Furthermore, it is stored in different kind of systems and it is represented in many formats. In this paper, different techniques have been studied to make easier the data mining process in a distributed environment. Our approach proposes the use of grid to improve the data mining process due to the features of this kind of systems. In addition, we show a flexible architecture that allows data mining applications to be dynamically configured according to their needs. This architecture is made up of generic, data grid and specific data mining grid services.

The Italica Project is the implementation of an Electronic Health Record system at the Italian Hospital of Buenos Aires. The present work shows the implementation of a Medical Signal Grid Repository module and its integration to the Italica Project. We also describe the architecture of that module and explain the design decisions taken for its construction. The integration to the preexisting system is also presented, together with the difficulties that arose from the insertion of a data grid in an environment not used to this technology. Several issues about the security and reliability of the medical data are treated as well. In this paper we also discuss the use of the grid repository as a starting point for the development of distributed processing applications.

Data integration over multiple heterogeneous data sources has become increasingly important for modern applications. The integrated data is usually stored in materialized views for better access, performance and high availability. Such views must be maintained after the data sources change. In a loosely-coupled environment, such as the Data Grid, the source updates are autonomous and may cause erroneous maintenance results. State-of-the-art maintenance strategies apply compensating queries to correct such errors. However, they assume that the source schema remain static. This is an unrealistic assumption for such dynamic environments, where the data sources may change not only their data but also their schema, query capabilities or semantics. Consequently, either the maintenance or compensating queries may fail. In this paper, first, we analyze the maintenance errors and classify them into different classes of dependencies. We then propose Dyno, a two-pronged strategy composed of dependency detection and correction algorithms to handle these new classes of concurrency. Our techniques are not tied to specific maintenance algorithms nor to a particular data model. To our knowledge, this is the first complete solution to the view maintenance concurrency problems for both data and schema changes. Our experimental results illustrate that Dyno imposes an almost negligible overhead on existing maintenance algorithms for data updates while now allowing for this extended functionality.

Data integration over multiple heterogeneous data sources has become increasingly important for modern applications. The integrated data is usually stored in materialized views (MV) to allow better access, performance and high availability. MV must be maintained after the data sources change. In a loosely-coupled environment, such as the Data Grid, the data sources are autonomous. Hence the source updates can be concurrent and cause erroneous maintenance results. State-of-the-art maintenance strategies apply compensating queries to correct such errors, making the restricting assumption that all source schemata remain static over time. However, in such dynamic environments, the data sources may change not only their data but also their schema, query capabilities or semantics. Consequently, either the maintenance queries or compensating queries would fail. In this paper, we propose a framework called DyDa that overcomes these limitations and handles both source data updates and schema changes. First, we identify three types of maintenance anomalies, caused by either data updates and/or rename and/or drop schema changes. We propose a compensation algorithm to solve the first two types of anomalies. We identify that the third type of anomaly is caused by the violation of dependencies between the maintenance processes. We propose a detection and correction algorithm to remove such anomalies based on the formalisms of dependencies. A new view adaptation algorithm is designed to incrementally adapt some complex updates introduced by the correction algorithm. Put together, these algorithms are the first complete solution to the concurrency problems for MV maintenance in loosely-coupled environments. We have implemented the DyDa system. The experimental results show that our new concurrency handling strategy imposes a minimal overhead on normal data update processing while allowing for the extended functionality to maintain the materialized views even under concurrent schema changes.

Data integration over multiple heterogeneous data sources has become increasingly important for modern applications. The integrated data is usually stored as materialized views to allow better access, performance, and high availability. In loosely coupled environments, such as the Data Grid, the data sources are autonomous. Hence, the source updates can be concurrent and cause erroneous results during view maintenance. State-of-the-art maintenance strategies apply compensating queries to correct such errors, making the restricting assumption that all source schemata remain static over time. However, in such dynamic environments, the data sources may change not only their data but also their schema. Consequently, either the maintenance queries or the compensating queries may fail. In this paper, we propose a novel framework called DyDa that overcomes these limitations and handles both source data updates and schema changes. We identify three types of maintenance anomalies, caused by either source data updates, data-preserving schema changes, or non-data-preserving schema changes. We propose a compensation algorithm to solve the first two types of anomalies. We show that the third type of anomaly is caused by the violation of dependencies between maintenance processes. Then, we propose dependency detection and correction algorithms to identify and resolve the violations. Put together, DyDa extends prior maintenance solutions to solve all types of view maintenance anomalies. The experimental results show that DyDa imposes a minimal overhead on data update processing while allowing for the extended functionality to handle concurrent schema changes.

The Grid-based Virtual Laboratory AMsterdam (VLAM-G) provides a science portal for distributed analysis in applied scientific research. By facilitating access to distributed compute and information resources held by multiple organizations, and by providing remote experiment control, data management and information retrieval capabilities, it allows scientists to better analyse their data. The ability to use data from multiple sources and correlating these datasets without in-depth domain expertise is a prime goal of the system. This paper describes the design and an implementation prototype of the VLAM-G platform. The feasibility of the system is demonstrated by a generalized sample scenario from the chemo-physical analysis domain.

Data Grid environment is a geographically distributed that deal with date-intensive application in scientific and enterprise computing. Dealing with large amount of data makes the requirement for efficiency in data access more critical. The goal of replication is to shorten the data access not only for user accesses but enhancing the job execution performance. In this paper, we proposed a new approach to replication based on organizing the data in Data Grid based on its property. In this paper, we organized the data in to several data categories that it belongs to. And this information is used to help improving data replication placement strategy. We study our approach and evaluate it through simulation. The result shows that our algorithm has improved 30% over the current strategies.

Recently, Grid computing activities in Asia-Pacific have been drawn attention, includes in high learning education institutes in Malaysia. Many university and institute in Malaysia are started to build their campus grid infrastructure. One of the main problems that most institution faced when deploying campus grid is interconnectivity between cluster/grid sites. Thanks to EGEE gLite middleware project which break this barrier. The gLite distribution is an integrated set of components designed to enable resource sharing. The distribution model is to construct different services ('node-types') from these components and then ensure easy installation and configuration on the chosen platforms (currently Scientific Linux versions 4 and 5). It gives flexibity to the grid administrator on how they want to customize their cluster. We present a proposal of deploying a scalable campus grid infrastructure configuration based on gLite middleware concept, focused on typical Malaysian campus grid deployment and its challanges. The main goal of this proposal is to fostering the adoption of E-Science Application widely among Malaysian institute/university.

Summary: Timely worldwide distribution of biosequence and bioinformatics data depends on high performance networking and advances in Internet transport methods. The Bio-Mirror project focuses on providing up-to-date distribution of this rapidly growing and changing data. It offers FTP, Web and Rsync access to many high-volume databanks from several sites around the world. Experiments with data grids and other methods offer future improvements in biology data distribution. Availability:  http://www.bio-mirror.net/

Data Grids deal with geographically-distributed large-scale data-intensive applications. Schemes scheduled for data grids attempt to not only improve data access time, but also aim to improve the ratio of data availability to a node, where the data requests are generated. Data replication techniques manage large data by storing a number of data files efficiently. In this paper, we propose centralized dynamic scheduling strategy-replica placement strategies (CDSS-RPS). CDSS-RPS schedule the data and task so that it minimizes the implementation cost and data transfer time. CDSS-RPS consists of two algorithms, namely (a) centralized dynamic scheduling (CDS) and (b) replica placement strategy (RPS). CDS considers the computing capacity of a node and finds an appropriate location for the job. RPS attempts to improve file access time by using replication on the basis of number of accesses, storage capacity of a computing node, and response time of a requested file. Extensive simulations are carried out to demonstrate the effectiveness of the proposed strategy. Simulation results demonstrate that the replication and scheduling strategies improve the implementation cost and average access time significantly.

An SDDS-2000 server currently manages only buckets in its RAM storage [C01]. Several buckets can coexist. When many files are created however, RAM storage space may not be sufficient for all the buckets simultaneously. When an application requests a bucket, but there is not enough room in RAM for it, one can evict some buckets to the disk. We define a number of promising eviction strategies. Our goal is to maximize the throughput of the server. 1.The Model We call cache the RAM storage available for the buckets at an SDDS Server. The cache of an SDDS server will contain buckets of files created by LH* and RP* insertions and splits. We envision that during a particular time period we will need to keep at a server only a small number of rather large buckets, // how large are they ?// say of the order of 20-30 buckets. The pattern of accesses will most likely consist of some steady accesses to a number of favorite files. In addition, we envision that some files will exhibit a burst of accesses during short periods, while for some other files we may have to deal with a temporal pattern, where the files go through peaks and valleys of accesses periodically. We would also like to consider the case where for some files, for example payroll files, we can anticipate this cyclical pattern. A bucket consists of a number of pages. Observe that when a page is not in the cache, the system needs to bring to memory the entire bucket (pointer swizzling does not work here). However, when a bucket is written to disk we only need to write the pages that have been modified since the last time the bucket was written to disk.

The CMS experiment at CERN is setting up a Grid infrastructure required to fulfil the needs imposed by Terabyte scale productions for the next few years. The goal is to automate the production and at the same time allow the users to interact with the system, if required, to make decisions which would optimise performance. We present the architecture, design