Outlier detection

The paper presents model based on fuzzy methods for churn prediction in retail banking. The study was done on the real, anonymised data of 5000 clients of a retail bank. Real data are great strength of the study, as a lot of studies often use old, irrelevant or artificial data. Canonical discriminant analysis was applied to reveal variables that provide maximal separation between clusters of churners and non-churners. Combination of standard deviation, canonical discriminant analysis and k-means clustering results were used for outliers detection. Due to the fuzzy nature of practical customer relationship management problems it was expected, and shown, that fuzzy methods performed better than the classical ones. According to the results of the preliminary data exploration and fuzzy clustering with different values of the input parameters for fuzzy cmeans algorithm, the best parameter combination was chosen and applied to training data set. Four different prediction models, called prediction engines, have been developed. The definitions of clients in the fuzzy transitional conditions and the distance of k instances fuzzy sums were introduced. The prediction engine using these sums performed best in churn prediction, applied to both balanced and non-balanced test sets.

RESUMO O presente artigo pretende apresentar os principais resultados de um estudo empírico de modelação da série cronológica da taxa de juro nominal da operação activa do crédito a particulares em Portugal realizado por Caiado (1997). Com base na metodologia dos modelos ARIMA com variáveis de intervenção e detecção de outliers , vai avaliar-se o impacto de alterações de relevo

All known robust location and scale estimators with high breakdown point for multivariate sample's are very expensive to compute. In practice, this computation has to be carried out using an approximate subsampling procedure. In this work we describe an alternative subsampling scheme, applicable to both the Stahel-Donoho estimator and the estimator based on the Minimum Volume Ellipsoid, with the property that the number of subsamples required is substantially reduced with respect to the standard subsampling procedures used in both cases. We also discuss some bias and variability properties of the estimator obtained from the proposed subsampling process.

Spatial data mining is the process of discovering interesting and previously unknown, but potentially useful, patterns from large spatial datasets. Extracting interesting and useful patterns from spatial datasets is more difficult than extracting the corresponding patterns from traditional numeric and categorical data due to the complexity of spatial data types, spatial relationships, and spatial autocorrelation. This chapter will discuss some of accomplishments and research needs of spatial data mining in the following categories: location prediction, spatial outlier detection, co-location mining, and clustering.

The analysis of UCR data provides a basis for crime prevention in the United States as well as a sound decision making tool for policy makers. The decisions made with the use of UCR data range from major funding for resource allocation down to patrol distribution by local police departments. The FBI collects and maintains the database of the Uniform Crime Reports (UCR), from 18,000 reporting police agencies nationwide. However, many of these data sets have missing, incomplete, or incorrect data points that render crime analysis less effective. UCR experts have stated that in the current form UCR data is unreliable and sporadic. Efforts have previously been made to design a software application to correct these necessary problems, but the application was deemed insufficient due to limited portability and usability. Software requirements restricted potential users and the user interface was ineffective. However, this previous work describes the functions needed to effectively clean and assess UCR data. This paper describes the design of an application used to clean, process, and correct UCR data so that ideal policy decisions can be made. Erroneous portions of the data will be found using the outlier detection function that is based on a statistical model of anomalous behavior. These methods incorporate sponsor specifications and user requirements. This project builds upon the GRASP (Geospatial Repository for Analysis and Safety Planning) project's goal of sharing information between law enforcement agencies. Eventually

A distance-based outlier detection method that finds the top outliers in an unlabeled data set and provides a subset of it, called outlier detection solving set, that can be used to predict the outlierness of new unseen objects, is proposed. The solving set includes a sufficient number of points that permits the detection of the top outliers by considering only a subset of all the pairwise distances from the data set. The properties of the solving set are investigated, and algorithms for computing it, with subquadratic time requirements, are proposed. Experiments on synthetic and real data sets to evaluate the effectiveness of the approach are presented. A scaling analysis of the solving set size is performed, and the false positive rate, that is, the fraction of new objects misclassified as outliers using the solving set instead of the overall data set, is shown to be negligible. Finally, to investigate the accuracy in separating outliers from inliers, ROC analysis of the method is accomplished. Results obtained show that using the solving set instead of the data set guarantees a comparable quality of the prediction, but at a lower computational cost.

Clustering is an extremely important task in a wide variety of application domains especially in management and social science research. In this paper, an iterative procedure of two-way clustering method based on multivariate outlier detection was proposed using the famous Squared Mahalanobis distance or T 2-distance and step-wise multiple discriminant analysis. At first, Squared Mahalanobis distance or T 2 –distance should be calculated for the entire sample, then using T 2 – statistic fix a UCL. Above the UCL are treated as outliers which are grouped as outlier cluster and the remaining inliers are treated as inlier cluster. Now, perform a step-wise multiple discriminant analysis by consider the cluster membership as dependent variable and utilizing the variables already used to detect multivariate outliers as independent variables. Moreover, retain the dominant subset of discriminant independent variables and leave the remaining variables. The dominant subset of variables are treated as variables cluster-1.Now, remove the outliers from the entire sample, leave the subset of dominant variables from the original set and repeat the same procedure for the remaining inliers and variables, until one of the following results will be attain i) there are no outliers in the final cluster ii) the set of predictors in the last iteration will be empty or there are no subsets emerged and iii) the variance-covariance matrix for the variables in the final cluster achieved singularity. At each iteration, multivariate test of mean used to check the discrimination between the outlier clusters and the inliers. Moreover, multivariate control charts also used to graphically visualizes the iterations and clustering process. This paper employed this procedure for clustering 275 customers of a famous four-wheeler in India based on 19 different attributes of the four wheeler and its company. The result of the proposed technique confirms there exist 6 clusters of customers, 5 variable clusters at 5% significance level and 5 clusters of customers, 5 variable clusters at 1% significance level respectively.

Outlier (or anomaly) detection is an important problem for many domains, including fraud detection, risk analysis, network intrusion and medical diagnosis, and the discovery of significant outliers is becoming an integral aspect of data mining. This paper presents CURIO, a novel algorithm that uses quantisation and implied distance metrics to provide a fast algorithm that is linear for the number of objects and only requires two sequential scans of disk resident datasets. CURIO includes a novel direct quantisation technique and the ...

When a system fails to function properly, healthrelated data are collected for troubleshooting. However, it is challenging to effectively identify anomalies from the voluminous amount of noisy, high-dimensional data. The traditional manual approach is time-consuming, error-prone, and even worse, not scalable. In this paper, we present an automated mechanism for node-level anomaly identification in large-scale systems. A set of techniques are presented to automatically analyze collected data: data transformation to construct a uniform data format for data analysis, feature extraction to reduce data size, and unsupervised learning to detect the nodes acting differently from others. Moreover, we compare two techniques, principal component analysis (PCA) and independent component analysis (ICA), for feature extraction. We evaluate our prototype implementation by injecting a variety of faults into a production system at NCSA. The results show that our mechanism, in particular, the one using ICA-based feature extraction, can effectively identify faulty nodes with high accuracy and low computation overhead.

Techniques based on agglomerative hierarchical clustering constitute one of the most frequent approaches in unsupervised clustering. Some are based on the single linkage methodology, which has been shown to produce good results with sets of clusters of various sizes and shapes. However, the application of this type of algorithms in a wide variety of fields has posed a number of problems, such as the sensitivity to outliers and fluctuations in the density of data points. Additionally, these algorithms do not usually allow for automatic clustering.

Detecting outliers in data is an important problem with interesting applications in a myriad of domains ranging from data cleaning to financial fraud detection and from network intrusion detection to clinical diagnosis of diseases. Over the last decade of research, distance-based outlier detection algorithms have emerged as a viable, scalable, parameter-free alternative to the more traditional statistical approaches.

Outlier detection is an important task in many applications; it can lead to the discovery of unexpected, useful or interesting objects in data analysis. Many outlier detection methods are available. However, they are limited by assumptions in distribution or rely on many patterns to detect one outlier. Often, a distribution is not known, or experimental results may not provide enough information about a set of data to be able to determine a certain distribution. Previous work in outlier detection based on area-descent focused on detecting outliers which are solely isolated; it can not detect the outliers clustered together. In this paper, we propose a new approach for outlier detection based on two-stage area-descent of convex-hull polygon. It not only detects outliers clustered together but also shows their location related to the data set. Instead of removing the outlier, this relative location provides a suitable direction for moving the outlier to reduce its effects to linear regression. In addition, this method does not depend on the distribution of data set.

Special thanks are given to my supervisors in Lisbon and Stavanger, Prof. Paulo Urbano and Prof. Chunming Rong, who accepted supervise this master thesis project in two different universities, on two countries so far away, for the orientation and suggestions, patience on my writing process and precious corrections.

Before implementing any multivariate statistical analysis based on empirical covariance matrices, it is important to check whether outliers are present because their existence could induce significant biases. In this article, we present the minimum covariance determinant estimator, which is commonly used in robust statistics to estimate location parameters and multivariate scales. These estimators can be used to robustify Mahalanobis distances and to identify outliers. Verardi and Croux (1999, Stata Journal 9: 439–453; 2010, Stata Journal 10: 313) programmed this estimator in Stata and made it available with the mcd command. The implemented algorithm is relatively fast and, as we show in the simulation example section, outperforms the methods already available in Stata, such as the Hadi method.

Given the widespread use of modern information technology, a large number of time series may be collected during normal business operations. We use a fast-food restaurant franchise as a case to illustrate how data mining can be applied to such time series, and help the franchise reap the benefits of such an effort. Time series data mining at both the store level and corporate level are discussed. Box-Jenkins seasonal ARIMA models are employed to analyze and forecast the time series. Instead of a traditional manual approach of Box-Jenkins modeling, an automatic time series modeling procedure is employed to analyze a large number of highly periodic time series. In addition, an automatic outlier detection and adjustment procedure is used for both model estimation and forecasting. The improvement in forecast performance due to outlier adjustment is demonstrated.

Outlier is an object which is different from any objects in one dataset. In data mining, outlier detection is one of growing researches. Generally, outlier detection methods find exception or rare cases in a dataset without considered class label as an important thing and only can be used on dataset that have single datatypes. In fact, real world dataset usually have mixed datatypes. On outlier detection using Enhanced Class Outlier Distance Based (ECODB) algorithm, data which is different from its class can be found. ECODB algorithm can be applied on dataset that have numerical and categorical attributes. ECODB algorithm count the Class Outlier Factor (COF) from each instances based on k and top N value. COF is the probability/degree from an instance to be considered as outlier. Outlier is data which have least COF value. The testing can be done by counting credit data of BPR XYZ using ECODB algorithm with various input of k and top N. The determination of k and top N value influence the results of outlier detection. Very small or very high of k and top N value cause unoptimal outlier detection. Keyword : data mining, outlier detection, ecodb, enhanced class outlier distance based

Increased interest in the opportunities provided by artificial intelligence and machine learning has spawned a new field of healthcare research. The new tools under development are targeting many aspects of medical practice, including changes to the practice of pathology and laboratory medicine. Optimal design in these powerful tools requires cross-disciplinary literacy, including basic knowledge and understanding of critical concepts that have traditionally been unfamiliar to pathologists and laboratorians. This review provides definitions and basic knowledge of machine learning categories (supervised, unsupervised, and reinforcement learning), introduces the underlying concept of the bias-variance trade-off as an important foundation in supervised machine learning, and discusses approaches to the supervised machine learning study design along with an overview and description of common supervised machine learning algorithms (linear regression, logistic regression, Naive Bayes, k-nearest neighbor, support vector machine, random forest, convolutional neural networks).

Empirical QSAR models are only valid in the domain they were trained and validated. Application of the model to substances outside the domain of the model can lead to grossly erroneous predictions. Partial least squares (PLS) regression provides tools for prediction diagnostics that can be used to decide whether or not a substance is within the model domain, i.e. if the model prediction can be trusted. QSAR models for four different environmental end-points are used to demonstrate the importance of appropriate training set selection and how the reliability of QSAR predictions can be increased by outlier diagnostics. All models showed consistent results; test set prediction errors were very similar in magnitude to training set estimation errors when prediction outlier diagnostics were used to detect and remove outliers in the prediction data. Test set prediction errors for substances classified as outliers were much larger. The difference in the number of outliers between models with a randomly and systematically selected training illustrates well the need of representative training data.

Outlier detection has been used for centuries to detect and, where appropriate, remove anomalous observations from data. Outliers arise due to mechanical faults, changes in system behaviour, fraudulent behaviour, human error, instrument error or simply through natural deviations in populations. Their detection can identify system faults and fraud before they escalate with potentially catastrophic consequences. It can identify errors and remove their contaminating effect on the data set and as such to purify the data for processing. The original outlier detection methods were arbitrary but now, principled and systematic techniques are used, drawn from the full gamut of Computer Science and Statistics. In this paper, we introduce a survey of contemporary techniques for outlier detection. We identify their respective motivations and distinguish their advantages and disadvantages in a comparative review.

Every second plethora of reviews on various product lines are being posted in a trending e-commerce website. The objective of a review section in such websites is to analyze customer satisfaction for sales growth and to aid buyers make right purchase decisions. This becomes a herculean task for the business analyst when the search space for interesting patterns is vast. Buyers could benefit from a trusted recommendation label for the product and an overall product rating purely based on data mining of the customer generated live review dataset. The proposed sentiment analysis system makes use of data mining and natural language processing algorithms. The words in the corpus selected for the sentiment classification is associated with fuzzy scores to indicate degree of polarity strength. Naïve bayes algorithm forms the basis for the project. It is simple yet yields effective results. Sentiment scoring of the review is used for automated five star rating that replaces the conventional manual rating. Context based anomaly detection is done to remove irrelevant portions of each review. Selective-feature analysis is performed to assist vendors as well as buyers. A threshold value is compared against the overall rating for the recommendation label. The polarity detection gave fair results. It was observed that the richness of corpus and grammar rules contributed to the accuracy of the model. Threshold chosen for product recommendation gave mediocre credence when it was solely based on reviews. Therefore Threshold was set using more complex criteria that involves purchase patterns. Results obtained from the proposed model is ideal for e-commerce websites. Model can be improved by addition of sarcasm detection algorithm which is quite challenging.

There is an increasing concern about the control of customs operations. While globalization incentives the opening of the market, increasing amounts of imports and exports have been used to conceal several illicit activities, such as, tax evasion, smuggling, money laundry, and drug traffic. This fact makes it paramount for governments to find automatic or semi-automatic solutions to guide the customs' activities in order to minimize the number of manual inspections of goods. In this context, this paper presents an overview of some approaches developed in the HARPIA project that is a partnership between universities and the Brazilian Federal Revenue for the development of computational intelligence solutions to the management of customs risk.

When analyzing data, outlying observations cause problems because they may strongly influence the result. Robust statistics aims at detecting the outliers by searching for the model fitted by the majority of the data. We present an overview of several robust methods and outlier detection tools. We discuss robust procedures for univariate, low-dimensional, and high-dimensional data such as estimation of location and scatter, linear regression, principal component analysis,

Outlier detection is an important branch in data mining, which is the discovery of data that deviate a lot from other data patterns. Outlier identification can be classified in to formal and informal methods. This paper deals the informal methods also called as labeling methods. Identification of outliers in real time medical data using outlier labeling methods was studied. There are several labeling methods applying in practical situation in the dataset are computed. Finally the estimated results of the outliers are more appropriate way to resolving the large populations.

By comparing historical data of trading like daily Open, High, Low, Close, Volume, Number of Trades, Turnover, Delivery percentage etc. of a particular stock with its Peer Group companies and Non Peer Group companies stocks for a particular period, we can find some unusual observations which are also known as outliers. In this paper we have tried to detect the observations, which are very different from the other observations using a Data Mining Technique for Outlier Detection-"Multiple Linear Regression Analysis".

The present article discusses various preprocessing techniques suitable for dealing with time series data for environmental science-related studies. The errors or noises due to electronic sensor fault, fault in the communication channel, etc., are considered here. Such errors or glitches that occur during the data acquisition or transmission phases need to be eliminated before it fed to the forecasting or classification systems. Computationally simple and efficient techniques are discussed here so that they can even be adopted for a hard real-time system environment. While adopting these techniques, we may also end up with some of the real genuine values, which may consider as an outlier. A special indicator function, the moving Inter Quartile Range (MIQR) algorithm, is proposed to overcome such special cases.

Seiring dengan semakin diminatinya minuman wine, banyak negara yang mendukung pertumbuhan industri minuman ini. Sertifikasi guna meyakinkan konsumen akan kualitas dan untuk mencegah pemalsuan terhadap produk anggur juga diperlukan. Sehingga diperlukan penilaian kualitas anggur, di mana variabel yang masuk dalam penelitian ini adalah tingkat keasaman (pH), alkohol, dan sulfat. Dalam penelitian ini, akan dibahas mengenai pre-processing pada data penilaian kualitas anggur Portugis "Vinho Verde" di tahun 2009, untuk mendeteksi adanya missing value dan data outlier. Setelah dideteksi, ditemukan terdapat missing data yang kemudian diatasi dengan teknik imputasi listwise dan pairwise. Dalam proses pre-processing, data terdeteksi oleh boxplot dan Mahalanobis distance bahwa mengandung outlier, sehingga diputuskan untuk menghapus data tersebut. Setelah mengatasi kedua hal tersebut, peneliti melanjutkan menguji asumsi normalitas dari data yang diuji secara univariat menggunakan Kolmogorov-Smirnov dan Liliefors. Dan untuk pengujian normalitas secara multivariat digunakan uji QQ-Plot dan Saphiro-Wilk.

In this paper we focus on the impact of additive level outliers on the calculation of risk measures, such as minimum capital risk requirements, and compare four alternatives of reducing these measures' estimation biases. The first three proposals proceed by detecting and correcting outliers before estimating these risk measures with the GARCH(1,1) model, while the fourth procedure fits a Student's t-distributed GARCH(1,1) model directly to the data. The former group includes the proposal of Grané and Veiga (2010), a detection procedure based on wavelets with hard-or soft-thresholding filtering, and the well known method of . The first results, based on Monte Carlo experiments, reveal that the presence of outliers can bias severely the minimum capital risk requirement estimates calculated using the GARCH(1,1) model. The message driven from the second results, both empirical and simulations, is that outlier detection and filtering generate more accurate minimum capital risk requirements than the fourth alternative. Moreover, the detection procedure based on wavelets with hard-thresholding filtering gathers a very good performance in attenuating the effects of outliers and generating accurate minimum capital risk requirements out-of-sample, even in pretty volatile periods.

Credit card frauds are at an ever-increasing rate and have become a major problem in the financial sector. Because of these frauds, card users are hesitant in making purchases and both the merchants and financial institutions bear heavy losses. Some major challenges in credit card frauds involve the availability of public data, high class imbalance in data, changing nature of frauds and the high number of false alarms. Machine learning techniques have been used to detect credit card frauds but no fraud detection systems have been able to offer great efficiency to date. Recent development of deep learning has been applied to solve complex problems in various areas. This paper presents a thorough study of deep learning methods for the credit card fraud detection problem and compare their performance with various machine learning algorithms on three different financial datasets. Experimental results show great performance of the proposed deep learning methods against traditional machine learning models and imply that the proposed approaches can be implemented effectively for real-world credit card fraud detection systems.

Raw data collected through surveys, experiments, coding of textual artifacts or other quantitative means may not meet the assumptions upon which statistical analyses rely. The presence of univariate or multivariate outliers, skewness or kurtosis in a distribution, and heteroscedasticity or multicollinearity among variables may compromise data analysis. Scholars have identified a variety of techniques to discern such problems and address them.

These days, with the popularity and significant advancements of emerging technologies such as Internet of Things (IoT), Cyber-Physical-Systems (CPS), and other wireless sensor technologies, the huge volume of sensor data has generated for IoT devices is vast. In these data, identification and detection of outliers/anomalies is a challenging issue and raised as the primary importance of data analysis. In the olden days, the conventional outlier detection techniques are not effectively applied to deal with outliers over IoT data. Therefore, this paper explores a comprehensive survey of the latest Machine Learning (ML)-based outlier detection techniques for handling outliers in IoT data. Also, surveyed the various smart city based use cases related to IoT applications more significantly. Besides, the required performance evaluation metrics have been addressed for validating the results of ML-based outlier detection techniques. Finally, this article also addressed the possible open research issues that are necessary to deal with outliers in IoT sensor data

The minimum covariance determinant (MCD) estimator is a highly robust estimator of multivariate location and scatter. It can be computed efficiently with the FAST-MCD algorithm of Rousseeuw and Van Driessen. Since estimating the covariance matrix is the cornerstone of many multivariate statistical methods, the MCD has also been used to develop robust and computationally efficient multivariate techniques.

Center of Mass (CoM) estimation realizes a crucial role in legged locomotion. Most walking pattern generators and real-time gait stabilizers commonly assume that the CoM position and velocity are available for feedback. In this thesis we present one of the first
3D-CoM state estimators for humanoid robot walking. The proposed estimation scheme fuses effectively joint encoder, inertial, and feet pressure measurements with an Extended Kalman Filter (EKF) to accurately estimate the 3D-CoM position, velocity, and external
forces acting on the CoM. Furthermore, it directly considers the presence of uneven terrain and the body’s angular momentum rate and thus effectively couples the frontal with the lateral plane dynamics, without relying on feet Force/Torque (F/T) sensing.
Nevertheless, it is common practice to transform the measurements to a world frame of reference and estimate the CoM with respect to the world frame. Consequently, the robot’s base and support foot pose are mandatory and need to be co-estimated. To this end, we extend a well-established in literature floating mass estimator to account for the
support foot dynamics and fuse kinematic-inertial measurements with the Error State Kalman Filter (ESKF) to appropriately handle the overparametrization of rotations. In such a way, a cascade state estimation scheme consisting of a base and a CoM estimator
is formed and coined State Estimation RObot Walking (SEROW). Additionally, we employ Visual Odometry (VO) and/or LIDAR Odometry (LO) measurements to correct the kinematic drift caused by slippage during walking. Unfortunately, such measurements suffer from outliers in a dynamic environment, since frequently it is assumed that only the
robot is in motion and the world around is static. Thus, we introduce the Robust Gaussian ESKF (RGESKF) to automatically detect and reject outliers without relying on any prior knowledge on measurement distributions or finely tuned thresholds. Therefore, SEROW
is robustified and is suitable for dynamic human environments. In order to reinforce further research endeavors, SEROW is released to the robotic community as an open-source ROS/C++ package.
Up to date control and state estimation schemes readily assume that feet contact status is known a priori. Contact detection is an important and largely unexplored topic in contemporary humanoid robotics research. In this thesis, we elaborate on a broader question: in which gait phase is the robot currently in? To this end, we propose a holistic frame-
work based on unsupervised learning from proprioceptive sensing that accurately and efficiently addresses this problem. More specifically, we robustly detect one of the three gait-phases, namely Left Single Support (LSS), Double Support (DS), and Right Single Support (RSS) utilizing joint encoder, IMU, and F/T measurements. Initially, dimensionality reduction with Principal Components Analysis (PCA) or autoencoders is performed to extract useful features, obtain a compact representation, and reduce the noise. Next, clustering is performed on the low-dimensional latent space with Gaussian Mixture Models (GMMs) and three dense clusters corresponding to the gait-phases are obtained. Interestingly, it is
demonstrated that the gait phase dynamics are low-dimensional which is another indication pointing towards locomotion being a low dimensional skill. Accordingly, given that the proposed framework utilizes measurements from sensors that are commonly available
on humanoids nowadays, we offer the Gait-phase Estimation Module (GEM), an open-source ROS/Python implementation to the robotic community.
SEROW and GEM have been quantitatively and qualitatively assessed in terms of accuracy and efficiency both in simulation and under real-world conditions. Initially, a simulated robot in MATLAB and NASA’s Valkyrie humanoid robot in ROS/Gazebo were employed to establish the proposed schemes with uneven/rough terrain gaits. Subsequently,
the proposed schemes were integrated on a) the small size NAO humanoid robot v4.0 and b) the adult size WALK-MAN v2.0 for experimental validation. With NAO, SEROW was implemented on the robot to provide the necessary feedback for motion planning and real-
time gait stabilization to achieve omni-directional locomotion even on outdoor/uneven terrains. Additionally, SEROW was used in footstep planning and also in Visual SLAM with the same robot. Regarding WALK-MAN v2.0, SEROW was executed onboard with kinematic-inertial and F/T data to provide base and CoM feedback in real-time. Furthermore, VO has also been considered to correct the kinematic drift while walking and facilitate possible footstep planning. GEM was also employed to estimate the gait phase in WALK-MAN’s dynamic gaits.
Summarizing, a robust nonlinear state estimator is proposed for humanoid robot walking. Nevertheless, this scheme can be readily extended to other type of legged robots such
as quadrupeds, since they share the same fundamental principles.

An outlier is an observations which deviates or far away from the rest of data. There are two kinds of outlier methods, tests discordance and labeling methods. In this paper, we have considered the medical diagnosis data set finding outlier with discordancy test and comparing the performance of outlier detection. Most of the outlier detection methods considered as extreme value is an outlier. In some cases of outlier detection methods no need to use statistical table. The suggested outlier detection methods using the context of detection sensitivity and difficulties of analyzing performance for outlier detections are compared.

Outliers in a set of data represent observations that are distinguished from the expected patterns in the observed data. We are attracted to them because they are somehow atypical of what we expect to see in the distribution of the data that we have collected or generated. We may be troubled by them because they may contain important information about the process which we are attempting to model, or they may simply be spurious noise or even data points that have been corrupted by noise. In any case, we must be careful in our treatment of them because we cannot easily decide whether they represent some valuable aspect of the process under study or are simply aberrations.

Pemodelan menggunakan analisis regresi linier berganda memiliki tindak lanjut setelah full model terbentuk. Tindak lanjut dilakukan untuk mendapatkan hasil evaluasi yang terbaik untuk model. Pemodelan menggunakan analisis regresi berganda menghasilkan nilai Residual Standard Error dan Koefisien Determinasi R2 yang dapat dijadikan evaluasi model yang sudah terbentuk (James, 2013). Metode tindak lanjut setelah full model terbentuk adalah dengan penerapan model dengan deteksi outliers dan transformasi pada variabel respon. Pada studi kasus menggunakan data PDAM Provinsi Jawa Barat, Jawa Timur, D.I Yogyakarta dan Jawa Timur pada Data Rekapitulasi Kinerja PDAM di Indonesia Tahun 2017. Berdasarkan 4 metode yang sudah digunakan, maka model yang memiliki Residual Standard Error terendah dan nilai koefisien determinasi tertinggi adalah Model dengan Metode Regresi berganda dengan menghilangkan observasi yang dianggap Outliers dengan memperhatikan nilai Externally Studentized Residuals.

Outlier detection has recently become an important problem in many industrial and financial applications. In this paper, a novel feature bagging approach for detecting outliers in very large, high dimensional and noisy databases is proposed. It combines results from multiple outlier detection algorithms that are applied using different set of features. Every outlier detection algorithm uses a small subset of features that are randomly selected from the original feature set. As a result, each outlier detector identifies different outliers, and thus assigns to all data records outlier scores that correspond to their probability of being outliers. The outlier scores computed by the individual outlier detection algorithms are then combined in order to find the better quality outliers. Experiments performed on several synthetic and real life data sets show that the proposed methods for combining outputs from multiple outlier detection algorithms provide non-trivial improvements over the base algorithm. AS t,k … AS t,4 AS t,3 AS t,2 AS t,1

For many KDD applications, such as detecting criminal activities in E-commerce, finding the rare instances or the outliers, can be more interesting than finding the common patterns. Existing work in outlier detection regards being an outlier as a binary property. In this paper, we contend that for many scenarios, it is more meaningful to assign to each object a degree of being an outlier. This degree is called the local outlier factor (LOF) of an object. It is local in that the degree depends on how isolated the object is with respect to the surrounding neighborhood. We give a detailed formal analysis showing that LOF enjoys many desirable properties. Using realworld datasets, we demonstrate that LOF can be used to find outliers which appear to be meaningful, but can otherwise not be identified with existing approaches. Finally, a careful performance evaluation of our algorithm confirms we show that our approach of finding local outliers can be practical.

Most outlier detection rules for multivariate data are based on the assumption of elliptical symmetry of the underlying distribution. We propose an outlier detection method which does not need the assumption of symmetry and does not rely on visual inspection. Our method is a generalization of the Stahel-Donoho outlyingness. The latter approach assigns to each observation a measure of outlyingness, which is obtained by projection pursuit techniques that only use univariate robust measures of location and scale. To allow skewness in the data, we adjust this measure of outlyingness by using a robust measure of skewness as well. The observations corresponding to an outlying value of the adjusted outlyingness are then considered as outliers. For bivariate data, our approach leads to two graphical representations. The first one is a contour plot of the adjusted outlyingness values. We also construct an extension of the boxplot for bivariate data, in the spirit of the bagplot [1] which is based on the concept of half space depth. We illustrate our outlier detection method on several simulated and real data.

In this paper we propose a method for correctly detecting outliers based on a new technique developed to simultaneously evaluate mean, variance and outliers. This method is capable of self-regulating its robustness to suit the experimental data set under analysis, so as to overcome shortcomings of: (i) nonrobust methods such as the least sum of squares; (ii) the need of the user in defining a trimmed sub-set of experimental points such as in least trimmed sum of squares; and (iii) the possibility to read the data set only once to evaluate the mean, variance, and outliers of a population by preserving robustness.

Plagiarism detection can be divided in external and intrinsic methods. Naive external plagiarism analysis suers from computationally demanding full near- est neighbor searches within a reference corpus. We present a conceptually simple space partitioning approach to achieve search times sub linear in the number of ref- erence documents, trading precision for speed. We focus on full duplicate searches while achieving

Multivariate calibration 1 (MVC1), a MatLabR toolbox for implementing up to 12 different first-order calibration methodologies through easily managed graphical user interfaces, is presented. The toolbox accepts different input data formats (either arranged as matrices or vectors contained in raw data files or in already existing MatLab variables) and incorporates many preprocessing algorithms in order to improve prediction capabilities. The development and validation of each model and its subsequent application to unknown samples are straightforward. Prediction results are produced along analytical figures of merit and standard errors calculated by uncertainty propagation. Moreover, the toolbox allows one to manually select working sensor regions, or to automatically find which region provides the minimum error. It also generates many different plots regarding model performance, including outliers detection, facilitating both model evaluation and interpretation. D

Cognitive radio is an enabling technology that allows opportunistic users to reuse licensed spectrum in order to overcome the artificial spectrum scarcity. In cognitive radio networks, opportunistic users collaboratively perform spectrum sensing to detect the presence of incumbent users. Collaborative Spectrum Sensing (CSS) performance suffers due to the presence of malicious users. We propose a robust malicious user detection algorithm which exploits inherent statistical moments of the sensing observations from collaborating users to identify malicious users among them. In this way, CSS performance is improved by detecting and marginalizing the effects of malicious users.