Table of Contents
show
Big data has shown phenomenal growth over the past decade and it’s widespread application by businesses as a growth catalyst continues to deliver positive results. The scale of data is massive and the volume, velocity and variety of data calls for more efficient processing to make it machine-ready. Although there are a multitude of ways to extract data such as public APIs, custom web scraping services, internal data sources, etc., there would always remain the need to do some pre-processing to make the data perfectly suitable for business applications.
Pre-processing of data involves a set of key tasks that demand extensive computational infrastructure and this in turn will make way for better results from your big data strategy. Moreover, cleanliness of the data would determine the reliability of your analysis and this should be given high priority while plotting your data strategy.
Data pre-processing techniques
Since the extracted data tend to be imperfect with redundancies and imperfections, data pre-processing techniques are an absolute necessity. The bigger the data sets, the more complex mechanisms are needed to process it before analysis and visualization. Pre-processing prepares the data and makes the analysis feasible while improving the effectiveness of the results. Following are some of the crucial steps involved in data pre-processing.
Data cleansing
Cleansing the data is usually the first step in data processing and is done to remove the unwanted elements as well as to reduce the size of the data sets, which will make it easier for the algorithms to analyze it. Data cleansing is typically done by using instance reduction techniques.
Instance reduction helps reduce the size of the data set without compromising the quality of insights that can be extracted from the data. It removes instances and generates new ones to make the data set compact. There are two major instance reduction algorithms:
Instance selection: Instance selection is used to identify the best examples from a very large data set with many instances in order to curate them as the input for the analytics system. It aims to select a subset of the data that can act as a replacement for the original data set while completely fulfilling the goal. It will also remove redundant instances and noise.
Instance generation: Instance generation methods involve replacing the original data with artificially generated data in order to fill regions in the domain of an issue with no representative examples in the master data. A common approach is to relabel examples that appear to belong to wrong class labels. Instance generation thus makes the data clean and ready for the analysis algorithm.
Tools you can use: Drake, DataWrangler, OpenRefine
Data normalization
Normalization improves the integrity of the data by adjusting the distributions. In simple words, it normalizes each row to have a unit norm. The norm is specified by parameter p which denotes the p-norm used. Some popular methods are:
StandardScaler: Carries out normalization so that each feature follows a normal distribution.
MinMaxScaler: Uses two parameters to normalize each feature to a specific range – upper and lower bound.
ElementwiseProduct: Uses a scalar multiplier to scale every feature.
Tools you can use: Table analyzer, BDNA
Data transformation
If a data set happens to be too large in the number of instances or predictor variables, dimensionality problem arises. This is a critical issue that will obstruct the functioning of most data mining algorithms and increases the cost of processing. There are two popular methods for data transformation by dimensionality reduction – Feature Selection and Space Transformation.
Feature selection: It is the process of spotting and eliminating as much unnecessary information as possible. FS can be used to significantly reduce the probability of accidental correlations in learning algorithms that could degrade their generalization capabilities. FS will also cut the search space occupied by features, thus making the process of learning and mining faster. The ultimate goal is to derive a subset of features from the original problem that describes it well.
Space transformations: Space transformations work similar to feature selection. However, instead of selecting the valuable features, space transformation technique will create a fresh new set of features by combining the originals. This kind of a combination can be made to obey certain criteria. Space transformation techniques ultimately aim to exploit non-linear relations among the variables.
Tools you can use: Talend, Pentaho
Missing values imputation
One of the common assumptions with big data is that the data set is complete. In fact, most data sets have missing values that’s often overlooked. Missing values are datums that haven’t been extracted or stored due to budget restrictions, a faulty sampling process or other limitations in the data extraction process. Missing values is not something to be ignored as it could skew your results.
Fixing the missing values issue is challenging. Handling it without utmost care could easily lead to complications in data handling and wrong conclusions.
There are some relatively effective approaches to tackle the missing values problem. Discarding the instances that might contain missing values is the common one but it’s not very effective as it could lead to bias in the statistical analyses. Apart from this, discarding critical information is not a good idea. A better and more effective method is to use maximum likelihood procedures to model the probability functions of the data while also considering the factors that could have induced the missingness. Machine learning techniques are so far the most effective solution to the missing values problem.
Noise identification
Data gathering is not always perfect, but the data mining algorithms would always assume it to be. Data with noise can seriously affect the quality of the results, tackling this issue is crucial. Noise can affect the input features, output or both in most cases. The noise found in the input is called attribute noise whereas if the noise creeps into the output, it’s referred to as class noise. If noise is present in the output, the issue is very serious and the bias in the results would be very high.
There are two popular approaches to remove noise from the data sets. If the noise has affected the labelling of instances, data polishing methods are used to eliminate the noise. The other method involves using noise filters that can identify and remove instances with noise from the data and this doesn’t require modification of the data mining technique.
Minimizing the pre-processing tasks
Preparing the data for your data analysis algorithm can involve many more processes depending on the application’s unique demands. However, basic processes like cleansing, deduplication and normalization can be avoided in most cases if you choose the right source for data extraction. It’s highly unlikely that a raw source can give you clean data. As far as web data extraction is concerned, a managed web scraping service like PromptCloud can give you clean and ready to use data that’s ready to be plugged into your analytics system. As the data provided by our DaaS solution is clean, you can save your best efforts for your application-specific data processing tasks.
