EDA in Machine Learning

Everyone wants an ultra-accurate, super-smart assistant, and this is why machine learning systems are gaining popularity across industries. Machine learning is a problem-solving tool. It is considered to be the Future for reasons such as computational speed and predictive analysis.

EDA in machine learning refers to Exploratory Data Analysis, a critical initial step in data processing. It involves examining and visualizing datasets to understand their main characteristics, often using statistical graphics and other data visualization methods.

This process helps identify patterns, anomalies, trends, and relationships within the data, guiding the selection of appropriate models and algorithms for machine learning tasks.

Machine learning is like an assistant that can work much faster than a human and is incredibly accurate at providing analytical approach to solving difficult problems.

A popular example of an industry that relies on machine learning is autonomous or driverless vehicles. Although autonomous vehicles are in their early stages, by 2035, autonomous driving could generate $300 billion to $400 billion in revenue, according to McKinsey & Company.

This innovation is influenced by machine learning algorithms, which collect data from the environment with the help of cameras and sensors, interpret it, and decide what to do with it. For instance, Radar sensors algorithm is essential for monitoring the position of nearby vehicles to avoid accidents.

Machine learning models rely on data to understand how humans do things and complete tasks. However, to make these models capable of making decisions, we need to teach them through a process called model training.

Training a machine learning model involves a series of specific steps, which include:

• Data: A large amount of data is required to effectively teach a system. This material should be properly prepared by removing any irregularities, inconsistencies, or garbage information.
• Pattern Identification: Data is exposed to a machine-learning algorithm for analysis and learning purposes.
• Prediction: MIL (Machine instance Learning) is used to make predictions that take different forms based on the specific task.

Data is significant because machine learning algorithms use it to predict or make data-based decisions. This means that without data, we are unable to train any model.

Data can take several forms, including numerical and categorical data. Age, income, and time are examples of numerical data or data in the form of numbers. While categorical data represents categories (e.g., gender, race, sex, etc.), numeric data represents numbers.

The focus of this article will be on exploratory data analysis. We'll look at the basics and importance of exploratory data analysis in the context of machine learning, a practice that dictates how algorithms use raw data to make judgments.

The lifecycle of data science projects requires exploratory data analysis. Data scientists use this process to carefully review, analyze, and understand datasets. The insights from data sets can be used to create a data model to spot patterns in a business.

The data scientist or analyst involved in the process of EDA manipulates data to find trends, such as ‘finding’ a high churn rate or gauging a company's activity level across different regions. To get these insights, exploratory data analysis (EDA) employs strategies like statistical summaries and graphical representations.

Explanatory data analyses are performed prior to data modelling to fully comprehend the data sets. There are several reasons why explanatory Data analysis is important to ensure that the data used for modelling is as accurate as possible, and they include the following:

• Discover relevant patterns in data for improved insights.
• Effectively detect and correct data discrepancies and inaccuracies.
• Determine and correct any missing or missed data points.
• Validate theories through in-depth data analysis.
• During the analysis, confirm or question initial assumptions.

Data modelling in machine learning involves creating a visual representation of data patterns to help machines do intelligent tasks. Google Assistant, for example, uses natural machine learning models and algorithms to understand and respond to user requests.

There are different learning models:

• Supervised Learning Model: This model gains knowledge from labelled datasets. Suppose we aim to train the algorithm to recognize and distinguish between chairs and tables. In this scenario, all chairs and tables would come with clear labels. This process is also referred to as annotation. There are different types of supervised learning, and they are:

  • Classification: Involves predicting a class or discrete values, e.g., true or false, male or female. Examples of classification algorithms used in machine learning are Logistic Regression, Decision Tree, Random Forest, K-nearest Neighbor, etc.

  • Regression: Involves predicting a quantity or continuous value, e.g., Price, age, or temperature. Examples of regression algorithms used in machine learning are Linear regression, Lasso regression, Random Forest regression, and Support Vector Machine regression.

• Unsupervised Learning Model: The model learns from unlabeled data sets or training models that are not supervised using training data sets.

  • Clustering: Involves grouping of data into several categories based on similarities or differences.

  • Association: Involves finding relationships between variables in a database. For instance, people who have cats tend to purchase cat meals.

Explanatory Data Analysis can be divided into two sub-groups;

1. Univariate: This is data that can be described using one variable. There are two classifications under univariate, and they are:

• Univariate non-graphical analysis: Univariate non-graphical analysis is a method whereby the data being analyzed consists of a single variable. This is considered to be the simplest form of data analysis.

• Univariate graphical analysis: This method of analysis doesn’t give a full picture of the data because it is limited to a single variable, unlike multivariate analysis. It’s also said to be quantitative and objective because it uses numerical data and visualization to give meaningful and measurable representations of variable attributes. Examples of univariate graphic analysis include:

  • Histograms
  • Box Plots
  • Bar Charts
  • Frequency Tables
  • Stem and Leaf plot

2. Multivariate: This is data that can be described using more than one variable. There are two classifications under Multivariate EDA, and they are:

• Multivariate non-graphical: This method of analysis shows the correlation between two or more variables by cross-tabulation or statistics.

• Multivariate graphical: This method of analysis involves analyzing multiple variables or sets of data at the same time to find the relationship between them. E.g. groups, bar charts, bubble charts, heat maps, etc.

The steps required when carrying out explanatory data analysis vary, and data scientists or analysts may tailor steps depending on the data set and problem they’re trying to solve. However, there are common EDA techniques, which may include:

Data collection: It involves gathering large volumes of data from different mediums, e.g. surveys, APIs, databases, etc.

Data Inspection: Here, the structure and content of the data set are examined.

Handle Missing Values: Missing values are Identified using standard deviation.

Data visualization: Plots are created to find the relationship between data and data distributions.

Treating Outliers: Identify potential outliers in data

Test Hypothesis: Conduct a test to authenticate assumptions

Segment Data: Employ data division based on categorical variables

Explore Data: Employ interactive tools to explore data

Reporting: Communicate results to the analytics manager or other stakeholders.

Prior to diving into a dataset, it's crucial to begin with thoughtful questions. These inquiries help frame your data research and guide you towards meaningful and relevant insights and solutions. Some questions you might ask are:

• What are we trying to achieve with this data set?
• What models can we build with the available data?
• Is the data available enough to scale a machine-learning model?
• What insights can we gain from the data set?
• What visualization tool can we use to show insights to business stakeholders?

In the example below, we’ll perform a simple exploratory data analysis with a population data set. We’ll show you how to read a CSV file in a Jupyter Notebook using Panda. Also, we’ll be using Jupyter Notebook hosted in Anaconda’s cloud environment.

• Import Libraries

• Download the data - upload the CSV file and load the file by typing the following command

df = pd.read_csv( "world_population.csv")

df

• Understand the data

Show the information of the data set using df.info() allows us to see how many columns we have in our data

df. head() displays the five first columns; we can see the top column has information like country, capital, continent, population by year, etc. This information would be important when working on any model or making assumptions. Some information can be removed if we find that it is not necessary for our data set. We can also use df. tail() to display the bottom of the column.

df. shape to check the shape of our data, this shows us that our data has 234 rows and 17 columns.

df.columns to check the columns separately

We can search for unique values using df. nunique

You can also check the unique columns separately using df['columnName'].unique()

• Manipulate data

We can remove data that we do not want from our data set. In the image below, we want to delete the column Rank del df['Rank']

• Visualize data

There are different methods for visualizing data; it could be a histogram, scatterplot, box plot, etc. Here’s what visualizing data using diff.hist() looks like.

These basic examples are to show you what you can do with EDA tools. Learning from data can be very complex. To further assist, here are sites where you can get free data sets for your project:

• Kaggle
• Datahub.io
• Google Data Search
• FiveThirtyEight
• data.world

Exploratory data analysis can be performed using different tools depending on the complexity of the analysis and your preferences.

• Python: Python can be used with EDA to identify missing data in a data set. This helps to decide how to handle data for machine learning.

• R: R language is used by statisticians in data science in developing statistical observation and data analysis.

• Tableau: This is a visualization tool to create interactive and sharable dashboards

• Power BI: This is a visual analytical tool for data visualization and sharing

• Jupyter Notebook: This is an IDE [Integrated developer environment] for Python for data analysis, visualization, and collaboration.

• Rstudio: This is an IDE [Integrated developer environment] for R that facilitates data analysis and visualization.

• Seaborn: This serves as an Interface for creating statistical graphics

• MatplotLib: Creates static, animated, and interactive visualizations

• Pandas: Python library for analyzing, cleaning, and manipulating data sets.

After Exploratory Data Analysis is performed, there are other steps involved;

• Feature selection: This involves selecting the most relevant features from data sets to use as inputs for the machine learning model.

• Model Selection: This involves choosing an appropriate machine learning model.

• Hyperparameter tuning: Adjust the model’s parameters to improve performance.

• Model deployment: After the model meets the criteria, deploy it in a production environment for making predictions.

• Incremental learning: Continuously monitoring model performance, relearning new data to make sure it’s accurate and up to date.

• Documentation: Record results to provide comprehensive insights during the EDA process.

In simple terms, to arrive at a solution, one must first understand the problem. EDA is a method for delving into data to comprehend it fully and draw essential conclusions and insights. Once EDA is done, these insights can inform the development used to create more elaborate modelling or data analysis. In essence, Data scientists can understand patterns, uncover anomalies, and find relationships within data that can be leveraged by businesses for informed decision-making.