Understanding data analysis

 Understanding data analysis

The 21st century is the century of information. We are living in the age of information,

which means that almost every aspect of our daily life is generating data. Not only this, but

business operations, government operations, and social posts are also generating huge data.

This data is accumulating day by day due to data being continually generated from

business, government, scientific, engineering, health, social, climate, and environmental

activities. In all these domains of decision-making, we need a systematic, generalized,

effective, and flexible system for the analytical and scientific process so that we can gain

insights into the data that is being generated.

In today's smart world, data analysis offers an effective decision-making process for

business and government operations. Data analysis is the activity of inspecting, preprocessing, exploring, describing, and visualizing the given dataset. The main objective of

the data analysis process is to discover the required information for decision-making. Data

analysis offers multiple approaches, tools, and techniques, all of which can be applied to

diverse domains such as business, social science, and fundamental science.

Let's look at some of the core fundamental data analysis libraries of the Python ecosystem:

NumPy: This is a short form of numerical Python. It is the most powerful

scientific library available in Python for handling multidimensional arrays,

matrices, and methods in order to compute mathematics efficiently.

SciPy: This is also a powerful scientific computing library for performing

scientific, mathematical, and engineering operations.

Pandas: This is a data exploration and manipulation library that offers tabular

data structures such as DataFrames and various methods for data analysis and

manipulation.

Scikit-learn: This stands for "Scientific Toolkit for Machine learning". It is a

machine learning library that offers a variety of supervised and unsupervised

algorithms, such as regression, classification, dimensionality reduction, cluster

analysis, and anomaly detection.

Matplotlib: This is a core data visualization library and is the base library for all

other visualization libraries in Python. It offers 2D and 3D plots, graphs, charts,

and figures for data exploration. It runs on top of NumPy and SciPy.

Seaborn: This is based on Matplotlib and offers easy to draw, high-level,

interactive, and more organized plots.

Plotly: Plotly is a data visualization library. It offers high quality and interactive

graphs, such as scatter charts, line charts, bar charts, histograms, boxplots,

heatmaps, and subplots.

The standard process of data analysis

Data analysis refers to investigating the data, finding meaningful insights from it, and

drawing conclusions. The main goal of this process is to collect, filter, clean, transform,

explore, describe, visualize, and communicate the insights from this data to discover

decision-making information. Generally, the data analysis process is comprised of the

following phases:

1. Collecting Data: Collect and gather data from several sources.

2. Preprocessing Data: Filter, clean, and transform the data into the required

format.

3. Analyzing and Finding Insights: Explore, describe, and visualize the data and

find insights and conclusions.

4. Insights Interpretations: Understand the insights and find the impact each

variable has on the system.

5. Storytelling: Communicate your results in the form of a story so that a layman

can understand them.

The KDD process

The KDD acronym stands for knowledge discovery from data or Knowledge Discovery in Databases. Many people treat KDD as one synonym for data mining. Data mining is referred to as the knowledge discovery process of interesting patterns. The main objective of KDD is to extract or discover hidden interesting patterns from large databases, data warehouses, and other web and information repositories. The KDD process has seven major phases:

1. Data Cleaning: In this first phase, data is preprocessed. Here, noise is removed, missing values are handled, and outliers are detected.

2. Data Integration: In this phase, data from different sources is combined and integrated together using data migration and ETL tools.

3. Data Selection: In this phase, relevant data for the analysis task is recollected.

Data Transformation: In this phase, data is engineered in the required appropriate form for analysis.

5. Data Mining: In this phase, data mining techniques are used to discover useful and unknown patterns.

6. Pattern Evaluation: In this phase, the extracted patterns are evaluated.

7. Knowledge Presentation: After pattern evaluation, the extracted knowledge needs to be visualized and presented to business people for decision-making purposes.

SEMMA

The SEMMA acronym's full form is Sample, Explore, Modify, Model, and Assess. This

sequential data mining process is developed by SAS. The SEMMA process has five major

phases:

1. Sample: In this phase, we identify different databases and merge them. After this, we select the data sample that's sufficient for the modeling process.

2. Explore: In this phase, we understand the data, discover the relationships among variables, visualize the data, and get initial interpretations.

3. Modify: In this phase, data is prepared for modeling. This phase involves dealing with missing values, detecting outliers, transforming features, and creating new additional features.

4. Model: In this phase, the main concern is selecting and applying different modeling techniques, such as linear and logistic regression, backpropagation networks, KNN, support vector machines, decision trees, and Random Forest.

5. Assess: In this last phase, the predictive models that have been developed are evaluated using performance evaluation measures

The preceding diagram shows the steps involved in the SEMMA process. SEMMA emphasizes model building and assessment. Now, let's discuss the CRISP-DM process.

CRISP-DM

CRISP-DM's full form is CRoss-InduStry Process for Data Mining. CRISP-DM is a welldefined, well-structured, and well-proven process for machine learning, data mining, and

business intelligence projects. It is a robust, flexible, cyclic, useful, and practical approach to

solving business problems. The process discovers hidden valuable information or patterns

from several databases. The CRISP-DM process has six major phases:

1. Business Understanding: In this first phase, the main objective is to understand

the business scenario and requirements for designing an analytical goal and

initial action plan.

2. Data Understanding: In this phase, the main objective is to understand the data

and its collection process, perform data quality checks, and gain initial insights.

3. Data Preparation: In this phase, the main objective is to prepare analytics-ready

data. This involves handling missing values, outlier detection and handling,

normalizing data, and feature engineering. This phase is the most timeconsuming for data scientists/analysts.

4. Modeling: This is the most exciting phase of the whole process since this is

where you design the model for prediction purposes. First, the analyst needs to

decide on the modeling technique and develop models based on data.

5. Evaluation: Once the model has been developed, it's time to assess and test the

model's performance on validation and test data using model evaluation

measures such as MSE, RMSE, R-Square for regression and accuracy, precision,

recall, and the F1-measure.

6. Deployment: In this final phase, the model that was chosen in the previous step

will be deployed to the production environment. This requires a team effort from

data scientists, software developers, DevOps experts, and business professionals.

The following diagram shows the full cycle of the CRISP-DM process:

The standard process focuses on discovering insights and making interpretations in the

form of a story, while KDD focuses on data-driven pattern discovery and visualizing this.

SEMMA majorly focuses on model building tasks, while CRISP-DM focuses on business

understanding and deployment. Now that we know about some of the processes

surrounding data analysis, let's compare data analysis and data science to find out how

they are related, as well as what makes them different from one other.

Comparing data analysis and data science

Data analysis is the process in which data is explored in order to discover patterns that help

us make business decisions. It is one of the subdomains of data science. Data analysis

methods and tools are widely utilized in several business domains by business analysts,

data scientists, and researchers. Its main objective is to improve productivity and

profits. Data analysis extracts and queries data from different sources, performs exploratory

data analysis, visualizes data, prepares reports, and presents it to the business decisionmaking authorities.

On the other hand, data science is an interdisciplinary area that uses a scientific approach to

extract insights from structured and unstructured data. Data science is a union of all terms,

including data analytics, data mining, machine learning, and other related domains. Data

science is not only limited to exploratory data analysis and is used for developing models

and prediction algorithms such as stock price, weather, disease, fraud forecasts, and

recommendations such as movie, book, and music recommendations.

The roles of data analysts and data scientists

A data analyst collects, filters, processes, and applies the required statistical concepts to capture patterns, trends, and insights from data and prepare reports for making decisions.

The main objective of the data analyst is to help companies solve business problems using discovered patterns and trends. The data analyst also assesses the quality of the data and handles the issues concerning data acquisition. A data analyst should be proficient in writing SQL queries, finding patterns, using visualization tools, and using reporting tools Microsoft Power BI, IBM Cognos, Tableau, QlikView, Oracle BI, and more. Data scientists are more technical and mathematical than data analysts. Data scientists are research- and academic-oriented, whereas data analysts are more application-oriented. Data scientists are expected to predict a future event, whereas data analysts extract significant insights out of data. Data scientists develop their own questions, while data analysts find answers to given questions. Finally, data scientists focus on what is going to happen, whereas data analysts focus on what has happened so far. We can summarize these two roles using the following.

Features	Data Scientist	Data Analyst
Background	Predict future events and scenarios based on data	Discover meaningful insights from the data.
Role	Formulate questions that can profit the business	Solve the business questions to make decisions.

Type of data	Work on both structured and unstructured data	Only work on structured data
Programming	Advanced programming	Basic programming
Skillset	Knowledge of statistics, machine learning algorithms, NLP, and deep learning	Knowledge of statistics, SQL, and data visualization
Tools	R, Python, SAS, Hadoop, Spark, TensorFlow, and Keras	Excel, SQL, R, Tableau, and QlikView

Now that we know what defines a data analyst and data scientist, as well as how they are different from each other, let's have a look at the various skills that you would need to become one of them.

Test your knowledge on taking action with data

 A data analytics team works to recognize the current problem. Then, they organize available information to reveal gaps and opportunities. Finally, they identify the available options. These steps are part of what process?

1 / 1 point

Applying the SMART methodology

Using structured thinking

Making connections

Categorizing things

Correct

This describes structured thinking. Structured thinking begins with recognizing the current problem or situation. Next, information is organized to reveal gaps and opportunities. Finally, the available options are identified.

2.

Question 2

In which step of the data analysis process would an analyst ask questions such as, “What data errors might get in the way of my analysis?” or “How can I clean my data so the information I have is consistent?”

1 / 1 point

Analyze

Process

Prepare

Ask

Correct

An analyst asks questions such as, “What data errors might get in the way of my analysis?” or “How can I clean my data so the information I have is consistent?” during the process step. This is when data is cleaned in order to eliminate any possible errors, inaccuracies, or inconsistencies.

3.

Question 3

A data analyst has entered the analyze step of the data analysis process. Identify the questions they might ask during this phase. Select all that apply.

1 / 1 point

What story is my data telling me?

Correct

The analyze step involves thinking analytically about data. Data analysts might ask how the data can help them solve the problem and what story the data is trying to tell.

How will my data help me solve this problem?

Correct

The analyze step involves thinking analytically about data. Data analysts might ask how the data can help them solve the problem and what story the data is trying to tell.

How can I create an engaging presentation to stakeholders?

What is the question I’m trying to answer?

4.

Question 4

A data analyst is trying to understand their target audience. They’re asking questions such as, “How can learning more about my target audience help me figure out how to solve this problem?” and “What research do I need to do about my target audience?” The data analyst is in which phase of the data analysis process?

1 / 1 point

Ask

Act

Share

Prepare

Correct

The data analyst is in the prepare step. This is when analysts consider what information to gather and what research they can do to help problem-solve.

Weekly challenge 2

 

*Weekly challenge 2*

LATEST SUBMISSION GRADE

90%

1.

Question 1

Fill in the blank: Analytical skills are defined as _____.

1 / 1 point

qualities and characteristics associated with solving problems using facts

the management of people, processes, and tools

an analyst's intuition, inner voice, or gut instinct

the ability to break things down into smaller steps

Correct

Analytical skills are qualities and characteristics associated with solving problems using facts.

2.

Question 2

Seeking out new challenges and experiences in order to learn is an example of which analytical skill?

1 / 1 point

Having a technical mindset

Curiosity

Understanding context

Data strategy

Correct

Curious people seek out new challenges, which leads to knowledge.

3.

Question 3

Identifying the motivation behind data collection and gathering additional information are examples of which analytical skill?

0 / 1 point

A technical mindset

Understanding context

Data design

Data strategy

Incorrect

Review the section on analytical skills for a refresher.

4.

Question 4

Having a technical mindset is an analytical skill involving what?

1 / 1 point

Managing people, processes, and tools

Understanding the condition in which something exists or happens

Balancing roles and responsibilities

Breaking things down into smaller steps or pieces

Correct

Having a technical mindset involves the ability to break things down into smaller steps or pieces and work with them in an orderly and logical way.

5.

Question 5

Fill in the blank: Data strategy involves _____ the people, processes, and tools used in data analysis.

1 / 1 point

visualizing

choosing

managing

supervising

Correct

Data strategy is the management of the people, processes, and tools used in data analysis.

6.

Question 6

Fill in the blank: Being able to identify a relationship between two or more pieces of data describes _____.

1 / 1 point

correlation

problem-orientation

detail-oriented thinking

visualization

Correct

Fill in the blank: Being able to identify a relationship between two or more pieces of data describes correlation.

7.

Question 7

When executing a plan, a data analyst always double-checks their work, reads emails twice before sending them, and makes sure instructions are clear and correct the first time. They save their company time, money, and effort by paying attention to the specifics. This is an example of what?

1 / 1 point

Problem-solving

Strategy

Context

Detail-oriented thinking

Correct

Detail-oriented thinking is about all of the specifics that will help you execute a plan.

8.

Question 8

The five whys is a technique that involves asking, “Why?” five times in order to achieve what goal?

1 / 1 point

Use facts to guide business strategy

Visualize how a process should look in the future

Identify the root cause of a problem

Put a plan into action

Correct

In the five whys, you ask, “Why?” five times to reveal the root cause of a problem.

9.

Question 9

What method involves examining and evaluating how a process works currently in order to get it where you want it to be in the future?

1 / 1 point

Strategy

The five whys

Gap analysis

Data visualization

Correct

Gap analysis is a method for examining and evaluating how a process works currently in order to get where you want to be in the future.

10.

Question 10

Data-driven decision-making involves the five analytical skills: curiosity, understanding context, having a technical mindset, data design, and data strategy. Each plays a role in data-driven decision-making.

1 / 1 point

True

False

Correct

Data-driven decision-making involves curiosity, understanding context, having a technical mindset, data design, and data strategy.