banner image

What is Big Data

Overview

Big Data is a field dedicated to the analysis, processing, and storage of large collections of data that frequently originate from disparate sources. Big Data solutions and practices are typically required when traditional data analysis, processing and storage technologies and techniques are insufficient. Specifically, Big Data addresses distinct requirements, such as the combining of multiple unrelated datasets, processing of large amounts of unstructured data and harvesting of hidden information in a time-sensitive manner.

Although Big Data may appear as a new discipline, it has been developing for years. The management and analysis of large datasets has been a long-standing problem—from labor-intensive approaches of early census efforts to the actuarial science behind the calculations of insurance premiums. Big Data science has evolved from these roots.

In addition to traditional analytic approaches based on statistics, Big Data adds newer techniques that leverage computational resources and approaches to execute analytic algorithms. This shift is important as datasets continue to become larger, more diverse, more complex and streaming-centric. While statistical approaches have been used to approximate measures of a population via sampling since Biblical times, advances in computational science have allowed the processing of entire datasets, making such sampling unnecessary.

The analysis of Big Data datasets is an interdisciplinary endeavor that blends mathematics, statistics, computer science and subject matter expertise. This mixture of skillsets and perspectives has led to some confusion as to what comprises the field of Big Data and its analysis, for the response one receives will be dependent upon the perspective of whoever is answering the question. The boundaries of what constitutes a Big Data problem are also changing due to the ever-shifting and advancing landscape of software and hardware technology. This is due to the fact that the definition of Big Data takes into account the impact of the data’s characteristics on the design of the solution environment itself. Thirty years ago, one gigabyte of data could amount to a Big Data problem and require special purpose computing resources. Now, gigabytes of data are commonplace and can be easily transmitted, processed and stored on consumer-oriented devices.

Data within Big Data environments generally accumulates from being amassed within the enterprise via applications, sensors and external sources. Data processed by a Big Data solution can be used by enterprise applications directly or can be fed into a data warehouse to enrich existing data there. The results obtained through the processing of Big Data can lead to a wide range of insights and benefits, such as:

  • Operational optimization
  • Actionable intelligence
  • Identification of new markets
  • Accurate predictions
  • Fault and fraud detection
  • More detailed records
  • Improved decision-making

Scientific discoveries Evidently, the applications and potential benefits of Big Data are broad. However, there are numerous issues that need to be considered when adopting Big Data analytics approaches. These issues need to be understood and weighed against anticipated benefits so that informed decisions and plans can be produced.

Concepts and Terminology

As a starting point, several fundamental concepts and terms need to be defined and understood.

Datasets

Collections or groups of related data are generally referred to as datasets. Each group or dataset member (datum) shares the same set of attributes or properties as others in the same dataset. Some examples of datasets are:

  • Tweets stored in a flat file
  • A collection of image files in a directory
  • An extract of rows from a database table stored in a csv formatted file
  • Historical weather observations that are stored as xml files

Data Analysis

Data analysis is the process of examining data to find facts, relationships, patterns, insights and/or trends. The overall goal of data analysis is to support better decision-making. A simple data analysis example is the analysis of ice cream sales data in order to determine how the number of ice cream cones sold is related to the daily temperature. The results of such an analysis would support decisions related to how much ice cream a store should order in relation to weather forecast information. Carrying out data analysis helps establish patterns and relationships among the data being analyzed

Data Analytics

Data analytics is a broader term that encompasses data analysis. Data analytics is a discipline that includes the management of the complete data lifecycle, which encompasses collecting, cleansing, organizing, storing, analyzing and governing data. The term includes the development of analysis methods, scientific techniques and automated tools. In Big Data environments, data analytics has developed methods that allow data analysis to occur through the use of highly scalable distributed technologies and frameworks that are capable of analyzing large volumes of data from different sources.

The Big Data analytics lifecycle generally involves identifying, procuring, preparing and analyzing large amounts of raw, unstructured data to extract meaningful information that can serve as an input for identifying patterns, enriching existing enterprise data and performing large-scale searches.

Different kinds of organizations use data analytics tools and techniques in different ways. Take, for example, these three sectors:

  • In business-oriented environments, data analytics results can lower operational costs and facilitate strategic decision-making.
  • In the scientific domain, data analytics can help identify the cause of a phenomenon to improve the accuracy of predictions.
  • In service-based environments like public sector organizations, data analytics can help strengthen the focus on delivering high-quality services by driving down costs.

Data analytics enable data-driven decision-making with scientific backing so that decisions can be based on factual data and not simply on past experience or intuition alone. There are four general categories of analytics that are distinguished by the results they produce:

  • Descriptive analytics
  • Diagnostic analytics
  • Predictive analytics
  • Prescriptive analytics

The different analytics types leverage different techniques and analysis algorithms. This implies that there may be varying data, storage and processing requirements to facilitate the delivery of multiple types of analytic results.

 

Descriptive Analytics

Descriptive analytics are carried out to answer questions about events that have already occurred. This form of analytics contextualizes data to generate information.

Sample questions can include:

  • What was the sales volume over the past 12 months?
  • What is the number of support calls received as categorized by severity and geographic location?
  • What is the monthly commission earned by each sales agent?

It is estimated that 80% of generated analytics results are descriptive in nature. Value-wise, descriptive analytics provide the least worth and require a relatively basic skillset.

Descriptive analytics are often carried out via ad-hoc reporting or dashboards.

The reports are generally static in nature and display historical data that is presented in the form of data grids or charts. Queries are executed on operational data stores from within an enterprise, for example a Customer Relationship Management system (CRM) or Enterprise Resource Planning (ERP) system.

Diagnostic Analytics

Diagnostic analytics aim to determine the cause of a phenomenon that occurred in the past using questions that focus on the reason behind the event. The goal of this type of analytics is to determine what information is related to the phenomenon in order to enable answering questions that seek to determine why something has occurred.

Such questions include:

  • Why were Q2 sales less than Q1 sales?
  • Why have there been more support calls originating from the Eastern region than from the Western region?
  • Why was there an increase in patient re-admission rates over the past three months?

Diagnostic analytics provide more value than descriptive analytics but require a more advanced skillset. Diagnostic analytics usually require collecting data from multiple sources and storing it in a structure that lends itself to performing drill-down and roll-up analysis. Diagnostic analytics results are viewed via interactive visualization tools that enable users to identify trends and patterns. The executed queries are more complex compared to those of descriptive analytics and are performed on multi-dimensional data held in analytic processing systems.

Predictive Analytics

Predictive analytics are carried out in an attempt to determine the outcome of an event that might occur in the future. With predictive analytics, information is enhanced with meaning to generate knowledge that conveys how that information is related. The strength and magnitude of the associations form the basis of models that are used to generate future predictions based upon past events. It is important to understand that the models used for predictive analytics have implicit dependencies on the conditions under which the past events occurred. If these underlying conditions change, then the models that make predictions need to be updated.

Questions are usually formulated using a what-if rationale, such as the following:

  • What are the chances that a customer will default on a loan if they have missed a monthly payment?
  • What will be the patient survival rate if Drug B is administered instead of Drug A?
  • If a customer has purchased Products A and B, what are the chances that they will also purchase Product C?

Predictive analytics try to predict the outcomes of events, and predictions are made based on patterns, trends and exceptions found in historical and current data. This can lead to the identification of both risks and opportunities.

This kind of analytics involves the use of large datasets comprised of internal and external data and various data analysis techniques. It provides greater value and requires a more advanced skillset than both descriptive and diagnostic analytics. The tools used generally abstract underlying statistical intricacies by providing user-friendly front-end interfaces.

Prescriptive Analytics

Prescriptive analytics build upon the results of predictive analytics by prescribing actions that should be taken. The focus is not only on which prescribed option is best to follow, but why. In other words, prescriptive analytics provide results that can be reasoned about because they embed elements of situational understanding. Thus, this kind of analytics can be used to gain an advantage or mitigate a risk.

Sample questions may include:

  • Among three drugs, which one provides the best results?
  • When is the best time to trade a particular stock?

Prescriptive analytics provide more value than any other type of analytics and correspondingly require the most advanced skillset, as well as specialized software and tools. Various outcomes are calculated, and the best course of action for each outcome is suggested. The approach shifts from explanatory to advisory and can include the simulation of various scenarios.

This sort of analytics incorporates internal data with external data. Internal data might include current and historical sales data, customer information, product data and business rules. External data may include social media data, weather forecasts and government-produced demographic data. Prescriptive analytics involve the use of business rules and large amounts of internal and external data to simulate outcomes and prescribe the best course of action.

Business Intelligence (BI)

BI enables an organization to gain insight into the performance of an enterprise by analyzing data generated by its business processes and information systems. The results of the analysis can be used by management to steer the business in an effort to correct detected issues or otherwise enhance organizational performance. BI applies analytics to large amounts of data across the enterprise, which has typically been consolidated into an enterprise data warehouse to run analytical queries. The output of BI can be surfaced to a dashboard that allows managers to access and analyze the results and potentially refine the analytic queries to further explore the data.

Key Performance Indicators (KPI)

A KPI is a metric that can be used to gauge success within a particular business context. KPIs are linked with an enterprise’s overall strategic goals and objectives. They are often used to identify business performance problems and demonstrate regulatory compliance. KPIs therefore act as quantifiable reference points for measuring a specific aspect of a business’ overall performance. KPIs are often displayed via a KPI dashboard. The dashboard consolidates the display of multiple KPIs and compares the actual measurements with threshold values that define the acceptable value range of the KPI.

 

What is Big Data What is Big Data Reviewed by Car Insurance Expert on June 10, 2018 Rating: 5
Powered by Blogger.