10- Big Data

You may be wondering about the term BIG DATA which you may have heard by many technology freaks and some other peoples too. Is it really the topic that a common man could understand in a simple concept? Well, I think it is like introducing the child to sea and explaining its depth, size and expect him to understand which he would better understand when he himself experienced it.

The messages that we send, the pics and videos we store, the other information related to work like excel sheets, word documents, etc are referred as DATA. The combination of such data's under some special instructions is called as data sets.


Big data is data sets that are so voluminous and complex that traditional data-processing application software is inadequate to deal with them. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy and data source. There are five concepts associated with big data: volume, variety, velocity, veracity and value. It can be explained below:

Volume:

The quantity of generated and stored data. The size of the data determines the value and potential insight, and whether it can be considered big data or not.

Variety

The type and nature of the data. This helps people who analyze it to effectively use the resulting insight. Big data draws from text, images, audio, video; plus it completes missing pieces through data fusion.

Velocity

In this context, the speed at which the data is generated and processed to meet the demands and challenges that lie in the path of growth and development. Big data is often available in real-time.

Variability

Inconsistency of the data set can hamper processes to handle and manage it.

Veracity

The data quality of captured data can vary greatly, affecting the accurate analysis.

Big data is an old concept which is used today at the larger scale and seems simple while explaining but is actually a very complex and proves a programmer a lot of challenges for arranging, sorting and replacing certain data. Even though machine learning(artificial intelligence) performs these tasks, it needs to be instructed by the programmer at a certain level because machine learns from experience and corrections made during its execution.

Big data repositories have existed in many forms, often built by corporations with a special need. Commercial vendors historically offered parallel database management systems for big data beginning in the 1990s. For many years, WinterCorp published the largest database report.

Let's try to understand what actually big data is!
In the late 60's, the computers of a size of some kilobyte were introduced where people would work with some data. Now, many countries were introduced to the computers and people used it to its maximum capacity. As the time flew, there was a need for sharing the works to develop efficient solutions, which led to an introduction of the Internet.

The language barrier was a problem with sharing and needed some solution. There was a need for some centre which would store those data, analyze it and convert it to computer language and then further process it to the regional languages hence leading to the concept of BIG DATA.

It is nothing but a machine instructed with some primary instructions and workings for some tasks. When data is passed to it, it checks the available processes to perform and in case of unavailability of the process, it asks for the solutions to the programmer.

The photos, videos or any file you upload over cloud storage is processed, converted to the zip file and labelled it with some tags by the machine and when you access the data, you actually access the tags which show you the processed output data. As far as we are concerned about a simple user, the big data has no direct connection with the user and it clearly needs the virtuoso mind to give solutions to a machine.