In recent years, with the rapid development and popularization of computer and information technology, the scale of industrial application system has expanded rapidly, and the data generated by industrial application has exploded. Industry/enterprise big data often reaches the scale of hundreds of TB or even tens to hundreds of PB, which has far exceeded the processing capacity of existing traditional computing technologies and information systems. Therefore, it has become an urgent need to seek effective big data processing technologies, methods and means in the real world. At present, the total data volume of Baidu has exceeded 1000 Pb, and the web page data that needs to be processed every day reaches10pb ~100pb; The accumulated transaction data of Taobao is as high as100 Pb; Twitter publishes more than 200 million messages every day, and Sina Weibo publishes 80 million messages every day; The telephone communication record data of a province of China Mobile can reach 0.5 Pb ~1Pb per month; The monitoring data of road vehicles in the public security bureau of a provincial capital city can reach 20 billion in three years, with a total amount of 120TB. According to the research report of IDC, an authoritative IT information consulting and analysis company in the world, in the future,1zb =10000bb = 100000 PB), the global data volume will increase from 0.8ZB in 2009 to 35 ZB in 2020, and100000pb will increase by 40.
A few years ago, people called large-scale data "massive data", but in fact, the concept of big data was put forward as early as 2008. In 2008, on the occasion of Google's 10 anniversary, the famous Nature magazine published a special issue to discuss a series of technical problems and challenges related to big data processing in the future, in which the concept of "big data" was put forward.
With the popularization of the concept of big data, people often ask, how big is big data? In fact, it is difficult to have a very quantitative definition of big data. Wikipedia gives a qualitative description: Big data refers to data sets that cannot be obtained, managed and processed by using traditional and commonly used software technologies and tools in a certain period of time. Furthermore, the focus of the word "big data" today is not only the definition of data scale, but also represents that the development of information technology has entered a new era, representing the technical challenges and difficulties brought by explosive data information to traditional computing technology and information technology, representing new technologies and methods needed for big data processing, and also representing new inventions, new services and new development opportunities brought by big data analysis and application.