Key technologies for big data processing generally include: big data collection, big data preprocessing, big data storage and management, big data analysis and mining, big data presentation and application (big data retrieval, big data visualization, Big data applications, big data security, etc.).
1. Big data collection technology
Data refers to various types of structured and Semi-structured (or weakly structured) and unstructured massive data are the foundation of the big data knowledge service model.
2. Big data preprocessing technology
Mainly complete operations such as analysis, extraction, and cleaning of received data. 1) Extraction: Because the acquired data may have multiple structures and types, the data extraction process can help us transform these complex data into a single or easy-to-process configuration to achieve rapid analysis and processing. 2) Cleaning: Not all big data is valuable. Some data are not what we care about, while other data are completely wrong interference items. Therefore, the data must be filtered and "denoised" to extract the Valid data.
3. Big data storage and management technology
Big data storage and management requires the use of memory to store the collected data, establish a corresponding database, and manage and call it. Focus on solving complex structured, semi-structured and unstructured big data management and processing technologies. It mainly solves several key issues such as the storage, representation, processing, reliability and effective transmission of big data.
4. Big data analysis and mining technology
Big data analysis technology. Improve existing data mining and machine learning technologies; develop new data mining technologies such as data network mining, specific group mining, and graph mining; break through big data fusion technologies such as object-based data connection and similarity connection; break through user interest analysis, network Domain-oriented big data mining technologies such as behavioral analysis and emotional semantic analysis.
6. Big data presentation and application technology
Big data technology can dig out the information and knowledge hidden in massive data, provide a basis for human social and economic activities, thereby improving The operational efficiency in various fields has greatly improved the intensification of the entire social economy. In our country, big data will be mainly used in the following three major areas: business intelligence, government decision-making, and public services. For example: business intelligence technology, government decision-making technology, telecommunications data information processing and mining technology, power grid data information processing and mining technology, meteorological information analysis technology, environmental monitoring technology, police cloud application system (road monitoring, video monitoring, network monitoring, Intelligent transportation, anti-telecom fraud, command and dispatch and other public security information systems) This answer is compiled based on Baidu library information. For the original text, please see "Big Data Key Technologies"