What is the difference between the functions of a big data engineer and a big data development engineer?

There is no difference between big data engineers and big data development engineers. A big data engineer refers to a big data development engineer. Big data engineers (that is, big data development engineers) are engaged in technical research such as big data collection, cleaning, analysis, governance and mining, and use, manage, maintain and serve them.

The functions of a big data engineer (that is, a big data development engineer) are as follows:

1, big data collection (crawler), big data cleaning (ETL engineer), big data modeling (algorithm engineer), big data analysis (data analyst).

2. Technologies such as management, analysis, presentation and application (big data development engineer).

3. Research and apply the architecture, technology and standards of big data platform.

4. Design, develop, integrate and test big data software and hardware systems.

5. Manage, maintain and ensure the stable operation of big data systems.

6. Monitor, manage and guarantee the security of big data.

7. Provide technical consultation and technical services for big data.

Extended data:

Skills requirements of big data engineers (that is, big data development engineers):

1, proficient in Java technology knowledge, familiar with the design and development of Spark, kafka, Hive, HBase, zookeeper, HDFS, MR and other applications.

2. Understand scripting languages such as python/shell.

3. Familiar with the architecture of big data platform, and have a certain understanding of ETL and data warehouse.

4. Experience in data visualization, data analysis and mathematical modeling is preferred.

5. Experience in developing crawler system is preferred.