Research on flink streaming data processing and MySQL storage scheme

Dianhui Wang; Rui Liu; Tao Sun; Jianyu Guo; Meng Li; Zhuangzhuang Tian; Li Jianyong; Chao Zhang

doi:10.1117/12.2652754

10 November 2022 Research on flink streaming data processing and MySQL storage scheme

Dianhui Wang, Rui Liu, Tao Sun, Jianyu Guo, Meng Li, Zhuangzhuang Tian, Li Jianyong, Chao Zhang

Proceedings Volume 12331, International Conference on Mechanisms and Robotics (ICMAR 2022); 123313B (2022) https://doi.org/10.1117/12.2652754
Event: International Conference on Mechanisms and Robotics (ICMAR 2022), 2022, Zhuhai, China

Abstract

This article introduces the research process and solution of Flink stream process data and storing parsed data into MySQL, which means that the one-stop process of obtaining, analyzing and storing is realized in the field of vehicle big data. The algorithm obtains real-time information from the target (Kafka) through Flink, then passes the information to parsing program, and finally imports the parsing result into MySQL. Since the MySQL is designed as a storage method of sub-databases and tables, by setting the time period and then landing the data in HBase, we don’t need to worry about the storage performance of MySQL when the amount of data is large.

Citation Download Citation

Dianhui Wang, Rui Liu, Tao Sun, Jianyu Guo, Meng Li, Zhuangzhuang Tian, Li Jianyong, and Chao Zhang "Research on flink streaming data processing and MySQL storage scheme", Proc. SPIE 12331, International Conference on Mechanisms and Robotics (ICMAR 2022), 123313B (10 November 2022); https://doi.org/10.1117/12.2652754

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available