Research on Hadoop-based massive short text clustering algorithm

Qiang Zhao; Yuliang Shi; Zepeng Qing

doi:10.1117/12.2540380

31 July 2019 Research on Hadoop-based massive short text clustering algorithm

Qiang Zhao, Yuliang Shi, Zepeng Qing

Proceedings Volume 11198, Fourth International Workshop on Pattern Recognition; 111980A (2019) https://doi.org/10.1117/12.2540380
Event: Fourth International Workshop on Pattern Recognition, 2019, Nanjing, China

Abstract

Many clustering algorithms work well on small data sets of less than 200 data objects. However, a large database may contain millions of objects, and clustering on such a large data set may lead to biased results. As data volumes and availability continue to grow, so does the need for large dataset analytics. Among the most commonly used clustering algorithms, K-means proved to be one of the most popular choices to provide acceptable results in a reasonable amount of time. In this paper, we present an improved k-means algorithm with better initial centroids. Also, we implement this modified algorithm on Hadoop platform. Experiments show that the improved k-means algorithm converges faster than the classic k-means and the average execution time is reduced compared to the traditional k-means.

Citation Download Citation

Qiang Zhao, Yuliang Shi, and Zepeng Qing "Research on Hadoop-based massive short text clustering algorithm", Proc. SPIE 11198, Fourth International Workshop on Pattern Recognition, 111980A (31 July 2019); https://doi.org/10.1117/12.2540380

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available