DBSCAN algorithm is one of the most typical representative algorithms in this
kind of methods. Its core idea is to find the points with higher density first, then connect
the similar high-density points step by step, and then generate various clustering.
3.2 Clustering Ensemble
In the process of ensembling, the result is easy to be affected by low-quality clustering.
In order to solve this problem, this paper proposes to evaluate the reliability of each
base cluster based on Gini coefficient and consider the internal structure of the cluster,
instead of simply treating each cluster as an indivisible individual, so as to give them
weight to achieve the goal of improving the credibility of consistent clustering. Finally,
a consistent result is obtained by combining the hierarchical clustering method.
3.2.1 Gini Coefficient
Gini coefficient reflects the probability that two samples are randomly sampled from the
sample set. It is a kind of index that can be used to show the purity of data. The smaller
the Gini coefficient is, the purer the representative data is, that is to say, the higher the
certainty of classification. It is defined as:
GiniðpÞ ¼
X K
k¼1
p k ð1 À p k Þ ¼ 1 À
X K
k¼1
p
2
k
ð1Þ
where K denotes the number of classification, the probability of object belongs to the
kth category is p k .
3.2.2 Formulation of the Ensemble Clustering Problem
In this section, we introduce the formulation of the clustering ensemble problem. Let
D ¼ fd 1 ; d 2 ; . . .; d N g be the dataset, where d i is the ith data and N is the number of
data. For the dataset D, there are M partitions, each of which is treated as a base
clustering and consists of a certain number of clusters. It can be defined as:
P ¼ fp
1
; p
2
; . . .; p
M
g
ð 2Þ
where
p
m
¼ fC
m
1 ; . . .; C
m
k g
ð 3Þ
denotes the mth base clustering in P; C
m
i and denotes the ith cluster in p
m . Obviously,
the union of all clusters in the same base clustering covers the entire data set. Let
F
m
ðd i Þ denote the cluster in p
m that object d i belongs to. That is, if d i belongs to the kth
cluster in p
m , i.e., d i 2 C
m
k , then we have F
m
ðd i Þ ¼ C
m
k . For convenience, we present
the set of all clusters in the ensemble P as r ¼ fC 1 ; . . .; C n g.
3.2.3 Reliability of Clusters
As introduced in Sect. 3.2.1, Gini coefficient is a measure of uncertainty associated
with a random variable. Each cluster is a set of data objects. Given a cluster C i does not
Research on Chinese Short Text Clustering …
625
kind of methods. Its core idea is to find the points with higher density first, then connect
the similar high-density points step by step, and then generate various clustering.
3.2 Clustering Ensemble
In the process of ensembling, the result is easy to be affected by low-quality clustering.
In order to solve this problem, this paper proposes to evaluate the reliability of each
base cluster based on Gini coefficient and consider the internal structure of the cluster,
instead of simply treating each cluster as an indivisible individual, so as to give them
weight to achieve the goal of improving the credibility of consistent clustering. Finally,
a consistent result is obtained by combining the hierarchical clustering method.
3.2.1 Gini Coefficient
Gini coefficient reflects the probability that two samples are randomly sampled from the
sample set. It is a kind of index that can be used to show the purity of data. The smaller
the Gini coefficient is, the purer the representative data is, that is to say, the higher the
certainty of classification. It is defined as:
GiniðpÞ ¼
X K
k¼1
p k ð1 À p k Þ ¼ 1 À
X K
k¼1
p
2
k
ð1Þ
where K denotes the number of classification, the probability of object belongs to the
kth category is p k .
3.2.2 Formulation of the Ensemble Clustering Problem
In this section, we introduce the formulation of the clustering ensemble problem. Let
D ¼ fd 1 ; d 2 ; . . .; d N g be the dataset, where d i is the ith data and N is the number of
data. For the dataset D, there are M partitions, each of which is treated as a base
clustering and consists of a certain number of clusters. It can be defined as:
P ¼ fp
1
; p
2
; . . .; p
M
g
ð 2Þ
where
p
m
¼ fC
m
1 ; . . .; C
m
k g
ð 3Þ
denotes the mth base clustering in P; C
m
i and denotes the ith cluster in p
m . Obviously,
the union of all clusters in the same base clustering covers the entire data set. Let
F
m
ðd i Þ denote the cluster in p
m that object d i belongs to. That is, if d i belongs to the kth
cluster in p
m , i.e., d i 2 C
m
k , then we have F
m
ðd i Þ ¼ C
m
k . For convenience, we present
the set of all clusters in the ensemble P as r ¼ fC 1 ; . . .; C n g.
3.2.3 Reliability of Clusters
As introduced in Sect. 3.2.1, Gini coefficient is a measure of uncertainty associated
with a random variable. Each cluster is a set of data objects. Given a cluster C i does not
Research on Chinese Short Text Clustering …
625