Concept Drift Detection Based
on Kolmogorov–Smirnov Test
Zhixiong Wang and Wei Wang
(&)
Tianjin Key Laboratory of Wireless Mobile Communications and Power
Transmission, Tianjin Normal University, Tianjin 300387, China
weiwang@tjnu.edu.cn
Abstract. With the advancement of information society, a large amount of
data, which is in the form of stream, has been produced in many fields. As a
result of its extensive application in the fields of sensor networks, banking and
telecommunications, data stream mining is obtaining more attention. One of the
most challenging steps to learn from data stream is to react to concept drift, as
most of the existing data stream algorithms only deal with abrupt or gradual
concept drifts. The existing work of detecting concept drift is mostly based on
the changing of error rate of single window, making it difficult to be universally
applied to different types of concept drifts. A method of detecting concept drift
is proposed in this paper based on Kolmogorov–Smirnov test (K–S test).
Keywords: Data stream Á Concept drift Á Kolmogorov–Smirnov test
1 Introduction
With the development of economy and society, a large amount of data, which is in the
form of stream, has been continuously produced in fields such as financial data analysis, network monitoring, sensor network data processing, credit card fraud monitoring,
weather forecasting and electricity price forecasting. This fast-reaching, real-time,
continuous, and unbounded data sequence is called data stream. It makes data classification, is based on data stream system, which becomes a research focus. On one hand,
because it’s fast, infinite, and unreproducible, data mining technology in a relatively
mature and static system can’t be directly transferred to data stream system. On the
other hand, due to the data stream system and application, the concepts in data stream
are dynamically changing or even full of noise; as a result, the distribution of data will
change with time, which aggravates the complexity of data stream classification. For
example, the rules of weather forecast may change with seasons; Methods of Customer
online shopping preference analysis may change with customer groups, seller reputation, service type, and other factors; industrial electricity consumption will change
periodically as seasons alternate. Therefore, research on detecting concept drift is being
become more and more important in data streams. According to the changing speed, we
can subdivide concept drift into gradual drift and abrupt drift. If the data distribution in
data stream is suddenly replaced by another completely different data distribution in a
relatively short period of time, then we can think that abrupt concept drift occurs.
Gradual concept drift, however, can only be observed after a rather long period of time
© Springer Nature Singapore Pte Ltd. 2020
Q. Liang et al. (Eds.): Artificial Intelligence in China, LNEE 572, pp. 273–280, 2020.
https://doi.org/10.1007/978-981-15-0187-6_31
on Kolmogorov–Smirnov Test
Zhixiong Wang and Wei Wang
(&)
Tianjin Key Laboratory of Wireless Mobile Communications and Power
Transmission, Tianjin Normal University, Tianjin 300387, China
weiwang@tjnu.edu.cn
Abstract. With the advancement of information society, a large amount of
data, which is in the form of stream, has been produced in many fields. As a
result of its extensive application in the fields of sensor networks, banking and
telecommunications, data stream mining is obtaining more attention. One of the
most challenging steps to learn from data stream is to react to concept drift, as
most of the existing data stream algorithms only deal with abrupt or gradual
concept drifts. The existing work of detecting concept drift is mostly based on
the changing of error rate of single window, making it difficult to be universally
applied to different types of concept drifts. A method of detecting concept drift
is proposed in this paper based on Kolmogorov–Smirnov test (K–S test).
Keywords: Data stream Á Concept drift Á Kolmogorov–Smirnov test
1 Introduction
With the development of economy and society, a large amount of data, which is in the
form of stream, has been continuously produced in fields such as financial data analysis, network monitoring, sensor network data processing, credit card fraud monitoring,
weather forecasting and electricity price forecasting. This fast-reaching, real-time,
continuous, and unbounded data sequence is called data stream. It makes data classification, is based on data stream system, which becomes a research focus. On one hand,
because it’s fast, infinite, and unreproducible, data mining technology in a relatively
mature and static system can’t be directly transferred to data stream system. On the
other hand, due to the data stream system and application, the concepts in data stream
are dynamically changing or even full of noise; as a result, the distribution of data will
change with time, which aggravates the complexity of data stream classification. For
example, the rules of weather forecast may change with seasons; Methods of Customer
online shopping preference analysis may change with customer groups, seller reputation, service type, and other factors; industrial electricity consumption will change
periodically as seasons alternate. Therefore, research on detecting concept drift is being
become more and more important in data streams. According to the changing speed, we
can subdivide concept drift into gradual drift and abrupt drift. If the data distribution in
data stream is suddenly replaced by another completely different data distribution in a
relatively short period of time, then we can think that abrupt concept drift occurs.
Gradual concept drift, however, can only be observed after a rather long period of time
© Springer Nature Singapore Pte Ltd. 2020
Q. Liang et al. (Eds.): Artificial Intelligence in China, LNEE 572, pp. 273–280, 2020.
https://doi.org/10.1007/978-981-15-0187-6_31