Scalable change and anomaly detection in cross-correlated data

Abstract

Both in science and industry, the sizes of data sets are growing. It is not uncommon to encounter sets containing millions or even billions of measurements. Without appropriate tools for turning such enormous amounts of data into insight, however, the data’s value is severely limited. Apart from consisting of many measurements, a common feature of big data sets is that some properties of the data change over time. Determining whether and when changes have taken place is important in many scientific problems. For example: Is the climate changing? Has the covid-19 reproduction number changed? Is the quality of manufactured cars stable? Moreover, monitoring changes in network traffic data can be used to detect cyber attacks. Therefore, in this thesis, I have studied statistical methods for detecting changes and estimating when they have occurred. My collaborators and I have constructed efficient computer programs both for retrospective analysis of large data sets as well as for real-time analysis of streaming data. We have also demonstrated that detecting changes in a stream of data from temperature sensors could have prevented a costly and dangerous overheating event in a ship motor.