1. 程式人生 > >Quiet log noise with Python and machine learning

Quiet log noise with Python and machine learning

Continuous integration (CI) jobs can generate massive volumes of data. When a job fails, figuring out what went wrong can be a tedious process that involves investigating logs to discover the root cause--which is often found in a fraction of the total job output. To make it easier to separate the most relevant data from the rest, the Logreduce machine learning model is trained using previous successful job runs to extract anomalies from failed runs' logs. This principle can also be applied to other use cases, for example, extracting anomalies from Journald or other systemwide regular log files. A typical log file contains many nominal events ("baselines") along with a few exceptions that are relevant to the developer.