A Framework for a Network Anomaly Detection Architecture

1.1. Motivation

When operational or configurational changes in connectivity services are happening, it is crucial for network operators to detect interruptions within the network faster than the users utilizing the connectivity services.¶

In order to achieve this objective, automation in network monitoring is required. The amount of people operating the network are today simply outnumbered by the amount of people utilizing connectivity services.¶

This automation needs to monitor network changes holistically by supervising all 3 network planes simultaneously for a given connectivity service on the OSI (Open Systems Interconnection) layer 3. The monitoring system needs to detect whether configurational or operational State changes, an interface was shutdown by an operator versus an interface State went down due to loss of signal on the optical layer and wherever it disrupted the service, e.g. the received packets from customers are no longer forwarded to the desired destination, or not.¶

Management plane relates to network node entities. Where control plane in turn propagates a subset o the management plane entities, the path reachability, to its neighboring network nodes accross the network. The forwarding plane requires a previously converged network topology and received packets to export metrics.¶

A State change in control and management plane which are related to each other indicate a network topology State change while a State change in the forwarding plane describes how the packets are being forwarded. In other words, control and management plane State changes can be attributed to network topology State changes whereas forwarding plane State changes are related to the outcome of these network topology State changes.¶

Since changes in networks are happening all the time due to the vast number of dependencies, most of the changes are not negatively affecting the end to end connectivity due to redundancies in networks, a scoring system is needed to indicate how disruptive the change is considered. The scoring system needs to take into account the amount of transport sessions, the amount of affected flows and whether the detected interruptions are usual or exceptional.¶

1.2. Scope

Such objectives can be achieved by applying checks on network modeled time series data that contains semantics describing their dependencies across network planes. These checks can be based on domain knowledge or using outlier detection techniques. Domain-knowledge-based techniques applies the expertise of network engineers operating a network to understand whether there is an issue impacting the customer or not. On the other hand, outlier detection techniques identify measurements that deviate significantly from the norm and therefore are considered anomalous.¶

The described scope does not take the connectivity service intent into account nor does it verify whether the intent is being achieved all the time. Changes to the service intent causing service disruptions are therefore considered service disruptions. On monitoring systems which take the intent into account, this is considered as intended.¶

Also out of scope of this document are a gradual degredation of a connectivity service over a long period of time. An example would be optical fiber degredation which lead to malform packets on IP layer and therefore increases packet drops steadily. Outlier detection techniques can be applied here as well but instead of taking the network model, the component type and characterstics would be taken into context.¶

2.1. Terminology

This document defines the following terms:¶

Outlier Detection: Is a systematic approach to identify rare data points deviating significantly from the majority.¶

Service Disruption Detection (SDD): The process of detecting a service degradation by discovering outliers in network monitoring data.¶

Service Disruption Detection System (SDDS): A system allowing to perform SDD.¶

Rules: Refers to rules defined by domain experts or artificial intelligence in context of detection strategies. See Section 3.5.1.1 for details on domain expert rules.¶

Additionally it makes use of the terms defined in [I-D.ietf-nmop-terminology], [I-D.ietf-nmop-network-anomaly-lifecycle] and [RFC8969].¶

The following terms are used as defined in [I-D.ietf-nmop-terminology] :¶

• Resource¶

• Event¶

• State¶

• Relevance¶

• Problem¶

• Symptom¶

• Alarm¶

Figure 2 in Section 3 of [I-D.ietf-nmop-terminology] shows characteristics of observed operational network telemetry metrics.¶

Figure 4 in Section 3 of [I-D.ietf-nmop-terminology] shows relationships between, state, relevant state, problem, symptom, cause and alarm.¶

Figure 5 in Section 3 of [I-D.ietf-nmop-terminology] shows relationships between problem, symptom and cause.¶

The following terms are used as defined in [I-D.ietf-nmop-network-anomaly-lifecycle] :¶

• False Positive¶

• False Negative¶

• Confidence Score¶

• Concern Score¶

The following terms are used as defined in [RFC8969] :¶

• Service Model¶

2.2. Outlier Detection

Outlier Detection, also known as anomaly detection, describes a systematic approach to identify rare data points deviating significantly from the majority. Outliers can manifest as single data point or as a sequence of data points. There are multiple ways in general to classify anomalies, but for the context of this document, the following three classes are taken into account:¶

Global outliers:

An outlier is considered "global" if its behavior is outside the entirety of the considered data set. For example, if the average dropped packet count is between 0 and 10 per minute and, in a small time-window, the value gets to 1000, this data point is considered a global anomaly.¶

Contextual outliers:

An outlier is considered "contextual" if its behavior is within a normal (expected) range, but it would not be expected based on some context. Context can be defined as a function of multiple parameters, such as time, location, etc. An example of a contextual outlier is when the forwarded packet volume overnight reaches levels which might be totally normal for the daytime, but anomalous and unexpected for the nighttime.¶

Collective outliers:

An outlier is considered "collective" if the behavior of each single data point that are part of the anomaly are within expected ranges (so they are not anomalous in either a contextual or a global sense), but the group, taking all the data points together, is. Note that the group can be made within a single time series (a sequence of data points is anomalous) or across multiple types of metrics (e.g. if looking at two metrics together, the combined behavior turns out to be anomalous). In Network Telemetry time series, one way this can manifest is that the amount of network paths and interface State changes matches the time range when the forwarded packet volume decreases as a group.¶

For each outlier a Confidence and a Concern Score between 0 and 1 is being calculated. The higher the Confidence Score value, the higher the probability that the observed data point is an outlier. The higher the Concern Score value, the higher the probability that observed outlier is impacting the forwarding of the customer packets negatively. Combined together raising the Relevance of the observed events. Anomaly detection: A survey [VAP09] provides and discusses an overview on different anomaly detection techniques and the outlier detection approach adopted by each.¶

2.3. Knowledge Based Detection

Knowledge-based anomaly detection, a superset of rule-based anomaly detection and a subset of semantic-based, Knowledge-based anomaly detection: Survey, challenges, and future directions [ASNL25], is a technique used to identify anomalies or outliers by comparing them against predefined rules or patterns. This approach relies on the use of domain-specific knowledge to set standards, thresholds, or rules for what is considered "normal" behavior. Traditionally, these rules are established manually by a knowledgeable network engineer. Forward-looking, these rules can be expressed using human and machine readable network protocol derived Symptoms and patterns defined in ontologies.¶

Additionally, in the context of network anomaly detection, the knowledge-based approach works hand in hand with the deterministic understanding of the network, which is reflected in network modeling. Components are organized into three network planes: the Management Plane, the Control Plane, and the Forwarding Plane [RFC9232]. A component can relate to a physical, virtual, or configurational entity, or to a sum of packets belonging to a flow being forwarded in a network.¶

Such relationships can be modelled in Service and Infrastructure Maps (SIMAP) to automate that process. [I-D.ietf-nmop-simap-concept] defines the concepts for the SIMAP and [I-D.havel-nmop-digital-map] defines an application of the SIMAP to network topologies.¶

These relationships can also be modeled in Knowledge Graphs Section 5 of [I-D.mackey-nmop-kg-for-netops] using semantic triples [W3C-RDF-concept-triples], where with ontologies, due to its declarative form, those semantic triples are machine and human readable. See Section 2.5.2 as an example for an ontology describing symptoms.¶

2.4. Machine Learning

Machine learning is commonly used for detecting outliers or anomalies. Typically, unsupervised learning is widely recognized for its applicability, given the inherent characteristics of network data. See [VAP09]. Although machine learning requires a sizeable amount of high-quality data and considerable advanced training, the advantages it offers make these requirements worthwhile. The power of this approach lies in its generalizability, robustness, ability to simplify the fine-tuning process, and most importantly, its capability to identify anomaly patterns that might go unnoticed to the human observer.¶

2.5. Data Mesh

The Data Mesh [Deh22] Architecture distinguishes between operational and analytical data. Operational data refers to collected data from operational systems. While analytical data refers to insights gained from operational data.¶

3.1. Service Inventory

A service inventory, (1) in Figure 1, is used to obtain a list of the connectivity services for which Anomaly Detection is to be performed. A service profiling process may be executed on the operational network data of the service in order to define a configuration of the service disruption detection approach and parameters to be used.¶

3.2. Service Disruption Detection Configuration

Based on this service list and potential preliminary service profiling, a configuration of the Service Disruption Detection, (2) in Figure 1, is produced. It defines the set of approaches that need to be applied to perform SDD, as well as parameters, grouped in templates, that are to be set when executing the algorithms performing SDD per se.¶

As the service lives on, the configuration may be adapted, (3) in Figure 1, as a result of an evolution of the profiling being performed. Postmortem analysis are produced as a result of Events impacting the service, or the occurrence of false positives raised by the Alarm system. These postmortem analysis can improve the deployed profiles parameters and creation of new customer profiles. See upcoming section Section 3.5.1.3 for details on profiling.¶

3.3. Operational Data Collection

Collection of network monitoring data, (4) in Figure 1, involves the management of the subscriptions to network telemetry on nodes of the network, and the configuration of the collection infrastructure to receive the monitoring data produced by the network.¶

The monitoring data produced by the collection infrastructure is then streamed through a message broker system, for further processing.¶

Networks tend to produce extremely large amounts of monitoring data. To preserve scaling and reduce costs, decisions need to be made on the duration of retention of such data in storage, and at which level of storage they need to be kept. A retention time need to be set on the raw data produced by the collection system, in accordance to their utility for further used. This aspect will be elaborated in further sections.¶

3.4. Operational Data Aggregation

Aggregation, (5) in Figure 1, is the process of producing data sets based on collected network monitoring data upon which detection of a service disruption can be performed by filtering or aggregating.¶

Pre-processing of collected network monitoring data is usually performed to reduce input for the Service Disruption Detection component since not all metrics are relevant for this use case. This can be achieved in multiple ways, depending on the architecture of the SDD component. As an example, the granularity or cardinality at which forwarding plane data is produced by the network may be too high for the SDD algorithms, and instead be aggregated into a coarser dimension for SDD execution.¶

A retention time for the operational data needs to be decided on Aggregated data and should reflect the expected further use. As example, the retention time must be set in accordance with the replay ability requirement discussed in Section 3.8.¶

3.5. Service Disruption Detection

Service Disruption Detection processes, (6) in Figure 1, decide whether a service might be degraded to the point where network operation needs to be alerted of an ongoing Problem within the network.¶

Two key aspects need to be considered when designing the SDD component. First, the way the data is being processed needs to be carefully designed, as networks typically produce extremely large amounts of data which may hinder the scalability of the architecture. Second, the algorithms used to make a decision to alert the operator need to be designed in such a way that the operator can trust that a targeted Service Disruption will be detected (no false negatives), while not spamming the operator with Alarms that do not reflect an actual issue within the network (false positives) leading to Alarm fatigue.¶

Two approaches are typically followed to present the data to the SDD system. Classically, the aggregated data can be stored in a database that is polled at regular intervals by the SDD component for decision making. Alternatively, a streaming approach can be followed so as to process the data while they are being consumed from the collection component.¶

For SDD per-se, two families of algorithms can be decided upon. First, knowledge based detection approaches can be used, mimicking the process that human operators follow when looking at the data. Second, Machine Learning based approaches to detect outliers based from prior trained operational network data.¶

3.6. Alarm

When the SDD component decides that a service is undergoing a disruption, an aggregated relevant-state change notification, taking the output of multiple Service Disruption Detection processes into account, MUST be sent to the Alarm and Problem management system as shown in Figure 4 in Section 3 of [I-D.ietf-nmop-terminology] and (11) in Figure 1. Multiple practical aspects need to be taken into account in this component.¶

When the issue lasts longer than the interval at which the SDD component runs, the relevant-state change mechanism should not create multiple notifications to the operator, so as to not overwhelm the management of the issue. However, the information provided along with the Alarm should be kept up to date during the full duration of the issue.¶

3.7. Postmortem

   Network Anomaly
     Detection             Symptoms
+-------------------+         &
|   +-----------+   | Network Anomalies
|   | Detection |---|---------+
|   |   Stage   |   |         |
|   +-----------+   |         v
+---------^---------+    +-------------------+   Labels  +------------+
          |              | Anomaly Detection |---------->| Validation |
          |              |   Label Store     |<----------|   Stage    |
          |              +-------------------+  Revised  +------------+
   +------------+             |                 Labels
   | Refinement |             |
   |   Stage    |<------------+
   +------------+    Historical Symptoms
                              &
                      Network Anomalies

Validation and refinement are performed during Postmortem analysis, (12) in Figure 1.¶

From an Anomaly Detection Lifecycle point of view, as described in [I-D.ietf-nmop-network-anomaly-lifecycle], the Service Disruption Detection Configuration evolves over time, iteratively, looping over three main phases: detection, validation and refinement.¶

The Detection phase produces the Alarms that are sent to the Alarm and Problem Management System and at the same time it stores the network anomaly and Symptom labels into the Label Store. This enables network engineers to review the labels to validate and edit them as needed.¶

The Validation stage is typically performed by network engineers reviewing the results of the detection and indicating which Symptoms and network anomalies have been useful for the identification of Problems in the network. The original labels from the Service Disruption Detection are analyzed and an updated set of more accurate labels is provided back to the label store for version-control.¶

The resulting labels will be then provided back into the Network Anomaly Detection via its refinement capabilities: the refinement is about the update of the Service Disruption Detection configuration in order to improve the results of the detection (e.g. false positives, false negatives, accuracy of the boundaries, etc.).¶

3.8. Replaying

When a service disruption has been detected, it is essential for the human operator to be able to analyze the data which led to the raising of an Alarm. It is thus important that a SDDS preserves both the data which led to the creation of the Alarm as well as human understandable information on why the data led to the raising of an Alarm.¶

In early stages of operations or when experimenting with a SDDS, it is common that the parameters used for SDD are to be fined tuned. This process is facilitated by designing the SDDS architecture in a way that allows to rerun the SDD algorithms on the same input.¶

Data retention, as well as its level, need to be defined in order not to sacrifice the ability of replaying SDD execution for the sake of improving its accuracy.¶

4.1. Cosmos Bright Lights

This architecture have been developed as part of a proof of concept started in September 2022 first in a dedicated network lab environment and later in December 2022 in Swisscom production to monitor a limited amount of 16 L3 VPN connectivity services.¶

At the Applied Networking Research Workshop at IRTF 117 the architecture was the first time published in the following academic paper: [Ahf23].¶

Since December 2022, 20 connectivity service disruptions have been monitored and 52 false positives due to time series database temporarily not being real-time and missing traffic profiling, comparing to previous week was not applicable, occurred. Out of 20 connectivity service disruptions 6 parameters where monitored and 3 times 1, 8 times 2, 6 times 3, 2 times 4 parameters recognized the service disruption.¶

A real-time streaming based version has been deployed in Swisscom production as a proof of concept in June 2024 monitoring approximate >13'000 L3 VPN's concurrently. Improved profiling capabilities are currently under development.¶