IPv6 Operations K. Ouellette Internet-Draft UNH-IOL Intended status: Informational 4 March 2024 Expires: 5 September 2024 IPv6 Hop-by-hop and Destination Options Forwarding In Routers draft-ouellette-v6ops-eh-router-forwarding-00 Abstract It has become accepted that packets containing IPv6 Extension Headers, especially Hop-by-hop Options, are frequently dropped on the Internet. However, the question still remains as to why they get dropped and what type of devices may be dropping them. This document describes research conducted to isolate routers in a simple topology, with minimal configuration, and shows that router implementations alone are likely not the cause of packets containing IPv6 Extension Headers being dropped on the Internet. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 5 September 2024. Copyright Notice Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved. Ouellette Expires 5 September 2024 [Page 1] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Layer 3 Topology . . . . . . . . . . . . . . . . . . . . . . 3 3. Routers Under Test (RUTs) . . . . . . . . . . . . . . . . . . 3 3.1. Configuration . . . . . . . . . . . . . . . . . . . . . . 4 4. Experiments . . . . . . . . . . . . . . . . . . . . . . . . . 4 4.1. Performance and Diagnostics Metrics (PDM) Destination Option . . . . . . . . . . . . . . . . . . . . . . . . . 4 4.2. Variable Length Hop-by-hop and Destination Options . . . 4 5. Results and Findings . . . . . . . . . . . . . . . . . . . . 5 5.1. Performance and Diagnostics Metrics (PDM) Destination Option . . . . . . . . . . . . . . . . . . . . . . . . . 5 5.2. Variable Length Hop-by-hop and Destination Options . . . 5 5.2.1. Single Extension Header . . . . . . . . . . . . . . . 5 5.2.2. Multiple Extension Headers . . . . . . . . . . . . . 7 5.3. Notable Findings . . . . . . . . . . . . . . . . . . . . 10 6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 11 7. Future Work . . . . . . . . . . . . . . . . . . . . . . . . . 11 7.1. Determine Causes of Failed Scenarios . . . . . . . . . . 11 7.2. Additional Routers . . . . . . . . . . . . . . . . . . . 11 7.3. Additional Extension Headers and Chains . . . . . . . . . 11 7.4. Testing Under Load . . . . . . . . . . . . . . . . . . . 12 8. Security Considerations . . . . . . . . . . . . . . . . . . . 12 9. Privacy Considerations . . . . . . . . . . . . . . . . . . . 12 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 12 11.1. Normative References . . . . . . . . . . . . . . . . . . 12 11.2. Informative References . . . . . . . . . . . . . . . . . 12 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 13 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 13 Ouellette Expires 5 September 2024 [Page 2] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 1. Introduction It has become well known that packets containing IPv6 [RFC8200] Extension Headers (EHs), especially Hop-by-Hop Options, are often dropped when traversing the Internet. However, the question still remains as to what are the primary contributors to these packet drops and under what circumstances are the packets dropped. Some of the potential culprits include ACLs, firewalls, routers, load balancers, and more. It also is not known whether these packets are dropped intentionally (e.g., ACLs) or unintentionally (e.g., firmware bugs). This document describes the research conducted and results of isolating routers to better understand what role they may play, if any, in the dropping of EHs. Specifically, this research is aiming to understand whether or not router implementations may be the cause of dropped packets containing EHs. 2. Layer 3 Topology The layer 3 topology used for these experiments can be seen below in Figure 1. Layer 3 is specifically noted because layer 2 switches exist in the topology as well as some components being virtualized, however, this is not expected to have an effect on the traversal of packets containing EHs. +--------+ +--------+ +--------+ | Client |-----| Router |-----| Server | +--------+ +--------+ +--------+ Figure 1: Simple Two Network Topology Both client and server are virtual machines running Ubuntu 22.04 and are configured to inject EHs into all egress packets. Additionally, the server has an out-of-the-box installation of an apache2 web server. Details of the routers used can be found below in Section 3. 3. Routers Under Test (RUTs) Six routers in total were tested and are comprised of three major router manufacturers. Of the six routers, five are physical and the sixth is virtual. Due to confidentiality requirements, the specific manufacturers and models of the routers tested cannot be disclosed, therefore, the routers will be referred to with a numerical value from here on. Ouellette Expires 5 September 2024 [Page 3] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 3.1. Configuration For these experiments, all RUTs have very minimal configurations. No routing protocols are enabled and no ACLs are configured. The routers are configured to simply forward IPv6 traffic between the two interfaces. 4. Experiments For all experiments, both the client and server are configured to inject varying EHs into all egress packets. A single HTTP request is then made from the client to the server and it was observed that the client received an HTTP reply containing the expected EHs. This indicates that the RUT properly forwarded all packets associated with both the HTTP request and reply that contained EHs without dropping the packets or stripping the EHs from them [I-D.herbert-eh-inflight-removal]. To execute the HTTP request, the curl utility was used. 4.1. Performance and Diagnostics Metrics (PDM) Destination Option [RFC8250] defines the PDM destination option (DO), an option used for collecting performance metrics such as round-trip delay and server delay. For this experiment, a PDM implementation leveraging Extended Berkley Packet Filter (eBPF) was used [I-D.elkins-v6ops-bpf-pdm-ebpf]. The PDM DO was chosen to test with as it is a proposed standard and access to an existing implementation was provided for the use of this research. 4.2. Variable Length Hop-by-hop and Destination Options The second type of experiment run was testing how the RUTs processed both HBH and DO of varying header sizes. For both HBH and DO, sizes of 8, 16, 32, 64, 128, 256, and 512 bytes were tested. In addition to this, tests with an EH chain of both HBH and DO for all enumerations of the 8, 16, 32, 64, 126, and 256 header sizes were also run. Overall, there were 48 unique scenarios tested. To achieve this, the IPv6 Extension Headers Injection with eBPF (https://github.com/IurmanJ/ebpf-ipv6-exthdr-injection) project was used which allows for injecting many different types of extension headers with control over the size of each header. Ouellette Expires 5 September 2024 [Page 4] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 5. Results and Findings The following section reports on the findings of these experiments. Tables in this section will denote whether the router properly forwarded packets containing the EH(s) or not. Scenarios in which the router forwards the packets will be denoted with a check mark (✓) and scenarios where the router does not forward the packets will be denoted with an X (X). 5.1. Performance and Diagnostics Metrics (PDM) Destination Option As can be seen in Table 1, all six of the routers tested properly forwarded packets containing the PDM DO. +==========+==========+==========+==========+==========+==========+ | Router 1 | Router 2 | Router 3 | Router 4 | Router 5 | Router 6 | +==========+==========+==========+==========+==========+==========+ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+----------+----------+----------+----------+ Table 1: PDM Experiment Results 5.2. Variable Length Hop-by-hop and Destination Options 5.2.1. Single Extension Header Table 2 shows the results of a single variable sized HBH or DO EH. Seen below, all but one router forwards the EHs in every scenario. Ouellette Expires 5 September 2024 [Page 5] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 +==========+==========+========+========+========+========+========+ | Scenario | Router 1 | Router | Router | Router | Router | Router | | | | 2 | 3 | 4 | 5 | 6 | +==========+==========+========+========+========+========+========+ | HBH 8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | HBH 16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | HBH 32 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | HBH 64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | HBH 128 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | HBH 256 | ✓ | ✓ | ✓ | ✓ | X | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | HBH 512 | ? | ? | ✓ | ? | ? | ? | +----------+----------+--------+--------+--------+--------+--------+ | DO 8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | DO 16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | DO 32 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | DO 64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | DO 128 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | DO 256 | ✓ | ✓ | ✓ | ✓ | X | ✓ | +----------+----------+--------+--------+--------+--------+--------+ | DO 512 | ✓ | ✓ | ✓ | ✓ | X | ✓ | +----------+----------+--------+--------+--------+--------+--------+ Table 2: Single EH Experiment Results Ouellette Expires 5 September 2024 [Page 6] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 There are two observations to note regarding these results. The first is that five of the six routers have a question mark (?) as their result for the 512-byte HBH scenario. This is because this scenario was unable to be tested for these five routers. The reasoning for this can be found in Section 5.3. The other observation to make is that out of the six routers, Router 5 was the only router unable to forward packets for each scenario. Based on these results, it suggests that Router 5 has a limitation on the maximum size of an EH in a packet. If a packet contains an EH larger than this ceiling, it is dropped. In fact, further testing has shown that a 176-byte HBH or DO is the largest Router 5 is able to properly forward. When increasing to 184-bytes, the initial TCP SYN packet is dropped and therefore the TCP connection is not established and the HTTP request cannot be made. 5.2.2. Multiple Extension Headers The following section includes a matrix for each RUT indicating whether or not it forwarded the HTTP traffic when the packets include both a HBH and DO of varying lengths. +=====+=====+===+====+====+====+=====+=====+ | DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 | +=====+=====+===+====+====+====+=====+=====+ | 8 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 16 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 32 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 64 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 128 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 256 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ Table 3: Router 1 HBH and DO Results Ouellette Expires 5 September 2024 [Page 7] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 +=====+=====+===+====+====+====+=====+=====+ | DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 | +=====+=====+===+====+====+====+=====+=====+ | 8 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 16 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 32 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 64 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 128 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 256 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ Table 4: Router 2 HBH and DO Results +=====+=====+===+====+====+====+=====+=====+ | DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 | +=====+=====+===+====+====+====+=====+=====+ | 8 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 16 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 32 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 64 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 128 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 256 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ Table 5: Router 3 HBH and DO Results Ouellette Expires 5 September 2024 [Page 8] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 +=====+=====+===+====+====+====+=====+=====+ | DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 | +=====+=====+===+====+====+====+=====+=====+ | 8 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 16 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 32 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 64 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 128 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 256 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ Table 6: Router 4 HBH and DO Results +=====+=====+===+====+====+====+=====+=====+ | DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 | +=====+=====+===+====+====+====+=====+=====+ | 8 | | ✓ | ✓ | ✓ | ✓ | ✓ | X | +-----+-----+---+----+----+----+-----+-----+ | 16 | | ✓ | ✓ | ✓ | ✓ | ✓ | X | +-----+-----+---+----+----+----+-----+-----+ | 32 | | ✓ | ✓ | ✓ | ✓ | ✓ | X | +-----+-----+---+----+----+----+-----+-----+ | 64 | | ✓ | ✓ | ✓ | ✓ | X | X | +-----+-----+---+----+----+----+-----+-----+ | 128 | | ✓ | ✓ | ✓ | X | X | X | +-----+-----+---+----+----+----+-----+-----+ | 256 | | X | X | X | X | X | X | +-----+-----+---+----+----+----+-----+-----+ Table 7: Router 5 HBH and DO Results Ouellette Expires 5 September 2024 [Page 9] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 +=====+=====+===+====+====+====+=====+=====+ | DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 | +=====+=====+===+====+====+====+=====+=====+ | 8 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 16 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 32 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 64 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 128 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ | 256 | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +-----+-----+---+----+----+----+-----+-----+ Table 8: Router 6 HBH and DO Results Of the six routers, Router 5 (Table 7) was the only router that did not properly forward packets for all scenarios. As was seen in Section 5.2.1, Router 5 did not forward packets for the 256-byte and 512-byte scenarios which suggests that Router 5 only processes packets containing EHs up to a certain length. However, in conjunction with these results, it seems as if Router 5 may have limitations based on the total EH chain length rather than individual EHs. It does however properly process smaller EH configurations. 5.3. Notable Findings In these results, there are two findings to note. The first is that as seen in Section 5.2.1 and Section 5.2.2, a router was discovered (i.e., Router 5) that dropped packets in certain scenarios. However, the exact cause of this is still unknown as additional testing not described in this document has shown that whether or not packets are dropped is more nuanced than simply just being based on the length of the EH chain. Ouellette Expires 5 September 2024 [Page 10] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 The second notable finding is that as seen in Table 2, the 512-byte HBH scenario was unable to be performed for five out of the six routers. The exact cause is still being investigated, however, packets specifically with a 512-byte HBH option were dropped by the layer 2 switching infrastructure. Notably, both 504-byte and 520-byte HBH options traversed the infrastructure without a problem, and only 512-byte HBH options were affected. Although not recorded in the tables above, both 504-byte and 520-byte HBH scenarios were tested on Routers 1, 2, 3, 4, and 6 and they all forwarded the traffic as expected. Because of this, it is believed this is due to a firmware bug in one of the switches and was an unexpected finding. Router 3 was the only router not affected by this due to it having a different layer 2 path and not traversing the problematic switch. 6. Conclusion Even though this research was conducted on a relatively small sample size of manufacturers and routers, the results indicate that router implementations in general are likely not the culprits for dropping packets containing HBH or DO EHs. One router did face limitations when processing larger EHs, however, EH chains of that size may not be practical when used in a real-world network and therefore may not be a concern. The most surprising finding was that a layer 2 switch was the cause of dropped packets containing 512-byte HBH options. 7. Future Work 7.1. Determine Causes of Failed Scenarios Currently it is not known what the limitation in Router 5 is caused by and it is also not known why a switch in the layer 2 infrastructure is dropping packets specifically with 512-byte HBH options. Next steps will include trying to determine why these behaviors are occurring and making the manufacturers aware of these limitations with the hopes that they address them if possible. 7.2. Additional Routers A logical next step would be to test more routers, especially across a wider breadth of manufacturers. Additionally, more testing could be performed with varying connectors and link speeds. 7.3. Additional Extension Headers and Chains This research investigated how routers handle varying sizes of single HBH and DO EHs as well as chains of the two. However, more tests could be run with other standalone EHs as well as in various chain configurations. Ouellette Expires 5 September 2024 [Page 11] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 7.4. Testing Under Load As was mentioned in Section 4, these experiments were run with a single HTTP request and therefore were not run under load which may be the case for routers on the open internet. It is possible that a router's EH processing varies based on whether it is under load or not, especially in the context of processing packets on the fast vs slow path. Because of this, running these same experiments at higher throughputs may be interesting to explore. 8. Security Considerations This document has no security considerations. 9. Privacy Considerations This document has no privacy considerations. 10. IANA Considerations This document has no IANA actions. 11. References 11.1. Normative References [RFC8200] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", STD 86, RFC 8200, DOI 10.17487/RFC8200, July 2017, . [RFC8250] Elkins, N., Hamilton, R., and M. Ackermann, "IPv6 Performance and Diagnostic Metrics (PDM) Destination Option", RFC 8250, DOI 10.17487/RFC8250, September 2017, . 11.2. Informative References [I-D.elkins-v6ops-bpf-pdm-ebpf] Elkins, N., Sharma, C., Umesh, A., V, B., and M. P. Tahiliani, "Implementation and Performance Evaluation of PDM using eBPF", Work in Progress, Internet-Draft, draft- elkins-v6ops-bpf-pdm-ebpf-00, 20 February 2024, . Ouellette Expires 5 September 2024 [Page 12] Internet-Draft IPv6 HBH and DO Forwarding In Routers March 2024 [I-D.herbert-eh-inflight-removal] Herbert, T., "Infight Removal of IPv6 Hop-by-Hop and Routing Headers", Work in Progress, Internet-Draft, draft- herbert-eh-inflight-removal-04, 22 February 2024, . Acknowledgments The author would like to acknowledge and thank the following individuals. Nalini Elkins and Michael Ackermann for their guidance and support throughout this research. Balajinaidu V, Amogh Umesh, and Chinmaya Sharma for developing and providing access to their PDM DO implementation. Justin Iurman for developing and providing access to their IPv6 Extension Headers Injection with eBPF project. Michayla Newcombe and Ben Patton for their review and feedback of this document. Author's Address Kyle Ouellette University of New Hampshire InterOperability Laboratory United States of America Phone: +1 603 862 3941 Email: kouellette@iol.unh.edu Ouellette Expires 5 September 2024 [Page 13]