One of the common questions asked by people who begin their BGP journey is related to BGP ‘Next-Hop Self’ configuration option. What does it do? Should I use it on my network? What will happen if I forget to configure it? Today we’ll try to answer these questions. Continue reading “BGP Next-Hop Self Explained”
As a BGP admin, you will often need to make decisions on the ways to partition your IP Space, which routes to advertise to the Internet and which routes to suppress.
Ideally, you’d want to aggregate your IPv4/IPv6 Space as much as possible, by only advertising aggregate prefixes (also known as supernets and summary routes) to the Internet. Practically, this would mean that your Autonomous System (AS) will originate IP prefixes assigned to you by Regional Internet Registries (RIRs) or delegated to you by Upstream Providers, while suppressing all other advertisements. If everybody were to follow this rule, the Internet routing table would be much smaller and we would not have issues with FIB exhaustion. Continue reading “Advertising Aggregates Routes”
In this example, we will show recommended configuration for a Single-homed Single CE device using private AS with an upstream ISP. It is assumed that management of this device will be performed from a dedicated server residing within Customer’s Network.
This type of setup is quite common in an environment where a dedicated firewall performing source NAT function is setup to protect customer infrastructure.
Please note, that the Management Station is connected directly to the LAN interface for illustration purpose only. In real production deployments, Management Station must be protected by a firewall.
BGP Configuration
BGP configuration can be split in the following tasks:
Accept the default route from the ISP while discarding all other advertisements that might be sent to your CE
Advertise your subnet (120.0.50.0/24) while making sure that no other routers are erroneously injected
Secure BGP session by configuring a MD5 key
The actual configuration is comprised of the following blocks:
Configure Two prefix lists – one with the subnet you’ll advertise upstream and the other one with the default route you’ll be receiving from your ISP:
Next step is to secure the router itself. But default, it will pass any traffic (with some exceptions, not covered in this article) and accept connections from anywhere on the Internet. Your job is to make sure that only trusted sources can communicate with your device (control plane protection) and spoofed traffic is not allowed in and out of your network (data plane protection).
Data Plane Protection
Configure access-list to block spoofed traffic originated on the Internet:
ip access-list extended martians deny ip host 255.255.255.255 any deny ip 0.0.0.0 0.255.255.255 any deny ip 127.0.0.0 0.255.255.255 any deny ip 10.0.0.0 0.255.255.255 any deny ip 172.16.0.0 0.15.255.255 any deny ip 192.168.0.0 0.0.255.255 any deny ip 196.18.0.0 10.1.255.255 any deny ip 240.0.0.0 15.255.255.255 any deny ip 224.0.0.0 15.255.255.255 any deny ip 169.254.0.0 0.0.255.255 any deny ip 192.0.0.0 0.255.255.255 any deny ip 198.0.0.0 0.255.255.255 any deny ip 203.0.0.0 0.255.255.255 any deny ip 100.64.0.0 0.0.63.255 any! ßLocal Traffic, should not be arriving from the Internet à deny ip 120.0.50.0 0.0.0.255 any permit ip any any
Configure interface with security commands and uRPF on the LAN interface. Note that ‘no ip unreachables’ will block traceroute.
interface GigabitEthernet2 description 'CE5->PE2' ip address 120.0.4.18 255.255.255.252 no ip redirects no ip unreachables no ip proxy-arp ip access-group martians in negotiation auto!interface GigabitEthernet3 description 'LAN Segment' ip address 120.0.50.1 255.255.255.0 no ip redirects no ip unreachables no ip proxy-arp ip verify unicast source reachable-via rx negotiation auto
!
Control Plane Protection
Configure Logging; Enable SSH and SNMP access-lists, disabled unnecessary services and protocols:
no ip http server
no ip http secure-server
ip route 192.168.74.0 255.255.255.0 192.168.3.18
ip ssh rsa keypair-name ssh-key
ip ssh version 2
logging host 120.0.50.10
access-list 10 permit 120.0.50.10
snmp-server community t0ps3crrr3t RO 10
line vty 0 4
access-class 10 in
exec-timeout 11 0
password d0ntt3ll
login local
transport input ssh
!
Configure Control Plane (CPP) Protection
! Routing Protocols (BGP)
access-list 120 permit tcp any gt 1024 host 120.0.4.18 eq bgpaccess-list 120 permit tcp any eq bgp host 120.0.4.18 gt 1024 established
! Management Protocols (SSH, SNMP)
access-list 121 permit tcp host 120.0.50.10 host 120.0.50.1 eq 22access-list 121 permit tcp host 120.0.50.10 eq 22 host 120.0.50.1 establishedaccess-list 121 permit udp host 120.0.50.10 host 120.0.50.1 eq snmp
! Ping / Traceroute LAN Interface
access-list 122 permit icmp any host 120.0.50.1 echoaccess-list 122 permit icmp any host 120.0.50.1 echo-replyaccess-list 122 permit icmp any host 120.0.50.1 ttl-exceededaccess-list 122 permit icmp any host 120.0.50.1 packet-too-bigaccess-list 122 permit icmp any host 120.0.50.1 port-unreachableaccess-list 122 permit icmp any host 120.0.50.1 unreachable
! Ping/Traceroute WAN Interface
access-list 122 permit icmp any host 120.0.4.18 echoaccess-list 122 permit icmp any host 120.0.4.18 echo-replyaccess-list 122 permit icmp any host 120.0.4.18 ttl-exceededaccess-list 122 permit icmp any host 120.0.4.18 packet-too-bigaccess-list 122 permit icmp any host 120.0.4.18 port-unreachableaccess-list 122 permit icmp any host 120.0.4.18 unreachable
! Undesired Traffic
access-list 123 permit icmp any any fragmentsaccess-list 123 permit udp any any fragmentsaccess-list 123 permit tcp any any fragmentsaccess-list 123 permit ip any any fragmentsaccess-list 123 permit tcp any any eq bgp rst
! All Other Traffic
access-list 124 permit tcp any anyaccess-list 124 permit udp any anyaccess-list 124 permit icmp any anyaccess-list 124 permit ip any any!
! Define Class-Maps
class-map match-all Catch-All-IP match access-group 124class-map match-all Management match access-group 121class-map match-all Normal match access-group 122class-map match-all Undesirable match access-group 123class-map match-all Routing match access-group 120!
! Configure CoPP Policy
policy-map RTR_CoPP class Undesirable police 8000 1500 1500 conform-action drop exceed-action drop class Routing police 100000 5000 5000 conform-action transmit exceed-action transmit class Management police 100000 20000 20000 conform-action transmit exceed-action drop class Normal police 50000 5000 5000 conform-action transmit exceed-action drop class Catch-All-IP police 50000 5000 5000 conform-action transmit exceed-action drop class class-default police 8000 1500 1500 conform-action transmit exceed-action drop
! Apply CoPP Policy
control-plane service-policy input RTR_CoPP!
Complete Router Configuration
service timestamps debug datetime localtime show-timezoneservice timestamps log datetime localtime show-timezoneservice password-encryption!hostname CE11!boot-start-markerboot-end-marker!!enable secret 5 $1$9Ah6$7tFkcd/bJRrHSx0grfmYA1!no aaa new-modelno ip source-routeno ip domain lookup!username cisco privilege 15 secret 5 $1$ZJAP$Hmq/nCv7qQcwPHyB4Ixdo0!!class-map match-all Catch-All-IP match access-group 124class-map match-all Management match access-group 121class-map match-all Normal match access-group 122class-map match-all Undesirable match access-group 123class-map match-all Routing match access-group 120!policy-map RTR_CoPP class Undesirable police 8000 1500 1500 conform-action drop exceed-action drop class Routing police 100000 5000 5000 conform-action transmit exceed-action transmit class Management police 100000 20000 20000 conform-action transmit exceed-action drop class Normal police 50000 5000 5000 conform-action transmit exceed-action drop class Catch-All-IP police 50000 5000 5000 conform-action transmit exceed-action drop class class-default police 8000 1500 1500 conform-action transmit exceed-action drop!!interface GigabitEthernet1 description 'Out-of-Band Management' ip address 192.168.3.231 255.255.255.0 no ip redirects no ip unreachables no ip proxy-arp negotiation auto!interface GigabitEthernet2 description 'CE5->PE2' ip address 120.0.4.18 255.255.255.252 no ip redirects no ip proxy-arp ip access-group martians in negotiation auto!interface GigabitEthernet3 description 'LAN Segment' ip address 120.0.50.1 255.255.255.0 no ip redirects no ip unreachables no ip proxy-arp ip verify unicast source reachable-via rx negotiation auto!router bgp 111100 bgp log-neighbor-changes network 120.0.50.0 mask 255.255.255.0 neighbor 120.0.4.17 remote-as 100 neighbor 120.0.4.17 description PE2 neighbor 120.0.4.17 password 7 14141B180F0B neighbor 120.0.4.17 soft-reconfiguration inbound neighbor 120.0.4.17 prefix-list default-only in neighbor 120.0.4.17 prefix-list originated-out out!virtual-service csr_mgmt!ip forward-protocol nd!no ip http serverno ip http secure-serverip route 192.168.74.0 255.255.255.0 192.168.3.18ip ssh rsa keypair-name ssh-keyip ssh version 2!ip access-list extended martians deny ip host 255.255.255.255 any deny ip 0.0.0.0 0.255.255.255 any deny ip 127.0.0.0 0.255.255.255 any deny ip 10.0.0.0 0.255.255.255 any deny ip 172.16.0.0 0.15.255.255 any deny ip 192.168.0.0 0.0.255.255 any deny ip 196.18.0.0 10.1.255.255 any deny ip 240.0.0.0 15.255.255.255 any deny ip 224.0.0.0 15.255.255.255 any deny ip 169.254.0.0 0.0.255.255 any deny ip 192.0.0.0 0.255.255.255 any deny ip 198.0.0.0 0.255.255.255 any deny ip 203.0.0.0 0.255.255.255 any deny ip 100.64.0.0 0.0.63.255 any deny ip 120.0.50.0 0.0.0.255 any permit ip any any!!ip prefix-list default-only seq 10 permit 0.0.0.0/0!ip prefix-list originated-out seq 10 permit 120.0.50.0/24logging host 120.0.50.10access-list 10 permit 120.0.50.10access-list 10 permit 192.168.0.0 0.0.255.255access-list 120 permit tcp any gt 1024 host 120.0.4.18 eq bgpaccess-list 120 permit tcp any eq bgp host 120.0.4.18 gt 1024 establishedaccess-list 121 permit tcp host 120.0.50.10 host 120.0.50.1 eq 22access-list 121 permit tcp host 120.0.50.10 eq 22 host 120.0.50.1 establishedaccess-list 121 permit udp host 120.0.50.10 host 120.0.50.1 eq snmpaccess-list 122 permit icmp any host 120.0.50.1 echoaccess-list 122 permit icmp any host 120.0.50.1 echo-replyaccess-list 122 permit icmp any host 120.0.50.1 ttl-exceededaccess-list 122 permit icmp any host 120.0.50.1 packet-too-bigaccess-list 122 permit icmp any host 120.0.50.1 port-unreachableaccess-list 122 permit icmp any host 120.0.50.1 unreachableaccess-list 122 permit icmp any host 120.0.4.18 echoaccess-list 122 permit icmp any host 120.0.4.18 echo-replyaccess-list 122 permit icmp any host 120.0.4.18 ttl-exceededaccess-list 122 permit icmp any host 120.0.4.18 packet-too-bigaccess-list 122 permit icmp any host 120.0.4.18 port-unreachableaccess-list 122 permit icmp any host 120.0.4.18 unreachableaccess-list 124 permit tcp any anyaccess-list 124 permit udp any anyaccess-list 124 permit icmp any anyaccess-list 124 permit ip any any!snmp-server community t0ps3crrr3t RO 10!!control-plane service-policy input RTR_CoPP!banner motd ^CDisconnect IMMEDIATELY if you are not an authorized user!^C!line con 0 exec-timeout 11 0 password d0ntt3ll stopbits 1line vty 0 4 access-class 10 in exec-timeout 11 0 password d0ntt3ll login local transport input ssh!!end
This document provides a cheat sheet of commonly used troubleshooting commands used in Cisco and Juniper environments. The list is incomplete. Please send us a note if you want to contribute.
Management
Cisco IOS
Juniper JunOS
Description
show tech-support
request support info
request support information | save /var/tmp/RSI.txt
file archive compress source RSI.txt destination RSI.txt.tgz
Gather support info for vendor’s TAC
show hardware
show chassis hardware
Show hardware-related info
show version
show system uptime
Show system’s uptime
show processes cpu
show processes cpu sorted
show processes cpu history
show chassis routing-engine
show system processes extensive
show system threads
Verify CPU Utilization
show processes memory
show memory summary
show system processes extensive
show task memory detail
Verify Memory Utilization
dir bootflash:
show system core-dumps
Check for crash files / core dumps
dir
file list
Show directory structure
show system storage
Verify available storage space
show users
show system users
List connected users
clear line X
request system logout user ABC
Disconnect user
start shell
Enter Unix Shell
monitor traffic interface ge-0/0/1
Monitor traffic on the interface (will not show transit packets)
In this article, we will focus on the RFC 7547. This RFC covers BGP Operations and Security best current practices and needs to be understood and implemented by any organization running BGP in production.
Introduction
RFC 7547 recommendations can be split into the following categories:
BGP Session Protection
Prefix Filtering Recommendations
AS-Path Filtering Recommendations
Next-Hop Filtering
Optional BGP Community Scrubbing
Traffic Filtering Recommendations
In this article, we will use Roman Numerals (I, II, etc) to identify BGP protection mechanisms, Arabic Numerals (1,2, etc) to identify Traffic Filtering, Uppercase Letters (A, B, etc) to identify Prefix Filtering, and Lowercase Letter (a,b, etc) to identify AS-Path filtering and Greek Letters (α, β) to identify BGP scrubbing.
Figure below shows depicts peering routers connected to upstream, private, IXP and downstream peers.
As most of the modern routers do, our sample router has a dedicated forwarding engine responsible for forwarding packets and a dedicated routing engine responsible for participating in routing protocols, building Routing Information Base (RIB) and Forwarding Information Base (FIB) tables. While actual vendors’ implementations will vary between routers’ models, best practices discussed in this article are generic enough to be applicable to the majority of vendors.
BGP Protection
Group of BGP Protection mechanisms is responsible for maintaining stability of BGP sessions, as well as providing anti-spoofing and bogus route-injection protection mechanisms. We will also add “maximum-prefix” protection mechanism to this category, as it helps to protect against operators’ mistakes.
I. GTSM (TTL Security)
GTSM – Generalized TTL Security Mechanisms, also known as TTL security, defined in RFC 5082. GTSM (TTL Security) is a mechanism that checks TTL value of incoming IP Packets in order to make sure they have not been spoofed. Directly connected BGP peers will set IP TTL value to 255, making it impossible to deliver spoofed IP with TTL=255 packets via non-directly connected interfaces. As per section 5.2 of RFC 7454 GTSM should be implemented.
TCP-AO – TCP Authentication Option is a stronger protection mechanism than traditionally used MD5, it is described in RFC 5925. At some point, it is expected to replace MD5 for session protection. It has not been widely adopted due to the lack of implementation from equipment vendors.
Section 5.1 of RFC 7454 recommends, although does not require, leveraging either MD5 or TCP-AO for session protection.
No configuration examples due to lack of vendors’ implementation.
III. MD5
MD5 – Protection of the TCP session header, described in RFC 2385. MD5 is a TCP session protection mechanism that has been available for many years and is supported by the vast majority of equipment manufacturers. It has become the de-facto standard for BGP session protection. Although it has been made obsolete by TCP-AO protection, it is still used for the majority of BGP peering sessions.
Configuration Examples (Simple Key and Key-Chains):
Maximum-Prefix Limit is one of the commonly used safety mechanisms that will bring down BGP session if the number of routes advertised by the peer exceeds pre-configured limit. Section 8 of RFC 7454 provides the following recommendations:
From public and private peers, it is recommended to have the limit set to either a lower than the number of routes on the Internet, or to a specific number for each peer based on the advertised number of routes plus some headroom. From the author’s experience, setting the number to below the number of routes on the Internet is too risky and should be avoided. There have been situations where public and private peers would make an error and leak the entire BGP table to their peering partners, causing major network instability. Author prefers setting session reset limit to 2x the number of routes normally advertised by the peer and session warning limit to 1.5x number of routes. Your NOC should monitor logs for warning threshold violations and adjust limits accordingly.
From upstream, the number of routes should be set higher than the number of routes on the Internet, but not higher that the capabilities of your routers. For example, if FIB tables of your devices can support up to 1 Million IPv4 routes, you can set the limit to be 950,000 routes. While resetting BGP sessions with your upstream providers is never a good thing, damage caused by reset is much lower than that caused by FIB exhaustion. For more information, please refer to our article on BGP Table Size analysis (http://www.bgphelp.com/2017/01/01/bgpsize/).
MD5, TCP-AO and GTSM have to be configured on both sides of the BGP session. Max-Prefix can be configured on one side only.
Prefix Filtering
Prefix-filtering policies are responsible for discarding bogus route-advertisements to and from BGP peers. Examples of these bogus advertisements are prefixes from RFC1918 address space, to specific routes (>24), unallocated prefixes.
Route-filtering should be implemented on each BGP session maintained by the service provider:
A. Private/Public/Transit Inbound Prefix Filtering
B. Private/Public/Transit Outbound Prefix Filtering
C. Downstream Inbound Prefix Filtering
D. Downstream Outbound Prefix Filtering
A. Inbound Prefix Filtering from Private/Public/Transit Peers
RFC 7475 provides similar recommendations for route filtering from Upstream Providers (section 6.2.3) and route-filtering from private and public peers (section 6.2.1). Because of this, there is very little difference in filtering policies, allowing us to combine them in one recommendation.
As per Section 6.2.1.1.1 of RFC 7475, the following prefixes should not be accepted from peers
Special-Purpose Prefixes (RFC 7475 Section 6.1.1)
Unallocated Prefixes (RFC 7475 Section 6.1.2)
Prefixes that are too specific (RFC 7475 Section 6.1.3)
Prefixes belonging to the local AS (RFC 7475 Section 6.1.4)
IXP LAN Prefixes (RFC 7475 Section 6.1.5), other than authorized ASes (RFC 7475 Section 6.1.5)
The Default Route (RFC 7475 Section 6.1.6)
Section 6.2.1.1.2 of RFC 7475 also provides recommendations for “Strict” inbound filtering option, which we consider to be too risky and will not cover in this document.
B. Outbound Prefix Filtering towards Private/Public/Transit Peers
As per Section 6.2.1.2 of RFC 7475, the following prefixes should not be accepted from peers
Special-Purpose Prefixes (RFC 7475 Section 6.1.1)
Prefixes that are too specific (RFC 7475 Section 6.1.3)
IXP LAN Prefixes (RFC 7475 Section 6.1.5)
The Default Route (RFC 7475 Section 6.1.6)
You also need to make sure that only authorized prefixes (those advertised by your AS and downstream customers) are being sent.
C. Inbound Prefix Filtering from Customers
General recommendations provided in Section 6.2.2.1 of RFC 7475 state that “only customer prefixes SHOULD be accepted, all others SHOULD be discarded.” The list of allowed prefixes should be manually built by the network provisioner after validating that customer prefixes are indeed allocated to the client by IP address management authorities.
In some cases, if customer advertises too many prefixes or has BGP clients of their own, customer-specific filters can be replaced with generic filters previously described in “Inbound Filtering from Private/Public/Transit Peers” section of the paper.
D. Outbound Prefix Filtering towards Customers
Depending on the customer preferences, they might want to receive
The default route only
Full Internet routing table
Subset of the Full Internet table (e.g. only the routes received via public and private peers, but not the transit routes)
The default route in addition to the Full or Partial Internet view
Generic recommendation described in Section 6.2.2.2 of RFC 7454 states that the following prefixes should not be sent to the customer:
Special-Purpose Prefixes (RFC 7475 Section 6.1.1)
Prefixes that are too specific (RFC 7475 Section 6.1.3)
The Default Route (RFC 7475 Section 6.1.6), for those customers not willing to receive it
AS-Path Filtering
Section 9 of RFC 7454 provides a number of AS-Path Filtering recommendations that should be implemented on upstream/private/public peering sessions and customer sessions.
Similar to how we analyzed Prefix Filtering recommendations in the previous chapter, we will review AS-Path Filtering recommendations below.
a. Inbound AS-Path Filtering from Private/Public/Transit Peers
Section 9 of RFC 7454 recommends the following:
Private AS numbers should not be accepted, unless used for special purposes such as black-hole origination
AS Paths with the first AS number not the one of the peer should not be accepted, unless originated by IXP’s router server
Do not accept your own AS number in the AS path
b. Outbound AS-Path Filtering from Private/Public/Transit Peers
Section 9 of RFC 7454 recommends the following:
Do not originate prefixes with nonempty AS Paths, unless you intend to provide transit for these prefixes
Do not originate prefixes with upstream AS numbers in the AS Path, unless you intend to provide transit to these prefixes
Do not advertise Private AS Paths, unless there is a special “private” arrangement with your peers
c. Inbound AS-Path Filtering from Downstream Customers
Section 9 of RFC 7454 recommends the following:
Only accept 2-byte and 4-byte AS paths containing ASNs belonging to the customer.
If this is not possible, accept only path lengths relevant to the type of the customer, while discourage excessive prepending
Do not accept your own AS number in the AS path
d. Outbound AS-Path Filtering from Downstream Customers
Do not advertise Private AS Paths, unless there is a special “private” arrangement with your customers
Next-Hop Filtering
BGP can advertise prefixes with a third-party next hop, thus directing packets not to the peer announcing the prefix but somewhere else. This mechanism is commonly used at Internet Exchange Points, where prefixes will be announced by IXP’s route-server.
Section 10 of RFC 7545 recommends the following policies at IXP locations:
For direct peering (without router-server), apply inbound BGP policy that would set next-hop for the accepted prefix to BGP peer IP address
For indirect peering (with IXP’s route-server), accept next-hop attribute advertised by the route-server
BGP Community Scrubbing
Section 11 of RFC 7454 provides the following optional community scrubbing recommendations.
Ingress BGP peering policy applied to transit/public/private and downstream peers should remove all inbound communities with SP’s number in the high-order bits, except for the ones used for signaling (e.g. setting BGP Local Preference).
Ingress BGP Policy should not remove other communities, as those communities can be used to communicate with upstream providers.
Traffic Filtering
Section 4 of RFC 7454 provides basic recommendations when it comes to traffic filtering and BGP.
All packets destined to TCP Port 179 and not originated from addresses of configured BGP peers should be discarded. If supported, Control Plane ACL (point 3 on the diagram) should be used. If not supported, ACL applied to each peer-facing port (point 1) should be used.
If supported, BGP Rate-Limiting (point 4) should also be implemented, to make sure that the number of BGP packets per second does not exceed platform’s capability.
Generic Control Plane protection recommendations are out of RFC 7454 scope and are covered in RFC 6192.
In this article, we will attempt to forecast the size of global internet routing table and analyze the potential impact of aforementioned routing growth on the stability of Internet infrastructure.
Global routing infrastructure is comprised of IPv4 and IPv6 routes advertised by BGP-speaking service providers and enterprises. These BGP advertisements are processed by the routers and eventually programmed into special tables called Forwarding Information Table (FIB). There is a limit a number of FIB entries a particular system can support before running out of FIB capacity. The maximum FIB capacity of the platform is determined by such factors as ASIC, amount of memory, software license, etc.
Even within a single vendor’s portfolio, the maximum FIB size of available platforms varies dramatically, from a few thousand entries in a low-cost top or rack switch up to millions of entries in an expensive Internet router. It is important to note, that advertised FIB numbers may only be applicable to certain (typically IPv4) routes. Other route types, such as MPLS VPN and IPv6, might require more memory per entry, decreasing the overall FIB capacity.
For example, Cisco’s Catalyst 6500 / 7600 with 3BXL supervisor can support 1 Million IPv4 routes, but only 512K IPv6 routes.
It is also important to note, that not all vendors will support dynamic allocation of FIB entries between route-types. Instead, FIB might be pre-partitioned to support some arbitrary number of entries of a certain type. Previously mentioned 3BXL supervisor comes preconfigured to support 512K IPv4 + MPLS entries and 256K IPv6 + Multicast entries. It is easy to spot that in Cisco’s SUP720 implementation IPv6 routes take twice as much space as IPv4 entries.
Historic perspective
The problem of FIB capacity and growing Internet size is not new.
Multiple outages were reported back in 2008 when Internet BGP table size crossed 256K limit and again in 2014 when 512K entries limit was exceeded.
Service Providers and BGP-speaking enterprises had to take remedial actions in order to maintain Internet stability. We will discuss these actions later.
Internet Growth
There are two major forces that drive Internet table size growth – IPv4 space partitioning and new IPv6 advertisements.
IPv4 address exhaustion (https://en.wikipedia.org/wiki/IPv4_address_exhaustion) that occurred before 2011 and 2015 did not slow down the speed of IPv4 table growth, instead it accelerated the fragmentation of IPv4 space.
IPv4 Table Size Projection
As mentioned previously, IPv4 table size continues its rapid expansion, demonstrating approximately 10% year-over-year growth over the past few years.
2009 to 2017 IPv4 Table Size Growth:
2009
2010
2011
2012
2013
2014
2015
2016
2017
Table
Size (Thousand Routes)
286
316
345
409
466
499
536
591
648
Year
over
Year (%)
18
10
9
19
14
7
7
10
10
2017 IPv4 Table Size Growth to Date:
Month
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Table Size (Thousand Routes)
648
653
663
663
673
676
679
684
688
691
Month over Month (%)
0.7
1.5
0.1
1.5
0.5
0.4
0.7
0.5
0.5
Compared to January (%)
0.7
2.2
0.3
3.9
4.4
4.8
5.5
6.1
6.6
Our statistical model shows that if this growth continues, global Internet table will surpass 1 Million entries sometime in 2020.
IPv6 Table Size Projection
As IPv6 gets adopted by Service Providers and Enterprises, IPv6 table size is also expected to continue to raise. The current year-over-year growth is about 30% with no signs of deceleration.
2009 to 2017 IPv6 Table Size Growth:
2009
2010
2011
2012
2013
2014
2015
2016
2017
Table Size (Thousand Routes)
1.6
2.5
4.1
7.7
12
17
22
27
37
Year
over
Year (%)
65
52
65
86
56
41
29
25
35
2017 IPv6 Table Size Growth to Date:
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Table Size (Thousand Routes)
36
37
38
39
40
40
42
43
44
44
Month over Month (%)
2.7
3.0
0.8
2.2
1.1
3.1
2.7
2.0
0.4
Compared to January (%)
2.7
5.7
6.6
9.0
10.3
13.7
16.7
19.0
19.5
While IPv6 table is not expected to grow to the same size as IPv4 table due to much bigger initial block allocations by the registries, ongoing IPv6 adoption will nonetheless lead to the table size increase.
FIB Utilization
IPv4 and IPv6 table size increases will translate into FIB size increase. The actual impact on your router will depend on a specific vendor’s implementation. In the best-case scenario, you will observe one-to-one correlation between the combined size of IPv4 and IPv6 tables and FIB table. More common scenario might be IPv6 entries using twice as much space as IPv4 entries. This later scenario is depicted below:
As you can deduce from the graph, routers that are capable of supporting 1Mln routes, will run out of FIB space sometime in 2019. In fact, you might run into problems much earlier than that, if you have
Large number of disaggregated internal routes, such as loopbacks, point-to-point IPs and customer routes
BGP policy allowing to accept long (>24) prefixes from external peers
Extensive public and private peering with partners who might advertise more specific routes not otherwise visible in the public Internet
Provide other services that require FIB space, such as Mutlicast, MPLS VPN, L2 VPN, etc.
What to expect
Assuming that the FIB size of your Internet-facing router is limited by 1 Mln entries, you can expect to run into issues sometime in 2019. The actual impact will depend on the platform in use. Some systems might attempt to fall back to RE-based forwarding for the destinations which could not be programmed in hardware. This might lead to high CPU utilization on the entire system and general instability of the router.
Other systems will simply drop traffic to such destinations. This scenario can manifest itself by customers unable to reach some sites on the Internet, while accessing other sites. You should monitor system logs and FIB utilization to spot the issue.
How to prepare
As an administrator, there are a few things you should do to be ready to withstand Internet size growth:
Understand your system’s FIB capacity to make sure you have enough room to accommodate expected Internet growth
If your system allows changing FIB partitioning, make sure it is set up in the most optimal way. For example, you might want to allow for up to 800K IPv4 and 100K IPv6 routes
If possible, upgrade your systems to support at least 2Mln FIB entries. This is applicable to both Routing Engine and Line Cards
If upgrade is not viable at the moment, consider inbound route-filtering to decrease the number of routes accepted from your peers. The general consensus is that you can safely drop all IPv4 /25 and longer prefixes while maintaining full reachability of Internet destinations.
Conclusion
Internet global routing table continues to grow with no signs of slowing down. The major contributor to this growth is an ongoing IPv4 disaggregation, as well as a proliferation of IPv6 Internet. As a network administrator, you need to be prepared to protect your network from negative consequences of this growth by optimizing your routing policies and upgrading physical infrastructure.