Campus Network Design
Access: Where end users are connected. Provides L2 connectivity between users. Low cost per switch, high port density, scalable uplinks to higher layers, high availability (redundant power and supervisors, FHRP protocols), ability to converge network services (phones, PCs, APs), security features (ARP Inspection, DHCP Snooping, BPDU Guard), and QoS.
Distribution: Interconnection between the network’s Access and Core layers. Aggregation of multiple Access layer switches. High L3 routing throughput. Segmentation and isolation of workgroups using a combo of L2 and L3 switching. Security and policy-based connectivity functions, QoS, and scalable. Routing policies such as route selection, filtering, and summarization. Can be default gateway for Access devices. High availability through redundant Distribution switches providing dual paths to Access and Core.
Core: Sometimes called the backbone. Provides connectivity between all Distribution layer devices. Must be capable of switching traffic as efficiently as possible. Very high L3 routing throughput, no costly packet manipulations (ACLs, packet filtering), resilience through redundant devices, components, and paths. Advanced QoS. Scalability through routing protocols and Gigabit+ connectivity. Core should consist of two multilayer switches that connect two or more switch blocks in a redundant fashion. Sometimes called a dual Core, because it is usually built from two identical switches. Multinode Core = expanded Core.
Collapsed Core: When the Distribution and Core layers are combined into a single layer of switches.
Switch Block: Group of Access layer switches, together with their Distribution switches.
If all links between Distribution and Access switches are L2, loops are possible, and the entire topology is a single failure domain. A better design prevents L2 loops by utilizing a unique VLAN per Access switch, and making links between Distribution switches L3. Also allows for load balancing.
Switch Port Configuration
To detect speed/duplex mismatches, use sh interface. Will show the port’s settings, number of runts, input errors, queue, and drops.
By default, switches detect an error condition on every port for every cause. If an error is detected, the port is made “err-disabled.”
config)# (no) errdisable detect cause [ all | cause-name ] !!! to disable or define causes. all by default.
sh interface (fx/x) status (err-disabled) !!! to see which are disabled, regular sh int won’t show.
By default, “err-disabled” ports must be re-enabled manually. To setup auto recovery (globally):
config)# errdisable recovery cause [ all | cause-name ] !!! psecure = port security.
config)# errdisable recovery interval [ seconds ] !!! default of 300 secs.
sh errdisable recovery !!! to see which errors have recovery enabled.
CDP: L2 protocol. Rapid error tracking, HW platform, interfaces, encapsulation, MTU, 01-00-0c-cc-cc-cc multicast in clear text. Version 2 is default (cdp advertise-v2), adds native vlan, port duplex, and VTP domain. 1 min interval, 3 min hold-down defs.
sh cdp | sh cdp neighbors (fx/x) (detail) | sh cdp entry (id*) | (no) cdp run {global} | (no) cdp enable {interface} | cdp timer [#] | cdp holdtime [#]
LLDP: Standard alternative to CDP, extensible with type-length-values (TLVs). Cannot send both LLDP-MED and basic LLDP TLVs at the same time! Does not provide VTP info!
sh lldp | sh lldp neighbors (fx/x) (detail) | (no) lldp run | (no) lldp (receive | transmit)
LLDP Media Endpoint Discovery (MED): Extension of LLDP, sent between network and endpoint devices (switches and phones). Auto-discovers LAN policies: VLAN, Diffserv settings, PnP networking, device location, E911, L2 priority, Plug N Play, PoE, etc.
POE: DC power is supplied over Ethernet cables. Switch keeps power disabled when port is down, but continually tries to detect whether a powered device is connected to a port. If so, the switch begins providing power so the device can initialize. The switch begins by supplying a small voltage across the tx/rx pairs. If resistance is measured, it knows power is being drawn. The switch can apply predetermined voltages to test for the power class of device, to use the appropriate maximum power. Switch begins with 15.4W, once the device is started, it can use CDP or LLDP to request more power. On by default. To configure:
config-if)# power inline (auto | static) (max [mw]) (consumption [mw]) !!! (static) for high priority, (consumption) for initial power send to non cdp devices.
config-if)# power inline never !!! to disable.
sh power inline (fx/x) (detail)
Cisco InLine Power (ILP) = 7W | 802.3af (PoE) = 15.4W | 802.3at (PoE+) = 25.5W | Cisco Universal PoE (UPoE) = 60W.
Switch Operation
As a frame is received on a switch port, the switch inspects the source MAC; if it is not known, the source MAC, port, VLAN, and a timestamp are recorded in the forwarding table (Content Addressable Memory, CAM). Incoming frames also include the destination MAC; the switch looks this address up in the CAM to find the port/VLAN to send the frame out. If the address is not in the table, the frame is flooded out all switch ports assigned to the source VLAN (Unknown unicast flooding). CAM commands:
config)# mac address-table static [mac] vlan [vlan #] interface [fx/x] !!! static entry.
config)# mac address-table notification [ mac-move | threshold | change ] !!! notifications.
sh mac address-table dynamic ( address [mac] | interface [fx/x] | vlan [vlan #] ) !!! to view the cam table.
sh mac address-table count !!! to see the cam size.
clear mac address-table dynamic [ address (mac) | interface (fx/x) | vlan (vlan #) ] !!! clear a cam entry.
The CAM table stores entries by default for 300 seconds (5 mins), after which, they are purged. Every time a frame is received, the timestamp is renewed. To change the timer for purging stale entries:
config)# mac address-table aging-time [seconds] !!! 0 disables.
Multilayer Switching (MLS) has two types/generations:
Route Caching: First gen, requires a route processor (RP, hardware), and a switching engine (SE, software). The RP processes a traffic flow’s first packet to determine a destination. The SE listens to that first packet and sets up a “shortcut path” in the cache so that subsequent packets of the same flow can be switched directly to the destination port without passing through the RP. Called NetFlow LAN switching, demand-based switching, fast switching, or route once, switch many. Slower.
Topology Based: Second gen, utilizes specialized hardware with distinct RP and SE functions. Uses L3 routing info to build and prepopulate a single database of the entire known network topology, known as the FIB. The FIB is used for efficient table lookup in hardware by the SE. Known as Cisco Express Forwarding (CEF).
Ternary CAM (TCAM): Contains ACLs in a compiled form so that a decision can be made on whether to forward a frame, in a single table lookup. Acts as an L3 version of the CAM, uses ASICs combined with the CAM memory to route, do ACLs, and QoS via hardware. ACLs can be used to identify frames according to their MACs, IPs, protocols, and L4 port numbers. Most switches have multiple TCAMs so both in/outbound security and QoS ACLs can be evaluated simultaneously.
After TCAM, CAM, and FIB lookups to determine egress, some portions of the frame must be modified/rewritten such as the destination and source MACs. L3 packets additionally have their TTL value decremented and checksum calculated.
Feature Manager (FM): After an ACL has been created, the FM compiles/merges the entries into the TCAM. If an ACL has a gt/lt/range operator for port numbers, the FM compiles the entry to include the operator in a Logical Operation Unit (LOU).
The TCAM is organized by masks. Entries are composed of Value, Mask, and Result (VMR) combinations:
Value: 134-bit quantities of source/dest addresses and other protocol info, components depend on ACL type.
Masks: 134-bit quantities that select the bits of interest. A mask bit is set to mark a value bit to be exactly matched or is not set to mark a value bit that doesn’t matter.
Results: Numeric values representing what action to take after the TCAM lookup occurs. Multiple results/actions: Permit/deny, QoS policer value, pointer to a next-hop etc.
The TCAM is self-sufficient, to display utilization and max values:
sh platform tcam utilization
SDM-Templates: Most non high-end switches have fixed architecture where the CAM, FIB, and other tables share resources. You can select a preferred type of switching to alter the sizes of the tables, depending on the desired functions, including dual stack IPv4/6. L2 switching should have a large CAM while L3 routing should have a large FIB. To see and change the current setting:
sh sdm prefer (?) (option) !!! (?) will list the options, add in (option) to see values. blank for running.
config)# sdm prefer [template] !!! requires reboot.
access: maximizes system resources to accommodate a large number of acls.
default: default template, gives balance to all functions.
routing: maximizes system resources for ipv4 unicast routing, typically required for a router or aggregator in the center of a network.
vlan: disables routing and supports the maximum number of unicast mac addresses. typically selected for a layer 2 access switch.
dual-ipv4-and-ipv6: dual stack, doesn’t support ipv6 multicast.
L3 Switching
Switch Virtual Interface (SVI): Logical interface that represents one VLAN, used for inter-VLAN routing. The assigned L3 address becomes the default gateway for the VLAN. The VLAN must be defined and active (no shut), and at least one L2 port assigned to the VLAN must be active and STP converged before the SVI can be used. Not considered a routed port, but supports both routing and switching protocols. A sh interface will display the hardware type as “EtherSVI.” VLANs and interface VLANs are separate!
config)# int vlan [#] !!! doesn’t need to match an existing vlan, and won’t create a new vlan.
config-if)# ip add [ip] [mask]
config-if)# no shut
config)# ip routing !!! to enable inter-vlan routing on the switch.
SVI Autostate: Automatically keeps the SVI down until the VLAN is ready so no other switching or routing can attempt to use the SVI prematurely. To disable:
config-if)# switchport autostate exclude !!! per physical interface.
Routed Port: A physical L3 port on a switch that acts like a port on a router. Supports all routing protocols and can have an IP address assigned. Does not support VLAN subinterfaces, is not associated with a VLAN, nor does it support any L2 protocols (STP). If an L3 address is assigned to an EtherChannel, it’s assigned to the whole Port Channel, not just an individual port
config)# int (fx/x) !!! physical interface.
config-if)# no switchport
config-if)# ip add [ip] [mask]
Process Switching: Used in routers only. Each packet must be examined by the CPU and handled in software. Slowest method.
Cisco Express Forwarding (CEF): Topology based. High performance packet forwarding through the use of dynamic lookup tables.
config)# (no) ip cef !!! global, on by default.
config-if)# (no) ip route-cache cef !!! per interface.
Forward Information Base (FIB): The L3 routing table, reformatted into an ordered list of destination networks, masks, and next-hop IP addresses for each IP destination subnet in the table. Most specific routes are placed at the top. Also contains host routes for directly connected hosts. Show FIB with:
sh ip cef (fx/x) (vlan [x]) (detail)
version number: # of times the cef entry has been updated.
epoch: # of times the cef table has been flushed and regenerated as a whole.
sh ip cef [prefix ip] [prefix mask] (longer-prefixes) !!! to search per network.
Adjacency Table: The L2 MAC for each next-hop IP in the FIB that is a single hop away. Eliminates lag for ARP requests. Null adj. used to switch packets destined for the null interface. Drop adj. used to switch packets that can’t be forwarded normally. Discard adj. used when packets must be discarded due to an ACL or other. Punt adj. used when packets must be punted to L3 engine.
sh adjacency (fx/x) (vlan [x]) (summary) (detail) !!! address string: destmac sourcemac ethertype (0800=ip)
CEF Glean: If there is no L2 next-hop in the Adjacency table, the FIB entry is marked as “glean” because the L3 Forwarding Engine can't forward the packet in hardware. The packet is sent to the L3 Engine so that it can “glean” the MAC via an ARP request/reply.
sh ip cef adjacency (glean | punt)
sh ip cef [ip] [mask] detail
Layer 3 Engine (Software): Acts like a router; builds the routing information using static routes or routing protocols, and ARP.
Layer 3 Forwarding Engine (Hardware): Uses the Layer 3 Engine's routing info and ARP table to switch packets in hardware.
Normal CEF Behavior: Ingress packet → FIB → Adjacency Table → Rewrite Engine → egress packet.
If CEF Punted to L3 Engine: Ingress packet → Routing Table → ARP Table → egress packet.
Reasons for Punting: Entry not in FIB, FIB or TCAM are full, TTL expired, MTU exceeded (must fragment packet), ICMP redirect, encap. type not supported, packets tunneled/require compression/encryption, ACL with log, or NAT. Wildcard entry to RE.
sh cef (not-cef-switched | drop) !!! drop are dropped, not switched are punted.
Packet Rewrite: Header must be rewritten. Change L2 dest & source address, decrement L3 TTL, and recalc IP and frame checksum.
Accelerated CEF (aCEF): Distributed across multiple L3 engines, each engine stores the most often used entries.
Distributed (dCEF): Replicated across multiple L3 engines. Central engine maintains the routing table and generates the FIB, which is downloaded to each line card. Better performance of the two.
VLANs and Trunks
VLANs are logical networks, they form a single broadcast domain; devices connected to different VLANs will not receive the same broadcasts. A VLAN can have members located anywhere in the campus network as long as VLAN connectivity is provided among all members. L2 switches are configured with VLAN mappings and provide the logical connectivity among the VLAN members. 1002-1005 are reserved.
Static VLANs: Manually assigned, normal range = 1-1005, extended range = 1006-4094 (with VTP in transparent mode or in v3).
Dynamic VLANs: Provide membership based on the MAC of an end-user device by querying a database (VMPS).
End-to-End VLANs: Span the entire switch fabric of the network, are flexible, and offer mobility. Users can be assigned to VLANs regardless of their physical location. Users should be grouped according to common requirements, policies, and access to resources. All users in a VLAN should have roughly the same traffic flow patterns. Not recommended because broadcast traffic is carried all over, allowing a broadcast storm/L2 loop to spread far. More difficult to administrate.
Local VLANs: Should use 20/80 rule, 20% of traffic is local within the VLAN, while 80% is destine for a remote network. Local VLANs are designed to contain user communities based on geographic boundaries. Allows L3 functionality to handle inter-VLAN traffic, provides redundancy with multiple paths, scalability, and easier management.