Intro: Broadcast Loops v Broadcast Storms
Wherever there is a broadcast domain there exists the opportunity for some idiot to create a loop. It happens in the simplest of environments where a misunderstanding or mis-configuration creates a logical loop where BUM traffic (broadcast, unknown-unicast, multicast) is allowed to go round and round like a merry-go-round. Except this ride doesn’t put smiles on anyone’s faces.
If you delve deep into L2 loops (other loops are available) you find there is some technical complexity involved. They are often called broadcast loops even though it isn’t always Ethernet broadcast frames that are the culprit. Any type of BUM can bring a network down given the opportunity.
I think every network engineer should create a loop in a lab environment and observe precisely what happens. It isn’t what you expect. Taking two switches with a blank config and putting two wires between them is normally enough. However, to really understand the phenomena and to be able to do something about it, you need to add some clients/uplinks to the lab so you can see the effect on users rather than enjoy the disco effect of the port lights. The first thing you’ll notice when you plug in that second link is that nothing really happens. This comes as a surprise to some who expect network death instantly. If you do this in a live DC you do get a more instant effect. In some configurations nothing will happen at all if you do nothing further leaving the inexperienced engineer to think they are so rubbish they can’t even break a network. #selfconfession
In this basic setup you have a physical loop with vlan 1 able to flow in a loop between the two switches. So you have one part of the cause. Assuming the switch doesn’t have a loop prevention mechanism in place it is fair to question why the solid activity lights associated with broadcast loops aren’t evident. You’ll see the occasional flash as the switches protocols are put to work like LLDP. The reason no loop exists is that no BUM traffic is present in this network that could spin round and around. The only traffic being transmitted in this lab is from the switches. Things like CDP/LLDP are broadcast in nature but they are also special cases that a receiving switch will accept but then destroy. So in some way there is a loop prevention in place per protocol. So even without classic mechanisms specifically designed to stop loops a basic loop will sit there for ever without much more than the occasional flash.
This is a key point: You can have a loop without a broadcast storm. The two are not implicitly linked.
With this lab you could then assign an IP to one of the switches. This gives that switches the power of ARP. At this point you might still see no storm. This is because the switch has no reason to ARP. If you then try and ping an address on the same subnet it will try and resolve the address with an ARP request which is a broadcast packet sent out of every interface in the VLAN.
It is fair to assume that when you press enter for said ping command that the reward of a storm should come instantaneously. Most of the time this isn’t the case. You’ll see no storm. This is because what happens under the hood is that the single ARP packet is transmitted from the source switch to the other switch, which then floods it out of every interface (makes one copy) and sends it back to the source. And so on. and so on. So now you have a physical loop with a logical loop configured and for the first time actual traffic that is looping. You’ve achieved a broadcast loop. You haven’t achieved a broadcast storm.
At this point you would be forgiven for giving up and heading out of the lab for a brew. This isn’t a bad idea. Take five and return to the lab and you might get a surprise. Someone has done something that has created the storm you’ve been chasing all day. All the connected interface lights are flashing constantly and switch logs are full of complaints.
What has happened is that the tiny amount of BUM traffic that was generated with the ping has multiplied by the normal behavior of the switch as the frames go round and round and are flooded in all directions. After some time the storm grows and eventually saturates the links. This is another good lesson, some broadcast storms aren’t evident immediately.
Classic Solutions
It is seen as a bit of a filthy word in modern data centres but Spanning Tree should be understood by everyone in the same way as everyone should learn the details of why the world wars start. Understand STP so that you don’t end up using it.
STP at its very basic level is a technology that lives in the software of network switches. It aims to create a loop free topology by sending information between network devices that enable each one to block/permit traffic on each of its ports. Entire sections of basic networking courses/exams such as Cisco CCNA are devoted to it. It has existed for decades and in theory should be a standard that is completed and faultless.
It isn’t. Partly due to vendor interpretation partly due to the complexity that comes with the various extensions (e.g. MSTP) and partly because of the intrinsic limitation of an in-band control mechanism, the technology is the cause of as much grief as it has saved. In a simple, single vendor environment it can work well. It can allow path redundancy without the risk of loops. Once you add different operating systems you run into situations where the assumed standard breaks down.
One example I observed was when a network of the same switch type was playing happiliy together and when a small switch with a different OS was added, the change in STP priority assigned by default causes the whole ‘tree’ to re-converge. Other failures occur when single devices have technical issues such as CPU overloads which lead to the communication path breaking down and the flapping of paths. STP introduces security vulnerabilities which need to be dealt with by additional switch security features (BPDU guard) which in themselves could introduce a bug or be misconfigured.
However, this isn’t the main problem with STP. The main issue that most tight fisted network engineers observe is that if you have two possible paths and STP correctly permits traffic on the preferred one, you have half of your paths not in use. Twenty years ago this was acceptable in the same way as we accept many active/passive technologies today. But times have changed.
It is worth mentioning that a lot of designs include protection from loops even though they don’t contain any physically. Within a tree structure many engineers will choose to employ some protection against a colleague connecting wires in the future that cause a loop. STP can be used for this and loop would be prevented if configuration was spot on. More simplistic mechanisms exist on most switch models which are often called loop protect. On Comware this involves a propriety L2 frame being sent with the base MAC address in the payload. If the switch receives said frame on another port it knows the loop exists. This works well. Until it doesn’t. One issue with this mechanism is that it requires you to choose between permanently disabling the port upon detection or to shut it down for a set time period. The former kills a part of the network and while DC wide disaster may be prevented, there is still a partial outage. If the port is automatically enabled after thirty seconds then problem is that twice a minute you have a mini storm until such time that someone notices the strange behavior.
A different approach
What if we allow the BUM storms?
When presented with technology that stops a bad thing happening it is intuitive to want to stop the bad thing. But maybe that isn’t the best approach. If when a loop occurs, and BUM traffic rises to storm level it might be possible to permit that traffic and to not block and interface.
If every interface within a DC had BUM suppression set then under this circumstance all non BUM traffic would carry on as normal. Broadcast suppression etc tend to act on the physical interface and act like QoS, dropping packets that exceed a threshold (in this case percentage of the physical bandwidth).
There would be no immediate effect on UDP/TCP sessions if this feature was invoked. A maximum of 5% for interfaces would mean that under good conditions all BUM traffic passed as normal (except where there are strange reasons for there being a very large amount of this type of traffic). Under loop conditions the storm rises to the point where 5% of interface bandwidth is saturated (i.e. not much). a lot, but not all, BUM traffic is dropped. SNMP traps are sent to indicate a problem which because of the clever engineering you have already done are set with the highest alert category and thus trigger immediate action.
When testing this in a small DC my concern was that while immediate disaster would be averted things like ARP & MAC tables would fail leading to eventual disaster. My observation is that under storm conditions with suppression MAC tables don’t change as the switch hears all the same unicast traffic. ARP tables will persist for several minutes depending on settings and even then an ARP exchange may be one of the lucky packets to make it through the storm. Remember the answer to an ARP request is a unicast L2 frame.
Where this setup fails is in networks where L2 multicast makes up a lot of the traffic. Or when there are non typical applications where there are a lot of unknown unicast packets flying around. Also in corner cases where, due to some other events, a large amount of broadcast is occasionally required.
Conclusion
This approach is at first glance a little difficult to swallow for the traditionalist. The idea of turning off perfectly good technology seems at first odd. However it can be tested in a lab (or a full DC in your last week before a new job). If the network has the hardware that allows this approach and there are no corner cases where loosing BUM traffic for a period would be a big issue, then this is worth considering.
To set suppression at 5% on comware7 switches using the following commands:
broadcast-suppression 5
multicast-suppression 5
unicast-suppression 5
Constantpinger 
Leave a comment