August 02, 2018
Nobody likes network outages, but sometimes the root cause can be tricky to find, especially with newer technology. Recently, I ran into a difficult challenge with Cisco Meraki Firewalls.
A client was experiencing network and internet outages after deploying their Meraki device. Since the firewall on the Meraki doesn’t follow the traditional out-of-the-box configurations that I am used to seeing with Cisco Adaptive Security Appliances (ASA’s), it took quite some time to troubleshoot the issue, and we eventually needed a joint phone call with representatives from both a traditional Cisco Technical Assistance Center (TAC), and from a Cisco Meraki TAC to resolve the issues we were seeing.
Traditionally with a Cisco ASA, all clients and physical (MAC) address associations are kept in the Address Resolution Protocol (ARP) table, whether the ASA is used as a default gateway for one or more subnets or if it is simply the next hop from the layer 3 device downstream. Best practices would suggest that all Virtual Local Area Networks (VLANs) and broadcast domains should be segregated from the firewall, from the DMZ’s, and from Guest Access, on a downstream layer 3 device. Restricted access from these areas to the main network would terminate directly at the ASA firewall interfaces.
With a Cisco ASA, there is no need to configure the firewall to track clients by IP address instead of MAC address. We found that this does not hold true for Meraki Firewalls.
Meraki Firewalls have a setting that changes the way it tracks clients. The default configuration for a Meraki is to track clients by MAC address. It assumes that the firewall is the default gateway for all VLANs/subnets in the network. What does this mean specifically for traffic trying to get to the internet? In a network where the Meraki firewall is part of the same broadcast domain and serves as the default gateway for all MAC addresses on the subnets, it treats clients on the network the same way a switch does. It maintains an ARP table but clients are sent by MAC address to the next hop device and it shares MAC address tables with the downstream switch. See the Meraki Firewall online documentation for more details.
Let’s assume though, that you don’t follow the default configuration described above, and decide to do this in a more traditional setting. The only traffic sent to the firewall would be traffic that needs to go to the internet or traffic that crosses subnets from the internet network to a protected network like a DMZ. This would suggest that you have a layer 3 device behind your firewall managing the default gateways for all of your internal subnets along with inter-VLAN routing. In order to get to the internet, the layer 3 device would have a gateway of last resort that would point to the inside IP address of your firewall. This internal IP address would be a unique subnet all its own, used specifically to route traffic to the firewall and out to the internet or over to the DMZ.
At this point it is important to review how MAC addresses and IP addresses are handled in a routed network. When a destination IP address is sent to a default gateway, a routing decision must be made. During that routing decision, the source and destination IP addresses are kept, but the source and destination MAC addresses are changed. When a layer 3 device routes to the next hop device, the source MAC address is stripped and a new source MAC address of the directly connected interface is added before it is sent on. Because layer 3 deals in IP addresses, a full MAC address table of every client is not kept, only the directly connected MAC addresses are stored in that table between layer 3 devices.
Back to Meraki firewalls. In the more traditional setting described above, there is a layer 3 device behind a Meraki and the Meraki is kept in its default configuration, which is to track clients by MAC address. That means every time a routed packet is sent to the Meraki it sees only the MAC address of the downstream router, not the actual client that sent the original request.
Now let’s assume that you have Advance Malware Protection (AMP) enabled on the Meraki. All of the traffic that is sent to the firewall comes from the same MAC address. Should there be any kind of Malware event on the network, the firewall is going to recognize it as a single source. Eventually the AMP will shut down that source. As a result, all traffic coming from the layer 3 device will be shut down as well, causing a network outage or an inability to reach the internet from within the network.
In order to prevent this type of issue it is important to change the default setting on the Meraki from tracking clients by MAC address to tracking clients by IP address. Once this setting is changed, the Meraki firewall will be able to identify individual clients from the downstream router by their unique IP address and track their activity that way. Now the AMP will be able to identify users and clients not by the MAC address of the downstream layer 3 device but by their client IP address and populate the ARP table correctly. Should an event occur that triggers the AMP to shut down a single device, it does not cause a locally global outage of the entire private network.
This is in no way a standard configuration requirement for those who are used to working with a traditional enterprise firewall. Not just Cisco ASAs, but Palo Alto and Check Point as well, to name a few. Meraki is a great product and has a growing market in the mid-size company base, but there are some non-standard design considerations that need to be made before deployment in order to avoid network issues like this one.