Today I had an experience that is not too far off from experiences I’ve seen and experienced in the past, though, not directly caused by me. As a typical (non-ma’ pa’) business setup, it is perfectly normal to have 2 or more domain controllers with DNS on a Windows based network. You ALWAYS want to point your DNS settings in DHCP to your domain controllers (Note the s in that last word). The experience was of maintenance on a server that caused a partial network ‘outage’ as it wasn’t able to find a second DNS server when maintenance was started on the Domain Controller 2. The problem came when, out of our 4 domain controllers, the one I happened to be doing maintenance on was the only one in use by the network…… (and mind you this is a 2008 domain controller, not our Server 2016 domain controller, yes that is an 8 year old server).
This is why redundancy is SO important. You should ALWAYS specify 2 DNS servers if you are in charge of a network. And you should be very conscious of what servers could be decommissioned in the future as well. You should feel a nasty feeling in your gut until you have a second one in there. You should always have more than 1 uplink to your VMWare hosts preferably on a secondary network card. Hardware failure isn’t super common these days, but it does happen. If your company is larger, you want to be sure you have HA (high availability) running on your core network equipment. You want to pay for a second internet provider.
Whether it is a network, a virtual machine, or something else that is business critical, PLEASE make sure you have redundancy built in to prevent issues like this. And if you are the one performing maintenance, you do always want to go over a mental checklist of what a server is performing for the business. If you are also controlling the network, you should NEVER have holes as giant as specifying only 1 DNS server in your network.
Please note, I was not responsible for the network in this particular scenario.