Dreamersi Server Down due to the system trouble

At around 4:30pm on April 7th, 2020, it was reported to me that email was not working. Soon after, I began receiving reports that our web servers were responding very slow. This is what went on behind the scenes for the next 9 hours.

We started searching for the problem, Internet traffic was unusually high which caused a delay in everything we did. At first we suspected that something went wrong with the DNS server - we have 3 DNS servers. We shut down No. 1 hoping it would result in No. 2 and No. 3 resolving the issue. Unfortunately, after this was done nothing was improved.

Next, we suspected some kind of attack was happening. This is called a DDOS attack targeting our DNS server. We have 2 primary Internet lines coming to our datacenter. We found 1 suspicious IP that was connecting to our DNS, we blocked that IP using the Firewall but nothing changed. We asked our backbone provider to block the DDOS attack, however they did not see much traffic coming into our datacenter. The traffic problem was inside, not from the outside. So we began checking the health of the Router, Firewall, and Switches.

At the same time, we decided to look at our internal servers. Is it possible that an internal server was attacking our DNS server? By this time we had brought in most of our technical staff to see what was happening. At this point, we still had no idea what was happening.

Some of our customers in Japan were waking up to see that their email and web were not working. We had been receiving many calls from customers in the US, and now we had people calling us from overseas. We were not able to announce the progress or problem to the outside world because our datacenter was not operating.

I had decided to just sit in my room and let smart people figure this out. We had people working at our HQ Network Monitoring Center as well as our Seattle Datacenter. It had already been over 7 hours since this issue started. Since then, our staff had narrowed the problem down to the DNS servers. We then started rebuilding a new DNS server. At the same time, 1 of our best Server Engineers started to work on the down DNS server.

He noticed the configuration file on the DNS server was blank, which caused all of the traffic to be unresolved and bounce all over the computers. He was able to modify the configuration file with the appropriate information. When he did that, we started to see traffic slowly normalize and people were able to access their web servers. Emails began to be delivered, but it took another 2 hours to go back to normal.

We started investigating why the configuration file went blank. The report came back but it was too technical for me to understand, but I am sure our people will not make same mistake again. Now we know what it means to be an essential service. There are so many people and businesses relying on us to provide service.

I am sorry for what happened. If you allow us to take care of your Internet web and email, we will make sure we provide the best service we can. This is a brief explanation of what went on during the outage.

Kenichi Uchikura
President
ken.uchikura@pspinc.com
Pacific Software Publishing, Inc.
https://www.pspinc.com

Pacific Software Publishing, Inc. 1404 140th Place N.E., Bellevue, WA 98007

PSPinc Creates Tools For Your Business

Pacific Software Publishing, Inc. is headquartered in Bellevue, Washington and provides domain, web, and email hosting to more than 40,000 companies of all sizes around the world. We design and develop our own software and are committed to helping businesses of all sizes grow and thrive online. For more information you can contact us at 800-232-3989, by email at info@pspinc.com or visit us online at https://www.pspinc.com.