I had a fun Friday. I woke up at my usual time (6am) and dove into my usual routine. While the wife is in the shower I jumped on the computer and my browsers default home page (www.carknee.com) wasn't coming up. humm. Checked a different site on the same server (www.rswarrior.com) and that wasn't responding. Can I ping it? Nope. *dang* I tried pinging a couple other servers in the facility, but they weren't responding either. Double dang. I went to work and figured a bad switch was being replaced and it would be back up when I got to the office.

At 7:30am, still no server so I called the web hosting facility... no answer and no voice mail!?!?!? At 8am I called the owner on his cell phone (it is nice to work with small companies), but no answer. At 8:30am I called again, but this time I got an answer! He said he can't get into the building because the Arson squad is still investigating. SAY WHAT? Apparently the web hosting facility had a gas explosion.

My co-located web server is now off the grid sitting in a war zone. Initial thoughts went as follows.

  • Shit.
  • Was there, and if so how bad was the, fire?
  • If it survived the fire, would there be water damage?
  • If there was no fire or water damage, would it survive the pressure of the explosion?
  • If it survived all that, would it be ok in a room that could have collapsed in on itself?
  • Oh shit, I'm f'ed.
  • When were my last backups? Where can I host a backup server before end of business on Friday?
  • Oh shit... I'm really f'ed

So, around 10am I drove to the web hosting facility (again, nice to work with small local companies). The owner was there, but he had yet to go in. The scene was covered with the local Gas companies workers and the fire marshal still wasn't letting the owner in. I left around 10:45 since there was no way I was getting to get what ever was left of my box.

GasExplosion_1 GasExplosion_2 GasExplosion_3

Numerous conversations with owner over the course of the afternoon, but he was still not allowed in the building. I showed up back at the facility around 5:15pm and he still wasn't allowed in. An asbestos removal team was finishing up and he said I can get in around 7pm. Rather than showing up at 7pm I figured I would just call this time... he picked up and I could sense the relieved tone in his voice! He said, "oh, you didn't know? Your server came back online at 6:30!" Wow. He boarded up the blown out windows and doors, brought in ancillary generators and got some of his facility back up and running. His main generator was out of commission since it ran on natural gas.

I was down for about 12 hours total, but what would I have done if my box hadn't survived this? I had a database backup from a month ago. I had critical web applications on my development PC that I could have deployed, but most everything else would have been gone. Years of data up in a cloud of smoke.

Why don't I have a disaster recovery plan or backup strategy in place you ask?

  1. I never expected my facility to blow up. I have RAID'ed drives and figured worse case a drive would go bad on me. Two can't go bad at once, right?
  2. I did have a backup server running at my home. However, I never stuck to my manual backup process and I ran out of disk space!

Lessons learned:

  1. Shit happens! You know that worse case scenario you think about, but figure it can't happen? It happened.
  2. Backup everything --everyday-- and keep it offsite! RAID is great, but it won't save you if your entire server blows up (or gets stolen -- that's another story). Amazon S3 time perhaps?
  3. Stay friendly with the people at your web hosting facility! I was one of the first servers he powered on. If I wasn't onsite, and calling him regularly, my server might have been sitting under rubble for the weekend.

Someone posted this as a joke when they heard of my misfortune

An unwritten rule of thumb in IT management:
if an organization has no database backup
and no database backup strategy,
and no database backup contingency strategy
then the organization has
no organizational intelligence.


Live and learn I guess.

One of my web servers had been having all sorts of problems over the last few months. It turned out to be the RAID controller, but that is another post. Because of the problems we were having I wanted a simple way to get a notification of when the site we were hosting went down. I didn't want anything fancy, just a simple email when the sites stopped responding. Here is what I found...

First, I downloaded SystemMonitor from MSDN's Coding4Fun. This is an extensible application that can be run in your system tray and notifies you of events. I then wrote up a quick "Monitor" to send a request to the sites. The monitor makes an HttpWebRequest to a site looking for a specific file. Any errors returned (404/500/etc...) will fire off a notification from the SystemMonitor. If a status code of 200 is returned, the monitor will check for "site running" in the content of the page. If "site running" is not found then a notification will be sent.

The advantage with this method is that you can specify the URI of the page you want to test. This page can be .txt, .htm, .asp, .aspx, .whatever. So, if your web server is ok, bu[tags:Web Server, Programming]t your database server has a habit of going down then you can point the SystemMonitor to test for a  page that will test your DB connection. As long as the response status code 200 and the page returned "site running" in the content of the page it will not fire off a notification.

To install the Site Monitor you need to download the SystemMonitor from Coding4Fun. Once downloaded, add my SiteMonitor (and any needed references) to the project and recompile. Once recompiled you just need to add the settings to the App.config file. The config section, as shown below, is pretty self explanatory. The one setting that may be a little abstract is the localIP variable. The machine I installed this on has multiple IP addresses assigned to the NIC. One has access to the Internet and the other doesn't. By setting the localIP variable you can control what IP the monitor will bind to when trying to make a request.

      <monitor runFrequency="00:10"  type="SystemMonitor.Monitors.SiteMonitor,SystemMonitor">
        <settings>
          <setting name="url" value="http://www.example.com/Utility/uptime.aspx" />
          <setting name="localIP" value="" />
          <setting name="proxy.enable" value="false" />
          <setting name="proxy.ip" value="192.168.0.1" />
          <setting name="proxy.port" value="8080" />
        </settings>
        <notifiers>
          <notifier type="SystemMonitor.Notifiers.MessageBoxNotifier,SystemMonitor" />
        </notifiers>
      </monitor>

The Code for the Site Monitor is show below, and available for download here.

class SiteMonitor : MonitorBase
    {
        string _url;
        string _localIP;
        bool _enableProxy = false;
        string _proxyIP;
        int _proxyPort = -1;

        #region Binding the WebRequest to a Local IP with a Delegate
        public delegate IPEndPoint BindIPEndPoint(ServicePoint servicePoint, IPEndPoint remoteEndPoint, int retryCount);
        private IPEndPoint BindIPEndPointCallback(ServicePoint servicePoint, IPEndPoint remoteEndPoint, int retryCount)
        {
            if (retryCount < 3)
                return new IPEndPoint(IPAddress.Parse(_localIP), 0);
            else
                return new IPEndPoint(IPAddress.Any, 0);
        }
        #endregion

        public override void Execute()
        {
            HttpWebRequest request;
            HttpWebResponse response;

            string errTitle = String.Empty, errMessage = String.Empty;
            
            try
            {
                request = (HttpWebRequest)WebRequest.Create(_url);
                request.Timeout = 1000;
                
                // bind to a specific local ip
                if (!String.IsNullOrEmpty(_localIP))
                    request.ServicePoint.BindIPEndPointDelegate = new System.Net.BindIPEndPoint(BindIPEndPointCallback);

                if (_enableProxy && !String.IsNullOrEmpty(_proxyIP) && _proxyPort >= 0)
                    request.Proxy = new System.Net.WebProxy(_proxyIP, _proxyPort);

                response = (HttpWebResponse)request.GetResponse();
                HttpStatusCode status = response.StatusCode;

                if (status != HttpStatusCode.OK)
                {
                    errTitle = "Failed Contacting Web Server";
                    errMessage = String.Format("{0} Returned Http Status Code: {1}, {2}", _url, (int)status, status.ToString());
                }
                else
                {

                    StreamReader reader = new StreamReader(response.GetResponseStream());
                    string content = reader.ReadToEnd();

                    // LOOK FOR [SITE RUNNING]
                    if (content.ToLower().IndexOf("site running") < 0)
                    {
                        errTitle = "Web Server Returned Invalid Response";
                        errMessage = String.Format("{0} Returned Invalid Response", _url);
                    }
                    reader.Close(); reader.Dispose(); reader = null;
                }

            }
            catch (WebException ex) // this will catch 404's
            {
                errTitle = "Failed Contacting Web Server";
                if (ex.Response != null)
                {
                    HttpWebResponse r = (HttpWebResponse)ex.Response;
                    errMessage = String.Format("{0} Returned {1}, {2}", _url, (int)r.StatusCode, r.StatusDescription);
                }
                else
                    errMessage = ex.Message;

            }
            catch (Exception ex)
            {
                errTitle = "Failed Monitoring Site";
                errMessage = string.Format("'{0}' failed with an exception: {1}", _url, ex.Message);
            }
            finally
            {
                response = null;
                request = null;
            }

            // fire off the Notifier
            if (!String.IsNullOrEmpty(errTitle))
                Notify(errTitle, errMessage);
        }

        protected override void Initialize(Dictionary<string, string> settings)
        {
            _url = settings["url"];
            _localIP = settings["localIP"];

            _proxyIP = settings["proxy.ip"];
            try{_enableProxy = bool.Parse(settings["proxy.enable"]);}
            catch { _enableProxy = false; }

            try { _proxyPort = int.Parse(settings["proxy.port"]); }
            catch { _proxyPort = -1; }

        }

        public override string Description
        {
            get { return string.Format("Testing '{0}'.", _url); }
        }

        public override MonitorType MonitorType
        {
            get { return MonitorType.Scheduled; }
        }

        public override Icon Icon
        {
            get { return Properties.Resources.PingIcon; }
        }
    }

Defrag your server

January 18, 2007

It seems that my fancy dancy server needs to run defrag every couple days. The site is running a well visited site (and some other not so visited sites) and is creating 300MB Log Files from IIS, as well as user uploaded files, etc... All of this is really fragmenting my nice and speedy data drive (actually, 2 drives in a RAID 1 configuration).

I hear a lot about the Windows Defrag program is an outdated POS and that you should  consider using DiskKeeper's newer products. It seems reasonable since Diskeeper is constantly running making sure your files aren't fragmented, but is it worth the $300 MSRP?

Does anyone have any thoughts to this?  [tags:Server]

Drivers on a Floppy?

January 15, 2007

A little rant here...

I purchased a Zonet GigE NIC (ZEN3301E) card a couple months back and finally got around to installing it. Installing a NIC shouldn't be too difficult, right? I threw it in, powered up the PC (actually, my backup web server) and let Windows (Server 2000) configure it. Windows didn't have the driver for it, but the NIC came with a floppy disk with the drivers.

Side Note:  I am not a big fan of floppies. In fact, I can't remember the last time I used one successfully! It seems that every floppy I get is busted, or all of my floppy drives are busted.

Well, my backup web server doesn't have a floppy drive installed on it. (It doesn't even have a CD-ROM drive in it.) Off to Zonet's web site to download their latest drivers. Guess what. No drivers listed. (They have since posted them). Now, being a person who has built many PC's for people I knew that chipset manufacturer's drivers typically work. I looked at the chip on the NIC and downloaded their driver and it works no problem.

If you are a manufacturer, try to post your drivers before you start releasing your products to major manufacturers!

[tags:Rants, Web Server]

Server Configuration: FTP

December 19, 2006

In the spirit of trying to keep writing, I figured I would write about by new found love of FTP. Well that is a bit of an exaggeration, but on my previous box I used the FTP service built into IIS 5. What a nightmare! Don't get me wrong, the service worked fine and it was stable. The problem came out only when I needed to give someone else access. You need to create an account on the computer, deal with permissions, blah blah blah. I felt it was overly complicated.

In walks in FileZilla FTP Server. The thing is great. Installation and set it up are a breeze. The real advantage comes into play when you need to add users and groups! You set directory alias' easily and things are great. It has all the bells and whistles such as Logging, IP blocking, passive/active, etc... You can even go so far as creating a web front end to manage users; how simple is that?

I highly suggest you check the program out if you are running a single server with no active directory to manage your users.

[tags:Web Server]

I have been running my own web server for a few years now and like everyone who does this I have my own ways to do things. One of the things I wanted to revisit, before setting the new machine out into the wild, was my typical directory structure. While this may seem like a trivial concept to many, things can get out of control quikly when you start throwing in many domains, sub domains, Log files, databases, etc...

Here is my current directory structure.

X:\Websites\
  |--domain.com
     |--subDomain
         |--Logs
         |--Web
         |--Any Other Directory Needed (client files, a private directory (data), etc...)
     |--subDomain2
         |--[see above]

So far I am liking this setup! I could have taken it up a level and gone so far as:
X:\Websites\
  |--com
     |--domain
        |--subDomain
           |--[see above]

I felt that for the most part any domain where I am running the .com, the .net will be the same content. My chosen setup would probably not work for everyone, and the second methos may be how [cs] runs things since their CommunityServer.com and CommunityServer.org are two separate sites.

I don't know how web hosts setup their servers, but their requirements would be MUCH different than mine. The other thing I do like about my new setup is how I can configure FTP to give access to the website and logs easily! I will get into FTP setup in another post.

<<DirectoryStructure.gif>>

DirectoryStructure.gif

New Server

December 18, 2006

In my first post, all of an hour ago, I mentioned that a new server was one of the reasons I renewed my interest in blogging.

Just to give you an idea of what I was running:
It was a "homemade" computer:
CPU: Celeron (2GHz maybe?)
RAM: Started at 512MB and was upgraded to 2GB (motherboard max)
Drives: 2x 40GB IDE drives running RAID 1
OS: Windows Server 2000

That machine served me well! I made it for under $600 about 3 years ago and it worked great up until I started trying to run [cs] sites, DotNetNuke sites, etc... all of the ASP.NET Web applications really started bogging the machine down. I didn't see myself getting away from [cs] anytime soon, so it was time for an upgrade!

Here are some specs of the new server:
Dell PowerEdge 2900
CPU: Dual Core Xeon Processor 5130 4MB Cache, 2.00GHz, 1333MHz FSB
RAM: 4GB @ 667MHz
OS Drives: 2x 36GB SAS, 15K RPM running RAID 1
Data Drives: 2x 300GB SAS, 10K RPM running RAID 1
OS: Windows Server 2003 R2 Standard Edition
* Dual onboard Gb Ethernet
* Redundant power supplies

I must say that the sites that run on the server are much happier now! Here are a couple of them:
http://www.rswarrior.com
http://digitaldjpool.com
(I will get into some site details in another post.)