Last night there were three of us in the Chat room when it began freezing up. After logging off to log back in (to see if it made any difference), the entire board was off line, so I guess the server had some kind of problem.

I'm thinking it was JW's fault, but I'm not allowed to talk about the chat room.

Mike

Hi All,

Let me address some misinformation above about this site.  Our recent downtime has been due to three main issues.

1) Last week we had a lot of bad storms in the area.  We are in a heavily wooded area -- if a tree falls in the forest, and no one is around to hear it, it still falls on the utility lines.  Short of running underground cabling for 12 miles or so, that stuff is going to occassionally happen, despite our best efforts.  If someone wants to donate a 3+kW diesel cutover generator and a satellite dish ISP backup service, PM me.

2) There is a known issue with the beta firewall we are using that affects DNS queries.  That should be taken care of later this week with an upgrade.

3) I had to upgrade our version of ESX (VMware; a virtualized server operating system) and couldn't do it at 2-5 am as is my normal maintenance window time, due to commitments at work. I couldn't just vMotion the server to the other box as I had to upgrade the VMware tools and the virtual hardware, which required like 4 reboots total.

As far as the 'plainly ridiculous' 'BS' of it costing around $250 per month to run this site, I'll ignore the insult and break it down for those who choose to complain:  $125/mo goes to the ISP for a 10-down / 3-up line.  The electric, on a separate meter, runs in the neighborhood of $65/mo.  I also upgraded all of our 5-year old equipment this winter to a mix of new and used enterprise-level hardware -- all to the tune of around $6,000 (which, btw, is not yet included on the 'what it takes to run a site like this' page). 

Fully loaded DL380 G4 (around $1200), a fully-loaded DL360 G4p (around $600), a maxed-out DL380 G3 ($400), a new LTO drive and 6 tapes for offline backup ($800 + $180), a new rack ($400), a new UPS ($600), 2 SCSI adapters and cables ($150), a 2TB NAS for online backup($1100), a 24-port gigabit switch ($350) and I'm not even going to enumerate all of the miscellaneous costs either.  Amortize that out over 5 years and we're over $300 a month, not $250.  I have to upgrade the G3 to a G4 sometime soon as well for ESX vSphere 4.0, but I'll do it gladly to keep eveything in top shape. The users here are great with donations, and with the eBay links we usually cover costs, so I don't complain. 

There are 20 sites running in this farm, some for 11+ years.  Many are for other car clubs for whom I provide their site without cost.  I'm just as familiar with the LAMP platform (Linux, Apache, MySQL, PERL/PHP) and run both for a living at a private investment firm.  Windows has been a rock-solid platform for many years, and I've simply found no reason to take the time to port it over to LAMP just because of some anti-MS sentiment out there.  This board software was chosen by a test group of users several years ago.  Its multi-skin, multi-language capabilities and features were far advanced beyond any other board at the time.  As the years have gone by, MegaBBS has become something of a dead platform.  I have talked to the folks at vBulletin and they will be our eventual upgrade.  We can move over most of the posts, but cannot import the photo albums at this time.  I'm not willing to take that loss, so things are on hold until I have the time to work on a conversion plan.

As the father of an 18-month old, and having just purchased a new home, my free time is extremely limited. I have not had any of my cars out in about 2 years now, as often I have to make the choice between doing things for this site or another activity. This year I have even had to take vacation days to sleep after staying up all night to do site upgrades so as to impact the least number of users. I try not to inconvenience folks with downtime (we were still over 99.9% hardware uptime last year, even with the power outage last June and the upgrades. 8 hours downtime in a year is great.)

As the content of this and the other sites is around 1 MILLION posts/pages/archived mailing list posts (not to mention pictures), it is too big to do backups from a cloud virtual server environment on any kind of a regular basis.  After having seen friends' sites simply vaporize when a provider went belly-up, I will not allow that to happen.  That is why we have spent such a great deal on redundant hardware and online/offline backups.  We also push over 300GB up per month between all of the sites (not counting cached traffic from the proxy), so it isn't a walk in the park...

There is a lot that goes on behind the scenes, and I have very little time to do it. I'll try and post the scheduled outages when I can.

-Dave