September 1995:
VectorStar Communications was started as a web design business. We made websites for customers of City-Guide Internet Services. Our first server, Vector, was comprised of an AMD 5x86/133 with 32mb of RAM and a 1GB drive, running Windows NT server 4 and IIS. I think we ended up making 5 websites: VectorStar, City-Guide, one for a local Hungry Howies store, one for Josh's Dad's business, and one other which nobody can seem to remember.
At this point, we gave up on a web design business, though it did last long enough to get Benny and Josh through a big school project on business.
It sat around, sucking up 5 static IP's, doing practically nothing but being an idle NT server ("the only non-intel box in my garage!" as Charles called it), occasionally being used for test ideas.
Laird met Mike Solomon, who lived down the dorm hall from him. Mike had done some web design and knew some people looking for space to work on websites. He became a major member of the team for about a year.
Mike and Laird were sitting around, BS'ing like any typical day, when Mike mentioned the idea of making VectorStar a free service for anyone, and not just friends of ours. The idea sat well, but Laird knew that it would be completely unmanageable with the current system: Windows NT 4, IIS 4, and MailMAX mail server. Frank had played around with Linux a little bit (Slackare 3.2 and 3.4 I believe), got rooted and fdisk'd once, but more-or-less gave up on using it for a desktop OS. Laird mentions the public free hosting idea to Frank, who thinks it's "straight". Laird tells him that they would need to redo some things to make the system manageable. Frank is really eager to try Linux (I think Slackware 4 had either just been released or was about to be released) but Laird knew they weren't ready to actually run a UNIX system that does anything that could be considered useable. Instead, he got off IIS and started using Apache (1.3.4 or 1.3.6 I think) on NT. We stopped using Berzerker for production, and just did testing with him. VectorStar started accepting new users, by email-request to Laird.
Around April 10th, Laird's dorm ethernet connection went dead. Since he no longer worked for the University, he had no protection from being shut off for totally violating school policy (running DNS servers in particular). VectorStar was down for about 2 weeks while Laird moved nameserver records to a few IP's he could use at work. VectorStar ran from there (multiple T3's) for about 5 months. Those were the really good days. Berzerker came home and became an abuse-to-death machine.
After 7 weeks of extremely intense installation testing, tweaking, securing, and documenting, Frank and Laird stick a new hard drive in Laird's computer and start building it to be a complete replacement for Vector's NT setup. After about 2 weeks of getting Slack 4 clean, tweaked, secured, and building Apache, BIND, proftpd, and qmail, we were ready to run. One Sunday evening, Frank and Laird go to Laird's work and gleefully (rather scared, actually) swapped the hard drives, copied user data, and cheered REALLY loudly when everything worked perfectly. We immediately gave all of our users shell accounts. Laird made the signup system a lot better than it was: a web form, filling in username, real name, email address. It just emailed him, because he was still pretty new at perl. Dave Mead was a co-worker of Laird's back in the computer retail days, and he was a major user of VectorStar. We were running everything on one Pentium 133 and 32mb RAM with a (new) 4gb drive. Although it ran ten times as efficiently as it did on NT, we knew we needed to upgrade this machine. Dave contributed a nice ATX case, a BX Pentium II motherboard, 64mb RAM, an 8gb IDE drive, and I believe a Celeron 333 (original, no cache) which Frank commandeered for his own machine (the rest of us had PII-350 workstations OC'd to 467 while frank was still pushing his p200mmx). I bought an ECC Pentium II-266 (quite a rare chip!) and we once again made a late-sunday-night upgrade party with Vector. VectorStar reached something like 80 users around this point in time. We had some sort of announcement when we crossed the 75-user milestone.
Laird's Christmas-day publicly-posted rant about his job was the dawn of the perilous, funny, tactless, filthy messageboard of VectorStar.
When VectorStar finally came back up, not all of its 100 users were still around. Many figured we were done for good and never came back. The task of getting new users became really difficult, since the world basically thought we were dead. With IntNet, we had 5 IP's to play with, so Berzerker came back with a vengeance. It was basically all of the old Vector (Pentium 133), but with one IDE 3gb and one SCSI hard drive. It ran Linux, FreeBSD, and OpenBSD and was used heavily for perl development. Laird learned MySQL. We also got some more servers:
Laird had taken on a little SQL/perl development project to make money (he wasn't working since he quit City-Guide), and used/abused the oblivion out of Spooky for that project. That project paid for VectorStar's operations from about March through July. Things got tough in July, and the still very- appreciated help from Pogo and Forge kept VectorStar going. In this same month, Jason Daunt made his debut into the bizarre world of VectorStar. By June he was officially Staff. By October he was officially co-owner.
Vector got upgraded to Differential Ultra SCSI. two 4gb drives, mirrored. Crappy overheating Seagate Barracudas. We also retired spooky to being a firewall for Frank's parents around this time. Pogo, Jason, and Laird acquired Mexicanhouseboy, a SparcStation 5/170mhz with 128mb RAM and a QuadFastEthernet card. In other words, it was a geek's dream toy. We had no idea what to do with it yet, but since Laird was familiar with Sparc hardware and OpenBSD, we made it a firewall for Jason to use. We were up to about 80 users.
Server Upgrades: We also registered "vectorstar.net" in november.
Mexicanhouseboy was retired immediately after that migration. It just didn't have the power needed to handle the 175-and-growing userbase. Berzerker was replaced with Chester, a single Xeon 400 with 128mb RAM. It also took over the SQL services previously running on "holocaust". Our users page had historically listed every user of VectorStar. At this point, we implemented an opt-in users page.
VectorStar Networks serviced over 200 users. We had a lot of problems with shell users, and decided to stop user signups until we could address some of the security troubles we were having.
On July 15th (Laird's birthday), Laird went into a perl frenzy and finished the shiny-new new user signup system he had been working on. Signups resumed on the 18th. VectorStar Networks saw 50 new users in 2 weeks.
VectorStar Networks saw more users in August than it had in total. 178 new users signed up in August.
We moved all vectorstar.net services (FTP, SQL, webcams) on Chester to Apathy. Chester got a very-needed fresh wipeout of the OS. It had been running the same botched and hacked up Slackware install that ran Berzerker in February 2000. Plans were for Chester to take over operations of VectorStar Communications (vectorstar.com) so we could prepare for some for-profit ventures. We moved the servers on the 23rd, and had 3 days of DNS outage due to some miscommunication with Network Solutions. Services resumed on the 26th.
Both Chester and Apathy were upgraded to 512mb of Crucial ECC SDRAM. This would tide over Apathy until further upgrades were done. Chester got two brand new Pentium II Xeon-450 CPUs with 1mb cache. These upgrades were to prepare Chester to run some high-power services for VectorStar Communications.
Planning began for a BIG network overhaul. The overhaul should bring VectorStar Networks to the point of handling 10,000+ users, as well as new services. It would give a lot of room for VectorStar Communications to grow as well.
We also crossed 1000 users. Work started to isolate and remedy some performance problems seen on the web server, due to the extremely high load.
After 4 months of no Staff Meetings, we started meeting again and discussing all the things that needed work. The decision was made to focus first and foremost on strengthening our existing infrastructure and services before adding new services.
Late in July, we started ordering new hardware for our infrastructure upgrades. After an incredible amount of trouble trying to purchase hard drives for the server, we finally got all of our orders completed. The SSL mail system we put together in April was found to have some serious problems. We redid the SSL mail system from the ground up with much better success. We discovered that the RAID cards we bought for the servers did not work with any of the BSD or Linux kernels available. We quickly wrote a patch for Linux to get the cheap FastTrak cards running. Late in the month, we finally managed to put together a working kernel patch for the SuperTrak card. We ran into more data corruption problems on the ext2 filesystem holding all the home directories. With some very creative work, we recovered about 75% of the data 'lost' in the filesystem. The NIS structure for our new servers got set in stone in September. At the end of the month, another user abused a perl module and completely halted our network for about 4 hours by trying to DoS an IRC server. And that was the end of letting everyone use Socket.pm.
Chester's motherboard fried. The replacement board we bought was dead too. The Dual PII-266 we planned on using was officially deemed worthless, as it took way too long to boot and can't boot from CD (disaster recovery). So we've got 2 out of the 3 new servers completely dead in the water. We had to buy hardware to build two whole new servers. To really top things off, the main SCSI hard drive in Apathy (the only production server at the time) ground itself to a halt. However, it took 3 days to grind down, and partially corrupted our backups before we noticed that the drive was about to fail. Nonetheless, we switched to the backup drive, fixed all the corrupt file permissions we can find, and call it a loss. Right before the end of the month, we got all the new hardware in for the servers. Hardware was built and final buildout begins. RAID problems left and right. Lots of RAID problems. We didn't end up figuring out until January that we just didn't have a big enough power supply in the file/backup server. We also bought our first-ever piece of Cisco networking equipment. To top it off, it was a brand-new, new-model Catalyst 2950-24 switch! We also got all of the core systems and services running on the new servers. The new web server config (using suexec for PHP) was completed and worked exceptionally well. The new all-SSL mail server was run through full simulation testing and performed perfectly.
We go buy a new, bigger, badder power supply for the file server and magically all of our tape backup quirks and RAID array deaths go away! We were running out of power. On February 15th, we go live with the new servers! Finally, the new servers first dreamt of in November 2001 make it into production! ... And the problems started to appear quickly. First, the DNS servers didn't like the switch. Then the RAID card started acting funny, so we got a new RAID card. Turns out the RAID card wasn't the problem - the motherboard had blown its voltage-regulating capacitors. The venerable Vector, 5-year servant of VectorStar, had given up the ghost. Normally we'd rename a machine when it got a new motherboard (our server names represent the "personality" of the hardware), but since it would be such a pain in the rear to rename it, it kept the name while stepping up to a 1.2ghz Athlon.
The power outages/spikes/whatever blew up the batteries in our 3000VA UPS. Because the UPS was sitting up on its side (with the batteries on top), the battery acid trickled down on the electronics and completely ruined the UPS. The UPS problems smoked the board in Vector - not enough to make it outright fail, but enough that it was crashing on a near-daily basis. We ended up cannibalizing Apathy (which was a hardcore, built-up, all SCSI-RAID Dual Xeon meant for running web services) to get Vector running. Jeff & Jeremy (our very gracious colocating hosts) let us borrow their 3000, plus they hooked up a generator for good measure. Finally, stability was ours again!
We experienced plenty of hard drives dying in Vector, and intermittent, unexplainable crashes of Chester. Aside from that, everything went on smoothly and quietly.
After lots of searching for a colocation provider, we got a fantastically generous offer to let us host VectorStar in the corner of a room at one of our users' houses.
That's all that needs to be said. If you weren't watching the news, well, let's just say that Florida got royally smacked around by the forces of nature. We had some downtime due to massive power outages. Two more drives failed in Vector. We were down to about 230 users.
On April 17th, VectorStar once again reached 1000 users!
In June we had to kill new user signups again, due to rampant abuse. People were using VectorStar webspace to host virus payloads that were being retrieved by viruses! Our webspace was also used in phishing scams against a major bank. That kind of notoriety does NOT sit well with us. At the end of October, angstful (web/file/ftp server) died. It was replaced by a rejuvinated Apathy (still dual Xeon) with Apache 2.0 and PHP 5.
Work on the new servers resumed, CODA was cancelled, and an old staff member re-joined the ranks.
A MASSIVE overhaul of the email system took place. The new email system does a lot of anti-spoofing checks and is better behaved overall. SpamAssassin was introduced, doing both automatic system-wide filtering and optional per-user bayesian filtering. Our custom-written mailfilter was also simplified and improved and automatically runs for all user accounts. Users now have the ability to configure what happens to spam messages - they can deliver them to the folder of their choice, bounce them back to the sender, delete them, or any combination. By default all spam is delivered to the user's "spam" folder, instead of their INBOX.
In February, early development started on a potential IP Telephony service. In March, VectorStar moved to a new, probably permanent home.
|