About the Network

Hardware, Software and Design

Hardware

Rack and Such:
  • 12U 19" rolling cabinet
  • APC SmartUPS 3000-RM 3U
  • Belkin OmniView PRO-8 KVM
  • 8000 BTU portable air conditioner
Networking:
  • 7M/768kbps Business Cable, through Bright House Networks. We used to run through Intelligence Network, a great local provider.
  • Cisco Catalyst 2950-24 Switch
    * Separate VLANs for private and public networks

Servers

gw: (firewall, gateway)
  • 1U KRI GH-188 w/ EMacs 230w Power
  • Tyan Thunder LE-T (S2518) Motherboard
  • Single PIII Tualatin 1.2ghz
  • 512mb Crucial PC133 Registered ECC SDRAM
  • 256mb CompactFlash card for HD
  • Two Intel 82559 NICs (onboard)
  • ATI Rage XL Video (onboard)
  • Running OpenBSD 4.0
vm1: (VMWare Server #1)
  • 1U KRI GH-188 w/ EMacs 230w Power
  • Tyan Thunder LE-T (S2518) Motherboard
  • Dual PIII 800mhz
  • 1gb Crucial/Infineon PC133 Registered ECC SDRAM
  • 3ware Escalade 7000-2 IDE RAID card
  • Two Maxtor 40gb IDE drives (RAID 1)
  • Two Intel 82559 NICs (onboard)
  • ATI Rage XL Video (onboard)
  • Running CentOS 4.x for the moment, but it sucks
vm2: (VMWare server #2)
  • 1U case
  • Intel's "My First P4" server Motherboard
  • Single P4 2.4ghz
  • 1gb PC133 Registered DDR PC3200 RAM
  • 3ware Escalade 7000-2 IDE RAID card
  • Two Seagate 120gb IDE drives (RAID 1)
  • Two Intel 82559 NICs (onboard)
  • ATI Rage XL Video (onboard)
  • Running Slackware 11.0
bulldog: (VectorStar sites, email)
  • 1U case
  • Intel's "My First P4" server Motherboard
  • Single P4 2.4ghz
  • 1gb PC133 Registered DDR PC3200 RAM
  • 3ware Escalade 7000-2 IDE RAID card
  • Two Seagate 120gb IDE drives (RAID 1)
  • Two Intel 82559 NICs (onboard)
  • ATI Rage XL Video (onboard)
  • Running Slackware 10.2
web: (user web sites, ftp server)
  • Virtual Machine
  • 256mb RAM
  • Running Slackware 11.0
god: (backend file, NIS, SQL server)
  • Virtual Machine
  • 256mb RAM
  • Running Slackware 11.0
Last updated 4-3-2007.

Software

UNIX:
  • We love BSD, especially OpenBSD. Eventually we hope to use it on every single server we run. So far we've had issues with NIS support not working as expected, which has delayed our migration toward it.

    For the moment, our big production servers run Linux.
  • As far as Linux goes, we use Slackware. We have used Debian in the past, but if we're going to put up with a Linux we might as well enjoy Slackware. It is significantly lower-maintenance, higher default security and higher flexibility than any other Linux distro.
    We have a bit of a history with Linux: We have written a few Linux Kernel patches in order to get our hardware running (mainly drivers for Promise IDE RAID cards). Our biggest patch is to make the SuperTrak SX6000 compile, boot and work cleanly into the official Linus Torvalds kernel tree. It's not as fast as we'd like, but it works. :)

    Atop the UNIX core, we use a universal filesystem structure, shared across all servers by NFS. We were trying to replace this with a distributed Coda filesystem, but Coda is a royal pain in the ass. Oracle just released some new stuff that may let us do distributed filesystems in the future.
    User accounts are managed on the back-end via custom software and access is granted to front-end servers via NIS.

    DNS: We run ISC BIND 9, since prior architectures of BIND are inexcusably poor and djbdns still seems overly complicated.

    Mail: We run a highly-customized implementation of Qmail. We have our implementation using SSL for every mail protocol, and requiring SMTP AUTH in order to send non-local mail. Our latest implementation also includes serious anti-spam capabilities built in. All mail is filtered for spam using SpamAssassin. We have a system-wide bayesian filter, and are working on providing automated per-user bayesian filtering as well. By default, spam is automatically delivered to a "spam" folder, separating it from good email. This can be configured for each mailbox, though. The system can automatically delete spam messages, bounce them back to the sender, or deliver them to another folder (even your INBOX). We also provide secure webmail with an open-source package called AeroMail, which we help develop.
    Qmail is an excellent mail system, but it has an extremely high learning curve. Building a major qmail system is not for the faint of heart!

    Web: Of course, Apache! We now run Apache 2.0. It's customized, as is our implementation of PHP. On our system, both CGIs and PHP scripts actually run with the security credentials of the user (YOUR account). This allows for better security and privacy for users' data that they access or manage through scripting.
  • Last updated 4-3-2007.

    Design

    Some of the design has been covered in the Software section above.. This section is more to mention the general design principles we follow in building and running our network.

    1. Expect things to break. Build your systems so that they can withstand the loss of other parts of your network and still (at least partially) run.
    2. Expect people to be mischevious. Well, this wasn't always common-sense, but to anyone these days who spends 30 seconds with a packet sniffer connected to an unfiltered internet connection, it's a very obvious statement. Basically, expect people to snoop around and try to test the limits of your system. Beat them to it - create yourself a few test accounts and do everything you can to try to circumvent things. Be especially careful about permissions to shared services (web server security in particular).
    3. Keep it simple. Now this statement can be deceiving... "keep it simple" can mean to write a little shell script instead of a full-blown application that stores stuff in SQL... but if you get 20 of these little scripts trying to work on the same chunk of data, it may be simpler to set up that SQL database and write that application. Try this rule of thumb: Do things in such a way that you can fully explain what you're doing to a stanger in as few words as possible.
    4. Avoid setuid-root programs. You'd be amazed how many programs come straight from the vendor/distro as setuid root that don't need to be. For a single-user workstation it might make sense, but for a server or otherwise secure machine, there are VERY, VERY few times where a setuid-root program is needed to do the job. Go through the distribution and find all the setuid-root programs. Figure out how to either eliminate the program or make it work correctly without being setuid-root.
    Last updated 7-20-2006.