Computing Facilities

 Design philosophy

A major goal of our design philosopy is to have all data products readily accessible online, rather than some form of media archive, while maintaining data integrity and minimizing costs.  To meet this goal we are:

• Exploring various configurations of RAID (Redundant Array of Independent Disks) technology.
  
  
• Spreading the computing services of the lab over many PC-based machines, each offering one or two services, rather than having a few expensive server machines offering many services.
  
  
• Using redundant servers to guard against interruption of computing services.
  
  
• Using open source software, whenever possible to lower costs and increase robustness; this includes software for operating systems (Linux), web servers (Apache), database management systems (MySQL, PostGRESQL), network monitoring (Big Brother) and for the management of our computing cluster (NPACI ROCKS).

 Compute Cluster

• We are currently using the NPACI ROCKS software package to manage our compute cluster: http://www.rocksclusters.org.
  
  
• ROCKS allows us to easily update, add, and remove systems to our cluster of inexpensive, heterogeneous PCs.
  
  
• Cluster nodes use private network addreses and are isolated on a dedicated switch to segregate potentially heavy network traffic and reduce security risks.
  
  
• The cluster can only be accessed via a single "front-end" system which is also behind a firewall.
  
  
• Each of our disk servers has a second network card connected to the cluster switch which allows the cluster to access data on these machines.

 Disk storage notes:

 Requirements:

• Inexpensive
• Fault tolerant (we run everything at RAID5)
• Online backups (tape is too slow and unreliable)
• Large partition sizes to keep datasets together
• Dense form-factor (server room space and cooling is an issue)
• Easily expandible

 History:

• First disk servers consisted of RedHat Linux server PCs with SCSI
  disks run at RAID5 on Mylex ExtremeRAID hardware RAID PCI cards.
  • The SCSI disks and enclosures are very expensive.
• We then attached IDE-SCSI hardware RAID towers attached to Sun servers.
  • This is fast and reliable but expensive considering we have not
    been able to run more than 2 disk towers on a single SCSI chain.
• We then put the same hardware RAID5 towers on RedHat Linux PC based
  servers.
  • This solution is slower but cheaper and allows us to run the
    IPTABLES based firewalls on each server.
• We have experimented with the 3Ware Escalade hardware RAID PCI
  cards with success.
• 200302- running software RAID5 on fully contained RedHat linux
  servers packed with large (200GB) IDE disks on PCI-IDE controllers
  • 200302: put together 9 WD2000JB 200GB Western Digital IDE
    hard drives with an old dual PIII 450 MB and 1GB of ram running RedHat
    8.0 at software RAID5 to get 1.5TB for at a cost of ~$2.33/GB (not
    counting cost of MB or case) to act as BOS-BUB (Barely Online
    Storage-BackUp-Brick) to provide R-RAID5 (Redundant-RAID5).
  • Currently spec'ing out and building a 3TB disk server and BUB
    pair (6TB total)
  • 20030220-Current Pie-in-the-Sky disk server (3TB):
    Case: www.servercase.com: RMC-4D-E 18bay redundant 460Wps Price: $1,300.00
    http://servercase.com/cgi-bin/miva.cgi?/Merchant2/merchant.mv+Screen=PROD&Store_Code=SC&Product_Code=RMC-4D-E&Category_Code=4UBKBLN
    Slim Floppy from www.servercase.com:
    IDE Cables from www.servercase.com:
    MotherBoard:
    http://www.supermicro.com/PRODUCT/MotherBoards/E7501/X5DPL-iGM.htm Price $415 from www.anacapamicro.com
    CPU: Dual Intel 2.4GHz Xeon Socket 603 (533MHz) at $284.50 each from anacapamicro.com = $569
    Memory: 512MB PC266 DDR Registered ECC Memory x 6 at $154each from anacapamicro.com = $924
    Four Maxtor PCI ATA EIDE Controler Cards at $25each from anacapamicro.com = $100
    16 WD2000JB Western Digital IDE ATA/100 Hard Drives at $258each from www.mwave.com = $4096
    IDE Disk for OS:

Description of Barely-Online-Storage BackUpBricks (BOS-BUB):
The idea here is to get backups of our data online in the least expensive
manner possible. Our most recent solution has been to retro-fit an old dual
PIII450 with a new power supply, 1GB of RAM, a PCI-IDE expansion card, and
9 200GB WD2000JB ide hard drives running software RAID5.

This gives us 1.5TB at a total cost under $4000.00 (not including the original
price of the carcass system).

We use rsync over nfs to backup our disk servers to the BOS-BUBs nightly.
Each BOS-BUB is configured with dual NICs as are the Disk Servers so that they
can quickly fill in for failed disk servers, simply at a lower performance
level.

REFERENCES:
http://research.microsoft.com/~gray/papers/TeraScaleSneakerNet.doc