The Perfect Linux Firewall Part I -- IPCop | HowtoForge - Linux Howtos and Tutorials

The Perfect Linux Firewall Part I -- IPCop | HowtoForge - Linux Howtos and Tutorials: "This document describes how to install the GNU/Linux GPL IPCop firewall and create a small home office network. In the second installment we cover creating a DMZ for hosting your own web server or mail server and the Copfilter proxy for filtering web and email traffic. "

Zimbra on gentoo at Elfenbeinturm.cc

Zimbra on gentoo at Elfenbeinturm.cc
Zimbra is a collaboration suite which includes an ajax based webmailer with an address book and calendar. It uses many popular open source components like postfix, openldap, spamassasin, apache and tomcat. Because its an integrated solution, installing on gentoo is not easy. But the author of this doc shows how to run Zimbra debian package under Gentoo's chroot.

Gentoo Linux Documentation -- Upgrading Apache (have you done it?)

Gentoo Linux Documentation -- Upgrading Apache:
then Upgrading PHP
and making the server serves both PHP4 and PHP5

GCC upgrade guide

Hercules Load Balancer Virtual ApplianceVMTN Virtual Appliances Directory

Hercules Load Balancer Virtual ApplianceVMTN

Hercules Load Balancer Virtual Appliance

Hercules is a tiny but mighty load balancer appliance. It can be used to balance requests using any of these TCP protocols:

• HTTP
• HTTPS
• SMTP
• FTP
• POP3
• LDAP

It runs on a linux os built from scratch to have a minimal
footprint. It can run with just 32MB memory for normal usage and you can increase memory requirements for the appliance as you need. It also has a built in web server, ssh server, dhcp client. You can administer the machine remotely via ssh. It is setup for dhcp so the networking is setup automatically. A webserver is installed to both serve up this page and to help you test different configurations. The load balancing functionality is provided by the excellent open source project pen.

[Note: VERY simple!!! - pizzaman]

Pound and Pen - Simple Load Balancing

Pound and Pen - Simple Load Balancing: "Pound and Pen are simple web server load balancers licensed under the GPL that run mainly on Unix based operating systems. Pound is a web (port 80) load balancer only, unlike Pen which is designed to balance any simple TCP/IP protocol, including http, smtp, ftp, and others. Before you decide to load balance your website with one of these software programs there are several other considerations you may want to understand."
...
Pen Load Balancer - http://siag.nu/pen/
Pound Load Balancer - http://www.apsis.ch/pound/
...

Extracting remote_ip is easily done with mod_extract_forwarded
Pound mailing list has this information.
Or this one is good, with mod_rpaf.

Backup using SystemRescueCD & Partimage

Partimage Partition Image is a Linux/UNIX utility which saves partitions in many formats (see below) to an image file.

SystemRescueCd is a linux system on a bootable cdrom for repairing your system and your data after a crash. It also aims to provide an easy way to carry out admin tasks on your computer, such as creating and editing the partitions of the hard disk. It contains a lot of system utilities (parted, partimage, fstools, ...) and basic ones (editors, midnight commander, network tools). It aims to be very easy to use: just boot from the cdrom, and you can do everything. The kernel of the system supports most important file systems (ext2/ext3, reiserfs, reiser4, xfs, jfs, vfat, ntfs, iso9660), and network ones (samba and nfs).

Today's the day

Today's also the first day that I've ever linked to this site. The first link to this site comes from ThaiHostTalk.com.

The scaling experience of ThaiLE.com

My presious post regarding Web Server Performance Comparison ends with a note about ThaiLE.com. Now's the time to write about it some more.

ThaiLE.com is the largest banner exchange network in Thailand with over 2 million banner exchange requests a day, as of April'06. This means basically about 10 million SQL queries per day. It comes with real-time transaction update and a lot of other features and resides on a single Dell server (P4D 3.0G, 2G RAM, 70G SCSI 15000rpm). The server runs Gentoo Linux with NPTL and a lot of other optimizations.

It used to be LAMP (Linux, Apache2, MySQL and PHP) and now it's 3LMP (Linux, LSWS, Lighty, MySQL and PHP).

During May we saw a traffic growth of over 30% and it's time to do something about the server.

Toward the end of April, server load average was pretty high, and I started to play with lighty first becuase of its open source nature. Lighty was able to cut down on our LA about 20-30% during peak hours, while memory pressure was reduced about as much as 30-40%. But there were problem in a prior version of lighty that forced me to investigate lsws. With lsws, the lightened feeling that I'd experienced with lighty has carried on, and since its watchdog process is better that lighty, there have been just slight problems with lsws since a month ago. Now with its latest version, I'd be able to sleep soundly at night and the system is able to scale some more without adding any hardware. ;)

Oh well, lsws STD edition wouldn't give you more than 300 concurrent connections, I have to add lighty running on a second IP on the same machine to serve just banners. Swap space usage is almost nil (while it was couple hundred M during Apache2 tenure) and the system is able to serve about 3 million banner exchange transactions a day (or ~ 15 million MySQL queries) without any problem, web-server-wise.

Next thing to tinker with was MySQL and it is now fixed by migrating a couple high traffic tables to memory (heap) tables with cron jobs to update the on-disk tables with current values from memory.

End note, Apache is good, Lighty is good, LSWS is good. Choose the ones that best suits your needs. Also, I don't notice speed advantage of lsws over lighty but the experiment shows that lsws is of better stability than lighty when running with php-fastcgi (phplsapi in case of lsws).

Cudos to all people that make possible all those great software.

High Availability, and a lot... (work in progress)

High Availability
As more and more mission-critical applications move on the Internet, providing highly available services becomes increasingly important. One of the advantages of a clustered system is that it has hardware and software redundancy, because the cluster system consists of a number of independent nodes, and each node runs a copy of operating system and application software. High availability can be achieved by detecting node or daemon failures and reconfiguring the system appropriately, so that the workload can be taken over by the remaining nodes in the cluster.

pizzaman: "There are so many possible scenarios for doing HA stuff, hope that I'll be able to play with more of them soon."


Edit: Aug 16, 2006:

During this 3 days holiday, I've read through a lot and found many interesting solutions including:

• LVS itself (of course)
• Linux-HA
  
• pound
• pen
• HAProxy

Several nice articles:

• How to setup HA-LB apache cluster (using LVS)
• How to setup HA NFS server with DRBD
• HA NFS from linux-ha, including lock test results (however, the article is quite old and those problem should be solved by now)
  
• How to aggregate apache log files from several nodes
• NFS and LVM: We will need to work with shared NFS resource in our cluster
  
• RHEL4 NFS reference
  
• DRBD itself, so-called network RAID-1 data replicator(?)
  
• RedHat cluster suite + GFS

I later stumbled upon:

• Kevin Minnick's comments on these solutions (although I think pound doesn't provide HA on its own). Kevin touches upon perlbal, a good perl-based RP and load balancer that I've never heard of until now. Got to have a look later.
• At danga, I saw its DFS called MogileFS. Looks cool, altought it lacks POSIX compliance that I need.

The adventures of scaling, Stage 1

The adventures of scaling, Stage 1

Linux performance tuning

The C10K problem
Old doc but very useful
RH system stat tools