I recently deployed a Django application with mod_wsgi on my server which runs Fedora 12. Since this required a bit more configuration than a standard Apache virtual host I thought it might be useful to document the configuration for others.
SELinux
While SELinux can be a little annoying if you don’t understand how it works it is a very powerful security layer that should not be disabled. In order to get the Django/mod_wsgi application working I had to modify a couple of SELinux booleans which give Apache extra permissions.
setsebool httpd_tmp_exec on
setsebool httpd_can_network_connect on
mod_wsgi configuration
The default configuration tries to create the mod_wsgi sockets in a directory that SELinux does not allow Apache access to. You can change this by adding the following line to /etc/httpd/conf.d/wsgi.conf.
WSGISocketPrefix run/mod_wsgi
Apache virtual host configuration
Below is the Apache virtual host configuration. Note that I have chosen to use mod_wsgi’s daemon mode and processes instead of threads because some of the libraries I’m using are not thread safe.
This weekend I’ve been doing some network experimentation on my little DSL connection. I’ve learned a couple of things the hard way so I figured a quick blog post is in order in the hopes that it will save someone else time.
PPP interface errors
Over the last while my Internet connection has been a little slow. I noticed that there were occasionally packet drops but I didn’t take the time to figure out where they were occurring. The testing I was doing this weekend was very sensitive to packet loss so I had to get to the bottom of this.
There were two symptoms. The first was a bunch of log entries like the following.
Apr 19 12:03:21 titan pppoe[26690]: Bad TCP checksum 109c
Apr 19 12:10:35 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:10:35 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:10:36 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:10:36 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:24:50 titan pppoe[26690]: Bad TCP checksum 3821
Apr 19 12:31:54 titan pppoe[26690]: Bad TCP checksum 9aeb
Apr 19 12:33:22 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:33:49 titan pppd[26689]: Protocol-Reject for unsupported protocol 0xb00
Apr 19 12:33:57 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x2fe5
Apr 19 12:33:58 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:34:01 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:34:02 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:34:12 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x58e6
Apr 19 12:34:14 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:34:17 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:34:27 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:34:29 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:34:30 titan pppd[26689]: Protocol-Reject for unsupported protocol 0xb00
Apr 19 12:34:31 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x800
Apr 19 12:34:33 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x0
Apr 19 12:34:36 titan pppd[26689]: Protocol-Reject for unsupported protocol 0x7768
The bad TCP checksum entries hinted at some kind of packet corruption. However, I didn’t know if this was coming from packets being transmitted or received. Since I don’t know the inner workings of PPP as well as I’d like, the Protocol-Reject messages were harder to get a handle on. I grabbed a capture on the Ethernet interface underlying ppp0 so I could look at the PPP messages in Wireshark.
Suspect PPP message
My PPPoE client sent a message with the protocol field set to 0. Wireshark doesn’t know what 0 is supposed to mean.
PPP rejection message
And the remote PPPoE device is sending a message back rejecting the transmitted message. And it’s even nice enough to return the entire payload thereby wasting download bandwidth as well. From this packet capture I became pretty confident that the problem was on my end not the ISP’s. After this I wasted a bunch of time playing around with the clamp TCP MSS PPP option because the data size in the above messages (1412) matched clamp TCP MSS setting in my PPP interface configuration file.
The second symptom was a large number of receive errors on the ppp0 interface – the underlying Ethernet interface did not have any errors. Opposite to the PPP errors above, the receive errors made it look like the problem was in the PPP messages being received by my PPPoE client.
After several unsuccessful theories I finally figured out what the problem was. The PPPoE implementation on Linux has two modes: synchronous and asynchronous. Synchronous mode uses less CPU but requires a fast computer. I guess the P3-450 that I use as a gateway doesn’t qualify as fast because as soon as I switched to the asyncronous mode all of the errors went away.
Fixing the problem was good but this still didn’t make sense to me because I’ve been using this computer as a gateway for years. Then I discovered this Fedora bug. It turns out that Fedora 10 shipped with a version of system-config-network which contained a bug that defaulted all PPPoE connections to synchronous mode. This bug has since been fixed and pushed out to all Fedora users but that didn’t fix the problem for me because the PPP connection configuration was already generated.
In summary, this was a real pain but I did learn more about PPP than I’ve ever had reason to in the past.
Dropping PPP connections
Some of the experimentation I’ve been doing this weekend required completely congesting the upload channel of my DSL connection. I don’t just mean a bunch of TCP uploads; this doesn’t cause any problems. What I was doing is running three copies of the following.
ping -f -s 1450 alpha.coverfire.com
This generates significantly more traffic than my little 768Kbps upload channel can handle. During these tests I noticed that occasionally the PPPoE connection would die and reconnect. Examples of the log entries associated with these events are below.
Apr 19 20:02:31 titan pppd[15627]: No response to 3 echo-requests
Apr 19 20:02:31 titan pppd[15627]: Serial link appears to be disconnected.
Since I had already been looking at PPP packet captures in Wireshark I recognized the following.
PPP echo
It appears that too much upload traffic causes enough congestion that the PPP echos fail and the PPP connection is dropped after a timeout. I would have thought the PPP daemon would prioritize something like this over upper layer packets but nevertheless this appears to be the case. For the purposes of my testing this problem was easy to avoid by modifying the following lines in /etc/sysconfig/network-scripts/ifcfg-INTERFACE. I increased the failure count from 3 to 10.
A few years ago I wrote about the backup script that I use to do daily and weekly backups of my computers. Since this script must run unattended it makes use of a passphrase-less SSH key. The SSH key in question only exists on my main workstation and is used to login as a user which does not own any other files. While this isn’t a big security problem it would be nice to limit the privileges of this user. To this end I started using scponly some time ago. Scponly is a restricted shell which limits a logged in user to only executing a few commands such as scp, sftp and rsync. This small set of available programs greatly reduces the chances that the user will be able to find a local exploit. Scponly is already packaged for Fedora so installing it is simple.
yum install scponly
Setting an user’s shell to scponly is accomplished with the usermod command.
usermod -s /usr/bin/scponly backup
Like any shell, scponly must also be added to /etc/shells. Just add “/usr/bin/scponly” (without the quotes) to the end of this file.
As I mentioned when describing the backup script, the script works great except for large amounts of data such as media collections. Over time my photo collection has grown to over nine thousand images and now consumes more than eighteen gigabytes of disk space. So today I decided to cron up rsync to synchronize my photos to the same location where my backups are sent every night. Unlike my backup script, rsync will only send the changes to the remote server not the entire archive.
After much debugging I discovered that the most recently released version of scponly does not work with rsync. The thread where this problem was first discussed started in March 2006. More related posts can be found in subsequent months. Fortunately the scponly authors have fixed this bug in their CVS repository so I built a RPM for the CVS version.
