CS-TN-60

DCS Mail System: Sendmail v8 and Procmail

Neil Clifford

(Currently maintained by Ross Hamilton)

Last updated: 20th March 1998

In early 1998 the DCS mail system was upgraded from the ageing, slow and poorly featured PP mail transport agent (MTA) to the most recent version (8.8.8) of Berkeley Sendmail integrated with Procmail 3.10 as the local delivery agent (LDA), in order to provide fast, efficient, flexible mail delivery with filtering and anti-spam features.

The mail relay

The PP mail relay rainich (a SPARCstation 5) has been replaced with a new machine, muck (an UltraSPARC 150E). This is the most visible SMTP server (ie is most obvious from outside the department being the most prefered host specified in the departmental MX record, dcs.ed.ac.uk, and the only one listed in that record that is located in the department). The role of this machine is to accept mail from outside for delivery to local aliases (be they mailboxes, mailing lists, programs or simply redirection to other sites), or to accept mail from other departmental machines for relay outwards to remote sites. The main program that performs this task is sendmail.

Sendmail

The local version of sendmail on muck is the most recent release (at the time of the mail system development) - namely 8.8.8. It is run as a daemon, bound, listening to the SMTP port and processing the local (to muck) mail queue at least every minute:

/opt/mail/sbin/sendmail -bd -q1m 
                -C/opt/mail/etc/mail/sendmail.cf

Another sendmail process

/opt/mail/sbin/sendmail -q1m 
                -C/opt/mail/etc/mail/sendmail-lists.cf

looks after the lists queue making sure that any straggler traffic gets kicked at least every minute (since bulk_mailer, used to dispatch list mail, instructs sendmail to queue mail first then process it on a subsequent run).

On muck any binaries associated with mail are stored in

/opt/mail/sbin

whilst the configuration and database files are stored in

/opt/mail/etc/mail

The /opt/mail directory is mirrored to /export each night with lfu.

This version of sendmail has been enhanced beyond the standard Berkeley version. The following features should be noted:

it has tcp wrappers integrated with it such that access can be denied to domains or hosts through the standard /etc/hosts.{allow,deny} mechanism before the remote site is even able to interact with the MTA itself. In the allow/deny file the service should be refered to as sendmail.
a further code change to stock sendmail provides a new internal map to match regular expressions used as usernames in From addresses (this can be used to filter out the long numeric fantastical usernames so beloved of many spammers). The regular expression is set in the sendmail configuration file, sendmail.cf.
additions to the sendmail configuration file define databases for sites/hosts allowed to use muck for relaying mail and sites denied any access for spamming activities. Furthermore a DNS lookup is performed on any incoming mail to verify that the originator actually exists in the DNS. Another lookup checks to see whether the originator is currently listed in an international spammers blacklist (the RBL). A facility also exists to exempt domains from these checks or to allow mail straight through with no checking for specified local users. All these features perform the checks prior to the actual mail message being transfered to muck - thus avoiding the expense (in local resources) of having to accept, process and (attempt to) bounce any unsolicited mail.
another change adds the spamcan tool to sendmail providing a further line of defence against unsolicited mail. This acts after receipt of the entire mail message but prior to delivery of it. If any of a set of given regular expressions is matched in the headers or body of the mail then it is filtered out and stored in the spamcan (~ /spamcan/.mail) which can be reviewed at suitable intervals. Exceptions to the spamcan can also be defined in order to bypass this mechanism.
This installation makes use of the sendmail restricted shell, smrsh. This restricts what files sendmail will execute from an alias or .forward file. Allowed programs are indicated by placing a symlink from /opt/mail/etc/mail/sm.bin/ to the program in question.
A modification to add a maximum recipients feature configurable from the sendmail.cf file (MaxRecipientPerMessage).
Minor changes to hide version numbers.

Sendmail hands mail for local delivery to procmail which will write the mail to the users mail inbox (.mail) after acting on any global and user procmailrc/forward file.

The database files

In /opt/mail/etc/mail you will find the sendmail configuration file, database files for controlling relaying, spamming, aliases and tables for routing mail and creating virtual users. Those files are as follows:

aliases-*

- these files are various classes of aliases (eg local manually crafted ones, aliases for majordomo managed lists, aliases for news traffic sent to the mail2news gateway, etc) which are compiled into the master alias files, aliases, by the method described in the makefile, Makefile (which also defines the dependancies for the other database files in this directory). The file is converted to a hashed file by the make process (in this case by sendmail itself, for the other files, by makemap; both use the new Berkeley database, db).

mailertable

- this defines any mail routes that you need to manually hard code. For example to bypass broken MX records or poorly performing relays. The format is

domain.to.mail transport:relay.host.name

To explicitly deliver mail to a given host, bypassing MX records, you should use the syntax:

domain.to.mail transport:[relay.host.name]

Note that matching is done in order of most-to-least qualified. Also, a %1 syntax may be used to interpolate the wildcarded part of the host name. For more details and examples see cf/README.

The file is converted to a hashed file by the make process.

domaintable

- this maps partially qualified domains to actual FQDN's. Use it for preserving local mail shortcuts (eg user@dai or user@ed). The format is eg

dcs dcs.ed.ac.uk

It is advised to avoid this and discourage use of it due to conflicts which may arise in the future with the increasing pressure to deregulate and expand the top level domain namespace. Instead, fully qualified domain names should be used. The file is converted to a hashed file by the make process.

virtusertable

- this is used to create virtual (or rather non-existant) users and route mail for them to a real user. It can be used to catch mail for a virtual user or an entire virtual domain eg if we have an MX for the virtual domain wherever.org pointing to our relay we can route mail for a user in that virtual domain to a real user:

fred@wherever.org user1@dcs.ed.ac.uk

or we can route mail for any user in that virtual domain to a real user:

@wherever.org user2@dcs.ed.ac.uk

or we can map the username from one domain to the same in another:

@foo.org %1@elsewhere.com

This is detailed further in the sendmail source tree cf/README. The file is converted to a hashed file by the make process.

genericstable

- this file is used to rewrite mail in the from header of any mail routed through this MTA. A string on the left handside is rewritten to that given on the right:

username initial.surname@dcs.ed.ac.uk

user@dcs.ed.ac.uk a.n.other@wherever.org

The domains for which we will perform this rewrite are specified in the generics_domainfile, one per line. This is detailed further in the sendmail source tree cf/README. The genericstable file is converted to a hashed file by the make process. generics_domainfile is a flat ASCII file. Any virtual users (virtusertable) for who you want From lines preserved should be added to this file as well.

sendmail.cw

- list of pseudonyms for this machine (class w). The relay will process any mail addressed to alias@pseudonym for local delivery instead of relaying it (ie interpret the aliases files). All local DCS machines and hosts/domains we act as an MX for (for the purposes of local delivery) should be added to this file. All hosts/domains we want to perform genericstable rewrites for should also be added to this file. If a host/domain we MX for needs both genericstable and virtusertable facilities (but we want to relay incoming mail to a different restricted machine - ie no interpretation of usernames/aliases) then an explicit route should be added to the mailertable.

spamcan.cf

- a flat ASCII file containing regular expressions (one per line) which you wish the spamcan filter to match (the file spamcan-exceptions.cf contains regular expressions for exceptions to this).

sendmail.cf

- this is the classic sendmail configuration file. It is built from m4 macros in the sendmail source tree (cf/cf). The file used for the relay was dcs-mailhub.muck.mc. Besides the checks performed on mail based around the other files here, there are three internal checks performed. First the originators domain/host is checked in the DNS (if it doesn't exist a temporary error code is returned. Secondly a regexp map is checked. This traps ridiculous numerical usernames/domains that many spammers generate on the fly. Finally a lookup on the originating host/domain is performed using the Realtime Black List (http://maps.vix.com/rbl/). This is an independantly maintained list of recent spam offenders used by many sites.

sendmail-lists.cf

- this is a copy of sendmail.cf but modified to process mail in the /var/spool/listsmqueue directory. This is for use by majordomo. The lists mail is kept separate from the personal mail in order that queue processing parameters can be tuned separately. In this respect the file has the following set:

CheckpointInterval=3 (to avoid resends after a crash)

QueueLA=5 (stop processing the lists queue)

DefaultUser=31056:31000 (majordom:local)

MaxRecipientPerMessage (unlimited on lists, 500 otherwise)

MaxMessageSize=15000000 (bytes; 150Mb on main queue)

QueueSortOrder=host (sort on hostname)

StatusFile=/opt/mail/etc/mail/sendmail-lists.st

Using a separate queue also has the advantage that a different MTA could be used in future for handling the lists (for example, exim or vmailer, drop in replacements for sendmail, which are acknowledged to have better performance).

sendmail.st

- this is a statistics file to which sendmail dumps summary data concerning throughput. The mailstat tool is used to read this.

procmailrc

- this is the global system wide procmailrc file which is read first prior to any users procmailrc file. Here recipes could be inserted for automating replies to mail sent to users or groups, or recipies for trapping mail loops and other situations (currently not used).

LocalIP

- list of local IP addresses, one wire per line, consequently defining what machines are external to the department and will thus be refused the right to use muck to relay mail traffic out to other external sites.

RelayTo

- destination domains/hosts that are exempt from the refuse to relay policy. These would be, for example, those we MX for, ie tardis.ed.ac.uk (in other words external hosts are allowed to use us as a relay in order to reach these domains). Flat ASCII file, one domain/relay per line.

SpamJunk

- Defines spammers (spammer@spam.domain) or spam domains (spam.domain or IP addresses W.X.Y.Z, W.X.Y, W.X, W) from which (if indiciated in the SMTP envelope) we will refuse mail either permanently (5xx error code) or temporarily (4xx error code). By refusing the traffic with a temporary code you can tie up the spammers mail relay. The syntax is key, tab, data, where the data is the error code/string that is returned during the SMTP session. This file could be used to deny users the capability to send mail:

user@dcs.ed.ac.uk "550 You are not permitted to send mail"

(whilst you could also deny incoming traffic by modifying aliases).

NoFilter

- A file containing local users/aliases, one per line, for whom we will allow traffic to bypass the sendmail based checks (SpamJunk, the regexp map, DNS lookups and the RBL). This contains postmaster so that remote sites may attempt to plead their case for exemption. ASCII file, one user per line.

AllowThruUser

- Remote users allowed to bypass filtering checks. ASCII file, one user per line.

AllowThruDomain

Remote domains allowed to bypass filtering checks. ASCII file, one user per line.

AllowDuffDNS

- Domains, not in the DNS, that we might (temorarily) wish to allow through. ASCII file, one user per line.

sendmail.oE

- header to add to permanent errors

The spam filtering route

A few words about the filtering and anti-spoofing/abuse methods that are available. These are multi-layered and outlined below:

At the tcp/ip level a connection from a host or domain may be refused by using the hosts.{allow,deny} mechanism of the tcp wrapper package since sendmail is linked against, and uses, (libwrap).
The strongest privacy options are set in sendmail.cf in order to enforce strict adherence to [E]SMTP and catch out possible door knob twisting events.
After the initial SMTP greeting the remote host should define the originator (MAIL FROM) and recipient(s) (RCPT TO) for this message. If the recipient is defined in AllowThru then no further sendmail filtering will occur and the mail will (providing it doesn't violate the RFCs) be handed off to the LDA for delivery. Otherwise a check is made to make sure that this is a valid use of the relay (ie if this is an external site that they are trying to send to an internal address, unless that address is defined in RelayTo).
The originator address is checked. It must exist in the DNS, not exist in the RBL and not be listed in the local known spammers file (SpamJunk). It must also not match the internal regexp map. If it fails any of these then an error code is returned (configurable in the sendmail.cf and SpamJunk files) and then the SMTP conversation is terminated. No DATA (ie mail message) has been transferred at this point (thus minimising local resource wastage). At this level checks on maximum message size and number of recipients are performed (edit the sendmail.cf file to control this).
If the mail has satisfied the checks thus far, the DATA will be accepted and the SMTP conversation then terminated. Local aliases are interpreted (generics rewrites and virtusertable routings applied). If the destination is non-local then the mail is dispatched via [E]SMTP, otherwise the mail is delivered locally. First, if the mail originated outside the local network, the spamcan mail filtering rules are applied (unless the user has a ~ /.nospamcan file). If the user has created a ~ /.spam file then the matching mail will be delivered to this file. Otherwise it will end up in the system spamcan ~ /spamcan/.mail. This file should be checked regularly in order to tune the spamcan spam filter expressions and ensure that valid mail has not been accidentally trapped. The triggering condition for each message so filtered is identified by the extra mail header field, X-Spamcan-Reason, that spamcan adds to the filtered spam message.
If the user uses a .forward file it will be processed at this point. This might involve forwarding the mail message or it could result in piping the message to another program. Only certain programs are allowed to handle mail traffic in this manner. Apart from procmail (which always acts under the uid of the recipient), these allowed programs are defined by creating symlinks to them in /opt/mail/etc/mail/sm.bin, thus indicating to the sendmail restricted shell, smrsh, what it can and can't execute (this same restriction applies to programs specified in sendmail aliases).
Finally, if the message is not spamcan filtered, or does not trigger that filter, or is not handed to another program to process (via a .forward file) it may be subject to filtering by procmail as part of either a global procmailrc or a users own procmailrc.

Mail to News gateway

The aliases in aliases-news define the newgroups for which the system provides a mail to news gateway (local eduni.dcs.* newsgroups). This uses the perl script /usr/local/etc/mail2news which in turn makes use of inews to dispatch the news posting vial NNTP to the news server.

Majordomo and lists

Majordomo 1.94 manages all the official departmental lists. The aliases for this are in aliases-majordomo and the list files are in /opt/mail/etc/mail/majordomo/lists (the majordomo tools and config files are in /usr/local/lib/majordomo-1.94). The lists use a separate queue /var/spool/listsmqueue into which list mail is delivered using the bulk_mailer program (this pre-sorts the list mail and chops it up in order to aid sendmail in processing it more efficiently). The advantage of this scheme is that it can be migrated to a faster MTA (eg replace sendmail with exim, or perhaps vmailer) should traffic levels require this.

The lists queue can be examined by using the command:

/opt/mail/sbin/sendmail -bp 
                -C/opt/mail/etc/mail/sendmail-lists.cf

Users who want to run their own lists (especially small or short lived ones such as those managed by students) should be encouraged to use procmail for this (either through hand crafted recipes - refer to the procmailex man page - or using a package such as SmartList).

Daily operations

Outlined below are several of the most common tasks.

The queue

The main mail queue can be examined using mailq (check /opt/mail/sbin is at the front of your PATH). The output looks like:

                Mail Queue (4 requests)
--Q-ID-- --Size-- -----Q-Time----- ------------Sender/Recipient------------
RAA04895*    2709 Fri Jan 30 17:03 <skeletons-request@dcs.ed.ac.uk>
                                   mmarr@lehman.com
PAA00816      325 Fri Jan 30 15:05 <hn@dcs.ed.ac.uk>
                 (Deferred: Connection refused by gns001.gulfnet.co.jp.)
                                   <aba8cnla@GNS001.GULFNET.CO.JP>
OAA00084*     115 Fri Jan 30 14:47 <ewd@dcs.ed.ac.uk>
                 (host map: lookup (ox.ios.ac.cn): deferred)
                                   <xw@ox.ios.ac.cn>
KAA01102     2680 Thu Jan 29 10:21 <wb@dcs.ed.ac.uk>
                 (host map: lookup (condor.daltek.net): deferred)
                                   <roxxyy@condor.daltek.net>

An asterisk (*) indicates that that mail message is currently being processed. Other messages may be delayed for several reasons - most commonly remote mail server problem (error message: connection refused, or perhaps more meaningful such as an indication that the recipients is over quota or there is a lack of disk space), poor or non-existant network connectivity (connection timed out) or DNS lookup problem (map: lookup deferred). In this last case you should make sure that the local (muck) named is still running and it is not returning corrupted information. Often this problem arises due to a lack of an A or MX record for the domain/host concerned - either due to incorrect DNS configuration at the remote site or because that host/domain was never intended to receive mail (for example a user trying to mail username@host.domain.net instead of username@domain.net; this can also lead to the connection refused error). In these cases you can do one of two things. Either let the mail bounce as it will eventually do (the timeout is 5 days, as recommended by the rfc's and set in the sendmail.cf file), or you could help the mail to its destination by hard wiring a suitable mail relay/server into the mailertable. For example in the above case you might check that domain.net has suitable MX records and then add

.domain.net smtp:domain.net

to mailertable and rebuild it. Alternatively judicious examination of traceroute, whois and name server records may reveal a suitable intermediate host (with better connectivity with yourself and with the target destination machine) to which you can build a route and dispatch the mail (though due to the proliferation of spamming this option is now more often than not closed to you). You should now be able to kick the mail queue and dispatch the mail.

You can kick the mail queue by message ID or by sender or recipient string. The syntax is (for recipient, sender and ID respectively):

/opt/mail/sbin/sendmail -qR<string>
/opt/mail/sbin/sendmail -qS<string>
/opt/mail/sbin/sendmail -qI<messageID>

add the -v flag if you wish to be voyeuristic or are trying to figure out why mail is sticking in the queue (it will print the SMTP conversation with the remote server).

Use the same approach if a local home directory server has been down - procmail will signal temporary errors to sendmail causing mail to be requeued until the server is back up.

If a host or network is down for a long period of time (eg a country or perhaps a machine we MX for such as tardis) you might want to move the queue elsewhere and process it separately, leaving the main mail queue free for other traffic (if a very large number of mail messages build up in the queue, sendmail will spend a lot of its time stating those files and sorting the queue in order to prioritise the mail correctly; by large I mean several hundred plus). You should first stop sendmail. Next copy the contents of the main queue, mqueue, to the spare queue, mqueue2. For example as root:

bash% purgestat bash% cd /var/spool/mqueue bash% lfu * ../mqueue2/

this will copy the current mail spool to the backup spool area. The purgestat removes the persistant host information (see below) which doesn't need to be moved. Restart sendmail on the main queue, and then start a separate sendmail to process the backup queue:

bash% /opt/mail/sbin/sendmail -q5m 
                      -C/opt/mail/etc/mail/sendmail-backupq.cf

Keep an eye on this queue:

bash% /opt/mail/sbin/sendmail -bp 
                      -C/opt/mail/etc/mail/sendmail-backupq.cf

to make sure that it eventually clears.

As mentioned above sendmail stores persistant hosts information in /var/spool/mqueue/.hoststat . This database contains the status of every relay that sendmail has contacted. If sendmail needs to contact one of these relays again (before it reaches the hoststat timeout value given in sendmail.cf) it checks the database to see if that relay was reachable or accepting mail. If not it queues the mail for a subsequent mail relay attempt (instead of wasting resources trying to contact that relay). The database can be viewed with the command hoststat and purged using the command purgestat. The purging is performed nightly by cron in order to avoid the database growing too large and impacting on disk I/O (it stores its data in what can become a very large directory hierarchy). If mail is stuck in the queue and attempts to kick it appear to die silently check that the mail relay concerned isn't marked as bad in the hoststat database.

The queue files

The files, on disk, in the queue are labelled like

<file-type><sendmail message ID>

The sendmail message ID is like RAA26891 whilst the file types are qf, df, tf, xf and Qf. The first two are the most common. Each message consists of a queuefile (qf) containing the SMTP envelope data plus sendmail queue processing status information and a datafile (df) containing the body of the message itself. When moving or deleting messages from the queue you should always manipulate them in qf/df pairs. The tf files are temporary copies of the qf files, existing momentarily each time the queue is being processed and the staus of each message is being updated. xf files are transcripts of on going SMTP conversations, transcripts of sendmail sessions whilst mail is being received. The Qf files are bogus files; sendmail carries out a number of security checks on qf files and if such a file fails then it is renamed to a Qf file (see the bat book for further details). Hopefully few if any of these will appear.

The queue should periodically be cleaned in order to remove accumulated cruft (zero length files, convert tf to qf files, remove df files for which there is no qf file). The script /opt/mail/sbin/cleanq can be used to do this (run it with the path to the queue to clean as an argument).

See appendix B of the sendmail operations guide for more details.

Interpreting the sendmail traffic logs

The sendmail information that is sysloged is similar to this extract (wrapped for clarity):

Jan 25 12:13:53 muck.dcs.ed.ac.uk sendmail[23891]: RAA23891: 
from=<abc@dcs.ed.ac.uk>, size=402, class=0, pri=30402, nrcpts=1, 
msgid=<1544.199801278713@scaraig.dcs.ed.ac.uk>,
proto=SMTP, relay=abc@scaraig.dcs.ed.ac.uk [129.215.64.71]
Jan 25 12:13:54 muck.dcs.ed.ac.uk sendmail[23893]: RAA23891: 
to=<zaza@informatik.tu-muenchen.de>, ctladdr=<abc@dcs.ed.ac.uk> (28315/28031), 
delay=00:00:01, xdelay=00:00:01, mailer=esmtp, 
relay=tuminfo2.informatik.tu-muenchen.de. [131.159.0.81], 
stat=Sent (2.6.0 S.onVLG110513 message accepted)

A line is logged upon receipt of the message and then one is logged for each delivery attempt (of that message). The full details are in the sendmail operation guide (section 2.1.1) but the key items are the sendmail message ID (RAA23891, in the extract above) which allows you to cross reference the logged data with the queue information and the files on disk in the queue spool, the message ID itself (msgid) which matches that in the headers of the actual message itself, the time the message spent on our relay being processed (delay, should be low), the time taken trying to deliver the message (xdelay, usually indicates how good connectivity to the remote machine is or how responsive it is), the relays involved and the status of the message (stat). Note that for local delivery xdelay should be low unless a home directory server is down or procmail is having difficulties locking files.

Adding a Majordomo mailing list

On the mail relay (muck) nsu to majordom then go to the master lists directory:

cd /opt/mail/etc/mail/majordom/lists

Now create a file called <name-of-list> and an associated password file called <name-of-list>.passwd (chmod 600 this to protect it). Move back to the main mail configuration directory and become user sendmail. Now edit aliases-majordomo putting the appropriate aliases in that file (copy the aliases for another list and simply edit references to that list). Don't forget to set the list owner correctly. Now nsu to root and run make to build the final aliases file. You can create a config file for the list by mailing majordomo with

writeconfig <listname> <password>

in the body of the message.

Building mail traffic statistics

There are several programs/scripts for collecting or processing mail related statistics. Apart from mailstats which comes with sendmail (it provides a summary of traffic), there are several scripts under ~ mailutil/various which could be used to build statistics for the web (for example).

`username`		`initial.surname@dcs.ed.ac.uk`
`user@dcs.ed.ac.uk`		`a.n.other@wherever.org`

`CheckpointInterval=3`	(to avoid resends after a crash)
`QueueLA=5`	(stop processing the lists queue)
`DefaultUser=31056:31000`	(majordom:local)
`MaxRecipientPerMessage`	(unlimited on lists, 500 otherwise)
`MaxMessageSize=15000000`	(bytes; 150Mb on main queue)
`QueueSortOrder=host`	(sort on hostname)
`StatusFile=/opt/mail/etc/mail/sendmail-lists.st`