Procmail FAQ

This is a draft FAQ for Procmail, the mail processing utility for Unix.

The official URL of this page is http://www.iki.fi/~era/procmail/mini-faq.html -- this is a "virtual" URL which resolves to a different host. Please use the "virtual" URL rather than whatever your browser thinks it is you are currently looking at. This site can and will move in the relatively near future.

The following mirror sites are available:

United States
http://n.base.org/procmail/
United kingdom
http://www.dcs.ed.ac.uk/~procmail/faq/
Please use one of the mirrors if you can.

This Procmail FAQ is an attempt at answering the most often asked questions and straightening out the most frequent misconceptions about Procmail. This is no substitute for the manuals, and indeed often will direct you to the manuals and other similar resources. There is also a small collection of links to more information.

If you feel you have trouble understanding the tips in this FAQ and/or the manual pages, please be invited to check out the newbie links on the companion link page, which also has a lot of other Procmail-related links for you to investigate.

If you are looking for a quick way to get started with Procmail, you should probably be reading the Procmail links page, too -- it has links to some good tutorials.

The author wanted to call this a "Mini-FAQ" but it keeps getting bigger. There are plans to rename it the "Bronto-FAQ".

As this document has changed and grown (it is currently about eight times the size of the arguably more elegant original version 1.0) it has become a bit hard to know where exactly to expect information about some things. I apologize for this. In the Contents, I try to include a mention of all very frequently asked questions, even if they're in a subsection of a subsection (further adding to the bloat, I'm afraid). The below table of contents is an abridged "best of" instead of a full TOC. (This makes little sense on the web page but is currently made to fit into various versions; I'll fix it. Someday. I think.)

The author's contact information is at the bottom of this page.

Contents (abridged "best of")

Version information:
$Id: mini-faq.prep,v 1.182 1998/03/20 15:05:26 reriksso Exp $
The version history details recent developments.

What is Procmail?

Description, availability, and installation

Procmail is a mail processing utility, which can help you filter your mail; sort incoming mail according to sender, Subject line, length of message, keywords in the message, etc; implement an ftp-by-mail server, and much more.

Procmail is also a complete drop-in replacement for your MDA. (If this doesn't mean anything to you, you don't want to know.)

Procmail runs under Unix. See Infinite Ink's Mail Filtering and Robots page for information about related utilities for various other platforms, and competing Unix programs, too (there aren't that many of either).

You can download Procmail from the main Procmail site or a number of mirrors. The Links page has a listing of well-established mirror sites.

In ancient times, then-current versions were posted to comp.sources.misc (vol 43, July 1994, is the newest I could find).

The recommended version of Procmail to install is 3.11pre7 . [The previous stable version was 3.11pre4. Don't use pre5 or pre6.]

Please note that some of the tips in this FAQ (or elsewhere) might not work for versions older than 3.11pre4. In particular, a great number of sites seem to be stuck on 3.10, which is a bad choice for a number of reasons. There should be plenty of incentive to upgrade to 3.11pre7 (or pre4, if you're chicken -- this was the recommended version for a good one and a half years before the issue of pre7. And don't let that "pre" scare you).

The installation procedure is fairly straightforward but probably not the first thing you should attempt after you get a Unix account.

If you feel adventurous, and have a friend with a working copy of Procmail for your type of operating system and hardware, you can just snatch her/his binary. However, you need to be aware that this defeats some checks which the installation program performs, such as determine where your mail spool is, what kinds of file locking should be employed, etc. Be particularly wary if you use NFS mounted mail spool directories.

The distribution comes with a simple FAQ (locally produced HTML version -- the text-only original is also available) which covers some issues faced when first getting acquainted with Procmail, such as how to view the manual pages, but it primarily addresses various installation problems. There's also answers to some very frequently asked questions, some of which are not dealt with in the document you're reading now. Please look at least at the TOC of the "original" FAQ as well.
(An abridged table of contents of the "original" Procmail FAQ is in the "Quick Questions" section below. Various files in the Procmail distribution package also contain nuggets of very valuable information, especially for the administrator setting up Procmail for the first time.)

If you can't find the answer to your question in either of these FAQs, please take a look at the Links section towards the end of this document -- but first, make sure you have found all of Procmail's manual pages; there are several.

If you are new to Unix, you should probably read up on regular expressions (grep/sed/awk/perl etc) and a little on mail handling before attempting to tackle the Procmail manual pages.

Related quick questions

Q: How can I run an arbitrary Perl or shell script on all or selected incoming mail?

A: Install Procmail. Read the manual pages (there are several). Thank you.

	:0
	* conditions, if any
	| your-script-here
The conditions, in their simplest form, are regular expressions to match against the header of each incoming mail message. You can leave out the condition lines completely if you want to do your action (in this case, run a shell script) unconditionally.

More-complicated conditions can also be exit codes of other shell scripts or programs, or tests against the full body of the message, or against Procmail variables (Procmail's variables are also exported to the environment of subprocesses, so they are essentially environment variables).

Actions can also be to save the message to a folder (appended to a Unix mailbox file, or written to a new file in a directory) or to forward the message to one or more other addresses. Finally, the action can be a nested block of more "recipes", as these condition-action mappings are called in Procmail jargon, to try if the outer condition is met. The procmailrc(5) manual page has the full scoop.

Obviously, you are not restricted to Perl or shell scripts. Anything you can run from a Unix command prompt can be run from Procmail, in principle, although running interactive programs doesn't usually make much sense.

Q: I can't read. What's in the "original" Procmail FAQ?

A: Here's an abridged listing.

Not every question is a Frequently Asked Question, of course. That's why there is a mailing list, after all.
(Conversely, not all answers in the FAQs are Frequently Wanted and/or Understood. Sigh. ;-)

Q: What if Procmail is already installed by another user on my host?

A: Could be. Ask around. Yes, one installation per site should suffice.

Q: How do I know I have found all the manual pages?

A: You read them; most of them contain useful information. Start with the procmail(1) manual page; it contains pointers to the others under the "SEE ALSO" heading.

Q: How do I know which version of Procmail I have?

A: You examine the output of the following command:

$ procmail -v
procmail v3.11pre4 1995/10/29 written and created by Stephen R. van den Berg
							<srb@cuci.nl>

Submit questions/answers to the procmail-related mailinglist by sending to:
	<procmail@informatik.rwth-aachen.de>

And of course, subscription and information requests for this list to:
	<procmail-request@informatik.rwth-aachen.de>

Locking strategies:	dotlocking, fcntl()
Default rcfile:		$HOME/.procmailrc
System mailbox:		/var/spool/mail/$LOGNAME

For Formail and Lockfile, the companion utilities, it's harder to say as they won't tell you themselves.

Q: Please please tell me the address of the Procmail mailing list!

A: See answer to previous question.

There is also a competing / complementary / whatever list. Rhett 'Jonzy' Jones writes:

I have just created two new mailing lists, which are Procmail/SmartList supported, along with my modifications to prevent UCE/SPAM. These mailing lists are called:
smartlist@lists.utah.edu
procmail@lists.utah.edu
To subscribe to these mailing lists send the Subject: subscribe to:
smartlist-request@lists.utah.edu
procmail-request@lists.utah.edu
NOTE - You will not be able to post to these mailing until your address has been verified. Furthermore, you will NOT be able to subscribe to either one of these mailing lists if your domain is one of the 2850+ domain which I deny both subscription and postings from.

Q: I subscribed to the list at Aachen earlier but haven't seen any messages for a while. Did the listserv crash?

A: Try subscribing yourself anew and see if things start to change. SmartList is rather paranoid and will easily unsubscribe you if it gets bounces from your address.


How do I use wildcards in Procmail?
Explain file locking, please.

... and other syntax issues

Procmail uses regular expression syntax, which is more complicated but also vastly more versatile than the "glob" wildcards used in many shells for matching file names (where pr*mail would match anything which starts with "pr" and ends with "mail").

Briefly, the regular expression to match anything that begins with "pr" and ends with "mail" is ^pr.*mail$ but for practical purposes, e.g. in a Subject: line, you might want to try the following recipe:

	:0:
	* ^Subject: pr.*mail
	procmail-mail

This says: Anything that begins with "Subject: pr" and contains the string "mail" somewhere after that is to be saved in the folder procmail-mail.

Note that there is no need for a trailing .* wildcard (and certainly not a sole *) -- the recipe will match regardless.

The neat stuff starts when you want to ignore "pro-mail" and "ProMail" while still looking for anything else that begins with "pro" and also contains "mail", but this example ends here.

Any beginners' book about Unix will contain a more detailed tutorial on regular expressions, most likely in conjunction with examples using the grep and sed programs. For reference, the manual page for egrep (an extended grep) at your site should contain a concise listing of regular expression operators. (Note that the manual's assertion that Procmail is 100% egrep compatible is not strictly speaking true. For one thing, there are many egrep implementations.)

One frequently seen Procmailism is this:

	[ 	]*
The brackets contain one space and one tab, in any order. This regular expression will catch any sequence of horizontal whitespace (including none at all -- change the * to a + if you want to make sure there's at least one whitespace character).

Some mailer programs use tabs instead of spaces in some places, others will attempt to "pad" all header fields with spaces to make the headers somewhat more readable. Finally, Subject headers are written by mere humans who sometimes press the space bar twice, perhaps only because they want to see if that will break your autoresponder script. Therefore, your Procmail recipes frequently need to match arbitrary runs of whitespace characters if you want them to work in all situations.

File locking

Under Unix and other multitasking operating systems, several processes can be running at once. This means several mail messages can be in delivery at the same time. Without Procmail, the default system mailer hopefully handles this just fine when mail is delivered to a system mailbox, but if two Procmails are delivering two messages to the same file more or less at the same time, you end up with problems. (A typical symptom is that the first message gets a FFrom in its first line, and the second gets a rom without the F. This messes up the mailbox format thoroughly.)

Rule of thumb: Use file locking when delivering to a file. Don't use file locking when delivering to /dev/null (because then it doesn't matter if the message gets mangled, and you might not have the permission to acquire a lock on a device), forwarding to another address, or piping into a program. A pipeline which ends up appending to a file should still use a lock, of course, since there is the same race condition as when delivering straight to a file.

Rule of index finger: Using an unnecessary extra lock seldom hurts. (When it does hurt, you'll notice. :^)

Examples:

	:0:   # Deliver to a file, let Procmail figure out how to lock it
	* ^From scooby
	scooby

	:0    # Forwarding; no locking required
	* ^TO_dogbert
	! bofh@dilbert.com

	:0:snoopy.lock  # Explicitly name a file to use as a lock
	* ^Subject:.*snoopy
	| $HOME/bin/woodstock-enhancer.pl >>snoopy.mbox

The last one might need a little elaboration. When mail with a Subject: line containing the word "snoopy" anywhere in it arrives, Procmail will create a lock file called snoopy.lock and hold on to that file while executing the recipe (in this case, piping the mail to a program called woodstock-enhancer.pl and appending the program's output to the file snoopy.mbox). If a second message matches the same recipe, the second instance of Procmail which attempts to deliver that message will politely wait until the first Procmail lets go of the lock file.

(You don't really need to name the lock file in this isolated case. Procmail would automatically deduce a lock file based on the name of the file you're appending to, snoopy.mbox. But you might have several recipes which do separate things which need to be serialized somehow, so that one doesn't occur before the previous one has finished its critical section. Say, maybe you have a second recipe which modifies the program woodstock-enhancer.pl itself, or, more realistically, modifies a separate data file used by that program. You can't have that running while this recipe runs, so they have to share a lock, and you have to tell Procmail the name of the lock file to use.)

The manual really does explain most of this, although it's not very explicit about why you would or wouldn't want to use locking in the first place.

A typical newbie problem is placing a redundant but harmless lock on a forwarding recipe. Here's an example:

	:0:
	* ^TO_johnny@lunatix.com
	! jjasmith@ppp.home.in.isp.net

This will forward to the address jjasmith any mail addressed to johnny. The trailing colon says to use a lock file, but Procmail can't figure out what file you want to lock because there is no file involved anywhere. Indeed, locking doesn't make any sense here, and you'll get warning messages in the log stating that Procmail "couldn't determine implicit lockfile". Remove the second colon on the first line and all will be fine.

(If you think you want to use a variant of this recipe, don't. Read the "virtual domain" question below.)

Related quick questions

Q: "Regular expressions?" What's that? Wouldn't it be much simpler to just use glob-like patterns? Does this mean I have to learn something new again? Augh.

A: Yes. Don't bother with Procmail if you want something simple and stupid. (But getting started really doesn't take that much, all things counted.)

Q: How can I forward to many addresses?

A: In the simple case, just add more addresses to the action line:

	:0
	* ^Subject: result from cgi-bin/www-feedback$
	! first@one.com second@two.net third@three.org
If you have a largish list of recipients, you might prefer to store the addresses in an external file you can edit without mucking with your Procmail filters:
	# The file $MAILDIR/addresses.txt contains the recipients, one per line
	:0
	* ^Subject: result from cgi-bin/www-feedback$
	! `cat addresses.txt`
	# ^ Make sure those ^ are backticks, BTW (ASCII 96)
This will break as soon as the output of the backticks is too large for your shell to handle. You can use more trickery to get around that, but why not resort to a ready-made solution instead? A mailing list manager will scale to thousands of recipients and have some provisions for handling bounces, preventing mail loops, and automatically adding and deleting subscribers. (You might want to look at SmartList, which runs on top of Procmail.)

By the way, if mail to one address should always be redirected to one or more other addresses, you should read the "virtual domain" section below. (Don't use Procmail for this. The universe will implode if you do.)

Q: What does the second colon in :0: mean? Should I worry?

A: You just missed it. It tells Procmail to use locking on this recipe. Go back and read the above snippet about file locking. See the next question.

Q: What does "Couldn't determine implicit lockfile" mean?

A: Briefly, that you have :0: where you should have either a named lock file or just :0 (ignoring any possible flags here). See previous question. Hope this helps.

Q: What does "Extraneous locallockfile ignored" mean?

A: See previous question. Hope this helps.

Q: What does "Extraneous filter-flag ignored" mean?

A: Briefly, that you have :0f where you should have :0 (ignoring any other flags and possible lock file you might have). The f flag is perhaps a bit poorly explained in the manual, seeing as lots of people are using it where they shouldn't (probably "filtering" sounds too enticing in a filtering program -- of course you want to filter your mail, dear).

Q: Okay, then what is the significance of the number zero on the :0 line?

A: Nowadays, none really. In old versions of Procmail, you had to tell it how many condition lines your recipe contained (yes, "bletch"). The syntax was later extended and the number zero special-cased to mean, all the following lines which begin with asterisks are condition lines. (The asterisk at the beginning of each condition is obviously also part of this extended syntax.)

Q: What's this From_ header everyone seems to be talking about? I can't find a header like that on a single one of all the messages I have on store.

A: This is just a conventional way to talk about the line that starts with the five characters "From " at the very beginning of the headers of a mail message. This is a tricky line for several reasons:

  1. It's not really a header, but rather added by your MTA at delivery, so it should properly be referred to as a "pseudo-header". The contents of this pseudo-header are the envelope sender and a time stamp. Some programs will accept basically anything that starts with "From ", others will get confused if e.g. the time stamp is missing.
  2. It's very easy to mix up with the From: header (note the trailing colon on this one). The underscore is used partially to underscore [sic] the fact that we are talking about this colonless pseudo-header, not the From: header proper.
  3. Many MUA:s will get extremely confused if a line in the body of a message starts with the word "From". Often, this is escaped by adding a wedge in front of it, something that in and of itself is a source of confusion (you don't really want your MUA to muck with the body of your message, ordinarily). Unescaped From_ lines often lead to messages being split where the unescaped From was, because the MUA genuinely thinks this marked the start of a new message.
The format that uses the From_ line as message separator is commonly referred to as "Berkeley mbox format". Not all systems use this format, but it's pretty much a de facto standard on Unix. A related format uses the Content-Length: header instead, but keeps the From_ line, leading to utter and hopeless confusion.

Q: How can I do a logical OR of two or more conditions?

A: The usual reason people are asking this is because the following, which is basically the way to do it, feels clumsy or something.

	:0
	* condition1|condition2|condition3|condition4
	{ ... do something about it ... }
There are situations where you can't do this, such as when one of the conditions is a negated condition. You can try to fool around with de Morgan's laws to get you somewhere where this doesn't play in, or use scoring:
	:0
	* 1^0 condition1|condition2
	* 1^0 ! not condition 3
	* 1^0 ? /path/to/external-program whose exit code we want to look at
	{ ... now do something about that instead ... }
Some side effects are different when you resort to scoring; for instance, if you are also using the \/ operator to grab stuff into $MATCH, scoring will generally grab the last matching line, whereas a straightforward regex OR will stop already at the first one. (You can change the 1 in 1^0 to some really big number to prevent this.)

Q: How can I test the value of a variable or an argument? Let's say I manage to pass Procmail a user name using the -a switch, how do I look at the value of $ARG in a recipe?

A: Thusly:

	:0
	* ARG ?? regex
	action
You can test against any Procmail regular expression. You might want to anchor the beginning and end of the match with the ^^ special anchor (so to match "str" but not "string", "Strauss", or "tapestry", you'd say ^^str^^. Also note the absence of a dollar sign on the variable name. Of course, this is all in the manual, by the way ;^)

Q: I want to strive for elegance and efficiency in my regular expressions, and ultimately achieve Procmail guru status. What typical newbie mistakes should I try to avoid?

A: Good girl/boy. Here's a partial list.

Q: I'm having a hard time getting this \/ and $MATCH thing to work for me. I can't seem to get $MATCH to contain what I expect it to. Sometimes the whole regular expression fails to match when I include the \/ token. What gives?

A: This is a bit complicated and basically belongs in the "advanced" section. But here's a pretty good explanation. (Thanks to Philip Guenther.)


Help, I get this error message ...

Troubleshooting tips

The light bulb contains the seeds of its own revolution.

Before you panic, try adding VERBOSE=yes to your .procmailrc, send yourself a test message, and examine the information about Procmail's actions that are written to the log. This tells you, in excruciating detail, about every decision Procmail makes while processing your rc file.

Some common error messages are explained briefly above. Go back and look for "couldn't determine implicit lockfile". (Search for the error you're getting instead if this is not it.)

All of Procmail's own error messages and some plausible explanations are listed on the Procmail manual page. If you haven't before, now is the time to read the manual pages. Note that there are several; procmailrc(5) explains the syntax of the rc files while procmailex(5) contains some good real-world examples. There is also procmailsc(5) which discusses the scoring system.

Free tip: You'll avoid a lot of sarcastic replies if you make sure you try this before you post to the Procmail mailing list or some newsgroup about your problems.

Another free tip: Here's an alternate rc file you can use as a skeleton for interactive experiments. It will

Do note that an interactive Procmail gets a different environment and possibly runs on a different machine than the Procmail in your .forward, though.
	TMPDIR=/tmp/$USER
	MAILDIR=$TMPDIR/procmail.out
	DEFAULT=$MAILDIR/$USER
	VERBOSE=yeah
	SHELL=/bin/sh
	
	:0
	* ? test -d $TMPDIR || mkdir $TMPDIR
	* ? test -d $MAILDIR || mkdir $MAILDIR
	{ }
	:0E
	{
	    # Bail out if either directory didn't exist and couldn't be created
	    EXITCODE=127
	    HOST
	}
	
	# ... your experimental recipes here
With this recipe, you can use Procmail directly from the command line, typing in experimental messages as you go along, or feeding it a message on standard input, like this:
	procmail experiments.rc <test.mbox

(assuming you had saved the above example rc file in experiments.rc and have a test message for it in the file test.mbox).

[Click here to download a copy of experiments.rc]

Known bugs, common gotchas, and funny quirks

This listing is by no means exhaustive.

Known bugs

This is mostly for version 3.11pre4 and newer. If you have an older version, you should probably upgrade. The HISTORY file in the distribution package contains a list of fixed bugs in earlier versions (although it's not very detailed).
Not all delivering recipes are logged after all (3.11pre7)
With LOGABSTRACT=all Procmail is supposed to log all deliveries. This is not working correctly in 3.11pre7; expect a fix in the next version.
Procmail always requires enough memory to read in the entire message it's processing.
This is rather self-evident, given the way Procmail works, but it has some unfortunate consequences (especially considering the next item). It would be nice if there was a back door of some sort, so you could say e.g. "if a message is bigger than 32 megabytes, just deliver it, okay?" (or maybe look at only the first 32 megabytes of it when reading the recipes).
Procmail's memory handling is poison to some *BSD systems.
This is true at least for at least FreeBSD systems, and possibly other related *BSD:s as well (NetBSD?, OpenBSD?, others?). More information about this would be welcome. A "quick and dirty" patch was posted to the list in October 1997. Allegedly, you could also try simply changing malloc's options (Michael S. Fischer writes: "Changing malloc's options seemed to do the trick for now (ln -s '<' /etc/malloc.conf)." See the malloc(3) man page if you're on FreeBSD and wonder what this means.)
Exceeding LINEBUF can give you core dumps.
I'm not sure on what systems and under what circumstances exactly this happens. You should definitely avoid grabbing huge amounts of data into MATCH and then evaluating it, unless you know that your LINEBUF can accommodate all the data.
formail -r breaks RFC822
Always use formail -rt if you don't know what this means. Perhaps you should always use it anyway. Never mind that the documentation says something funny about how the -t option implies "trusting" the sender. Without -t, you stand a good chance of replying to the envelope sender, which is a violation of the standard. (See next item.)
The precedence of different headers when doing a formail -r or formail -rt is not documented
The source is worth a look. Really, this part is actually understandable even if you don't speak C.
The manual says the regexp engine is egrep compatible
'Tain't so. This statement is probably a holdover from older times when basically it was, or tried to be.

Common gotchas

The following things are frequently regarded as unintuitive or even "broken" by many people.
There is no ^FROM macro
Many people incorrectly infer that since there is a special ^TO macro which catches all variants of To:, Cc:, Resent-To:, etcetera, there also ought to be a ^FROM which would catch, say, From:, Sender:, Reply-To:, and so forth. But this is in fact not the case, and Stephen van den Berg has repeatedly stated that he will not add such a special macro.
Of course, rolling your own isn't too hard. Just define a variable which contains the stuff you think you want in there, e.g.
	FROM="^(From[ 	]|(Old-|X-)?(Resent-)?(From|Reply-To|Sender):)(.*\<)?"

	# Use thusly:
	:0
	* $ ${FROM}billg@microsoft\.com
	| $HOME/bin/throw-virtual-cake
This is not completely orthogonal to what ^TO does; you use it at your own risk, of course. (This is provided as an example. Compare to the documentation for ^TO_ and pick the things you like from both.)
(Oh, and by the way, that "mysterious extra dollar sign" is required to actually expand the FROM variable.)
Procmail strips off all whitespace after backslash plus newline
This frequently bites newbies. Most other programs will treat backslash plus newline as a token delimiter. Not Procmail.
	:0
	| formail -AMoo: -IFoo: -r\
		-aBoo:
will actually be glued together as
	:0
	| formail -AMoo: -IFoo: -r-aBoo:
which will not work as expected. Solution: If you want a space, put it before the line-continuation backslash.
Backslashes at the beginning of a line are parsed differently from other backslashes
Get used to it, or use some workaround. Here's one idea:
	* Condition which wants to continue \
	()	with leading whitespace
You can use something else besides the empty parens. The basic idea is just to avoid a backslash in the first column, which is what you'd ordinarily expect to work to tell Procmail that the following whitespace is to be taken literally. The parens work well as a substitute for a backslash in this particular case.
Backslash as the first character of a condition has the same problem; similar workarounds should do:
	* VAR ?? ()\$string which starts with a literal dollar sign
	* VAR ?? [$]string which also starts with a literal dollar sign
	* VAR ?? (\$)third alternative solution
The empty parentheses are almost an idiom among veteran Procmail users, but you see all variants in regular use.
Getting a newline into the action line is tricky
Sometimes, you'd want to write a recipe with an action line which contains a newline. Let's say you have a sed script to trim off a pompous .signature and leave a notice about what happened:
	sed '/^-- $/,$c\
	-- \
	sed: .signature removed'
and you'd like to run this over mail from a particular user. But Procmail won't easily permit you to split the action line over several physical lines. How do you fix that?
One approach is to define the newline as a variable, and then interpolate that variable when you need it:
	NL="
"
	:0bfwi
	* ^From:(.*\<)?dquayle?@aol\.com\>
	| sed '/^-- $/,$c\'"$NL"'-- \'"$NL"'sed: .signature removed'
(Don't let that quoting scare you. If you look at it closely, you'll notice that those are just several quoted strings glued together in order to pass it all as a single argument to sed. The newlines are in double quotes while all the other stuff is in single quotes. Do pay attention to the first lines which define the NL variable, though -- that's double quote, newline, double quote.)
You can, in principle, use backslashes to escape a newline on the action line, but the number of backslashes you need for that is not easily deduced (see previous notes about Procmail's quirky treatment of backslashes in some situations). It's often easier to just try to think of a workaround.
If your login shell is a C shell (csh or tcsh), prepare for havoc.
Procmail works very badly with the C family of shells (you could almost say it doesn't, plain and simple, but the symptoms are murky and hard to put your finger on). As a precaution, always set SHELL=/bin/sh at the beginning of your .procmailrc
$@ is only available to external programs
This is actually documented (all right, in the BUGS section, so perhaps it's not a feature :-) -- if you need to use $@ internally, you'll have to go via an external program such as echo:
	:0ir
	ARGS=| echo "$@"
Using "$ARGS" subsequently is still not exactly the same as "$@" is to the shell.
Diagnostics in the log file don't tell you which line in the .rc file the error is happening on
Run the problem message through Procmail again with VERBOSE=yes and it should become apparent where the problem is.
formail -A or -I don't imply -X
If you use formail's -A or -I flags in a command which also uses -X, you have to explicitly add an -X for the fields you just added.
In other words,
	formail -rt -XTo: -A"X-Loop: me@my.isp"
will actually kill the X-Loop header immediately after inserting it. You have to extract it explicitly, even though you just added it:
	formail -rt -XTo: -A"X-Loop: me@my.isp" -XX-Loop:
formail has a fixed order for parsing its arguments
This means you sometimes have to feed the output of formail to another instance of formail to achieve the end result you want.
formail and lockfile won't tell you what their version number is.
No fix has been scheduled. You can look inside the binary with strings(1) -- however, you will probably only find a date but no version number. (Dates in April 1997 seem to correspond to the versions that come with version 3.11pre7 of Procmail. [pre5 and pre6 were also issued around April.])

Funny quirks

Related quick questions

Q: What are some typical installation problems?

A: Here are some of the most typical problems with getting Procmail to run in the first place:

See also various points in Stephen's original Procmail FAQ.

Q: What are all those mysterious things I'm supposed to stick in my .forward file?

A: First off, for the record, you don't need a .forward if Procmail is your local MDA. Ask your admin.

If Procmail is not your local MDA, you need something more or less like what the manual suggests. Here's one popular variation:

"|IFS=' '&&p=/usr/local/bin/procmail&&test -f $p&&exec $p -f-||exit 75#whatever"
(That should all be on a single line. Unwrap it if it appears wrapped. Unwarp it if it appears warped. The surrounding double quotes are significant.)
This is just a simple (but slightly convoluted) shell script.
Whitespace is often left out as much as possible, partly out of paranoia, partly because people want it all to fit onto a single line in their editor.
Please note that this is not guaranteed to work out of the box. Your first attempt at a .forward should be what your local manual page suggests (and please, people, change #whatever to your own login name with a hash mark in front).

Here's what it does:

The Filtering Mail FAQ has a section with more variants you can try if the one in the manual page and this one don't seem to work.

Q: I have this recipe which I basically copied from someone's working .procmailrc but it gives me funny errors such as "Badly placed ()'s" or "foo: not found". What's happening here?

A: The first error message is a Csh error message (also "Too many ('s") -- add the following to your .procmailrc and try again:

	SHELL=/bin/sh
The second error comes from a Bourne shell which can't find the command named foo. It probably means your PATH is not set up properly, or a program the recipe needs doesn't exist on your system, or the recipe author used the (somewhat dubious) convention of defining commands as Procmail variables and using these variables in the actual recipes:
	:0
	* ? $FORMAIL -XFrom: -XSender: | $FGREP cyberpromo.com
        /dev/null
The above will only work if the variables FORMAIL and FGREP are set to suitable values such as FORMAIL=/usr/local/bin/formail and FGREP=/usr/bin/fgrep earlier in your .procmailrc. (And you can't just snatch these values from here either -- ask your local gurus [or the recipe author] to help you out if you don't understand what this does.)

Q: Yikes! Where did my mail go??

A: Usually, to a file in your MAILDIR. Look in the log for clues (you were using the logging facility, yes? Yes?) and in your MAILDIR for files called something funny such as "*" or "# I wonder if Procmail allows comments here" (the quotes are not part of the file names and these file names are not necessarily representative).

Q: I copied the recipe from the procmailex manual page for checking for duplicates, but now I get a lot of log entries saying this invocation of formail is failing. What's up?

A: It's supposed to work that way. What the recipe does is, if formail detects a duplicate, it will succeed, which makes Procmail think the message is now delivered. In other words, the message was just delivered to the bit bucket because formail gulped it and didn't save it anywhere. When a message is not a duplicate, formail fails, Procmail notices delivery was not successful, and Procmail falls back to continued processing.

Q: I have a filter which works fine most of the time, but sometimes it will fail and I get a log message that Procmail has "rescued" lost data. Huh? Why is that?

A: It's a feature. Normally, you want Procmail to keep track of what your recipes are doing, and salvage your data if something goes wrong with one of them. If you have a recipe like this:

	# Old Stan always writes boring messages, 
	# we only really need to see the first five lines of'em
	:0fbw
	* ^From: stanislaus@poland\.com
	| head -5
it will work some of the time, when the message from Stan is short enough, but that's a coincidence. With a longer message, though, Unix starts paying attention to what is happening, because it will have to buffer some of the data, and then when the buffered data is never read, an error occurs. The error is passed back to Procmail, and Procmail tries to be nice and give you back your original message as it was before this malicious program truncated it. Never mind that in this case you wanted to truncate the data. Anyway, the fix is easy: Just add an :i flag to the recipe (:0fbwi instead of :0fbw) to make Procmail ignore the error.

Q: biff works just fine for notifying me of mail in my system mailbox, but I don't seem to get notifications for mail delivered to mailboxes in my home directory, or it keeps telling me I have mail in the system mailbox when in fact it was delivered to a different mailbox. How can I fix that?

A: Procmail by default sends a notification message for all delivered messages. The COMSAT pseudovariable controls this; see the documentation for details. So for starters, make sure you have COMSAT set to the right thing -- in a verbose log, you should see something like

	procmail: Notified comsat: "you@offset:file"
for each delivered message.

Unfortunately, many systems come with an ancient version of comsat(8) which doesn't pay attention to the last part of this notification. Thus, you will see apparently random biff notices, or none at all for mail delivered outside your system spool.

The solution is to [get your admin to] install a comsat daemon which understands the newer, extended form of the comsat protocol.


How do I ...?

Running Procmail

This section merely consists of quick questions. Okay, some of them are not so quick. Don't try to set up virtual domains if you don't know what you are doing.

Put another way, this is where the really Frequently Asked Questions are answered.

Put another way, this is the kitchen sink.

"Related" quick questions

Q: Why can't I match on the BCC: header?

A: You can't match on the BCC: header because it is not present in the message you receive. If the sending program is functioning correctly, it will copy the recipient data from the BCC: header into the "envelope", so it knows where to send the message, and then completely discard the BCC: header from the message. This is what the "Blind" in "Blind Carbon Copy" stands for; none of the recipients are supposed to see who's been BCC:ed. (Okay, so the spec has a few more twists. This is how Sendmail does it. Some other mailers will show the BCC list to the people who are being BCC:ed.)

Q: The Received: header seems to often contain something like Received: from elsewhere (...) by somewhere (...) for <recipient@host.domain> even if the recipient was Bcc:ed -- can't I rely on that?

A: No. Next question?

The problem is that (a) not all MTA:s add this information, and (b) if the message was sent to several recipients at your site, this information will not be there.

Q: I have a recipe which I would like to run on many messages which are already in one big happy mailbox file. How can I tell Procmail to not treat them all as a single long message?

A: You don't, you use formail to split it it back into separate messages and feed each into its own little procmail process.

	formail -ns procmail experiments.rc < test.mbox

(This will use the recipes in the rc file experiments.rc.)

Be careful with the -n switch if you have a large amount of messages to process.

Q: I can extract the sender's email address with formail -zxFrom: but this still often results in something like Donaldus Anas <don@see.va> or don@see.va (Donaldus Anas). How can I trim this down to just don@see.va?

A: The canonical way to get the sender's trimmed-down address is

	formail -rtzxTo:
This looks a bit counter-intuitive, but it works fine, provided that you don't mind the fact that formail -rt will prefer e.g. the value of Reply-To: over From:.

This will in fact do two things, which is not apparent if you're not too familiar with formail. One: It will generate a reply header (formail -rt). Two: Out of this new header, it will extract the To: field (formail -zxTo:). By happy coincidence, the generated reply address will only contain the actual email terminus, without the full name or other comments. Voila.

None of this is important if you merely want to find an address to send off a reply to (all you need for that is formail -rt and pass it to Sendmail). An address with a full name will do just fine for replying; any decent mail program should accept that (although you probably need to put it in double quotes to pass it as a single argument). For keeping a database of addresses, though, the stripped-down version is better (but still not perfect, and keep in mind that the host.domain part is not case sensitive).

Typical newbie mistake: formail -rtzxFrom:; this will actually return nothing at all, but you could in fact expect it to return your own address.

If this approach doesn't suit you for some reason, you could always post-process the address through some simple script which successively strips off layers of comments and whitespace. A fully RFC822-compliant script isn't trivial to write, but for many purposes, a simple sed script will probably do. (If you're truly masochistic, try to figure out how to get sendmail to strip the address and hand it back to you :-)

Perhaps you also want to look at formail -rD which keeps record of a limited number of addresses and expires the oldest ones as it gets fed new addresses.

Q: I've seen all these recipes which use EXITCODE= but how can I tell what exit codes to use?

A: The file /usr/include/sysexits.h should contain a good listing. If not, ask your system administrator where to get this information. These exit codes are fairly standardized but you should still not be surprised if something turns out to be different at your site. Some things are also probably more or less Sendmail-centric.

(The mechanism when you use this from Procmail is that Procmail quits with the specified error code, and the calling program is expected to catch that and act accordingly, i.e. generate a bounce message in most situations. If you are calling Procmail from a program which doesn't do this, there will obviously be no bounce. So if you test Procmail from your shell prompt and you see **Bounced** in the log, it only really means Procmail quit with a non-zero exit code.)

Not all programs return sensible error codes. If you see "mailer died with error 12 and a half" in a bounce message, it probably means the recipient ran his own home-grown filter on your message and it just happened to die with a more or less random and nonstandard exit code. Procmail, of course, is an excellent wrapper to put around such misbehaving programs.

If your /usr/lib/sendmail is in fact not a real Sendmail, it probably doesn't care about Procmail's exit code. At least Smail is rumored to behave this way. Bottom line, as always: test your recipes.

Q: How can I prevent my .forward from being included in sendmail's bounce messages?

I am rejecting some unwanted messages by setting EXITCODE to a suitable value (see separate question about that) but the generated bounce will include my .forward file, which will reveal to a human reader that my account in fact does exist, and it won't really allow me to explain the reason for the bounce, either. Can I make the bounce include a better explanation to the receiver of the bounce, or otherwise customize it?

A: If your site is using Procmail as your local MDA, you shouldn't have this problem. Perhaps you can persuade your admin to install Procmail site-wide, in which case you will have more control over generated bounce messages in general.

If that's not an option, your best bet is probably to simply generate an autoreply which looks like a bounce message, instead of letting sendmail generate the bounce message for you. Required reading: RFC1894, or get one of the existing bounce generators (there's at least one; see the Links page for more information).

Q: How can I extract (or kill) MIME parts from composite messages?

A: While there is nothing inherently very complicated about MIME multipart messages, the problem is not particularly a Procmail problem. You cannot write a Procmail recipe or a sed script which can parse them on its own because the MIME format is -- after all -- complicated enough to require a bit more muscle than simple regular expressions.

You can write your own Perl script to pick apart the messages (recommended: get the Perl MIME module instead of inventing your own wheel) or use a dedicated program such as munpack to handle them. (See the MIME FAQ for pointers.)

An unsound thing to do, which people ask about a lot anyway, is to kill all MIME messages. This should be easy (all compliant MIME messages contain a Mime-Version: header) but not smart, because MIME is merely an extension technology, not a sign that the message contains anything in particular that people usually want to filter with this measure (such as a huge file attachment, or HTML-encrypted message text, or text in the Latin-1 character set or a number of other character sets, including the default US-ASCII, or anything else that MIME can be a vehicle for). Again, reading up on MIME would probably be a good idea if you are thinking about this.

Q: How do I implement a virtual domain in Procmail?

Or, is Procmail even the right tool for this? (The short answer is no.)

A: It has been asked a number of times if it's possible to implement a good forwarding scheme which can take care of things like Cc:s to different users in the same virtual domain.

Here's what you might have started out with:

	:0
	* ^TO_jill@mydomain\.virtual
	! jill@real.address

	:0
	* ^TO_jane@mydomain\.virtual
	! jane@somewhere.else

The problem now is that if something comes in To: both jane and jill, (with infinite variations of Cc:s, Resent-To:, etc) it will only be forwarded to jill. You might try to generate a duplicate for each respondent you see, and kill off the message when it doesn't contain any more, but that doesn't really solve anything, either, and you'll have to get creative again in order to stop Procmail from generating too many duplicates. It all gets pretty hairy.

The short answer is that Procmail is not the right tool for this.

If you really want this to work, you will also want BCC:s to work (see the BCC question above), which is usually not possible without tweaking your Sendmail configuration. In which case you might as well implement some or all of the actual forwarding mechanism in Sendmail, which is the right tool anyway.

The basic problem is that when mail gets delivered into mailboxes at the upstream, the envelope information (like who the message is for) is usually discarded. So you shouldn't deliver into a mailbox that is not the correct mailbox, you should continue to transport the mail until you are ready to deliver to the final destination.

You can tweak Sendmail to copy selected envelope fields to the message header, but you don't have to do that if you don't deliver the message prematurely.

In a typical my-home-computer-is-my-domain setting, you could instead transport mail to your home computer using UUCP. fetchmail can allegedly also handle this for you, allowing you to have all mail for your domain delivered to a single POP account with the envelope information intact.

Q: Procmail is broken in this respect, isn't it?

A: No. If every problem begins to look like a nail, you need to have more tools than just the hammer in your toolbox.

Q: So how do I configure Sendmail to handle virtual domains?

A: Now, this is a Sendmail topic, but there's an instructive message about this from Philip Guenther which is available from the archives of the Procmail mailing list, volume 96 no 169. Or you can look at this local copy

Also check out the Sendmail FAQ, which discusses this at some length. And please visit http://www.sendmail.org/ if you haven't already.

Q: How do I implement a "vacation" program? A spam auto-responder?

A: Look at the procmailex man page. For an even simpler solution (in that it's less reading for you and presumably a better and more stable solution than you could build on your own in a couple of hours), get Alan K. Stebbens's Procmail library (see below) and tweak to suit your needs.

If unsolicited bulk e-mail, aka spam, is a problem for you, your best defense is to get your ISP to filter your mail at the SMTP and/or router level. The next best option is to install one of the many existing Procmail antispam packages. The Links page has pointers to a host of'em. If you want to discuss the minutia of detecting spam, a dedicated spam list is probably a better forum than the Procmail-L mailing list.

Bouncing unwanted messages is not necessarily a smart thing to do.


Where can I learn more?

A small links collection

Kudos
This Procmail FAQ draws heavily on Nancy McGough's excellent collection of mail filtering resources. If you only bookmark one web page this year, make sure it's an Infinite Ink one!
This FAQ would also be impossible -- and perhaps unnecessary :-) -- were it not for the Procmail-L mailing list. If you're serious about Procmail, you need to be on this list. It also happens to be one of the most helpful bunches of people on the 'Net. No names mentioned, in order not to offend those I'd no doubt forget to remember -- you know who you are!
Finally, none of this would be even remotely possible without the skill, dedication, and genius of Procmail's author, Stephen R. van den Berg. The 'Net community owes you one, Stephen.


The official location of this page is http://www.iki.fi/~era/procmail/mini-faq.html -- please use this redirector URL instead of whatever it happens to point to when making references to this page.

HTML 3.2 Checked!

To Links page * Feedback and comments are most welcome!

The cover page has a "to do" section with some ideas. If you think you can help out with any of them, it would be very much appreciated.

A text-only version of this Procmail FAQ is also available by e-mail (via Procmail of course); send a message to era+pr@iki.fi with a Subject: header containing only the words send procmail-faq
... to get a copy of the experiments.rc file, use send experiments.rc