Simscan/Guide

From Qmailwiki
(Difference between revisions)
Jump to: navigation, search
(Do not run spamassassin)
(spelling corrections)
 
(14 intermediate revisions by 6 users not shown)
Line 4: Line 4:
  
 
If you need more features than this you should consider using qmail-scanner. It is a perl program with many excellent features. However it places a higher load on the system. This is mostly due to using perl. The system has to envoke the perl process which is fairly large. simscan is less than 100K and uses the standard system libraries so it loads and runs much faster.
 
If you need more features than this you should consider using qmail-scanner. It is a perl program with many excellent features. However it places a higher load on the system. This is mostly due to using perl. The system has to envoke the perl process which is fairly large. simscan is less than 100K and uses the standard system libraries so it loads and runs much faster.
 +
 +
= Attachment Processing =
 +
To enable attachment processing in simscan use the following configure option
 +
 +
  --enable-attach
 +
 +
You also need to install ripmime to extract the mime parts of the e-mail. If you installed ripmime in a place other than /usr/local/bin/ripmime, specify it's PATH in the following configure option:
 +
 +
  --enable-ripmime=PATH
 +
 +
Place your list of attachments to be blocked in the /var/qmail/control/ssattach file.
 +
The list should look something like this:
 +
 +
.scr
 +
.pif
 +
.bat
 +
 +
Each time an email comes in and simscan is started by qmail-smtpd, this file is read into an array. After simscan calls ripmime to break the email MIME parts into separate files in the tmp directory /var/qmail/simscan, and then simscan will check the list of attachments against the list of file names in the email. If the attachments are set to be blocked, Simscan will remove the temporary files and tell qmail-smtpd to reject the message.
 +
 +
In order to make file names and attachment names case insensitive, simscan forces everything to lower case. Then it does a reverse string comparison for each file name against each attachment name. If there is a match then the email is rejected and control passes back up to qmail-smtpd for the final email rejection.
 +
 +
Simscan will then report to the smtp log a message similar to:
 +
  simscan: IP-of-sender pid pid-of-simscan: invalid attachment: FileName from: FromAddress to: ToAddress
 +
 +
If custom rejection messages are enabled then qmail-smtpd will report a failure message similar to:
 +
  Your email was rejected because it contains a bad attachment: FileName
 +
 +
The main advantages of using attachment blocking are:
 +
* You can block files regardless of whether they are virus infected or not. This becomes handy when new undetectable viruses appear on the Net and start spreading through, for example, .bat .scr and .pif files. Block these extensions permanently, users never need these files. If you are not using attachment blocking, these viruses will sneak in untill your antivirus software updated its virus database.
 +
* attachment blocking is *very* lightweight. You don't have to spawn clamdscan, it doesn't have to scan stuff; It's just simply rejected.  If you have a large volume of email going through your server this can make the difference between being fine with your current hardware, or needing to buy another server.
 +
* Its an opportunity to reduce bandwidth and employees "play time" if you block the funny .mp3, .avi, .mpeg, etc, files.
 +
 +
 +
If you are not interested in attachment blocking, just don't use '--enable-attach' configure option. Attachment blocking is disabled by default. You should then enable one antivirus scanner to scan your messages for viruses.
  
 
= ClamAntiVirus Processing =
 
= ClamAntiVirus Processing =
ClamAV processing is one of the simplist parts of simscan. The simscan configure script defaults to doing just clamav processing. If you are testing simscan for the first time you might want to start with just clamav processing. In the source code directory run just: ./configure ; make ; make install-strip. That should be enough to just enable clamav processing.  
+
ClamAV processing is one of the simplest parts of simscan. The simscan configure script defaults to doing just clamav processing. If you are testing simscan for the first time you might want to start with just clamav processing. In the source code directory run just: ./configure ; make ; make install-strip. That should be enough to just enable clamav processing.  
  
 
It is probably worth while to understand the steps simscan will go through at this point. After qmail-smtpd receives the email and starts handing it to simscan, the following will happen.
 
It is probably worth while to understand the steps simscan will go through at this point. After qmail-smtpd receives the email and starts handing it to simscan, the following will happen.
Line 14: Line 48:
 
# The email is written to another file
 
# The email is written to another file
  
Under default settings simscan then calls ripmime to break up mime encoded parts of the email. This step might not be needed but you should check your clamav settings. Clamav can scan email messages. Check your clamav.conf (pre 0.80 version) or your clamd.conf file. Look for the following section:
+
Clamav can break up mime encoded parts of the email and scan inside  them to find viruses. Check your clamav.conf (pre 0.80 version) or your clamd.conf file to see if Clamav e-mail scanner is enabled. Look for the following section:
  
 
  # Enable internal e-mail scanner.
 
  # Enable internal e-mail scanner.
Line 20: Line 54:
 
  #ScanMail
 
  #ScanMail
  
This is from a 0.80 installation. As you can see, email scanning is enabled by default. If you have disabled email scanning you will need ripmime processing. However, if email scanning is enabled, you can save alittle processing time by telling simscan to NOT run ripmime. Reconfigure your simscan with the --disable-ripmime option. Go back into your simscan source directory and run these commands:
+
This is from a 0.80 installation. As you can see, email scanning is enabled by default. If you have disabled email scanning you will need ripmime processing. However, if email scanning is enabled and you're not interested in performing attachment blocking, you can save a little processing time by telling simscan to NOT run ripmime. Reconfigure your simscan with the --disable-ripmime option. Go back into your simscan source directory and run these commands:
  
 
# make distclean
 
# make distclean
Line 31: Line 65:
 
In any case, after the email files are written and ripmime is called or not called, simscan forks a copy of clamdscan. Using pipes, simscan will read clamdscan's output. clamdscan will run, passing the file information to your clamd daemon. Make sure your clamd process is running. clamd will scan the files in the temporary working directory and report back to clamdscan. clamdscan will then report the results which simscan reads. simscan parses the results and looks for reports of a virus.
 
In any case, after the email files are written and ripmime is called or not called, simscan forks a copy of clamdscan. Using pipes, simscan will read clamdscan's output. clamdscan will run, passing the file information to your clamd daemon. Make sure your clamd process is running. clamd will scan the files in the temporary working directory and report back to clamdscan. clamdscan will then report the results which simscan reads. simscan parses the results and looks for reports of a virus.
  
If a virus is found then simscan removes the temporary working directory and all the files and reports a permenent failure back to qmail-smtpd. qmail-smtpd will then report the pemement error back to the sender and clean up all of the data that came in for that email. The result is the data with the virus is removed from the machine before it can cause any problems.
+
If a virus is found then simscan removes the temporary working directory and all the files and reports a permanent failure back to qmail-smtpd. qmail-smtpd will then report the permanent error back to the sender and clean up all of the data that came in for that email. The result is the data with the virus is removed from the machine before it can cause any problems.
  
 
== Optional smtp reject message ==
 
== Optional smtp reject message ==
 
If you patch your qmail source with the smtp reject message patch from the simscan/contrib directory, and you re-configure simscan with the --enable-custom-smtp-reject option then the smtp reject message returned by qmail-smtpd will contain the name of the virus that was found.  
 
If you patch your qmail source with the smtp reject message patch from the simscan/contrib directory, and you re-configure simscan with the --enable-custom-smtp-reject option then the smtp reject message returned by qmail-smtpd will contain the name of the virus that was found.  
  
How does this work? If simscan detects a virus it exits with a special error number: 82. simscan will also write a special reject message to file descriptor 4. If qmail-smtpd receives that return code then it will attempt to read the reject message from file descriptor 4 and report that message back to the sender. This is a pretty nifty feature. What I like about it is, if the email is  being sent by a real user, as opposed to a zombie pc sending out viruses, then the real user should receive back a bounce message  telling them which virus they have on their machine. I know I would like to know if my machine had a virus that was going out in my email. But then again I do not use Microsoft products..
+
How does this work? If simscan detects a virus it exits with a special error number: 82. simscan will also write a special reject message to file descriptor 4. If qmail-smtpd receives that return code then it will attempt to read the reject message from file descriptor 4 and report that message back to the sender. This is a pretty nifty feature. What I like about it is, if the email is  being sent by a real user, as opposed to a zombie pc sending out viruses, then the real user should receive back a bounce message  telling them which virus they have on their machine. I know I would like to know if my machine had a virus that was going out in my email. But then again I do not use Microsoft products...
  
 
= SpamAssassin Processing =
 
= SpamAssassin Processing =
Line 43: Line 77:
 
There are four ways to use spamassassin with simscan.
 
There are four ways to use spamassassin with simscan.
 
== Do not run spamassassin ==
 
== Do not run spamassassin ==
The simplist way to use spamassassin and simscan is not to use it at all.
+
The simplest way to use spamassassin and simscan is not to use it at all.
 
Some sites prefer to have spamassassin run on local email delivery. vpopmail
 
Some sites prefer to have spamassassin run on local email delivery. vpopmail
 
has an --enable-spamassassin option that tells vdelivermail, the local vpopmail delivery agent, to
 
has an --enable-spamassassin option that tells vdelivermail, the local vpopmail delivery agent, to
 
send the email through spamassassin with vpopmail user options so individual users can set their
 
send the email through spamassassin with vpopmail user options so individual users can set their
own perferences. maildrop can do this as well (I think).
+
own preferences. maildrop can do this as well (I think).
  
 
By having your local delivery agent run spamc you avoid the multiple recipient problem.
 
By having your local delivery agent run spamc you avoid the multiple recipient problem.
Line 60: Line 94:
 
system wide spamassassin preferences.  
 
system wide spamassassin preferences.  
  
But if there is more than one recipient then which user perference should simscan set?
+
But if there is more than one recipient then which user preference should simscan set?
 
The solution simscan uses is if there is more than one recipient and the --enable-spamc-user
 
The solution simscan uses is if there is more than one recipient and the --enable-spamc-user
 
option is set, then it skips adding the -u user@domain option to spamc.
 
option is set, then it skips adding the -u user@domain option to spamc.
Line 107: Line 141:
  
 
   --enable-spam-passthru
 
   --enable-spam-passthru
 
= Attachment Processing =
 
To enable attachment processing in simscan use the following configure option
 
 
  --enable-attach
 
 
Place your list of attachments in the /var/qmail/control/ssattach file.
 
The list should look something like this:
 
 
.jpg
 
.gif
 
.exe
 
 
Each time an email comes in and simscan is started by qmail-smtpd, this
 
file is read into an arrary. After simscan calls ripmime to break the
 
email MIME parts into separate files, simscan then checks the list of
 
attachments against the list of file names in the email.
 
 
In order to make file names and attachment names case insensitive, simscan
 
forces everything to lower case. Then it does a reverse string comparision
 
for each file name against each attachment name. If there is a match then
 
the email is rejected and control passes back up to qmail-smtpd for the
 
final email rejection.
 
 
Simscan will then report to the smtp log a message similar to:
 
  simscan: IP-of-sender pid pid-of-simscan: invalid attachment: FileName from: FromAddress to: ToAddress
 
 
If custom rejection messages are enabled then qmail-smtpd will report a failure message similar to:
 
  Your email was rejected because it contains a bad attachment: FileName
 
  
 
= Per domain processing =
 
= Per domain processing =
Line 165: Line 170:
 
First the sender address will be looked up and then the recipients. Without any matches, no scans will be done.
 
First the sender address will be looked up and then the recipients. Without any matches, no scans will be done.
  
Then run /var/qmail/bin/simscanmk to build the simcontrol.cdb file. You can rebuild this files at any time. The simscanmk program can safely update the cdb files while the system is running.
+
Then run /var/qmail/bin/simscanmk to build the simcontrol.cdb file. You can rebuild this file at any time. The simscanmk program can safely update the cdb files while the system is running.
  
 
Qmail extensions are handled like this: the address is broken up into its parts and each one is looked up. For example, test-list-owner@example.com looks up:
 
Qmail extensions are handled like this: the address is broken up into its parts and each one is looked up. For example, test-list-owner@example.com looks up:
  
   test@test.ch
+
   test@example.com
   test-list@test.ch
+
   test-list@example.com
   test-list-owner@test.ch
+
   test-list-owner@example.com
  
 
== regex matching ==
 
== regex matching ==
Line 197: Line 202:
 
* If one of these three regexes matches against any line of the mail, the mail is refused
 
* If one of these three regexes matches against any line of the mail, the mail is refused
 
* This regex was created to block some sober variant cheaper (system resource wise) than with a virus scanner.
 
* This regex was created to block some sober variant cheaper (system resource wise) than with a virus scanner.
 +
  
 
== attachment blocking ==
 
== attachment blocking ==
Line 202: Line 208:
  
 
The ''attach'' option is for banning specific attachment types, and is only available if --enable-attach was used at compile-time. The string must be a colon-separated list of extensions, like in the first line of the example above.
 
The ''attach'' option is for banning specific attachment types, and is only available if --enable-attach was used at compile-time. The string must be a colon-separated list of extensions, like in the first line of the example above.
 +
 +
 
== refusing spam ==
 
== refusing spam ==
 
   spam_hits=number
 
   spam_hits=number
Line 208: Line 216:
  
 
   spam_passthru=(yes|no)
 
   spam_passthru=(yes|no)
 +
 +
This option requires simscan to be configured with --enable-spam-passthru.
  
 
The ''spam_passthru'' option is equivalent to the --enable-spam-passthru configure option.
 
The ''spam_passthru'' option is equivalent to the --enable-spam-passthru configure option.
Line 218: Line 228:
 
Use this option when configuring simscan
 
Use this option when configuring simscan
  
   --enable-drop-msg
+
   --enable-dropmsg
  
 
= Custom Reject Option =
 
= Custom Reject Option =
Line 225: Line 235:
  
 
This option has to be used with the custom reject message patch to qmail-smtpd. The patch is available in the simscan tar ball contrib directory.
 
This option has to be used with the custom reject message patch to qmail-smtpd. The patch is available in the simscan tar ball contrib directory.
 
 
..
 

Latest revision as of 20:32, 24 August 2009

Contents

Introduction

simscan was written for people who need a fast and efficent way to do virus scanning or spamassassin processing or attachment blocking


If you need more features than this you should consider using qmail-scanner. It is a perl program with many excellent features. However it places a higher load on the system. This is mostly due to using perl. The system has to envoke the perl process which is fairly large. simscan is less than 100K and uses the standard system libraries so it loads and runs much faster.

Attachment Processing

To enable attachment processing in simscan use the following configure option

 --enable-attach

You also need to install ripmime to extract the mime parts of the e-mail. If you installed ripmime in a place other than /usr/local/bin/ripmime, specify it's PATH in the following configure option:

 --enable-ripmime=PATH

Place your list of attachments to be blocked in the /var/qmail/control/ssattach file. The list should look something like this:

.scr
.pif
.bat

Each time an email comes in and simscan is started by qmail-smtpd, this file is read into an array. After simscan calls ripmime to break the email MIME parts into separate files in the tmp directory /var/qmail/simscan, and then simscan will check the list of attachments against the list of file names in the email. If the attachments are set to be blocked, Simscan will remove the temporary files and tell qmail-smtpd to reject the message.

In order to make file names and attachment names case insensitive, simscan forces everything to lower case. Then it does a reverse string comparison for each file name against each attachment name. If there is a match then the email is rejected and control passes back up to qmail-smtpd for the final email rejection.

Simscan will then report to the smtp log a message similar to:

 simscan: IP-of-sender pid pid-of-simscan: invalid attachment: FileName from: FromAddress to: ToAddress

If custom rejection messages are enabled then qmail-smtpd will report a failure message similar to:

 Your email was rejected because it contains a bad attachment: FileName

The main advantages of using attachment blocking are:

  • You can block files regardless of whether they are virus infected or not. This becomes handy when new undetectable viruses appear on the Net and start spreading through, for example, .bat .scr and .pif files. Block these extensions permanently, users never need these files. If you are not using attachment blocking, these viruses will sneak in untill your antivirus software updated its virus database.
  • attachment blocking is *very* lightweight. You don't have to spawn clamdscan, it doesn't have to scan stuff; It's just simply rejected. If you have a large volume of email going through your server this can make the difference between being fine with your current hardware, or needing to buy another server.
  • Its an opportunity to reduce bandwidth and employees "play time" if you block the funny .mp3, .avi, .mpeg, etc, files.


If you are not interested in attachment blocking, just don't use '--enable-attach' configure option. Attachment blocking is disabled by default. You should then enable one antivirus scanner to scan your messages for viruses.

ClamAntiVirus Processing

ClamAV processing is one of the simplest parts of simscan. The simscan configure script defaults to doing just clamav processing. If you are testing simscan for the first time you might want to start with just clamav processing. In the source code directory run just: ./configure ; make ; make install-strip. That should be enough to just enable clamav processing.

It is probably worth while to understand the steps simscan will go through at this point. After qmail-smtpd receives the email and starts handing it to simscan, the following will happen.

  1. A temporary working directory is created and simscan changes it's working directory there.
  2. The envelope (to and from addresses) are written to a file
  3. The email is written to another file

Clamav can break up mime encoded parts of the email and scan inside them to find viruses. Check your clamav.conf (pre 0.80 version) or your clamd.conf file to see if Clamav e-mail scanner is enabled. Look for the following section:

# Enable internal e-mail scanner.
# Default: enabled
#ScanMail

This is from a 0.80 installation. As you can see, email scanning is enabled by default. If you have disabled email scanning you will need ripmime processing. However, if email scanning is enabled and you're not interested in performing attachment blocking, you can save a little processing time by telling simscan to NOT run ripmime. Reconfigure your simscan with the --disable-ripmime option. Go back into your simscan source directory and run these commands:

  1. make distclean
  2. ./configure --disable-ripmime
  3. make
  4. make install-strip

If everything goes well, you will now have disabled ripmime processing and saved yourself some system load.

In any case, after the email files are written and ripmime is called or not called, simscan forks a copy of clamdscan. Using pipes, simscan will read clamdscan's output. clamdscan will run, passing the file information to your clamd daemon. Make sure your clamd process is running. clamd will scan the files in the temporary working directory and report back to clamdscan. clamdscan will then report the results which simscan reads. simscan parses the results and looks for reports of a virus.

If a virus is found then simscan removes the temporary working directory and all the files and reports a permanent failure back to qmail-smtpd. qmail-smtpd will then report the permanent error back to the sender and clean up all of the data that came in for that email. The result is the data with the virus is removed from the machine before it can cause any problems.

Optional smtp reject message

If you patch your qmail source with the smtp reject message patch from the simscan/contrib directory, and you re-configure simscan with the --enable-custom-smtp-reject option then the smtp reject message returned by qmail-smtpd will contain the name of the virus that was found.

How does this work? If simscan detects a virus it exits with a special error number: 82. simscan will also write a special reject message to file descriptor 4. If qmail-smtpd receives that return code then it will attempt to read the reject message from file descriptor 4 and report that message back to the sender. This is a pretty nifty feature. What I like about it is, if the email is being sent by a real user, as opposed to a zombie pc sending out viruses, then the real user should receive back a bounce message telling them which virus they have on their machine. I know I would like to know if my machine had a virus that was going out in my email. But then again I do not use Microsoft products...

SpamAssassin Processing

SpamAssassin section

There are four ways to use spamassassin with simscan.

Do not run spamassassin

The simplest way to use spamassassin and simscan is not to use it at all. Some sites prefer to have spamassassin run on local email delivery. vpopmail has an --enable-spamassassin option that tells vdelivermail, the local vpopmail delivery agent, to send the email through spamassassin with vpopmail user options so individual users can set their own preferences. maildrop can do this as well (I think).

By having your local delivery agent run spamc you avoid the multiple recipient problem.

Multiple Recipient Problem with Per User simscan option

You can enable the use of spamassassin's per user option with this option:

 --enable-spamc-user

If there is a single recipient in the email then simscan adds the -u user@domain option to spamc. This tells spamassassin to look up the users preferences to over ride the system wide spamassassin preferences.

But if there is more than one recipient then which user preference should simscan set? The solution simscan uses is if there is more than one recipient and the --enable-spamc-user option is set, then it skips adding the -u user@domain option to spamc.

This problem is avoided by having your local delivery agent do the spamassassin processing.

Scan and pass through to users

You can also set simscan to pass the email through spamassassin and send the result on to qmail. This is done with two options:

 --enable-spam
 --enable-spam-passthru

Do not use this option:

 --enable-spam-hits

The --enable-spam-passthru option is ignored if you use:

 --enable-per-domain

Reject anything marked as spam

Another option is to have simscan reject email (smtp failure 500 response) when spamassassin detects anything marked as spam. Use this option

 --enable-spam

and do not use these options

 --enable-spam-hits
 --enable-spam-passthru

With this setup, any email designated as spam with the X-Spam flag set to Yes, will be rejected at the smtp level.

Reject really bad spam and pass through anything else

Perhaps the most popular method is a mix of the others. Which is to reject email that scores very high and to pass through other email to the user even if it is marked as spam. This seems to help with the "false positive" issue; email marked as spam but not really spam. Use these options:

 --enable-spam
 --enable-spam-hits=number
   where number is the spamassassin score at which you want to reject the email

Do not use this option:

 --enable-spam-passthru

Per domain processing

The per domain processing feature allows you to fine-tune scanning parameters on a per domain or per account (email address) base.

Enable this using --enable-per-domain when compiling simscan.

When this option is enabled, simscan is controlled by the simcontrol.cdb file, usually stored in /var/qmail/control. This file can be built, using simscanmk, from /var/qmail/control/simcontrol

Here is an example simcontrol file:

 postmaster@example.com:clam=yes,spam=no,attach=.txt:.com
 example.com:clam=no,spam=yes,attach=.mp3
 :clam=yes,spam=yes,trophie=yes,spam_hits=20.1

The options that are available (in version 1.1) are:

 clam=(yes|no)
 spam=(yes|no)
 trophie=(yes|no)

The clam, spam, and trophie options toggle whether or not to use those scanners for scanning - the binaries must have been available and their locations must have been configured when simscan was compiled.

Processing order of options

The order of precedence is:

 email address (overrides all)
 domain (overrides default)
 default (only used if not overridden by domain or email address.

First the sender address will be looked up and then the recipients. Without any matches, no scans will be done.

Then run /var/qmail/bin/simscanmk to build the simcontrol.cdb file. You can rebuild this file at any time. The simscanmk program can safely update the cdb files while the system is running.

Qmail extensions are handled like this: the address is broken up into its parts and each one is looked up. For example, test-list-owner@example.com looks up:

 test@example.com
 test-list@example.com
 test-list-owner@example.com

regex matching

 regex=perl_regex1:perl_regex2:perl_regex3:...

To enable regex matching you have to configure simscan using --enable-regex. You can have only one regular expression per "perdomain" match line. You can however seperate multiple regexes using a semicolon (:).

If you want to match on a semicolon you have to use \072 or \x3a in the regex.

The regex engine is libpcre, so refer to the pcre documentation for the patterns. Basically it takes perl regexes.

The regex option tries to match every line of the mail with all the regular expressions. If it matches, it refuses the mail. If you enabled the custom reject option, it tells you the regex number it matched.

A single regex can be 500 chars at most, and you can have 100 regexes per run applied.

Examples

 :regex=(?i)^\+\-\+\-\+.*Service:(?i)^\*-\*-\*.*Virus:(?i)^\*\-\*\-\*.*Service
  • Creates three regexes to match against
    1. (?i)^\+\-\+\-\+.*Service
    2. (?i)^\*-\*-\*.*Virus
    3. (?i)^\*\-\*\-\*.*Service
  • The (?i) at the beginning of the regex makes the regex a case insensitive match
  • If one of these three regexes matches against any line of the mail, the mail is refused
  • This regex was created to block some sober variant cheaper (system resource wise) than with a virus scanner.


attachment blocking

 attach=string

The attach option is for banning specific attachment types, and is only available if --enable-attach was used at compile-time. The string must be a colon-separated list of extensions, like in the first line of the example above.


refusing spam

 spam_hits=number

The spam_hits option changes the threshold spamassassin spam value. If a mail gets a spam score that number or above, it is rejected, otherwise it is accepted.

 spam_passthru=(yes|no)

This option requires simscan to be configured with --enable-spam-passthru.

The spam_passthru option is equivalent to the --enable-spam-passthru configure option.

IMPORTANT: if you use the --enable-per-domain option, the --enable-spam-passthru option is IGNORED. You MUST add spam_passthru=yes to any and all domains you wish to use it with.

Drop Message option

Some sites have security policies in place which require them to accept every email. For these sites there is an option to do all the normal simscan processing, but if a virus or spam is detected the message is not handed to qmail-queue for local delivery. Instead it is silently dropped.

Use this option when configuring simscan

 --enable-dropmsg

Custom Reject Option

 --enable-custom-reject

This option has to be used with the custom reject message patch to qmail-smtpd. The patch is available in the simscan tar ball contrib directory.

Personal tools