Email Processing
at Smith College
In the past twelve months, Smith College has received approximately
82 million email messages. In an ideal world all of these
would have been legitimate - unfortunately we do not live
in an ideal world.
Email
has become the major threat vector for computer virus delivery,
but it is the flood of unsolicited commercial email (or UCE – commonly
known as spam) that taxes email systems world wide. Apart
from the annoyance factor, some spam is of an explicit, distasteful,
and sometimes disturbing nature.
To combat these threats and
abuses, Smith College scans incoming messages. When viruses
or unquestionable spam is detected, the messages are deleted. If a message
contains some spam identifiers, but not enough to be absolutely positive,
then the message is either quarantined or delivered, depending
on the level of uncertainty.
From a user's perspective, email is simple – one
sends and receives messages. However, the choice of email client,
address format, and message content can have a profound effect on the
successful delivery of email.
The purpose of this document is to explain
in detail the message delivery processes used at Smith College
in order to help email users better understand the complexities of the
systems, as well as the factors under their control that can affect
message delivery.
The Big Picture
Smith College Email Overview
What Affects Message Delivery?
WebShield Actions
1 - Recipient Domain Check
2 - Permit Sender Check
3 - Deny Sender Check
4 - RBL (Real-time Blackhole List) Check
5 - Content Scanning
1 - User settings
2 - Recipient Verification
3 - System/User Trusted/Blocked Sender List Processing
4 - Real-time Blackhole List (RBL)
5 - Real-time Pattern Analysis (RPA)
6 - Virus Scanning
7 - Attachment analysis
8 - Remove active content
9 - Miscellaneous Scoring Parameters
10 - Unsolicited Commercial Email (UCE) Rules
11 - Explicit Content Rules
12 - Content Score Analysis & Action
The Big Picture
Of the approximately 82 million messages received at Smith in the past 12 months:
- 36.9 million were deleted because they were sent to invalid recipients
- 24.3 million were deleted because they came from known spam sources
- 7.4 million were deleted because their content identified them
as spam
- 3.2 million were quarantined as possible spam
- 10.2 million were delivered directly to users mailboxes
- 151 thousand were subsequently released from quarantine by users
Smith College Email Overview
The core email system in use
at Smith College is Novell GroupWise. Every faculty member, administrative staff
member, and student has a GroupWise account.
GroupWise runs as a distributed system spanning multiple
servers. Email delivery between GroupWise accounts is very
robust and efficient. Hardware issues can occasionally impact
the users of a particular post office, but system-wide downtime
is rare and more often as a result of planned maintenance
than technical problems.
Internal email delivery using GroupWise is very fast. Delivery
between users on the same post office is accomplished in
under 1 second, between users on different post offices in
30 seconds or less.
Most ‘internal’ email delays are incurred when the choice of email client,
addressing method, or use of an account on a foreign email system cause the message
to travel outside of GroupWise and be treated as external email.
The diagram below illustrates how GroupWise interfaces to
external systems. Note that Envoy (@smith.edu) and Scinix
(@scinix.smith.edu, @math.smith.edu etc) are all external
systems.
The server Envoy accepts all email addressed to auser@smith.edu. Envoy has special rules to redirect email for some recipients to different on and off campus addresses. If no such rule is found the recipient address is merely changed to auser@email.smith.edu and delivered to WebShield.
WebShield
The Smith Domain Name Service (DNS) has been configured to route mail for most Smith Mail systems to WebShield. The main exception being the @smith.edu domain which is routed first to Envoy as described above.
The primary function of WebShield is to detect and block virus infected messages from entering Smith. If a virus is detected the message is dropped and a rejection code is sent back to the sending mail system. No alert emails are generated for either the sender or recipient as in most cases the sender address is forged or invalid.
WebShield also performs preliminary blocking of spam from confirmed spam sources through the use of a list of rampant spam sender addresses and reference to the Spamhaus Real time Black hole List (RBL) system.
Spamhaus is an organization that analyzes millions of messages and identifies major sources of spam. It identifies the IP address of the known source rather than the address which can be forged or morphed to attempt to avoid detection.
MessageScreen
The MessageScreen server only accepts email from WebShield. MessageScreen scans for and blocks spam and explicit content.
As a result of the various processing modules a message may be passed for delivery, deleted or have a scoring points added or subtracted. The final scores are then assessed and the message is then deleted, quarantined or delivered.
Back to top >
What Affects Message Delivery?
The following sections describe the tests and actions of WebShield and MessageScreen. Items in red text indicate an action that will stop a message from being delivered.
1 - Recipient Domain Check
Domains Accepted:
- email.smith.edu
- scinix.smith.edu
The following domains are merely aliases of their parent domain, and their use is not recommended:
- [science.smith.edu]
- [felix.smith.edu]
- [math.smith.edu]
- [turing.smith.edu]
- [cs.smith.edu]
- [earth.ast.smith.edu]
- [ast.smith.edu]
- [halka.smith.edu]
All other recipient domains: Reject email and close connection.
2 - Permit Sender Check
Sender on Permit list: Skip Deny sender and RBL check;
Continue at step 5 - Content Scanning.
[eDigest sender domain]
3 - Deny Sender Check
Sender address or domain on list: Reject email and close
connection.
[Confirmed
spammer domain and email addresses]
4 - RBL (Real-time Blackhole List) Check
Sender on RBL list: Reject email and
close connection.
[references sbl-xbl.spamhaus.org]
- Virus detected: Refuse data and return a rejection code.
- Unwanted program detected (Spyware, Adware, Remote Admin tools, Dialers, Password Crackers): Refuse data and return a rejection code.
- Content keyword scanning: Disabled.
- Corrupt content: Refuse data and return a rejection code; Deliver notification email to sender.
- Disclaimer Text: Disabled.
- Encrypted content: Allow through; Deliver an annotated original message to system admin.
- File filtering: Disabled.
- MIME Partial Message: Refuse data and return rejection code; Deliver notification email to sender.
- MIME External body message: Refuse data and return rejection code; Deliver notification email to sender.
- MIME Null characters in headers: Treat and handle as corrupt content; Refuse the data and return a rejection code; Deliver notification email to sender.
- Number of MIME parts exceeds 1000: Treat and handle as corrupt content; Refuse the data and return a rejection code; Deliver notification email to sender.
- Header corruption: Continue processing.
- Mail size filtering - message is larger than 26,000 kilobytes: Refuse data and return a rejection code; Deliver notification email to sender; Deliver notification email to original recipients.
- Mail size filtering - attachment is larger then 26,000 kilobytes: Refuse data and return a rejection code; Deliver notification email to sender; Deliver notification email to original recipients.
- Mail size filtering - more than 500 attachments: Refuse data and return a rejection code.
- Protected content: Continue processing.
- Scanner Denial of Service detection:
- Nesting depth exceeds 100: Replace content with an HTML alert.
- Expanded file size exceeds 50MB: Refuse data and return a rejection code.
- Scan time exceeds 8 minutes: Replace content with an HTML alert.
- Signed Content: Allow changes to break the signed email; Deliver notification email to sender; Deliver notification email to original recipients.
- Anti-Relay: Deny routing characters - *!* *%* *|*
- Deferred email (retry period - 27 minutes): Retry lifetime greater than 48 hours: Discard message.
Enable Filtering is unchecked – user chose not to scan email for spam: Deliver Email.
2 - Recipient Verification
Target mail server returns undeliverable response: Discard Email.
3 - System/User Trusted/Blocked Sender List Processing
Rules can be applied to all or part of the sender address. Conflicting rule actions are resolved using the following priority levels:
Priority Address Granularity: Example -
1=Full sender address aspammer@mailer2.buymyjunk.com
2=Email domain @mailer2.buymyjunk.com
3= IP address of sending server 192.168.12.12
3= FQDN of sending server mailer2.buymyjunk.com
4= Network IP of sending server 192.168.12.1/24
4= Domain of sending server buymyjunk.com
On Admin Blocked Sender list, and:
- Not on Admin Trusted Sender list:
Discard Email.
- On Admin Trusted Sender list:
- Lower or equal priority to Admin Blocked Sender list: Discard Email.
- Higher Priority than Admin Blocked Sender List, and:
- On User Blocked Sender List:
- Lower or equal priority to Admin Trusted Sender List: Deliver Email.
- Higher priority than Admin Trusted Sender List: Discard Email.
- On User Blocked Sender List:
On User Blocked Sender List, and:
- Not on User Trusted Sender list: Discard Email.
- On User Trusted Sender list, and:
- Lower or equal priority to User Blocked Sender list: Discard Email.
- Higher priority than User Blocked Sender list: Deliver Email.
- On Admin Trusted Sender list: Deliver Email.
- On User Trusted Sender List: Deliver Email.
4 - Real-time Blackhole List (RBL)
Disabled – processed on WebShield.
5 - Real-time Pattern Analysis (RPA)
- Identified as Definite Spam: Discard Email.
- Identified as Probable Spam: Add 1500 points to UCE score.
- Identified as Possible Spam: Add 5 points to UCE score.
6 - Virus Scanning
Disabled – processed on WebShield.
7 - Attachment Analysis
Attachment File size check: Disabled.
Attachment
File extension check:
- Extension is .BAT, .CMD, .COM, .CPL, .EXE, .HTA, .PIF, .SCR, .VBE, .VBS, .WMF:
Discard Email.
- Extension is .ZIP: Park attachment; Prepend original email body with attachment park message; Include link for file download or deletion by recipient.
8 - Remove Active Content
Active content such as JavaScript and VBScript is removed from message body.
9 - Miscellaneous Scoring Parameters
Anti-Phishing Configuration:
- Invalid ‘From’ address in header: Add 25 points to UCE score.
- Invalid ‘From’ address in SMTP Envelope: Add 25 points to UCE score.
- Header and envelope ‘From’ do not match: Add 25 points to UCE score.
- No reverse resolution for sending mail Server: Add 10 points to UCE score.
- Reverse resolution for sending server does not match SMTP Envelope. Add 10 points to UCE score.
- For each embedded image: Add 20 points to UCE score.
10 – Unsolicited Commercial Email (UCE) Rules
Header rules: Over 2,000 rules. Add points to UCE score.
Body Rules: Over 30,000 rules. Add points to UCE score.
11 - Explicit Content Rules
Header rules: Over 2,000 rules. Add points to UCE score.
Body Rules: Over 10,000 rules. Add points to UCE score.
12 – Content Score Analysis and Action
Rules may add a few to several thousand points depending
on the confidence with which it identifies spam content.
Some rules assign a negative score in order to offset partial
content mismatches.
For example, one rule adds 100 points for ‘cialis’ the
drug name; another subtracts 100 points for ‘specialist’. In addition,
special rules look for certain ‘pass
phrases’ and assign huge negative scores in order to
guarantee the email will not be deleted or quarantined.
Content Score actions:
UCE score is greater than or equal to 2500 points and:
- Explicit Content Score greater than or equal to 2500 points: Discard Email.
- Explicit Content score greater than or equal to 150 and less than 2500 points: Quarantine Email.
- Explicit Content score is less than 150 points: Discard Email.
UCE score greater than or equal to 170 and less than 2500 points and:
- Explicit Content Score greater than or equal to 2500 points: Discard Email.
- Explicit Content score greater than or equal to 150 and less than 2500 points: Quarantine Email.
- Explicit Content score is less than 150 points: Quarantine Email.
UCE score is less than 170 points and:
- Explicit Content Score greater than or equal to 2500 points: Discard Email.
- Explicit Content score greater than or equal to 150 and less than 2500 points: Quarantine Email.
- Explicit Content score is less than 150 points: Deliver Email.
- User junk mail rules: Disabled.
- Other user rules: User-defined rules may delete messages without notification.
- Clean up Policy: User may define ‘Delete after n days’ policy which will cause the Post office to delete messages without user involvement.