MemeStreams | MemeStreams Discussion

Create an Account

This page contains all of the posts and discussion on MemeStreams referencing the following web page: Spam Filtering. You can find discussions on MemeStreams as you surf the web, even if you aren't a MemeStreams member, using the Threads Bookmarklet.

Spam Filtering
by Decius at 9:44 am EDT, Jul 20, 2003

I thought I would post and see what people's experience has been with various spam filtering tools. A few comments:

1. I see spam as a law enforcement problem. The VAST majority of the spam I receive contains forged headers and is being relayed through systems without permission. If the government simply enforced the laws it already has, it could prosecute these people for computer fraud. This would eliminate most of the problem that I see. The reason that I get all this spam is because the government won't enforce their laws. (Although they are happy enough to raid internet "bong" dealers. Gosh I'm glad to be safe from them.)

2. I have one email address that is basically useless because of spam. For every legitimate email I receive there, I get 30 or more spams. My other email address is rapidly approaching this state.

3. The basic requirement for a spam filtering solution that I have is no false positives. I can deal with deleting some spam. Its not THAT big of a deal. If I could delete LESS spam, and still get all of my legitimate email, I think I'd be alright.

4. I don't trust RBL based solutions. RBLs block legitimate mail. Lots of it. I'm interested in blocking SPAM, not people who are using SMTP relay. This effort to lock down SMTP has been going on for years, and the amount of spam has not been reduced.

5. My only experience with "AI" like filters has been the spam filter in MacOSX. It doesn't work. It ids spam as legit. More importantly, it regularily IDs legit email as spam. I'm not sure how it measures up with other filters. If I put more effort into training it, it might get better, but I could never TRUST that if I turned it on it would never block legitimate email.

6. This morning I was considering implementing Challenge Response for all of my email. This sounds like an effective solution. Unfortunately, its not. The problem is that there are a number of bots out there, mostly related to ecommerce sites, that I probably do need to see email from. I can try to list them in my whitelist, but I risk missing something.

7. It occurs to me that what might work better then these solutions is something that relies on a network. If 100 people get the same email, its probably spam. This, I imagine, is what yahoo is doing. I think I've heard of systems that allow large numbers of people to coordinate to filter spam, but I don't recall what it is.

What systems are you using? How effective are they?

RE: Spam Filtering
by Dolemite at 12:06 pm EDT, Jul 20, 2003

Decius wrote:

] What systems are you using? How effective are they?

Personally, I'm using SpamAssassin with both the Bayesian Filters and Vipul's Razor. Razor is the implementation that creates a checksum on the message and compares it against a database of reported spam from others. This may be the collaborative system you were trying to remember the name of.

I have only once gotten a false positive with SpamAssassin, and that was in a message from Network Solutions. Go figure. Their email asking me to confirm that I wanted to transfer my domain away from them was ridden with spam-terms.

I still get about 10 spams per day that make it through the filters. If you're concerned about legitimate mail not making it through, you can change the sensitivity level of SpamAssassin from the default, which is 5.0 on a 10.0 scale, I believe.

Dolemite

RE: Spam Filtering
by leed25d at 4:49 pm EDT, Jul 20, 2003

Decius wrote:
] I thought I would post and see what people's experience has
] been with various spam filtering tools. A few comments:
]

I read my mail with emacs gnus. I split mail from known senders/domains first then run the remainder through SpamAsassin and run Vipul's razor on the rest. I'd say that 10-15 spams per day make it through. I have never found a false positive in over a year.

--lee

RE: Spam Filtering
by Laughing Boy at 8:45 am EDT, Jul 21, 2003

Decius wrote:
]What systems are you using? How effective are they?

Xwall for MS Exchange. Has many mechanisms for spam filtering, including black lists, Bayesian filter, verification of MX, blocking IP subnets, etc...

How well does it work? Not very. I have it configured to grab suspected spam and forward it to the recipient as an attachment. This means the user still gets all the spam, but they can set up rules with "inbox assistant" to delete these messages if they desire. This insures that nothing that is incorrectly detected as spam still gets to the intended recipient.

In my experience, with all the configuration I've done, Xwall grabs about 50% of the spam. Thats not bad, but I'd really like to see it catch closer to 90% and improve its accuracy at what is spam and what is not.

I had users dragging their spam out to a public folder on the exchange server. I would then go thru and manually extract IP addresses to build our own custom black list. I quickly realized this was a full-time job and abandoned it after a few weeks.

Legislation will work for spammers based here in the USA, but what about the scores of spammers in other countries?

Laughing Boy

RE: Spam Filtering
by bucy at 2:38 pm EDT, Jul 21, 2003

Decius wrote:
] I thought I would post and see what people's experience has
] been with various spam filtering tools. A few comments:
]
] 1. I see spam as a law enforcement problem. The VAST majority
] of the spam I receive contains forged headers and is being
] relayed through systems without permission. If the government
] simply enforced the laws it already has, it could prosecute
] these people for computer fraud. This would eliminate most of
] the problem that I see. The reason that I get all this spam is
] because the government won't enforce their laws. (Although
] they are happy enough to raid internet "bong" dealers. Gosh
] I'm glad to be safe from them.)

Agreed. The big problem is that a lot of it comes from abroad and you need to get the US govt to armtwist Korea, China, Russia, etc.

] 6. This morning I was considering implementing Challenge
] Response for all of my email. This sounds like an effective
] solution. Unfortunately, its not. The problem is that there
] are a number of bots out there, mostly related to ecommerce
] sites, that I probably do need to see email from. I can try to
] list them in my whitelist, but I risk missing something.

PGP is a similar solution -- only accept signed mails -- but lacks critical mass to work. A little weaker than this: I use qmail and often give ecommerce sites addresses like bucy-amazon@gloop.org, etc. Sendmail supports bucy+amazon ... don't know about other MTAs.

] What systems are you using? How effective are they?

I've been using SpamAssassin for awhile now. With a threshhold of 5, I don't think I've gotten a single false positive and I'd guess that the false negative rate is maybe 10%.