Best settings & usage for Junk Mail filter?

[LGM]

Hello ... I have junk mail filtering on and have done my best to try to train it. But apparently I have something wrong as it keeps letting certain emails through, even though I keep 1) blacklisting the domains of the offending emails and 2) elect to "Classify as Junk and Move" emails I find to be junk. Even with these steps I keep on getting emails with the same or very similar Name in the "From" column and with similar subject lines. This should not be happening with a decent junk mail filter. So I figure I have something set wrong.

For example, mail with "acai berry" or "car insurance quote" or "slim shake" or "low carb shake" keep getting through. I hope I'm not expected to go into the settings and type in certain words to blacklist them myself. The program should be doing this for me automatically. If I have to type them in myself, that takes too long.

Any ideas?

[Tomas]

Hello,

it is bayesian filtering based antispam, thus it has to be taught for some time. It might happen that you would have to classify seemingly similar messages repeatedly before they will be catched in Junk folder.

Judging from the past conversations, people keep getting mixed results, some great, some less so, with the built-in filtering. Some reported that it takes some time to train it. Experiences might differ also due to the fact that we all are receiving different types of messages, and we all have different ways of training the filters.

You could also consider specialized antispam solutions as an alternative to the built-in filtering. Just from what I remember, there are many more, these are not recommendations, just a selection of what some of the users reported that worked for them:
Freeware: SpamBayes, K9, Popfile, ...
Commercial:
Mailshell Anti-Spam Desktop from http://www.mailshell.com/
ChoiceMail from http://www.digiportal.com/

[LGM]

I've had the baysesian filters running for a long time. I've had to intervene repeatedly for email with similar messages. For example, an email with the subject line of "Remove harmful errors that slow down your PC" or very very close variants of it has been getting through for weeks. A decent junk filter would have fixed this by the 2nd or 3rd instance. But emails like this are getting through 20 or 30 or more times.

Something is definitely wrong. I"ve had another junk filter program before and it worked much better than this. You could actually notice it learning something.

The way my filter is set up now it is very DUMB. So I figured I didn't have correct settings and was hoping someone could suggest some settings and/or tweaks.

Anyone?

For now I have taken it off default settings and chose High Sensitivity (it was on Medium before) and selected "Strict Bayesian" under the Bayesian tab. I'll see if that makes a difference.

But if anyone has any suggestions in the meantime, I'd like to know. If this doesn't work I'm considering trying a different mail program.


I'm using PocoMail 4.5.0.3910

[Tomas]

I expect you already have at least 1000 words trained for both junk and good mail (which can be found in Status tab of Junk Mail Filters), thus here are some more data about setting tweaking the settings:

Junk threshold in Bayesian tab - each mail is classified on scale from 0 to 0.99. You will set junk threshold to 0.7 if you wish all emails classified as 0.7 and more, to be considered as junk and moved to junk folder.

Good mail bias - the higher you set email bias, the less frequently the word has to occur to be considered in calculations. Some users reported that increasing the value led to better results.

Also in Bayesian tab, you can try to use strict bayesian filtering (click on the Strict Bayesian button).

So far, there doesn't seem to be any consensus about what works best.

If this doesn't work I'm considering trying a different mail program.

While I am biased, I'd dare to suggest to consider separating the decision on the email program and antispam program, as they are usually two different pieces of software.

I'm not aware of any other developers of an email software that at the same time offer their own antispam solution. Usually they offer some interface to third party antispam. Same as the antispam guys don't offer their own email programs. Both are experts in their own field. Poco Systems are experts in emailing, and while the antispam solution seems OK to me, it's by no means in the top antispam league.

Shall you decide not to use Pocomail's built-in antispam, it's true that other email software might offer more comfortable usage of third party antispam modules (through plugins etc.), but even with Pocomail it's not that much harder to setup a third party antispam.

Usually you just change the incoming server setting in Pocomail (to something like localhost:100 - just an example, exact details you can find in antispam documentation, or ask here on the forum), and setup the original incoming server in the antispam program.

[LGM]

Tomas, thanks for you helpful reply.

Junk threshold in Bayesian tab - each mail is classified on scale from 0 to 0.99. You will set junk threshold to 0.7 if you wish all emails classified as 0.7 and more, to be considered as junk and moved to junk folder.

I've had it on default settings which sets the junk threshold to 0.9. So I actually had it higher than 0.7.

Good mail bias - the higher you set email bias, the less frequently the word has to occur to be considered in calculations. Some users reported that increasing the value led to better results.

OK, I will try this to see if this makes a difference. I have now set it to 3.5.

Also in Bayesian tab, you can try to use strict bayesian filtering (click on the Strict Bayesian button).

I reset it to strict Bayesian yesterday early morning. It did not seem to make a difference on certain emails I continued to get throughout the day and last night.

I'm wondering if selecting "Classify as Junk and Move" is worthless. Does it simply filter the message content and not the subject and sender fields? If so, that may explain why the spam filtering was failing so miserably for me. Most of my junk mail these days is simply a gif or jpeg attachment which contains the message. So there is no text to analyze. If so, every time I selected "Classify as Junk and Move" it was not analyzing anything of significance. I would also choose to "Ban Sender's Domain" even though I know that has limited use since spammers change their address so often. But that is a necessary step nonetheless.

What I wasn't doing was selecting "Ban Sender" and "Ban Subject" for each offending piece of mail. Now I'm beginning to think I need to do that for every email which would be a pain. So my question is -- is there a way to automate the filter where in one click I can do all these things at once:

--Ban Subject
--Ban Sender
--Ban Sender's Domain
--Classify as Junk and Move

To do that for each piece of junk mail that gets through is tedious. But if I can combine those four commands into one single step ..,. that's doable. What about a script? Aren't there a few ways to automate things in PocoMail? Can you help me here?

While I am biased, I'd dare to suggest to consider separating the decision on the email program and antispam program, as they are usually two different pieces of software.

I'm not aware of any other developers of an email software that at the same time offer their own antispam solution. Usually they offer some interface to third party antispam.

I read somewhere that Outlook's filtering is now very good. But then their interface is a dog. Which is why I've always preferred a client like PocoMail .... and still do, except that the failure to catch repeated spam is driving me crazy,

Shall you decide not to use Pocomail's built-in antispam, it's true that other email software might offer more comfortable usage of third party antispam modules (through plugins etc.), but even with Pocomail it's not that much harder to setup a third party antispam.

If these steps don't work, then I will first try a third party anti-spam program that will intervene between the server and PocoMail before I jump ship. Any suggestions?

But first I would really like to see if I can get PocoMail's filtering to work better. I'm really interested to see if there is a way to combine the 4 steps mentioned above into one,

[LGM]

This morning I selected "Ban Sender" for a couple of spam emails with the "From" field having the email address of info@pupilwrite.com. The little box popped up confirming that it was banning that email address. Now, a couple of hours later, I am getting mail from info@pupilwrite.com in my regular inbox. What's going on? Is it my settings? What setting do I have to tweak to make the program do what it says it will do?

Also, is there a way to tell the program to ban the name that appears in the "From" field in addition to the underlying email address without having to manually enter the name in the blacklist?

[chrisretusn]

This morning I selected "Ban Sender" for a couple of spam emails with the "From" field having the email address of info@pupilwrite.com. The little box popped up confirming that it was banning that email address. Now, a couple of hours later, I am getting mail from info@pupilwrite.com in my regular inbox. What's going on? Is it my settings? What setting do I have to tweak to make the program do what it says it will do?

First make sure that "Enable automatic Junk Mail filtering" is checked in the Junk Mail Filters to turn on filtering.

Second make sure you have "Run standard non-Bayesian filters" checked on the General Setting tab of Junk Mail Filters or the Word List filters will not be enabled.

With the Word List filters there are no settings to tweak, any message that triggers a hit from the Banned Senders get a +100 score. That should be more than enough to send the message to Junk Mail.

Also, is there a way to tell the program to ban the name that appears in the "From" field in addition to the underlying email address without having to manually enter the name in the blacklist?

Yes. In the Mail directory look for "JunkSender.txt", edit that file to add the name that appears in the From field.

Using this spam address as an example (Kathleen Sewell <erotica1@pmcgroup.com>) entries in JunkSender.txt that will hit on it are:

Kathleen Sewell
Kathleen
Sewell
erotica1@pmcgroup.com
erotica1
pmcgroup.com

If any one is interested. A hit from a key word in Banned Senders and Banned Subjects is a score +100. Allowed Senders and Allowed Receivers is a score -100. The Message Body filter is unique in that you can assign a score to each key word or sentence e.g., "online degree",+20; you could also assign a score of +200 or -200.

Positive is a junk score, Negative is a good score.

[LGM]

First make sure that "Enable automatic Junk Mail filtering" is checked in the Junk Mail Filters to turn on filtering.

Second make sure you have "Run standard non-Bayesian filters" checked on the General Setting tab of Junk Mail Filters or the Word List filters will not be enabled.

With the Word List filters there are no settings to tweak, any message that triggers a hit from the Banned Senders get a +100 score. That should be more than enough to send the message to Junk Mail.

I have had both options checked. But that didn't make a difference. Mail from the same email address gets through AFTER I elect to Ban Sender. That's why I thought there must be some unobvious tweak/option/setting that I'm missing. The filtering in PocoMail is simply not working as it should. It is frankly abysmal.

Also, is there a way to tell the program to ban the name that appears in the "From" field in addition to the underlying email address without having to manually enter in the name in the blacklist?

Yes. In the Mail directory look for "JunkSender.txt", edit that file to add the name that appears in the From field.

Using this spam address as an example (Kathleen Sewell <erotica1@pmcgroup.com>) entries in JunkSender.txt that will hit on it are:

Kathleen Sewell
Kathleen
Sewell
erotica1@pmcgroup.com
erotica1
pmcgroup.com

Are you saying that for every junk mail that gets through I should go open that file and manually enter in the name of the sender? As I originally said, I was looking for a way for the program to automatically enter this in for me "without having to manually enter in the name" myself each and every time. To do that would be unreasonable given the amount of junk mail I am receiving these days. With all due respect, you must be kidding!

I am at wit's end. PocoMail cannot even ban a simple Sender's name without manual intervention every time.

I am now in the process of evaluating third party anti-spam solutions to work with PocoMail but am not having much luck.

So far I really lke Cactus Spam Filter but for some reason, when it is enabled my main POP3 account hangs. I have tried SpamEater Pro but the program itself freezes and hangs while trying to process the mail on the server. I tried Mailwasher Pro, but every time you go to check mail, it processes every single item on the server -- even mail that's been processed before. I can't find a setting in the program to tell it to just process new mail.

I will continue to evaluate third-party solutions. Does anyone have any suggestions?

I love the concept of Cactus and it would be almost ideal for me if only it would work with my main POP3 account without hanging.

[chrisretusn]

I have had both options checked. But that didn't make a difference. Mail from the same email address gets through AFTER I elect to Ban Sender. That's why I thought there must be some unobvious tweak/option/setting that I'm missing. The filtering in PocoMail is simply not working as it should. It is frankly abysmal.

Are there other filters that may hit on these messages. Junk Mail filtering takes place last in the scheme of things so if any regular filters act on these message the Junk Mail filters will not be run on the messages.

Another thing is you might have conflicting entries in the Allowed Senders or Allowed Receivers word list.

One thing I find useful it to the select problematic message(s) and use Apply and Test. The resulting pop up gives you the total junk score (XPS) and also the details that resulted in getting that score.

For example some spams put your address in the From: header, if you add your self to your address book, that message will get a -20 score to figure in with the total score.

Are you saying that for every junk mail that gets through I should go open that file and manually enter in the name of the sender? As I originally said, I was looking for a way for the program to automatically enter this in for me "without having to manually enter in the name" myself each and every time. To do that would be unreasonable given the amount of junk mail I am receiving these days. With all due respect, you must be kidding!

As I look at my spam collection, I see as varied number of spam names as I do spam addresses. Selecting Ban Sender essentially accomplishes the same thing. I see very little gain in adding the name too. If that is what you want then the only what to add them is manually. Sorry about that. You could also use Ban Sender's Domain, that would take care of the domain, those those are also many variations.

I just retrained my Bayesian filters using 882 junk messages and 199 good messages. I then applied the Junk Mail filters to the messages. Out of 882 messages 22 were not filtered to Junk Mail.

[LGM]

Are there other filters that may hit on these messages. Junk Mail filtering takes place last in the scheme of things so if any regular filters act on these message the Junk Mail filters will not be run on the messages.

There had been another filter, but roughly 25% of junk mail was properly being flagged and routed to the Junk folder so I doubt that was the case.

Another thing is you might have conflicting entries in the Allowed Senders or Allowed Receivers word list.

One thing I find useful it to the select problematic message(s) and use Apply and Test. The resulting pop up gives you the total junk score (XPS) and also the details that resulted in getting that score.

For example some spams put your address in the From: header, if you add your self to your address book, that message will get a -20 score to figure in with the total score.

Interesting. I never used the Apply and Test function before but will utilize it in the future for especially persistent spam. I tried it on one piece of spam that keeps getting through no matter how many times I classify it as junk in PocoMail. Here are the results:



I don't understand the language. Can you interpret this for me?

BTW, the "SPAM" designation in the subject field was put there by Fastmail on the server side. They do their own filtering but I have the controls on the server side set to just medium. If I set it higher, too much of my non-spam mail gets caught in their spam filter.

I do know for a fact that some of the mail I have been having problems with did not have my email address in the "From" field. When you choose an action, a little window pops up to tell you what's been banned. That's how I know that some email addresses supposedly banned by PocoMail were continuing to get through.

As I look at my spam collection, I see as varied number of spam names as I do spam addresses. Selecting Ban Sender essentially accomplishes the same thing. I see very little gain in adding the name too.

I have been getting many emails lately that have the exact same name but with different email addresses for each one. For example, I'll get 8 emails a day from "Slim Shake" or "Winter BooK" or "New Car" but there will be a different email address for each one. When this happens, the built in junk filters are worthless since none of them will automatically ban the name. And it's simply not feasible for anyone to have to manually type in these names in a text document every day. If PocoMail's junk filter when applied would simply automatically ban the name, that might improve it's success rate greatly. Why it won't do that is beyond me.

If that is what you want then the only what to add them is manually. Sorry about that. You could also use Ban Sender's Domain, that would take care of the domain, those those are also many variations.

I have been using Ban Sender's Domain quite often. But as I explained in my original post, it did almost nothing as the spammers are changing the originating e-mail address with almost every piece of mail. And, as I subsequently mentioned, even when I did ban a specific email address, it would still mysteriously get through afterwards.

I just retrained my Bayesian filters using 882 junk messages and 199 good messages. I then applied the Junk Mail filters to the messages. Out of 882 messages 22 were not filtered to Junk Mail.

If I can get Pocomail's filter to start working better I'll be happy. That's why I'm here now. It seems there is a new version of PocoMail in the works and now in beta testing? I do hope the developers are addressing the issue of junk mail filtering.

The good news is that Cactus Spam Filter is actually doing a pretty good job of filtering spam for me in the very short time I've been using it (just two days). Within 24 hours it was doing what PocoMail has been failing to do over weeks and even months. Cactus within 24 hours was catching over 90% of the spam. The problem is sometimes I have to turn it off as certain pieces of mail gets "stuck" while processing and PocoMail times out. I then have to shutdown Cactus and then PocoMail will finish processing new mail on the server normally.

The ideal of course is to get PocoMail's filtering to work better.

[chrisretusn]

Interesting. I never used the Apply and Test function before but will utilize it in the future for especially persistent spam. I tried it on one piece of spam that keeps getting through no matter how many times I classify it as junk in PocoMail. Here are the results:



I don't understand the language. Can you interpret this for me?

Hehe, not sure I do either, but here follows my interpretation:

The [XPS1] is the X-Poco-Score(?) The sensitivity value (High =10, Medium =12, Low = 15 or Custom) looks to be what determines if message is flagged as junk. Anything lower is good, anything equal to or over is junk.

That first line is the Bayesian filter it only appears if the Bayesian filter scores it. I am guessing you are using Strict Bayesian since the Bayesian score is -100 (all messages classified as Good are given this score). If your Good Score is 0, the Bayesian filter results will never show up in those Information pop-ups. I prefer not to use Strict Bayesian because it essentially overrides PocoMail's non-Bayesian filters. If not using strict Bayesian you can use the scores to better interact with the non-Bayesian filters.

The P=0, I guess is the Bayesian probability or percent score. I have only seen this as P=100 (junk) or P=0 (good). The T=90 is your "Junk threshold" (.90) value. The BIAS=+35 is your "Good mail bias" (3.5) value.

The rest are PocoMail's non-Bayesian filter scores. The +3 X-MAILER score I have determined to be given if there is no X-Mailer: header; those address scores the -4 TO looks to be if your address is in the TO header. The +2 FROM is because it's not in your address book.

That +100 SUBJECT means that a subject word/phrase is listed in Banned Subjects. Guessing that you have "spam" as a key word in you Banned Subjects word list so your messages fagged by Fastmail also flagged by PocoMail.

There are other PocoMail filter results that show up, some are self explaining some I have not figured out yet.

I have a message scored similar to yours . I changed my Bayesian setting to match yours and added a key word to Banned Subjects. One way to get that message flagged as junk is to not use Strict Bayesian. If you use PocoMail's default Junk Score of 20 and Good Score of 0 the message will get a XPS101 score. Note the missing Bayesian score, this is because the Good Score is set to 0.


Just dropping the Good Score to 90 would give it a score of XPS11 junk at high sensitivity.

I have been getting many emails lately that have the exact same name but with different email addresses for each one. For example, I'll get 8 emails a day from "Slim Shake" or "Winter BooK" or "New Car" but there will be a different email address for each one. When this happens, the built in junk filters are worthless since none of them will automatically ban the name. And it's simply not feasible for anyone to have to manually type in these names in a text document every day. If PocoMail's junk filter when applied would simply automatically ban the name, that might improve it's success rate greatly. Why it won't do that is beyond me.

Ok, I see what you mean. Maybe the next version will have some improvements in the area. I agree being able to add the senders name to the Banned Senders list via context menu would be a plus. I get similar messages but my Bayesian filter catches them.

If I can get Pocomail's filter to start working better I'll be happy. That's why I'm here now. It seems there is a new version of PocoMail in the works and now in beta testing? I do hope the developers are addressing the issue of junk mail filtering.

Agree I would like to see better capabilities and flexibility in PocoMail's non-Bayesian filtering.

[speerga]

Not meaning to sound negative here. But I and others have struggled literally for YEARS to get PocoMail's Junk Filtering process to work.

Some report glowing success.

I have NEVER been able to get it to work even moderately well. And, yes, I've tried virtually every combination of filters and Bayesian stuff, etc., that's been suggested over the years for this version of PocoMail and earlier versions.

I really miss PocoMail, and keep checking here hoping there'll be some news from developers about a sort of Homer Simpson "DOH!" moment when they get their junk filtering to work well and work easily.

About 8 months ago I actually quit using Poco and went over to another email client (won't name it here, but it involves birds). I really, really like many of the other features of Pocomail better, but got too weary of trying to get the simplest consistent results from the junk mail filtering.

*big sigh*

[LGM]

I would just like to update and say unfortunately, I have given up on Poco mail because of its inability to filter junk properly OR work with Cactus. If it could have done either one, I would have stuck with it.

Instead, like speerga, I went with Thunderbird. I had no choice as the time management trying to deal with junk using Poco was just too wasteful. Cactus works flawlessly with Thunderbird too.

I have never seen a junk filter as smart or easy to use as Cactus. In less than a week of training, it was catching virtually 100% of junk mail with no false positives. It is the best junk mail filter I've ever used of any kind, paid or non-paid. The fact that it is free just makes it even more amazing.

I hope that Poco fixes its junk filtering and then I might revisit it one day.

[chrisretusn]

I get a lot of messages that would be considered junk (newsletters, Yahoo! Group messages, plus a few "forwarding" friends etc. , so I have filters that take care of those before they hit the Junk Mail filters.

It's been working well for me; 88% of my incoming messages have been filtered as junk, with a 97.95% accuracy and 0.28% false positives.

[Tomas]

OR work with Cactus.

Perhaps it might help in Options > Advanced, to increase the timeout from 60 seconds up to 120 and/or decrease the maximum number of accounts to check at the same time. Not sure which other setting could affect it.

Unfortunately Cactus will "break into" the connection that Pocomail makes with the server, instead of technically more cleaner solution like working as a proxy server between email client and the server, I guess that's where some friction could occur.

Having no false positives is a very good reason to make the decision on the "what works with Cactus" basis, that's for sure - I have ended with SpamBayes for exactly same reason (no false positives) and nothing would made me switch as that's where the time savings occur. Fortunately SpamBayes works flawlessly for me with Pocomail.