IP address filtering

[FieldDir121]

For those of you interested in the scripts I have posted so far that have figured out how to use them, a false positive presents a problem, removing the false positive address from the file. My address list has close to 2500 addresses, 2400 of them added manually before I wrote the script. As such, there seem to be no false positives [yet]. Now that the process is automated I anticipate it might happen occasionally [to others <g>].

The script below will ease removal of the IP address of the current false positive message from SpamAddress.txt (or some other file). This script is intended to be run manually, such as from a button, not automatically, at least not without some modification.

The line numbers where the address is found can give some indication of when the most recent and oldest use of this address occurred. this isn't really necessary here, but I wanted to debug the technique for use elsewhere.

Code: Select all

{  RemoveIPAdrFromFile - Version 1.10
{  Author: Scott Taylor - January 24, 2006
{  Also see scripts DisplayMostRecentIPAddress, MoveIPAddressToStartOfFile
{  and AddIPAdrToFile (formerly AddIPAdrToAdrFile).
{
{  Purpose: Remove all occurrences of the most recent IP address in the current message from the file containing
{  banned IP addresses. Note: This program is not limited to only using this file, but it is the original
{  purpose of this script. This script is intended top be run manually such as from a button.
{
{  Method: The script gets the most recent IP address of the current message, opens the specified file and
{  removes all occurrences of that IP address from the file, overwriting the original file with the remaining text.

{ Get the IP address from the current message
ReadAllHeaders $headers %message     { put entire header of current message into variable "$headers"
Set $MostRecentIPAddr ""             { clear variable $MostRecentIPAddr
LocateLine #z "Received:" $headers   { finds the first occurrence of "Received:" in $headers
If #z < 0 Then NoIPAdr               { skip adding new address if not found
  GetLine $line #z $headers          { put line number in $line
  StringPos #z "[" $line             { find opening bracket
  If #z = 0 Then NoIPAdr             { skip adding new address if not found
    Dec #z                           { decrement to leave opening bracket
    ChopString $line 1 #z            { delete characters before opening bracket
    StringPos #z "]" $line           { find closing bracket
    If #z = 0 Then NoIPAdr           { skip adding new address if not found
      Inc #z                         { increment to leave closing bracket
      ChopString $line #z 9999       { delete everything after closing bracket
      Set $MostRecentIPAddr $line    { this is a bit redundant but I like the string name

{ Open the file [with the list of banned IP addresses]
Set $FileName "..\SpamAddress.txt"   { use Pocomail main directory
OpenBody $ExistingAdrs $FileName     { get existing file contents

{ initialize variables
Set #n 0                             { counter to track how mnay times address is found
Set #x 0                             { working copy of #n
Set $a "Newest occurrence found at line "
Set $b "Oldest occurrence found at line "
Set $c "Single occurrence found at line "

{ search for the IP address and delete all occurrences from the file
:SearchLoop
  LocateLine #z $MostRecentIPAddr $ExistingAdrs  { serach for IP address
  If #z < 0 Then LoopDone            { if not found quit
    GetLine $line #z $ExistingAdrs   { get line number, put  in #z
    DeleteLine $ExistingAdrs #z 1    { delete that line
    Inc #n                           { increment counter
    Set #x #z                        { save a copy of the line number
    If #n ! 1 Then SearchLoop
      AddStrings $a #x               { save first occurrence in string $a (newest)
      AddStrings $c #x               { save first occurrence in string $c (single)
  GoTo SearchLoop

:LoopDone
AddIntegers #x #n                    { add how many lines have been deleted (added v1.10)
Dec #x                               { do not include the current line (added v1.10)
AddStrings $b #x                     { save last occurrence in string $b (oldest)
{ Comment out the next line if you want to locate the addresses and not remove them (added v1.10)
SaveBody $ExistingAdrs $FileName     { Save remaining text, overwritting the old file.
GoTo finish

:NoIPAdr
Set $msg "Received IP Address of current message not found" { insert error message
MessageBox $msg                          { display error message
GoTo Exit

:finish
Set $TmpName $Filename               { make a working copy of the file name
ChopString $TmpName 1 3              { get rid of the leading "..\" to reduce confusion of message
Set $msg "Found "
AddStrings $msg #n " occurrences of IP address " $MostRecentIPAddr " in file " $TmpName
If #n ! 1 Then NotSingle             { see if #n is not equal to 1
  InsertLine $msg 99 $c              { add single line number to end of message
  GoToSkipLine

:NotSingle
If #n < 2 Then SkipLine
  InsertLine $msg 99 $a              { add newest line number to end of message
  InsertLine $msg 99 $b              { add oldest line number to end of message

:SkipLine
MessageBox $msg                      { display informative message
:Exit


[FieldDir121]

After I posted the code I realized that by commenting out a single line the same code could be used to locate the addresses without deleting them from the file on the disk. This might be useful to determine how long ago the address was used.

For instance, if you get 50 spams per day and the address is found in the first few hundred addresses the message isn't very old. If the address is found at line 1857 then it is quite a bit older.

The comment was added just below the label :LoopDone

Scott

[tribble]

Ok, I've been using the AddIPAddress for a few days and have built a list of over 100 unique IP's from spammers.

Now, how do I assign this as a filter option? What I've done is to create a filter that searches 'entire message' for %filename%:"drive:\path\filename". If found, move to junk.

Is that correct?

Thx
Gene

[FieldDir121]

Gene,

What you are doing should work. I used to use 'Entire Message' but decided 'Message Headers' will work just as well and may be a little faster for messages with large bodies.

Initially I deleting message caught by the filter. Occasionally I sent them to the junk mailbox. After a while I wondered how many messages were being caught by this filter so I created a special mailbox, SpamAdr. Now I can see for sure. The price is having to manually delete them.

At first the IP address filter was catching the majority of the incoming spam. I have another filter using the same technique that uses keywords and phrases, SpamEntire.txt. As that list has improved the number of messages that make it to the SpamAdr mailbox has decreased. Keyword messages are placed in the mailbox SpamEntire.

I was wondering if anyone was utilizing what I have done so far, wondering if I should continue to post the scripts I am working on. One person is enough since it doesn't take much extra effort to share.

My current effort is a script that will search the SpamAddress.txt file for the IP address of the current [incoming] message and insert the line number(s) where the address is found at the beginning of the message, InsertAdrLineNumberIntoBody.

The technique is only difficult because html and plain text must be handled differently. Determining which type is the tricky part. I just found a script by Pete that adds attachment names to the message body. He had one additional test criteria than I did. I added it this morning. After some additional testing I may be ready to post the result.

This new script will allow me to quickly determine how many of the messages in SpamEntire would have been caught by SpamAdr if they hadn't been caught by SpamEntire first. If the IP address is not found in the SpamAddress.txt nothing is added to the message body. This makes the yes/no determination very quick.

Another interim script will be to add the current IP address to the beginning of the SpamAddress.txt file and delete any other other occurrences of the address from the file. Otherwise, some addresses could (and do) appear many times. This will also put the most recent and most active addresses at the beginning of the file.

Since some addresses used by spammers will be spoofed, not their actual address but the address of an innocent, periodic manual truncation of the SpamAddress.txt file from the end will eventually remove addresses used only once.

Eventually I plan to integrate several of the scripts plus a few more I have in mind into one or two more comprehensive scripts, assuming I don't run out of spare time first.

For instance, once duplicate addresses are eliminated from the SpamAddress.txt file, there will be no need to display more than a single line number of where the address is found.

Scott

[tribble]

Scott,

Sometimes I read a message and say "D'uh! - Why didn't I think of that"

Moving the message to a NEW directory is a good approach, it will allow me to isolate what filters are working and which aren't.

I like the idea of a filter to clean up duplicate addresses, currently I do that manually every 2 weeks or so. Copy/paste to Excel, sort, delete dups, copy/paste back....

I'm curious though, what is the advantage if your script: InsertAdrLineNumberIntoBody? What would you do with it afterwards?

Thanks,

Gene

[FieldDir121]

Gene,

I wasn't able to get Excel to sort IP addresses. I didn't try very hard though. I originally wanted to sort numerically but eventually decided that sorting by most recent might be more useful. That is why I gave up on using Excel to eliminate duplicates.

>> InsertAdrLineNumberIntoBody?

What do I do with it?

Why put the [spam] messages in their own mailbox? Same idea. I want to see positive results. If catching a few more percent of the spam messages using an automated technique, spam addresses, is possible I would like to do it. I can add that technique to the system used by my wife and kids. If I have to maintain it manually it will not be very up to date. I delete the messages once I look to see if they have an IP address inserted or not. The line number where found indicates how long ago the address was last used. This gives me some idea of how long addresses not recently used should be kept.

I was even thinking along the lines of writing a script that identifies a specific attachment. When that attachment is found in an e-mail to one of the accounts on that system the script would replace the filters.ini, SpamAddress.txt, SpamEntire.txt files and any scripts. This is the height of lazyness since I walk by the room with that system many times each day.

Scott

[FieldDir121]

Here is a script I just finished. Finished means I have put it into general use on my system. Whether it will work for you, or do anything you find useful is for you to decide.

Code: Select all

{  InsertAdrLineNumberIntoBody - Version 1.01
{
{  Author: Scott Taylor - January 29, 2006
{  Also see scripts DisplayMostRecentIPAddress, MoveIPAddressToStartOfFile, RemoveIPAdrFromFile
{  and AddIPAdrToFile (formerly AddIPAdrToAdrFile).
{
{  Version 1.01: Reformatted comments so they wouldn't wrap when posted in the Pocomail forum.
{
{  Notice: I have done some testing on incoming mesages. This script appears to work properly for
{  the types of e-mails I receive. That doesn't mean it will work properly for all e-mails or any
{  of your e-mails. Problems may include, but are not limited to, complete loss of an incoming
{  e-mail and/or corruption of the e-mail contents.
{
{  Purpose: This script is for informational purposes. Inserting a marker, in this case the line
{  number in Spam Address.txt, at which the IP address was found, provides a visual indication of
{  messages that would have been caught by SpamAddress.txt and the associated filter if the message
{  had not been caught by another filter first. The actual line number provides some indication of
{  how recently the address had previously been used. Since these messages will be ultimately be
{  deleted, altering the contents is acceptable (in my case). Ultimately the goal is to increase
{  the chances of catching e-mails from repeat addresses that are not caught by other filters.
{
{  Method: This script gets the most recent IP address of the incoming e-mail message and a copy of
{  the body of the message to work with. SpamAddress.txt is searched to see if the IP address of
{  this message is present and at which line number(s). Each line number at which the address is
{  found is tghen inserted into the beginning of the incoming message body. The working copy of
{  the message body is then used to replace the original message body and processing continues as
{  if nothing has happened.
{
{  HTML versus non-HTML are both accomodated. If the incoming IP address is not found, or any
{  similar IP address errors occur, the script exits leaving the message untouched.
{
{  This script doesn't change the ultimate destination of the incoming message. Also, it won't work
{  on messages that are already in a mail box. It will work on messages that are being moved from
{  one mailbox to another, such as by a filter.

{ Get the IP address from the current message
ReadAllHeaders $headers %message     { put entire header of current message into variable "$headers"
Set $MostRecentIPAddr ""             { clear variable $MostRecentIPAddr
LocateLine #z "Received:" $headers   { finds the first occurrence of "Received:" in $headers
If #z < 0 Then Exit                  { skip adding new address if not found
  GetLine $line #z $headers          { put line number in $line
  StringPos #z "[" $line             { find opening bracket
  If #z = 0 Then Exit                { skip adding new address if not found
    Dec #z                           { decrement to leave opening bracket
    ChopString $line 1 #z            { delete characters before opening bracket
    StringPos #z "]" $line           { find closing bracket
    If #z = 0 Then Exit              { skip adding new address if not found
      Inc #z                         { increment to leave closing bracket
      ChopString $line #z 9999       { delete everything after closing bracket
      Set $MostRecentIPAddr $line    { this is a bit redundant but I like the string name

{ Get the message body from the current incoming message (raw allows html or text only format).
ReadRawBody $body %message           { get message body
Set $LowerBody $body                 { make an expendable copy
LowerCase $LowerBody                 { set to all lower case to make searching for strings easier

{ Determine if HTML or plain text. My original method evolved to something close to this. A few
{ messages were still getting through incorrectly identified. Attachments Lister.poc by Pete had
{ a similar approach but appeared more comprehensive so I copied that portion his code here.
Set &noHTML false                    { default to HTML style
Set $LineTerm "<br>"                 { set line terminator to HTML style
ReadHeader $contentType "Content-Type:" %message
Lowercase $contentType
If $contentType = "text/html" Then UseHTML

StringPos #z "</body>" $LowerBody
If #z > 0 Then UseHTML

StringPos #z "</html>" $LowerBody
If #z > 0 Then UseHTML
  Set &noHTML true
  Set $LineTerm ""                   { set line terminator to plain text style (nothing required)

:UseHTML
{ Open the file [with the list of banned IP addresses]
Set $FileName "..\SpamAddress.txt"   { use Pocomail main directory
OpenBody $ExistingAdrs $FileName     { get existing file contents

{ initialize variables
Set #n 0                             { counter to track how many times address is found

{ search for the IP address in the file
:SearchLoop
  LocateLine #z $MostRecentIPAddr $ExistingAdrs  { search for IP address
  If #z < 0 Then LoopDone            { if not found quit
    GetLine $line #z $ExistingAdrs   { get line number, put  in #z
    DeleteLine $ExistingAdrs #z 1    { delete that line
    Inc #n                           { increment counter
    AddIntegers #z #n                { add how many lines have been deleted
    Dec #z
    Set $msg "Line number "
    AddStrings $msg #z               { construct line to be inserted
    AddStrings $msg $LineTerm        { add line terminator
    Set #pos #n                      { make a working copy of find count
    Dec #pos                         { decrement since line numbers start at 0
    InsertLine $body #pos $msg
  GoTo SearchLoop

:LoopDone
If #n = 0 Then Exit                  { if IP address not found in file leave message alone and exit
  Set $msg "Found IP Address "       { prepare a summation description
  AddStrings $msg  #n " time(s)"
  AddStrings $msg $LineTerm          { add line terminator
  InsertLine $body 0 $msg            { insert the description on line 0 (the first line)
  Inc #n                             { adjust for inserted lines
  { insert a dividing line to indicate what has been added to the message by this script
  Set $msg "-----------------------"
  AddStrings $msg $LineTerm          { add line terminator
  InsertLine $body #n $msg           { add to body

If &noHTML Then SavePlain
  AssignStyledBody %message $body    { save in HTML format
  GoTo Exit

:SavePlain
AssignBody %message $body            { save in non-HTML format
:Exit


[FieldDir121]

Here is the script that will delete all occurrences of the IP address in a file and insert a single copy as the first line of the file.

It can be tested by using AddIPAdrToFile to put multiple copies into the file. LocateIPAdrInFile will then show how many copies exist and the line numbers of the first and last occurrence. After running this script LocateIPAdrInFile will indicate a single copy at line 1 (remember line 0 has been intentionally left blank).

Code: Select all

{  MoveIPAdrToStartOfFile - Version 1.00
{  Author: Scott Taylor - January 30, 2006
{
{  Also see scripts DisplayMostRecentIPAddress, InsertAdrLineNumberIntoBody, RemoveIPAdrFromFile and
{  AddIPAdrToFile (formerly AddIPAdrToAdrFile).
{
{  Purpose: This script has the same result as if RemoveIPAdrFromFile had been run followed by
{  AddIPAdrToFile, without the message box that requires human intervention. The result will be that
{  all copies of the IP address of this message will be removed from the file, SpamAddress.txt, with
{  a single copy of the address being added as the first line of the file. Eventually, addresses not
{  used for the longest time will be at the end of the file.
{
{  Method: The script gets the most recent IP address of the current message, opens the file
{  SpamAddress.txt and removes all occurrences of that IP address from the file. A new copy of the
{  address is inserted into the file at the first line.

{ Get the IP address from the current message
ReadAllHeaders $headers %message     { put entire header of current message into variable "$headers"
Set $MostRecentIPAddr ""             { clear variable $MostRecentIPAddr
LocateLine #z "Received:" $headers   { finds the first occurrence of "Received:" in $headers
If #z < 0 Then Exit                  { skip adding new address if not found
  GetLine $line #z $headers          { put line number in $line
  StringPos #z "[" $line             { find opening bracket
  If #z = 0 Then Exit                { skip adding new address if not found
    Dec #z                           { decrement to leave opening bracket
    ChopString $line 1 #z            { delete characters before opening bracket
    StringPos #z "]" $line           { find closing bracket
    If #z = 0 Then Exit              { skip adding new address if not found
      Inc #z                         { increment to leave closing bracket
      ChopString $line #z 9999       { delete everything after closing bracket
      Set $MostRecentIPAddr $line    { this is a bit redundant but I like the string name

{ Open the file [with the list of banned IP addresses]
Set $FileName "..\SpamAddress.txt"   { use Pocomail main directory
OpenBody $ExistingAdrs $FileName     { get existing file contents

{ search for the IP address and delete all occurrences from the file
:SearchLoop
  LocateLine #z $MostRecentIPAddr $ExistingAdrs  { serach for IP address
  If #z < 0 Then LoopDone            { if not found quit
    GetLine $line #z $ExistingAdrs   { get line number, put  in #z
    DeleteLine $ExistingAdrs #z 1    { delete that line
  GoTo SearchLoop

:LoopDone
InsertLine $ExistingAdrs 1 $MostRecentIPAddr  { insert new address at the beginning of the file
SaveBody $ExistingAdrs $FileName     { Save remaining text, overwritting the old file.

:Exit


[FieldDir121]

To anyone interested in these scripts,

Wanting to know how well my filters are doing, the next script I work on is likely to be one that allows statistics to be derived. I was thinking of updating various information in a text file each time a message comes in. The actual percentage math, a ratio of spam to non-spam, may not be included in the script, leaving that for a handy calculator. The reason for this is explained below.

The first script will count the total number of incoming messages, either on all accounts or on a specific account. A second script will count how many messages are stored in the SpamAdr mailbox (the mailbox names can be changed to the names you use). Either the script will have to be run after the message has been placed in the mailbox, assuming it isn't too difficult to determine the mailbox name within the script, or a slightly modified version of the script will need to be created for each mailbox, ie., SpamAdr, SpamEntire, Junk Mail and the main mailbox.

Here is what I am thinking of having so far:
nnn Total number of incoming messages
nnn Number of messages placed in SpamAdr
nnn Number of messages placed in SpamEntire
nnn Number of messages placed in Junk Mail
nnn Number of messages placed in UnKnown Sender

The way I have my system configured, only e-mails from people in my address book are guaranteed to be placed in the main mailbox. Those not placed in the main mailbox and not determined to be spam are placed in the Unknown Sender mailbox. I have to manually look through these for non-spam e-mails.

This arrangement makes determining the spam to valid e-mail ratio a bit more tricky, especially if messages are deleted directly out of Unknown Sender. Those messages will not be added to the other counts. This is one reason I may not add an automatic ratio calculation.

What isn't clear, yet, is which column, if any, to add the number of spam messages from the Unknown Sender mailbox. They didn't get caught by any filters and they weren't put in the main mailbox. Adding them to any of the above catagories after manually sorting the messages in Unknown Sender will distort the results. Perhaps using a third script, activated by a button, that increments another catagory:
nnn Manually designated spam from Unknown Sender

On my system a script counting the messages actually placed in my main mailbox would need to be called from several filters, so I may not include that, instead figuring anything not put somewhere else ended up there.

Comments and suggestions welcome as I haven't started this script yet, and it will more than likely evolve as time goes on.

Scott

[FieldDir121]

Update 31-Jan-06 regarding InsertAdrLineNumberIntoBody:

I have received around 100 spam e-mails since I put this script into use. Today was the first time an HTML e-mail appeard to have been significantly altered, other than adding the intended text. I think the problem has to do with the text being added above the HTML header.

If someone knows why things are not as expected I would appreciate comments. Otherwise, since the e-mail will be deleted anyway I may not spend much time revising the script.

Here is what the first few lines look like in a raw file:

Found IP Address 1 time(s)<br>
Line number 1<br>
-----------------------<br>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=koi8-r">
<META content="MSHTML 6.00.2900.2722" name=GENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=#ffffff>


Here is how the first two lines appear in the message:

Found IP
Address 1
time(s)
Line Number 1
----------------
-----


This is how they should look:

Found IP Address 1 time(s)
Line Number 1
-----------------------

Scott

[tribble]

Using the AddIPAddr script, I have built a listing of over 250 unique IP addresses. However, as suggested earlier any match against these IPs is routed to a specific folder. Although I have since added duplicate IPs, not a single message is being filtered based on IPs. In the filter list, I have this one set as the FIRST filter...

That got me to thinking, Scott, you add [] to the IP address yet that would NEVER be found in the headers. To test this, I deleted all []'s in the text file but for 48 hours, no hts, yet more duplicate addresses have been added.

What could I be doing wrong?

[tribble]

Um, nevermind. Operator error. I misspelled the path to the file

[FieldDir121]

Tribble,

If you turn on the header you will see that the IP address is [always] enclosed in square brackets.

As I mentioned earlier in this topic, or perhaps another, you need string terminators at both the start and finish of the address. Otherwise you could get unexpected results.

12.1.1.192 would also appear to be 112.1.1.192 and 212.1.1.192. A similar case exists for the end of the address string, 192.168.1.1 will be a hit for 192.168.1, 192.168.1.10 through 192.168.1.19 and 192.168.1.100 through 192.168.1.199.

I have a separate filter watching for groups of addresses:
[60.
[61.
etc.
I keep these in a separate file and use a separate filter. Individual IP addresses are considered definite spam. I am sure enough that at times those messages are automatically deleted by the filter rather than stored. The same goes for messages in SpamEntire. I look through the SpamBlock messages to be sure there are no legitimate messages.

As to not getting many hits by SpamAddress, my SpamEntire filter catches about 2/3 of the incoming spam. 1/3 ends up in the Unknown Sender mailbox. SpamAddress gets an e-mail every day or two. I am hoping my recent automation of adding IP addresses to SpamAddress.txt will improve this. I stopped manually adding addresses many weeks (months?) ago.

SpamAddress.txt has grown from 38kB to 48kB in the last couple of weeks. At a maximum of 19 charaters per line, "[nnn.nnn.nnn.nnn]<CR><LF>" that is over 500 new IP addresses. For the past couple of days duplicate addresses of hits are also being removed. Note: Existing duplicates remain in the file until a new message arrives using that IP address.

I attribute the lack of benefit form SpamAddress to the success of SpamEntire rather than to the failure of SpamAddress. My justification for this view is that early on SpamAddress caught more than SpamEntire, which comes first. I do this because keywords can catch spam from many addresses. IP addresses can only catch spam from a single address.

For any message that makes it to SpamAddress I manually extract keywords. That means that next time the message that made it to SpamAddress will have been caught by SpamEntire, assuming there were unique keywords.

About half of the messages that still make it to Unknown Sender have no unique keywords. To me unique means they are unlikely to ever occur in a legitimate message, hence my to date 0% false positives.

Scott

[tribble]

Ok, I have confirmed that my filter is not working and now need some help to isolate why.

Below is a snapshot of how I have the filter configured:


The file has entries that look like this:
[12.217.58.78]
[81.218.92.56]
[200.101.123.105]

About 350 of them...

I have confirmed that there are NEW duplicate entries in the list. Building the list works flawlessly, matching an entry against the list doesn't work at all.

Any ideas?

Thanks,
Gene

[FieldDir121]

tribble,

Please copy the entire path and file name from the "for" and post it. Also, a screen grab of the root directory of your F: drive. No insult intended, but let's eliminate the obvious first.

Here is one of mine:
%file%:"SpamAddress.txt"

Notice that I have "file" where you have "filename". Not sure if that matters or not. See the 7th message in this thread for a link to the FAQ about using a file within a filter.

Next a screen grab of your filter screen.
(Note from just after I posted this: A screen grab might show too much of your personal information for an open forum like this. What I was after was other filters that might affect where the message ends up.)

Question: Do you now have one mailbox called "Junk Mail" and another called "Junk Mail\Spam IP Addre..."? (... since I cannot see what comes next)

If so, just to eliminate a possibility, change the name of the "Junk Mail\Spam IP Addre..." mail box to something that does not contain the exact phrase "Junk Mail".

Scott