[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/session.php on line 2208: Array to string conversion
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 4688: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3823)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 4690: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3823)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 4691: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3823)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 4692: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3823)
Poco Forums • View topic - Another wild idea about cause of "lost messages"

Another wild idea about cause of "lost messages"

Help and advice on using PocoMail

Moderators: Eric, Tomas, robin

Another wild idea about cause of "lost messages"

Postby frazmi » Tue Aug 10, 2004 6:58 pm

Here's another wild idea that perhaps is the result of eating too many jalapeño peppers...

Given that Poco keeps the mailbox and index files in memory during operation, what happens if the user shuts down the system without explicitly shutting Poco down first?

I realize that the shutdown sequence should send a message to Poco saying "imminent shutdown -- clean up now!" However, what if something goes wrong here?

(1) Perhaps Poco does not receive, or ignores the notification.
(2) Perhaps Poco does not respond to the OS properly. For example, it might send back an "OK" response before all the in-memory files have been written fully.
(3) Perhaps Poco saves file data after Windows thinks it has flushed all in-memory data out to disk.
(4) Perhaps Poco hands off the saving of file data to Windows, which in turn drops the ball somehow.

Admittedly, this is all speculation, since I don't know enough about the details of the shutdown process. But I thought the ideas were interesting enough to waste some pixels.
frazmi
Poco Enthusiast
 
Posts: 248
Joined: Tue Jul 27, 2004 1:27 am
Location: South Korea

Postby robin » Tue Aug 10, 2004 10:13 pm

It's a fair thought frazmi, and you seem to be becoming the de facto expert on this. I guess how likely this scenario is depends upon whether, as well as keeping it in memory, Poco also writes it to disk when a change is made - only PSI could tell us that for certain; however, some investigation...

I moved a message into a mailbox while monitoring the file in an explorer window (the benefits of two monitors). After I dropped the message, the .mbx file on disc was immediately updated but the .idx file was not. So I killed Poco and the .idx file stayed un-updated. This should mean that when I restart it then the message should not show in the index pane. Let's have a look...

Yup, sure enough, not there in the index. But Compress Mailbox finds the message and creates the new .idx file with it in. It also, incidentally, copies the old .idx file to .~idx and the old .mbx to .~idx. This means that the backup is out if synch - the backup index does not reflect what is in the backup mailbox (because they were out of synch when I killed Poco). Now, does this tell us anything? I think a qualified "maybe" has to be the answer.

The frustrating thing (for me - because I can't help sort it out) is that I have never experienced loss of messages that Compress Mailbox has been unable to recover, and many other people don't either.

Here's another though though; supposing a disc utility of some kind was checking the .mbx file and had it open at the time that Poco wanted to write to it. If Poco were to try and write to it, but couldn't, what would it do? Wait and try again? Remember to try later and forget? Or simply not do the write? PSI need to answer that one. I've tried to hold the file open by opening it n notepad but that doesn't work, I'll have to knock up a quick utility to open a file and hold it open exclusively to see what happens then. Watch this space...

Do you have any disc utilities running (real-time monitoring with n AV for example, or an automatic backup facility?), or anything that might want to access a disc file? Can you get it to ignore the .idx, .mbx, .~... files?
robin
 

Postby frazmi » Wed Aug 11, 2004 1:23 am

The only thing I have running in the background that might access a file is message indexing service, which should be read-only. I generally run disk utilities (and even virus checkers) in a standalone mode.

I've been able to duplicate your scenario, by the way.
frazmi
Poco Enthusiast
 
Posts: 248
Joined: Tue Jul 27, 2004 1:27 am
Location: South Korea

Postby robin » Wed Aug 11, 2004 1:33 am

Well, I knocked up a "quick-and-dirty" program in Delphi 7 that opens a file for exclusive access - i.e. it stops any other application being able to open the file. Then I tried Poco out with one of the .mbx files that I had opened and "locked". Results...

When I copied a message to the "locked" mailbox, the mailbox didn't get updated.
When I then closed Poco it didn't tell me, or complain, that it was not able to write the mailbox out.
When I reopened Poco with the mailbox unlocked, the messages that I had dropped there before were missing.
When I tried to view the contents of the locked mailbox, all I got was the headers with no content.
When I unlocked the mailbox and then returned to the mailbox in Poco I could see all the content again.

So if something else has a mailbox open in exclusive mode when Poco wants to write to it, Poco can't. frazmi - try disabling the indexing service for a while huh?

The display issue is exactly the problem that I have seen sometimes in the past but never been able to pin down, that I can only see the headers and the detail of the message disappears.

I'll pm you the address of the exe file that I have knocked up and you can try it for yourself if you like - let me know.

Be interesting to see what Jim has to say on this one.
robin
 

Postby Jim » Wed Aug 11, 2004 2:01 am

Good one guys. This is being looked into :) . I have a question though, in the absence of reverse engineering or staged situations, what are some of the real scenarios in which another application will need to write into and lock PocoMail's mail directory? One thing that comes to mind is a manual run of an AV on a mailbox while PocoMail is writing to it, but I doubt that scenario fits Robin's simulation.
Last edited by Jim on Wed Aug 11, 2004 5:49 am, edited 1 time in total.
Jim
 

Postby robin » Wed Aug 11, 2004 2:11 am

That is a fair question Jim and I don't have a real answer - merely speculation.

what are some of the real scenarios in which another application will need to write into and lock PocoMail's mail directory?
However anything that locks the file in these two modes (and it might only be read, does not have to be write - the utility I wrote only opens the file for read)...
Code: Select all
fmShareExclusive - Read and write access is denied.
fmShareDenyWrite - Write access is denied.

at the exact moment that Poco wants access could probably set up the scenario, and that would depend on the way each application is written.

Possibles? Indexing service, anti-virus service, automatic back-up service...just to start with.

Maybe under certain circumstances Poco does not release the lock on a file, maybe if someone is moving quickly between mailboxes. The situation where only the header is visible not the message I have seen before - changing to another mailbox displays the new mailbox but going back to the "locked" one does not redisplay it until Poco is restarted.

As you will have seen, I've suggested that frazmi turns off his message indexing service and see what happens.
robin
 

Postby waratel » Wed Aug 11, 2004 2:32 am

The situation contrived and described by Robin, is effectively the same as the NAV lockout which I described in the earlier thread.

In my case NAV blocked access to the .mbx file because it thought there was a virus in it. Same as Robin's program opening file with exclusive access. Same result, data doesn't get written, and Poco appears to be blissfully unaware.

Note that in the case of NAV blocking access as above, this block is enduring, not fleeting.

Bill
waratel
Drop-in Visitor
 
Posts: 11
Joined: Wed Jul 28, 2004 4:11 am
Location: Melbourne, Australia

Postby Pete » Wed Aug 11, 2004 5:08 am

Regarding the file locking, I have seen similar behavior with the DBGood.ini and DBSpam.ini files. PocoMail keeps the content of these files in memory. When I add or remove good words or spam words to these lists, PocoMail does not write the changes to disk until either I close PocoMail or a certain amount of (idle?) time has passed. The problem that I encountered was that when I was looking at either of these files with a read-only file-viewer program* and closed PocoMail, PocoMail did not write the changes to the disk files and it did not show me an error message.

Regarding shutting down Windows without first manually closing PocoMail, Slaven said (a while ago) that PocoMail didn't handle the shutdown event (actually he might have been referring to the logout event). I don't know if this was ever fixed. I've never seen anything about it in the release notes for PocoMail betas, so it might not have been fixed yet. Note that this problem would only occur in the situation where PocoMail hasn't had a chance to save cached mailbox information to disk before a logout and/or shutdown.

*The file-viewer program that I use is read-only and it does not use exclusive locks. When I'm looking at any file with it, I can open a text editor and save changes to the disk file with no problem. In this case, when I return to the file-viewer program, it prompts me that the file has changed and then it re-loads the file. FYI, it's called V (from http://fileviewer.com )
Pete
 

Postby robin » Wed Aug 11, 2004 6:01 am

Pete wrote:Note that this problem would only occur in the situation where PocoMail hasn't had a chance to save cached mailbox information to disk before a logout and/or shutdown.
My experience from my experimentation earlier is that Poco writes the new information out to the end of the .mbx file as soon as it receives it (when I did the copy to mailbox, the updated time on the file (as shown by Explorer) changed within a second). This suggests that while Poco may keep a cached copy in memory for its use, it also makes sure that the disc copy is kept up to date.
robin
 

Postby Pete » Wed Aug 11, 2004 12:10 pm

Good point, Robin. I should have been more specific. I think that the issue is with the index files (.idx). If they are not updated before the logout/restart/shutdown, then that would be one cause of messages that do not appear in the GUI after re-starting PocoMail until after the mailbox is compressed.
Pete
 

Postby Sandy » Wed Aug 11, 2004 12:46 pm

In my case NAV blocked access to the .mbx file because it thought there was a virus in it. Same as Robin's program opening file with exclusive access.


Are you sure you are not making a logical jump here? How do we know that NAV uses the same windows locking mechanism that robin's program is using? Furthermore what is the evidence that NAV blocks access to directories at all? How would you determine that?

I've used NAV for years and have found no conflict btwn Poco and NAV. But then I only get one virus/etc hit per month or so. I have NAV set to "repair then quarentine if unsuccessful" so I end up with a NAV.txt type files substituted for the actual attachment when I do get a hit.
Sandy
 

Postby frazmi » Wed Aug 11, 2004 1:17 pm

Pete wrote
Note that this problem would only occur in the situation where PocoMail hasn't had a chance to save cached mailbox information to disk before a logout and/or shutdown.
If Poco is running in the tray, and is checking mail frequently, then it seems highly likely that a Windows shutdown could occur before Poco updates the index file. The question then becomes, "so what?" When the index file is corrupt, for any reason, is it always the case that compressing (rebuilding) gives a valid index file?

Frankly, I missed Slaven's comment about handling Windows shutdown messages. I'd sure appreciate a confirmation about this, as it would have a radical impact on how I use Poco. At the moment, I am assuming that Poco is "well behaved" and handles shutdown properly. If it's confirmed that Poco does not handle shutdown messages, then I think there should be a bug report in Bug Traction, and I'll try to remember to stop Poco before shutting down.

Regarding Sandy's point about NAV, I;ve also used NAV for years (about a decade) and finally gave up on it after I confirmed that (1) NAV2004 was missing incoming virus-infected messages, and (2) was interacting poorly with Poco (in terms of causing mbx files to become out of synch with the index files. Granted, the second problem might not be a NAV bug per se, since there's no way for NAV to know that the Poco index file pointers were being invalidated. However, the first problem is unforgivable, and Symantec tech support, after over 3 months of dialog, was unable to give any kind of satisfaction. Note: my virus activity level was about 50 virus messages per day.
frazmi
Poco Enthusiast
 
Posts: 248
Joined: Tue Jul 27, 2004 1:27 am
Location: South Korea

Postby tribble » Wed Aug 11, 2004 1:21 pm

Very interesting thread. 99.99% of the time, I shutdown all open applications prior to shtting down Windows, this of course includes Poco. I use NAV, Poco and Barca and can find no indication/record of EVER losing a message. I check email on 3 accounts every 5 minutes, 1 account every hour and 1 account every 24 hours.

Now duplicate messages or lost header information is another story :)

It appears the only way to isolate this particular problem is with empirical data. Perhaps a listing from any user that suffers from this problem describing some level of detail about their system config.
tribble
Poco Enthusiast
 
Posts: 430
Joined: Wed Jul 28, 2004 8:55 am

Postby Jim » Wed Aug 11, 2004 2:15 pm

tribble wrote:It appears the only way to isolate this particular problem is with empirical data. Perhaps a listing from any user that suffers from this problem describing some level of detail about their system config.


That would be nice.
Jim
 

Postby Pete » Wed Aug 11, 2004 2:42 pm

tribble wrote:99.99% of the time, I shutdown all open applications prior to shtting down Windows, this of course includes Poco.

Do you mean that (1) you manually shutdown all open apps first, and then you shutdown Windows second? Or do you mean that (2) you just shutdown Windows and let Windows shutdown all of your open applications?

The problem (if there is one) wouldn't happen with (1).


frazmi wrote:Frankly, I missed Slaven's comment about handling Windows shutdown messages. I'd sure appreciate a confirmation about this, as it would have a radical impact on how I use Poco.

The discussion was not in the public forums. Bruce (a PocoMail Expert at the time) reported it and Slaven responded. To be fair, it didn't sound like Slaven had spent any time to verify the problem, it was more of an "off-the-top-of-his-head" comment that suggested that maybe it was possible that PocoMail wasn't listening to that/those system events.

I'll go and gather some empirical data now...
Pete
 

Next

Return to PocoMail Help and How-To

Who is online

Users browsing this forum: No registered users and 4 guests

cron