How to compose double-byte character messages

[frazmi]

One of the limitations of Poco (at least for now) is its inability to compose messages with double-byte characters, such as are found in many Asian languages.

The following script gets around that limitation to a large extent, by calling an external editor to do the double-byte work, and then using the mechanism Slaven wrote for the load HTML script.

Coupled with the script to view double-byte messages in an external editor, we have a workable (if somewhat clumsy) way to compose, edit, view etc. messages in almost any language.

Please, anybody feel free to extend this script by making it easier to select the right external editor, making provision for mulitple edit files, etc. This is just a starting point.

Please see this link for instructions on viewing messages with double-byte and other character-encodings.

Code: Select all

{ Simple script designed to allow the sending of arbitrary message content from Poco.
{ This content can include double-byte characters, up to the limit of your computer's
{ ability to compose such a message and save it in HTML format.
{ Script inspired by Load HTML as Body script by Slaven, with modifications.
{ Place in public domain by frazmi 17 August 2004
{ Please test on your system!
{ Modifications and enhancements welcome!

IntToChar $QUOTE 34
IntToChar $CR 13
IntToChar $LF 10
AddStrings $CRLF $CR $LF

External $SimpleInstructions "Include simple instructions in blank email?" "YES"
External $SimpleInstructions2 "Show simple instructions at start of script?" "YES"
External $SendImmediate "Send edited message immediately?" "YES"

IF $SimpleInstructions2 = "YES" THEN ShowScriptInstructions
      GOTO "Continue"

:ShowScriptInstructions
      Set $Msg "Start a new message before you run this script. Queue it for sending."
      AddStrings $Msg $CRLF "That will put it in the Out mailbox. Select the message."
      AddStrings $Msg $CRLF "Then run this script with the message selected."
      AddStrings $Msg $CRLF "Your word processor will open. Edit the message as you wish"
      AddStrings $Msg $CRLF "and then save it as HTML format. Make sure you don't change"
      AddStrings $Msg $CRLF "the file name. This simple script will break if you do."
      MessageBox $Msg

:Continue


Set $Content $mailpath
AddStrings $Content "DoubleByte.html"

FileExists &FileExists $Content
If &FileExists Then UsePrevious

IF $SimpleInstructions = "YES" THEN "SetInstructions"
      {Just insert a carriage return (so the file is not null)
      Set $Explain $CRLF
      GOTO "SaveBody"

:SetInstructions
      {Put simple instructions into body of message. User will delete.
      Set $Explain "Enter your formatted message here."
      AddStrings $Explain $CRLF "Don't change the filename or the script will break."
      AddStrings $Explain $CRLF "Make sure file is saved in RTF format."
      AddStrings $Explain $CRLF "Delete these lines (or else they will appear in the message!"
      AddStrings $Explain $CRLF "When you are finished editing, save the file and exit Word."

SaveBody $Explain $Content

:UsePrevious
Set $a $QUOTE
AddStrings $a $Content $QUOTE

ExecuteAndWait "C:\Program Files\Microsoft Office\Office10\WINWORD.EXE" $a

OpenBody $body $Content
AssignStyledBody %message $body

IF $SendImmediate = "YES" THEN "SendRightNow"
      Set $Msg "The edited message has been inserted in the Out box."
      AddStrings $Msg $CRLF "You need to send the message by executing the command"
      AddStrings $Msg $CRLF "to send queued messages. Note: If you open the message in"
      AddStrings $Msg $CRLF "in a compose window, the formatting will be lost."

      SaveMessage %message Out
      DeleteMessage %message
      GOTO "Finalize"

:SendRightNow
      SendMessage %message
      DeleteMessage %message
      MessageBox "Edited message sent."

:Finalize
Set $Msg "Do you want to delete the temporary edit file "
AddStrings $Msg $Content "?"
      InputBox $Answer $Msg "No|Yes"
      LowerCase $Answer
      If $Answer = "Yes" Then DeleteEditFile
      
      EXIT

:DeleteEditFile
      DeleteFile $Content

[DavidMD]

Greetings, Frazmi!

Thank you for your helpful script!

Although HTML e-mail messages can be aesthetically pleasing and even serve utilitarian purposes -- allowing one to format text (e.g., bold or italic) and embed images, as well as long URLs, simply by highlighting a single word and using the A HREF tag, for example -- I prefer to send (and to receive) plain-text messages whenever possible.

In addition, all of my friends in Odessa, Ukraine, have relatively slow PCs and modems -- including a business with which I deal -- and the Soviet era telephone lines are hard enough to use for conversations, let alone transferring data!

I started trying to modify your script, Frazmi, to send Russian/Cyrillic (KOI8-R) text using Emurasoft's EmEditor (developed in Japan, <http://www.emeditor.com/index.htm>) -- a Windows text editor that supports Unicode (which is just one of its many great features) -- but I realized that linking the script to the EmEditor Professional excecutable (instead of Microsoft Word) and changing the composing file's name to 'DoubleByte.txt' did not change the fact that PocoMail does not support Unicode. (I am displaying my ignorance with no inhibitions, as you can see!)

Obviously, my experiment was destined to fail, Frazmi, but your script certainly does a nice job of sending HTML e-mail messages!

The message sent from PocoMail to another account I have that uses Outlook XP, looks like Cyrillic gibberish in Outlook, which is to be expected. (Oddly, for outgoing Russian text, Outlook XP defaults to KOI8-R and not Windows Cyrillic! When I first made that discovery, I admit that I was shocked. Checking the encoding on almost any Russian Web site reveals that it is 'Cyrillic (Windows)'. Please note that I am NOT "bashing" Microsoft, but simply noting a contradiction between Outlook XP's default for Russian and the choice that most people make when creating Russian Web sites.

Regarding the encoding function of the script, Frazmi, I assume that the reason that 'Content-Type: text/html; charset="koi8-r"' appears in the header (i.e., the 'html') is because your script is based upon Slaven's mechanism to load HTML.

I realize that, until PocoMail offers at least limited Unicode support, I cannot send plain text messages containing Russian. I must use HTML or attach the Russian message as a text file in Unicode format.

I hope that I do not seem overly demanding on my specific desire to send Cyrillic as plain text (i.e., Unicode), but I have compared several messages with the exact same content in text format versus (MS)HTML, and Microsoft's added bloat to standard HTML dramatically increases the size of an e-mail file (and the download time for friends in Ukraine with old copper telephone lines, slow modems, and older Pentium processors).

I can quickly create a Russian text message in The Bat! by always using the KOI8-R character set as my default for e-mail (which causes no problems for English text, that I have ever seen) -- but PocoMail offers so many advantages over The Bat! (specifically in terms of documentation, support, and this very helpful forum -- as opposed to The Bat!'s e-mail-based discussion group, for example).

I will continue to hope that PocoMail will offer Unicode support in the near future, or -- in the meantime -- the ability to let the user paste text from a non-Latin character set into a message being composed, in addition to allowing direct input into the composition window of PocoMail. (When I try to past KOI8-R text into PocoMail, each Russian character becomes a question mark.) I also hope that I will be able to help in some way with the Unicode effort.

Anyway, Frazmi (and Slaven), I appreciate all of your hard work in this area! I know that PocoMail's quality will ensure its growing popularity!

Cordially,

David

[frazmi]

DavidMD wrote

Regarding the encoding function of the script, Frazmi, I assume that the reason that 'Content-Type: text/html; charset="koi8-r"' appears in the header (i.e., the 'html') is because your script is based upon Slaven's mechanism to load HTML.

Right.

One thing I have not tried, due to lack of proper keyboard and fonts installed, is to set Poco's default character set. Have you tried that yet? (Options | General Options | Encoding Options | Character-set)

[DavidMD]

Hello, 'framzi'!

One thing I have not tried, due to lack of proper keyboard and fonts installed, is to set Poco's default character set. Have you tried that yet? (Options | General Options | Encoding Options | Character-set)

Thank you for the suggestion, Framzi! I apologize for my delay in replying!

I believe that I tried the character-set option, but I will check again. If I recall correctly, I had a real problem using plain text and KOI8-R; HTML seems to be the default, but I will reinstall PocoMail and try your suggestion. I do not have PocoMail installed right now.

I do not have a proper keyboard for Russian, Framzi, although I do have the proper fonts. I am running Windows XP Professional and the Russian keyboard layout, as shown on the On-Screen Keyboard application, does not match any Russian/English keyboard made for Russian; I find this fact baffling, and I wonder who is buying these keyboards. Perhaps after some version of Windows (95 or 98), or with Windows XP, the Windows Russian key layout changed. (I would even be willing to purchase a Russian keyboard, and using a KVM switch with two keyboards (English and Russian), if I could find the proper Russian keyboard -- but I am getting off-topic, now, for which I apologize!

Because I keep my keyboard very clean, Russian stickers for the keys are not practical. They would not stay on the keys very long. (There is a smoker in the house, so I have to spend a lot of time keeping my PC, including the mouse and keyboard, clean.)

I have been looking for a Web site with information to change the keymapping for Russian text only, but I have had no success after searching the Web, on and off, for over two years! I seem to be facing several obstacles. How to do you handle the keyboard issue, Frazmi?

Whenever I type Russian, I display the On-Screen Keyboard application, so that I can see where the Russian letters are. The OSK application is buggy, however. It "broke" under XP's SP1 and I have not yet installed SP2, although I plan to do so after I reinstall XP Pro (which was corrupted, but made much worse by Microsoft Support's suggestions, although I realize that the representative was only doing his best to help me).

I am the only English speaker whom I know who wants an e-mail application that supports Unicode (especially, Russian/Cyrillic)! Such a feature is not high on the list of software developers, because of lack of demand, I assume.

I am impressed by every feature of PocoMail that I see, Frazmi, but I need support for Russian. Given the fact that an on-line manual for PocoMail is supposedly in the works, the problem with Russian is the only real problem that I have with adopting PocoMail (and jettisoning Outlook XP).

I know of one e-mail application, besides Outlook (ugh), that is Unicode compliant, but it lacks many features (including filtering, scripts, and "spam-fighting," and it simply pales in comparison to PocoMail!

Thank you, again, Frazmi!

Cordially,

David

[DavidMD]

Hello, again, Frazmi.

I just wanted to let you know that I followed your suggestion of selecting the character set, but without success.

If I select KOI8-R and switch from English to Russian input in Windows, my Russian text appears as gibberish in the PocoMail message that I am composing. (At least PocoMail recognizes the characters as individual letters; otherwise, each Russian letter would be replaced by a question mark.)

If I paste Russian text into a new PocoMail message, however, each Cyrillic character is replaced by a single question mark, so pasting Russian text into PocoMail 3.2 definitely does not work (or I am doing it wrong).

If there is a way to send Russian plain-text in PocoMail, even when specifying a Russian character set, I apparently am too inept to implement such functionality.

I seem to be on a quest for an e-mail program that does not exist, unless I want to use Outlook. (The e-mail program, which I mentioned, that I thought offered Unicode support actually does not, according to the support staff for Courier 3.5, the latest version of the program.)

I posted a similar query to the discussion list of users of The Bat and received no responses at all. (I am going to stop subscribing, because of the overall tone of the discussions; PocoMail's forums are much friendlier!) Given the popularity of The Bat in Russia and Ukraine, to name two countries, I cannot understand why I am having such trouble with that program either.

The bottom line is that I would prefer to use PocoMail, but I realize that Unicode support could be many, many months away. At this point, I am totally stuck.

Thanks!

Cordially,

David

[frazmi]

Just a couple of thoughts...

1) For my "Korean" keyboard, I bought a regular keyboard and some dry transfer letters. I popped the keys off one at a time, and rubbed the letters onto the front side (not the top) of the key. They are totally legible, and I don't have any problem with wear, since my fingers never touch the dry-transfer letters. (If I had need to clean the keys often, I would spray clear matte laquer over the keys after adding the letters -- and before re-inserting the keys on the keyboard switches.)

2) It sounds like you probably should send the bulk of your messages as in-line attachments, rather than trying to compose them directly in the Poco editor. You could use the scripts that invoke the external editor, and change the result to add an attachment instead of replacing the message body.

Good luck...

[DavidMD]

Hello, Frazmi!

Thank you for your suggestions:

Just a couple of thoughts...

1) For my "Korean" keyboard, I bought a regular keyboard and some dry transfer letters. I popped the keys off one at a time, and rubbed the letters onto the front side (not the top) of the key. They are totally legible, and I don't have any problem with wear, since my fingers never touch the dry-transfer letters. (If I had need to clean the keys often, I would spray clear matte laquer over the keys after adding the letters -- and before re-inserting the keys on the keyboard switches.)

2) It sounds like you probably should send the bulk of your messages as in-line attachments, rather than trying to compose them directly in the Poco editor. You could use the scripts that invoke the external editor, and change the result to add an attachment instead of replacing the message body.

Good luck...

I have been searching for a Russian/English keyboard that matches the Russian ('RU') keyboard layout of Windows XP (Professional) for a couple of years now. For some reason, every Russian/English keyboard made for Windows, even supposedly XP, does not match Microsoft's layout. I did receive a reply from a company at <http://www.russiankeyboards.com/> indicating that they have a Russian/English keyboard that matches XP's layout. I have checked their Web site, however, and found three keyboards that supposedly are for Windows XP. One has the standard Russian layout and the other has the Russian typewriter layout -- which means that the company wants you to spend $79.99 for a package that includes software, which I assume includes layout software, in addition to the two sets of fonts. (I do not need the Russian system fonts, at this point, anyway.)

I would probably be better off using the Cyrillic Starter Kit (http://www.cyrillic.com/cyrstart/index.html) with one of my two Russian/English keyboards. It includes keyboard layout software, is about $30.00, and I would not need to buy a keyboard. (I have two R/E keyboards, because I occasionally had startup problems with the first one. Replacing it stopped the problems, although my step may have been a coincidence.) In addition, I can try the Cyrillic Starter Kit for free, to make sure that it works. (In other words, there still is no Russian/English keyboard whose layout matches the RU layout in Windows XP.)

Your idea of the dry transfer letters on the fronts of the keys, Frazmi is an excellent one, although I would have to find a source for the Cyrillic letters. (Eventually, I would like to start learning Ukrainian, simply because it is Ukraine's official language and Russian is no longer taught in Ukrainian schools -- part of the government's "de-Russification" program.)

I have pretty much given up on using the Poco editor directly to type Russian text, Frazmi, and I have given up on using a signature file with Russian text in it, with PocoMail, which is a major disappointment. (It took me forever to achieve this goal in The Bat! 3.0. I finally figured out that I needed to compose the Russian text in Windows 1251 format and send the message in KOI8-R format. I can only presume that TB! is like many e-mail programs, converting Windows 1251 encoding to KOI8-R "on the fly" as it sends the message, but I have just sent such a message to my Linux system, which uses KMail, as a test.)

I will try to find the scripts to use an external editor, Frazmi, and to try your suggestion. I was going to try your suggestion of using Word as an external editor, but I cannot find this script, perhaps because I am using an evaluation copy of PocoMail.

I have gotten The Bat! to allow me to compose Russian text directly in TB!'s editor, with KOI8-R as my default encoding, but I cannot begin to estimate how long all of these efforts took with TB! -- and I very, very much want to use PocoMail! (Typing Russian in the Poco editor, however, even with KOI8-R encoding, simply does not work, and Russian text pasted into the Poco editor causes each Cyrillic letter to be replaced by a question mark.)

I will give your suggestions a try, Frazmi, and I truly appreciate your time and patience with me! Unicode support is my number-one wish for PocoMail. I know that I can get support with PocoMail, and "support" is basically nonexistent with TB! (I just hope that I can give your suggestions a try in the evaluation version of PocoMail; as I say, the option to use Word as the editor is not available to me in my evaluation copy.)

I could get around all of this heartache by continuing to use Outlook, but I have about reached the limit for Outlook XP/2002's .PST file and would have to upgrade to Outlook 2003. I do not need to elaborate about my security concerns with using Outlook! (Microsoft makes it "easy," but not safe!)

Finally, speaking of the Cyrillic Starter Kit, the information about it refers to Eudora, which I know supports US ASCII text only. Perhaps the CSK would resolve my problems with PocoMail, without having to jump through so many proverbial hoops. I realize that spending $30 in order to be able to register and use PocoMail would make using PocoMail a very costly investment, but PocoMail has "good karma" that I do not sense with competitors, such as TB!

Thanks, again, Frazmi!

Cordially,

David

P.S. -- I apologize for the length of this reply, and I thank you for your patience!

[DavidMD]

Hello, everyone!

I am still struggling with getting PocoMail to handle Russian/Cyrillic text.

Today (18 October), I partially solved the problem of using Russian with PocoMail.

In order to have a signature file that contains Russian text (a version by Pushkin), I used an evaluation copy of EmEditor Professional to encode a text file as Windows-1251/Cyrillic. I have my default message encoding as KOI8-R, but PocoMail apparently converts all Russian text in a message to KOI8-R before transmitting it.

This step is a major victor for me and puts PocoMail "back in the running" against The Bat, whose development team at RITLabs apparently plans to have Unicode support enabled by their traditional "Christmas release."

Anyway, the key to getting PocoMail to import a signature file containing Russian text is to encode the file as Windows-1251 (Cyrillic), even though my messages' default encoding is KOI8-R.

Most of the time, I have been able to paste Russian text into a PocoMail message, but as styled text only. (I was able to do so successfully for a while today, but suddenly started getting a question mark in place of each Cyrillic character, but I will try again after rebooting.)

Apparently, PocoMail converts all Cyrillic text to KOI8-R plain text before sending the messages, even if I paste a small portion of Russian text, such as a short poem, into a message as styled text. Is my assumption correction? (I prefer plain text always, but simply copying and pasting Russian text into PocoMail causes each Cyrillic character to be replaced by a question mark.)

I would like to use an external text editor, such as EmEditor, to write the body of my Russian/English messages, in Windows-1251/Cyrillic format. In theory, PocoMail would handle the imported "message-body text file" as it does my signature, and the entire message would be sent in KOI8-R format. I have been searching the on-line help to create some sort of %body%='text-file-name.txt' function, but I am making no progress.

Frazmi, I tried to adapt your script at the beginning of this thread for use with EmEditor, but the result was HTML with no spaces between paragraphs. I obviously have no business trying to do any scripting with PocoMail at this early stage! Besides, this approach overwrites my signature file, which I finally got working today. (I am not ruling out that the Cyrillic Starter Kit may be causing a conflict, which would truly be a sad irony.)

I would appreciate any suggestions from anyone. I am about to give up.

Thank you, very much, in advance!

Cordially,

David