Unicode type coersion - revisited

Question

I am still having a heck of a time with reading and comparing stuff from text files using vanilla AppleScript (I would prefer not to use 3rd party additions or applications). The file contents are basically ASCII text, but they can be stored as either plain text or Unicode files. Strings can be converted to Unicode, but there doesn't appear to be any coersion from Unicode back to plain text. I have a cheapo conversion subroutine that just strips out the null characters of a Unicode string, but there has to be a better way to do it. Since it looks like Unicode is the way things are going, is it better to just use everything as Unicode text? Before I start tearing up my script, are all the various places like dialogs, the Finder, shell scripts and the like Unicode savvy? Tracking all the ApleScript changes is a bit tricky, since the last manual is something like 1.3.7. If someone is familiar with this, I would appreciate the help.
G4 Tower 733MHz   Mac OS X (10.4.8)  

Answer 1

Hello
Here's a little more consistent version which will always replace any character in the source Unicode text that cannot be represented in system's primary encoding with "?", consequently will not yield raw byte sequence extracted from intermediate styled text which may be improperly interpreted in system's primary encoding.
Tested with AS1.8.3 under OS9.1 (Engilsh).
Hope this may be of some help.
H
-- SCRIPT
  utxt2string
  v0.2
  * solely using special behaviour of 'as string' coercion for Unicode text that contains character
  which can not be represented by any of installed fonts.
set s to "abcdefghijklmnopqrstuvwxyzáàâäãåçéèêëíìîïñóòôöõúùûüæøœÿ" -- string in System's primary encoding (e.g. MacRoman)
set ut to s as Unicode text -- UTF-16
set cyYa to «data utxt042F» as anything -- U+042F : CYRILLIC CAPITAL LETTER YA
set cmd to «data utxt2318» as anything -- U+2318 : PLACE OF INTEREST SIGN (= COMMAND KEY)
set ut to ut & cyYa & cmd
set s1 to utxt2string(ut)
return {ut, s1}
on utxt2string(ut)
  Unicode text ut : source unicode text
  return string : string (in system's primary encoding) that corresponds to ut. -- [*0]
local x
if ut's class is not Unicode text then error "parameter error" number 9000 -- for safety
set x to «data utxtD800» as anything -- an invalid surrogate pair (an orphan high surrogate)
return ((ut & x) as string)'s text 1 thru -2 -- [*1]
  NOTES.
  [0] This string will be plain string («class TEXT») (not styled text («class STXT»)) where
    each character unrepresentable in system's primary encoding will be replaced by "?".
  [1] This 'as string' coercion for Unicode text that contains character
    which can not be represented by any of installed fonts will yield plain string (not styled text),
    where each character that is not representable in system's primary encoding will be replaced by "?".
    (Here, the character x appended at the end of ut is unrepresentable (invalid), which will result in
    this coercion of desired behaviour.)
end utxt2string
-- END OF SCRIPT
  Mac OS 9.1.x