How to Determine Text File Encoding is UNICODE
Hi Gurus,
How to determine whether the file is a UNICODE format or not?
I have the file stored as a BLOB column in a table
Thanks,
Sombit
That's a rather hard problem. You would, realistically, either have to make a bunch of simplifying assumptions based on the data or you would want to buy a commercial tool that does character set detection.
There are a number of different ways to encode Unicode (UTF-8, UTF-16, UTF-32, USC-2, etc.) and a number of different versions of the Unicode standard. UTF-8 is one of the more common ways to encode Unicode. But it is popular precisely because the first 127 characters (which is the majority of what you'd find in English text) are encoded identically to 7-bit ASCII. Depending on the size and contents of the document, it may not be possible to determine whether the data is encoded in 7-bit ASCII, UTF-8, or one of the various single-byte character sets that are built off of 7-bit ASCII (ISO 8859-15, Windows-1252, ISO 8859-1, etc).
Depending on how many different character sets you are trying to distinguish between, you'd have to look for binary values that are valid in one character set and not in another.
Justin
Similar Messages
-
Convert Text file encoding in perticular format(Unicode)
Hi Expert,
I have requirement of transfering text file (encoding) in perticular file format to Application server ,by default SAP system generates in ANSI ,is it possible to convert it to Unicode format like UTF-8.If possible then how to generate the text file in unicode.
Thanks,
RegardsCheck
Note 752835 - Usage of the file interfaces in Unicode systems
Markus -
OPEN DATASET file FOR OUTPUT IN TEXT MODE ENCODING NON-UNICODE
Hi There,
I also have the similar issue. I am able to write the data into appliaction server in Chinese Characters using :OPEN DATASET datei FOR OUTPUT IN TEXT MODE ENCODING DEFAULT or OPEN DATASET datei FOR OUTPUT IN TEXT MODE ENCODING UTF-8. But when i save that file into my presentation server manually, all the chinese characters are showing as Junk.
When i use OPEN DATASET datei FOR OUTPUT IN TEXT MODE ENCODING NON-UNICODE, giving runtime error and when i use OPEN DATASET datei FOR OUTPUT IN TEXT MODE ENCODING NON-UNICODE IGNORING CONVERSION ERRORS, No error but application server output itself showing as Junk characters.
Could you please suggest me what you have done?
Regards,
Chaitanya AHi,
Use this
OPEN DATASET File_path FOR OUTPUT IN TEXT MODE ENCODING NON-UNICODE
WITH SMART LINEFEED
it will definitely work.
Regards,
Manesh. R -
Difference between IN LEGACY TEXT MODE & TEXT MODE ENCODING NON-UNICODE
Hi,
We're upgrading to ECC5 and the 'open dataset' command needs amending if the program is flagged for Unicode (which usually occurrs in user/fm exits). Therefore is ECC5 this command is no longer valid:
"open dataset DSN in text mode"
We currently interface with systems that may not have unicode enabled. Yet we have not enabled unicode in our own system just yet.
So we think these two commands are the most approriate for replacing the 'old' open dataset command:
"open dataset DSN for input in TEXT MODE encoding NON-UNICODE"
"open dataset DSN in LEGACY TEXT MODE for input"
However we're not really sure what the difference between these two commands is?
Has anyone worked with these commands?
Could you offer some help as to their differences and when each should be used?
Many thanks!Hi Robert,
Here is an excerpt from sap documentation.
... TEXT MODE ENCODING {DEFAULT|UTF-8|NON-UNICODE}
Effect:
The addition IN TEXT MODE opens the file as a text file. The addition ENCODING defines how the characters are represented in the text file. When writing in a text file, the content of a data object is converted to the representation entered after ENCODING, and transferred to the file. If the data type is character-type and flat, trailing blanks are cut off. In the data type string, trailing blanks are not cut off. The end-of-line marking of the relevant platform is applied to the transferred data by default. When reading from a text file, the content of the file is read until the next end-of-line marking, converted from the format specified after ENCODING into the current character format, and transferred to a data object.
The end-of-line marking depends on the operating system of the application server. In the MS Windows operating systems, the markings "CRLF" and " LF" are possible, while under Unix, only "LF" is used. If, when using Windows, an existing file is opened without the TYPE addition (see os_addition), the first end-of-line marking is found and used for the whole file. If a new file is created without the TYPE addition, the content of the profile parameter abap/NTfmode is used. If the profile parameter is not set, "CRLF" is used. If a file with the TYPE addition is opened and a valid value is contained in attr, this value is used.
In Unicode programs, only the content of character-type data objects can be transferred to text files and read from text files. The addition ENCODING must be specified in Unicode programs, and can only be omitted in non-Unicode programs.
The additions after ENCODING determine in which character representation the content of the file is handled.
DEFAULT
In a Unicode system, the designation DEFAULT corresponds to the designation UTF-8, and the designation NON-UNICODE in a non-Unicode system.
UTF-8
The characters in the file are handled according to the Unicode character representation UTF-8.
NON-UNICODE
In a non-Unicode system, the data is read or written without being converted. In a Unicode system,the characters in the file are handled according to the non-Unicode-codepage that would be assigned to the current text environment according to the database table TCP0C, at the time of reading or writing in a non-Unicode system.
If the addition ENCODING is not specified in non-Unicode programs, the addition NON-UNICODE is used implicitly.
... LEGACY TEXT MODE [{BIG|LITTLE} ENDIAN] [CODE PAGE cp]
Effect:
Opening a Legacyfile. The addition IN LEGACY TEXT MODE opens the file as a legacy text file. As with legacy binary files, the byte order and the codepage with which the content of the file should be handled can also be specified. The syntax and meaning of {BIG|LITTLE} ENDIAN and CODE PAGE cp are the same as for legacy binary files.
In contrast to legacy binary files, the trailing blanks in a legacy file are cut off when writing character-type flat data objects in a legacy text file. As for a text file, an end-of-line marking is also applied to the transferred data. In contrast to text files opened with the addition INTEXT MODE, Unicode programs do not check whether the data objects used for reading or writing are character-type. Furthermore, the LENGTH additions of the statements READ DATASET and TRANSFER are used for counting in bytes in legacy text files and in the units of a character represented in the memory for text files.
Note:
As with legacy binary files, text files that have been written in a non-Unicode system can be accessed in Unicode systems as legacy text files, and the content is converted accordingly.
Example
A file test.dat is created as a text file, filled with data, changed, and exported. As every TRANSFER statement applies end-of-line marking to written content, after the change, the content of the file has two lines. The first line contains "12ABCD". The second line contains "890". The character "7" has been overwritten by the end-of-line marking of the first line.
DATA: file TYPE string VALUE `test.dat`,
result TYPE string.
OPEN DATASET file FOR OUTPUT IN TEXT MODE ENCODING DEFAULT.
TRANSFER `1234567890` TO file.
CLOSE DATASET file.
OPEN DATASET file FOR UPDATE IN TEXT MODE ENCODING DEFAULT
AT POSITION 2.
TRANSFER `ABCD` TO file.
CLOSE DATASET file.
OPEN DATASET file FOR INPUT IN TEXT MODE ENCODING DEFAULT.
WHILE sy-subrc = 0.
READ DATASET file INTO result.
WRITE / result.
ENDWHILE.
CLOSE DATASET file.
Regards,
Ravi -
How to read text file line by line...?
how to read text file line by line, but the linefeed is defined by user, return list of string, each line of file is a item of list?
please help me.
Thanks very muchBrynjar wrote:
In Groovy, you would do something like:
linefeed = "\n" //or "\r\n" if the user chose so
lines = new File('pathtofile').text.split("${linefeed}")This is one of the things that has always annoyed me about Sun's sdk, i.e. the lack of easy ways to do things like that. You always end up making your own utilities or use something like Apache's commons.io. Same goes for jdbc and xml - I'll wait for appropriate topics to show how easy that is in Groovy :)I generally agree, but what I really don't like about the Groovy text-file handling niceties: They don't care about encoding/always use the default encoding. And as soon as you want to specify the encoding, it gets a lot more complex (granted, it's still easier than in Java). -
What determines the file encoding for
${C:file.txt} = 'abc'
I'm always getting ASCII as the encoding for file.txt after executing that assignment.Thanks so much. I'll keep looking for the MSFT doc on this. I scanned Bruce Payette's book and did not find anything there.
It turns out to be one of those "by rote" things you have to learn about PowerShell.
My concern about the lack of documentation is that MSFT might change the underlying code in the future to use Unicode and that might break some existing code. If there was some MSFT provided documentation declaring ASCII as the intended encoding they
might provide plenty of warning if they do a switch in encoding.
I note also that if you try to write characters outside the ASCII set (see example below) that character substitution happens to find an ASCII character to use in place of the one outside the ASCII set. In the example below a 'v' is substituted for
the '√' character:
${C:xo.txt} = '√' -
Text File Encoding used by TextEdit/OS X
Hi all folks,
does someone know the code page are used with the text file encoding "Western (EBCDIC US)"
available from the "Customize Encodings List" in the TextEdit "Plain Text File Encoding" Preferences.
The text file encoding "Western (EBCDIC Latin 1)" works well, but "EBCDIC US" does not,
the character set is very limited.
Thanks for any help,
LutzYeah unfortunately they're all listed as 0kb files. I guess that means the faulty hard drive didn't transfer them properly, even though the Mac did the copy confirmation sound.
Hundreds of folio files... all gone. ;___; -
How to determine the file system on Solaris
Friends,
How to determine which file system I have installed UFS or ZFS on Solaris
ThanksOther methods would include looking at the /etc/vfstab if it's in there or fstyp(1M):
System Administration Commands fstyp(1M)
NAME
fstyp - determine file system type
SYNOPSIS
fstyp [-a | -v] special [:logical-drive] -
How to write text file in Shockwave?
Does anybody know how to write text file in Shockwave to
user's disk?
Thanks in advance.Those Xtras can wreak to much havoc when used with the wrong
intent.
What you can do is write with setpref and store a list of
saves and the
saves itself seperatly. Then you'd have to build your own
save/open
dialog to let the user:
* pick a previously saved file to load or overwrite
* have the user type the name of a new file to save.
Only thing that remains is that the user cannot decide where
the files
are saved.
Manno
SiuLinda wrote:
> Thanks a lot for your reply.
> Yes, cookies is good but I have to write a program to
save the text file in
> where the user wants, user can open these files later if
they like, like using
> Filextra and Fileio, but I found all of these xtras seem
to be not supported in
> shockwave.
>
Manno Bult
[email protected] -
How to load text file data to Oracle Database table?
By using Oracle Forms, how to load text file data to Oracle Database table?
Metalink note 33247.1 explains how to use text_io as suggested by Robin to read the file into a Multi-Row block. However, that article was written for forms 4.5 and uses CREATE_RECORD in a loop. There was another article, 91513.1 describing the more elegant method of 'querying' the file into the block by transactional triggers. Unfortunately this more recent article has disappeared without trace and Oracle deny its existence. I know it existed as I have a printed copy in front of me, and very useful it is too.
-
plz tell me .. how to load and compare two text files using file popup's . example file i have attached..
Attachments:
testW_FF.txt 2 KBI don't understand whether your question is on how to load text files or how to show them on a panel or how to compare them... or all aspects together!
The first operation (loading the file) can be accomplished with functions included in the Formatting and I/O Library like OpenFile, ReadFile and so on; with a file like yours even FileToArray could be an option.
How to show the data on screen is heavily dependent on what you intend to do with them: data can be shown in textboxes, listboxes, tables or graphs so... what do you want to do?
The same applies with comparison: without additional details is difficult to give you the proper hint.
Proud to use LW/CVI from 3.1 on.
My contributions to the Developer Zone Community
If I have helped you, why not giving me a kudos? -
Step by Step"How JSP read text file :
Hi ,
Any one know or have a good site to show step by step how JSP read text file.
TQ.There is no difference Between reading a text file from JSP and reading a text file from Java.
Just follow the same steps for JSP also. -
How to call text file using Script in Data Integrator
Dear All,
Can any one assit me in how to call a text file using script with the help of Data Integrator.
and one question ?
M having 32 csv files i want to club thos 32 csv files into one table with the help of Data Integrator, can
any one assist me.mary,
since you knew the file name ,when clicked in name send to server,read the file and write to servlet outputstream.
I think this would help you.
If anything wrong in mycode ..forums will help you further
BufferedInputStream bis=null;
BufferedOutputStream bos=null;
int bytesRead=0;
byte buff[]=new byte[1024];
File f=new File(test.txt);
try{
bis= new BufferedInputStream(new FileInputStream(f));
bytesRead=bis.read(buff,0,buff.length);
if(bytesRead!=-1){
// create a BufferedOutputStream from ServletOutputStream
bos=new BufferedInputStream(response.getOutputStream());
do{
bos.write(buff,0,bytesRead);
}while((bytesRead=bis.read(buff,0,buff.length))!=-1)
}catch(Exception e){
////error handling
} -
How to set the file.encoding in jvm?
I have some error in showing Chinese by used servlet,
some one tole me that I can change the file.encoding in jvm to zh_CN, how can I do that?Add the java argument in your servlet engine.
e.g
java -Dfile.encoding=ISO8859-1
garycafe -
How to read text file content in portal application?
Hi,
How do we read text file content in portal application?
Can anyone forward the code to do do?
Regards,
AnaghaCheck the code below. This help you to know how to read the text file content line by line. You can display as you require.
IUser user = WPUMFactory.getServiceUserFactory().getServiceUser("cmadmin_service");
IResourceContext resourceContext = new ResourceContext(user);
String filePath = "/documents/....";
RID rid = RID.getRID(filePath);
IResource resource = ResourceFactory.getInstance().getResource(rid,resourceContext);
InputStream inputStream = resource.getContent().getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
String line = reader.readLine();
while(line!=null) {
line = reader.readLine();
//You can append in string buffer to get file content as string object//
Regards,
Yoga
Maybe you are looking for
-
Dear Apple support team I have just bought an application, it is an image converter, but it did not mention that it doesn't support a specified image formats. When I bought it I could not get any use of it because it does not dupport NTF format. Can
-
Problem installing Flash v10.1
Hi everyone, I am new to this forum and am doing this because I have been trying to install flashplayer10_1r102_64_ub_mac.dmg to my (now quite old) 1.42 power pc G4. I have read a few old posts from these pages and followed instructions that have hel
-
Flashing 19 and 36 error code on my motherboard led display
Hello, When I try to restart my computer after it has been on for awhile, it wont restart. The fans remain on and the power light too. On my mobo where the led display is, has a blinking 19 and 36 along with a red flashing light. Anyone know what the
-
Any way to change the Accept-Encoding in apache/jsp?
I have a jsp page which needs to not be delivered using gzip compression and currently my apache is configured to compress any text content. IS there anyway to diable the gzip encoding through a request, or response command OR an directive at the top
-
Anamorphic still not exporting properly. This never seems answered
Hi I shot a video on a Canon XM2 in anamorphic 16:9, edited it on avid, now iv moved it home to my mac and added effects on FCP. I want to export it, among other things to put it on youtube. It always exports as anamorphic. I know FCP allows you to v