OPEN DATASET... in a Unicode system

Question

I'm discussing with a developer @ SAP about the correct use of OPEN DATASET in a Unicode system. I'm not sure I'm correct with my opinion so maybe someone could shed a light on it.
The source of discussion is the (SAP standard) program RSQUEU01. This program is used to download data from the TemSE; in our case we use it to produce a file that is to be sent to auditors (component FI-AIS).
Our system is a little endian Unicode system (codpage 4103). If we download the created data using that program we get a dump with "CONVT_CODEPAGE" because a character could not be converted from 4103 to 1100.
The proposed correction by the developer is changing
OPEN DATASET EXP_FILENAME FOR OUTPUT IN legacy BINARY MODE.
to
OPEN DATASET exp_filename                              
FOR OUTPUT IN LEGACY BINARY MODE                     
IGNORING CONVERSION ERRORS.
I think, that correction is wrong. Since we have 12 languages in the system including some Asian the output might get corrupted.
When I asked him about that I was told use an application server with a "correct codepage" then - I'm not sure what that means since I can't connect an ASCII application server to a Unicode system.
I guess the statement should be
OPEN DATASET EXP_FILENAME FOR OUTPUT IN BINARY MODE ENCODING DEFAULT.
This makes sure that no data is cut (like doublebyte) and makes sure, the appropriate codepage (LE/BE) is used.
Are my assumptions right?
Markus
(OSS 323320/2010)

Answer 1

Hi Markus,
Let's first clarify the difference ways for writing files:
<ul style="list-style:circle!important;">
<li>BINARY MODE: Means that we essentially dump a sequence of bytes, which isn't necessarily related to any code page (and characters). I.e. if I'd want to save for example an executable program, the individual bytes have no meaning when interpreted as characters (unless we look at strings stored in the program). Note that legacy binary mode actually allows you to specify a code page though, but in general the recommendation is not to use the legacy option.</li>
<li>TEXT MODE: Here we have text information that has to be interpreted using a specific code page; thus usually the additional parameter ENCODING should be given, which specifies which code page is used.</li>
</ul>
Now, let's clear up a small typo in Ajay's response:
Yes you are right if you want to have all double byte characters too then you need to use ENCODING DEFAULT which would use 4103 in Unicode system.
That is incorrect. In a Unicode system [encoding default|http://help.sap.com/abapdocu_70/en/ABAPOPEN_DATASET_ENCODING.htm] corresponds to UTF-8, not UTF-16.
Back to your problem. Your suggestion doesn't work, because you cannot specify encoding default for a binary output (the legacy binary mode allows you to specify a code page, but that's misleading and I wouldn't use any legacy mode). So when you try to use the syntax you proposed, you'd get a syntax error.
Generally the recommendation is for Unicode enabled applications to use UTF-8 files with byte order mark, i.e. something like
open dataset EXP_FILE in text mode encoding utf-8 with byte-order mark.
However, the real question is what your external audit application expects and it sounds as if it's not Unicode enabled...
Enough blabber, here's what I'd do: Since you're having issues with a audit-related standard SAP program I'd post the question in forum - other people must have run into that problem. Also, I checked OSS, but couldn't make much sense out of the few notes I've found (and nothing seemed relevant). Check what the expected input format is in the external audit system; possibly post a message to OSS.
Cheers, harald