Transformation of characters in records

Hello,
at the testing applications written in dot.net 2.0 (System.Data.OracleClient provider from Microsoft) for one customer who uses Oracle 9i (client and server) appeared the following problem: from a explicit record with text fields began to transform into another character set (differed according to select table and the number of columns in select too). Example:
Correct item on line number 159: prices1
Corrupt item on line number 160 (to be prices1 too): 散 ㅮ
Items are copied from the file that is in UTF16LE. In hexa (via PSPad) file looks like this:
FFFE - BOM introductory document
followed lines seems OK, up to line number 160:
line 159: 6300 6500 6E00 3100 (prices1 in UTF - correct)
line 160: 6365 6E31 (散 ㅮ - ASCII chars, corrupt)
All following lines are corrupted.
On line 160 are characters like from line 159, but trimmed the 8bit (00) - this is ASCII characters.
Records in DB are correct. If I select all records via SQLdeveloper, PLSQL developer etc..., all records are correct.
If I change ORDER BY in SELECT clause in .net app, the problem starts on the same line (data that was previously corrupted, but now get to a lower position, are now correct)
If I made a select from another table, a problem occurs, but on another line. Example (other table and I changed encoding in output file to UTF-8):
all lines to line number 965 are OK
line 965: 37 33 32 (732 - correct)
line 966: E3 8C B3 (ãŒ³ - corrupt, should be 733)
If someone have some idea, thanks very much.
dan.

Similar Messages

Error: 'InfoObject ZDATE contains invalid characters in record 1 in value'

Dear friends,
When I am uploading data from oracle datasource (in BW 3.5) I am getting an error 'InfoObject ZDATE contains invalid characters in record 1 in value'. There are no illegal characters in the date instead the date field is blank in datasource. There are other date fields technically the same as ZDATE with blank values but they dont throw an error when scheduler runs.
Though when I manually upload the data, it's loaded successfully.
Please suggest where can the error be?
Regards,
Amit Srivastava

You have 2 options:
1. Corect your data for the characteristics that give errors.
Or
2. go to SPRO in BW - Netweaver - BW - general settings - maintain permitted characters.
In this screen, you maain tain any special characteristic that you the system to recognise and accept. In your case, take your characteristics that give error and add to this.
Ravi Thothadri

Lookup in transformation not fetching all records

Hi Experts,
In the routine of the transformation of a DSO (say DSO1), i have written a look-up on other DSO (say DSO2) to fetch records. I have used all the key fields of DSO2 in the Select statement, Still the look-up is not fetching all the records in DSO1. There is difference in the aggregated value of the Key Figure of both the DSOs. Please suggest, how can i remove this error.
Thanks,
Tanushree

hi tanushree,
The code which yu have written in the field routine for lookup is not fetching the data. you can debug the field routine code in the simulation mode of execution of DTP by keeping a break point after the transformation.
you can test the routine with out actually loading the data..
double click rule where you have routine and in the below you have option called test routine.
here you can pass input parameters..
i hope it will give you an idea.
Regards
Chandoo7

Problems with transforming special characters

Hi,
I develop a small educational application ( http://sourceforge.net/projects/pauker/ ). I work with JDK-1.4.0 on Mandrake Linux 8.2. At first I used serialized objects to save the lessons to a file. This worked well until I wanted to change some public members of the involved classes. That's why I switched over to the new and shiny XML. Now I have a different problem!
Pauker saves its lessons in gziped XML files. Users from all over the world can create lessons containing very different characters. There are European characters like �� and asian characters. Loading this lessons on a system with a different encoding works fine. Saving such a lesson on a system with a different encoding can destroy the lesson.
Example:
On a german system a user creates a lesson with the letter � on a card side and saves it. A different user working on an english system loads this lesson. The character "�" is displayed correctly. The english user saves the lesson. The character "�" will be replaced by a question mark in the xml file. Next time the english user loads the lesson she will not see "�" but "?" on the display.
Here is a little example program that does the transformation in exactly the way Pauker does. Please test it out.
import java.io.*;
import javax.xml.parsers.*;
import javax.xml.transform.*;
import javax.xml.transform.dom.*;
import javax.xml.transform.stream.*;
import org.w3c.dom.*;
public class XMLTest {
public XMLTest() {
try {
// create document
DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder documentBuilder = documentBuilderFactory.newDocumentBuilder();
Document document = documentBuilder.newDocument();
// fill document
Element element = document.createElement("Element");
document.appendChild(element);
element.appendChild(document.createTextNode("��"));
// transform to XML
TransformerFactory transformerFactory = TransformerFactory.newInstance();
Transformer transformer = transformerFactory.newTransformer();
//transformer.setOutputProperty(OutputKeys.ENCODING, "ISO-8859-1");
transformer.setOutputProperty(OutputKeys.INDENT, "yes");
transformer.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", "2");
DOMSource source = new DOMSource(document);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
StreamResult result = new StreamResult(outputStream);
transformer.transform(source, result);
System.out.println("original: ��");
System.out.println(outputStream.toString());
} catch (Exception e) {
e.printStackTrace();
System.out.println();
public static void main(String[] args) {
new XMLTest();
So what do I have to do to fix this problem? I have a lot of new features waiting in the CVS but this bug is still open and discussed. I dont want to release a new version with such a gaping hole in it...
Thanks a lot!
If you prefer to reply peronally, please use Ronny.Standtke at gmx.de

I don't see how you can say that there's a problem with saving your XML files when the code you post doesn't actually save it to a file. Your transform is being written to a ByteArrayOutputStream, which isn't used except for this statement which I assume is for debugging:System.out.println(outputStream.toString());Of course that is useless for debugging the problem you describe, for two reasons:
1. the toString() method uses your system's default encoding, which may be ISO-8859-1 but is definitely not UTF-8. You could write toString("UTF-8") but that is a waste of time because:
2. You use System.out.println() to examine the data, which writes it to a console that also probably does not use UTF-8. I don't know what encoding it does use, but UTF-8 is unlikely.
So, save your files using the UTF-8 encoding as robadmin suggested. And to test the result, make sure you use a tool that understands the UTF-8 encoding.

Row to Column Transformation for Millions of records

Hi Members,
I need to transform data from two stage tables which has data in PIM structure (data stored as separate records ) to target table which has a flat structure (data stored in a single record). One of the stg tables has data volume of 45M and other has 5M records.The challenge I am seeing here is to transform such huge data into single table with considerable performance.What would be the ideal way to transform such huge data?Also can we have multiple programs running at the same time to achieve transformation for such huge data load quicker?
Just to add my Oracle Version is 10g.
Thanks
Edited by: Sonic on Jul 12, 2010 1:33 AM

Still no version number, still no code, and no explain plan report.
Is there a better, faster way to do it?, I don't know ... how could I or anyone else as you've not told us what you are doing beyond the level of "my car won't start tell me why?"
This should help you understand the issue from my keyboard.
http://www.builderau.com.au/strategy/developmentprocess/soa/How-developers-should-ask-for-help/0,339028278,339299138,00.htm

Simulate update in Transformation using a single record

In an update rule, it was possible from the monitor to simulate the update of an infoProvider. During the simulation, you could choose to select only certain records of the PSA for simulation. This was very useful if you wanted to simulate only a single problematic record.
In a transformation using a DTP, I can find out how to add the breakpoints and debut a start routine in a transformation, but I cannot find a way to do that simulation using only a single record from the PSA.
The only option I can find is to create a DTP that has a filter that restricts the PSA down to a single record, but this is time consuming as you first have to find a way to filter the PSA to select a unique record.
I have tried to use the ENTER DATA PACKAGE when setting the breakpoints, but this does not seem to allow this to happen.
Is there a similar function in the DTP that allows you to simulate the update on a single selectable record?

Hi Kevin,
this function is not available from the transformation, yet check the DTP, where you can set the processing to serial. You can then change breakpoints at the processing steps of the DTP, and will also be asked to enter for which package you want to simulate the DTP or use the expert mode [next to processing mode] (I hope this is correct since I couldn't verify it in the system due to time constraints).
Please search also the forum, since the topic simulation of DTP was already a topic here.
Cheers
SAP NetWeaver BI Organisation

Maximum record length in sender file adapter

Hi,
My requrement is to read a test file which contains the records of fixed length of 4096 charcters. I tried a quick test but sender adapter seems to be terminating the records.
Is there any limitation on the number of characters in records which file adapter can read? Please suggest. I am putting togather a prototype for a potential client and it will have a big impact on decision whether PI can be used in project or not.
Thanks for help!

Never tried this one. How ever, were you trying with content conversion? When you say the records are being truncated, can you tell after how many characters the adapter is truncating the record?
VJ

Transformation Rule: Error while loading from PSA to ODS using DTP

Hi Experts,
I am trying to load data from PSA to ODS using DTP. For about 101 records I get the following error:
"Runtime error while executing rule -> see long text     RSTRAN     301"
On further looking at the long text:
Diagnosis
    An error occurred while executing a transformation rule:
    The exact error message is:
    Overflow converting from ''
    The error was triggered at the following point in the program:
    GP4808B5A4QZRB6KTPVU57SZ98Z 3542
System Response
    Processing the data record has been terminated.
Procedure
      The following additional information is included in the higher-level
     node of the monitor:
     o   Transformation ID
     o   Data record number of the source record
     o   Number and name of the rule which produced the error
Procedure for System Administration
When looking at the detail:
Error Location: Object Type    TRFN
Error Location: Object Name    06BOK6W69BGQJR41BXXPE8EMPP00G6HF
Error Location: Operation Type DIRECT
Error Location: Operation Name
Error Location: Operation ID   00177 0000
Error Severity                 100
Original Record: Segment       0001
Original Record: Number        2
Pls can anyone help in deducing and pointing this error to the exact spot in the transformation rule
Thanks & Regards,
Raj

Jerome,
The same issue.
Here are some fields which are different in terms of length when mapped in transformation rules
ODS                    |Data Source
PROD_CATEG     CHAR32           |Category_GUID      RAW 16
CRM_QTYEXP     INT4          |EXPONENT      INT2
CRM_EXCRAT     FLTP16          |EXCHG_RATE     Dec 9
CRM_GWEIGH     QUAN 17, 3     |Gross_Weight     QUAN 15
NWEIGH          QUAN 17, 3     |Net_Weight     QUAN 15
CRMLREQDAT     DATS 8          |REQ_DLV_DATE     Dec 15
The difference is either some dats field are mapped to decimal, or the char 32 field is mapped to raw 16 OR Calweek, Calmonth is mapped to Calday
Both mostly all the ods field size is greater than the input source field.
Thanks
Raj

Error in Transformation Rules - Runtime Error

Hi Experts,
I am trying to load data from PSA to ODS using DTP. For about 101 records I get the following error:
"Runtime error while executing rule -> see long text     RSTRAN     301"
On further looking at the long text:
Diagnosis
    An error occurred while executing a transformation rule:
    The exact error message is:
    Overflow converting from ''
    The error was triggered at the following point in the program:
    GP4808B5A4QZRB6KTPVU57SZ98Z 3542
System Response
    Processing the data record has been terminated.
Procedure
      The following additional information is included in the higher-level
     node of the monitor:
     o   Transformation ID
     o   Data record number of the source record
     o   Number and name of the rule which produced the error
Procedure for System Administration
When looking at the detail:
Error Location: Object Type    TRFN
Error Location: Object Name    06BOK6W69BGQJR41BXXPE8EMPP00G6HF
Error Location: Operation Type DIRECT
Error Location: Operation Name
Error Location: Operation ID   00177 0000
Error Severity                 100
Original Record: Segment       0001
Original Record: Number        2
Pls can anyone help in deducing and pointing this error to the exact spot in the transformation rule
Thanks & Regards,
Raj

Hi Rajesh,
Why don't you debug that program.
You can go to his program : GP4808B5A4QZRB6KTPVU57SZ98Z 3542
How :
1. tcode : se38
2. type : GP4808B5A4QZRB6KTPVU57SZ98Z 3542
3. Set highlight there,
Then setup with debug mode, finnaly you execute the dtp.
And watch carefully, in what part you get an error.
Hopefully it can help you a lot.
Regards,
Niel
thanks a lot for any points you choose to assign.

Can't get affine transforms do anything on an AttributedString

Hello,
I'm trying to transform individual characters in an AttributedString using the respective attribute, but nothing happens as though the transforms simply were ignored by the renderer. Other attributes like underline and strikethrough work fine, however. Am I missing something?
import java.awt.*;
import java.awt.font.*;
import java.text.*;
import java.applet.Applet;
import java.awt.geom.*;
public class test4 extends Applet {
     public void paint(Graphics g) {
          Graphics2D g2 = (Graphics2D) g;
          String s = "abc";
          Font f = new Font("Times New Roman", Font.PLAIN, 24);
          AttributedString as = new AttributedString(s);
          as.addAttribute(TextAttribute.FONT, f);
          as.addAttribute(TextAttribute.UNDERLINE, TextAttribute.UNDERLINE_ON, 0,
                    1);
          as.addAttribute(TextAttribute.TRANSFORM, new TransformAttribute(
                    AffineTransform.getScaleInstance(0.5, 3)), 1,3);
          g2.drawString(as.getIterator(new
                    TextAttribute[]{TextAttribute.TRANSFORM,TextAttribute.UNDERLINE}),30,40);
}

Hi,
I'm not sure why the code you have does not work but a quick search for bugs revealed that there are some outstanding issues relating to AttributedString. It may be worth looking into a little more.
The following code seems to get the desired result if it helps:
String s = "abc";
Font f = new Font("Times New Roman", Font.PLAIN, 24);
AttributedString as = new AttributedString(s);
as.addAttribute(TextAttribute.FONT, f);
as.addAttribute(TextAttribute.UNDERLINE, TextAttribute.UNDERLINE_ON, 0, 1);
AffineTransform transform = AffineTransform.getScaleInstance(0.5, 3);
f = f.deriveFont(transform);
as.addAttribute(TextAttribute.FONT, f, 1, 3);
g2.drawString(as.getIterator(), 30, 100);

Suggestion: interactive update of text while its path is transformed

I've been using Ps CS6 for a couple of months now, and I occasionally used CS5.1 before that. I'm becoming increasingly frustrated with its lack of interactive feedback during many manipulations. A lack of feedback which is provided in applications costing a small fraction of the price. This is 2012, Ps is a leading graphics app in some ways, but in other ways it is pitifully like a relic from the early 1990s.
The latest case. You have text on a path and you decide to transform that path with Free Transform (Cmd+T). The positions of the text characters don't update while you manipulate the path! You don't see the changes until confirming the transform command. So you blindly transform again, confirm, and again, confirm, and again... again... again... until you chance upon the desired result.
Please improve this, Adobe.

Hi Noel
I think you're transforming the characters and their path.
I want the form and scale of each character to be unchanged. It's the path along which the characters flow that I'm transforming. Below is a trivial before and after example.
The characters' positions do not change during use of the transform command, e.g. Free Transform (Cmd+T), when it's only the path that's being transformed. It's like working with a blindfold on.

Creating email attachment with 255 char per record

Hi,
We want to generate an email with excel attachment through ABAP program. To do the same, we are trying to use FM 'SO_DOCUMENT_SEND_API1'.
It looks like it only allows 255 characters per record however we want include upto 700 chracters per record. Does anyone know a way out to overcome this problem?
Note - We do not want to wrap the text to next line.
Thank you

Hi:
I would advice to make the copy of this function module SO_DOCUMENT_SEND_API1 and change the packing_list-doc_size = '700'..
Hopefully it would be helpful to solve the problem.
Regards
Shashi

SQL job running DTEXEC.EXE via PowerShell step produces wrong characters in result tables. Running the same DTEXEC.EXE in PowerShell x86 window produces correct results.

I've created a package in SSDT (Visual Studio 2012) to import DBF tables into MSSQL 2014 via a wizard.
Source DBF files contain tables with Russian (Cyrillic) characters in records fields.
Created package works fine, produces correct results (imports Cyrillic characters as needed) while:
•debugging in VS2012
•deploying package in SSISDB and running the package via PowerShell x86:
&"c:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\DTExec.exe" /Server "server\mssql2014" /ISServer "\SSISDB\import\Integration Services Project16\Package1.dtsx";
But when I create a SQL Agent job with a single PowerShell step that contains the same command and run the job it produces wrong characters in resulting tables. So, the problem is in wrong code page (or encoding). The correct tables with records are
produced and job runs without errors.
I tried
chcp 1251; # 855, 866 -- all of them
&"c:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\DTExec.exe" /Server "server\mssql2014" /ISServer "\SSISDB\import\Integration Services Project16\Package1.dtsx";
in SQL Agent PowerShell step, but with no positive result.
Anyone encountered such a behavior? How to fix it?
v

What if you run this package without PowerShell?
Arthur My Blog

How to split this string into 4 sections to a max 35 characters

Hello,
Does anyone have an idea how I can acheive this please.
I have this string
Expense_Inv_8- ExpenseInv_7- Exp001- Expense_Inv_6- Expense_Inv_5- Expense_Inv_4- Expense_Inv_3- Expense_Inv_2- Expense_inv1
and I need to display them in sections seperated by ';' as explained below
Section 1 Section 2 Section 3 Section 4
Expense_Inv_8- ExpenseInv_7- Exp001;Expense_Inv_6- Expense_Inv_5;Expense_Inv_4- Expense_Inv_3;Expense_Inv_2;
need to split this string into 4 sections seperated by ';' and each section should be of no more than 35 characters, if null should end ;;;
Section 1, 35 Character ended by;
Section 2, broken off after Expense_Inv_5 because Expense_Inv_4 will take it over 35 chracters)
Section 3, should only take Expense_Inv_4- Expense_Inv_3, because adding Expense_Inv_2 will take it over 35
characters, each record in the string is seperated by '-'
Section 4, dispays the reminder of the string
regards
Ade

Hi,
Welcome to the forum!
Whenever you ask a question, it helps if you post a little sample data (CREATE TABLE and INSERT statements, relevant columns only) and the results you want from that data.
I think I understand the problemk well enough to attempt a solution, but if the query below isn't right, please post that information.
WITH     cntr     AS
     SELECT     LEVEL     AS n
     FROM     dual
     CONNECT BY     LEVEL     <= ( SELECT MAX (LENGTH (txt))
                         FROM    table_x
,     got_best_path     AS
     SELECT     id
     ,     txt
     ,     MAX ( SYS_CONNECT_BY_PATH ( TO_CHAR (c.n, '99')
                  ) AS best_path
     FROM     cntr     c
     JOIN     table_x     x     ON     c.n <= LENGTH (x.txt)
     START WITH     c.n     = 1
     CONNECT BY     c.n - PRIOR c.n     BETWEEN 1
                             AND      :section_length
          AND     x.id          = PRIOR     x.id
          AND     SUBSTR ( x.txt
                             , c.n
                             , 1
                             )     = '-'
     AND     LEVEL          <= :section_cnt
     GROUP BY id
     ,            txt
,     got_pos     AS
     SELECT     id
     ,     REPLACE ( txt
               ) || ';'                         AS txt
     ,     best_path
     ,     TO_NUMBER (REGEXP_SUBSTR (best_path, '[0-9]+', 1, 2))     AS pos_2
     ,     TO_NUMBER (REGEXP_SUBSTR (best_path, '[0-9]+', 1, 3))     AS pos_3
     ,     TO_NUMBER (REGEXP_SUBSTR (best_path, '[0-9]+', 1, 4))     AS pos_4
     FROM     got_best_path
SELECT id
,     SUBSTR (txt,     1    , NVL ( pos_2         , :section_length))     AS section_1
,     SUBSTR (txt, pos_2 + 1, NVL ((pos_3 - pos_2), :section_length))     AS section_2
,     SUBSTR (txt, pos_3 + 1, NVL ((pos_4 - pos_3), :section_length))     AS section_3
,     SUBSTR (txt, pos_4 + 1,                        :section_length )     AS section_4
FROM     got_pos
;As written, this requires SQL*Plus 9 (or higher). You can have multiple versions or SQL*Plus on the same client, if you really need to keep the older version.
:section_length is the maximum length of each section (35, as you stated the problem).
:section_cnt is the number of sections. In the query above, this is 4. If you change it, you not only have to change the bind variable, but you have to change the hard-coded SELECT clauses of the main query and the last sub-query (that is, got_pos).
MODEL or PL/SQL would probably be better ways to solve this problem.

Problem with code: reading a string

Hi,
I am very new to java so excuse my naivety. i need to write some code which will decode an encrypted message. the message is stored as a string which is a series of integers which. Eg. Nick in this code is 1409030110 where a = 1, b = 2,... and there is a 0 (zero) after every word. at the moment the code only goes as far as seperating the individual characters in the message up and printing them out onscreen.
Heres the code:
int index = 0;
while(index < line.length() - 1) {
char first = line.charAt(index);
char second = line.charAt(index + 1);
if(second == 0) {
char third = line.charAt(index + 2);
if(third == 0) {
System.out.print(first + "" + second);
index = index + 3;
else {
System.out.print(first);
index = index + 2;
else {
System.out.print(first + "" + second);
index = index + 3;
Basically its meant to undo the code by reading the first two characters in the string then determining whether the second is equal to 0 (zero). if the second is 0 then it needs to decide whether the third character is 0 (zero) because the code could be displaying 10 or 20. if so the first two characters are recorded to another string and if not only the first is recorded. obviously if the second character is not a zero then the code will automatically record the first 2 characters.
my problem is that the code doesnt seem to recognise the 0 (zero) values. when i run it thru with the string being the coded version of Nick above (1409030110) the outputted answer is 1490010 when it should be 149311. as far as i can see the code is right but its not working! basically the zeros are begin removed but i need each group of non-zero digits to be removed seperately so i can decode them.
if anyone has understood any of this could you give me some advice, im at my wits end!
Thanks, Nick

Try something like this:
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.IOException;
import java.util.Properties;
public class SimpleDecoder
    /** Default mapping file */
    public static final String DEFAULT_MAPPING_FILE = "SimpleDecoder.properties";
    /** Regular expression pattern to match one or more zeroes */
    public static final String SEPARATOR = "0+";
    private Properties decodeMapping = new Properties();
    public static void main(String [] args)
        try
            SimpleDecoder decoder = new SimpleDecoder();
            decoder.setDecodeMapping(new FileInputStream(DEFAULT_MAPPING_FILE));
            for (int i = 0; i < args.length; ++i)
                System.out.println("Original encoded string: " + args);
System.out.println("Decoded string : " + decoder.decode(args[i]));
catch (Exception e)
e.printStackTrace();
public void setDecodeMapping(InputStream stream) throws IOException
this.decodeMapping.load(stream);
public String decode(final String encodedString)
StringBuffer value = new StringBuffer();
String [] tokens = encodedString.split(SEPARATOR);
for (int i = 0; i < tokens.length; ++i)
value.append(this.decodeMapping.getProperty(tokens[i]));
return value.toString();
I gave your "1409030110" string on the command line and got back "nick". The SimpleDecoder.properties file had 26 lines in it:
1 = a
2 = b
3 = c
// etc;Not a very tricky encryption scheme, but it's what you asked for. - MOD

Transformation of characters in records

Similar Messages

Maybe you are looking for