File Content Conversion: Remove reserved characters from Content

Hi PI Gurus,
we're facing a little problem with FCC in File Receiver Adapter.  Sometimes the defined fieldSeparator ("|") comes within the field content.
e.g.
<?xml version="1.0" encoding="UTF-8"?>
<ns0:Positions xmlns:ns0="http://test.com">
<Position>
<Value1>001</Value1>
<Value2>Test|Content</Value2>
</Position>
</ns0:Positions>
So, my first thought was to replace all "|" with a litte java mapping beind the actual mapping.
But, is there any possibility to achieve this via configuration of FCC (something like fieldSeperatorSubstitution)?
Help is as always highly appreciated!
Cheers,
Matthias Kral

Hello Hareenkumar,
yes you're totally right. I ran in several OutOfMemoryError while testing the code above. The errors started with messages over 15 MB.
Here's the code, that is productive now. It replaces all Pipes "|" by Slashes "|" and really works fine and performant.
public void execute(InputStream in, OutputStream out)
          throws StreamTransformationException {
          try {
               int totalCount = 0;
          byte[] buf = new byte[40960];
          int count = 0;
               while (count >= 0) {
                    count = in.read(buf);
                    totalCount += count;
                    if (count > 0) {
                         for (int i=0; i<count; i++){
                              if (buf<i>==124){
                                   buf<i> = 47;
                         out.write(buf, 0, count);
          } catch (IOException e) {
               throw new StreamTransformationException(e.getMessage());
Thank you all again!!!
Cheers Matthias

Similar Messages

  • Remove LF characters from file names

    I have a folder full of files with filenames that contain LF character (ASCII code 10).  I want to use Automator's "Replace Text" funcion to remove these non printing characters from file names.  Is there a way to do it?
    If automator is not able to do this task, I will take a bash script or applescript solution as well...

    Take a look at: http://stackoverflow.com/questions/4417588/sed-command-to-fix-filenames-in-a-dir ectory
    (I changed  tr -d "\r\n" to tr -d "\n", but try both)
    for f in ~/Desktop/*
    do
        new="$(printf %s "$f" | tr -d "\n")"
        if [ "$f" != "$new" ]; then
            mv "$f" "$new"
        fi
    done

  • Removing offending characters from string...

    Hello. I have some code that receives a csv file that is generated by an AS/400 - info about store orders, etc. However occasionally there is a character that causes an exception when I try to send the data into XML (prep for a web service call).
    So, I added some code to my process to check the individual bytes. It's a byte value of 28 that causes me problems. I sometimes find it where I am expecting an apostrophe. So for testing purposes I added a routine that replaces byte=28 with an apostrophe, and all is well.
    My question revolves around the fact that I don't have a great understanding of character encoding, etc. Essentially, what should I be doing? Should I be using different character encoding when reading the file (currently using ISO8859-1)? I don't have much control over the actual XML generation, so my hands are tied on that end. It just seems that the solution that I implemented is less-than-ideal. (Although searching through this forum, I did find it as a suggestion...)
    Any help is greatly appreciated.
    Chris

    So any thoughts on the question of how to get rid of it?
    Just iterate over all bytes as I am now? Or is there a better solution?I assume you are using something like a buffered reader to read in the CSV file contents. Does the ISO 8859-1 encoding have problems with the FS control code byte or does it simply read it in and decode it into \u0028 ? if the latter, my own inclination would be to handle it on the char level, (by replacement or removal) before you generate your XML. What you replace it with depends on its original function within the CSV file, which I don't know.
    Joe

  • Sql to remove special characters from my search

    Hi everyone.  I'm very new to sql and have hit another road block.  I am doing a query on my database in oracle sql developer.  I want to search manufacturer numbers but sometimes they were entered with dashes ( 999-99-9999) and other times not (999999999)  is it possible to apply a function to overlook the dash in both my query numbers and in the database mfr_nbr column?
    any help would be appreciated.
    Kelly

    ok,  I have built a nesting string of replaces to remove all of my special characters and it worked perfectly but now I am not sure where to place the nest later in the string to remove it from my search of mfr numbers.  The reason I need to do it again is because I want to remove the characters so I am searching in terms of " apples to apples" so to speak.   here is my string so far. I still need to add the part where I put in my search for the manuf_item_nbr.  my question is  where do I need to place the nested replace's to remove it from my numbers I'm going to search?
    SELECT  MAX(item_nbr) ,REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE( manuf_item_nbr, ' '), ','), '<'), '.'), '>'), '?'), ''), '"'), ';'), ':'), '\'), '|'), ']'), '}'), '['), '{'), '='), '+'), '_'), '-'), ')'), '('), '*'), '&'), '^'), '^'), '%'), '$'), '#'), '@'), '!'), '~'), '`'),
      manuf_item_nbr  ,MAX(description), MAX(description2), MAX(GHX_FULL_ITEM_DESCR), MAX(Cntrct_nbr_txt), MAX(uom_cd)   ,
    MAX(item_qty), MAX(tier_descr), MAX(tier_prc_amt),MAX(list_prc_amt), MAX(vndr_nm), MAX(vndr_id), MAX(unspsc_nbr),MAX(iss_account)   FROM
    ( SELECT '' AS item_nbr, manuf_item_nbr,'' AS description, '' AS description2,'' AS GHX_FULL_ITEM_DESCR, Cntrct_nbr_txt, uom_cd, CAST(item_qty AS VARCHAR (255)) AS item_qty,tier_descr, CAST ( tier_prc_amt AS VARCHAR (255)) AS tier_prc_amt, CAST (list_prc_amt AS VARCHAR (255)) AS list_prc_amt,
    vndr_nm, '' AS vndr_id, '' AS unspsc_nbr,'' AS iss_account FROM ROI.CNTRCT_PRC_LIST
    WHERE ACTN_CD <> 'D'
    AND ROW_UPDT_TSP IS NULL 
    UNION ALL
    SELECT item_nbr, manuf_item_nbr,'', '', GHX_FULL_ITEM_DESCR,'',  purch_uom_txt  AS uom_cd,
      purch_qoe_txt  AS item_qty, '',  '' AS tier_prc_amt,'' AS list_prc_amt,
    vndr_nm, vndr_id, unspsc_nbr,
      gl_cd  AS iss_account
      FROM ROI.ROI_ITEM_ENRCHD_NUVIA
       UNION ALL
    SELECT  trim(item)  AS item_nbr,
       trim(manuf_nbr)  AS manuf_item_nbr,
       trim(description),
       trim(description2), '' AS GHX_FULL_ITEM_DESCR, '',
        trim(stock_uom ) AS uom_cd,
        ''  AS item_qty,'', '','' AS tier_prc_amt, '' AS list_prc_amt,'' AS vndr_id, '' AS unspsc_nbr,
        CAST( trim(iss_account) AS VARCHAR(255))
          FROM ITEMMAST_LAW
    )GROUP BY manuf_item_nbr
       ORDER BY manuf_item_nbr

  • Problem with macro expansion removing ? characters from macros

    Hi,
    When compiling the following code (CC ex.cpp), we always receive the error line 28: Error: Unexpected ":" found.
    ex.cpp
    #include <stdio.h>
    #include <stdarg.h>
    #include <string.h>
    #define ONE 1
    #define MCheck(n) (n == ONE ? "One" : "???")
    #define F1(buff, size, ...) func(buff, size, __VA_ARGS__)
    #define F(...) F1(acBuff, sizeof(acBuff), __VA_ARGS__)
    void func(char* buff, size_t len, const char* fmt, ...) {
        va_list args;
        va_start(args, fmt);
        size_t nWrote = vsnprintf(buff, len, fmt, args);
        va_end(args);
    int main(void) {
        char acBuff[10];
        // Parsed OK
        F1(acBuff, sizeof(acBuff), "%s\n", MCheck(1));
        printf("%s", acBuff);
        // Causes error
        F("%s\n", MCheck(1));
        printf("%s", acBuff);
        return 0;
    }This appears to be because the pre-processed code (found using CC -E ex.cpp) has removed the '?' characters from the parsed macro 'F' expansion as shown:
    (Code inserted from headers)
    #13 "ex.cpp"
    void func ( char * buff , size_t len , const char * fmt , ... ) {
    va_list args ;
    ( void ) ( args = ( __va_list ) & __builtin_va_alist ) ;
    size_t nWrote = vsnprintf ( buff , len , fmt , args ) ;
    ( void ) 0 ;
    int main ( void ) {
    char acBuff [ 10 ] ;
    func ( acBuff , sizeof ( acBuff ) , "%s\n" , ( 1 == 1 ? "One" : "???" ) ) ;
    printf ( "%s" , acBuff ) ;
    func ( acBuff , sizeof ( acBuff ) , "%s\n" , ( 1 == 1 "One" : "???" ) ) ;
    printf ( "%s" , acBuff ) ;
    return 0 ;
    }Is this an error with the compiler, or is there some flag we need to set to ensure the macros are expanded to the correct level?

    Bwadly wrote:
    Hi Maxim,
    Thanks for the feedback.
    We are using Solaris 8, so I'm guessing we should use Sun Studio 11? Correct, the latest release that supports Solaris 8 is Sun Studio 11. Be sure to patch both Solaris and Sun Studio once you install it.
    BTW, I was able to reproduce the problem using Studio 9 compiler (CC 5.6). Apparently, it got fixed in later versions of compiler - patched version of CC 5.6 does not have it.
    Edited by: MaximKartashev on 15.08.2008 12:55

  • Remove special characters from incoming data

    Hi Varun, You could use either of below.. REG_REPLACE(YOUR_INPUT_STR,^[A-Za-z0-9 ],NULL)  -- Replaces all non-alphanumeric with null
       REG_EXTRACT(YOUR_INPUT_STR,[A-Za-z0-9 ]) -- Extracts only alphanumeric data -Rajani

    i have special character coming in the source data and i want to remove it before loading into target, currently i am getting one special character , it may come as some other type of special character  other than alpha numeric. so how to remove those special characters from data and load the alphanumeric data into target.

  • Excel - Remove specific characters from entire worksheet if they are the first 2 in the cell

    Hello,
    I have a very large worksheet that is used to populate a SQL database.
    I need to have the data clean before I pass it on to the owner of that database.
    The data is riddled with "na" where the supplier of the data sometimes leaves it blank, like they should, but instead sometimes puts in "na" as either "not applicable" or "information not available". 
    I just need the all "na" removed.  If I highlight the entire sheet and replace "na" with nothing then where it says "international" it will say "inertional" and I can't have that.
    I need a routine, with an on-click for a button, probably, that removes all of the "na" that comes first in the cell.
    I've seen where you can do it with individual cells, but not looping through the whole spreadsheet.
    Thank you in advance for any help I can get! 
    Gee

    >>then where it says "international" it will say "inertional"
    Selecting the "match entire cell contents" option may give you what you want.

  • Removing unwanted characters from imported string

    Hello,
    I have a tab-delimited .txt file which I have to import into Indesign for further processing.
    The file is composed by a 3 columns header row at the beginning (Code, Description, price) followed by a series of 3 columns data rows.
    The problem is that sometimes, depending on the way the txt/csv file has been created, may include unwanted characters (such as spaces, double spaces, etc.).
    Is there a way to "clean" the imported strings from these unwanted characters?
    This is my starting code:
    function processImportedTxt(){
        //Open .csv file
        var csvFile = File.openDialog("Open file .csv","tab-delimited(*.csv):*.csv;");
        datafile = new File(csvFile);
        if (datafile.exists){
            datafile.open('r');
       var csvData = new Array();
       while(!datafile.eof){//read every row till the end of file
            csvData.push(datafile.readln());
        datafile.close();
        for(a=1; a<csvData.length; a++){
            var myRowData = csvData[a];//row of data
            var mySplitData = myRowData.toString().split("\t");//divide columns
            var myRowCode = mySplitData[0];
            var myRowDesc = mySplitData[1];
            var myRowPrice = mySplitData[2];
            // Here goes code for cleaning strings from unwanted characters
    processImportedTxt();
    Any help would be much appreciated
    Thanks in advance

    Hi,
    If you want to safe 1-space occurences just a small correction:
    i.e.:
    var myRowCode = mySplitData[0].replace(/\s\s+/g,'');
    Jarek

  • [Forum FAQ] How to remove div characters from multiline textbox field in SharePoint 2013

    Scenario:
    Need to avoid the div tags and get data alone from multiline textbox field using JavaScript Client Object Model in SharePoint 2013.
    Solution:
    We can use a regular expression to achieve it.
    The steps in detail as follows:
    1. Insert a Script Editor Web Part into the page.
    2. This is the complete code, add it into the Script Editor Web Part and save.
    <script type="text/javascript">
    ExecuteOrDelayUntilScriptLoaded(retrieveListItems, "sp.js");
    function retrieveListItems() {
    // Create an instance of the current context to return context information
    var clientContext = new SP.ClientContext.get_current();
    //Returns the list with the specified title from the collection
    var oList = clientContext.get_web().get_lists().getByTitle('CustomListName');
    //use CAML to query the top 10 items
    var camlQuery = new SP.CamlQuery();
    //Sets value that specifies the XML schema that defines the list view
    camlQuery.set_viewXml('<View><RowLimit>10</RowLimit></View>');
    //Returns a collection of items from the list based on the specified query
    this.collListItem = oList.getItems(camlQuery);
    clientContext.load(this.collListItem, 'Include(Title,MultipleText)');
    clientContext.executeQueryAsync(Function.createDelegate(this, this.onQuerySucceeded), Function.createDelegate(this, this.onQueryFailed));
    function onQuerySucceeded() {
    //Returns an enumerator to iterate through the collection
    var listItemEnumerator = this.collListItem.getEnumerator();
    //Remove div tag use a regular expression
    var reg1 = new RegExp("<div class=\"ExternalClass[0-9A-F]+\">[^<]*", "");
    var reg2 = new RegExp("</div>$", "");
    //Advances the enumerator to the next element of the collection
    while (listItemEnumerator.moveNext()) {
    //Gets the current element in the collection
    var oListItem = listItemEnumerator.get_current();
    alert(oListItem.get_item('MultipleText').replace(reg1, "").replace(reg2, ""));
    function onQueryFailed(sender, args) {
    alert('Request failed. ' + args.get_message() + '\n' + args.get_stackTrace());
    </script>
    Result:<o:p></o:p>
    References:
    http://www.w3schools.com/jsref/jsref_obj_regexp.asp
    Please click to vote if the post helps you. This can be beneficial to other community members reading the thread.

    Nice article :)
    If this helped you resolve your issue, please mark it Answered

  • How to remove the characters from first and last position of a string?

    Hi all,
    I am creating an application where i am using vb script to export the data to excel.
    In the excel sheet most of the values are entered with double " quotes.
    so the entry looks like ""http://www.sap.com"".
    Notice that it is starting with "" and not " . so the vb takes it as a line feed and shows an error .
    It will be great if anybody can tell me how to remove these set of quotes ,so that the value should come as "http://www.sap.com" . A sample code wil be of great help.
    Thanks in advance,
    mahima.

    Hi,
    To modify you excel you have 2 options, 1) you can filter them in excel itself by using filter ...etc in excel itself. or 2) you can do it by using ABAP code.
    Below code may help you.
    Get the data from Excel to Internal table using FM :
    call function 'ALSM_EXCEL_TO_INTERNAL_TABLE'
      exporting
        filename                = 'c:\temp\test.xls'
        i_begin_col             = 1
        i_begin_row             = 1
        i_end_col               = w_values
        i_end_row               = 10
      tables
        intern                  = t_alsmex_tabline
      exceptions
        inconsistent_parameters = 1
        upload_ole              = 2
        others                  = 3.
    Now replace all unwanted "" in internal table with space.
    replace all occurrences of '""' in itab-field with '   '.
    Again you can export to Excel using
      call function 'GUI_DOWNLOAD'
        exporting
          filename                = lv_file
          filetype                = 'DAT'
          append                  = ' '
          write_field_separator   = 'X'
        tables
          data_tab                = itab.

  • How can I remove special characters from spreadsheet filename?

    Hi
    Having a spreadsheet filename coded (on the Report Attributes page) as &VARIABLE..csv works fine unless the VARIABLE contains special characters such as space, lt, gt and so on.
    In the case of space, it is replaced by %20 in the generated filename. Is it possible convert the %20s to underscores (or remove them)?
    thanks
    Trevor

    Trevor,
    I would use a computation on that page to alter the value of variable, perhaps a SQL Expression such as:
      replace (:VARIABLE, ' ', '_')Alternatively, look at the process that puts the value into VARIABLE in the first place and ensure no spaces &gt;, etc are assigned.
    Sergio

  • Removing special characters from a string

    Hi All,
    I have this in one my column in my table
    ABC-DEF
    ABC/DEF
    ABC,DEF
    ABC DEF
    I want to remove -, /, comma and space so i want the result like this
    ABCDEF
    ABCDEF
    ABCDEF
    ABCDEF
    How can I do this in PL/sql.
    Thanks.

    Use REGEXP_REPLACE:
    with sample_table as (
                          select 'ABC-DEF' str from dual union all
                          select 'ABC/DEF' from dual union all
                          select 'ABC,DEF' from dual union all
                          select 'ABC DEF' from dual
    -- end of on-the-fly sample table
    select  str,
            regexp_replace(str,'[-/, ]') new_str
      from  sample_table
    STR     NEW_STR
    ABC-DEF ABCDEF
    ABC/DEF ABCDEF
    ABC,DEF ABCDEF
    ABC DEF ABCDEF
    SQL> SY.

  • Re: remove special characters from incoming data

    Dear Varun Kumar, Please use this logic. REG_REPLACE(INPUT FIELD,'[^A-Za-z0-9]' ,'')Only Characters and Numbers it will come. I Tried Same Logic it working. Please let me Know. Thanks & RegardsKasireddy+966545281845

    Hi Varun, You could use either of below.. REG_REPLACE(YOUR_INPUT_STR,^[A-Za-z0-9 ],NULL)  -- Replaces all non-alphanumeric with null
       REG_EXTRACT(YOUR_INPUT_STR,[A-Za-z0-9 ]) -- Extracts only alphanumeric data -Rajani

  • How do you removing special characters from a string

    hi,
    i'm a novice to java, if you couldn't tell. i need to write a program to recognise palindromes (eg. madam etc.) but i can't work out how to strip or clean the input string of white space or punctuation. any help appreciated. by the way, any other ideas on how to go about it?? (i don't expect it to be done, just after guidance)
    my thanks in advance

    OK Here you are:
    protected static String removeJunk(String string)
              int i, len = string.length();
              StringBuffer dest = new StringBuffer(len);
              char c;
              for (i = (len - 1); i >= 0; i--)
                   c = string.charAt(i);
                   if (Character.isLetterOrDigit(c))
                        dest.append(c);
              return dest.toString();
    Klint

  • Removing unwanted characters..

    Hey guys,
    I'm back for help again. Unfortunately my brain isn't creative enough, so please help! :-)
    Ok, I need to remove unwanted characters from a file...the problem is that the characters look like this:
    in any text editor. I'm using JEdit, and it's ISO-8859-1 encoding. The text was initally from a html file, and i think that most of the html is displayed well as text in JEdit. But these squares, which are bits of info that I don't need, are making it a little trick to do my extraction.
    Ex: the word I want to extract is "trouble". But in the file, it looks like this:
    troble....
    Anyone know how to get rid of all that stuff???
    Thanks in advance.
    ...DJVege...

    you could try to set a filter on the characters you accept. Process each character and only accept those that fall into some ASCII boundary. If you accept ASCII characters that have values between 33-255, most blocks should be eliminated.
    Something like this should help:
    import java.io.*;
    public class Example {
         protected static final int MIN_ASCII = 33;
         protected static final int MAX_ASCII = 255;
         public Example(String file) throws IOException {
              BufferedReader b = new BufferedReader(new InputStreamReader(new FileInputStream(file)));
              PrintStream p = new PrintStream(new FileOutputStream(file+"_fix.txt"));
              String s = "";
              int j;
              while ((s = b.readLine()) != null) {
                   for (j = 0;j < s.length();j++) {
                        if (valid(s.charAt(j))) {
                             p.print(s.charAt(j));
                   p.println();
              b.close();
              p.close();
         protected boolean valid(char c) {
              int asc = (int)c;
              // allow for tabs and spaces
              if (asc == 9 || asc == 32) {
                   return true;
              return (asc >= MIN_ASCII && asc <= MAX_ASCII);
         public static void main(String args[]) {
              if (args.length > 0) {
                   try {
                        new Example(args[0]);
                   catch (IOException e) {
                        e.printStackTrace();
    }useage: java Example <file>
    of course, something like this will probably only work on "english" files as I dont have an understanding on how foreign characters are encoded..
    if this doesnt solve your problem, you might want to adjust the range to 33-127, which will eliminate all "block" characters and all "special formatted" characters (i.e. accented characters, currency signs, etc.)
    see http://asciitable.com/ for more information on ASCII characters

Maybe you are looking for