Square boxes in file content

Question

Hi,
We have got a text file containing 'square boxes' in the field 'Item_description'.
The file content has to be passed to BW.
How can we eliminate these wild characters from being passed into BW?
First option is to omit them while reading the file. Here can we use the file encoding option? If yes, please confirm the correct way to do that.
The second option is to eliminate that in mapping. But I wonder how we can identify a 'square box' and then replace them.
Kindly help.
Thanks,
John

Answer 1

It might be too late, but here is a java code that removes all bad characters from the payload. Should be imported in PI and added to Operation Mapping before the Mesage Mapping:
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import com.sap.aii.mapping.api.AbstractTransformation;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.sap.aii.mapping.api.TransformationInput;
import com.sap.aii.mapping.api.TransformationOutput;
public class RemoveBadCharacters extends AbstractTransformation {
     public void transform(TransformationInput arg0, TransformationOutput arg1)
               throws StreamTransformationException {
          getTrace().addInfo("JAVA program RemoveBadCharacters is Initiated");
          String inputPayload = convertInputStreamToString(arg0.getInputPayload().getInputStream());
          String outputPayload = cleanUp(inputPayload);
          try {
               arg1.getOutputPayload().getOutputStream().write(
                         outputPayload.getBytes("UTF-8"));
          } catch (Exception exception1) {
     public String convertInputStreamToString(InputStream in) {
          StringBuffer sb = new StringBuffer();
          try {
               InputStreamReader isr = new InputStreamReader(in);
               Reader reader = new BufferedReader(isr);
               int ch;
               while ((ch = in.read()) > -1) {
                    sb.append((char) ch);
               reader.close();
          } catch (Exception exception) {
          return sb.toString();
     public String cleanUp(String instructions){
          String res = "";
          String c = "";
          int hexCode = 0;
          if (instructions != null && instructions.length()>0){     
               for (int i=0; i<instructions.length(); i++){
                    c = instructions.substring(i, i+1);
                    hexCode = c.hashCode();
                    if (hexCode>31 && hexCode<127 && hexCode != 96){
                         res = res.concat(c);
          return res;