Stop processing the duplicate record values

Question

Hi Experts,
   I have a requirement that i need do not process the duplicate values from the file place in FTP.
Scenerio is File->PI->RFC.
For ex:
Source Data:
      Name,Emp_id,DOB,Designation,Location,Joining_Date,Time_Stamp
      Moni,654654,11-09-1980,Developer,TN,20-02-2008,24-03-2014:3.38pm
      Shiva,654612,21-02-1982,Developer,TN,15-08-2009,24-03-2014:3.38pm
      Venkat,654655,19-01-1983,Developer,TN,28-10-2010,24-03-2014:3.38pm
      Moni,654654,11-09-1980,Developer,TN,20-02-2008,24-03-2014:9.38pm
If next time the same record comes like Moni,654654,11-09-1980,Developer,TN,20-02-2008,24-03-2014:9.38pm,no need to process this record values.
How to stop processing the duplicate record.Kindly share some ideas to achieve this requirement by using PI 7.1.
Best Regards,
Monikandan.

Answer 1

Hi ,
Here is one of the clean sol:
1)Using FCC read record by record by giving end separator as newline in FCC.
Reading a delimeter separated file whose columns may jumble
2)Use two mapping in one operation mapping .
1st Mapping :
Sourcefield()-->sort(ascending)-->splitbyvalue(valuechange)-->eachvalue-->collapsecontext-->udf-->
splitbyvalue(eachvalue)-->map to all target fields
Keep context of source field to top node to come all values in single array
for(int i=0;i<input.length;i++){
String temp[] = input[i].split(",");
if(temp.length == 7)
Name.addValue(temp[0]) ;
Emp_id.addValue(temp[1]) ;
DOB.addValue(temp[2]) ;
Designation.addValue(temp[3]) ;
Location.addValue(temp[4]) ;
Joining_Date.addValue(temp[5]) ;
Time_Stamp.addValue(temp[6]) ;
else{throw new  StreamTransformationException("field missing in col "+i)}// up to you
Mapping 2:Actual file structure to RFC
Regards
Venkat