Flat File Validation

Hi
I have a scenario where i get a flat file from a external vendor every month which has customer #
as one of the fields.
Vendor usually updates his customers information using a clients 3rd party software tool.( basically this tool assigns a unique customer number and updates the BI customer master).
Now my issue is I have to validate the customer #'s in the Flat file against customer master in BI. If a particular Customer in Flat file do not exist in BI customer master(when Vendor forgets to update the BI customer master) I should have a log of the non existing customer #'s so that I can follow up with the Vendor to update mater data using the tool for the customers in question.
Please share your ideas in validating the flat file.

Hi Sreenivas/ Venkatesh
Thanks for the response.
Well Sreenivas; I am dealing with 40 of such flat files and trying to lay out a process to handle them efficiently so that maintenance can be mininmized in the future.
Can you please give more insight on handling this situation technically.
This is the process I am thinking of from the discussion.
1.Feed the flat files to a custome Flat File Validation program(transaction) in BI
2. Split the file in to Error file and Confirmation file.
3. Load the Confirmation file in to BI
4. Send the Error file to Vendor for correction and load the corrected file sent from the vendor.
Can we make this process more efficient especially handling files more in number.
Venkatesh: your suggestion is also feasible but I am guessing it needs more effort as we are actually loading and trying to find the errors. what is your take on this?
please share your ideas.

Similar Messages

Error while validating flat file schema

Hi,
I have the flat file schema which is character delimited on '|'. It has Details and Tralier. Trailer starts with a "Tag Identifier" as 'T' and can have one value as "Total Records" i.e. T|3. Details is also delimited on the base of '|' and but has no "Tag Identifier".
Details have min occurs as 0.
Schema:
Eg of the valid values:
21788367|EN
21848269|EN
22004225|EN
T|3
This should also be the valid value (with no details and only trailer):
T|0
But, when I'm trying to validate this. I'm getting error:
Unexpected data found while looking for:
'\r\n'
The current definition being parsed is User_details. The stream offset where the error occurred is 2. The line number where the error occurred is 1. The column where the error occurred is 2.
I think it is trying to find Details record also. What setting I need to change?
Please help.
Kunal G

This is my schema:
<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://BTS_POC.Users" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://BTS_POC.Users" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation>
<xs:appinfo>
<schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
<b:schemaInfo standard="Flat File" codepage="65001" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" root_reference="Users_Root" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Users_Root">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xD 0xA" child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence minOccurs="0">
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element minOccurs="0" maxOccurs="unbounded" name="Users_Detail">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="char" child_delimiter="|" child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Name" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="1" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="Id" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="2" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="Users_Trailer">
<xs:annotation>
<xs:appinfo>
<b:recordInfo tag_name="T" structure="delimited" child_delimiter_type="char" child_delimiter="|" child_order="prefix" sequence_number="2" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="TotalNumber" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="1" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
Kunal G

Flat file connection: The file name "\server\share\path\file.txt" specified in the connection was not valid

I'm trying to execute a SSIS package via SQL agent with a flat file source - however it fails with Code: 0xC001401E The file name "\server\share\path\file.txt" specified in the connection was not valid.
It appears that the problem is with the rights of the user that's running the package (it's a proxy account). If I use a higher-privelege account (domain admin) to run the package it completes successfully. But this is not a long-term solution, and I can't
see a reason why the user doesn't have rights to the file. The effective permissions of the file and parent folder both give the user full control. The user has full control over the share as well. The user can access the file (copy, etc) outside the SSIS
package.
Running the package manually via DTExec gives me the same error - I've tried 32 and 64bit versions with the same result. But running as a domain admin works correctly every time.
I feel like I've been beating my head against a brick wall on this one... Is there some sort of magic permissions, file or otherwise, that are required to use a flat file target in an SSIS package?

Hi Rossco150,
I have tried to reproduce the issue in my test environment (Windows Server 2012 R2 + SQL Server 2008 R2), however, everything goes well with the permission settings as you mentioned. In my test, the permissions of the folders are set as follows:
\\ServerName\Temp --- Read
\\ServerName\Temp\Source --- No access
\\ServerName\Temp\Source\Flat Files --- Full control
I suspect that your permission settings on the folders are not absolutely as you said above. Could you double check the permission settings on each level of the folder hierarchy? In addition, check the “Execute as user” information from job history to make
sure the job was running in the proxy security context indeed. Which version of SSIS are you using? If possible, I suggest that you install the latest Service Pack for you SQL Server or even install the latest CU patch.
Regards,
Mike Yin
If you have any feedback on our support, please click
here
Mike Yin
TechNet Community Support

Validation Event Handler Not working with Flat File GTC Trusted Recon

We are created Event Handler for checking special characters in Telephone field.Its working fine when the user is created through Admin Console.but the event handler is not triggering while doing GTC Flat File Trusted recon.
Version: OIM 11.1.1.5.0
Can someone please help me out with this.
Thanks
Edited by: 790561 on 17-Feb-2013 09:01
Edited by: 790561 on Feb 17, 2013 9:35 PM
Edited by: 790561 on Feb 18, 2013 12:38 AM

Validation Event handlers will not work with your trusted recon. You can use the GTC Validation provider, it is nothing but the plugins which you can insert with in your source field in GTC. I think you can easily get the steps for how to create a custom GTC Providers.
Edited by: iam37 on Feb 16, 2013 4:09 PM

Help Required regding: Validation on Data Loading from Flat File

Hi Experts,
I need u r help in the following issue.
I need to validated the transactional data loading to the GL Cube from Flat file,
1) The transactional data to the Cube to be loaded only if master data record exists for the 0GL_ACCOUNT info object.
2) If the master data record does not exits then the record need to be skipped from the loading and after the loading the system should throw a message saying that these many records have been skipped (if there are any skipped records.).
I would really appriciate u r help and suggestions on solving this issue.
Regds
Hari

Hi, write a start routine in transfer rules like this.
DATA: l_s_datapak_line type TRANSFER_STRUCTURE,
 l_s_errorlog TYPE rssm_s_errorlog_int,
 l_s_glaccount type /BI0/PGLACCOUNT,
 new_datapak type tab_transtru.
 refresh new_datapak.
 loop at datapak into l_s_datapak_line.
 select single * from /BI0/PGLACCOUNT into l_s_glaccount
 where CHRT_ACCTS eq l_s_datapak_line-field name in transfer structure/datsource for CHRT_ACCTS
and GL_ACCOUNT eq l_s_datapak_line-field name in transfer structure/datsource for GL_ACCOUNT
and OBJVERS eq 'A'.
 if sy-subrc eq 0.
 append l_s_datapak_line to new_datapak.
 endif.
 endloop.
 datapak = new_datapak.
 if datapak[] is initial.
abort <> 0 means skip whole data package !!!
 ABORT = 4.
 else.
 ABORT = 0.
 endif.
i have already some modifications but U can slightly change it to suit your need.
regards
Emil

BDC (Flat File Data Validation) - Code

I am trying to validate flat file data BEFORE performing BDC (Call Trans. or Session)..
Pls help me out in below code for xk02..
DATA: BEGIN OF itab occurs 0, "ITAB having flat file data.
      lifnr(16) ,
      bukrs(4),
      ekorg(4),
      END OF itab.
DATA: BEGIN OF int_final occurs 0,
      lifnr(16) ,
      bukrs(4),
      ekorg(4),
      status(6),
      message(6),
      END OF int_final.
DATA: int_final TYPE TABLE OF int_final.
DATA: wa_itab TYPE TABLE OF itab.
DATA: validate_itab TYPE TABLE OF itab. "VALIDATE_ITAB having master data.
DATA: wa_validate_itab TYPE TABLE OF itab.
FORM data_validation .
SELECT LFB1LIFNR LFB1BUKRS LFM1~EKORG INTO TABLE validate_itab
         FROM LFB1 INNER JOIN LFM1 ON LFB1LIFNR = LFM1LIFNR.
IF sy-subrc = 0.
SORT validate_itab BY lifnr bukrs ekorg.
ENDIF.
LOOP AT itab INTO wa_itab.
READ TABLE validate_itab WITH KEY
lifnr = itab-lifnr
bukrs = itab-bukrs
ekorg = itab-ekorg
BINARY SEARCH.
IF sy-subrc NE 0.
PERFORM f_error_log USING text-005. "Invalid Value Set
CONTINUE.
ENDIF.
ENDLOOP.
ENDFORM.                    " data_validation
*&      Form f_error_log
FORM f_error_log USING l_message TYPE string.
CLEAR : fs_final.
fs_final-lifnr = itab-lifnr.
fs_final-bukrs = itab-bukrs.
fs_final-ekorg = itab-ekorg.
fs_final-status = text-014. "Error
fs_final-message = l_message.
APPEND fs_final TO int_final.
ENDFORM.                    " f_error_log
Thanks..

Hi GAurav,
I have a small question in th validation.
In LFM1~LIFNR does not contian any value how u r comparing both and one more thing After getting the data using GUI_upload u will get the data into validate_tab.
Loop at Vlidate_tab into wa_itab.
SELECT LFB1LIFNR LFB1BUKRS LFM1~EKORG INTO TABLE validate_itab
FROM LFB1 INNER JOIN LFM1 ON LFB1~LIFNR = wa_itab-lifnr.
endllop.
Thanks,

FIDCC1 Validations / Substitutions and flat files?

I am investigating the potential for two different types of functionality from using the FIDCC1 IDOC for transferring financial postings between two separate SAP FICO systems. My understanding at this point in time is that this IDOC won't perform any of custom or standard substitutions or validations of the receiving system. Is this correct? can anyone confirm? Also, I understand that some IDOC will generate a flat file on the receiving system in addition to the posting of data there. Will FIDCC1 do this as well?
My apologies if this information is readily available elsewhere. I need to obtain these answer quickly and find that I can usually get rapid answers from the SAP Development Community.
Thanks,
David

Ok. That one is nice, especially in terms of defining header and trailer records. I also found this one http://sourceforge.net/projects/flatworm/ which allows for multi-line flat-files, which is also nice.
The problem is both only deal with parsing. Granted, writing a serializer even via reflection would not be all that tough. But a few Dukes for you!
- Saish

Validation error using flat files..VLD-2407

Trying to build a simple flat file mapping to test out file locations and permissions at a new client and getting the following error: VLD-2407: SQL*Loader map must have a source file. OWB 10.2.0.3. I've added a filename for everything i could find in the mapping configuration screens. Any thoughts?
Thanks

It can happen if your location for deployment is different than that of development and you have forgotten to make sure the
source directory ane file exist in deployment location.
Regards
azaman

Flat File Records Validation

Hi All,
We have a requirement, every one hour we are getting a flat file. We are loading into a stagging tables , at the time of extraction we want to validate the count of records from source and count of records loaded into stagging tables. If the records count
not match delete the data from stagging tables.
How can we achieve the above one using C# Code.
Thanks in Advance....

Thanks Visakh for quick reply...
We want to use C# script here, it is our requirement.... How can we achieve the same....
Its straightforward
see
http://www.c-sharpcorner.com/uploadfile/muralidharan.d/how-to-use-rowcount-in-ssis/
Still I cant understand why you need to go for this when there is a standard way (unless this is an assignment!)
Please Mark This As Answer if it solved your issue
Please Vote This As Helpful if it helps to solve your issue
Visakh
My Wiki User Page
My MSDN Page
My Personal Blog
My Facebook Page

Non English Charaters in Flat file(Note pad)

Dear Friends,
I have a flat file from presentation server which needs to be uploaded and do some validations on that file based on the data existing.
I have some Russian language words in that file. E.g: "ОЩИ В ПОДГ". If this word exists in the file i need to get some other data from a database table. But when trying to check for the word, these characters are appearing as junk characters like ####&^^.
I tried logging into RU language also, but still not working. And also tried changing the file as type Unicode, ASCII and also not worked.
Please let me know how to get identify these words to see the exact russian words.
Thanks for your help.
Srinivas.

Hello,
Is your file .txt file.If yes,while saving it gives a pop up for file name.There you have option for encoding, there select encoding as
Unicode and try to upload.
It worked for me.I tried with the data provided by you.
data:begin of it_tab occurs 0,
 val type string,
 end of it_tab.
CALL FUNCTION 'GUI_UPLOAD'
EXPORTING
 filename = 'C:\Documents and Settings\xxx\Desktop\SDN1.txt'
FILETYPE = 'ASC'
HAS_FIELD_SEPARATOR = ' '
HEADER_LENGTH = 0
READ_BY_LINE = 'X'
DAT_MODE = ' '
CODEPAGE = ' '
IGNORE_CERR = ABAP_TRUE
REPLACEMENT = '#'
CHECK_BOM = ' '
VIRUS_SCAN_PROFILE = VIRUS_SCAN_PROFILE
NO_AUTH_CHECK = ' '
IMPORTING
FILELENGTH = FILELENGTH
HEADER = HEADER
TABLES
 data_tab = it_tab
EXCEPTIONS
 FILE_OPEN_ERROR = 1
 FILE_READ_ERROR = 2
 NO_BATCH = 3
 GUI_REFUSE_FILETRANSFER = 4
 INVALID_TYPE = 5
 NO_AUTHORITY = 6
 UNKNOWN_ERROR = 7
 BAD_DATA_FORMAT = 8
 HEADER_NOT_ALLOWED = 9
 SEPARATOR_NOT_ALLOWED = 10
 HEADER_TOO_LONG = 11
 UNKNOWN_DP_ERROR = 12
 ACCESS_DENIED = 13
 DP_OUT_OF_MEMORY = 14
 DISK_FULL = 15
 DP_TIMEOUT = 16
 OTHERS = 17
IF sy-subrc <> 0.
MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
 WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
ENDIF.
Thanks.
Ramya.

Error while uploading data from a flat file to the hierarchy

Hi guys,
after i upload data from a flat file to the hierarchy, i get a error message "Please select a valid info object" am loading data using PSA, having activated all external chars still get the problem..some help on this please..
regards
Sri

there is o relation of infoobject name in flat file and infoobjet name at BW side.
please check with the object in the BW and their lengths and type of the object and check your flat file weather u have the same type there,
now check the sequence of the objects in the transfer rules and activate them.
there u go.

APD - Query output on to a Flat file

All
We are on BI 7.0 with SP13 and NWS 12. I am trying to create a new APD with source as "Query" built on Info object and target as "Flat File". There is no requirement of transformation so I am trying to map directly from Source to Target with a direct link with out using any transformations in between,
I could check and found no errors in APD definition and when I run APD, the file is getting created both on application server and local workstation but with 0 records.
The volume of records are expected to be around 4 millions on this query and am just wondering why APD is not writing any data on CSV file.
Pls let me know the following;
- What is wrong in the above process and why it is not writing data on to flat file
- Can APD handle volume of 4 million rows even though job runs in the background?
Your inputs in this are highly helpful and thanks for your time to provide solution.
Thanks - Sriram

Hi
Thanks for your reply and I could solve this issue and found the root cause.
The very purpose of this APD is to provide a list of business partners to an upstream systerm with few attributes and in this process one of my consultant has deactivated the key figure in the query which is an input for APD process.
When the key figure is inactivated ( this is the only key figure in the query ), APD does not recognize query as a valid one and sends out an error message.
When we re activate the key figure ( I have hidden this later using APD transformations to hide the column and send output to a flat file on to unix server ) in the query, the APD runs fine and could deliver the output in flat file.
Thanks

Flat file to target table map in 11G

Hi,
I am havin many problem with my 10G R1 map that have been migrated to 11G.
all that my 10G map has is mapping from flat file to stage table. It also had one constant operator. i am getting error coz of this constant that i am setting up in the map.
when i validate this constant variable i get error mess:
API8534:Validation no supported for language SQLLOADER,property Expression for SAASD_MAP
-- Generator Version : 11.1.0.7.0
-- Created Date : Mon May 11 21:16:25 CDT 2009
-- Modified Date : Mon May 11 21:16:25 CDT 2009
-- Created By : OWB_WUSER
-- Modified By : OWB_WUSER
-- Generated Object Type : SQL*Loader Control File
-- Generated Object Name : "SADAD_MAP"
-- Copyright © 2000, 2007, Oracle. All rights reserved.
OPTIONS (BINDSIZE=50000,ERRORS=50,ROWS=200,READSIZE=65536)
LOAD DATA
CHARACTERSET WE8MSWIN1252
INFILE '{{ETL_FILE_LOC.RootPath}}{{}}dgp.dat'
BADFILE '{{ETL_FILE_LOC.RootPath}}{{}}dgp.bad'
DISCARDFILE '{{ETL_FILE_LOC.RootPath}}{{}}dgp.dis'
CONCATENATE 1
INTO TABLE "STAGING"."DGP_STAGG"
APPEND
REENABLE DISABLED_CONSTRAINTS
WHEN (1:2) = 'EM'
"PHONE_COUNTRY" CONSTANT ''Asia'',
"MRI" POSITION(1:2) CHAR(2) ,
if you note 11G some how does not allow me to have constant operator in flat file maps. and the constant for PHONE_COUNTRY column is enclosed in double quotes insted of single quotes.[should be 'Asia' and not ''Asia'']
can any one tell me why this is heppening?...i also have problem with the READ BUFFER property. which in 10G used to be defulted to '4' but ignored which generating the .ctl but in 11G this property creates READ BUFFERS statement in .ctl....any help in this will be grt.
Edited by: user591315 on May 12, 2009 7:39 AM

if you note 11G some how does not allow me to have constant operator in flat file maps. and the constant for >PHONE_COUNTRY column is enclosed in double quotes insted of single quotes.Hi
Please open the same mapping in 10g R2 its same .
It use Double code around the column name.
By the way execute the mapping and then check are you getting desired result.

VLD-6328 error : Use a flat file as Source in a mapping in OWB 10.2

Dear all,
I have created a simple mapping using a flat file in a remote server as a source and a simple table as a target.
When I try to validate the mapping I get the following warning:
"VLD-2398: No data file specified. Generated code will use the sampled file name and the file location xxx"
I use the version 10.2 and I cannot find "Sources" or "Targets" when I use the "Configure" options.
Any help will be appreciated

Hi all,
If you want to validate correctly this mapping you must to :
1. Right click on the mapping, then Configure
2. Right click on Sql Loader Data Files, then Create
3. Verify that the location name is correct and complete the Data File Name
4. OK
The validation is now OK.
I hope it will be help you
Best Regards
Samy

How to determine the size of a flat file attachment in a mail sender cc?

Hi
I use PI 7.11
I have a scenario where a flat file attachment is being picked up by a MailSender adapter and if the size of the attached flat file is larger that 500 bytes the receiver is A and if the attachment is less that 500 bytes the receiver is B.
I have checked the Context Object list in the Conditions section of Receiver Determination, and it seems, that only the file adapter allows for validation on the file size.
I contemplated an extended receiver determination but the attachment is a flat file which just needs to be passed thru without being converted to an XML document, so my source document is not of XML format.
An other but not very nice way would be to use an intermediate folder to drop the attachment in and from there use a file sender adapter and thus get access to the filesize attribute in my receiver determination, but I would like to avoid this.
Any suggestions?
BR MIkael

Hi
I have decided to make a module, where I plan to place the size of the attachment in the Dynamic Configuration variable "ProcessStep", but to my surprise, it seems that all the context objects in the receiver determination condition section are of type String and hence it is not possible to make a condition where I test for whether the value is smaller than say 500!? Only equal, NOT equal and LIKE are available (EX seems to be disabled as this probably only works in an xpath.
Would one way be to use the xpath expression like this ns1:Main\ProcessStep < 500 where ns1: http://sap.com/xi/XI/System?
MIkael

Flat File Validation

Similar Messages

Maybe you are looking for