Open Source API for Flat Files?

I did some searches on SourceForge for a flat-file serializer and deserializer. I know flat files are simple, but I did want to find something that allowed schema-like validations to be declared and enforced. The projects I saw were either abandoned or too simple. Does anyone know of a good open source flat file API? Thanks!
- Saish

Ok. That one is nice, especially in terms of defining header and trailer records. I also found this one http://sourceforge.net/projects/flatworm/ which allows for multi-line flat-files, which is also nice.
The problem is both only deal with parsing. Granted, writing a serializer even via reflection would not be all that tough. But a few Dukes for you!
- Saish

Similar Messages

Which is the best Open source API for PGP??

Hi All,
Please tell me
which is the best Open source API for PGP??
Please Help me thi is urgent!!
regards
harish

I am also looking at these topic at the moment.
I found Bouncycastle and hushSDK.
Have you found a good one and with example also?

How to create source system for flat file loads

How do I create a source system to load flat file?
I have a screen that asks for the following:
Logical system name
Source system name
Type and release
What should I enter for these?
I am not Basis and Basis was supposed to set this up.

hi sam.
STEPS TO CREATE FLAT FILE SOURCE SYSTEM
step:1 select source systems under modeling in AWB.
step:2 select Source Systems root node.
           |---> context menu
               |--->create......
step:3 select your required source system Icon.
          (in your case it is PC Icon).
there it askes for a logical.......and source system name...
here u can specify any name of ur wish
for example:-
logical sys name --- PC_FF
source system name --- flat file source system*
*press Continue * button.
Observe the activation icon to confirm the successfull creation(_glowing lamp icon_ symbol)

Source System for Flat Flie in QA?

Hi Expert!
I have defined a new source system for flat files in QA environment,
and mapped this in 'conversion of source system names after transport'
but the loads do not recognize this new SS...
Error Msg "Source does not exist"
What can I do?
Thank you in advance!

Please try to do export DataSource and see if you find it in QA.
Thanks
Arjun

Omit the Open Hub control file 'S_*' for flat file extracts

Hi Folks,
a quick question is it somehow possible to omit the control file generation for flat file extracts.
We got some unix scripts running that get confused by the S_* files and we where wondering if we can switch the creation of those files off.
Thanks and best regards,
Axel

Hi,
However, the S_ (structure) file does not reflect the proper structure. The S_ file lists all fields sorted in alphabetical order based on the field name, instead of listing all fields in the correct order that they appear in the output file.
As I know and checked this is not the case. The S_ file has the fields in the order of infospoke object sequence ( and in transformation tab, source structure).
I would suggest you to control it again.
Derya

Golden Gate for flat file

hi,
I have tried with GoldenGate for Oracle/ non-Oracle databases. Now, I am trying for flat file.
What i have done so far:
1. I have downloaded Oracle "GoldenGate Application Adapters 11.1.1.0.0 for JMS and Flat File Media Pack"
2. Kept it on the same machine where Database and GG manager process exists. Port for GG mgr process 7809, flat file 7816
3. Following doc GG flat file administrators guide Page 9 --> configuration
4. Extract process on GG manager process_
edit params FFE711*
extract ffe711
userid ggs@bidb, password ggs12345
discardfile E:\GoldenGate11gMediaPack\V26071-01\dirrpt\EXTFF.dsc, purge
rmthost 10.180.182.77, mgrport 7816
rmtfile E:\GoldenGate11gMediaPack\V26071-01\dirdat\ffremote, purge, megabytes 5
add extract FFE711, EXTTRAILSOURCE ./dirdat/oo*
add rmttrail ./dirdat/pp, extract FFE711, megabytes 20*
start extract FFE711*
view report ffe711*
Oracle GoldenGate Capture for Oracle
Version 11.1.1.1 OGGCORE_11.1.1_PLATFORMS_110421.2040
Windows (optimized), Oracle 11g on Apr 22 2011 03:28:23
Copyright (C) 1995, 2011, Oracle and/or its affiliates. All rights reserved.
Starting at 2011-11-07 18:24:19
Operating System Version:
Microsoft Windows XP Professional, on x86
Version 5.1 (Build 2600: Service Pack 2)
Process id: 4628
Description:
** Running with the following parameters **
extract ffe711
userid ggs@bidb, password ********
discardfile E:\GoldenGate11gMediaPack\V26071-01\dirrpt\EXTFF.dsc, purge
rmthost 10.180.182.77, mgrport 7816
rmtfile E:\GoldenGate11gMediaPack\V26071-01\dirdat\ffremote, purge, megabytes 5
CACHEMGR virtual memory values (may have been adjusted)
CACHEBUFFERSIZE: 64K
CACHESIZE: 1G
CACHEBUFFERSIZE (soft max): 4M
CACHEPAGEOUTSIZE (normal): 4M
PROCESS VM AVAIL FROM OS (min): 1.77G
CACHESIZEMAX (strict force to disk): 1.57G
Database Version:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
PL/SQL Release 11.1.0.7.0 - Production
CORE 11.1.0.7.0 Production
TNS for 32-bit Windows: Version 11.1.0.7.0 - Production
NLSRTL Version 11.1.0.7.0 - Production
Database Language and Character Set:
NLS_LANG environment variable specified has invalid format, default value will b
e used.
NLS_LANG environment variable not set, using default value AMERICAN_AMERICA.US7A
SCII.
NLS_LANGUAGE = "AMERICAN"
NLS_TERRITORY = "AMERICA"
NLS_CHARACTERSET = "AL32UTF8"
Warning: your NLS_LANG setting does not match database server language setting.
Please refer to user manual for more information.
2011-11-07 18:24:25 INFO OGG-01226 Socket buffer size set to 27985 (flush s
ize 27985).
2011-11-07 18:24:25 INFO OGG-01052 No recovery is required for target file
E:\GoldenGate11gMediaPack\V26071-01\dirdat\ffremote, at RBA 0 (file not opened).
2011-11-07 18:24:25 INFO OGG-01478 Output file E:\GoldenGate11gMediaPack\V2
6071-01\dirdat\ffremote is using format RELEASE 10.4/11.1.
** Run Time Messages **
5. on Flat file GGSCI prompt-->_*
edit params FFR711*
extract ffr711
CUSEREXIT E:\GoldenGate11gMediaPack\GGFlatFile\V22262-01\flatfilewriter.dll CUSEREXIT passthru includeupdatebefores, params "E:\GoldenGate11gMediaPack\GGFlatFile\V22262-01\sample-dirprm\ffwriter.properties"
SOURCEDEFS E:\GoldenGate11gMediaPack\V26071-01\dirdef\vikstkFF.def
table ggs.vikstk;
add extract ffr711, exttrailsource ./dirdat/pp*
start extract ffr711*
view report ffr711*
Oracle GoldenGate Capture
Version 11.1.1.0.0 Build 078
Windows (optimized), Generic on Jul 28 2010 19:05:07
Copyright (C) 1995, 2010, Oracle and/or its affiliates. All rights reserved.
Starting at 2011-11-07 18:21:31
Operating System Version:
Microsoft Windows XP Professional, on x86
Version 5.1 (Build 2600: Service Pack 2)
Process id: 5008
Description:
** Running with the following parameters **
extract ffr711
CUSEREXIT E:\GoldenGate11gMediaPack\GGFlatFile\V22262-01\flatfilewriter.dll CUSE
REXIT passthru includeupdatebefores, params "E:\GoldenGate11gMediaPack\GGFlatFil
e\V22262-01\sample-dirprm\ffwriter.properties"
E:\GoldenGate11gMediaPack\GGFlatFile\V22262-01\ggs_Windows_x86_Generic_32bit_v11
_1_1_0_0_078\extract.exe running with user exit library E:\GoldenGate11gMediaPac
k\GGFlatFile\V22262-01\flatfilewriter.dll, compatiblity level (2) is current.
SOURCEDEFS E:\GoldenGate11gMediaPack\V26071-01\dirdef\vikstkFF.def
table ggs.vikstk;
CACHEMGR virtual memory values (may have been adjusted)
CACHEBUFFERSIZE: 64K
CACHESIZE: 1G
CACHEBUFFERSIZE (soft max): 4M
CACHEPAGEOUTSIZE (normal): 4M
PROCESS VM AVAIL FROM OS (min): 1.87G
CACHESIZEMAX (strict force to disk): 1.64G
Started Oracle GoldenGate for Flat File
Version 11.1.1.0.0
** Run Time Messages **
Problem I am facing_
I am not sure where to find the generated flat file,
even the reports are showing there is no data at manager process
I am expecting replicat instead of extract at Flatfile FFR711.prm
I have done this much what to do give me some pointers.....
Thanks,
Vikas

Ok, I haven't run your example, but here are some suggestions.
Vikas Panwar wrote:
extract ffe711
userid ggs@bidb, password ggs12345
discardfile E:\GoldenGate11gMediaPack\V26071-01\dirrpt\EXTFF.dsc, purge
rmthost 10.180.182.77, mgrport 7816
rmtfile E:\GoldenGate11gMediaPack\V26071-01\dirdat\ffremote, purge, megabytes 5
ggsci> add extract FFE711, EXTTRAILSOURCE ./dirdat/oo
ggsci> add rmttrail ./dirdat/pp, extract FFE711, megabytes 20
ggsci> start extract FFE711
You of course need data captured from somewhere to test with. You could capture changes directly from a database and write those to a trail, and use that as a source for the flat-file writer; or, if you have existing trail data, you can just use that (I often test with old trails, with known data).
In your example, you are using a data pump that is doing nothing more than pumping trails to a remote host. That's fine, if that's what you want to do. (It's actually quite common in real implementations.) But if you want to actually capture changes from the database, then change "add extract ... extTrailSource" to be "add extract ... tranlog". I'll assume you want to use the simple data pump to send trail data to the remote host. And I will assume that some other database capture process is creating the trail dirdat/oo
Also... with your pump "FFE711", you can create either a local or remote trial, that's fine. But don't use a rmtfile (or extfile). You should create a trail, either a "rmttrail" or "exttrail". The flat-file adapter will read that (binary) trail, and generate text files. Trails automatically roll-over, the "extfile/rmtfile" do not (but they do have the same internal GG binary log format). (You can use a 'maxfiles' to force them to rollover, but that's beside the point.)
Also, <ul>
<li> don't forget your "table" statements... or else no data will be processed!! You can wildcard tables, but not schemata.
<li> there is no reason that anything would be discarded in a pump.
<li> although a matter of choice, I don't see why people use absolute paths for reports and discard files. Full paths to data and def files make sense if they are on the SAN/NAS, but then I'd use symlinks from dirdat to the storage directory (on Unix/Linux)
<li> both windows and unix can use forward "/" slashes. Makes examples platform-independent (another reason for relative paths)
<li> your trails really should be much larger than 5MB for better performance (e.g,. 100MB)
<li> you probably should use a source-defs file, intead of a dblogin for metadata. Trail data is by its very nature historical, and using "userid...password" in the prm file inherently gets metadata from "right now". The file-writer doesn't handle DDL changes automatically.
</ul>
So you should have something more like:
Vikas Panwar wrote:
extract ffe711
sourcedefs dirdef/vikstkFF.def
rmthost 10.180.182.77, mgrport 7816
rmttrail dirdat/ff, purge, megabytes 100
table myschema.*;
table myschema2.*;
table ggs.*;For the file-writer pump:
+5. on Flat file GGSCI prompt+
extract ffr711
CUSEREXIT flatfilewriter.dll CUSEREXIT passthru includeupdatebefores, params dirprm\ffwriter.properties
SOURCEDEFS dirdef/vikstkFF.def
table myschema.*;
table ggs.*;
ggsci> add extract ffr711, exttrailsource ./dirdat/pp
ggsci> start extract ffr711
Again, use relative paths when possible (the flatfilewriter.dll is expected to be found in the GG install directory). Put the ffwriter.properties file into dirprm, just as a best-practice. In this file, ffwriter.properties, is where you define your output directory and output files. Again, make sure you have a "table" statement in there for each schema in your trails.
Problem I am facing_
I am not sure where to find the generated flat file,
even the reports are showing there is no data at manager process
I am expecting replicat instead of extract at Flatfile FFR711.prm
I have done this much what to do give me some pointers.....The generated files are defined in the ffwriter.properties file. Search for "rootdir" property, e.g.,
goldengate.flatfilewriter.writers=csvwriter
csvwriter.files.formatstring=output_%d_%010n
csvwriter.files.data.rootdir=dirout
csvwriter.files.control.ext=_data.control
csvwriter.files.control.rootdir=dirout
...The main problem you have is: (1) use rmttrail, not rmtfile, and (2) don't forget the "table" statement, even in a pump.
Also, for the flat-file adapter, it does run in just a "extract" data pump; no "replicat" is ever used. The replicats inherently are tied to a target database; the file-writer doesn't have any database functionality.
Hope it helps,
-m

Open source projects for reading/wrting Excel, need your ideas

Hi all,
First of all, I am sorry that i didn't find the matched category under the listed forum topics.
I am evaluating some open source projects for processing(read/write) Excel files, so far i have found two: POI from apache and Java Excel API, however, i am wondering if there are any excellent projects that i missed, would anyone who have such experience please give me some ideas? Any better APIs?
Any helps are appreciated!
Thanks a lot,
thurm

hi macrules2,
thanks for you reply!
I think POI is not 'excellent' in some areas, for example, i found POI sometimes cannot read Date format collectly, and cannot support some kinds of formula either. So we want another more 'excellent' java-excel components.

What are the settings for datasource and infopackage for flat file loading

hI
Im trying to load the data from flat file to DSO . can anyone tel me what are the settings for datasource and infopackage for flat file loading .
pls let me know
regards
kumar

Loading of transaction data in BI 7.0:step by step guide on how to load data from a flatfile into the BI 7 system
Uploading of Transaction data
Log on to your SAP
Transaction code RSA1LEAD YOU TO MODELLING
1. Creation of Info Objects
In left panel select info object
Create info area
Create info object catalog ( characteristics & Key figures ) by right clicking the created info area
Create new characteristics and key figures under respective catalogs according to the project requirement
Create required info objects and Activate.
2. Creation of Data Source
In the left panel select data sources
Create application component(AC)
Right click AC and create datasource
Specify data source name, source system, and data type ( Transaction data )
In general tab give short, medium, and long description.
In extraction tab specify file path, header rows to be ignored, data format(csv) and data separator( , )
In proposal tab load example data and verify it.
In field tab you can you can give the technical name of info objects in the template and you not have to map during the transformation the server will automatically map accordingly. If you are not mapping in this field tab you have to manually map during the transformation in Info providers.
Activate data source and read preview data under preview tab.
Create info package by right clicking data source and in schedule tab click star to load data to PSA.( make sure to close the flat file during loading )
3. Creation of data targets
In left panel select info provider
Select created info area and right click to create ODS( Data store object ) or Cube.
Specify name fro the ODS or cube and click create
From the template window select the required characteristics and key figures and drag and drop it into the DATA FIELD and KEY FIELDS
Click Activate.
Right click on ODS or Cube and select create transformation.
In source of transformation , select object type( data source) and specify its name and source system Note: Source system will be a temporary folder or package into which data is getting stored
Activate created transformation
Create Data transfer process (DTP) by right clicking the master data attributes
In extraction tab specify extraction mode ( full)
In update tab specify error handling ( request green)
Activate DTP and in execute tab click execute button to load data in data targets.
4. Monitor
Right Click data targets and select manage and in contents tab select contents to view the loaded data. There are two tables in ODS new table and active table to load data from new table to active table you have to activate after selecting the loaded data . Alternatively monitor icon can be used.
Loading of master data in BI 7.0:
For Uploading of master data in BI 7.0
Log on to your SAP
Transaction code RSA1LEAD YOU TO MODELLING
1. Creation of Info Objects
In left panel select info object
Create info area
Create info object catalog ( characteristics & Key figures ) by right clicking the created info area
Create new characteristics and key figures under respective catalogs according to the project requirement
Create required info objects and Activate.
2. Creation of Data Source
In the left panel select data sources
Create application component(AC)
Right click AC and create datasource
Specify data source name, source system, and data type ( master data attributes, text, hierarchies)
In general tab give short, medium, and long description.
In extraction tab specify file path, header rows to be ignored, data format(csv) and data separator( , )
In proposal tab load example data and verify it.
In field tab you can you can give the technical name of info objects in the template and you not have to map during the transformation the server will automatically map accordingly. If you are not mapping in this field tab you have to manually map during the transformation in Info providers.
Activate data source and read preview data under preview tab.
Create info package by right clicking data source and in schedule tab click star to load data to PSA.( make sure to close the flat file during loading )
3. Creation of data targets
In left panel select info provider
Select created info area and right click to select Insert Characteristics as info provider
Select required info object ( Ex : Employee ID)
Under that info object select attributes
Right click on attributes and select create transformation.
In source of transformation , select object type( data source) and specify its name and source system Note: Source system will be a temporary folder or package into which data is getting stored
Activate created transformation
Create Data transfer process (DTP) by right clicking the master data attributes
In extraction tab specify extraction mode ( full)
In update tab specify error handling ( request green)
Activate DTP and in execute tab click execute button to load data in data targets.

When we go for Flat File creatin

Hi gurus,
When we go for flat file creation.Can any one give me one Scenario with example.
Thanks in advance

Hello Srikar,
How r u ?
We go for flat file in some scenarios like,
1. The source system could not be connected to the BW System (here u could generate a FlatFile)
2. If the Customer asks for a different kind of report (in our case we do FlatFile Creation for a complex time management - Employee Working Time is 1st merged with R/3 data, then we combine both together as 1 FlatFile)
3. Sometimes the customer will give the FlatFile data generated by them
4. We also take FlatFile reports from Data Targers, using InfoSpokes
More scenarios are there,
Best Regards....
Sankar Kumar
+91 98403 47141

Selection in IP for flat file Load

Hi Experts ,
I Want to know can I have selections available for an IP which is created for Flat file as data source.
your early response is highly appreciated.
Regards
Patil

Hi,
Yes You can. But at the data source level you have to tick the selection fields, so that at info pack level those fields will be available for selections.
Regards
Sankar

Delta Upload for Flat File

Hello Everyone
i am srikanth. i would like to know wheather do we have a facility for Delta Upload for flat file. If yes can u please give me the steps.
thanks in advance
srikanth

Hi Sabrina...thank you for ur help...i did load the data from cube to ods
steps.
1. i generated export data source on the cube
2. i found the name of the cube with prefix 8<infocube name> in the infosource under DM application component.
3. there are already communication structure and transfer rules activated but when i am creating update rules for the ods..i am getting the message 0recordmode missing in the infosource.
4. so i went to infosource and added 0recordmode in communication structure and activated but the transfer rules in yellow colour..there was no object assigned to 0recordmode but still i activated.
5.again i went to ods and created update rules and activated (tis time i didnt get any message about 0recordmode).
6.i created infopackage and loaded.
a)Now my question is without green signal in the transfer rule how data populated into the ods and in your answer you mentioned to create communication structure and transfer rules where in i didnt do anything.
b) will i b facing any problem if i keep loading the data into the ods from cube (in yellow signal) ..is it a correct procedure..plz correct me..thanks in advance

Oracle OWB integrator for Flat Files 3.0

Hi,
I have installed owb 11g r2 in my machine, but while creating flat files module for file import it is showing that
it does not find *"Oracle OWB integrator for Flat Files 3.0"*. so i cannot use "import metadata using flat file wizard".
do i need to install the owb again or the Oracle OWB integrator for Flat Files 3.0 alone..or how do i integrate it with owb....... please help if anyone can,,
Thanks..

Here is the certification matrix
https://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0CFEQFjAE&url=http%3A%2F%2Fwww.oracle.com%2Ftechnetwork%2Fmiddleware%2Fdata-integration%2Fodi-11gr1certmatrix-ps6-1928216.xls&ei=mtmUUcX7DoiJrQek04DwAg&usg=AFQjCNGoOUFQHdK7Ti2DLb6vz_3s-miP3A&sig2=q3rf2foe9bl4_WbsLPwWng&bvm=bv.46471029,d.bmk

Open Source solution for Map Display on Web

Hi
Does anybody knows an Open Source solution for displaying Maps on Web. Something like the MapXtreme equivalent of Open Source.
Regards,
N�stor Bosc�n

There are all sorts of solutions for displaying maps on the Web, starting from JPG files and going up from there. Would you reject JPG because it isn't open source? Or did you really mean to ask for no-payment solutions?

An open source framework for agile development with Sites

I am happy to announce the release of a new open source framework for sites development: AgileSites 1.0.0.beta1.
AgileSites is an open source framework built using standard and documented Sites API provinding a number of features meant to simplify development, mostly to make agile development and offshoring way easier and more manageable.
Some of the features:
- MVC with plain Java controller and pure HTML views
- Templating done in jQuery style (like client-side javascript templating)
- Integrated unit testing and build system
- Integrated csdt so everything is source and can be stored in Git or Subversion and rebuilt with jenkins in minutes
- Hot reloading of java classes, including the url assembler - you code in java without restarting the application server
- Single jar deployment - all your site code is in a jar that can be easily tracked and moved around
- A simplified API layer built on top of standard tag assets making development a breeze
- Complete and unlimited access to the full Sites API
- Keeps the JSP structure so the framework can be added to another site using JSP
AgileSites is available in his own dedicated site: www.AgileSites.org with a video showing his capabilites.

The framework is completely following WebCenter Sites standard. It is built on top of JSP and Java using only documented API and standard Java libraries. It is not different than any other site implementation. It is basically a pre-built site to make easier applying agile development practices. It is as supported by Oracle as any other custom website implementation on top of Sites.

How IE works for flat file

Hi all:
As we all know that, when IE gets a idoc's service name from SLD, then use it with idoc's message type and Idoc type to do receiver determniation, what about flat file ? how can we know its Service name and interface name if there is only a flat file on FTP? how IE works for Flat file ?
Couldn't thank you more

Hi,
For any idoc scenarious, you would use business systems rather than business service which is stored in SLD. So the IE would fetch it from SLD at runtime.
For file based scenarious also, you can create business system as type third party and use the same.
Is that answer your question?
Regards
Krish

Open Source API for Flat Files?

Similar Messages

Maybe you are looking for