Databse of 1888 Byte Per Record with 259 Fixed Fields Stumped CR?!?

My Database of 15 million records/rows at 1888 Byte Per Record with 259 Fixed Fields STUMPED CR?!
Executive Summary :
I have lots of data. There are 15 million of records in some of my database files. But I need to drill into the data and get counts for each field.
I need to know how many times each data field (row) is populated in a database file. For example, how many of my records have last name? how many have first name? date of birth? etc. I need a simple report made using SAP "Crystal Reports 2011" Workstation type edition that SUMS each data row into a single column output like csv.
My database sizes can range from 1,000 to 18 million records that are in fixed field asci format with 1888 bytes and 256 rows per record! The report needs to SUM each data field or, in the case of a data field with a minimum of twelve tables, SUM each value of a table.
My contracted CR programmer is an expert although she was new to data files this humongous. Can anyone here help us get CR to work on my data? Here was her conclusion, Crystal Reports Fail:
My file is 14.1 gb as well. However, when run through Crystal Reports, it
only returns 2,274,876 records. Therefore, I am going to assume that there
is a Record Limit for Text files of this. I searched through all the
registry settings and there is nothing that appears to be limiting it. I
went online for a while to see if there was any information on it, and just
like when I used to work there, there was nothing concrete. Business Objects
does not physically test this large of data retrieval, therefore they don't
know the exact record limits.
I'm not sure what to suggest, but I don't think Crystal Reports is going to
work for you and I wouldn't have known that from the beginning without
testing your data. I'm 99% sure there is a refund policy on the software, so
you shouldn't have any issues returning it.
Since I only specialize in Crystal Reports, I'm not sure what other software
can handle this amount of data. Possibly Cognos, but you may want to find
out from them or do some research.
If your data were in a database that can run stored procedures (like SQL
Server or Oracle) then you could write SQL to only return the counts. Then
when Crystal connected to the stored procedure, it should work, but I can't
say for sure without testing it. And I don't have SQL Server or Oracle
installed on my machine. I believe MYSQL (the free version) has a limit to
the number of records it can hold and I'm sure 8 million would be over the
limit. Also, I'm not sure if it supports stored procedures. I have used it
once before and found it extremely cumbersome and limited. I think SQL
Server and Oracle are quite pricey, so I don't think that this will even be
an option for you.

Hi Shawn,
I actually contacted the same person you did also. Summer was a Report Designer Resource here and knows her stuff and the Report Designer as much as our Developer who wrote it do.
As I suggested in my first reply is to get all that data into a real database, one that can run Stored Procedures to do all the crunching unfortunately is the only way I can see this ever working.
Windows 7 and 8 CPU's doesn't help, 64 bit may but the problem is CR Designer is a 32 bit application so it can't take advantage of the 64 bit memory addressing space.
What you could so is post your question to the forum:
[Business Decision Making with BI|Marketing Library;
This is a forum for managers or anyone who wants to have a non-technical discussion about making informed decisions with analytics tools. Ask questions and share your experiences on how to effectively utilize data to address real business problems. Experts from inside and outside SAP will be monitoring these conversations and offering their knowledge along with the broader community.
Someone in there may have another SAP product that might be able to load your flat file into a data tool that CR can report off of.
I still think a copy of MS SQL 2008 Express is your best and likely cheapest solution and then you can simply import the data into it which it has an Import Wizard that makes it relatively easy to do... Although you may have to break your file into pieces for it to be able to process also.
You need to be flexible on that data source, I doubt you'll find anything that can report off of that huge of text file.
Good luck
Don

Similar Messages

Records with a picklist field with empty values can't be seen from a report

Records with a picklist field with empty values can't be seen from a report. I've created a report that have a picklist as a column. When there is no values the record(s) associated to this picklist don't appear. Do you know what are the common causes for this event?
Regards
Arturo

Hi samrat chakraborty ,
go to SUIM > roles > roles by complexselection criteria>
under the selection according to authorization values give the object's form SU53 screen and press enter it will asks the values for given object pass the values as per the SU53 and execute it will give the list of roles.... capture the roles. The user gets access if you give the any one of the listed roles is assigned, assign the role as per your approval process.
Check with the below link for more assistence:
http://help.sap.com/saphelp_erp2004/helpdata/EN/71/8fba30840c6e4d90da3526971cc684/frameset.htm
Regards,
S.Manu.

Gl master record with authorization group field, how to download this data

Dear Experts
I would like to down load general ledger master record with the following details.
gl account
text
authorization group
i did not find a way to down load this information with the values in the authorization
group filed.
submitted for experts advise and help.
joyal
Moderator: Please, use standard SAP reports/tables and search before posting

Dear:
                   Please check
                   S_ALR_87012328 - G/L Account List
                   OB_GLACC13 - Descriptions
                   You can select Auth group from the selection in OB_GLACC13 - Descriptions and download the same.
                    Regards

BDC steps differ when inputing same record with different other field value

CODEDATE--
ETC...(there is of course other field)
11----01.10.2007   (do process A C D)
12----01.10.2007   (do process A C D)
12----01.11.2007   (Having same code field so do process A B C D)
13----01.11.2007   (do process A C D)
14----01.10.2007   (do process A C D)
15----01.11.2007   (do process A C D)
15----01.12.2007   (Having same code field so do process A B C D)
16----01.10.2007   (do process A C D)
hi i am doing a BDC batch my input table a bit like above.
first record will do process A C D
second record will do process A C D
thirt record will do process A B C D
second record will do process A C D
Any hint of how i can do that plzzzz . Urgent helppp needed

as per understanding,
if code = 11 and date = 01.10.2007 then u need process A C and D right.
define 4 subrountine with A B C D logics.
define one more subroutine with SR1 and SR2.
inside SR1 call 3 subroutines A C D
and SR2 call A B C D.
if code = 11 and date = 01.10.2007 then
call subroutine SR1,
if if code = 12 and date = 01.11.2007 then call Subroutine SR2.
if u have any issue, let me know the detailed requirement.

Inserting multiple records with many repeating fields

I trying to get my form to accept default values from previous entries on insert.
For instance a user has 10 records to enter into a table lets say it contains 8 fields. In these batches 3 of the 8 fields are always the same and the users don't want to repeat these 3 field entries for all 10 records. After the 1st insert they want these 3 fields defaulted to the previous entry's values.
Any ideas?

I'm using BC4J for development.
Also the records are going to be inserted one at a time.
So they will enter a record. Hit insert , add another...
and so on

Copy a record with URL - Address Fields

Hi All,
Is it possible when using a weblink to copy a record to copy the address fields also? I'm copying a contact record and my URL is as follows:
https://secure-ausomx###.crmondemand.com/OnDemand/user/DefContactInsert?
OMTGT=ContactEditForm&OMTHD=ContactNewNav&OMRET0=ContactHomePage&OCTYPE=&ContactEditForm.Primary Account Name=%%%Primary_Account_Name%%%&ContactEditForm.Account Id=%%%Account_Id%%%&ContactEditForm.Parent Alternate Address.Personal Country=%%%Personal_COUNTRY_Country%%%
This works and the contact country copies as expected. However when I try and add the other contact address fields which I believe are as follows it wont copy:
ContactEditForm.Parent Alternate Address.Personal Street Address
ContactEditForm.Parent Alternate Address.Personal Street Address 2
ContactEditForm.Parent Alternate Address.Personal Street Address 3
ContactEditForm.Parent Alternate Address.Personal City
ContactEditForm.Parent Alternate Address.Personal County
ContactEditForm.Parent Alternate Address.Personal State
ContactEditForm.Parent Primary Address.Primary Province
%%%Personal_ADDR_Address1%%%
%%%Personal_ADDR2_Address2%%%
%%%Personal_ADDR3_Address3%%%
%%%Personal_CITY_City%%%
%%%PERSONAL_LIST_COUNTY%%%
%%%PERSONAL_LIST_USSTATE%%%
%%%Personal_PROVINCE_Province%%%
Any Ideas?
Thanks

Thanks Venky,
We've been playing and this is possible, for example... In this scenario the user is on the Account Record and they wish to create a Contact for this account. The account and it's corresponding address details should be populated through into the new record, this will include the contact address and account address. You need to ensure the weblink is setup to open in the current window:
javascript:openQuickCreatePopup('https://secure-ausomxdsa.crmondemand.com/OnDemand/user/ContactQuickCreate?OMTHD=New&OMTGT=PopupNewForm&OCTYPE=&ophpd=2&OMCR0=%%%Row_Id%%%&OMCBO=Account&OCPRF=AlternateCountry:%%%Ship_To_COUNTRY_Country%%%|AlternateAddress1:%%%Ship_To_ADDR_Address1%%%|AlternateAddress2:%%%Ship_To_ADDR2_Address2%%%|AlternateAddress3:%%%Ship_To_ADDR3_Address3%%%|AlternateCity:%%%Ship_To_CITY_City%%%|AlternateZipCode:%%%SHIP_LIST_ZIP%%%|AlternateCounty:%%%SHIP_LIST_COUNTY%%%|AlternateStateProvince:%%%SHIP_LIST_USSTATE%%%|AlternateProvince:%%%Ship_To_PROVINCE_Province%%%',0,0)
The field names "AlternateCountry" etc come from the WSDL, the destination field names come from the HTML page source. Give it a try.. I will get around to documenting this as I have seen many different techniques and documents based on copying records, this seems to be the better of them all though.
Oli @ Innoveer

How to insert a record with date format field

I tried to execute the query of the below but it showed error and hence i did research and found out i can add a date syntax in front to format it properly.
INSERT INTO EMP (EmpNo, EmpName, Sex, DateOfBirth, Salary, DeptNo)
VALUES ('E001', 'Alex', 'M', '2006-11-08', '1500', 'D001');
When i executed the following query under SQL plus environment, it hang and gave no response. May i know how to solve this problem? Thank you
INSERT INTO EMP (EmpNo, EmpName, Sex, DateOfBirth, Salary, DeptNo)
VALUES ('E001', 'Alex', 'M', DATE '2006-11-08', '1500', 'D001');

Normally when we execute a query, it will show the result or response whether it is fail or success or what the error message is. But my situation is that it showed nothing and no response, and cannot execute new query, i cant do anything under that environment and i can only off it by using end task.

Export records with images - how to export the filename of the image?

Hi There,
When we export records from the data manager, the image field exports as a sequence number, not the image filename. Is there any way that i can change this? Alternatively, can i export image data from the image manager which might provide me this link of sequence number to image filename??
Thanks..

Hi Adam,
I agree that using MDM Data Manager, main table record with linked image field exports as a sequence number, not the image filename. But You can easily export image file name using MDM Syndicator functionality.
As per my understanding, Image Manager will not solve your purpose as i feel it is nothing but now a part of Data Manager. I mean if you select your Images Table as current table in your data manager not Main table, you will get the same look and feel of Image manager. So when you open current table as Images in Data Manager and go to file-->export-> you can export Image name but it will not let you know about the relationship with main table record. I feel same thing will happen when you use Image Manager. So use MDM Syndicator which you resolve your issue of exporting main table record with image file name not image sequence number. You can also export Original file name, Code etc using Syndicator functionality along with Image name and main table record.
Regards,
Mandeep Saini

How can I make a report of all records with the date in the last two weeks?

Hi!
I have records with a date field, I want to create a report that only includes records with a date which is within the last two weeks of the system date, is this possible? Surely this is a basic database function, I would have thought, but I can find no help about it and there's very little in Mac Help about doing anything other than simple stuff with reports.
I'm not a genius with databases, so help with any required formula or what have you would be great!

Hi Jonathan,
Here's a formula you can use with the Match function. 'Date' is the name of the date field, which must be a "Date" type field.
'Date'>NOW()-14
Match will select (highlight) all records matching the query. You'll then need to use Organize > Hide Unselected to show only the matches.
Unfortunately, I haven't found a way to make the NOW() function work in a Find request, and neither Match requests nor Hide Unselected can be included in a recorded search, either of which would reduce repeating the report to a one-button operation.
Regards,
Barry

Program to update info record with material master data

I am trying to determine if there is a standard SAP program that can be run as a batch file to update the fields in the info record with the same fields that are in the material master? ie: Planned delivery time, Tolerance, purchasing group etc.

Hi,
There is no standard report or transaction that can do this.
The data from these fields in the material master is brought in as a default to the info record when it is created. beyond this point the Info record is treated as being the correct data regardless of what happens to the material master data and so no updates occur in the standard system.
It would not be difficult to build an LSMW job to copy the data from the material master to the info records and this would be the recommendation if you want to do this.
Steve B

Creating email attachment with 255 char per record

Hi,
We want to generate an email with excel attachment through ABAP program. To do the same, we are trying to use FM 'SO_DOCUMENT_SEND_API1'.
It looks like it only allows 255 characters per record however we want include upto 700 chracters per record. Does anyone know a way out to overcome this problem?
Note - We do not want to wrap the text to next line.
Thank you

Hi:
I would advice to make the copy of this function module SO_DOCUMENT_SEND_API1 and change the packing_list-doc_size = '700'..
Hopefully it would be helpful to solve the problem.
Regards
Shashi

How to find no of bytes per character in a word - Urgent need

Hi,
I have some data in the database with chinese characters. I want to know how many bytes per each character in a word and get the total bytes.
Is there any built in Oracle function or anything where i can get no of bytes for each character or a word.
I am not able to find the solution anywhere.
Any help is appreciated.

Hi,
You can use LENGHTB function.
SQL>SELECT LENGTHB(dummy) FROM DUAL;
LENGTHB(DUMMY)
1
1 row selected.
SQL>Regards

There is already a record with duplicate customer/vendor reference number

Dear Expert,
Person "A" reject the Sale Order now Person "B" edit same SO and again send to "A",now second time "A" approval
"B" in aleart msg go to that approval,press "add" button got information (error)
"there is already a record with duplicate customer/vendor reference number"
what shud i do ?
and why in aleart message box two entry for same SO
1st is reject but show SO draft [approved]
2nd is approval ...
Thanks

hi,
U can change radio button option in per document tab of document settings for sales order,
Either change it to without warning / warning only When duplicated customer reference no. occurs.
Modify customer reference number by including dot at end.

Master data records with one key

Hello All,
I have scenario that I need to load the master data records with one key.
For ex--
Cluster            Dcode               version
5000                 DC1                  2008
5000                 DC1                  2008
5001                  DC1                  2009
Could you please let me know possible ways.
Thanks

In your case if Cluster is the key then the underlying attributes data will be overwritten when the same cluster key comes.
as per you example data finally it loads only two records in to the target, since first two records key is same.
Hope this helps.
Veerendra.

We have an iMac 24-inch Mid 2007 with maximum of 4G bytes of SDRAM, with OX X 10.9.4 Mavericks. It is routinely locking-up with the rainbow wheel. Especially with iTunes and Safari. Ideas?

We have an iMac 24-inch Mid 2007 with maximum of 4G bytes of SDRAM, with OX X 10.9.4 Mavericks. It is routinely locking-up with the rainbow wheel. Especially with iTunes and Safari.
I'm wondering it it that Mavericks simply uses more RA that the machine allows when also running application, but I don't see the RAM is being maxed out per se' when in Activity Monitor either. I've also ran the Disk Utility and have run scans wit Sophos, looking for malware, but have found no issues.
Any ideas? Any better Malware/Spyware applications?
Help!

Here is the report:
EtreCheck version: 1.9.14 (50)
Report generated August 18, 2014 at 6:05:39 PM PDT
Hardware Information: ?
iMac (24-inch Mid 2007) (Verified)
iMac - model: iMac7,1
1 2.8 GHz Intel Core 2 Duo CPU: 2 cores
4 GB RAM
Video Information: ?
ATI,RadeonHD2600 - VRAM: 256 MB
iMac 1920 x 1200
System Software: ?
OS X 10.9.4 (13E28) - Uptime: 1 day 4:11:27
Disk Information: ?
Hitachi HDS721010KLA330 disk0 : (1 TB)
S.M.A.R.T. Status: Verified
EFI (disk0s1) <not mounted>: 209.7 MB
Thomas iMac (disk0s2) / [Startup]: 999.35 GB (697.1 GB free)
Recovery HD (disk0s3) <not mounted>: 650 MB
USB Information: ?
MediaTek Inc MT1887
Mitsumi Electric Apple Optical USB Mouse
Apple Inc. Built-in iSight
Apple, Inc. Keyboard Hub
Apple, Inc Apple Keyboard
Apple Inc. Bluetooth USB Host Controller
Apple Computer, Inc. IR Receiver
Firewire Information: ?
LaCie d2 quadra (button) 800mbit - 800mbit max
S.M.A.R.T. Status: Verified
EFI (disk1s1) <not mounted>: 209.7 MB
Master Back-up (disk1s2) /Volumes/Master Back-up: 999.86 GB (640.35 GB free)
Gatekeeper: ?
Mac App Store and identified developers
Kernel Extensions: ?
[not loaded] com.aliph.driver.jstub (1.1.2 - SDK 10.7) Support
[loaded] com.eltima.SyncMate.kext (0.2.5b15) Support
[loaded] com.sophos.kext.sav (9.0.61 - SDK 10.7) Support
[loaded] com.sophos.nke.swi (9.0.53 - SDK 10.8) Support
Launch Daemons: ?
[loaded] com.adobe.fpsaud.plist Support
[loaded] com.google.keystone.daemon.plist Support
[loaded] com.oracle.java.Helper-Tool.plist Support
[loaded] com.oracle.java.JavaUpdateHelper.plist Support
[running] com.sophos.autoupdate.plist Support
[running] com.sophos.configuration.plist Support
[running] com.sophos.intercheck.plist Support
[running] com.sophos.notification.plist Support
[running] com.sophos.scan.plist Support
[running] com.sophos.sxld.plist Support
[running] com.sophos.webd.plist Support
[running] rapiback.plist Support
Launch Agents: ?
[loaded] com.google.keystone.agent.plist Support
[loaded] com.oracle.java.Java-Updater.plist Support
[running] com.sophos.uiserver.plist Support
[running] syncmateStarter.plist Support
User Launch Agents: ?
[failed] com.google.GoogleContactSyncAgent.plist Support
User Login Items: ?
None
Internet Plug-ins: ?
Flip4Mac WMV Plugin: Version: 2.3.8.1 Support
FlashPlayer-10.6: Version: 14.0.0.145 - SDK 10.6 Support
JavaAppletPlugin: Version: Java 7 Update 67 Check version
Flash Player: Version: 14.0.0.145 - SDK 10.6 Outdated! Update
Default Browser: Version: 537 - SDK 10.9
o1dbrowserplugin: Version: 5.4.2.18903 Support
QuickTime Plugin: Version: 7.7.3
googletalkbrowserplugin: Version: 5.4.2.18903 Support
Silverlight: Version: 4.0.51204.0 Support
iPhotoPhotocast: Version: 7.0
Audio Plug-ins: ?
BluetoothAudioPlugIn: Version: 1.0 - SDK 10.9
AirPlay: Version: 2.0 - SDK 10.9
AppleAVBAudio: Version: 203.2 - SDK 10.9
iSightAudio: Version: 7.7.3 - SDK 10.9
iTunes Plug-ins: ?
Quartz Composer Visualizer: Version: 1.4 - SDK 10.9
3rd Party Preference Panes: ?
Flash Player Support
Flip4Mac WMV Support
FUSE for OS X (OSXFUSE) Support
Java Support
MenuMeters Support
Time Machine: ?
Skip System Files: YES - System files not being backed up
Auto backup: YES
Volumes being backed up:
Thomas iMac: Disk size: 930.71 GB Disk used: 281.49 GB
Destinations:
Master Back-up [Local] (Last used)
Total size: 931.19 GB
Total number of backups: 154
Oldest backup: 2011-11-26 04:54:07 +0000
Last backup: 2014-08-19 00:06:13 +0000
Size of backup disk: Adequate
Backup size 931.19 GB > (Disk used 281.49 GB X 3)
Time Machine details may not be accurate.
All volumes being backed up may not be listed.
Top Processes by CPU: ?
      2% WindowServer
      0% sysmond
      0% SophosWebIntelligence
      0% InterCheck
      0% com.apple.WebKit.Networking
Top Processes by Memory: ?
156 MB SophosScanD
143 MB InterCheck
135 MB softwareupdated
98 MB mds_stores
98 MB com.apple.WebKit.WebContent
Virtual Memory Information: ?
89 MB Free RAM
1.61 GB Active RAM
1.44 GB Inactive RAM
516 MB Wired RAM
2.92 GB Page-ins
446 MB Page-outs

Databse of 1888 Byte Per Record with 259 Fixed Fields Stumped CR?!?

Similar Messages

Maybe you are looking for