How would java compare to Perl for processing large txt files

I guess what I am asking is that I write a lot of scripts in perl for processing txt files with large amounts of data in them ( >1 MB or >100K Lines of data )
Stuff like searching for strings, deleting chuncks of the file, replacing strings, extracting strings, lines etc.
The reason I used perl is that I was under the impression it would be the fastest for the job but I never considered java. Now I am trying to get up to speed in Java so want to use these small jobs as practice?
Thanks ... J

Mod (Compiled) Perl is faster than Java I think. Also, perl has been optimized for this sort of thing. Plus, if you're creating a lot of small to medium size scripts, as you mentioned, perl will be a lot easier to support for you and who ever comes after you. Stick with what you know

Similar Messages

Java Gurus: How does Java compare to 4GLs?

I've been using Java for a year now. I love Java. As a 3GL, how does Java compare to a 4GL?
(I can't say I know a whole lot about fourth generation languages. The only 4GL that I'm a little familiar with is Clarion.)
What are your thoughts?

What is 3GLs and 4GLs? You mean Programming Language and Scripting Language?

How would I create buttons states for MCs I am creating abstract MCs

I am creating a generic website that would have abstract MCs that would load jpg or TXT fields for the user interface to allow for fast updates to the site. The MCs that would make up the user interface would be empty and load things into it using XML or PHP later on. How would I create button states for the user interface MCs. I started creating a function for every MC button state but I thought there might be a more efficient way.
would creating a array help in this case?
and is using URLRequest the way to link to pages with in a movie well the flash movie that is the site it's self?
/*---------------------------- THE START OF MY ACTION SCRIPT ------------------------------*/
var waywardLogo_mc:MovieClip = new MovieClip;
var theCollection_mc:MovieClip = new MovieClip;
var newsPage_mc:MovieClip = new MovieClip;
var whatthe#$@!doyouwant!?_mc:MovieClip = new MovieClip;
/*---------------------------- onOver ------------------------------*/
waywardLogo_mc.addEventListener(MouseEvent.ROLL_OVER,onOver);
theCollection_mc.addEventListener(MouseEvent.ROLL_OVER,onOver);
newspage_mc.addEventListener(MouseEvent.ROLL_OVER,onOver);
whatthe#$@!doyouwant!?_mc.addEventListener(MouseEvent.ROLL_OVER,onOver);
/*---------------------------- onOut ------------------------------*/
waywardLogo_mc.addEventListener(MouseEvent.ROLL_OUT,onOut);
theCollection_mc.addEventListener(MouseEvent.ROLL_OUT,onOut);
newspage_mc.addEventListener(MouseEvent.ROLL_OUT,onOut);
whatthe#$@!doyouwant!?_mc.addEventListener(MouseEvent.ROLL_OUT,onOut);
/*---------------------------- onClick ------------------------------*/
waywardLogo_mc.addEventListener(MouseEvent.CLICK,onClick);
theCollection_mc.addEventListener(MouseEvent.CLICK,onClick);
newspage_mc.addEventListener(MouseEvent.CLICK,onClick);
whatthe#$@!doyouwant!?_mc.addEventListener(MouseEvent.CLICK,onClick);
function onOver(event:MouseEvent):void
event.target.alpha = .5;
function onOut(event:MouseEvent):void
event.target.alpha = 1;
function onClick(event:MouseEvent):void
event.target.URLRequest("");
/*---------------------------- onover, onOut, onClick buttonModes ------------------------------*/
waywardLogo_mc.buttonMode = true;
theCollection_mc.buttonMode = true;
newspage_mc.buttonMode = true;
whatthe#$@!doyouwant!?_mc.buttonMode = true;

click insert/new symbol, tick movieclip, assign a name, tick export for actionscript and in the class textfield enter a name (say ButtonClass) and click ok.
attached to the first frame of your new movieclip, type stop() in the actions panel. put whatever graphic you want on-stage for your button's up stage. create another keyframe, label it "over" and put whatever graphic you want for button's over state on-stage.
in a layer above those graphics, you'll probably want to add a dynamic textfield so each of your buttons can have different text. assign the textfield and instance name (say tf) and extend its timeline to the last frame of your movieclip button.
then when you want to create a button, on your timeline you can use:
var b:ButtonClass=new ButtonClass(); // these two lines need to be entered for each button
buttonhandlerF(b,someX,someY);
//-------code between dotted lines only needs to be entered once no matter how many buttons you add --------------------
function buttonhandlerF(b:ButtonClass,x:Number,y:Number){
b.addEventListener(MouseEvent.MOUSE_OVER,overF);
b.addEventListener(MouseEVent.MOUSE_OUT,outF);
b.x=x
b.y=y
addChild(b);
function overF(e:MouseEvent){
e.currentTarget.gotoAndStop("over");
function outF(e:MouseEvent){
e.currentTarget.gotoAndStop(1);
//-------code between dotted lines only needs to be entered once --------------------
// you'll also want to create a click listener and listener function
the code between the dotted lines is amenable to being added to a ButtonClass.as class file if want to expand your capabilities.

How to process Large Image Files (JP2 220MB+)?

All,
I'm relatively new to Java Advanced Imaging, so I need a little help. I've been working on a thesis that involves converting digital terrain data into X3D scenes for future use in military training and applications. Part of this work involves processing large imagery data to texture the previously mentioned terrain data. I have an image slicer that can handle rather large files (200MB+ jpeg files). But it can't seem to process jpeg 2000 data. Below is an excerpt from my code.
public void testSlicer(){
String fname = "file.jp2";
Iterator readers = ImageIO.getImageReadersByFormatName("jpeg2000");
ImageReader imageReader = (ImageReader) readers.next();
try {
ImageInputStream imageInputStream = ImageIO.createImageInputStream(new File(fname));
imageReader.setInput(imageInputStream, true);
} catch (IOException ex) {
System.out.println("Error: " + ex);
ImageReadParam imageReadParam = imageReader.getDefaultReadParam();
BufferedImage destBImage = new BufferedImage(256, 256, BufferedImage.TYPE_INT_RGB);
Rectangle rect = new Rectangle(0, 0, 1000, 1000);
//Only reading a portion of the file
imageReadParam.setSourceRegion(rect);
//Used to subsampling every 4th pixel
imageReadParam.setSourceSubsampling(4, 4, 0, 0);
try {
destBImage = imageReader.read(0, imageReadParam);
} catch (IOException ex) {
System.out.println("IO Exception: " + ex);
The images I am trying to read are in excess of 30000 pixels by 30000 pixels (15m resolution at 5 degrees latitude and 6 degrees longitude). I continually get an OutOfMemoryError, though I am pumping up the heap size to 16000MB when using the command line.
Any help would be greatly appreciated.

Hi,
In general, an extra sizing for XI memory consumption is not required. The total memory of the SAP Web Application Server should be sufficient except in the case of large messages (>1MB).
To determine the memory consumption for processing large messages, you can use the following rules of thumb:
Allocate 3 MB per process (for example, the number of parallel messages per second may be an indicator)
Allocate 4 kB per 1kB of message size in the asynchronous case or 9 kB per 1kB message size in the synchronous case
Example: asynchronous concurrent processing of 10 messages with a size of 1MB requires 70 MB of memory
(3MB + 4 * 1MB) * 10 = 70 MB With mapping or content-based routing where an internal representation of the message payload may be necessary, the memory requirements can be much higher (possibly exceeding 20 kBytes per 1kByte
message, depending on the type of mapping).
The size of the largest message thus depends mainly on the size of the available main memory. On a normal 32Bit operating system, there is an upper boundary of approximately 1.5 to 2 GByte per process, limiting the respective largest message size.
please check these links..
/community [original link is broken]:///people/michal.krawczyk2/blog/2006/06/08/xi-timeouts-timeouts-timeouts
Input Flat File Size Determination
/people/shabarish.vijayakumar/blog/2006/04/03/xi-in-the-role-of-a-ftp
data packet size - load from flat file
How to upload a file of very huge size on to server.
Please let me know , your problem is solved or not..
Regards
Chilla..

How to write a code for open new txt file in swing

hai all,
now i do one project in java.that project's GUI is Swing. But i don't known swing (basic).So how to write a code for open new txt file and "Open window " in menu item on swing.that means when i click the "New" on menu that time open a new txt file. open also like that type.
plz give me that code ! very urgent
Advance Thanks !
RSK

Swing Tutorial:
http://java.sun.com/docs/books/tutorial/uiswing/index.html
Since you don't know the basic of swing read the tutorial, it is for your own good because it is useless if we provide you with a code you don't even understand and how it works.
If you want a menu read the tutorial about using menus and for opening a file read using JFileChooser.
note: don't use the word urgent because it implies that your problem is more important than others.

How to process large data files in XI ? 100 MB files ?

Hi All
At present we have a scenario as follows
It is File to IDoc ....Problem is the size of the file
We need to transfer 100mb file to SAP R/3 system ? So this huge data how to
process ?
Adv thanx and regards
Rakesh

Hi,
In general, an extra sizing for XI memory consumption is not required. The total memory of the SAP Web Application Server should be sufficient except in the case of large messages (>1MB).
To determine the memory consumption for processing large messages, you can use the following rules of thumb:
Allocate 3 MB per process (for example, the number of parallel messages per second may be an indicator)
Allocate 4 kB per 1kB of message size in the asynchronous case or 9 kB per 1kB message size in the synchronous case
Example: asynchronous concurrent processing of 10 messages with a size of 1MB requires 70 MB of memory
(3MB + 4 * 1MB) * 10 = 70 MB With mapping or content-based routing where an internal representation of the message payload may be necessary, the memory requirements can be much higher (possibly exceeding 20 kBytes per 1kByte
message, depending on the type of mapping).
The size of the largest message thus depends mainly on the size of the available main memory. On a normal 32Bit operating system, there is an upper boundary of approximately 1.5 to 2 GByte per process, limiting the respective largest message size.
please check these links..
/community [original link is broken]:///people/michal.krawczyk2/blog/2006/06/08/xi-timeouts-timeouts-timeouts
Input Flat File Size Determination
/people/shabarish.vijayakumar/blog/2006/04/03/xi-in-the-role-of-a-ftp
data packet size - load from flat file
How to upload a file of very huge size on to server.
Please let me know , your problem is solved or not..
Regards
Chilla..

How would I do a http post with a xml file

How would I do a http post with a xml file.
I have a url called https://localhost:8443/wss/WSS and the XML file is sent as part of the HTTP POST body.
How would I send this XML file when posting?

most people just add feedback to the comments they've asked about already, rather than ignoring all the feedback and making a new post with the exact same information:
http://forum.java.sun.com/thread.jspa?threadID=5247331&messageID=10020973#10020973
If you want to add in an XML file to a POST command, you'd add it just like any other file you'd attach to a post request, especially (if you're request is the same as your last post) if you're just doing a jsp page. I'd do it something like this:
<form method=POST ENCTYPE="multipart/form-data" action="https://rcpdm.mnb.gd-ais.com/Windchill/servlet/IE/tasks/DJK/fcsAddContentComplete.xml">
<H3> File Name </H3>
<input type=file name=filename>
<input type=submit value="Do it!">
</form>Then again, this solution has literally nothing to do with java, and you still need to make sure that your servlet knows what to actually do with that information you're sending it.

Processing large xml file (500mb)? break into small part? load into jtree

hi,
i'm doing an assignment to processing large xml file (500mb) and
load into jree using JAVA.
can someone advice me on the algorithm to do this?
how can i load a 500mb xml in a jtree without system hang?
how to i break my file and do the loading?

1 Is the file schema based binary XML.
2. The limits are dependant on storage model and chacater set.
3. For all NON-XML content the current limit is 4GBytes (Where that is bytes not characters). So for Character content in an AL32UTF8 database the limit is 2GB
4. For XML Content stored as CLOB the limit is the same as for character data (2GB/4GB) dependant on database character set.
5. For SB Based XML content stored in Object Relatioanl storage the limit is determined by the complexity and structures defiend in the XML Schema

How to read and write a string into a txt.file

Hi, I am now using BEA Workshop for Weblogic Platform version10. I am using J2EE is my programming language. The problem I encounter is as the above title; how to read and write a string into a txt.file with a specific root directory? Do you have any sample codes to reference?
I hope someone can answer my question as soon as possible
Thank you very much.

Accessing the file system directly from a web app is a bad idea for several reasons. See http://weblogs.java.net/blog/simongbrown/archive/2003/10/file_access_in.html for a great discussion of the topic.
On Weblogic there seems to be two ways to access files. First, use a File T3 connector from the console. Second, use java.net.URL with the file: protocol. The T3File object has been deprecated and suggests:
Deprecated in WebLogic Server 6.1. Use java.net.URL.openConnection() instead.
Edited by: m0smith on Mar 12, 2008 5:18 PM

How would I search the hastable for homer?

This is an example that I did awhile back. I currently messing around with it just as a review. I got stuck on how I can implement a search for my hashtable. Here is what I have so far
import java.util.*;
public class SimpsonsClient {
    public static void main(String args[]) {
      Hashtable hash = new Hashtable();
      Student st1 = new Student("Homer", "Simpson", "0");
      Student st2 = new Student("Marge", "Simpson", "0");
      Student st3 = new Student("Bart", "Simpson", "0");
      Student st4 = new Student("Lisa", "Simpson", "0");
      Student st5 = new Student("Maggie", "Simpson", "0");
      System.out.println("Adding values to hash");
      hash.put(st1.getAge(), st1);
      hash.put(st2.getAge(), st2);
      hash.put(st3.getAge(), st3);
      hash.put(st4.getAge(), st4);
      hash.put(st4.getAge(), st5);
      System.out.println("Does the hash have st1?:" + hash.contains(st1));
      System.out.println("Is the hash code empty?" + hash.isEmpty());
      //Search for homer?
      System.out.println("Hash contain Homer?" + hash.get("Homer"));
    }Output Looks like this:
Adding values to hash
Does the hash have st1?:true
Is the hash code empty?false
Hash contain Homer?null
I just would like to know how I can do a search for homer and pull his records off like the name, age, and the field of 1.
Thanks

import java.util.*;
public class SimpsonsClient {
    public static void main(String args[]) {
      Hashtable hash = new Hashtable();
      Student st1 = new Student("Homer", "Simpson", "0");
      Student st2 = new Student("Marge", "Simpson", "0");
      Student st3 = new Student("Bart", "Simpson", "0");
      Student st4 = new Student("Lisa", "Simpson", "0");
      Student st5 = new Student("Maggie", "Simpson", "0");
      hash.put(st1.getAge(), st1);
      hash.put(st2.getAge(), st2);
      hash.put(st3.getAge(), st3);
      hash.put(st4.getAge(), st4);
      hash.put(st4.getAge(), st5);
    }I can't really tell actually. You haven't included a definition for the Student class. If it's something like this:
public class Student
public String getName();
public String getSurname();
public String getValue(); // the third field
}In the old jdk 1.1 API
public static boolean tableContainsStudent(Hashtable table, String name)
for(Enumeration students = table.elements(); students.hasMoreElements(); )
    if (((Student)students.nextElement()).getName().equals(name)) return true;
return false;
}It's quite similar with Collections API
for (Iterator students=table.values().iterator();students.hasNext(); )
if (((Student)students.nextElement()).getName().equals(name)) return true;
return false;
...I don't think the Hashtable is optimal at all for searching this way. It would be much nicer if you can do a binary search, but that requires more thinking when designing the storage class.
greets.

How will card payments be submitted for processing?

I’m setting up an online shop for my client, where products will be sold seamlessly through a payment gateway (SecurePay). My client is in the process of applying for a NAB merchant account. One of the questions on NAB’s application form is:
How will you be submitting your card payments for processing?
Virtual terminal or batched payments using tokenisation – you are not storing card details in your own systems.
Hosted payment page or transparent redirect – card details are not transmitted via your systems (eg. NAB Transact Direct Post or Hosted Payment Page).
API or batched payments – you store and/or transmit card details via your own systems.
Which of these options apply to collecting credit card payments using a payment gateway with Business Catalyst? Is it the first, “you are not storing card details in your own systems”?

Credit card details are not stored on the BC system - It complies to PCI Level 1 compliance.
Your payment type is dependant on your payment gateway, you mentioned seamless so that rules out 2 as well as 3 as no credit card details are stored.

How do I specify quicktime plugin for use by mp3 files on ppc mac

In the Applications section of Firefox Preferences I find no way to specify that the Quicktime PLUG-IN is to be used for mp3 files. Although this action option is a provided for other kinds of files, that action option is missing for MP3 Audio file (audio/mp3) and for MP3 Audio File (audio/x-mp3).
I can specify that the Quicktime PLAYER should be used, but not the plug-in. How can I specify that I want the plug-in to be used?

You need to set in the QuickTime Preferences what files types you want that plugin to handle.
See also http://kb.mozillazine.org/QuickTime

Upload/Download from unix - give easy exammple for tab delimited .TXT file

Hello experts,
I have 2 requirements.
1. I want to upload a tab delimited .txt file from local PC to UNIX server.
2. I want to Read same file, then some process and get result to the same location. (output file name should be diffrent)
How i can fulfill these two requirements?
Regards,
RH

Hi,
upload a tab delimited .txt file from local PC to UNIX server
Please go through the below logic this will help you,
*& Report Z10_SHAD_V_UPLD_PR_APP_SERV
report z10_shad_v_upld_pr_app_serv message-id zmessage.
*&--INCLUDE FOR DECLARATION
include z10_upld_top.
*&--INCLUDE FOR ALL FORMS
include z10_upld_forms.
start-of-selection.
*&--PERFORM FOR UPLOADING LOGIC.
perform z10_upld.
end-of-selection.
*& Include           Z10_UPLD_TOP
*&--STRUCTURE FOR INTERNAL TABLE.
*&--THAT IS GOING TO HOLD PRESENTATION SERVER FILE DATA.
data:begin of wa,
name(40) ,
end of wa.
*&--INTERNAL TABLE THAT IS GOING TO HOLD PRESENTATION SERVER DATA.
data: it_tab like wa occurs 0 with header line.
*&--SELECTION-SCREEN FOR GETTING THE PATH OF PRESENTATION LAYER
selection-screen begin of block abc with frame title text-001.
parameters: p_string(128) type c,
                  p_st(128) type c,
selection-screen end of block abc.
data: file_path type rlgrap-filename.
data: p_path type rlgrap-filename,
      filename1 type string,
      file_name like ibipparms-path,
      v_path like dxfields-longpath,
      l_path type string.
*&--PROVIDING F4 KEY FOR THE SCREEN FIELD THAT IS GOING TO
*&--HAVE THE PRESENTATION SERVER FILE PATH
at selection-screen on value-request for p_string.
call function 'F4_FILENAME'
   exporting
     program_name        = syst-cprog
     dynpro_number       = syst-dynnr
FIELD_NAME          = ' '
   importing
     file_name           = file_name
p_string = file_name.
file_path = p_string.
p_path = file_path.
filename1 = file_name.
*&--PROVIDING F4 KEY FOR THE SCREEN FIELD THAT IS GOING TO
*&--HAVE THE APPLICATION SERVER FILE PATH
at selection-screen on value-request for p_st.
call function 'F4_DXFILENAME_TOPRECURSION'
EXPORTING
I_LOCATION_FLAG       = ' '
I_SERVER              = '?'
I_PATH                =
FILEMASK              = '.'
FILEOPERATION         = 'R'
importing
O_LOCATION_FLAG       =
O_SERVER              =
   o_path                = v_path
ABEND_FLAG            =
exceptions
   rfc_error             = 1
   error_with_gui        = 2
   others                = 3
p_st = v_path.
if sy-subrc <> 0.
message id sy-msgid type sy-msgty number sy-msgno
         with sy-msgv1 sy-msgv2 sy-msgv3 sy-msgv4.
endif.
*&      Form Z10_UPLD
      text
--> p1        text
<-- p2        text
form z10_upld .
if not p_string is initial.
*&--FUNCTION MODULE THAT IS GOING TO UPLOAD THE PRESENTATION SERVER FILE TO
*&--APPLICATION SERVER.
call function 'GUI_UPLOAD'
exporting
   filename                      = filename1
   filetype                      = 'ASC'
HAS_FIELD_SEPARATOR           = ' '
HEADER_LENGTH                 = 0
READ_BY_LINE                  = 'X'
DAT_MODE                      = ' '
CODEPAGE                      = ' '
IGNORE_CERR                   = ABAP_TRUE
REPLACEMENT                   = '#'
CHECK_BOM                     = ' '
VIRUS_SCAN_PROFILE            =
NO_AUTH_CHECK                 = ' '
IMPORTING
FILELENGTH                    =
HEADER                        =
tables
    data_tab                      = it_tab[]
exceptions
   file_open_error               = 1
   file_read_error               = 2
   no_batch                      = 3
   gui_refuse_filetransfer       = 4
   invalid_type                  = 5
   no_authority                  = 6
   unknown_error                 = 7
   bad_data_format               = 8
   header_not_allowed            = 9
   separator_not_allowed         = 10
   header_too_long               = 11
   unknown_dp_error              = 12
   access_denied                 = 13
   dp_out_of_memory              = 14
   disk_full                     = 15
   dp_timeout                    = 16
   others                        = 17
if sy-subrc <> 0.
message id sy-msgid type sy-msgty number sy-msgno
         with sy-msgv1 sy-msgv2 sy-msgv3 sy-msgv4.
endif.
*path = filename1.
open dataset p_st in text mode for output encoding default.
if sy-subrc ne 0.
    close dataset p_st.
    exit.
endif.
if sy-subrc eq 0.
loop at it_tab.
    transfer it_tab to p_st.
endloop.
close dataset p_st.
message i000.
endif.
endif.
endform.                    " Z10_UPLD
Hope this helps.

How to extract a column out of a large ASCII file?

Hi all.
After searching the board and applying several solution approaches my problem still remains. Maybe you can help me.
The data source i've to deal with are large ASCII files (~540 MB) with 14 columns (delimiter: TAB). Each column represents one channel. The number of characters in each "field" is variable. I have to read user defined columns (=channels) out of each data set. Needless to say that reading the whole file runs into memory problems.
If anyone has an idea i would be happy
Thanks in advance.
Greets
Kane

I hate to defocus you, but there is a more efficient way to do this. My apologies that I do not have the time to write code, but here is the pseudo code.
Create an array for your output greater than or equal to what you think you will need.
Read a 65,000 character chunk from the file (or the rest of the file, whichever is smaller).
Use the string search functions functions to find successive line ends and the appropriate tab character delimiters for your column.
Convert and replace the element in your output array.
When done, trim your output array to the right size.
If you drop an LVM read, convert it to a regular VI, and dive in, you will see an example of this type of process. The idea is to keep disk reads, which are very inefficient, to a minimum. It also minimizes your memory allocations, because you do not need to resize your input buffer for every line. Problems you will need to deal with (which are handled by the LVM read) are such things as:
Your line crosses a chunk boundary.
The end-of-file creates a smaller chunk than 65,000 characters (the optimum chunk size for Win32 systems).
The end-of-line character is not well defined (in your case, this is probably not an issue)
Searching for a character can produce memory allocations
You may want to try reading the data as a U8 array instead of a string and doing your searches on that instead of the string.
I have always wanted to write the piece of code, but never had the time or reason to do so. Good luck. I will try to help if I can.
This account is no longer active. Contact ShadesOfGray for current posts and information.

COMPSUMM.BOX Slow to Process Larger SUM Files

Hey Everyone,
We have a large SCCM 2007 environment (over 2000 sites, 80,000+ clients) and we are finding the Central Site doesn't appear to be able to keep up with the influx of messages coming into COMPSUMM.BOX. We can move all of the files out and feed
them back in slowly, but even if we do nothing but clear out the files the inbox gets backlogged again.
In this Inbox there are a combination of SVF and SUM files. The SVF files are small and process without any issues. The problem is the SUM files. The SUM files up to about 500KB or less appear to process OK and don't hold
up the overall inbox processing too much, but the problem is when the 2MB, 3MB, 4MB and 5MB SUM files come in. As soon as they hit, the inbox comes to a crawl. It takes a significant amount of time to process each of these larger files (15mins+
each) and while it's processing one, more come in, then it gets to the next one, and then more come in and by the end of the day we can end up with as many as 300,000 files in this folder. Eventually, it seems to process a lot of them and occasionally
even gets caught up (down to only 10,000 files), but in busier periods it needs help by us moving some of the files out. Also, the large SUM files seem to process at a higher priority and leave other items sitting there not moving while the system works
on the larger files? What process generates the larger files flowing through here and how often?
My question is, is there anything we can do to reduce the amount of data flowing through this specific inbox so it is able to better keep up with the load? Can we change the processing priority so that the larger SUM files don't automatically jump
to the front of the queue? I know we can change the schedule for the "Site System Status Summarizer", but there doesn't seem to be such a schedule for the "Component Status Summarizer". I don't want to turn off replicating these
messages from the child primary sites, but we also don't want to be dealing with a constant backlog situation either.
Any suggestions are much appreciated.
Thanks!
-Jeff

Hi Garth,
I wasn't saying that the larger files process because they are larger files but rather that SUM files appear to process at higher priority than SVF files in this Inbox. Once one of the large SUM files hits it's turn in the queue, compsumm.box starts
growing. There were 88,000 files in here when I came in this morning and it still had files from the 17th that hadn't processed. Moved out all the large files (1000+ of them) and all of the other files in the Inbox processed. Move the 1000
back in... backlog begins again. We can have 300 SVF files come in and one SUM file and the SUM file will begin processing immediately and the SVF files don't appear to process until the SUM file is done. I was thinking more along the lines that
the content of the SUM files may be prioritized higher than others.
By 2000 Sites, yes CM07 Primary and Secondary Sites. 19 Primary and 2286 Secondary to be exact.
No errors in COMPSUMM.LOG. Just very busy processing.
HINV = Every 4 Days
SINV = Every 7 Days
SWM = Every 7 Days
Heartbeat = Every 7 Days
Discovery/Inventory isn't an issue... No DDR or MIF backlogs. We don't run AD System Discovery, only Group Discovery and it runs on the tier two primary sites once a month and are staggered by region.
CPU is steady at about 50% with SMSEXEC using about 30%.
Memory is steady at around 10GB of the 36GB Total
SQL is interesting... SQL is a cluster on a dedicated server with 48GB physical memory. SQL Server 2008 SP3 64-Bit SQL is configured to be able to use up to 42GB of memory leaving 6GB for the OS. In Task Manager, SQLServr.exe is showing
it is using about 800-900MB most of the time. However, looking in the status bar in Task Manager it shows Physical Memory: 98%, consistently. I ran a TASKLIST and exported the results to view in Excel but when I total everything there, it is only about
2.1GB. Hmm... Running RAMMap.exe it shows AWE allocated with 43GB of memory. AWE is NOT enabled in SQL. From another Google search, this appears to be something others have seen as well, but I'm not finding any good solution, or
any really clear indication this is actually a problem as opposed to how Win2K8 is managing memory. Don't like seeing the server showing 98% memory usage though. Going to continue to look at this further and feed the info back to the team.
I have not performed the DBCC commands on the SQL database (yet) but will do so.
Thanks!
-Jeff

How would java compare to Perl for processing large txt files

Similar Messages

Maybe you are looking for