Cutting first few byte from Large files

Hi,
I have a requirement to cut out first 20 bytes from very large binary files (in the order of 1-4 GB).
I was looking at RandomAccessFile and FileChannel to accomplish this. But still, it seems that there will be a lot of I/O involved as it will be practically copying over the remaining bytes to another file only.
Is there any other approach which doesn't cause a lot of disk I/O.
Thanks.

>
I have a requirement to cut out first 20 bytes from very large binary files (in the order of 1-4 GB).
>
Somehow I suspect your 'requirement' isn't telling us the whole picture. If you can explain what the purpose of that requirement is there might be a couple of options.
>
replace the first 20 bytes with values that won't interfere with how the file needs to be used.
>
One reason might be becaue the files need to be read and processed and the first 20 bytes will interfere with that processing. Depending on the tool that will read the file you might be able to replace the 1st 20 bytes with NULLS (0 hex), SPACES, or some other set of values that the tool will ignore or silently fail on and continue.
It would be very fast to replace 20 bytes in each file with some other value.
Second option could be use a file filter that strips off the first 20 bytes when the file is read. Again this depends on the tool being used to read the file.
In *nix systems you could have a small filter that reads the files, strips of the first 20 bytes and pipes its output into the tool you are using to read the actual file.
If Java were the tool doing the reading it is even simpler. Just open the RandomAccessFile and move the file pointer to where you want it before the first read.
A third option is to modify the process that delivers the files to you to strip of the first 20 bytes. If the files are being delivered by FTP you could easily add a filter to do this.
Fourth option is to have the file supplier zip up the files for you. You can easily add a filter to the process to strip the 20 bytes as the file is being zipped. If these are files that will be kept and moved around the compression will save space and transmission time and you can sneak a filter in for free.
Can you tell us how the files will actually be used?

Similar Messages

How to read a byte[] from a file

Hi can someone help me to read a byte[] from a file. I need to save a byte[] to a file and then retrieve it later can someone help in this..
Thanx

there are actually docs on the classes ByteArrayInputStream and ByteArrayOutputStream here:
http://java.sun.com/javase/6/docs/api/
go there (copy & paste in browser) and text search for the above classes.
Then read into it. That way, you will come to a better comprehension of the entire thing. You may as well want to read this:
http://java.sun.com/docs/books/tutorial/essential/io/index.html
And for the future: if you have general problems, search API docs and tutorials first and solve them yourself. The guys around here can get pretty annoyed about "Teach me the entire Java Thing" (also called "someone do my homework") requests.
If you have a special problem, ie an Exception you don?t understand or a method wich you don?t know how it works, everybody will like to help you.
Only few real serial killers frequent these forums.

IMAP mail - error 9 reading first 11 bytes of message file

I have just moved my mail from a POP account to an IMAP one.
I am not having any problems that I know of..
but I have noticed in my console the following message when receiving mail
2007-06-26 16:00:35.497 Mail[743] error 9 reading first 11
bytes of message file /Network/Servers/fileserver.company.co.uk/Users/ian/Library/Mail/IMAP-ian@mails erver/DrogBell/DrogBell.imapmbox/Messages/278185.partial.emlx
Has anyone seen this at all?
Is it something I should be worried about?
TIA
Tony

lfeltswcicc,
It appears that in the past few days you have not received a response to your
posting. That concerns us, and has triggered this automated reply.
Has your problem been resolved? If not, you might try one of the following options:
- Visit http://support.novell.com and search the knowledgebase and/or check all
the other self support options and support programs available.
- You could also try posting your message again. Make sure it is posted in the
correct newsgroup. (http://forums.novell.com)
Be sure to read the forum FAQ about what to expect in the way of responses:
http://forums.novell.com/faq.php
If this is a reply to a duplicate posting, please ignore and accept our apologies
and rest assured we will issue a stern reprimand to our posting bot.
Good luck!
Your Novell Product Support Forums Team
http://forums.novell.com/

Error 105, Could not read full block (2048 bytes) from checkpoint file ~/dirchk/sdfsdj.cpe

Hi expert,
i am getting below error in goldengate due to mount point full and i released the space and still the same error for all gg processes. i can see the *cpe cpr file become 0 bytes. so i deleted and re added the extract and repliacat and while adding the replcat i used add replicat checkpoint table because of that multiple entries of same replicat came in checkpoint table . my checkpoint details also present in ./GLOBALS. now my doubt is if add replicate with mentioning checkpoint table name will duplicate entry will be created or what is the work around for this.
MANAGER RUNNING
Invalid checkpoint for EXTRACT qqqq (error 105, Could not read full block (2048 bytes) from checkpoint file XXXXXXXXX)
Invalid checkpoint for EXTRACT qqq(error 105, Could not read full block (2048 bytes) from checkpoint file XXXXXXXXXX)
Invalid checkpoint for REPLICAT qqq (error 105, Could not read full block (2048 bytes) from checkpoint file XXXXXXXXXXX)

Hi Kariyath
Increase the page size of your windows machine.Check for the recommanded page size.Remember that the recommanded page size should be the lower limit.If you have any issue feel free to ask.Your prob will be solved
Award suitable points

Regarding ssis - Filter first 5 rows from flat file

Hi,
i have a requirement like this every day we receive 10-15 Flat files , we need to load these data into SQL Table ,here nothing is complex for this we need to use foreachloop Container,Dataflowtask..etc. here the issue is each flat file have top 5 rows has
Client information like ( We need to delete First 5 rows) and then insert into SQL Table . please suggest me how to develope the package..

You need to use a script task to read file one by one and remove the first 5 rows from the file. The file name/path can be stored as a variable and read it from the script code. Insert the script task inside the foreach loop container just before importing
the file.
Refer the link to remove a line in .net
http://stackoverflow.com/questions/7008542/removing-the-first-line-of-a-text-file-in-c-sharp
Regards, RSingh

Peek the first two bytes from socket

I have a socket connection, using DataInputStream to read the bytes. However, I don't want to modify the input stream, by reading the first two bytes. I only want to peek the first two bytes without modifing the input stream. How can I do that?

Instead of reading directly from the InputStream returned by the socket, go through a BufferInputStream:
InputStream in = new BufferedInputStream(sock.getInputStream());
in.mark(2); // allow a reset until at most two bytes have been read
in.read(); // read at most two bytes
in.read();
in.reset(); // reset the stream to the state prior to the two reads

Showing data from large file in JTree

HI all...
I have a text file, about 1 million (yes, 1,000,000) records long. I want to be able to display this information in a JTree (it is categorized in the file by some category column). However, as you might guess, it would be impractical to load all 1 million records into the system, and them display them in a JTree. I am guessing I should use some sort of modified TreeModel, but my main question is, what is the best approach to load such a large file for display in a JTree? How should I create a data structure that I can pass to a JTree so that it does not crash (don't know if this will happen, but could it?).
Thanks,
Bryan

HI all...
I have a text file, about 1 million (yes, 1,000,000) records long. I want to be able to display this information in a JTree (it is categorized in the file by some category column). However, as you might guess, it would be impractical to load all 1 million records into the system, and them display them in a JTree. I am guessing I should use some sort of modified TreeModel, but my main question is, what is the best approach to load such a large file for display in a JTree? How should I create a data structure that I can pass to a JTree so that it does not crash (don't know if this will happen, but could it?).
Thanks,
Bryan

How to delete first few characters from all the elements of an array of strings

hi
I have a 2-dimensional array of strings length 104. I have to delete first 6 characters from all the elements of array and then change the strings to int. how to do it?? plz help

Attached you can find the screenshot showing how to do so.
Bye!
Licia
Attachments:
array.jpg ‏14 KB

Java writing out bytes from one file, reading back in again

Hi, I'm having some trouble with bytes.
I have one program which outputs the following bytes into a file (using Printstream.write() - I'm sure that this program is working fine):
4 0 64 0 4 0 128 0 63 223 64 20 4 16 130 0 60 16 64 1 63 223 79 236 64 16 195 232 116 63 255 232 148 0 0 0
I'm then reading in the bytes using the code given on this page (http://www.exampledepot.com/egs/java.io/File2ByteArray.html)
and I get...
4 0 64 0 4 0 -128 0 63 -33 64 20 4 16 -126 0 60 16 64 1 63 -33 79 -20 64 16 -61 -24 116 63 -1 -24 -108 0 0 0
As you can see I now have negative numbers - I noticed that if I do 256 + (the negative number) then i get the number I originally wrote out
I tried fixing this when I read in by doing
bytes[i] = (byte) (256 + bytes);
but it doesn't change anything
how can I either change it so it reads in the bytes exactly as i wrote them out, or write a bit of code that takes the negatives and converts them to the original positives?
thx

Unconditional wrote:
Integer x1 = (bytes[i] << 24) & 0xFF;
                  Integer x2 = (bytes[i+1] << 16) & 0xFF;
                  Integer x3 = (bytes[i+2] << 8) & 0xFF;
                  Integer x4 = (bytes[i+3] << 0) & 0xFF;is what I'm doing
System.out.println("x1 " + x1 + " x2 " + x2 + " x3 " + x3 + " x4 " + x4);
returns
x1 0 x2 0 x3 0 x4 0
x1 0 x2 0 x3 0 x4 4
x1 0 x2 0 x3 0 x4 0
x1 0 x2 0 x3 0 x4 128
x1 0 x2 0 x3 0 x4 0
x1 0 x2 0 x3 0 x4 63
x1 0 x2 0 x3 0 x4 223
x1 0 x2 0 x3 0 x4 64
x1 0 x2 0 x3 0 x4 20What are you trying to do? Combine the bytes into an int?

Cutting and pasting tables from PDF files

Hi
We use large pdf files for survey data tables (typically 500-1000 pages). It would be very useful to be able to cut and paste tables into MS-Office applictions, but this is generally not available. However, I have been able to cut and paste from another pdf file.
Right click on the selected tables gives two different menus, even though the software is the same (Adobe reader 9). The file I can cut and paste from has a right-click menu 'Copy with formatting', which works fine - the survey tables don't have the 'Copy with formatting' option.
What causes this difference? Is it possible to get the copy with formatting option?
Apologies if this is really obvious - any help gratefully recieved.
Regards
Georges

"Copy with formatting" is available when working with a Tagged output PDF.
If the survey data tables' PDF is created via a reporting app it is most likely not a Tagged output PDF (and likely, not able to be processed out as a Tagged PDF).
Be well...

Load XML-data from large file into plain table

We have to load data from an XML file to a relational table row by row.
The structure of the data is very simple:
<ROOT>
<repReport>
<KNZ>XY</KNZ>
<LA>test123</LA>
<Ident>AA20</Ident>
</repReport>
<repReport>
</repReport>
</ROOT>.
The table econtains the fields KNZ, LA, Ident, ...
If the file is small the following PL/SQL-code works fine:
CREATE OR REPLACE PROCEDURE Loadxml_Epd AS
fil BFILE ;
len INTEGER;
insrow INTEGER;
ctxHdl NUMBER;
buffer_c CLOB ;
amt NUMBER := dbms_lob.lobmaxsize;
src_offset NUMBER := 1 ;
dst_offset NUMBER := 1 ;
lang_ctx NUMBER := dbms_lob.default_lang_ctx;
warning NUMBER;
errNo NUMBER;
errMsg VARCHAR2(1999);
BEGIN
fil := BFILENAME('XML_DIR','200412MKBReport.xml');
DBMS_LOB.OPEN(fil, DBMS_LOB.LOB_READONLY);
len := DBMS_LOB.GETLENGTH(fil);
INSERT INTO INIT_CLOB(init) VALUES( EMPTY_CLOB())
RETURNING init INTO buffer_c;
dbms_lob.loadfromfile(buffer_c,fil, amt, dst_offset, src_offset);
ctxHdl := dbms_xmlsave.newContext ('SST_XML_EPD');
dbms_xmlsave.setRowTag (ctxHdl , 'repReport');
dbms_xmlsave.SetIgnoreCase(ctxHdl , 1);
dbms_xmlsave.SetSqlToXmlnameEscaping (ctxHdl , TRUE);
dbms_xmlsave.propagateOriginalException(ctxHdl , TRUE);
dbms_xmlsave.SetCommitBatch(ctxHdl , 1);
dbms_xmlsave.SetBatchSize(ctxHdl , 1);
insrow := DBMS_XMLSAVE.insertXML(ctxHdl,buffer_c);
DBMS_OUTPUT.PUT_LINE('Anzahl SÃ¤tze:' || TO_CHAR(insrow,'999999999'));
IF DBMS_LOB.ISOPEN(fil) = 1 THEN
DBMS_LOB.CLOSE(fil);
END IF;
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE(SQLCODE);
errMsg := SQLERRM;
DBMS_OUTPUT.PUT_LINE(SUBSTR(errMsg,1,2000));
IF DBMS_LOB.ISOPEN(fil) = 1 THEN
DBMS_LOB.CLOSE(fil);
END IF;
END;
But if the file is larger (50 MB in our case) this does not work. In various documents we found that we have to use another sort of XML-parser but we did not find any example how this is really done.
Does anybody have an example how to load this data that is as simple as our problem?

We have to load data from an XML file to a relational table row by row.
The structure of the data is very simple:
<ROOT>
<repReport>
<KNZ>XY</KNZ>
<LA>test123</LA>
<Ident>AA20</Ident>
</repReport>
<repReport>
</repReport>
</ROOT>.
The table econtains the fields KNZ, LA, Ident, ...
If the file is small the following PL/SQL-code works fine:
CREATE OR REPLACE PROCEDURE Loadxml_Epd AS
fil BFILE ;
len INTEGER;
insrow INTEGER;
ctxHdl NUMBER;
buffer_c CLOB ;
amt NUMBER := dbms_lob.lobmaxsize;
src_offset NUMBER := 1 ;
dst_offset NUMBER := 1 ;
lang_ctx NUMBER := dbms_lob.default_lang_ctx;
warning NUMBER;
errNo NUMBER;
errMsg VARCHAR2(1999);
BEGIN
fil := BFILENAME('XML_DIR','200412MKBReport.xml');
DBMS_LOB.OPEN(fil, DBMS_LOB.LOB_READONLY);
len := DBMS_LOB.GETLENGTH(fil);
INSERT INTO INIT_CLOB(init) VALUES( EMPTY_CLOB())
RETURNING init INTO buffer_c;
dbms_lob.loadfromfile(buffer_c,fil, amt, dst_offset, src_offset);
ctxHdl := dbms_xmlsave.newContext ('SST_XML_EPD');
dbms_xmlsave.setRowTag (ctxHdl , 'repReport');
dbms_xmlsave.SetIgnoreCase(ctxHdl , 1);
dbms_xmlsave.SetSqlToXmlnameEscaping (ctxHdl , TRUE);
dbms_xmlsave.propagateOriginalException(ctxHdl , TRUE);
dbms_xmlsave.SetCommitBatch(ctxHdl , 1);
dbms_xmlsave.SetBatchSize(ctxHdl , 1);
insrow := DBMS_XMLSAVE.insertXML(ctxHdl,buffer_c);
DBMS_OUTPUT.PUT_LINE('Anzahl SÃ¤tze:' || TO_CHAR(insrow,'999999999'));
IF DBMS_LOB.ISOPEN(fil) = 1 THEN
DBMS_LOB.CLOSE(fil);
END IF;
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE(SQLCODE);
errMsg := SQLERRM;
DBMS_OUTPUT.PUT_LINE(SUBSTR(errMsg,1,2000));
IF DBMS_LOB.ISOPEN(fil) = 1 THEN
DBMS_LOB.CLOSE(fil);
END IF;
END;
But if the file is larger (50 MB in our case) this does not work. In various documents we found that we have to use another sort of XML-parser but we did not find any example how this is really done.
Does anybody have an example how to load this data that is as simple as our problem?

Removing first few characters from an infoobjects

Hi,
I have an Infoobject where we load 9 character long records like - Ab1234567.
I have to transfer everything to a new Infoobject but not the first 6 character. Could you please let me know the best code with all the details...I have limited ABAP knowledge
Thanks for you help.

Please search before you post -
transformation routine: delete 4 first characters..
First N characters

Large file folder with question mark is flashing on screen.

Just turned on computer for the first time and a large file folder with a question mark is flashing on the screen. I am unable to do anything and have restarted and it does the same thing. Please help!

It does this when it can't see the operating system. You didn't give much information but I'm assuming it's a brand new mini and you aren't using any external drives for the operating system. If that's the case then yes, reinstall OS X. Put your install DVD into the optical drive and reboot. As soon as you hear the chime, hold down the "C" key on your keyboard (or the Option key until the Install Disk shows up). That will force your Mac to boot from the install DVD in the optical drive.
If reinstallation isn't possible, or if the problem continues after reinstallation, then you need to exchange it for another, because something is keeping the operating system on the drive from being read.

[solved] How to remove specific pages from PDF file?

I'm looking for a way to remove few pages from PDF file.
Last edited by delor (2008-07-04 10:28:22)

To remove first page I did this:
pdftk original.pdf cat 2-end output result.pdf
Any other propositions ?

Replace first few letter in filePath

Hi,
Im trying to delete first few letters in Links file path and get alert.
for example;
Links in indesign doc are placed from the below location
Machintosh HD/Exert/Studio/1/2/Resource/ .
so I want to remove Macintosh HD/Exert and get alert only from Studio/1/2/Resource/.
imgs = app.activeDocument.allGraphics;
for (i = 0; i < imgs.length; i++)
img = imgs[i].itemLink
var area = img.filePath.split(img.name)[0].replace(/\:/g,"/"); (to replace : with /)
var pa = area.split("Macintosh HD/Exert/");
alert (pa);
and
the alert is : ,Studio/1/2/Resource/ (how can i remove "comma");
Many thanks.

Why so complicated?
Try this:
var linkText = "Macintosh HD/Exert/Studio/1/2/Resource/";
var result = linkText.replace('Macintosh HD/Exert/', '')
alert (result)
Have fun
Or better (for Macintosh)
var linkText = "Macintosh HD:Exert:Studio:1:2:Resource:"; // your img.filePath
var linkText = linkText.replace(/:/g, '/');
var result = linkText.replace('Macintosh HD/Exert/', ''); // your pa
alert (result);

Cutting first few byte from Large files

Similar Messages

Maybe you are looking for