Have a very large text file, and need to read lines in the middle.
I have very large txt files (around several hundred megabytes), and I want to be able to skip and read specific lines. More specifically, say the file looks like:
scan 1
scan 2
scan 3
scan 100,000
I want to be able to skip move the filereader immediately to scan 50,000, rather than having to read through scan 1-49,999.
Thanks for any help.
If the lines are all different lengths (as in your example) then there is nothing you can do except to read and ignore the lines you want to skip over.
If you are going to be doing this repeatedly, you should consider reformatting those text files into something that supports random access.
Similar Messages
-
I am a scientist and run my own business. Money is tight. I have some very large Excel files (~200MB) that I need to sort and perform logic operations on. I currently use a MacBookPro (i7 core, 2.6GHz, 16GB 1600 MHz DDR3) and I am thinking about buying a multicore MacPro. Some of the operations take half an hour to perform. How much faster should I expect these operations to happen on a new MacPro? Is there a significant speed advantage in the 6 core vs 4 core? Practically speaking, what are the features I should look at and what is the speed bump I should expect if I go to 32GB or 64GB? Related to this I am using a 32 bit version of Excel. Is there a 64 bit spreadsheet that I can us on a Mac that has no limit on column and row size?
Grant Bennet-Alder,
It’s funny you mentioned using Activity Monitor. I use it all the time to watch when a computation cycle is finished so I can avoid a crash. I keep it up in the corner of my screen while I respond to email or work on a grant. Typically the %CPU will hang at ~100% (sometimes even saying the application is not responding in red) but will almost always complete the cycle if I let it go for 30 minutes or so. As long as I leave Excel alone while it is working it will not crash. I had not thought of using the Activity Monitor as you suggested. Also I did not realize using a 32 bit application limited me to 4GB of memory for each application. That is clearly a problem for this kind of work. Is there any work around for this? It seems like a 64-bit spreadsheet would help. I would love to use the new 64 bit Numbers but the current version limits the number of rows and columns. I tried it out on my MacBook Pro but my files don’t fit.
The hatter,
This may be the solution for me. I’m OK with assembling the unit you described (I’ve even etched my own boards) but feel very bad about needing to step away from Apple products. When I started computing this was the sort of thing computers were designed to do. Is there any native 64-bit spreadsheet that allows unlimited rows/columns, which will run on an Apple? Excel is only 64-bit on their machines.
Many thanks to both of you for your quick and on point answers! -
Hi All,
powershell script Method required to read a value from registry and then taking the backup of that values in some text file.
For example the hive is
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment\Path
and under path i need to take back up of values in some text file and then put some value in the registry after back is taken in text file.
Also how to read the text file values so that we can again write to registry hive back from the back up text file.
Your help is much appreciated.
Umeed4uI think you need to read this first:
http://social.technet.microsoft.com/Forums/scriptcenter/en-US/a0def745-4831-4de0-a040-63b63e7be7ae/posting-guidelines?forum=ITCG
Don't retire TechNet! -
(Don't give up yet - 12,830+ strong and growing) -
How can FMS create a text file and write data into it in the Server application folders?
Recently, I writed a programe about creating a text file and writing data into it in the server application folder. My code is as following:
var fileObj = new File("/MyApp/test.txt");
if( fileObj != null)
if(fileObj.open( "text", "append"))
fileObj.write( " ———— Chat Info Backup ————\r\n" );
fileObj.close( );
trace("Chat info backup document :" + fileObj.name + " has been created successfully!");
But when I run it, FMS throw the error as following: File operation open failed ; TypeError: fileObj has no properties.
Can you help me ? Thanks in advance.
Supplement: The text file named test.txt doesn't exist before create the fileObj, an instance of File Class.Is MyApp the name of the application directory, or is it a child of the application directory? If myApp is the app name, just use test.txt as the path flag in the file constructor.
-
I have a MACBook and need to read documents from the government. Where do I get Pure edge viewer or something that will work? I have Adobe already.
Try this page http://net4geeks.com/index.php?option=com_content&task=view&id=69&Itemid=13
-
I have scratched off my Itunes code and now cannot read 2 of the digits. Is there a way to get it using the Card#?
If this page doesn't help then you will need to try contacting iTunes Support (you will probably need to give them images of the front and back of the card, and possibly its receipt) : http://www.apple.com/support/itunes/contact/ - click on Contact iTunes Store Support on the right-hand side of the page, then iTunes Cards And Codes
-
I am generating some very large XML files (600,000+ lines, 50MB+ characters). I finally have them all being valid XML and valid UTF-8.
But the files are so large Safari and Chrome will often not open them. FireFox will though.
Instead of these browsers, I was wondering if there are there any other recommended apps for the Mac for opening and viewing the XML, getting an error message if they are not valid for some reason and examing the XML tree?
I opened the file in the default app for XML which is Xcode, but that is just like opening it in a plain text editor. You can't expand/collapse the XML tree like you can with a browser, and it doesn't report errors.
Thanks,
DougHi Tom,
I had not seen that list. I'll look it over.
I'm also in touch with the developer of BBEdit (they are quite responsive) and they are willing to look at the file in question and see why it is not reporting UTF-8 errors while Chrome is.
For now I have all the invalid characters quashed and things are working. But it would be useful in the future.
By the by, some of those editors are quite pricey!
doug -
WebLogic writes to a text file and an applet reads the text file
Hi there,
I need some help here!
I have an applet that needs to read from a text file:
final URL prtDataURL = new URL("http://" + getCodeBase().getHost() + ":" + getCodeBase().getPort() + "/ESTC/lib/prtDataFile.txt");
final URLConnection urlConn = prtDataURL.openConnection();
urlConn.setUseCaches(false);
urlConn.setDoInput(true);
urlConn.setDoOutput(false);
final InputStreamReader is = new InputStreamReader(urlConn.getInputStream());
final BufferedReader in = new BufferedReader(is);
String inputLine;
while ((inputLine = in.readLine()) != null){
String[] tmp = inputLine.split("§");
this.dateTime = tmp[0];
this.quanTot = Integer.parseInt(tmp[1]);
this.quanSco = Integer.parseInt(tmp[2]);
this.quanSef = Integer.parseInt(tmp[3]);
this.valTot = Double.parseDouble(tmp[4]);
this.valSco = Double.parseDouble(tmp[5]);
this.valSef = Double.parseDouble(tmp[6]);
in.close();
is.close();and this is ok.
Now I have the web server (WebLogic) that must write, at regular time intervals, to that file (prtDataFile.txt). The web application writes to the file (which belongs to the application).
I've tried open the file with the File object but since the war file is unexploded I can't get the real path!
I've tried open an url connection,
final URL url = new URL("http://10.191.33.249:16801/ESTC/lib/prtDataFile.txt");
final URLConnection connection = url.openConnection();
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setDoInput(false);
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
out.write(currentDate + "§" + quantTot + "§" + quantSco + "§" + quantSef + "§" + valTot + "§" + valSco + "§" + valSef);;
out.flush();
out.close();which seems a little stupid since I'm opening an URLConnection to the same place where the application is. Anyway it doesn't work, i.e., it doesn't write to the file but it doesn't gives me any errors or exceptions.
Is there another way to get this two pieces communicating?
Thanks in advanceI have almost the exact same problem, and I am in the same situation as you are with respects to the language.
I am simply trying to create a file and output some garbage to it but my applet always spits back a security violation. I've tried eliminating the restrictions on the applet runner I use but I still get the error.
My method:
debug = new Label() ;
debug.setLocation( 20, 20 ) ;
debug.setSize( 500, 15 ) ;
add( debug ) ;
// output
try
OutputStream file = new FileOutputStream( new File( "" + getCodeBase() + "output.txt" ) ) ;
byte[] buffer = { 1, 2, 3, 4, 5 } ;
file.write( buffer ) ;
file.close() ;
} catch( Exception e )
debug.setText( e.toString() ) ;
Can anyone tell why this isnt working? -
What do I do?
Please help!iPad: Basic troubleshooting
http://support.apple.com/kb/TS3274
Update and restore alert messages on iPhone, iPad, and iPod touch
http://www.buybuyla.com/tech/view/012953a0d412000e.shtml
iOS: Resolving update and restore alert messages
http://support.apple.com/kb/TS1275
iPad: Unable to update or restore
http://support.apple.com/kb/ht4097
Cheers, Tom -
Best technology to navigate through a very large XML file in a web page
Hi!
I have a very large XML file that needs to be displayed in my web page, may be as a tree structure. Visitors should be able to go to any level depth nodes and access the children elements or text element of those nodes.
I thought about using DOM parser with Java but dropped that idea as DOM would be stored in memory and hence its space consuming. Neither SAX works for me as every time there is a click on any of the nodes, my SAX parser parses the whole document for the node and its time consuming.
Could anyone please tell me the best technology and best parser to be used for very large XML files?Thank you for your suggestion. I have a question,
though. If I use a relational database and try to
access it for EACH and EVERY click the user makes,
wouldn't that take much time to populate the page with
data?
Isn't XML store more efficient here? Please reply me.You have the choice of reading a small number of records (10 children per element?) from a database, or parsing multiple megabytes. Reading 10 records from a database should take maybe 100 milliseconds (1/10 of a second). I have written a web application that reads several hundred records and returns them with acceptable response time, and I am no expert. To parse an XML file of many megabytes... you have already tried this, so you know how long it takes, right? If you haven't tried it then you should. It's possible to waste a lot of time considering alternatives -- the term is "analysis paralysis". Speculating on how fast something might be doesn't get you very far. -
Read very very large excel file into 2d array (strings or varient)
Hi all,
Long time user, first time poster on these boards.
Looking at the copius amounts of great info related to reading Excel data from .xls files into labview, i've found that every one i've found from various people use the ActiveX method (WorkSheet.Range) which two strings are passed, namely excel's LetterNumber format to specify start and end.
However, this function does not work when trying to query huge amounts of information. The error returned is "Type Mismatch, -2147352571" I have a very large excel sheet i need to read data from and then close the excel file (original file remains unchanged). However this file is gigantic (don't ask me, I didn't make it and I can't convince them to use something more appropriate) with over 165 columns at 1000 rows of data.I can read a large number of columns, but only a handful of rows, or vice versa.
Aside from creating a loop to open and close the excel file over and
over reading pieces of it at a time, is there a better way to read more
data using ActiveX? Attached is code uploaded by others (with very minor modification) as an example.
Thanks,
Attachments:
Excel Get Data Specified Field (1-46col).vi 23 KBHi Maddox731,
I've only had a very quick glance through your thread, and I must admit I haven't really thought it through properly yet. Sounds like you've come up with your own solution anyway. That said I thought I'd take a bit of a scatter gun approach and attach some stuff for you regradless. Please forgive my bluntness.
You'll find my ActiveX Excel worksheet reader, which may or may not contain the problem you've come across. I've never tried it with the data size you are dealing with, so no promises. I've also attached my ADO/SQL approach to the problem. This was something I moved onto when I realised the limitations of AX. One thing I have noticed is that ADO/SQL is much faster than AX, so there may be some gains for you there with large data sets if you can implement it.
I should add that I'm a novice to all this and my efforts are down to bits I've gleamed from MSDN and others' LV examples. I hope it's of some use, if only to spark discussion. Your ctiticism is more than welcome, good or bad.
Regards.
Attachments:
Database Table Reading Stuff.zip 119 KB -
Read Text file and count occurences of certain string
Hello,
I have a text file with lines of data in it. I would like to read this text file and count how many lines match a certain string of text.
For example my text file has this data in it.
dog,blue,big
dog,red,small
dog,blue,big
cat,blue,big
If the certain string of text is "dog,blue,big" then the count would return "2".
Thanks for your helpHello,
Thank you for your post.
I am afraid that the issue is out of support range of VS General Question forum which mainly discusses the usage issue of Visual Studio IDE such as
WPF & SL designer, Visual Studio Guidance Automation Toolkit, Developer Documentation and Help System
and Visual Studio Editor.
I am moving your question to the moderator forum ("Where is the forum for..?"). The owner of the forum will direct you to a right forum.
In addition, if you are working with Windows Forms app. please consult on Windows Forms Forum:http://social.msdn.microsoft.com/Forums/windows/en-US/home?category=windowsforms
If you are working with WPF app, please consult on WPF forum:
http://social.msdn.microsoft.com/Forums/vstudio/en-US/home?forum=wpf
If you are working with ASP.NET Web Application, I suggest that you can consult your issue on ASP.NET forum:
http://forums.asp.net/
for better solution and support.
Visual Studio Language Forums:
http://social.msdn.microsoft.com/Forums/vstudio/en-US/home?category=vslanguages
Best regards,
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey. -
I have a string of text sms and iMessages that I would like to save and then delete them from my phone, but gain access to them later if I need them. I prefer to have the string with the date/time stamp vs copying and pasting into a doc or text file. Is this possible?
Not unless you buy an app that accesses data from an iTunes backup. Check in the app store.
-
Loading large text files into java vectors and outof memmory
Hi there,
need your help for the following:
i'm trying to load large ammoubnts of data into a Vector in order to concatenate several text files and treat them, but i'm getting outofmemory error. I even tried using xml structure and saving to database but the error is still the same. Can you help?
thanks
here's the code:
public void Concatenate() {
try {
//for(int i=0;i<1;i++) {
vEntries = new Vector();
for(int i=0;i<BopFiles.length;i++) {
MainPanel.WriteLog("reading file " + BopFiles[i] + "...");
FileInputStream fis = new FileInputStream(BopFiles);
BufferedInputStream bis = new BufferedInputStream(fis);
DataInputStream in = new DataInputStream(bis);
String line = in.readLine();
Database db = new Database();
Connection conn = db.open();
while(line != null) {
DivideLine(BopFiles[i], line);
line = in.readLine();
FreeMemory(db, conn);
MainPanel.WriteLog("Num of elements: " + root.getChildNodes().getLength());
MainPanel.WriteLog("Done!");
} catch (Exception e) {
e.printStackTrace();
public void DivideLine(String file, String line) {
if (line.toLowerCase().startsWith("00694")) {
Header hd = new Header();
hd.headerFile = file;
hd.headerLine = line;
vHeaders.add(hd);
} else if (line.toLowerCase().startsWith("10694")) {
Line entry = new Line();
Vector vString = new Vector();
Vector vType = new Vector();
Vector vValue = new Vector();
entry.name = line.substring(45, 150).trim();
entry.number = line.substring(30, 45).trim();
entry.nif = line.substring(213, 222).trim();
entry.index=BopIndex;
entry.message=line;
entry.file=file;
String series = line.substring(252);
StringTokenizer st = new StringTokenizer(series, "A");
while (st.hasMoreTokens()) {
String token=st.nextToken();
if(!token.startsWith(" ")) {
vString.add(token);
vType.add(token.substring(2,4));
vValue.add(token.substring(4));
token=null;
entry.strings= new String[vString.size()];
vString.copyInto(entry.strings);
entry.types= new String[vType.size()];
vType.copyInto(entry.types);
entry.values= new String[vType.size()];
vValue.copyInto(entry.values);
vEntries.add(entry);
entry=null;
vString=null;
vType=null;
vValue=null;
st=null;
series=null;
line=null;
file=null;
MainPanel.SetCount(BopIndex);
BopIndex ++;
public void FreeMemory(Database db, Connection conn) {
try {
//db.update("CREATE TABLE entries (message VARCHAR(1000))");
db.update("DELETE FROM entries;");
PreparedStatement ps = null;
for( int i=0; i<vEntries.size(); i++ ) {
Line entry = (Line) vEntries.get(i);
String value = "" + entry.message;
if(!value.equals("")) {
try {
ps = conn.prepareStatement("INSERT INTO entries (message) VALUES('" + Tools.RemoveSingleQuote(value) + "');");
ps.execute();
} catch(Exception e) {
e.printStackTrace();
System.out.println("error in number->" + i);
MainPanel.WriteLog("Releasing memory...");
vEntries = null;
vEntries = new Vector();
System.gc();
} catch (Exception e1) {
e1.printStackTrace();Well, i need to treat those contents, and calculate values withing those files, so wrinting files using FileInputstream wont do. for instance i need to get line 5 from file 1, split it, grab a value according to its class (value also taken) and compare it with another line of another file, adding those values to asingle file.
that's why i need vector capabilities, but since these files have more than 5 Mb each, an out of memory error is returned by loading those values into vector.
A better explanaition:
Each file has a line like
CLIENTNUM CLASS VALUE
so if the client is the same withing 2 files, i need to sum the lines into a single file.
If class is the same, then sum values, if not add it to the front.
we could have a final line like
CLIENTNUM CLASS1 VALUE1 CLASS2 VALUE2 -
I need to read data from a text file and display it in a datagrid.how can this be done..please help
hey ppl
i have a datagrid in my form.i need to read input(fields..sort of a database) from a text file and display its contents in the datagrid.
how can this be done.. and also after every few seconds reading event should be re executed.. and that the contents of the datagrid will keep changing as per the changes in the file...
please help as this is urgent and important.. if possible please provide me with an example code as i am completely new to flex...
thanks.....It's not possible to read from a file without using classes from the core API*. You'll have to get clarification from your instructor as to which classes are and are not allowed.
[http://java.sun.com/docs/books/tutorial/essential/io/]
*Unless you write a bunch of JNI code to replicate what the java.io classes are doing.
Maybe you are looking for
-
Hi there, when I click on COVER FLOW VIEW button, a message *+iTunes is unable to browse album covers on this computer+* appears on the screen. Does anyone know what's the problem? Any answers appreciated. thanks, Martin
-
How long does this update take? My screen went back and I'm worried the top on/off button got hit. I can;t get the screen to come back on. Even when the phone rings. Which means I can't answer phone. When I push the button on the front of the ph
-
Selecting Storage Type in Enterprise Manager
I've recently added a new enclosure, which has it's own storage type (Due to the fact they are different disks then the rest of the array). Using the "Storage Center Manager", when creating a new volume, I am asked to select one of my two different s
-
I am installing Oracle lite onto a laptop and need to have the smallest footprint possible. I know that I can remove many components of a regular install such as the documentation, but I would like to install only the components that are needed. I am
-
802.11n greenfield in a mixed environment
we plan to have mixed 802.11n deployment coexistence with a/b/g clients. But we want to have some classrooms or labs to run greenfield only where people request 100Mbps bandwidth to transfer large images. What is the best practice to design some smal