Alternative way to read data

I have extremely large data files in this format:
027/18/01 3:58 PM
0300010300000000000000000304000000000003010100000001
027/18/01 4:03 PM
0300020300000000000000000304000000000003010100000002
027/18/01 4:12 PM
0300030300000000000001000004000000000003010202000002
027/19/01 8:28 AM
0300040300000000000001000004000000000003010100000001
027/19/01 9:07 AM
0300050300000000000001020000000000000000010100000001
first 5 lines data out of possible 1000
first 6 places are just identifiers, the rest is data represented in binary form( respondent 1, question 1 had answer 03...etc)
This is my read code:
try{
       DataInputStream dis = new DataInputStream(new FileInputStream(DataFile));
                     while(!EOF){
                         try{
                              input = dis.readLine();
                                 if(input == null){
                                       EOF = true;
                                       break;
                              data = new char[input.length()];
                              data = input.toCharArray();
                              prefix = new String(data, 0, 2);
                                if(Integer.parseInt(prefix) == 1){
                                    CheckName();
                                else
                                if(Integer.parseInt(prefix) == 2){
                                    DateTime();
                                else
                                if(Integer.parseInt(prefix) == 3){
                                    ConvertData(data, numberOfQuestions, answers_per_question);
                        catch(IOException e){
                                  System.err.println(e);
                      }// close while
                         dis.close();
                catch(IOException e){System.err.println( e );}Problem is the larger the dat file gets the slower the read process gets, anyway to use a buffer in this instance( would it make it that much faster? ).
Thanks
Jim

Try adding a BufferedInputStream wrapping your FileInputStream. Could help.

Similar Messages

How is the best way to read data from an iphone if you lost your itunes data after a crash?

How is the best way to read data from an iphone if you lost your itunes data after a crash?

How is the best way to read data from an iphone if you lost your itunes data after a crash?

Best way to read data sources in parallel

Hi,
I'm looking for conceptual help as I start a project. I am trying to figure out the best way to get data from several sources at different timings and deliver them to a main vi.
I have 4 systems, which each work well on their own (OK, one doesn't work yet, but let's assume that can be fixed
One system reads from 2 pH meters on serial ports. The meters are slow to respond, so it takes about 2 minutes to read 4 channels of data. I save these data to a file every 10 minutes
One system reads from a CO2 meter on the USB port. It reads the data every second, and does some averaging. Every 2-10 minutes, it saves the average to a file and then sends a command to the parallel port to switch the input to the meter.
The third system reads from 6 valves, each on a serial port. These also take time, probably several minutes to poll all 6. These data will also be saved to a file.
The 4th system reads a bank of temperature probes on the USB port. These get polled every few seconds and saved to a file every few minutes.
Now that these individual routines are working, I am trying to create a front end that will display all the data in one place and allow me to set all the parameters from a single place. I would also like the possibility of using the data from one source at another place (for instance, having the output of the temperature probes sent to the pH meters to adjust their calibration). At this point, I get confused as to the best way to proceed.
It seems like if I just want to read the data from each source, I could simply put all 4 routines together in a single vi (oh, what a mess that would be to read). Maybe I should start this way?
However, if I want to have any communication between the different data sources, it seems like I will either need to use queues or VI server. I sort of envision a vi that lets me configure the various ports and the file operations and then can turn on monitoring of any or all of the various inputs. Each of them will do their thing at their own time and the main routine will simply display whatever data they deliver whenever they have new data. Fortunately, nothing is particularly time-critical, nor does it need to run fast.
My questions: Am I correct in how I'm thinking about getting this to work? Is there a clear choice between queues or vi server? I've looked at several examples of each, but without having done something like this before, it is hard for me to tell which is better.
Thanks for any suggestions.
mike

Hi Mike,
I think that you are on the right track with your thinking process. You might be able to implement this using queues. I'm not exactly sure how you would do it with VI server since it is just a set of functions that allows you to dynamically control front panel objects, VIs, and the LabVIEW environment. However, there are some great resources available with using queues for this type of application. I'm including the link to another discussion forum that had a very similar question to yours. There is a good example of using queues within this forum post. Also, there is a great example in NI Developer Zone about using queues and some other good ones in the NI Example Finder (just search 'queues' and you should get a few results). I hope this helps!
Carla
National Instruments
Applications Engineer

Is there a way to read data off drive, not relying on file structure somehow?

That might be the wrong question. But it kind of gets at what I'm after. I have a DVD disk which was written by my TV DVD machine. While it was recording onto the disk, it ran out of space. When this happens, when I put the disk into my computer, my computer show zero files on there. Whereas if it hadn't of run out of space, a folder and a few files inside would appear. How can I access what's on the disk in this state? Is there a way to get the data off and turn it into a file?; there is data on the disk but my computer is showing zero files. Thanks.

Some, if not all DVD Recorders (i.e. SA 8300 and Cablevision) use a proprietary file system that matches the drive to the device, so not only can it not be read by a computer, it won't even work when read by another SA 8300 device. (Found this out when my SA 83000 cable box crashed. My recording on an external drive could not be read by the new SA 8300).
If you want to backup the files to another drive, you may be able to do a block-by-block copy.

Does exist a way to read data from IP planning cube with ABAP?

Hello All.
My scenario is as follows:
I have an ODS where we store costcenters to be planned. This ODS is loaded via a manual falt file load (regular dta transfer process). In order to avoid inconsisten data in the planning cube, I want to check in my ODS load process, if any costcenter which is already planned (it has planned data in my planning cube) is being deleted in the loaded file.
To do this, I need to access my planning data cube ( i guess also planning buffer), so my question is if there is any function module which retrieves data from a planning level as reference data.
Thanks a lot and best regards,
Alfonso.

Hi Alfonso,
note 1101726 shows how the plan buffer can be read.
     l_r_plan_buffer = cl_rsplfa_plan_buffer=>if_rsplfa_plan_buffer~get_instance( i_infoprov ).
     l_r_plan_buffer->get_data( EXPORTING i_t_charsel = l_t_charsel
                                          i_include_zero_records = rc_false
                                          i_r_msg = l_r_msg
                                IMPORTING e_r_th_data = l_r_th_data
                                EXCEPTIONS OTHERS = 2 ).
Please take a look at the note. You do not need to implement the after_burn_selection exit, but you can find sample code how to read the planning buffer. Please give it a try.
Another solution would be to use the function module RSDRI_INFOPROV_READ. But you need to make shure that you first close the yellow request. This can be done using function module RSAPO_SWITCH_BATCH_TO_TRANS.
Hope this helps
Matthias Nutt
SAP Consulting Switzerland

Alternative ways to read excel in Sharepoint

Now I'm facing with a problem of reading excel file on SharePoint server.
In the old ways I usually use the OLEDB to read excel through the connection string, but it's just work on 32 bit system, then I attempt to convert my app to x86 platform target, but unfortunately my app (is a WebPart) which SharePoint is not allowed to work
with.
I do many googling and hearing SharePoint Module (Microsoft.SharePoint.dll) can do that but I can't find any references. Anyone know and can you help me?
Thanks.

Hi EngSoonCheah,
According to your description, my understanding is that you want to read excel file on the SharePoint Server.
I suggest you can use Microsoft OpenXML library to achieve it.
Before using OpenXML to read excel, you may need to download and install it firstly:
Download OpenXML for Office
Here is a code demo about reading excel using OpenXML for your reference:
Read and Write Microsoft Excel with Open XML SDK
Best Regards
Zhengyu Guo
TechNet Community Support

Efficient way to read CLOB data

Hello All,
We have a stored procedure in oracle with CLOB out parameter, when this is executed from java the stored proc is executed fast but reading data from clob datatype using 'subString' functionality takes more time (approx 6sec for 540kb data).
Could someone please suggest what is the efficient way to read data from Clob (We need to read data form clob and write into a file).
Thanks & Regards,
Prashant,

Hi,
you can try buffered reading / writing the data, it usually speeds the process up.
See example here:
http://www.oracle.com/technology/sample_code/tech/java/sqlj_jdbc/files/advanced/LOBSample/LOBSample.java.html

How to read data from a file in OSB

hi guys,
Recently, I've got a problem with reading file from specific location. I've actually followed this post OSB 11g - Read or Poll File in OSB - Oracle Fusion Middleware Blog, and then
I know how to read a file. However, it does not as expected. Because, I've found no way to read data from the file. Therefore, no chance to manipulate the data like assigning to a variable, or extracting ....
Hence, is there any way to read data from file by using proxy service in OSB ??? No Java code ???
by the way, supposed that there is no way to read data from a file in OSB. So, What purposes will the way in the post above be used for?
Many thanks in advance

http://jakarta.apache.org/poi/hssf/index.html
HSSF stands for Horrible Spreadsheet Format, but it still works!

Read data from maintenance view

hi all,
could anyone please tell me if there is any special way to read data from a maintenance view (if at all it is possible).
i wrote a simple select on it but it gave me a syntax error that the view is not defined in the dictionary. (i am able to see the view definition by double clicking and it is active too).
the following is my select.
select single lgnum from v_t320 into (w_lgnum) where werks eq p_i_werks and lgort eq p_i_lgort.
awaiting answers...
regards,
PJ

Thanx Rich,
but the documentation on maintenance views says that we can have read access to them. Does that mean the data can only be seen via the DDIC and not from a select written in a program??
this is what i read...
Maintenance Status
The maintenance status of a view controls whether data records can also be changed or inserted in the tables contained in the view.
The maintenance status can be defined as follows:
Read only: Data can only be read through the view.
Read, change, delete, insert: Data of the tables contained in the view can be changed, deleted, and inserted through the view.
Only read access is permitted for database views that are defined through several tables. If a database view contains only one single table, data can be inserted in this table through the view (see Inserts Through Database Views).
The following status definitions are also possible for maintenance views:
Read and change: Existing view entries can be changed. However, records cannot be deleted or inserted.
Read and change (time-dependent views): Only entries whose non-time-dependent part of the key is the same as that of existing entries may be inserted.
you surely have earned some points here....just let me be clear of the concept behind this
regards,
PJ

Read Data From DSO - In ABAP Program

HI Friends,
Can any one tell how to read Data from a DSO in a Abap Program on BI System itself......
Which is the better way to read data from DSO in Abap Program... Is it
1. Reading Data Directly From the Active Table Of the DSO ... or
2. Reading Data From DSO....
In case of reading data from DSO.... Which function module is used to do this work....When i searched the forum i came across 2 function Modules.... BAPI_ODSO_READ_DATA_UC and RSDRI_INFOPROV_READ..... Please let me know which one should be used....
Regards,
Shyam.

Hi Shyam,
The active content of any ods will be stored in the database tables( SE11) with the naming convention /BIC/A<odsname>00 for Custom-defined DSO and /BI0/A<odsname>00 for Business content delivered DSO.
For example if the technical name of the ods is TEST , then the active content table name in SE11 will be /BIC/ATEST00.
You can write your own program to read the contents of this database table.
Regards,
Krishna.

Urgent help on reading data into ByteBuffer

Hi,
I am trying to read data into ByteBuffer using SocketChannel.
The following code working fine but the problem is with ByteBuffer size.
The data varies from request to request .
How do I read data into ByteBuffer with out specifing size .
The problem with size is some records has more than 500 bytes and some are less than 500 bytes.
I want to read all the data .. ( even if it has more than 500 bytes ..)
How do i solve this problem ..
Is there any way to read data dynamically from socket channel ...
Any help on this please ...
The data I have to read is in the following format ..
XXX ... yyyyy
AAA
BBBBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDDDDDDDDDd
etc ...
Here is my Code:
public ByteBuffer getResponse(byte[] request) throws Exception {
int lenth = 500;
InetSocketAddress isa = new InetSocketAddress("ip", port);
SocketChannel channel = SocketChannel.open(isa);
ByteBuffer requestBuffer = ByteBuffer.wrap(request);
channel.write(requestBuffer);
ByteBuffer responseByteBuffer = ByteBuffer.allocate(length);
int num = 0;
while (num == 0 ) {
responseByteBuffer.rewind();
num = channel.read(responseByteBuffer);
channel.close();
return responseByteBuffer;
Thanks.

Two possible solutions
1. Read in as many bytes as you can until you hit the EOS OR you fill the byte buffer. If you fill the buffer (try checking for BufferUnderFlowException), you could move the read bytes into a dynamically allocated memory Collection such as an ArrayList. When you hit the EOF, you could then do whatever else you need to do from then on.
2. I use this option: Implement a site local protocol for transmitting data. Provide a header first containing 2 ints (compressed size, uncompressed size - so you can also use compression when sending), and send the remainder as a payload.
Therefore
Read in the header first (always 8 bytes)
Then extract those integer values. They will tell you how much is being sent. Set you byte buffer for the payload to be the correct size for this transmission.
Reset and wait for next connection

What is the easiest way to read fixed width ascii files?

There does not seem to be an easy way to read fixed with ascii files in labview. CSV files and other delimited files are easy, but I cannot think of a simple way to read data from a fixed width ascii file.

jgkempen wrote:
There does not seem to be an easy way to read fixed with ascii files in labview.
Reading a file does not depend on its contents. Unless the file is gigantic, you can just read it as one big string and parse it later, especiall if you want the entire contents of the file.
If you only need a subset, you can calculate the offsets and sizes as multiples of your record size. That's the beauty of fixed size records.
It is typically not efficient to read a file in a loop, small portions at a time, so I would not recommend that except in some very specific situations.
Could you be a bit more specific on what exactly you need to do? Is your problem with reading or with parsing the fixed sized records? Could you attach a sample file and some description on what you need to read from it.
Message Edited by altenbach on 09-05-2007 07:50 PM
LabVIEW Champion . Do more with less code and in less time .

Problem while reading data from AL11

Hi Experts,
I am using OPEN DATASET
READ DATASET
CLOSE DATASET to read data from the application sever. In application sever the file type is .CSV. So after reading data from application layer(AL11) to a text variable, using statement ' SPLIT AT ',' ' all the values passing into correspoding internal table fields. And it getting updated successfully into DB also.
In above scenario my question is, if aquantity figure with coma seperator( 95,000,000.000), then while SPLIT it's creating problem. Same case in .XLS file, it will be created # seperator in the application layer. if any field having # then it will create problem again.
So can anybody suggest how can we over come the problem. Or is there any other way to read data properly from application layer(AL11)?
Thanks and regards,
Phani.

to overcom this problem,
normally we use tab as sparator
so use CL_ABAP_CHAR_UTILITIES=>VERTICAL_TAB to separate your fields!
grx
Andreas

Some way to read datafiles ?

Hello All,
Might someone know a handy way to read data from data files? I have the database files from backup of my iPlanet Ldap server (like __db.001, __db.002 etc). The iPlanet installation is corrupted, and I'm looking for some way to dig out some data. I believe these are Berkeley database files. I wonder if it is possible to read these files somehow? Opening in some viewer or editor, import into some other BerkeleyDB instance ??
thanks,
JM

Hi,
Hello All,
Might someone know a handy way to read data from data
files? I have the database files from backup of my
iPlanet Ldap server (like __db.001, __db.002 etc).The above files are Berkeley DB region files those are not the database files.
The database files may be named something like .db or whatever convention the iPlanet Ldap application uses to name Berkeley DB Databases.
Did you look for specific recovery procedures for the Ldap server?
If you can find the database files, you can try to use the Berkeley DB utilities to read the data.
You might start with db_verify to verify the database and db_dump to dump out the data to a sequentail file. If the databases are corrupt, you may need to look at the -r or -R flags. Here is a documentation link for the current release:
http://www.sleepycat.com/docs/utility/db_dump.html
We didn't write the application but you should probably check any web sites support forums for LDAP since the may have documented procedures for doing this or other users that have posted what they did.
Regards,
Ron Cohen
Berkeley DB Support
Oracle Corporation

Alternative to webservice for data exchange in PDF (Form Export Rights in Reader)

Hello,
I'm exchanging data between my pdf and a web server. Basically, the pdf needs to get some xml string from the web server, alternatively send some xml string to it. This is now done using a webservice, and it works fine with the SOAP javascript object. However, it doesn't work in Adobe Reader, because (quote from the docs):
"The SOAP methods connect, request and response are available only for documents open in Acrobat Pro and Acrobat Standard and for documents with Form Export Rights open in Adobe Reader 6.0 or later."
It turns out I need to enable Form Export Rights to get webservices work in Reader.
Now, I was wondering if there's another way to exchange data between my pdf and a web server that doesn't involve enabling those Form Export Rights in Reader.
Note that I need to be able to parse the response string xml from the web server in pdf using its javascript API.
Thanks for any suggestions,
James

Your only other alternative is to POST th einformation to the web server via a submit operation. This is the same as POSTING on an HTML form.
Paul

Alternative way to read data

Similar Messages

Maybe you are looking for