Handling external text files

Hi experts!
Would you give me some advice to find the best way for handling the situation below?
Suppose I have a measuring system with 100 sensors. The measuring system can provide the measured values (with timestamp) through a simple text file (large amount of datas - one measuring session gives ~25,000 rows. The next session generates a new file: eg. data0001.csv, data0002.csv...).
Suppose, this file will be deposit:
A) somewhere on the db server (10g);
B) somewhere on the data collector PC.
The task is a data visualization on the datas already inserted into db table with Oracle ApEx. A package has to perform the observation of the new data files, the data insertion (and the visualization of course, but that I can manage).
Would you be so kind to help finding the shortest way for this?
What is the best practice (external table, sql*loader, utl_file...) for case A) and B), to create a "new data file listener" and "new data insertion" method?
Thanks, ML

user8353169 wrote:
Would you be so kind to help finding the shortest way for this?Not sure what you define as the "+shortest+" way - but the best way would be not to write to files, but to write the sensor data directly into the database.
The reasons are performance and scalability. One of the slowest operation we always had and still have on computer systems is I/O.
Let's say you need to process 6GB of sensor data per hour. Thus you will have 6GB worth of I/O per hour. This data then needs to be read (another 6GB I/O) and loaded (another 6GB written) into Oracle - where it is also indexed (more I/O) and processed (likely you will have reporting tables or materialised views for your Apex reports - and this also needs I/O).
Simply getting the data into the database using this method is 18+ GB I/O for 6 GB data. How well does this solution perform and how well does it scale when it has an I/O to data ratio of 3:1?
Thus the better option is to write that 6GB/hour of sensor data directly into the database. And not to go the write-to-file and then load-from-file approach. A single client process (collecting sensor data) can easily write over 1MB/sec (2000+ rows/sec) into the database (based on my experience doing a similar type of thing). Of course - table type, indexes, triggers and so on impacts insert performance.
The advantage of streaming the data directly into Oracle is, besides the reduction in total I/O, that the sensor data is almost immediately available for reporting and processing. Depending on the row/sec created, you may want to have the client commit every 10 to 60 seconds. Immediately after such a commit the database processes can access this data - process it for alarm notification, real-time reporting, aggregate it for a reporting data layer and so on.
Doing this via text files instead.. I do not see any advantage in it. The best place for data is inside the database. Not outside. The sooner and faster the data can be put into the database, the better.

Similar Messages

Find Change through external text file

Hello folks
I am bit pretty in InDesign scripting so could you please look into this.
How can i change any particular text field in Indesign CS3 document from text file.
I do have find change script but for each InDesign document specific text file is assigned.
So each time i have to modify find change GREP property that is also repetetive of work. Is there any way to get find change information should be extract from external text file.
Many Tanks in advance

In the FindChangeByList script, you could customize the function myFindFile(myFilePath) {...} as to search the FindChangeList text file in the document location rather than the script location. That's an example. The question is: given a document, where will you have the corresponding FindChangeList?
@+
Marc

Add one character at a time from External Text file

Hi,
I'm using Flash 8, and I have created a standard Dynamic
field that pulls the text from an external text file using the
LoadVars function and it works fine.
To get the correct effect for the design, I would like to be
able to create the illusion that the characters are being typed out
when the page loads as if someone has started typing and also hear
the "Typewriter Clack" each time a character appears.
The only place I have found an example is on the Syphon
Filter Game website which seems to have created the illusion
perfectly:
http://www.syphonfilterdarkmirror-thegame.com/en_IE/
(Check out the bottom right section)
I could create a Movieclip and add a character each frame
but, I would like to be able to edit the text when required via a
text file or even ASP or PHP if neccessary.
Any help would be grateful.
Cheers

or here:
http://www.actionscripts.org/tutorials/beginner/Scripted_Typerwriter/index.shtml

How to load external text file into a Form?

Hi,
I made a menu with 1-5 in a Form and 5 external text files. I want the user who is able to view the text content of the text file when he chooses one of them from the menu. And the text file screen has a "Back" command button to let him go back.
How can I do it and can it support other languages such as Chinese and Japanese?
Thanks for help.
gogo

Sorry, I made the mistake about the subject, it should be loading local file, not external file through http.
I wrote a method but it throwed an exception when the midlet was run.
private void loadText()
try {
InputStream is = this.getClass().getResourceAsStream("/text.txt");
StringBuffer sb = new StringBuffer();
int chr;
while ((chr = is.read()) != -1)
sb.append((char) chr);
is.close();
catch (Exception e)
System.out.println("Error occurs while reading file");
I put the text.txt file in the same folder with the main file (extends midlet). How can I load the text content and display it on StringItem?
Thanks for any help.
gogo

Appending an external text file

Hi everyone. I need a little help. I'm trying to append an external text file. I'm able to read it using the "in.readLine()" and "BufferedReader" commands but can't figure how to append it. The reason being is I'm trying to add new information to each new line. For example my ext. text file looks something like this:
Homer Simpson 3489 Evergreen Terrace Springfield
Marge Simpson 3489 Evergreen Terrace Springfield
Bart Simpson 3489 Evergreen Terrace Springfield
Please note that I'm not using a string tokenizer to read the text file. If someone could please give me an example to append information to each new line I would greatly appreciate it. Also please note that the text file is being read into a vector to hold the information. Thank you.

open the FileWriter or FileOutputStream you use in "append" mode..
http://java.sun.com/j2se/1.4.1/docs/api/java/io/FileOutputStream.html
FileOutputStream(String name, boolean append)
          Creates an output file stream to write to the file
          with the specified name.http://java.sun.com/j2se/1.4.1/docs/api/java/io/FileWriter.html
FileWriter(String fileName, boolean append)
          Constructs a FileWriter object given a file name
          with a boolean indicating whether or not to append
          the data written.

SSIS best practice on importing External text files

Hi -
I am a fairly seasoned SSIS/ETL developer and I am struggling with the best architecure on how to import vendor files into a shared database. I see there being 2 methods with in importing files, and I'm really wanting input from senior
level SSIS developers on their thoughts of importing vendor files, I see there being 2 methods:
- Set up the ETL to match the format of the file, and if the format is invalid, the entire ETL is unable to process the file, various error handling will be set up to log that the file import failed, but at this point, the ETL can no longer continue
- Set up the ETL to just take a text file, NOT looking at the format within the file, bring all of the data into ONE column, and then parse through the data, using the given file delimter and log any issues at that point
I have done both methods and I think there are advantages and disadvantages to both. Hopefully I explained that well enough. Can anyone give me their thoughts, suggestions, experience, etc. on these 2 approaches of importing a file? Any input
is greatly appreciated.
Thanks!
Jenna G

It depends on how much control you've on the source end. If you can ,best thing always would be to prefix the metadata and create your package based on that. Any violation should be flagged as an error and reported back for correction with row/column details
etc.
If you've no way to fix the source, then only go for single column approach. But this can still prove to be a nightmare if the source format keeps on changing. Giving that flexibility to users can prove to be costly sometimes and you may have to spend quite
a bit of time trying to fix inconsistencies in the source.
So in my opinion best would be the former approach. Only thing is the format (matadata) has to be fixed only after sufficient discussions with all involved and exception handling also has to be agreed upon.
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Reading an external text file in WAR file

Greetings,
I have a question. I have an web application that writes a text message and 0's and 1's based on checkboxes. Then I have a JSP that reads this text file and prints the message and enables/disables links based on the 1's and 0's. I don't have any problems with this when testing from localhost, but when I deploy to a server the application isn't reading the file. The text file and WAR file are in the same folder, same directory. I've tried using both relative and absolute paths to read the file, but to no avail. Is there an easy way to read an external file from a WAR file or is there an easier solution. Thanks
Nick

You should not try to force your text file to live in the same directory as the WAR file. Instead, you should remove that kind of dependency and instead be using something like System.getProperty("java.io.tmpdir") to use the temporary directory to create and delete these files.
Edit: I also bet you don't even need to create & read a file here. You could probably store the information in one of the contexts of the web app, such as in a Session. I'll assume you know what that means.
By creating a file, you are also running the risk of multiple threads (simultaneous users) vying for the same file name, thus walking on each other's data.
Message was edited by:
warnerja

Scroll bar with external text file

Hello all,
I need to make a flash movie with a graphic background with
text from an external file.
Ideally:
1) The text file would NOT have to be imported into the flash
movie. It would simply be in the same web folder as the flash file.
2) Even better the text file would have a separate css file
so I can change font size etc, while I tweak the movie, and get
client approval.
If 1 and 2 are not possible, then I can live with importing
the text file. But CSS would sure be a nice addition!
BTW I have Flaxh MX Professional 2004.
Thanks
Rowby

Hey, I had to do a VERY similar thing for a student project a
few months ago. If you go to
Flashkit.com, you can find
tutorials on how to import XML and CSS or XSL. I was able to do it
and I don't know a lot of actionscript. If you want to see how I
used it, go to
the University of Scranton's MIT
website.
Hope that helps!

Accented Characters Support for External Text Files

Hi there! I created a site which imports external text from a
TXT file. The text is in Spanish, so I need accented vowels and
Ñs. I've tried using simple text and HTML text... but when I
run my movie, it doesn't show these international characters.
Instead, I get some weird symbols or question marks... What should
I do? Thanks in advance!

Hi,
Try this link it may have what your looking for it worked for
me!
http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=tn_14143
Danielle

How to access an external text file in a Servlet class?

I do the regular file I/O in the Servlet class. But when I run Servlet class, text file is not loaded. Can sb tell me how to do it?
Thanx
Richard

Hi DrClap;
the following is the main servlet code, which calls other two classes.
===================================================================
import java.io.*;
import java.util.*;
import javax.servlet.*;
import javax.servlet.http.*;
public class TermsServlet extends HttpServlet {
          public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
          resp.setContentType("text/html");
          PrintWriter out = resp.getWriter();
          out.println("<HTML>");
          out.println("<TITLE>Computer Terms and Acronyms</TITLE>");
          out.println("<BODY>");
          out.println("<H1>Computer Terms and Acronyms</H1>");
          out.println("<TABLE BORDER=2>");
          out.println("<TR><TH>Term<TH>Meaning</TR>");
          TermsAccessor tax = new TermsAccessor("terms.txt");
          Iterator e = tax.iterator();
          while (e.hasNext()) {
               Term t = (Term)e.next();
               out.print("<TR><TD>");
               out.print(t.term);
               out.print("<TD>");
               out.print(t.definition);
               out.println("</TR>");
          out.println("</TABLE>");
          out.println("<HR></HR>");
          out.println("</BODY></HTML>");
=================================================
import java.io.*;
import java.util.*;
public class TermsAccessor {
     protected BufferedReader is;
     protected String ident;
     public TermsAccessor(String inputFileName) throws IOException {
          is = new BufferedReader(new FileReader(inputFileName));
          String ident = is.readLine();
     public String getIdent() {
          return ident;
     public Iterator iterator() {
     return new Iterator() {
          String line, term, defn;
public boolean hasNext() {
          try {
          return ((line = is.readLine()) != null);
          } catch (IOException e) {
          System.err.println("IO Error: " + e);
          return false;
          public Object next() {
          int i;
          while ((i = line.indexOf("\t"))<0 && hasNext());
          if (line == null)
          throw new IllegalStateException("Invalid EOF state");
                    term = line.substring(0, i);
                    defn = line.substring(i+1);
                    return new Term(term, defn);
          public void remove() {
          throw new UnsupportedOperationException();
=============================================================
public class Term {
          public String term;
          public String definition;
          public Term(String t, String d) {
          term = t;
          definition = d;
=================================================================
part of [terms.txt] file as following:
$Id: terms.txt,v 1.5 2000/03/17 04:08:55 $
AMD     American Micro Devices: a chip maker that competes with Intel.
API     Application Programmer Interface: a set of functions that a programmer can use
ASP     Active Server Pages, a Microsoft technology for imbedding certain commands in HTML pages
BSD     Berkeley Software Distribution, one of the two major flavors of UNIX. See OpenBSD
C     A computer programming language invented for writing UNIX in, and very popular in the late 1970's through the 1990's
C++     An Object-Oriented language heavily based on C; recently displaced by Java
CGI     Common Gateway Interface; a script or program used to handle a form on a web page
COBOL     COmmon Business Oriented Language, a programming language used mainly for large-scale financial applications.
CPU     Central Processing Unit, the "chip" that gives a computer its low-level personality ("machine instruction set") and performs individual instructions (add, multiply, compare...). Common CPUs include Intel/AMD "PC", SPARC, Alpha.
=================================================================
the "terms.txt" file and above class files are under the same directory. I use J2EE server to deploy the file, after run, the erro mesage is:"The requested resource (terms.txt (The system cannot find the file specified)) is not available." That means server can not locate the terms.txt file. So what file path should I use?
Thanx
Richard

Way to convert clip markers in FCP HD to an external text file?

I am using FCP Final Cut Pro HD 4.5 and wanted to know if anyone has a solution for converting text in marker fields to simple text for Word or other text editor? I am using the name field in markers that are used in a subclip in my viewer window.

Yeah...
Create a new bin.
Open the clip's or subclip's triangle next to it in the Browser to reveal the markers there.
Then drag them to the new bin. This creates subclips of these markers named the same...
Control click on the bin to open it in it's own window. Delete all columns you don't want to export.
With the bin open and active, choose file/export/batch list. This will send a text file out... if you don't neec all the columns, open it in excel to delete them, or 'hide' them in FCP by right clicking on the column name at the top of the bin...
Open that file in Word.
Jerry

How to create and handle a text file in CVI

Is there any way to create and handle (like...open,write,read,truncate,close) the .txt file in CVI?

Sure. CVI is a regular C compiler and hence you can use all ANSI C functions. In addition you can use the File I/O functions from the utility library.

How to handle External Jar files in EJB DC

I have created a EJB DC.
I have added the required MDM Jar files to the class path.
These MDM Jar files are supplied by SAP for integration with Master Data Management.
But the DC build fails.
If I create the same project as a <b>REGULAR</b> EJB (non-dc approach)project, everything works fine.
Qn:
How to bundle the MDM jars (External jars)in a DC?
How to create the Public Parts in the DC for the added MDM External Jars?
How to use the above DC in the EJB DC?
Basically, I am looking for a step by step procedure,
to add external jars in a DC.
And to use the above DC in a EJB DC.
Any assistance in this regard is appreciated.

You should not reference the MDM jar files by setting classpath directly. For setting references to it from your EJB DCs , you will have to create a DC of type "External Library". Deploy this DC also before your EJBs are deployed. You will then have to create a public part for this DC and the reference the public part from the DC metadata of the EJB DC ie set the Used DC list of EJB DC to the public part of the ext lib. You will get info about this in NDS help as well on help.sap.com

Import external text file into table in database using web form

I whant to import data from text delimited file located in os
into oracle table using web form.
I am using ORACLE DATABASE 8i, APPLICATION SERVER 9i
and ORACLE FORMS DEVELOPER AND FORMS SERVER 6i (Patch 2)
Is there anybody who know how can i do this?
Thank you!

WebUtil uploads the files as Binary - so yes you could have some issues if you have a Unix host - however, that would only mean that there is an extra character to trim off of the end of the line read by Text_io.

ECATT is skipping lines in external text file

Hi,
I am using eCATT for uploading data. However, the ECATT is skipping some of the ROWS in my external File
Example
[VARIANT] [DESCRIPTION] Z_VAR1
1                                       HELLO
2                                       HELLO2
3                                       HELLO3
it skips row 2.
Please advice
Thanks!

Hi,
This is difficult to judge where the problem is.
Please check whether you have '' at the beginning of the line. Lines starting with '' are regarded as comments and would not be uploaded. For example, the following line of data would not be uploaded:
*2 HELLO2
Also, to ensure that your file has the right format for the eCATT tool to use, the best is to maintain one or two variants in your test configuration, then download the data to a local txt file. Afterwards use MS Excel to open it and maintain your data there. The purpose of creating one or two variants in your test configuration is to just give you an idea where to maintain what when opening the txt file with MS Excel. If you do not want to do it, it should not be a big problem either.
Hopefully it helps.

Handling external text files

Similar Messages

Maybe you are looking for