Saving to file before storing into a database in a REST service

A mobile back-end I am building should receive a larger amount of data (some sensor recordings) from an Android phone. The phone does not have much use of it so it was most efficient to store the data into plain files (SQLite can get quite slow with larger
amounts). Now when we upload the data to a REST service backed with a distributed database, we have another issue. Storing to it takes some time, since it is really a lot. I thought of uploading the file to the web service as fast as possible, leave the phone
alone, cache the file somewhere on the server and then some long-running workers will pick the data up and chunk it into databases. We have some other mechanisms to verify the data is properly stored and so on, but that is not important.
I would like to know is the REST->file->database approach valid? My concern is where to store the file? Disk, some in-memory database, a cache? Web servers fail (we can request a re-upload, but I would rather mitigate the risk earlier), or the server
can get cluttered. I'm afraid the local storage on the web servers does not scale really well, if we have multiple workers and servers.
Thank you in advance.

You could save the data to a local file like people have indicated and then upload to storage as a blob, alternatively you could also write directly to Table Storage. We have getting starteds for both of these here:
http://azure.microsoft.com/en-us/documentation/services/storage/ - just click on the Java tab.
The Android client library and introduction can be found here:
https://github.com/Azure/azure-storage-android
Jason
jason

Similar Messages

Virus scanning of files before storing into SAP

<b>Hi,
I'm about to implement BAdi VSCAN_INSTANCE to integrate virus scanner into the virus scan interface. I would apprecaite If anyone can provide me information on how to configure scanner groups, virus scan servers, virus scan profiles and implement virus scanning BAdi.
Thanks in advance.</b>

The Online documentation might help you:
http://help.sap.com/saphelp_erp2004/helpdata/en/20/51773f12f14a18e10000000a114084/frameset.htm
Also check OSS note 807989 & 797108.
Peter

Create XLS file of 12 lakhs rows stored into oracle database.

Respected All,
   My requirement is :
Into my database , in one of my table has 12 lakhs rows. Now I want to create a PLSQL procedure which can convert all those rows into Microsoft .XLS file.
So I tried out with Procedure below : -
CREATE OR REPLACE PROCEDURE SCOTT.EMPLOYEE_REPORT(
DIR IN VARCHAR2, FILENAME IN VARCHAR2) IS
F UTL_FILE.FILE_TYPE;
    CURSOR AVG_CSR IS
    SELECT ENAME, DEPTNO, SAL
    FROM EMP;
BEGIN
F := UTL_FILE.FOPEN(DIR, FILENAME,'W');
UTL_FILE.PUT_LINE(F, 'REPORT GENERATED ON ' ||SYSDATE);
UTL_FILE.NEW_LINE(F);
FOR EMP IN AVG_CSR
LOOP
UTL_FILE.PUT_LINE(F,
RPAD(EMP.ENAME, 30) || ' ' ||
LPAD(NVL(TO_CHAR(EMP.DEPTNO,'9999'),'-'), 5) || ' ' ||
LPAD(TO_CHAR(EMP.SAL, '$99,999.00'), 12));
END LOOP;
UTL_FILE.NEW_LINE(F);
UTL_FILE.PUT_LINE(F, '*** END OF REPORT ***');
UTL_FILE.FCLOSE(F);
END EMPLOYEE_REPORT;
COMMAND TO EXECUTE THE PROCEDURE IS : - >
EXEC EMPLOYEE_REPORT('UTL_FILE','TEST.XLS')
   in this package , I used scott user table for the r & d.
While I execute this package , I got the error :
ORA-29280: invalid directory path
ORA-06512: at "SYS.UTL_FILE", line 18
ORA-06512: at "SYS.UTL_FILE", line 424
ORA-06512: at "SCOTT.EMPLOYEE_REPORT", line 8
ORA-06512: at line 1
   KINDLY HELP ME TO SOLVE THIS PROBLEM
THANKS/ REGARDS
HARSH SHAH
URMIN GROUP OF COMPANIES

Hi,
first of all try to use international measures. Any time I see lakh I have to google to find what it is.
So you are trying to put in an Excel file 1.200.000 records (more than 1 million).
Actually even if you would produce such file Excel will not be able to load it.
Even in the latest version of Excel (2013) the maximum number of rows per sheet are 1,048,576 (source: Excel specifications and limits - Excel - Office.com).
So if you will try to load in Excel such a big file you will get an error.
And, as Blue has said, you are not producing an Excel file with your statement.
Regards.
Al

Saving files as BLOB into oracle database

I have files to insert into the oracle database as a BLOB field. Please give me some code to display and store the files into the database.
Cheers..,
aruni.

Is your suggested approach limited by Oracle's 4k limit? No it isnt limited to 4k ...Somehow that 4k chunk is stuck in the back of my mind as a limit... Anyway, I recalled some one posted a question regarding Oracle's BLOB/CLOB and was having difficulty either reading or writing so I've decided to poke around a little and this is what I came up with:
1) Oracle's maximum size of a BLOB/CLOB is said to be 4gb -- but I failed to see how that can be accomplished thru Java since the setBinaryStream and setAsciiStream methods require a signed int as a length specification.
2) In reading a LOB, the input stream has an internal MAX_CHUNK_SIZE set to 32768, in order to read anything bigger, it must be done in multiple chunks.
;o)
V.V.

Saving a text file as CLOB into oracle database

Hi techies,
I have a text file, which I have to insert into the oracle database as a CLOB field. Please give me some code to read the file and store the contents of the file into the database.
Thanks in advance
Yogesh

Hi,
Please find attached sample code to read and write the data to the CLOB data type in the database.
Here, the column name called "DATA" is defined as CLOB data type in the Database.
Program Name : ClobSample.java
JDK Version : jdk1.4
import java.util.Properties;
import java.util.Hashtable;
import java.sql.*;
import javax.sql.DataSource;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.util.*;
import java.io.*;
class ClobSample
     public static void main(String[] args) throws Exception
          Connection con = null;
          try
               con = getConnection();
               PreparedStatement ps = con.prepareStatement("INSERT INTO CLOBTEST(DATA) VALUES(?)");
               String data = getMyString("c://mani.txt");
               Reader reader = new StringReader(data);
               ps.setCharacterStream(1, reader, data.length());
               ps.executeUpdate();
               System.out.println("DATA Inserted !!!!");
               ps = con.prepareStatement("SELECT DATA FROM CLOBTEST");
               ResultSet rs = ps.executeQuery();
               Clob clob = null;
               while (rs.next())
                    clob = rs.getClob("DATA");
                    System.out.println(clob.getSubString(1,(int)clob.length()));
          catch(Exception e)
               e.printStackTrace();
          finally
               if (con!= null)
                    con.close();
     private static Connection getConnection()
          Connection con = null;
          try
               String url = "t3://10.3.26.16:9001";
               Hashtable env = new Hashtable();
               env.put(Context.INITIAL_CONTEXT_FACTORY, weblogic.jndi.WLInitialContextFactory.class.getName());
               env.put(Context.PROVIDER_URL, url);
               InitialContext initialContext = new InitialContext(env);
               DataSource ob = (DataSource)initialContext.lookup("mcone.journal.txds.ds");
               System.out.println("DataSource Object obtained--->"+ob);
               con =ob.getConnection();
          catch(Exception exc)
               exc.printStackTrace();
               System.exit(0);
          return con;
     public static String getMyString(String File)
          byte a[] = null;
          try
               InputStream fso = new FileInputStream(File);
               a = new byte[fso.available()];
               fso.read(a);
               fso.close();
          catch(Exception e)
               e.printStackTrace();
          System.out.println("DATA READ FROM THE FILE");
          return (new String( a,0,a.length));

Storing value in a grid before storing to a database

Apex 4
Oracle 10g EE
Good Day I have this problem that states that when a user click the add button the value in the textfield will be stored in a grid, when he enter another value and click the ADD button it will store again in the next line of the grid so you have already two values in the grid. When the user press the SAVE button all values in the grid must be stored in my database two rows in the grid means two rows also on my database. How will I do that in apex? Any examples out their that will guide me? I really need your help. thanks a lot.

thanks for the reply, I've been searching for collection but I can't really understand what will be the process for it. Do you have any examples related to this problem? I guess no one can make this, Ive been browsing for whole day and no one clears my mind on how to deal with this problem. Im not good in jquery either im just a VB lover but apex seems to hit my nerves. dude. Any one out here who is will to give examples bout this problem? thanks so much..

Saving image file as binary into database + pulling it out + displaying

Hi guys, this is exactly I have to do, and have no ideal how to do.
It will be greatly appreciated if you can offer help on this.
I don't have a problem putting it into and pulling it out fo the database, it is really the jpg-> binary, and binary->display part that I need help on.
Btw, I am currently using "draw Flatten PixMap.VI" to display a picture in labview, from a jpg file. Can I still use this VI to achieve what I intend to do?

Have you tried this:
Saving a binary file to a database with the Database Connectivity Toolset
Adnan Zafar
Certified LabVIEW Architect
Coleman Technologies

Reading from a txt file and storing into a list

i want to read a list of counters(String) which are there in a flat txt file and store them into a List.
Any suggestion will be helpful
sanjeev

Try this
try {
          FileReader fr = new FileReader("C:/abc.txt");
          BufferedReader br = new BufferedReader(fr);
          String record = null;
          while ((record = br.readLine()) != null)
               System.out.println(record );

Problem in saving the image into SQL database..

Hello,
In my application I have to save image file (say jpg) into the database and retrieve back and display it on the applet. I am through with grabbing the pixels of the image & sending the pixel array to the servlet but I am struck in how to store the image into the database and retrieve the same.
Can anybody please help me in this regard... its really urgent...
Thanks in advance
Swarna

Hello.
I've been researching this problem (saving images in a MySQL database) in order to accomplish a task I was assigned to. Finally I was able to do it. I'd be glad if it will be of any use.
First of all: the original question was related to an applet. So, the post from rkippen should be read. It says almost everything, leaving the code job for us. Since I managed to write some code, I'll put it here, but I'm not dealing with the byte transferring issue.
To obtain a byte array from a file I'd open the file with FileInputStream and use a loop to read bytes from it and save them into a ByteArrayOutputStream object. The ByteArrayOutputStream class has a method named �toByteArray()� which returns an array of bytes (byte [] b = baos.toByteArray()) that can be transferred in a socket connection, as said by rkippen.
My problem was to save an image captured by a web camera. I had an Image object, which I converted into a byte array and saved into the database. Eventually I had to read the image and show it to the user.
The table in the MySQL database could be:
CREATE TABLE test (
id int(11) NOT NULL auto_increment,
img blob NOT NULL,
PRIMARY KEY (id)
)I had problems trying to use the �setBlob� and �getBlob� methods in the Statement object, so I used the �setBytes� and �getBytes� methods . In the end, I liked these methods most because they where more suitable to my application.
The database operations are:
    public int insertImage(Image image) throws SQLException {
        int id = -1;
        String sql = "insert into test (img) values (?)\n";
        PreparedStatement ps = this.getStatement(sql); // this method is trivial
        byte [] bytes = this.getBytes(imagem); // * see below
        ps.setBytes(1, bytes);
        ps.executeUpdate();
        id = ps.getGeneratedKeys().getInt(0); //Actually I couldn't make this line work yet.
        return id;
    public Image selectImage(int id) throws SQLException {
        Image img = null;
        String sql = "select img from test where id = ?\n";
        PreparedStatement ps = getStatement(sql);
        ps.setInt(1, id);
        ResultSet rs = ps.executeQuery();
        if (rs.next()) {
            byte [] bytes = rs.getBytes(1);
            img = this.getImage(bytes); // * see below
        return img;
    }* If the bytes are read directly from a file, I think it is not necessary to convert it into an Image. Just send the bytes to the database method would work. On the other hand, if the image read form the DB will be written directly into files, the bytes obtained from rs.getBytes(1) would be enough.
The image operations are:
    public byte [] getBytes(Image image) {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        try {
            ImageIO.write(this.getBufferedImage(image), "JPEG", baos);
        } catch (IOException ex) {
            throw new RuntimeException(ex);
        return baos.toByteArray();
    public Image getImage(byte [] bytes) {
        Image image = null;
        ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
        try {
            image = ImageIO.read(bais);
        } catch (IOException ex) {
            throw new RuntimeException(ex);
        return image;
    public BufferedImage getBufferedImage(Image image) {
        int width = image.getWidth(null);
        int height = image.getHeight(null);
        BufferedImage bi = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
        Graphics2D g2d = bi.createGraphics();
        g2d.drawImage(image, 0, 0, null);
        return bi;
    }That's it. I hope it is useful.

Reading content to insert into a database for file content searches

I have a new challenge that has been brought to me.
There is and existing application that has files being inserted into the database and the customer is having great difficulty in being able to have a search capability against those files. I was thinking that I could do a secondary lookup before the file is actually uploaded and there would be use of the cffile to read the file then insert that data from the read and insert that data into a column of that database the equiled that record. The most important part of this is that the file's content is read from cffile and inserted into a column that actually get's that data into the RBDMS system that will allow for a full text search that would point to that system.
Any other ideas of how to insert document content into a data record other than using the cffile tag would be most appreciated.
Thanks in advance.
Gene

Most, if not all, database management systems have their own features to read text data from files. You may prefer that functionality over piping the data through ColdFusion and the database drivers.
<cffile...> can "read" .doc and .pdf file as in it can read the file from the file system as the binary data they are, but that does not extract the text from the content. You can do some stuff with CFML to read the text in a pdf file with the <cfpdf...> and related tags and functions. I'm not aware of similar CFML features that work with doc files, but I know of Java tools that do such as POI. These are usually fairly straight forward to tap into with ColdFusion.

Moving saved .eml files into Outlook

I have thousands of saved .eml files, to move into Outlook 2010. However - when I drag and drop them into the folders on Outlook 2010 - every file has the "date received" as today's date ( in past scenarios where I've done this - it automatically shows the
original email date).
i.e. - every file moved today - shows October 9th, 2011 as the date received - even though the email may have been from August 2009 for example.
Any thoughts on how to move the eml files into Outlook 2010 - with the original receive dates being displayed.
Thanks for your help.
TmaxPa

Actually - found this info in this forum. Thanks very very much to Diane Poremsky - a lifesaver !
( Essentially - eml files from Windows Mail - need to be converted to .pst in Outlook - then imported into Outlook)
Her info :
Outlook doesn't use EML files. From Windows Mail, use File, Export. Choose Exchange Server (it exports to Outlook). You
will need to be using 32 bit Outlook to do this.

Data encryption within Azure website storing in Azure database

I am wondering what the best way to encrypt/decrypt my documents (which will be stored within my database) while at rest. My web app will allow user to upload document and share them with other people so they can download them.
I want the encryption keys/intial vectors to be generated from code so there is no user input. Here is my current thought which is a hybrid of AES/RSA.
Create a new table called keys within my database which will look like the following...
ID | Key | Vector
then when a document is uploaded the Key/Vector is generated and used to encrypt the document and then the document is saved into the document table returning the document ID, which will be stored within the new table in the ID field along with the key and
vector.
Then i somehow need to protect the keys as anyone who grabs my database will have access to all the doucments still as the keys are there!
So i was thinking i would upload a private certificate into Azure storage and store the public certificate inside my web app so before the key/vector is saved my application will encrypt them with the public certificate.
Then when a user downloads a document, the app will connect to my Azure storage account and grab the private key, it will grab the row from key table associated with the document id they are trying to download and then it will use the private key to decrypt
the key/vector then it will use the key/vector to decrypt the document.
Does this seem the best way forward for how to do it, am i way off, or is there a security hole within this solution?

All your keys are still the same so long as you use one cert to do the work. (and they're all still in one place). If the app downloads the private key, well - the app and anyone in the middle has the key to figure out the rest and decrypt all the
data. You'll struggle to make good security just by adding levels of effort or middle steps like that.
What you're doing is kind of like storing the key to your house in your car, and the car's key is in the car's door lock. You've just added a single step to get inside your home.
If I have your DB, I probably have your code, which means I know how to do the rest, and since everything is public, so is the resultant data.
What I'd do is have Azure Storage holding the encrypted data, and a key share table. Use the DB as an index if you feel you need, but it's cheaper to go with just storage.
Each app would create, have, and hold privately, an AES private key. The app would somehow initially upload the public key for the user to the key share table via your service.
When user A wanted to send to user B, they would get B's public key from the key share. The App would encrypt A's Document and only then send the encrypted data to B's 'mailbox'. Later, B would come along and see a file was waiting. They'd download
the encrypted data and then decrypt it with their private key which has never left their app/device/wherever.
Think about making every private key 'private' to the entity owning it. Don't ever store/share/transmit a private key where any public entity can get it. That's the only way it'll work.
To make things even harder, you can do things to the data before document encryption, (and after) but never mess with the one golden rule of AES:
Keep "private" private, private!
Darin R.

Where is the enjoylogo.gif stored in the database

Can anyone tell me where these type of files are stored in the database? Are they in a table?

If you have 2 lists, List A contains custom names, List B has a lookup field that points to List A. I go into list B and select a client name ("Contoso"), enter some info, and save it. On the back end, the list item will contain a value of something like:
34#;Contoso
In this scenario the 34 is the SPListItem.ID value of that item in List A. They start at 1 in each list and increment by 1.
Also, running queries against the database directly is not supported and can take your database out of Microsoft's supportability.
Dimitri Ayrapetov (MCSE: SharePoint)

Inserting Multiple Images into oracle database using JDBC

I wanted to insert multiple images into database using JDBC by reading it from the file... and i am passing photos.txt(my text file) as an input parameter... I have inserted all the values into the database except for the image part... this is my content of photos.txt file and i have copied all the images into the folder
*" C:\\photos "*
*1,in1.jpg,108,19,in-n-out*
*2,in2.jpg,187,21,in-n-out*
*3,in3.jpg,308,41,in-n-out*
*4,in4.jpg,477,52,in-n-out*
*5,in5.jpg,530,50,in-n-out*
and i want to store in1.jpg,in2.jpg,in3.jpg,in4.jpg,in5.jpg into the oracle databse using JDBC.... i have tried a lot using BLOB column.... and i have created my table as
CREATE TABLE PHOTO(
ID NUMBER NOT NULL PRIMARY KEY ,
Name BLOB,
X DOUBLE PRECISION,
Y DOUBLE PRECISION,
Tags VARCHAR2(40)
try { // for restaurant System.out.println();System.out.println();System.out.println(); System.out.print(" Creating Statement for Photo...\n"); stmt2 = con.createStatement (); stmt2.executeUpdate("delete from PHOTO"); stmt2.executeUpdate("commit"); PreparedStatement stmt3 = con.prepareStatement ("INSERT INTO PHOTO VALUES (?, ?, ?, ?, ?)"); System.out.print(" Create FileReader Object for file: " + inputFileName1 + "...\n"); FileReader inputFileReader2 = new FileReader(inputFileName1); System.out.print(" Create BufferedReader Object for FileReader Object...\n"); BufferedReader inputStream2 = new BufferedReader(inputFileReader2); String inLine2 = null; String[] tokens; // String[] imageFilenames = {"c:\\photos\\in1.jpg","c:\\photos\\in2.jpg","c:\\photos\\in3.jpg","c:\\photos\\in4.jpg","c:\\photos\\in5.jpg", // "c:\\photos\\in6.jpg","c:\\photos\\in7.jpg","c:\\photos\\in8.jpg","c:\\photos\\in9.jpg","c:\\photos\\in10.jpg","c:\\photos\\arb1.jpg","c:\\photos\\arb2.jpg", // "c:\\photos\\arb3.jpg","c:\\photos\\arb4.jpg","c:\\photos\\arb5.jpg","c:\\photos\\den1.jpg","c:\\photos\\den2.jpg","c:\\photos\\den3.jpg", // "c:\\photos\\den4.jpg","c:\\photos\\den5.jpg","c:\\photos\\hop1.jpg","c:\\photos\\hop2.jpg","c:\\photos\\hop3.jpg","c:\\photos\\hop4.jpg","c:\\photos\\hop5.jpg"}; File file = new File("C:\\photos\\in1.jpg"); \\ ( Just for example ) FileInputStream fs = new FileInputStream(file); while ((inLine2 = inputStream2.readLine()) != null) { tokens= inLine2.split(","); st2 = new StringTokenizer(inLine2, DELIM); stmt3.setString(1, tokens[0]); stmt3.setBinaryStream(2, fs, (int)(file.length())); stmt3.setString(3, tokens[2]); stmt3.setString(4, tokens[3]); stmt3.setString(5, tokens[4]); stmt3.execute(); //execute the prepared statement stmt3.clearParameters();
As i am able to enter one image file by above code in1.jpg in to the oracle database.... but i am not able to insert all the image file in to the database.....do tell me what should i do.... and can you give me the example on the basis of the above code of mine...
do reply as soon as possible..

jwenting wrote:
that depends. Putting the images in BLOBs prevents the file locations stored in the database from getting out of synch with the filesystem when sysadmins decide to reorganise directory structures or "archive" "old" files that noone uses anyway.True, but it really comes down to a business decision (cost-benefit analysis). If you have the bucks, the expertise, and the time, go with the Blobs, otherwise go with the flat files.

While inserting values from a xml file into the database.

Dear Forum Members,
While using Samp10.java (given in XSU!2_ver1_2_1/oracleXSU12/Sample)for inserting values from xml file Sampdoc.xml into database table xmltest_tab1,the error shown below appears on the DOS prompt.
The code for sam10 is:
import oracle.xml.sql.dml.*;
import java.sql.*;
import oracle.jdbc.driver.*;
import oracle.jdbc.*;
import java.net.*;
public class samp10
public static void main(String args[]) throws SQLException
String tabName = "xmltest_tab1";
String fileName = "sampdoc.xml";
try{
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");
Connection conn=DriverManager.getConnection("jdbc:odbc:BookingSealinerScott","scott","tiger");
OracleXMLSave sav = new OracleXMLSave(conn, tabName);
URL url = sav.createURL(fileName);
int rowCount = sav.insertXML(url);
System.out.println(" successfully inserted "+rowCount+
" rows into "+ tabName);
conn.close();
}catch (Exception e){e.printStackTrace();}
The Structure of Sampdoc.xml is:
<?xml version="1.0"?>
<ROWSET>
<ROW num="1">
<EMPNO>7369</EMPNO>
<ENAME>SMITH</ENAME>
<JOB>CLERK</JOB>
</ROW>
<ROW num="2">
<EMPNO>7499</EMPNO>
<ENAME>ALLEN</ENAME>
<JOB>SALESMAN</JOB>
</ROW>
<ROW num="3">
<EMPNO>7521</EMPNO>
<ENAME>WARD</ENAME>
<JOB>SALESMAN</JOB>
</ROW>
</ROWSET>
Description of table xmltest_tab1 is:
SQL> desc xmltest_tab1;
Name Null? Type
EMPNO NUMBER(4)
ENAME CHAR(10)
JOB VARCHAR2(9)
Error Displayed is:
A nonfatal internal JIT (3.00.078(x)) error 'Structured Exception(c0000005)' has
occurred in :
'oracle/xml/sql/dml/OracleXMLSave.cleanLobList ()V': Interpreting method.
Please report this error in detail to http://java.sun.com/cgi-bin/bugreport.cgi
oracle.xml.sql.OracleXMLSQLException: sun.jdbc.odbc.JdbcOdbcConnection
at oracle.xml.sql.dml.OracleXMLSave.saveXML(OracleXMLSave.java:1967)
at oracle.xml.sql.dml.OracleXMLSave.saveXML(OracleXMLSave.java:1880)
at oracle.xml.sql.dml.OracleXMLSave.insertXML(OracleXMLSave.java:1013)
at samp10.main(samp10.java:36)
Press any key to continue . . .
Please send me the solution as soon as possible.
Thanks,
Waiting for your Reply,
Bye,
Vineet Choudhary
Email id: [email protected]
null

Go and read about JDBC. You need to know some basics before asking such a st&*id questions.
Paul

Saving to file before storing into a database in a REST service

Similar Messages

Maybe you are looking for