Out.println() problems with large amount of data in jsp page

I have this kind of code in my jsp page:
out.clearBuffer();
out.println(myText); // size of myText is about 300 kbThe problem is that I manage to print the whole text only sometimes. Very often happens such that the receiving page gets only the first 40 kb and then the printing stops.
I have made such tests that I split the myText to smaller parts and out.print() them one by one:
Vector texts = splitTextToSmallerParts(myText);
for(int i = 0; i < texts.size(); i++) {
  out.print(text.get(i));
  out.flush();
}This produces the same kind of result. Sometimes all parts are printed but mostly only the first parts.
I have tried to increase the buffer size but neither that makes the printing reliable. Also I have tried with autoFlush="false" so that I flush before the buffer size gets overflowed; again same result, sometimes works sometimes don't.
Originally I use such a system where Visual Basic in Excel calls a jsp page. However, I don't think that this matters since the same problems occur if I use a browser.
If anyone knows something about problems with large jsp pages, I would appreciate that.

Well, there are many ways you could do this, but it depends on what you are looking for.
For instance, generating an Excel Spreadsheet could be quite easy:
import javax.servlet.*;
import javax.servlet.http.*;
import java.io.*;
public class TableTest extends HttpServlet{
     public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException {
          response.setContentType("application/xls");
          PrintWriter out = new PrintWriter(response.getOutputStream());
                out.println("Col1\tCol2\tCol3\tCol4");
                out.println("1\t2\t3\t4");
                out.println("3\t1\t5\t7");
                out.println("2\t9\t3\t3");
          out.flush();
          out.close();
}Just try this simple code, it works just fine... I used the same approach to generate a report of 30000 rows and 40 cols (more or less 5MB), so it should do the job for you.
Regards

Similar Messages

  • How do I pause an iCloud restore for app with large amounts of data?

    I am using an iPhone app which is holding 10 Gb of data (media files) .
    Unfortunately, although all data was backed up, my iPhone 4 was faulty and needed to be replaced with a new handset. On restore, the 10Gb of data takes a very long time to restore over wi-fi. If interrupted (I reached the halfway point during the night) to go to work or take the dog for a walk, I end up of course on 3G for a short period of time.
    Next time I am in a wi-fi zone the app is restoring again right from the beginning
    How does anyone restore an app with large amounts of data or pause a restore?

    You can use classifications but there is no auto feature to archive like that on web apps.
    In terms of the blog, Like I have said to everyone that has posted about blog preview images:
    http://www.prettypollution.com.au/business-catalyst-blog
    Just one example of an image at the start of the blog post rendering out, not hard at all.

  • Error in Generating reports with large amount of data using OBIR

    Hi all,
    we hve integrated OBIR (Oracle BI Reporting) with OIM (Oracle Identity management) to generate the custom reports. Some of the custom reports contain a large amount of data (approx 80-90K rows with 7-8 columns) and the query of these reports basically use the audit tables and resource form tables primarily. Now when we try to generate the report, it is working fine with HTML where report directly generate on console but the same report when we tried to generate and save in pdf or Excel it gave up with the following error.
    [120509_133712190][][STATEMENT] Generating page [1314]
    [120509_133712193][][STATEMENT] Phase2 time used: 3ms
    [120509_133712193][][STATEMENT] Total time used: 41269ms for processing XSL-FO
    [120509_133712846][oracle.apps.xdo.common.font.FontFactory][STATEMENT] type1.Helvetica closed.
    [120509_133712846][oracle.apps.xdo.common.font.FontFactory][STATEMENT] type1.Times-Roman closed.
    [120509_133712848][][PROCEDURE] FO+Gen time used: 41924 msecs
    [120509_133712848][oracle.apps.xdo.template.FOProcessor][STATEMENT] clearInputs(Object) is called.
    [120509_133712850][oracle.apps.xdo.template.FOProcessor][STATEMENT] clearInputs(Object) done. All inputs are cleared.
    [120509_133712850][oracle.apps.xdo.template.FOProcessor][STATEMENT] End Memory: max=496MB, total=496MB, free=121MB
    [120509_133818606][][EXCEPTION] java.net.SocketException: Socket closed
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at weblogic.servlet.internal.ChunkOutput.writeChunkTransfer(ChunkOutput.java:525)
    at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:504)
    at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:382)
    at weblogic.servlet.internal.ChunkOutput.checkForFlush(ChunkOutput.java:469)
    at weblogic.servlet.internal.ChunkOutput.write(ChunkOutput.java:304)
    at weblogic.servlet.internal.ChunkOutputWrapper.write(ChunkOutputWrapper.java:139)
    at weblogic.servlet.internal.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:169)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
    at oracle.apps.xdo.servlet.util.IOUtil.readWrite(IOUtil.java:47)
    at oracle.apps.xdo.servlet.CoreProcessor.process(CoreProcessor.java:280)
    at oracle.apps.xdo.servlet.CoreProcessor.generateDocument(CoreProcessor.java:82)
    at oracle.apps.xdo.servlet.ReportImpl.renderBodyHTTP(ReportImpl.java:562)
    at oracle.apps.xdo.servlet.ReportImpl.renderReportBodyHTTP(ReportImpl.java:265)
    at oracle.apps.xdo.servlet.XDOServlet.writeReport(XDOServlet.java:270)
    at oracle.apps.xdo.servlet.XDOServlet.writeReport(XDOServlet.java:250)
    at oracle.apps.xdo.servlet.XDOServlet.doGet(XDOServlet.java:178)
    at oracle.apps.xdo.servlet.XDOServlet.doPost(XDOServlet.java:201)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
    at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
    at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
    at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at oracle.apps.xdo.servlet.security.SecurityFilter.doFilter(SecurityFilter.java:97)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
    at weblogic.security.service.SecurityManager.runAs(Unknown Source)
    at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
    at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
    at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    It seems where the querry processing is taking some time we are facing this issue.Do i need to perform any additional configuration to generate such reports?

    java.net.SocketException: Socket closed
         at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
         at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
         at weblogic.servlet.internal.ChunkOutput.writeChunkTransfer(ChunkOutput.java:525)
         at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:504)
         at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:382)
         at weblogic.servlet.internal.CharsetChunkOutput.flush(CharsetChunkOutput.java:249)
         at weblogic.servlet.internal.ChunkOutput.checkForFlush(ChunkOutput.java:469)
         at weblogic.servlet.internal.CharsetChunkOutput.implWrite(CharsetChunkOutput.java:396)
         at weblogic.servlet.internal.CharsetChunkOutput.write(CharsetChunkOutput.java:198)
         at weblogic.servlet.internal.ChunkOutputWrapper.write(ChunkOutputWrapper.java:139)
         at weblogic.servlet.internal.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:169)
         at com.tej.systemi.util.AroundData.copyStream(AroundData.java:311)
         at com.tej.systemi.client.servlet.servant.Newdownloadsingle.producePageData(Newdownloadsingle.java:108)
         at com.tej.systemi.client.servlet.servant.BaseViewController.serve(BaseViewController.java:542)
         at com.tej.systemi.client.servlet.FrontController.doRequest(FrontController.java:226)
         at com.tej.systemi.client.servlet.FrontController.doPost(FrontController.java:128)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
         at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3498)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(Unknown Source)
         at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:17
    (Please help finding a solution in this issue its in production and we need to ASAP)
    Thanks in Advance
    Edited by: 909601 on Jan 23, 2012 2:05 AM

  • Working with large amount of data

    Hi! I am writing an peer-to-peer video player and I need to operate with huge amounts of data. The downloaded data should be stored in memory for sharing with other peers. Some video files can be up to 2 GB and more. So keep all this data in RAM - not the best solution i think =)
    Since the flash player does not have access to the file system I can not save this data in temporary files.
    Is there a solution to this problem?

    No ideas? very sad ((

  • Dealing with large amounts of data

    Hi
    I am new to using Flex and BlazeDS. I can see in the FAQ that binary data transfer from server to Flex app is more efficient. My question is: is there a way to build a Flex databound control (e.g. datagrid) which binds to a SQL query or web service or remoting on the server side and then display infinite amount of data as user pages or scrolls using scrollbar? or does the developer have to write code from scratch to deal with paging or deal with infinite scrollbar by asking server for chunks of data at a time?

    You have to write your own paginating grid. It's easy to do, just make a canvas, throw a grid and some buttons on it, than when user click to the next page, you make a request to the server and when you have a result, set it as a new data model for the grid.
    I would discourage you to return infinite amount of data to the user, provide a search functionality plus pagination.
    Hope that helps.

  • Ni reports crashes with large amounts of data

    I'm using labview 6.02 and am using some of the Ni reports tools to generate a report. I create, what can be, a large data array which is then sent to the "append table to report" vi and printed. The program works fine, although it is slow, as long as the data that I send to the "append table to report" vi is less than about 30kb. If the amount of data is too large Labview terminates execution with no error code or message displayed. Has anyone else had a similar problem? Does anyone know what is going on or better yet how to fix it?

    Hello,
    I was able to print a 100x100 element array of 5-character strings (~50 kB of data) without receiving a crash or error message. However, it did take a LONG time...about 15 minutes for the VI to run, and another 10 minutes for the printer to start printing. This makes sense, because 100x100 elements is a gigantic amount of data to send into the NI-Reports ActiveX object that is used for printing. You may want to consider breaking up your data into smaller arrays and printing those individually, instead of trying to print the giant array at once.
    I hope these suggestions help you out. Good luck with your application, and have a pleasant day.
    Sincerely,
    Darren N.
    NI Applications Engineer
    Darren Nattinger, CLA
    LabVIEW Artisan and Nugget Penman

  • XY graphing with large amount of data

    I was looking into graphing a fairly substantial amount of data.  It is bursted across serial.
    What I have is 30 values corresponding to remote data sensors.  The data for each comes across together, so I have no problem having the data grouped.  It is, effectively, in an array of size 30.  I've been wanting to place this data in a graph.
    The period varies between 1/5 sec and 2 minutes between receptions (its wireless and mobile, so signal strength varies).  This eliminates waveform graph as the time isn't constant and there's alot of data(So no random NaN insertion).  This leaves me with an XY graph.
    Primary interest is with the last 10 minutes or so, with a desire to see up to an hour or two back.
    The data is farily similar, and I'm tring to possibly split it into groups of 4 or 5 sets of ordered pairs per graph.
    Problems:
    1.  If data comes in slow enough, everything is ok, but the time needed to synchrounously update the graph(s) often can exceed the time it would take to fully recieve the chunk of data which contains these data points.  Thinking asynchrounously is useless, as the graphs need to be reasonably in tune with the most recent data recieved.  I can't have the an exponential growth in the delta of time represented on the graph and the time the last bit of data was recieved.
    2.  I could use some advice on making older data points more sparse to allow for older data to be viewed, but with a sort of 'decay' of old data I don't value that 1/5 second resolution at all.
    I'm most concerned with solving problem 1, but random suggestions on 2 are most welcome.

    I didn't quite get the first question. Could you try to find out where
    exactly the time is consumed in your program and then refine your
    question.
    To the second question (which may also solve the first question). You
    can store all the data to a file as it arrives. Keep the most recent
    data in a let's say shift register of the update loop. Add a data point
    corresponding the start of the mesurement to the data and wire it to a
    XY graph. Make X Scrollbar visible. Handle the event for XY graph:X
    scale change. When the scale is changed i.e. the user scrolls the
    scroll bar, load data from the file and display it on the XY graph.
    This was a little simplified, it's a little more complicated.
    In my project, I am writing an XY graph X Control to handle a bit
    similar issue. I have huge data sets i.e. multichannel recordings 
    with millisecond resolution over many hours. One data set may be
    several gigabytes, so the data won't fit on the main memory of the
    computer at once. I wrote an X control which contains a XY graph. Each
    time the time scale is changed i.e. the scroll bar is scrolled, the X
    control generates a user event if it doesn't have enough data to
    display the time slice requested. The user event is handled outside the
    X Control by loading the appropriate set of data from the disk and
    displaying it on the X Control. The user event can be generated already
    a while before the X Control is out of data. This way the data is
    loaded a bit in advance, which allows seamles scrolling on the XY
    graph. One must notice that the front panel updates must be turned off
    in the X Control when the data is updated and back on after the update
    has finnished. Otherwise the XY graph will flicker annoingly.
    Tomi Maila

  • Converting tdm to lvm/ working with large amount of data

    I use a PCI 6251 card for data aquisition, and labview version 8; to log 5 channels at 100 Khz for approximately 4-5 million samples on each channel (the more the better). I use the express VI for reading and writing data which is strored in .tdm format (file size of the .tdx file is around 150 MB). I did not store it in lvm format to reduce the time taken to aquire data.
    1. how do i convert this binary file to a .mat file ?
    2. In another approach,  I converted the tdm file into lvm format, this works as long as the file size is small (say 50 MB) bigger than that labview memory gets full and will not save the new file. what is an efficient method to write data (say into lvm format) for big size files without causing labview memory to overflow? I tried saving to multiple files, saving one channel at a time, increased the computer's virtual memory (upto 4880 MB) but i still have problems with 'labview memory full' error.
    3.  Another problem i noticed with labview is that once it is used to aquire data, it occupies a lot of the computer's memory, even after the VI stops running, is ther a way to refresh the memory and is this mainly due to  bad programming?
    any suggestions?

    I assume from your first question that you are attempting to get your data into Matlab.  If that is the case, you have three options:
    You can treat the tdx file as a binary file and read directly from Matlab.  Each channel is a contiguous block of the data type you stored it in (DBL, I32, etc.), with the channels in the order you stored them.  You probably know how many points are in each channel.  If not, you can get this information from the XML in the tdm file.  This is probably your best option, since you won't have to convert anything.
    Early versions of TDM storage (those shipping with LV7.1 or earlier) automatically read the entire file into memory when you load it.  If you have LV7.1, you can upgrade to a version which allows you to read portions of the file by downloading and installing the demo version of LV8.  This will upgrade the shared USI component.  You can then read a portion of your large data set into memory and stream it back out to LVM.
    Do option 2, but use NI-HWS (available on your driver CD under the computer based instruments tab) instead of LVM.  HWS is a hierarchical binary format based on HDF5, so Matlab can read the files directly through its HDF5 interface.  You just need to know the file structure.  You can figure that out using HDFView.  If you take this route and have questions, reply to this post and I will try to answer them.  Note that you may wish to use HWS for your future storage, since its performance is much better than TDM and you can read it from Matlab.  HWS/HDF5 also supports compression, and at your data rates, you can probably pull this off while streaming to disk, if you have a reasonably fast computer.
    Handling large data sets in LabVIEW is an art, like most programming languages.  Check out the tutorial Managing Large Data Sets in LabVIEW for some helpful pointers and code.
    LabVIEW does not release memory until a VI exits memory, even if the VI is not running.  This is an optimization to prevent a repeatedly called VI from requesting the same memory every time it is called.  You can reduce this problem considerably by writing empty arrays to all your front panel objects before you exit your top level VI.  Graphs are a particulary common problem.
    This account is no longer active. Contact ShadesOfGray for current posts and information.

  • Problem retrieving large amount of data!

    Hi,
    I'm currently working with a database application accessing very big amount of data. One query could result in 500 000+ hits!
    I would like to present this data in a JTable.
    When the query is executed, I create a two-dimensional array and store the data in it. The problem is that I get OutOfMemoryError when I reach the 150 000th row and add it to the array.
    I've looked into the design pattern "Value List Handler" and it seems it could be of use. But still I need to populate an array with all the data, and then I get the error.
    Is there somehow I could query the database, populate part of the data in a smaller array, use the "Value List Handler" pattern to access small portions of the complete resultset?
    Another problem is that the user wants ability to sort asc/desc by clicking columnheaders in the JTable. The I need to access all data in that table to make it be sorted right. Could re-query the database with a "ORDER BY <column> ASC" and use a modification of the value list handler pattern?
    I'm a bit confused, please help!
    Kind regards, Andreas

    The only chance that I you have: only select as many rows as you display on the screen. When the user hits "next page" retrieve the next rows.
    You might be able to do this with a scrollable resultset, but with that you are left to the mercy of the JDBC driver concerning memory management.
    So you need to think about a solution where you issue a new SELECT narrowing down the result set depending on the first/last row displayed
    Search this forum for pagewise navigation this question gets asked about 5 times a week (mostly in conjunction with web applications, but the logic behind it, should be the same).
    Tailoring your TableModel might be tricky as well.
    Thomas

  • How to deal with large amount of data

    Hi folks,
    I have a web page that needs 3 maps to serve. Each map will hold about 150,000 entries, and all of them will use about 100MB total. For some users with lots of data (e.g. 1,000,0000 entries), it may use up to 1GB of memory. If few of these high-volume users log in at the same time, it can bring the server down. The data is from the files, I cannot read it on demand because it will be too slow. Loading the data to maps offers me very good performance, but it does not scale. I am thinking to serialize the maps and deserialize them when I need. Is it my only option?
    Thanks in advance!

    JoachimSauer wrote:
    I don't buy the "too slow" argument.
    I'd put the data into a database. They are built to handle that amount of data.++

  • Putting different tables with large amounts of data together

    Hi,
    I need to put different kinds of tables together:
    Example 'table1' with columns: DATE, IP, TYPEN, X1, X2, X3
    To 'table0' with columns DATE, IP, TYPENUMBER.
    TYPEN in table1 needs to be inserted into TYPENUMBER in table0, but through a function which transforms it to some other value.
    There are several other tables like 'table1', but with slighty different columns, that needs to be inserted into the same table ('table0').
    The amount of data in each table is quite huge, so the procedure should be done in small pieces and efficiently.
    Should/Could I use data pump for this?
    Thank you!

    user13036557 wrote:
    How should I continue with this then?
    Should I delete the columns I don't need and transform the data in the table first and then use data pump,
    or should I simply make a procedure going through every row (in smaller pieces) of 'table1' and inserting it to 'table0'?You have both the options .. Please test both of them , calculate time to complete and implement the best .
    Regards
    Rajesh

  • Loosing indexes working with large amounts of data

    SCENARIO
    We are working on an Interface project with ORACLE 10g that works basically like this:
    We have some PARAMETER TABLES in which the key users can insert, update or delete parameters via a web UI.
    There is a download process that brings around 20 million records from our ERP system into what we call RFC TABLES. There are around 14 RFC TABLES.
    We developed several procedures that process all this data against the PARAMETER tables according to some business rules and we end up with what we call XML TABLES because they are sent to another software, completing the interface cycle. We also have INTERMIDIATE TABLES that are loaded in the middle of the process.
    The whole process takes around 2 hours to run.
    We had to create several indexes to get to this time. Without the indexes the process will take forever.
    Every night the RFC, INTERMIDIATE and XML tables need to be truncated and then loaded again.
    I know It might seem strange why we delete millions of records and then load them again. The reason is that the data the users insert in the PARAMETER TABLES need to be processed against ALL data that comes from the ERP and goes to the other software.
    PROBLEMS
    As I said we created several indexes in order to make the process run in less than 2 hours.
    We were able to run the whole process in that time for a few times but, suddenly, the process started to HANG forever and we realized some indexes were just not working anymore.
    When running EXPLAIN we figured the indexes were making no effect and we had some ACCESS FULLS. Curiously when taking the HINTS off and putting them back in the indexes started working again.
    SOLUTION
    We tried things like
    DBMS_STATS.GATHER_SCHEMA_STATS(ownname => SYS_CONTEXT('USERENV', 'CURRENT_SCHEMA'), cascade=>TRUE);
    dbms_utility.analyze_schema
    Dropping all tables and recreating the every time before the process starts
    Nothing solved our problem so far.
    We need advice from someone that worked in a process like this. Where millions of records are deleted and inserted and where a lot of indexes are needed.
    THANKS!
    Jose

    skynyrd wrote:
    I don't know anything about
    BIND variables
    Stored Outlines
    bind peeking issue
    or plan stability in the docs
    but I will research about all of them
    we are currently running the process with a new change:
    We put this line:
    DBMS_STATS.GATHER_SCHEMA_STATS(ownname => SYS_CONTEXT('USERENV', 'CURRENT_SCHEMA'), cascade=>TRUE);
    after every big INSERT or UPDATE (more than 1 million records)
    It is running well so far (it's almost in the end of the process). But I don't know if this will be a definitive solution. I hope so.
    I will post here after I have an answer if it solved the problem or not.
    Thanks a lot for your help so farWell, you best get someone in there that knows what those things are, basic development, basic performance tuning and basic administration all are predisposed on understanding these basic concepts, and patching is necessary (unless you are on XE). I would recommend getting books by Tom Kyte, he clearly explains the concepts you need to know to make things work well. You ought to find some good explanations of bind peeking online if you google that term with +Kyte.  
    You will be subject to this error at random times if you don't find the root cause and fix it.
    Here is some food for your thoughts:
    http://structureddata.org/2008/03/26/choosing-an-optimal-stats-gathering-strategy/ (one of those "what to expect from the 10g optimizer" links does work)
    http://kerryosborne.oracle-guy.com/2009/03/bind-variable-peeking-drives-me-nuts/
    http://pastebin.com/yTqnuRNN
    http://kerryosborne.oracle-guy.com/category/oracle/plan-stability/
    Getting stats on the entire schema as frequently as you do may be overkill and time wasting, or even counterproductive if you have an issue with skewed stats. Note that you can figure out what statistics you need and lock them, or if you have several scenarios, export them and import them as necessary. You need to know exactly what you are doing, and that is some amount of work. It's not magic, but it is math. Get Jonathan Lewis' optimizer book.

  • URLConnection with large amount of data causes OutOfMemoryError: Java heap

    I am setting up a system where my customers can sent me their data files to my server. If I use a simple socket with a server permanently running on a chosen port, my customers are able to transfer files of any size without problem. However, If I adopt a different architecture, using a web server and a CGI program to receive their submissions, the client program that they run will crash if the file they are sending is any larger than about 30 Mbytes, with the exception OutOfMemoryError: Java heap.
    The code in the two architectures is almost identical:
    Socket con = new Socket( host, portno);
    //URL url = new URL("http://"+host+":"+portno+path);
    //URLConnection con = url.openConnection();
    //con.setDoOutput(true);
    File source_file = new File(file_name);
    FileInputStream source = new FileInputStream(source_file);
    out = new BufferedOutputStream(con.getOutputStream());
    // First, Send a submission header.
    data_out = submit_info.getBytes();
    len = data_out.length + 1;
    data_len[0] = (byte)len;
    out.write(data_len, 0, 1);
    out.write(data_out, 0, data_out.length);
    // Then send the file content.
    buff = new byte[(int)buffSize];
    content_length = source_file.length();
    long tot = 0;
    while ( tot < content_length )
    // Read data from the file.
    readSize = (int) Math.min( content_length-tot, buffSize );
    nRead = source.read(buff, 0, readSize);
    tot += nRead;
    if ( nRead == 0 ) break;
    // Send data.
    out.write(buff, 0, nRead);
    "buffSize" is 4096. This code works fine, but if the first line is commented out and the next three are uncommented, the OutOfMemory exception is thrown within the loop when "tot" is around 30 million.
    I have tried calling the garbage collector within the loop but it makes no difference. I am unable to anticipate the size of files that my customers will submit, so I cannot set the heap size in advance to cope with what they will sent. Fortunately, using a simple Socket avoids the problem, so there seems to be something wrong with how URLConnection works.

    Set the URLConnection to use chunked mode. This saves it from having to buffer the entire contents before sending to ascertain its length.

  • Problems Removing Large Amounts of Data

    Hello Everyone,
    First let me state that i know the difference between removing and deleting and i was removing.
    So i have a huge lightroom library of about 300,000 images and i tried to remove 50% all at once.
    Heres my process
    1. Right click remove 150,000 images.
    2. Wait 48 hours.
    3. Lightroom locks up and takes over 11gb of memory. (I ran a sample process and that's whats posted below)
    4. Force Quit.
    5. Lightroom Library loads fine now.
    6. Optimize Catalog
    7. Wait 36 hours.
    8.  THE 150,000 IMAGES REAPPEAR IN THE CATALOG!!!!!!!
    9.  Currently... Repeat 1-3 and now waiting for lightroom to quit normally.... 
    Can anyone at adobe read this sample and tell me what lightroom is doing?  I suspect that if i force quit than all the removing i did will revert, is that correct? It's taking 11gb of 12gb of ram. Is it stuck in a loop?  Is this a normal problem or a bug?
    Sorry for the long sample data
    Computer specs:
    Imac, Intel i3, 3.06GHZ, 12gb of memory
    Sample Data
    Sampling process 193 for 3 seconds with 1 millisecond of run time between samples
    Sampling completed, processing symbols...
    Analysis of sampling Adobe Lightroom 3 (pid 193) every 1 millisecond
    Call graph:
        68 Thread_1252   DispatchQueue_1: com.apple.main-thread  (serial)
          68 start
            68 main
              68 NSApplicationMain
                68 -[NSApplication run]
                  68 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]
                    68 _DPSNextEvent
                      68 _NSHandleCarbonMenuEvent
                        68 _HandleMenuSelection2
                          68 MenuSelectCore(MenuData*, Point, double, unsigned int, OpaqueMenuRef**, unsigned short*)
                            68 FinishMenuSelection(SelectionData*, MenuResult*, MenuResult*)
                              68 SendMenuItemSelectedEvent
                                68 SendMenuCommandWithContextAndModifiers
                                  68 SendHICommandEvent(unsigned int, HICommand const*, unsigned int, unsigned int, unsigned char, void const*, OpaqueEventTargetRef*, OpaqueEventTargetRef*, OpaqueEventRef**)
                                    68 SendEventToEventTarget
                                      68 SendEventToEventTargetInternal(OpaqueEventRef*, OpaqueEventTargetRef*, HandlerCallRec*)
                                        68 DispatchEventToHandlers(EventTargetRec*, OpaqueEventRef*, HandlerCallRec*)
                                          68 NSSLMMenuEventHandler
                                            68 -[NSCarbonMenuImpl _carbonCommandProcessEvent:handlerCallRef:]
                                              68 -[NSMenu _internalPerformActionForItemAtIndex:]
                                                68 -[NSCarbonMenuImpl performActionWithHighlightingForItemAtIndex:]
                                                  68 -[NSMenuItem _corePerformAction]
                                                    68 -[NSApplication sendAction:to:from:]
                                                      68 -[NSApplication terminate:]
                                                        68 -[NSApplication _shouldTerminate]
                                                          68 -[NSDocumentController(NSInternal) _shouldTerminateWithDelegate:shouldTerminateSelector:]
                                                            68 -[NSDocumentController(NSInternal) _continueTerminationHavingClosedAllDocuments:context:]
                                                              68 -[NSApplication _docController:shouldTerminate:]
                                                                68 main
                                                                  68 willAutoChange
                                                                    68 AgShutdownNotification_appShouldTerminate
                                                                      68 lua_call
                                                                        68 luaF_recfillpcbase
                                                                          68 luaF_recfillpcbase
                                                                            68 luaL_newstate
                                                                              68 luaopen_package
                                                                                68 lua_pcall
                                                                                  68 lua_yield
                                                                                    68 luaL_buffinit
                                                                                      68 luaF_recfillpcbase
                                                                                        68 luaF_recfillpcbase
                                                                                          68 luaL_newstate
                                                                                            68 luaopen_package
                                                                                              68 luaL_error
                                                                                                 68 lua_resume
                                                                                                   68 luaL_buffinit
                                                                                                     68 luaF_recfillpcbase
                                                                                                       68 luaL_newstate
                                                                                                         68 AgDialogs_runModalL
                                                                                                           68 AgColorUtils_convertHSLtoRGB_L
                                                                                                             68 -[NSApplication runModalForWindow:]
                                                                                                               68 -[NSApplication _realDoModalLoop:peek:]
                                                                                                                 68 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]
                                                                                                                   68 _DPSNextEvent
                                                                                                                     68 BlockUntilNextEventMatchingListInMode
                                                                                                                       68 ReceiveNextEventCommon
                                                                                                                         68 RunCurrentEventLoopInMode
                                                                                                                           68 CFRunLoopRunSpecific
                                                                                                                             68 __CFRunLoopRun
                                                                                                                               68 __CFRunLoopDoSources0
                                                                                                                                 68 __NSThreadPerformPerform
                                                                                                                                   68 AgTransitQueue_enqueueToQueue
                                                                                                                                     67 lua_pcall
                                                                                                                                       67 lua_yield
                                                                                                                                         67 luaL_buffinit
                                                                                                                                           67 luaF_recfillpcbase
                                                                                                                                             67 luaF_recfillpcbase
                                                                                                                                               66 luaL_newstate
                                                                                                                                                 65 AgTransitQueue_enqueueToQueue
                                                                                                                                                   65 AgTransitQueue_enqueueToQueue
                                                                                                                                                     65 AgTransitValue_makeFromLuaState
                                                                                                                                                       65 AgTransitValue_push
                                                                                                                                                         65 lua_settable
                                                                                                                                                           65 luaL_loadstring
                                                                                                                                                             65 lua_yield
                                                                                                                                                               65 luaL_buffinit
                                                                                                                                                 1 luaopen_package
                                                                                                                                                   1 lua_pcall
                                                                                                                                                     1 lua_yield
                                                                                                                                                       1 luaL_buffinit
                                                                                                                                                         1 luaF_recfillpcbase
                                                                                                                                                           1 luaF_recfillpcbase
                                                                                                                                                             1 luaL_newstate
                                                                                                                                                               1 luaL_optlstring
                                                                                                                                               1 luaL_loadstring
                                                                                                                                                 1 lua_checkstack
                                                                                                                                                   1 lua_checkstack
                                                                                                                                                     1 lua_checkstack
                                                                                                                                                       1 lua_yield
                                                                                                                                                         1 luaL_buffinit
                                                                                                                                                           1 szone_free_definite_size
                                                                                                                                                             1 tiny_free_list_remove_ptr
                                                                                                                                     1 __compare_and_swap64
        68 Thread_1259   DispatchQueue_2: com.apple.libdispatch-manager  (serial)
          68 start_wqthread
            68 _pthread_wqthread
              68 _dispatch_worker_thread2
                68 _dispatch_queue_invoke
                  68 _dispatch_mgr_invoke
                    68 kevent
        68 Thread_1265: Worker Thread Dispatcher
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaF_recfillpcbase
                                      68 luaL_newstate
                                        67 AgTransitQueue_enqueueToQueue
                                          67 -[NSConditionLock lockWhenCondition:beforeDate:]
                                            67 -[NSCondition waitUntilDate:]
                                              67 _pthread_cond_wait
                                                67 semaphore_timedwait_signal_trap
                                        1 luaL_checkoption
                                          1 lua_gc
                                            1 lua_checkstack
                                              1 lua_checkstack
                                                1 lua_checkstack
                                                  1 lua_yield
                                                    1 luaL_buffinit
                                                      1 __spin_lock
        68 Thread_1266: ace
          68 thread_start
            68 _pthread_start
              68 PrivateMPEntryPoint
                68 private_load_AgColorProfile
                  68 private_load_AgColorProfile
                    68 MPWaitOnQueue
                      68 TSWaitOnConditionTimedRelative
                        68 TSWaitOnCondition
                          68 _pthread_cond_wait
                            68 __semwait_signal
        68 Thread_1267: ace
          68 thread_start
            68 _pthread_start
              68 PrivateMPEntryPoint
                68 private_load_AgColorProfile
                  68 private_load_AgColorProfile
                    68 MPWaitOnQueue
                      68 TSWaitOnConditionTimedRelative
                        68 TSWaitOnCondition
                          68 _pthread_cond_wait
                            68 __semwait_signal
        68 Thread_1268: ace
          68 thread_start
            68 _pthread_start
              68 PrivateMPEntryPoint
                68 private_load_AgColorProfile
                  68 private_load_AgColorProfile
                    68 MPWaitOnQueue
                      68 TSWaitOnConditionTimedRelative
                        68 TSWaitOnCondition
                          68 _pthread_cond_wait
                            68 __semwait_signal
        68 Thread_1269: cr_scratch
          68 thread_start
            68 _pthread_start
              68 private_load_AgColorProfile
                68 private_load_AgColorProfile
                  68 private_load_AgColorProfile
                    68 _pthread_cond_wait
                      68 __semwait_signal
        68 Thread_1270
          68 thread_start
            68 _pthread_start
              68 private_load_AgColorProfile
                68 private_load_AgColorProfile
                  68 semaphore_wait_trap
        68 Thread_1271
          68 private_load_AgColorProfile
            68 private_load_AgColorProfile
              68 semaphore_wait_trap
        68 Thread_1272
          68 private_load_AgColorProfile
            68 private_load_AgColorProfile
              68 semaphore_wait_trap
        68 Thread_1273
          68 private_load_AgColorProfile
            68 private_load_AgColorProfile
              68 semaphore_wait_trap
        68 Thread_1274
          68 private_load_AgColorProfile
            68 private_load_AgColorProfile
              68 semaphore_wait_trap
        68 Thread_1275: cr_negative_cache
          68 thread_start
            68 _pthread_start
              68 private_load_AgColorProfile
                68 private_load_AgColorProfile
                  68 private_load_AgColorProfile
                    68 _pthread_cond_wait
                      68 __semwait_signal
        68 Thread_1277: Worker Thread
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaL_newstate
                                      68 luaopen_package
                                        68 lua_pcall
                                          68 lua_yield
                                            68 luaL_buffinit
                                              68 luaF_recfillpcbase
                                                68 luaF_recfillpcbase
                                                  68 luaL_newstate
                                                    68 AgTransitQueue_enqueueToQueue
                                                      68 -[NSConditionLock lockWhenCondition:beforeDate:]
                                                        68 -[NSCondition waitUntilDate:]
                                                          68 _pthread_cond_wait
                                                            68 semaphore_timedwait_signal_trap
        68 Thread_1736: AsyncDataServer
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaF_recfillpcbase
                                      68 luaL_newstate
                                        68 luaopen_package
                                          68 lua_pcall
                                            68 lua_yield
                                              68 luaL_buffinit
                                                68 luaF_recfillpcbase
                                                  68 luaF_recfillpcbase
                                                    68 luaL_newstate
                                                      68 luaopen_package
                                                        68 lua_pcall
                                                          68 lua_yield
                                                            68 luaL_buffinit
                                                              68 luaF_recfillpcbase
                                                                68 luaF_recfillpcbase
                                                                  68 lua_checkstack
                                                                    68 luaL_loadstring
        68 Thread_1737: Queue Processor - AgPhotoAvailability
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaL_newstate
                                      68 luaopen_package
                                        68 lua_pcall
                                          68 lua_yield
                                            68 luaL_buffinit
                                              68 luaF_recfillpcbase
                                                68 luaF_recfillpcbase
                                                  68 luaL_newstate
                                                    68 AgLua_callWithAutoReleasePool
                                                      68 lua_call
                                                        68 luaF_recfillpcbase
                                                          68 luaL_newstate
                                                            68 AgTransitQueue_enqueueToQueue
                                                              68 -[NSConditionLock lockWhenCondition:beforeDate:]
                                                                68 -[NSCondition waitUntilDate:]
                                                                  68 _pthread_cond_wait
                                                                    68 semaphore_timedwait_signal_trap
        68 Thread_1739: Preview Server
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaF_recfillpcbase
                                      68 luaL_newstate
                                        68 private_load_AgEventLoopUtils
                                          68 lua_pcall
                                            68 lua_yield
                                              68 luaL_buffinit
                                                68 luaF_recfillpcbase
                                                  68 luaF_recfillpcbase
                                                    68 luaL_newstate
                                                      68 luaopen_package
                                                        68 luaL_error
                                                          68 lua_resume
                                                            68 luaL_buffinit
                                                              68 luaF_recfillpcbase
                                                                68 luaL_newstate
                                                                  68 AgSQLiteStatement_pushNewToLua
                                                                    68 sqlite3_step
                                                                      68 sqlite3_prepare
                                                                        68 sqlite3_result_int
                                                                          68 sqlite3_result_null
                                                                            68 sqlite3_result_null
                                                                              68 sqlite3_result_null
                                                                                68 sqlite3_result_null
                                                                                  68 sqlite3_thread_cleanup
                                                                                    68 read
        68 Thread_1740: Queue Processor - PreviewFileStore
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaL_newstate
                                      68 luaopen_package
                                        68 lua_pcall
                                          68 lua_yield
                                            68 luaL_buffinit
                                              68 luaF_recfillpcbase
                                                68 luaF_recfillpcbase
                                                  68 luaL_newstate
                                                    68 AgLua_callWithAutoReleasePool
                                                      68 lua_call
                                                        68 luaF_recfillpcbase
                                                          68 luaL_newstate
                                                            68 AgTransitQueue_enqueueToQueue
                                                              68 -[NSConditionLock lockWhenCondition:beforeDate:]
                                                                68 -[NSCondition waitUntilDate:]
                                                                  68 _pthread_cond_wait
                                                                    68 semaphore_timedwait_signal_trap
        68 Thread_1744: Queue Processor - HistogramQueue
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaL_newstate
                                      68 luaopen_package
                                        68 lua_pcall
                                          68 lua_yield
                                            68 luaL_buffinit
                                              68 luaF_recfillpcbase
                                                68 luaF_recfillpcbase
                                                  68 luaL_newstate
                                                    68 AgLua_callWithAutoReleasePool
                                                      68 lua_call
                                                        68 luaF_recfillpcbase
                                                          68 luaL_newstate
                                                            68 AgTransitQueue_enqueueToQueue
                                                              68 -[NSConditionLock lockWhenCondition:beforeDate:]
                                                                68 -[NSCondition waitUntilDate:]
                                                                  68 _pthread_cond_wait
                                                                    68 semaphore_timedwait_signal_trap
        68 Thread_1745: Queue Processor - RenderQueue_ReadPreview
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaL_newstate
                                      68 luaopen_package
                                        68 lua_pcall
                                          68 lua_yield
                                            68 luaL_buffinit
                                              68 luaF_recfillpcbase
                                                68 luaF_recfillpcbase
                                                  68 luaL_newstate
                                                    68 AgLua_callWithAutoReleasePool
                                                      68 lua_call
                                                        68 luaF_recfillpcbase
                                                          68 luaL_newstate
                                                            68 AgTransitQueue_enqueueToQueue
                                                              68 -[NSConditionLock lockWhenCondition:beforeDate:]
                                                                68 -[NSCondition waitUntilDate:]
                                                                  68 _pthread_cond_wait
                                                                    68 semaphore_timedwait_signal_trap
        68 Thread_1746: Queue Processor - RenderQueue_Render
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaL_newstate
                                      68 luaopen_package
                                        68 lua_pcall
                                          68 lua_yield
                                            68 luaL_buffinit
                                              68 luaF_recfillpcbase
                                                68 luaF_recfillpcbase
                                                  68 luaL_newstate
                                                    68 AgLua_callWithAutoReleasePool
                                                      68 lua_call
                                                        68 luaF_recfillpcbase
                                                          68 luaL_newstate
                                                            68 AgTransitQueue_enqueueToQueue
                                                              68 -[NSConditionLock lockWhenCondition:beforeDate:]
                                                                68 -[NSCondition waitUntilDate:]
                                                                  68 _pthread_cond_wait
                                                                    68 semaphore_timedwait_signal_trap
        68 Thread_1747: Queue Processor - NegativeQueue
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaL_newstate
                                      68 luaopen_package
                                        68 lua_pcall
                                          68 lua_yield
                                            68 luaL_buffinit
                                              68 luaF_recfillpcbase
                                                68 luaF_recfillpcbase
                                                  68 luaL_newstate
                                                    68 AgLua_callWithAutoReleasePool
                                                      68 lua_call
                                                        68 luaF_recfillpcbase
                                                          68 luaL_newstate
                                                            68 AgTransitQueue_enqueueToQueue
                                                              68 -[NSConditionLock lockWhenCondition:beforeDate:]
                                                                68 -[NSCondition waitUntilDate:]
                                                                  68 _pthread_cond_wait
                                                                    68 semaphore_timedwait_signal_trap
        68 Thread_1789: Worker Thread
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaL_newstate
                                      68 luaopen_package
                                        68 lua_pcall
                                          68 lua_yield
                                            68 luaL_buffinit
                                              68 luaF_recfillpcbase
                                                68 luaF_recfillpcbase
                                                  68 luaL_newstate
                                                    66 AgTransitQueue_enqueueToQueue
                                                      65 AgTransitQueue_enqueueToQueue
                                                        65 AgTransitValue_makeFromLuaState
                                                          65 AgTransitValue_push
                                                            64 AgTransitValueInternal_beginTransitState
                                                              64 AgMutex_lock
                                                                64 pthread_mutex_lock
                                                                  64 semaphore_wait_signal_trap
                                                            1 lua_settable
                                                              1 luaL_loadstring
                                                                1 lua_yield
                                                                  1 luaL_buffinit
                                                      1 -[NSObject(NSThreadPerformAdditions) performSelectorOnMainThread:withObject:waitUntilDone:]
                                                        1 __spin_lock
                                                    2 luaL_checkoption
                                                      2 lua_gc
                                                        1 luaL_loadstring
                                                        1 lua_checkstack
                                                          1 lua_checkstack
        68 Thread_1790: Worker Thread
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaL_newstate
                                      68 luaopen_package
                                        68 lua_pcall
                                          68 lua_yield
                                            68 luaL_buffinit
                                              68 luaF_recfillpcbase
                                                68 luaF_recfillpcbase
                                                  68 luaL_newstate
                                                    68 AgTransitQueue_enqueueToQueue
                                                      68 -[NSConditionLock lockWhenCondition:beforeDate:]
                                                        68 -[NSCondition waitUntilDate:]
                                                          68 _pthread_cond_wait
                                                            68 semaphore_timedwait_signal_trap
        68 Thread_1791: Worker Thread
          68 thread_start
            68 _pthread_start
              68 __NSThread__main__
                68 AgTransitQueue_enqueueToQueue
                  68 lua_pcall
                    68 lua_yield
                      68 luaL_buffinit
                        68 luaF_recfillpcbase
                          68 luaF_recfillpcbase
                            68 luaL_newstate
                              68 AgLua_callWithAutoReleasePool
                                68 lua_call
                                  68 luaF_recfillpcbase
                                    68 luaL_newstate
                                      68 luaopen_package
                                        68 lua_pcall
                                          68 lua_yield
                                            68 luaL_buffinit
                                              68 luaF_recfillpcbase
                                                68 luaF_recfillpcbase
                                                  68 luaL_newstate
                                                    68 AgTransitQueue_enqueueToQueue
                                                      68 -[NSConditionLock lockWhenCondition:beforeDate:]
                                                        68 -[NSCondition waitUntilDate:]
                                                          68 _pthread_cond_wait
                                                            68 semaphore_timedwait_signal_trap
        68 T

    Because the transaction is so unusually large, SQLite3 and/or LR are hitting a performance cliff.  When you force quit, the transaction never completed, so when you restart LR, SQLite3 restores the database to the state prior to the removal (as it should).
    The obvious workaround to try: Remove 10K images at a time, 15 times. 
    Yeah, i think your right.  The computer just can't handle that large of transaction.  I guess the answer is to get a new computer.  However, i'm still going to consider this a bug. 
    You should be able to remove as many files as you want without it shutting down your computer or needing a 12 core super computer.  I might be wrong, but i don't think that's too much to ask for.
    I just ended up creating a whole new catalog and adding only the folders i want, instead of pressing remove, waiting 3 hours and then pressing remove again 15 times.  That seems unbearable.
    Thanks for your comments everyone.  I hope LR checks this out.  Removing folders is a real pain, even when it's just a few.  Theres got to be an easier and less processor intensive way to remove.

  • [SOLVED] Problem transferring large amounts of data to ntfs drive

    I have about 150GB of files that I am trying to transfer from my home partition which is ext3 /dev/sdd2 to another partition which is ntfs mounted with ntfs-3g /dev/sde1 (both drives are internal sata). The files are sizes ranging from 200MB to 4GB. When the files start to move they transfer at a reasonable speed (10 to 60MB/s), but will randomly (usually after about 1 to 5GB transfered) slow down to about 500KB/s transfer speed. The computer becomes unusable at this point, and even if I cancel the transfer the computer will continue to be unusable (I must use alt+sysreq REISUB). I have tried transferring with dolphin, nautilus, and the mv command but they all will produce the same results. I have also tried this in dolphin as root with no change. If I leave dolphin running long enough I also get the message "the process for the file protocol died unexpectedly".
    There is nothing that I can tell is wrong with the drives, I've run disk checks on both, and checked the S.M.A.R.T. readings for both disks and everything was fine.
    My hardware is an intel X58 motherboard, core i7 processor, 6GB of RAM, 5 internal sata drives and 1 internal sata optical drive.
    Another thing to note is that every once in a while I will get an error message at boot saying "Disabling IRQ #19".
    This is driving me crazy as I have no idea why this is happening and when I search I can't find any solutions that work.
    If anybody knows how to solve this or can help me diagnose the problem please help.
    Thank you.
    Last edited by zetskee (2011-01-07 21:29:58)

    Primoz wrote:Do you use KDE (Dolphin) 4.6 RC or 4.5?
    Also I've noticed that if i move / copy things with Dolphin they're substantially slower than if I use cp/mv. But cp/mv works fine for me...
    Also run Dolphin from terminal to try and see what's the problem.
    Hope that help at least a bit.
    Could you explain why Dolphin should be slower? I'm not attacking you, I'm just asking.
    Cause I thought that Dolphin is just a „little" wrapper around the cp/mv/cd/ls applications/commands.

Maybe you are looking for

  • How do you stop your background from refreshing when a link is clicked on your muse page?

    The site I am creating is relying heavily on edge animate objects. I want to maintain smoothness and aesthetics by allowing buttons to animate (created in edge animate) when clicked then after their animation has concluded the page the button was lin

  • RIDC check in the file name encoding issue

    Hi, When I use the RIDC to upload the chinese file name, the native file field become ??.pdf. TransferFile tf = new TransferFile(new File("c:/中文.pdf"), "application/pdf"); dataBinder.addFile("primaryFile", tf); Thanks!

  • Reusing AIR Captive Runtime

    I have an application that is installed and acts as a loader, downloading different other applications that all use the captive runtime. However, considering the current folder structure of a "captive runtime" application: -> Adobe AIR -> META-INF ==

  • Badi Me_req_header_text

    hello all, My requirement is the header text on the purchse requsition should be copied on to the purchase order. Normally with standard SAP it is not possible. It is possible with the BADI Me_req_header_text and the method Copy_header_text, i should

  • Implement 2-way matching for SRM PO

    Hi Gurus,    We are in SRM5.0 Extended Classic and currently running 3-way match scenarios. Client wants to implement 2-way match for certain Vendors. My question is how to accomplish this. Below are my thoughts. Please let me know if I missed any. f