SQLite Inserts Taking Longer and Longer

The application I'm working on makes repeated calls to a webservice to get data that is then cached in a local sqlite db for the user. Once the db hits ~5mb it starts taking painfully long to run each set of inserts. Calls to the webservice remain quick. There had been an issue with XML not being garbage collected, but I fixed that and now the profiler shows consistent memory usage.
I've tried running the inserts with indexes on and off. I've tried batching the inserts in transactions of 100, or the entire set. The db calls are synchronous.
Running the queries against the database directly (not through the AIR application) suggest that there isn't a slowdown at the 5mb mark there, which is consistent with my experiences with SQLlite. Restarting the application and continuing to download data into an existing project does not resolve the issue, it starts off slow.
So...does anyone have any ideas of other things to try to get insert performance up to a reasonable level? Has anyone else run into similiar issues? Is anyone inserting into 20mb+ dbs and not seeing degrading performance?
Thanks for the help!

Guess I posted prematurely. Looking closer I realized there was a select happening during this process against a text column without an index. The slowdown was just the increasing cost of looping through the the entire dataset looking at strings that often shared a fairly sizable starting substring. Chalk another problem up to the importance of appropriate indexes in your db!

Similar Messages

Insert taking long time.

Hi,
I am inserting 10 milion records into a table
using procedure it's taking nearlly one hour
to insert all 10 million records.
So that I have used FORALL bulk collect along with APPEND hint and the
time is reduced to 20 minutes.
Is there any other method to reduce the time to less than 10 minutes.
please help me.
Thanks in advance.

"Batch processing" as you call it may be appropraite providing that the target system has enough spare processes to run five times the workload simultaneously. And providing the business rules allow for partial success/failures (i.e you now have five transactions). And providing the source query can be run in five separate threads without them interfering with each other.
Now if you had just said "parallel processing" that would have been a different matter. Definitely an option if the OP has an Enterprise Edition license, and the requisite spare CPUs.
Cheers, APC

Insert taking longer than usual

One of my single row INSERT operation is taking ~20 sec. It has a CLOB column (REQUEST_VALUE). This is a 2 node RAC setup.
Any pointers?
Here is the tablespace configuration.
dba_tablespaces:
PROD01_DATA
PROD01_LOB
dba_segments for the table
Tablespace: PROD01_DATA
dba_lobs for the table
SEGMENT NAME SYS_LOB0000033324C00003$$
COLUMN REQUEST_VALUE
TABLESPACE PROD01_DATA
dba_data_files: PROD01_DATA
BYTES 2147483648
BLOCKS 262144
STATUS AVAILABLE
RELATIVE_FNO 11
AUT YES
MAXBYTES 1.6106E+10
MAXBLOCKS 1966080
INCREMENT_BY 1
USER BLOCKS 262136
USER BYTES 2147418112
dba_data_files: PROD01_LOB
RELATIVE_FNO 16
MAXBYTES 1.0737E+10
MAXBLOCKS 1310720
(others same as PROD01_DATA)
From the AWR:
Cache Sizes
Buffer Cache:     640M     576M     Std Block Size:     8K
Shared Pool Size:     832M     896M     Log Buffer:     30,732K
Shared Pool Statistics
Memory Usage %:     90.75     87.80
% SQL with executions>1:     98.23     95.79
% Memory for SQL w/exec>1:     97.24     95.5
Top 5 Timed Events
Event     Waits     Time(s)     Avg Wait(ms)     % Total Call Time     Wait Clas
CPU time          583          49.1
reliable message     7,972     78     10     6.6     Other
db file sequential read     31,566     41     1     3.5     User I/O
control file sequential read     58,149     40     1     3.4     System I/O
latch: shared pool     188     34     179     2.8     Concurrency
Edited by: hardcodr on Jul 8, 2011 11:09 PM

If you really want to know where time a spent for a specific SQL statetemet you should use SQL trace and TKPROF for the related SQL statement.
See http://www.oracle-base.com/articles/10g/SQLTrace10046TrcsessAndTkprof10g.php.

Fwrite() and fread() of a shared FAT32 formatted file is taking long time in MAC osx Lion C program

Hi
Is there any provision or api in MAC to open a file in shared mode same as windows
hUSBdrive = CreateFile(pDriveName, GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_FLAG_NO_BUFFERING, NULL);
we have the follwing scenario where a file is shared among two processes for read/write.one is running on Linux and the other one is running on MAC.where both the processses are reading/writing into the same memory location in the file say "X"
FAT 32 formatted raw data file which is located on the device, is shared among two processes.
One process is running on Linux device which is connected to MAC book through usb.In this linux process, the file is opened using fopen() and we have used fcntl() with O_DIRECT flag.This process continuously reads/writes data on memory location "X" in the shared file .
The other process is running on Mac which has simple c program that opens the file on the connected device i.e from usb drive and reads/writes data using fread()/fwrite().fopen() is used to open the file and FILE_NOCACHE flag is used to avoid caching.
The value at memory location "X" is updated by mac by using fwrite() and the linux process reads the memory location "X" by using fread(). Linux process is taking around 30 sec to get the updated value.
If the value is updated by Linux process at memory location "X" by using fwrite() the MAC process is also taking long time more than a minute to read the updated value by usng fread().
fwrite()/fread() on mac is taking long time where as the windows application which uses the same apis is taking msec time.
Do we need to use other api s or flags to open file?
thanks in advance.......

does any one face this kind of problem?
fwrite() and fread() takes long time?
Is there any problem in read/write to a fat32 file on MAC?

HT201250 my time capsule is taking too much time indexing backup and then taking longer time to back up ( 207 days ) or longer !!! what shall i do ?

my time capsule is taking too much time indexing backup and then taking longer time to back up ( 207 days ) or longer !!! what shall i do ?

Try 10.7.5 supplemental update.
This update seems to have solved this problem for many.
Best.

I just received my i phone 5 and it seems to be taking long for the apps to download with iCloud. Is that right?

I just received my iphone5 and am downloading my apps with icloud and it seems to be taking long...is that right?

Hello JeenC,
I found an article with steps you can take when you experience issues with attachements on your iCloud calendar.
I recommend reviewing the steps in the section titled "Troubleshooting Calendar attachments" in the following article:
iCloud: Using and troubleshooting Calendar attachments
http://support.apple.com/kb/HT5373
Thank you for using Apple Support Communities.
Best,
Sheila M.

HT4623 ios update is taking longer hours, corrupting other applications and failing

ios update is taking longer hours, corrupting other applications and failing, please what can i do?

Make sure you're using the latest version of itunes.

After updating to latest firefox computer on internet running slower,taking longer to go to links and sites.also if i click on an email link the link keeps refreshing as mailto and wont stop opening new tabs very quickly,have to shut down to stop it.

since installing latest update,when using internet,email everything has slowed down and is taking long to go to websites to upload pictures to emails.
also when i click on an email link to send email the tab bar keeps refreshing and opening a new tab very quickly all i can read cos its going so fast is mailto this has happened several times on various websites,i clicked on a email us link on grangerhertzog.com yesterday and it did it.

You get that problem if you select the Firefox program to handle a file if you get an "open with" dialog.
*https://support.mozilla.org/kb/Firefox+keeps+opening+many+tabs+or+windows

Apps taking longer to startup and use

In my main OSX startup profile some apps Office, sometimes Safari, and others are taking longer than normal to startup and use. I booted into my administrator account and everything there is speedy so this appears to be a software issue somewhere. I am going to rebuild disk permissions, but I don think that will do the trick. It also takes years to perform this task in Snow Leopard. Tiger could do the job in half the time it takes Snow Leopard. Anyone got other suggestions to try? I am going to do some googling.
Thanks,
John

See:
Mac Maintenance Quick Assist,
Mac OS X speed FAQ,
Speeding up Macs,
Macintosh OS X Routine Maintenance
Essential Mac Maintenance: Get set up,
Essential Mac Maintenance: Rev up your routines,
Maintaining OS X, and
Myths of required versus not required maintenance for Mac OS X for information.

Y IPHONE KEEP ON SWTCHING OFF WHENEVER I INSERT MY SIMCARD AND LAST LONGER WHEN MY SIM CAD IS NOT INSERTED.

MY IPHONE KEEP ON SWTCHING OFF WHENEVER I INSERT MY SIMCARD AND LAST LONGER WHEN MY SIM CAD IS NOT INSERTED.PLEASE HELP ME:(((((

STOP SHOUTING...
The Basic Troubleshooting Steps are:
Restart... Reset... Restore from Backup... Restore as New...
Restart / Reset
http://support.apple.com/kb/ht1430
Backing up, Updating and Restoring
http://support.apple.com/kb/HT1414
If you try all these Steps and you still have issues... Then a Visit to an Apple Store or AASP (Authorized Apple Service Provider) is the Next Step...
Be sure to make an appointment first...

Deletion is taking long time using forall

Hi,
       i am inserting and deleting the rows using forall. insert taking less time to inset the rows but while coming to
deletion it is taking more than 5 days long time to delete 18.5 million rows in a table using forall.
the main table having 70 million rows.
the code is..
   FETCH ref_typ BULK COLLECT INTO l_id_tbl LIMIT 10000;
   begin
    FORALL i in 1..l_id_tbl.COUNT
       INSERT INTO   change_test (id,
                             history,
                             transaction,
                             date)
                     VALUES (seq.nextval,
                             'CHANGE_HIS',
                             l_id_tbl(i),
                             sysdate);
       exception
           when others then
            null;
    end;
    begin
        FoRALL i in 1..l_id_tbl.COUNT
          DELETE FROM change_his
                 where id = l_id_tbl(i);
       exception
           when others then
             null;
   end;
end loop;
so please give me a good solution to delete the rows less than 5 days..

Why are you wanting to do this using BULK COLLECT and FORALL?
Why not just "insert ... select ..." and "delete ..."?
Loading records into expensive PGA memory to insert them back on the database is bound to be slower (and use more server resources) than just doing a straight insert ... select ... statement.
Explain exactly what you are trying to do.
Re: 2. How do I ask a question on the forums?

Rtf template taking long time to load xml

Dear All
rtf template taking long time to load xml and Insert fields. I am using MS office2010. It was working before.
can any one help me out.
Regards
Arifuddin

Hi;
Pelase review below which could be helpful for you
XML Publisher Report Issues, Recommendations and Errors [ID 862644.1]
Regard
Helios

SSRS Reports taking long time to load

Hello,
Problem : SSRS Reports taking long time to load
My System environment : Visual Studio 2008 SP1 and SQL Server 2008 R2
Production Environment : Visual Studio 2008 SP1 and SQL Server 2008 R2
I have created a Parameterized report (6 parameters), it will fetch data from 1 table. table has 1 year and 6 months data,      I am selecting parameters for only 1 month (about 2500 records). It is taking almost 2 minutes and 30 seconds
to load the report.
This report running efficiently in my system (report load takes only 5 to 6 seconds) but in
production it is taking 2 minutes 30 seconds.
I have checked the Execution log from production so I found the timing for
Data retrieval (approx~)    Processing (approx~)           Rendering (approx~)
10 second                                  15 sec
        2 mins and 5 sec.
But Confusing point is that , if I run the same report at different time overall output time is same (approx) 2 min 30 sec but
Data retrieval (approx~)       Processing (approx~)                Rendering (approx~)
more than 1 min                            15 sec
more than 1 min
so 1 question why timings are different ?
My doubts are
1) If query(procedure to retrieve the data) is the problem then it should take more time always,
2) If Report structure is problem then rendering will also take same time (long time)
for this (2nd point) I checked on blog that Rendering depends on environment structure e.g. Network bandwidth, RAM, CPU Usage , Number of users accessing same report at a time.
So I did testing of report when no other user working on any report But failed (same result output is 2 min 30 sec)
From network team I got the result is that there is no issue or overload in CPU usage or RAM also No issue in Network bandwidth.
Production Database Server and Report server are different (but in same network).
I checked that database server the SQL Server is using almost Full RAM (23 GB out of 24 GB)
I tried to allocate the memory to less amount up to 2GB (Trial solution I got from Blogs) but this on also failed.
one hint I got from colleague that , change the allocated memory setting from static memory to dynamic to SQL Server
(I guess above point is the same) I could not find that option Static and Dynamic memory setting.
I did below steps
Connected to SQL Server Instance
Right click on Instance go to properties, Go to Memory Tab
I found three options 1) Server Memory   2) Other memory   3) Section for "Configured values and Running values"
Then I tried to reduce Maximum Server memory up to 2 GB (As mentioned above)
All trials failed, this issue I could not find the roots for this issue.
Can anyone please help (it's bit urgent).

Hi UdayKGR,
According to your description, your report takes too long to load on your production environment. Right?
In this scenario, since the report runs quickly in developing environment, we initially think it supposed to be the issue on data retrieval. However, based on the information in execution log, it takes longest time on rendering part. So we suggest you optimize
the report itself to reduce the time for rendering. Please refer to the link below:
My report takes too long to render
Here is another article about overall performance optimization for Reporting Services:
Reporting Services Performance and Optimization
If you have any question, please feel free to ask.
Best Regards,
Simon Hou

Connecting to the database taking long time to connect database server

Hi
When I execute procedure i am getting the below message at bottom of the Oracle SQL Developer
"Connecting to the database"
it is taking more than 10 min plz guide

Hi
have you installed a normal Oracle Client also on your Host? normal Oracle Client
Did you connect with host:port:sid or with a Oracle Naming Service? through TNS Service
Can you test tnsping <alias> yes, It is working fine
Did other user have the same problem? yes
Did you connect through WAN or LAN connection? LAN (Intranet)
Can you tell more about you client/database setup?
Database setup:
OS: Window 2008 server
version: 11.1.0
Client: 11.1.0
OS: Window 2008 server
Now I am not able to execute single select query which table contains 6 records and 15 columns it is taking long time I have waited 30 min still no resutls
only one table is behaving like this remaining is working fine
Edited by: user9235224 on Oct 6, 2012 7:06 PM

The ODS activation is taking long time

Hi,
We are on SAP NetWeaver BI 701 (Support Package 5).
We create a Z ODS, it will contain a lot of data (180.000.000 month-end) and we want to generate specific reports about it.
The activation is taking long time, I assume is because we checked the flag "SIDs Generation upon Activation". I am confused about this check. I really need it? is this check the only problem.
Thanks for you help.
Victoria

Hi Victoria:
If your Z DSO is used only for staging purposes (you don't have queries based on this DSO and you send the data to another DSO or to an InfoCube) then you don't need to check the "SIDs Generation Upon Activation" box.
Even more, to achieve better performance during data loads in this scenario, you might consider using a Write Optimized DSO instead of a Standard DSO, but if you decide to take this alternative don't forget to select the "Do Not check Uniqueness of Data" box if you need to write several records with the same Semantic Key.
Regards,
Francisco Milán.

SQLite Inserts Taking Longer and Longer

Similar Messages

Maybe you are looking for