How can I iterate over all data in database?

Hello,
I need to copy all data from one database to another. Unfortunately I can't.
1. When I use Cursor: database.openCursor, cursor.getFirst, cursor.getNext, cursor.getNext,.... after some period of time getNext returns me data which was loaded already and this operation never ends.
2. When I use Cursor: database.openCursor, cursor.getLast, cursor.getPrev, cursor.getPrev,.... here is no infinite loop as in first case, but not all data was retrieved from database.
3. When I use db dumb: same situation as in first case, operation never ends.
Any other solution are available?
p.s. database size is 18Gb, and we are using Berkeley DB 4.5.2 version.
Best regards,
Alexander

Hello Alex.
1. db_verify does not give anything. Just hangs up without any result.
2. My steps to identify that the cursor is in a loop:
- create HashSet for ids
- open cursor and iterate database using cursor.getNext or cursor.getPrev, as a result I have key and value
- check new key - contains in HashSet of ids or not
Example of results:
e.g.: I have ids in database: 1,2,3,4,5,6,7,8,9
Possible variants of result
1. Everything is OK
- Iterating database using cursor.getFirst, cursor.getNext returns me keys: 1,2,3,4,5,6,7,8,9
- Iterating database using cursor.getLast, cursor.getPrev returns me keys: 9,8,7,6,5,4,3,2,1
2. Fail - 1
- Iterating database using cursor.getFirst, cursor.getNext returns me keys: 1,2,3,4,5
- Iterating database using cursor.getLast, cursor.getPrev returns me keys: 9,8,7
3. Fail - 2
- Iterating database using cursor.getFirst, cursor.getNext returns me keys: 1,2,3,4,5,5,5,5,5
- Iterating database using cursor.getLast, cursor.getPrev returns me keys: 9,8,7
Situation with results Nr2 and Nr3 appears after few months working on production.
We are using BTREE database type.
Database configuration:
- page-size=16384
- cache-size=1073741824
- max-locks=20000
- max-lockers=10000
- max-lock-objects=5000
- log-size=20971520
- checkpoint-frequency=60
- max-transactions=1000
- database dir - memory disk
- database logs dir - VTRAK
- 1 master (Read/Write), 1 slave (for backup only)
Database size
- ~9.5GB
- ~8M keys
- ~1.2Kb average entry size
Database load
- ~4000 load operations/sec peak time, up to 160M loads per day
- ~1400 update operations/sec peak time, up to 55M updates per day
P.S. Newer version of Berkeley DB.
Currently we are testing on our production servers a new version of Berkeley DB 5.0.21. Same configuration, same load, same database size, same database type.
Result: works fine with one exception, but this exception does not allow us to migrate to the newest version.
Problem symptoms:
1. database started, everything is ok. average time of load operation 20K nanoseconds, save operation 75K nanoseconds
2. works smoothly about 20 minutes
3. suddenly all methods to DB read/write hangs up. As the result average time of read operation 110 milliseconds and max time 30 seconds, average time of save operation 50 milliseconds and max time 30 seconds
4. After some period of time (up to one minute) response time as it was in point 1
5. works smoothly for 5-10 minutes
6. see point 3
With no replication work without problems.
Here it is vmstat statistics where we can see the number of Processes increased that time when response time is higher.
>
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 19207660 279016 10752400 0 0 0 0 9142 3359 1 0 99 0
0 0 0 19207040 279016 10752652 0 0 0 160 9435 3786 1 0 99 0
0 0 0 19206948 279016 10752900 0 0 0 0 9165 3363 1 0 99 0
0 0 0 19206852 279016 10753244 0 0 0 0 9115 3710 1 0 99 0
0 0 0 19206148 279016 10753556 0 0 0 3876 9005 3275 1 0 99 0
0 0 0 19205404 279016 10753840 0 0 0 0 9321 3438 1 0 99 0
1 0 0 19205428 279016 10754120 0 0 0 92 9199 3422 1 0 99 0
104 0 0 19204684 279016 10754404 0 0 0 0 9289 4560 8 0 91 0
464 0 0 19204608 279016 10754464 0 0 0 0 6950 12271 44 0 56 0
611 0 0 19203740 279016 10754464 0 0 0 0 5381 12078 44 0 56 0
765 0 0 19203344 279016 10754464 0 0 0 32 5495 13164 44 0 56 0
857 0 0 19202596 279016 10754464 0 0 0 12 5349 13021 44 0 56 0
1027 0 0 19202152 279016 10754464 0 0 0 0 6135 13035 44 0 56 0
1232 0 0 19200120 279016 10754464 0 0 0 0 7392 14701 44 0 56 0
1443 0 0 19198584 279016 10754464 0 0 0 0 6185 14554 44 0 56 0
1444 0 0 19198096 279016 10754464 0 0 0 0 4773 14873 44 0 56 0
1444 0 0 19197848 279016 10754464 0 0 0 0 4409 14930 44 0 56 0
1444 0 0 19197352 279016 10754464 0 0 0 28 4656 14975 44 0 56 0
1444 0 0 19196360 279016 10754464 0 0 0 0 4836 14919 44 0 56 0
1444 0 0 19196112 279016 10754464 0 0 0 0 4841 14929 44 0 56 0
1445 0 0 19195864 279016 10754464 0 0 0 0 4553 14916 44 0 56 0
1445 0 0 19195740 279016 10754464 0 0 0 0 4551 14924 44 0 56 0
1444 0 0 19195616 279016 10754464 0 0 0 0 4493 14919 44 0 56 0
1444 0 0 19195616 279016 10754464 0 0 0 0 4439 14915 44 0 56 0
1444 0 0 19195616 279016 10754464 0 0 0 0 4433 14914 44 0 56 0
1444 0 0 19195616 279016 10754464 0 0 0 0 4451 14936 44 0 56 0
1 0 0 19198716 279016 10754464 0 0 0 44 11341 15083 36 0 64 0
1 0 0 19202048 279016 10754812 0 0 0 4 19893 19704 3 1 96 0
0 0 0 19202560 279016 10755404 0 0 48 0 19203 17524 3 1 96 0
3 0 0 19200180 279016 10756032 0 0 0 0 14340 9686 2 1 97 0
0 0 0 19199072 279016 10756376 0 0 0 2604 9914 3744 3 1 96 0
0 0 0 19198824 279016 10756628 0 0 0 16 9152 3485 1 0 99 0
0 0 0 19198692 279016 10756920 0 0 0 56 8502 6836 1 0 99 0
0 0 0 19198220 279016 10757200 0 0 0 0 8580 3178 1 0 99 0
>
Can you please suggest where to look or some changes in configuration, etc.?
I have logged all Berkeley DB internal statistics during the test: DatabaseStats, CacheStats, LockStats, LogStats, ReplicationStats, MutexStats, but there are plenty parameters. I do not know which parameters are interesting for me.
Best regards,
Alexander
Edited by: user12995955 on Jun 21, 2010 9:05 AM

Similar Messages

How can i get the meta data from database?

Hi, all java and db experts,
I need to write a tool to generate java file which will be used to hold the resultset of a stored procedure of Oracle. Is there any API call or tools to connect to db and then get the meta data of the return cursor instead of reading stored procedure definition on my own?
Please help, thanks a lot.
Hanna

if i execute a Oracle stored procedure, the resultset of a cursor is returned. It's easy to know the meta data at the runtime.
However, could i get the meta data about the resultset of a cursor before runtime? Such as by connecting to the database and ask it about meta data of a specified stored procedure?
Is it feasible?
DatabaseMetaData dbmd = conn.getMetaData();
ResultSet rs = dbmd.getProcedureColumns("", "%", "SP_NAME", "%");
while (rs.next()) {
String colName = rs.getString(4);
int colType = rs.getInt(5);
int colDataType = rs.getInt(6);
int colPrecision = rs.getInt(8);
int colLen = rs.getInt(9);
int colScale = rs.getInt(10);
long defaultValue = rs.getLong(11);
But what i get is a list of stored procedure parameters. In oracle, cursor is IN OUT parameter . How can i get the meta data about the resultset of cursor?

How can I iterate over the columns of a REF CURSOR?

I have the following situation:
DECLARE
   text   VARCHAR2 (100) := '';
   TYPE gen_cursor is ref cursor;
   c_gen gen_cursor;
   CURSOR c_tmp
   IS
        SELECT   *
          FROM   CROSS_TBL
      ORDER BY   sn;
BEGIN
   FOR tmp IN c_tmp
   LOOP
      text := 'select * from ' || tmp.table_name || ' where seqnum = ' || tmp.sn;
      OPEN c_gen FOR text;
      -- here I want to iterate over the columns of c_gen
      -- c_gen will have different number of columns every time,
      --        because we select from a different table
      -- I have more than 500 tables, so I cannot define strong REF CURSOR types!
      -- I need something like
      l := c_gen.columns.length;
      for c in c_gen.columns[1]..c_gen.columns[l]
      LOOP
          -- do something with the column value
      END LOOP;
   END LOOP;
END;As you can see from the comments in the code, I couln'd find any examples on the internet with weak REF CURSORS and selecting from many tables.
What I found was:
CREATE PACKAGE admin_data AS
   TYPE gencurtyp IS REF CURSOR;
   PROCEDURE open_cv (generic_cv IN OUT gencurtyp, choice INT);
END admin_data;
CREATE PACKAGE BODY admin_data AS
   PROCEDURE open_cv (generic_cv IN OUT gencurtyp, choice INT) IS
   BEGIN
      IF choice = 1 THEN
         OPEN generic_cv FOR SELECT * FROM employees;
      ELSIF choice = 2 THEN
         OPEN generic_cv FOR SELECT * FROM departments;
      ELSIF choice = 3 THEN
         OPEN generic_cv FOR SELECT * FROM jobs;
      END IF;
   END;
END admin_data;
/But they have only 3 tables here and I have like 500. What can I do here?
Thanks in advance for any help!

The issue here is that you don't know your columns at design time (which is generally considered bad design practice anyway).
In 10g or before, you would have to use the DBMS_SQL package to be able to iterate over each of the columns that are parsed from the query... e.g.
CREATE OR REPLACE PROCEDURE run_query(p_sql IN VARCHAR2) IS
v_v_val     VARCHAR2(4000);
v_n_val     NUMBER;
v_d_val     DATE;
v_ret       NUMBER;
c           NUMBER;
d           NUMBER;
col_cnt     INTEGER;
f           BOOLEAN;
rec_tab     DBMS_SQL.DESC_TAB;
col_num     NUMBER;
v_rowcount NUMBER := 0;
BEGIN
-- create a cursor
c := DBMS_SQL.OPEN_CURSOR;
-- parse the SQL statement into the cursor
DBMS_SQL.PARSE(c, p_sql, DBMS_SQL.NATIVE);
-- execute the cursor
d := DBMS_SQL.EXECUTE(c);
-- Describe the columns returned by the SQL statement
DBMS_SQL.DESCRIBE_COLUMNS(c, col_cnt, rec_tab);
-- Bind local return variables to the various columns based on their types
FOR j in 1..col_cnt
LOOP
    CASE rec_tab(j).col_type
      WHEN 1 THEN DBMS_SQL.DEFINE_COLUMN(c,j,v_v_val,2000); -- Varchar2
      WHEN 2 THEN DBMS_SQL.DEFINE_COLUMN(c,j,v_n_val);      -- Number
      WHEN 12 THEN DBMS_SQL.DEFINE_COLUMN(c,j,v_d_val);     -- Date
    ELSE
      DBMS_SQL.DEFINE_COLUMN(c,j,v_v_val,2000); -- Any other type return as varchar2
    END CASE;
END LOOP;
-- Display what columns are being returned...
DBMS_OUTPUT.PUT_LINE('-- Columns --');
FOR j in 1..col_cnt
LOOP
    DBMS_OUTPUT.PUT_LINE(rec_tab(j).col_name||' - '||case rec_tab(j).col_type when 1 then 'VARCHAR2'
                                                                              when 2 then 'NUMBER'
                                                                              when 12 then 'DATE'
                                                     else 'Other' end);
END LOOP;
DBMS_OUTPUT.PUT_LINE('-------------');
-- This part outputs the DATA
LOOP
    -- Fetch a row of data through the cursor
    v_ret := DBMS_SQL.FETCH_ROWS(c);
    -- Exit when no more rows
    EXIT WHEN v_ret = 0;
    v_rowcount := v_rowcount + 1;
    DBMS_OUTPUT.PUT_LINE('Row: '||v_rowcount);
    DBMS_OUTPUT.PUT_LINE('--------------');
    -- Fetch the value of each column from the row
    FOR j in 1..col_cnt
    LOOP
      -- Fetch each column into the correct data type based on the description of the column
      CASE rec_tab(j).col_type
        WHEN 1 THEN DBMS_SQL.COLUMN_VALUE(c,j,v_v_val);
                     DBMS_OUTPUT.PUT_LINE(rec_tab(j).col_name||' : '||v_v_val);
        WHEN 2 THEN DBMS_SQL.COLUMN_VALUE(c,j,v_n_val);
                     DBMS_OUTPUT.PUT_LINE(rec_tab(j).col_name||' : '||v_n_val);
        WHEN 12 THEN DBMS_SQL.COLUMN_VALUE(c,j,v_d_val);
                     DBMS_OUTPUT.PUT_LINE(rec_tab(j).col_name||' : '||to_char(v_d_val,'DD/MM/YYYY HH24:MI:SS'));
      ELSE
        DBMS_SQL.COLUMN_VALUE(c,j,v_v_val);
        DBMS_OUTPUT.PUT_LINE(rec_tab(j).col_name||' : '||v_v_val);
      END CASE;
    END LOOP;
    DBMS_OUTPUT.PUT_LINE('--------------');
END LOOP;
-- Close the cursor now we have finished with it
DBMS_SQL.CLOSE_CURSOR(c);
END;
SQL> exec run_query('select empno, ename, deptno, sal from emp where deptno = 10');
-- Columns --
EMPNO - NUMBER
ENAME - VARCHAR2
DEPTNO - NUMBER
SAL - NUMBER
Row: 1
EMPNO : 7782
ENAME : CLARK
DEPTNO : 10
SAL : 2450
Row: 2
EMPNO : 7839
ENAME : KING
DEPTNO : 10
SAL : 5000
Row: 3
EMPNO : 7934
ENAME : MILLER
DEPTNO : 10
SAL : 1300
PL/SQL procedure successfully completed.
SQL> exec run_query('select * from emp where deptno = 10');
-- Columns --
EMPNO - NUMBER
ENAME - VARCHAR2
JOB - VARCHAR2
MGR - NUMBER
HIREDATE - DATE
SAL - NUMBER
COMM - NUMBER
DEPTNO - NUMBER
Row: 1
EMPNO : 7782
ENAME : CLARK
JOB : MANAGER
MGR : 7839
HIREDATE : 09/06/1981 00:00:00
SAL : 2450
COMM :
DEPTNO : 10
Row: 2
EMPNO : 7839
ENAME : KING
JOB : PRESIDENT
MGR :
HIREDATE : 17/11/1981 00:00:00
SAL : 5000
COMM :
DEPTNO : 10
Row: 3
EMPNO : 7934
ENAME : MILLER
JOB : CLERK
MGR : 7782
HIREDATE : 23/01/1982 00:00:00
SAL : 1300
COMM :
DEPTNO : 10
PL/SQL procedure successfully completed.
SQL> exec run_query('select * from dept where deptno = 10');
-- Columns --
DEPTNO - NUMBER
DNAME - VARCHAR2
LOC - VARCHAR2
Row: 1
DEPTNO : 10
DNAME : ACCOUNTING
LOC : NEW YORK
PL/SQL procedure successfully completed.
SQL>From 11g onwards, you can create your query as a REF_CURSOR, but then you would still have to use the DBMS_SQL package with it's new functions to turn the refcursor into a dbms_sql cursor so that you can then describe the columns in the same way.
http://technology.amis.nl/blog/2332/oracle-11g-describing-a-refcursor
Welcome to the issues that are common when you start to attempt to create dynamic code. If your design isn't specific then your code can't be either and you end up creating more work in the coding whilst reducing the work in the design. ;)

Hi i have updated software in someone else's account and my iphone 4 turned in to there iphone so how can i get my all data back?

hi v have a same account me and my wife by mistake i have updated software in my wife's name so my iphone turned in to her's and now i can't find my data or my name on it so can u pls help me to get my data back there was an error (-50) appeared bu by mistake i override it so pls help me?

There's no way to get your data back unless you made a backup somewhere from which you can restore.

How can you transfer over all your pictures from icloud to andriod?

I just got a new aphone yesterday and its the new LG G2. I used to have an iphone and i have very important pictures on my iphone that i want on my new phone.. how would i go about doing that?

You can't do that via iCloud.
Connect the iPhone to a computer (Mac or PC) and copy the photos to it. Connect the new phone to the same computer and move the photos to it.

I am using iphone 3gs and have accidentally clicked "update" to software version 5. However, after I failed and all my data was lost. I manage to restore back to my previous situation. But all my apps has been gone. How can I get back all my aps?

I am using iphone 3gs and have accidentally clicked "update" to software version 5. However, after I failed and all my data was lost. I manage to restore back to my previous situation. But all my apps has been gone. How can I get back all my aps? I know I can get it by itunes but there are a few problems as below:
1. I have jailed break my iphone and I never purchased any apps from itunes
2. I have some important infomation saved in one of the apps called "awesome notes". How can I get it back? Is it store inside my computer? if so, how can i restore?
3. I am using window 7
I sincerely hope you guys can help me.
Regards,
Stephen Hong

alrite. I have made my mistake. I should not jail break my iphone. So if I purchase my apps again from itunes is that mean mean my data will be back?
I hope you guys can giv me some support here since I am Apple genuine buyer. I could be wrong last time and now I turn over a new leaf and learn a lesson.
Please noted that I can always create new account to ask for the same questions without mention about "jailbreak" if I want to but obviously I won't do that. Please provide me your support! Thank you!

How can I monitor my monthly data usage for all 3 computers in my house? I have an Airport base station and it seems there should be software to monitor it from that point rather than monitoring the usage for each computer and then adding it up.

How can I monitor my monthly data usage for all 3 computers in my house? I have an Airport base station and it seems there should be software to monitor it from that point rather than monitoring the usage for each computer and then adding it up.

The following example was one of dozens that showed up on a simple Google search of.....
monitor Internet data use on a Mac
Watch your Internet usage with NetUse Monitor | Macworld
Most service providers have an application for their users as well.

Hi,This morning I wanted to call, so I took my phone ( Iphone 4 S) ,but my All contacts list was empty! I checked in my computer ( windows 8) I have them in Iclould. How can I bring back my data?

Hi,This morning I wanted to call, so I took my phone ( Iphone 4 S) ,but my All contacts list was empty! I checked in my computer ( windows 8) I have them in Iclould. How can I bring back my data?

Hello Noushin,
It sounds like you are unable to see your contacts in the phone, but have confirmed they are still at http://www.icloud.com. I would next try these troubleshooting steps from the article named:
iCloud: Troubleshooting iCloud Contacts
http://support.apple.com/kb/ts3998
If you're using iOS 7, quit and restart the Contacts app on your iOS device:
Press the Home button twice to see preview screens of the apps you have open.
Find the Contacts preview screen and swipe it up and out of preview to quit the application.
Tap the Home button to return to your Home screen.
Wait a minute before reopening the Contacts app.
Turn iCloud Contacts off and back on:
Tap Settings > iCloud.
Turn Contacts off. Choose to delete data only if your data exists at icloud.com/contacts and on one or more of your devices. Otherwise, choose Keep Data.
Wait a few minutes before turning Contacts back on.
Restart your iOS device by holding down the Sleep/Wake button and then swiping the screen when prompted to power off. Then turn your device back on. This may sound simple, but it does reinitialize your network and application settings and can frequently resolve issues.
Thank you for using Apple Support Communities.
Regards,
Sterling

I have installed the latest version of iOS in my ipad but forgot to backup the content, what should i do!!! I don't want my media and other data erased, how can i find back all the data in my ipad? I haven't restore ipad yet, so any others way can be use?

I have installed the latest version of iOS in my ipad but forgot to backup the content, what should i do!!! I don't want my media and other data erased, how can i find back all the data in my ipad? I haven't restore ipad yet, so any others way can be use? Now my ipad is in recovery mode.

Once you are in Recovery Mode, it's too late to do any backup. I am afraid you are going to lose your data.

HT204150 i have deleted all contacts in icloud but still have them on my iphone (which is backed up as well to iTunes) how can i restore the iphone data to the iCloud so it will resend it to my mac book and iPad?

i have deleted all contacts in icloud but still have them on my iphone (which is backed up as well to iTunes) how can i restore the iphone data to the iCloud so it will resend it to my mac book and iPad? I have switched my iphone to airplane mode so it is not connected to the internet and so icloud can not wipe it at the moment.
thanks roy

There's a menu choice (I think under "Files") that lets you sync purchased items on a device back to itunes. Look for that.
iCloud does not store music. If you want to do that you'd have to subscribe to itunes Match, but to get music to the match servers, you need the music in the itunes library on your computer.

How can I take out all my HealthVault data?

As titled, how can I take out all my HealthVault data?

Hi,
I'm assuming by "take out" you mean you want to save copies of your data to keep outside of HealthVault.
In general, you want to use the Export feature. Find it by going to the Health Information page, then in the "More actions" menu, choose "Export information." If you're trying to send the data to another system, then you'll probably want to choose CCD or
CCR format. If you are keeping records for yourself, HTML would be the human-readable format. The different formats behave differently, so I'd recommend checking that the exported file contains the data you want.
After you export, then you'll want to go into the Files section of the Health Information page and download the files you want to keep. This will be especially important for files that came from a doctor, hospital, pharmacy, or lab.
If you want to close your HealthVault account, make sure you've got all the data you want before you close the account. By the way, if by "take out" the data you meant "delete" the data, when you close your account the data will be deleted from HealthVault
as per the Privacy Statement.
Hope this helps,
Kathy

Org Management - How can I change the start date of all the org units

How can I change the start date of all the org unit objects and its relationships at one shot ? I have created some 100 Org units with the wrong start date. Please suggest.
Thanks in advance..
Cheers
Vijay

Hello Archana,
Thanks a lot... This has solved my issue...
Cheers
Vijay

HT5625 I cannot send a text thru iMessage. I followed the directions over and over again but can't make it work. Also how can I find out all the apple ids I may have.

I cannot send a text thru iMessage. I followed the directions over and over again but can't make it work. Also how can I find out all the apple ids I may have.

A wet phone is out of warranty. This is considered user damage. Even if you were able to get it to start now, the chances of it working for long are slim. I suggest going to Apple and see about an OOW replacement. One for the iPhone 4 is only $149USD and it would come with a short warranty. It is a refurbished device and you would not be worried about encountering additional problems.

I just pressed the wrong button to erase of my IMac, how can I get back all information from my computer ?

I just pressed the wrong button to erase of my iMac, how can i get back all information ?

Possibly through a data rescue service (can cost a four-figure amount) if you didn't have an external backup.
But you must stop using that Mac immediately, or your data will be over-written.

ICloud storage space used up by old data from 1st gen iPad I traded in nearly 2 yrs ago for an iPad Mini. Can't backup any info for mini or new 6 plus. How can I access this old data to clear it out?

My iCoud storage space is being used up by old data from a 1st generation iPad I traded in nearly 2 yrs ago for an iPad Mini. The people at the sprint store where I made my trade were supposed to transfer everything over from my old iPad to the new Mini once they backed up the old device to iCloud. Well, long story short, none of the data ever got transferred to the new iPad, but it all got backed to iCloud, eating up all my storage space. Here I am almost 2 years later, and I've never been able to back-up my iPad mini. What's worse, My iPhone 4 wasn't backed up that whole time, and now I can't back-up my data from my new iPhone 6 Plus to iCloud either. Trying to find creative ways to store and save my information and data between all my devices is getting to be ridiculous. I don't want to upgrade my storage space though...just want to clear out a great deal of what is already being stored. How can I access this old data to clear it out?

The only way to access data from an iCloud backup is to restore it to your device. You could, for example, back up your iPad mini to your computer using iTunes (by connecting it, opening iTunes, then going to File>Devices>Back Up and choosing to back up apps and transfer purchases when prompted), then erase it and restore the old iPad backup (as explained here: iCloud: Restore or set up your iOS device from iCloud), save any recovered data you want to keep, then restore your iTunes backup back to your iPad mini (by connecting it to your computer, opening iTunes, then going to File>Devices>Restore from Backup).
Once you're done with the old backup, delete it from your account to free up the space.

How can I iterate over all data in database?

Similar Messages

Maybe you are looking for