Working with large database with multiple large tables ( orcale 11)

Hi all,
I'm trying to read and parse 3 tables:
NetworkEvent - 500000 entries
NewTrapEvent - 500000 entries
NewTrapVlaue - 2000000 entries
I I used 1 JDBC connection, 2 statements and 3 ResultSet( i used same statement for two queries).
My questions are:
==========
1. Is it reasonable? or should I add more connections? or more statements?
2. Additionally when creating the Result Set I used the following parameters:
ResultSet.TYPE_SCROLL_INSENSITIVE
ResultSet.CONCUR_READ_ONLY
and stmt.setFetchSize(1000)
is it valid parameters? should the fetch be larger?
is there any more important parameters that I missed?
3. When setting the SQL command, I used each command to retrieve all the table entries:
"select * from ana37.NetworkEvent order by detectiontime"
Should I take partial entries in a loop? (for example 100000 in each loop) or is it ok for the Result Set to hold all of the entries( 2 milion)
4. Can I clean the result set when I finished processing previous entries?
thx
Edited by: user9001513 on 06:59 23/11/2010

Why don't you use a stored proc?
Why are you ordering it?
Should I take partial entries in a loop? Yep. Because software isn't perfect. No point in attempting to process the universe when you know it will fail sometime and it is easier to handle smaller failures than large ones (and you won't have to redo everything.)

Similar Messages

Can we query through 5-6gb large database with AIR

As it creates one single file for whole database, can we have 5-6 GB large database if an AIR application requires

There's no arbitrary limit to the database size. It would depend on performance and the user's file system, I suspect. Only you could judge the performance aspect as it should depend on the complexity of your database and queries.

Web AS 6.40 Sneak Preview: Problems with monitoring Database with SQL Studi

Hello!
I'm new with SAP and SAP Web AS, and I installed the SAP Web AS 6.40 Sneak Preview recently. I'm exercizing with the tutorials, and in the Tutorial "Car rental Application", a "QCR_RESERVATION" database is created and deployed.
The deployment worked fine, but I'm having problems accessing the SAP DB with SQL Studio.
At first, I didn't find SQL studio in my SAP installation, so I installed it afterwards. I now tried to log on with the data mentioned in the tutorial (localhost, C11, SAPC11DB and "sap"), and got the error "Database not running (-709)". Looking at the SAP Management Console, I found a database with the name "J2E", so I tried this one.
But now the problem is, that with user SAPJ2EDB and password "sap", I get the error "(-4008) Unknown username and password", and the same error with the combination "control", "control".
What is my mistake? Could anybody give me a hint? Maybe there is something wrong with my Web AS 6.40 Sneak Preview Installation? I don't remember having changed any settings during installation ...
Thank you in advance
Bärbel Wolf

Dale, thank you very much.
Your hint really is useful, I know myself, I'd most probably need unlocking my admin user soon. )
But nevertheless, I'm still looking for the right password for the database user "SAPJ2EDB".
As I'm encountering problems with the "Quick Car Reservation" tutorial (the entity bean seems unable to access the database, too), I'd like to check the QCR_RESERVATION database with my SQL Studio. Maybe there's something wrong with it.
Greetings
Bärbel

Update question in a large database with partitions

Hello All!
Please help me in the following issue; I'm given a database that has partitions from 0 to 63. In each partition there can be 0 to any number of a given field like "NAME" (this may the primary key I think).
How can I update a field in a row corresponding to a "NAME" at a given partition range, if I don't know which NAME belongs to which partition.
Problems: there are around a million of this NAME field and the fastest method is only acceptable. I think I make an update command with a logical condition with a partition range (x<partition<y) and has a where clause like this:
NAME like value1 or like value2.... or like value(10^6) (here I enumerate all of the NAME field) : is it a good way?
How much is the longest where statement that can be given as a query to a database?
Or is it another way to do it?
Thank you for your kind help in advance,
Balage

I have a list that contains the NAMES, like:
110
111
112
113
The table is the following
NAME OLD PART
110 0 23
111 1 23
112 1 56
The NAME list is in a txt file, the OLD field should be updated according to the NAMES, and I'm writing a perl script to do the job. A thread is given a partition (PART) interval. Like the first thread works on
the PART numbers from 1 to 3... the 2nd is on from 4 to 6.. etc.
The problem is, that the NAME list is huge and I don't know which NAMEs are in a
given PART interval.
I thought this command will do the job:
UPDATE table SET old = new where 1<= part <=3 AND name LIKE 110 OR LIKE 111 OR LIKE... stc
how long can be the where clause after "name LIKE" as I have plenty of NAME records?
Hope you can understand it, sorry for my bad English,
regards,
Balage

Parsing XML document with nested elements into multiple db tables(master-detail)

Can you help me with storing xml document into master-detail tables?
I have two tables:
1) customers (customerid number primary key, firstname varchar2(30),lastname varchar2(30))
2) cust_addresses (customerid number references customers, street varchar2(30),city varchar2(30),state varchar2(10),zip varchar2(10))
I have XML document:
<?xml version="1.0"?>
<ROWSET>
<ROW num="1">
<CUSTOMERID>1044</CUSTOMERID>
<FIRSTNAME>Paul</FIRSTNAME>
<LASTNAME>Astoria</LASTNAME>
<HOMEADDRESS>
<STREET>123 Cherry Lane</STREET>
<CITY>SF</CITY>
<STATE>CA</STATE>
<ZIP>94132</ZIP>
</HOMEADDRESS>
<HOMEADDRESS>
<STREET>N.Fryda 4</STREET>
<CITY>CB</CITY>
<STATE>CZ</STATE>
<ZIP>37005</ZIP>
</HOMEADDRESS>
</ROW>
</ROWSET>
I know that I must use DBSM_XMLSave package, create view and instead of trigger but I did found no example in documentation.
Thanx.

Interested question; one I do not know the answer to at the moment, I am afraid. I would like to know your results.
To tell others, though, what I have done. I did finally adapt Muench's not-yet-published examples #71 to work with a table. Here the table has as it's value "#{backing_app_EPG_DAYPG.jobDayDriverTable[row.Id1]}"
This accesses a hash map defined in the backing bean as follows:
public Map jobDayDriverTable = new HashMap(){
@Override
public Object get(Object key) {
Number jobDayId = (Number)key;
if (getEpgDayPage().jobDayDrivers(jobDayId) != null) {
return getEpgDayPage().jobDayDrivers(jobDayId).getAllRowsInRange();
else return null;
jobDayDrivers returns RowIterator. From this I call getAllRowsInRange which returns a Row[]. The table consumes this as a value, and lists the values as I want.
Since the table is using Rows for its rows, I am guessing that it would have access to #{row.rowKeyStr}, or at least #{row.<pk>} which would allow you to programmatically set the current row using code like the following:
public static boolean setCurrentRow() {
// BindingContainer bindings = getBindings();
OperationBinding operationBinding =
// bindings.getOperationBinding("setCurrentRowWithKey");
(OperationBinding)EL.get("#{bindings.setCurrentRowWithKey}");
Object result = operationBinding.execute();
if (!operationBinding.getErrors().isEmpty()) {
return false;
return true;
You could call this as part of code in a pages backing bean behind a button or link.
Hope this helps somebody.
By the way example #71 was to get a detail set of rows from a master row value and display the detail set in the master row of an af:table.
You can find this and other examples at http://radio.weblogs.com/0118231/stories/2004/09/23/notYetDocumentedAdfSampleApplications.html

Database Design - Multiple Customer Tables

Scenario:
- Let's say there are various customer "types"
- Each one has a set of attributes which is only applicable to that customer type.
- You have a table for orders that is for all customers
- Want to treat it as one customer table for reports, updates, etc...
The solution we've come up with is to basically have a lookup table which allows you to lookup the table the customer record exists in. The order table would have a "customer type" column to key off of. You then use a view to fill in the missing columns from each table for say a global query.
I'm assuming this is a common business problem so thought I would ask the experts if there's a "best practice" for solving this problem.
Thanks

Think vertically ... not horizontally.
An infinite number of attributes can be stored in finite space if you do. For example:
CREATE TABLE horizontal (
customer_id NUMBER,
attribute_val1 VARCHAR2(20),
attribute_val1 VARCHAR2(20),
attribute_val1 VARCHAR2(20));
CREATE TABLE vertical (
customer_id NUMBER,
attribute_name VARCHAR2(20),
attribute_value VARCHAR2(20),The first example requires you to add a column and break everything every time you add a new attribute. The second keeps right on working.

Oracle E-Businesssuite with RAC Database with PCP enabled.

While setting up Transaction Managers for PCP, we need to set the following parameters
Could you please let mek now, what are these? why need to set?
Edit $ORACLE_HOME/dbs/_ifile.ora. Add the following parameters:
lmglobal_posts=TRUE
immediatecommit_propagation=TRUE

Hi,
lmglobal_posts if TRUE deliver global posts to remote nodes
immediatecommit_propagation if TRUE, propagate commit SCN immediately
These parameters are used for read consistency.
Regards,
Mithun
Refer:
http://www.recovery-for-oracle.com/Params/parm590.htm
http://www.orafaq.com/parms/parm761.htm
Hope this helps.

Can I use two different database with DBSystemLoginModule?

Finally I can login with Database user id by DBSystemLoginModule on Embedded oc4j server.
But I confused between [b]jazn-data.xml and system-jazn-data.xml.
Which xml do I have to edit for jdbcUrl and jdbcDriver?
Now I would like to use two different database(such as only sid is different).
In this case I want to show specific database name as realm name.
What should I do?
When I use DBSystemLoginModule, I edit JdbcUrl in system-jazn-data.xml.
So can I pass the parameter for sid programmatically?
And also I tried to run my application with other database with DBSystemLoginModule on Embedded oc4j server.
I changed few things(jdbcUrl at system-jazn-data.xml,jazn-data.xml in embedded-oc4j-server/config and database connection for BusinessComponents at Project Properties of Model/ViewController.
But I got some error.
Did I miss something?
----< my error messgae >------
500 Internal Server Error
JBO-30003: The application pool (customer.model.datamodel.CMAppModuleLocal) failed to checkout an application module due to the following exception:oracle.jbo.JboException: JBO-29000: Unexpected exception caught: java.lang.NullPointerException, msg=null at oracle.jbo.common.ampool.ApplicationPoolImpl.doCheckout(ApplicationPoolImpl.java:2002) Message was edited by:
Message was edited by:
sdcsdc

I think I understand now, you are modifing the custom login modules configuration within the system-jazn-data.xml file. This is a bit dangerous! Didn't you register the login module in your orion-application.xml? For example, this is the way I do it:
<web-module id="customProv-web" path="customProv-web.war" />
<persistence path="persistence" />

<security-role-mapping name="sr_developer">
<group name="developers" />
</security-role-mapping>
<security-role-mapping name="sr_manager">
<group name="managers" />
</security-role-mapping>
<jazn provider="XML">
<property name="custom.loginmodule.provider" value="true"/>
<property name="role.mapping.dynamic" value="true"/>
<property name="role.compare.ignorecase" value="true"/>
</jazn>

<jazn-loginconfig>
<application>
<name>customProv</name>
<login-modules>
<login-module>
<class>oracle.security.jazn.login.module.db.DBTableOraDataSourceLoginModule</class>
<control-flag>required</control-flag>
<options>
<option>
<name>data_source_name</name>
<value>jdbc/OracleDS2</value>
</option>
<option>
<name>roles_fk_column</name>
<value>userName</value>
</option>
<option>
<name>table</name>
<value>userinfo</value>
</option>
<option>
<name>groupMembershipTableName</name>
<value>groupinfo</value>
</option>
<option>
<name>groupMembershipGroupFieldName</name>
<value>role</value>
</option>
<option>
<name>usernameField</name>
<value>userName</value>
</option>
<option>
<name>user_pk_column</name>
<value>username</value>
</option>
<option>
<name>passwordField</name>
<value>passWord</value>
</option>
</options>
</login-module>
</login-modules>
</application>
</jazn-loginconfig>
<log>
<file path="application.log" />
</log>
<namespace-access>
<read-access>
<namespace-resource root="">
<security-role-mapping name="<jndi-user-role>">
<group name="administrators" />
</security-role-mapping>
</namespace-resource>
</read-access>
<write-access>
<namespace-resource root="">
<security-role-mapping name="<jndi-user-role>">
<group name="administrators" />
</security-role-mapping>
</namespace-resource>
</write-access>
</namespace-access>
</orion-application>

How to do mappings in owb with different databases

hi all,
I have two different databases with in same server one for staging and target.
How to connect both target and staging databses to do the mappings using OWB??
Is it possible?? if possible how to do the mappings??
for ex:
Now i registered my OWB with bcp database for both staging tables and target tables.
now i want to load data of staging tables into bcpsource database and
target tables into bcptarget database seperatly.

10gR2 does have connectors. They were introduced at the same time as locations.
The connection attributes (eg host, port, tnsname) are set against the location, not the connector. A connector simply defines the fact that there is a path from one location to another (eg via a database link). All the connection information remains on the location object.
The process flow packages within a location can each be deployed on their own simply by selecting the location in the Control Center Manager, and setting a deployment action against the package (or by using scripting, as mentioned). However, a process flow cannot be deployed independently of it's package - it's all or nothing.
Nigel.

Rman backup with standby database

Hi All,
I am looking for some suggestions for rman backup with standby database with datagaurd but no active DG. I wanted to do a rman tablespace backup. but I am stuck at sql 'alter system archive log current' in rman with standby as target. which gives following errors with or without noswitch clause
RMAN> sql 'alter system archive log current noswitch';
sql statement: alter system archive log current noswitch
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of sql command on default channel at 10/05/2013 18:15:12
RMAN-11003: failure during parse/execution of SQL statement: alter system archive log current noswitch
ORA-01109: database not open
Recovery Manager: Release 11.2.0.1.0
Oracle databaes Release 11.2.0.1.0 64 bit
OS:RHEL 6 x86-64 bit
Secondly, do I need to cancel manged recovery while taking rman backups on standby.
I scanned thru MOS but could find solutions for rman with active DG.
Thanks in advance

726d07e6-b870-431f-8c22-a499c3ec4e0c wrote:
Hi All,
I am looking for some suggestions for rman backup with standby database with datagaurd but no active DG. I wanted to do a rman tablespace backup. but I am stuck at sql 'alter system archive log current' in rman with standby as target. which gives following errors with or without noswitch clause
RMAN> sql 'alter system archive log current noswitch';
sql statement: alter system archive log current noswitch
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of sql command on default channel at 10/05/2013 18:15:12
RMAN-11003: failure during parse/execution of SQL statement: alter system archive log current noswitch
ORA-01109: database not open
Recovery Manager: Release 11.2.0.1.0
Oracle databaes Release 11.2.0.1.0 64 bit
OS:RHEL 6 x86-64 bit
Secondly, do I need to cancel manged recovery while taking rman backups on standby.
I scanned thru MOS but could find solutions for rman with active DG.
Thanks in advance
"I scanned thru MOS but could find solutions for rman with active DG."
Howto make a consistent RMAN backup in an Standby database in Active DataGuard mode (Doc ID 1419923.1)

Connect to oracle database with visual studio 2005

Hi, i have a oracle database called orcl (10.2). I am trying to connect with this database with visual studio 2005. I do this:
In Server Explorer, right click in Data Connection and click in add connection.
I choose oracle database and oracle provider.
In server name I write the name of my computer, and i write the user and pass of the data base (sys or system) but it gives me this error: "ora-12514 listener doesnt know the asket for service in the description one of connection" (translated of Spanish)
what i have to write in server name? host, host:port, LISTENER,.... what?
Thanks

Sometimes, you can check names using from the command line:
$> tnsping <orcl>
TNS Ping Utility for 32-bit Windows: Version 8.1.7.0.0 - Production on 10-ENE-20
06 09:01:04
(c) Copyright 1997 Oracle Corporation. All rights reserved.
Attempting to contact (ADDRESS=(PROTOCOL=TCP)(HOST=server1)(PORT=1521))
Realizado correctamente (0 mseg)
$>

Using older client 10201 with 11g database

Hello,
Are there any issues with using sqlplus client 10201 with 11g database with regard to performance? I would think using the 11g client with 11g database would be preferred but what would be the disadvantage in using sqlplus 10201 with 11g database?
Also, what would be disadvantage if using sqlplus 11g with an older database 10201/10202? Thank you.

I guess the most visible and the prominent issue of using an old version client with the more latest version db would be that the older version client won't be able to display/use some of the newest features. This sounds more like a usability issue than performance issue which you mentioned but that's what which came on top of the mind. Performance should not be affected as the client doesn' do much except to pull the data so if there would be any issues actualy , they would be with the server rather than the client.
Just my 2 cents.
HTH
Aman....

How to Connect Forms or any version with Oracle Databases 8/6i/9i/10i/11i

I want to know that how can i cannect Developer Forms / Reports with Oracle Databases with various versions and say developer 2000 to Oracel 6i/9i/10i/11i etc

Using the appropriate setting in tnsnames.ora file located in your [Forms_Home]\network\admin directory.
Sim

Problem with large databases.

Lightroom doesn't seem to like large databases.
I am playing catch-up using Lightroom to enter keywords to all my past photos. I have about 150K photos spread over four drives.
Even placing a separate database on each hard drive is causing problems.
The program crashes when importing large numbers of photos from several folders. (I do not ask it to render previews.) If I relaunch the program, and try the import again, Lightroom adds about 500 more photos and then crashes, or freezes again.
I may have to go back and import them one folder at a time, or use iView instead.
This is a deal-breaker for me.
I also note that it takes several minutes after opening a databese before the HD activity light stops flashing.
I am using XP on a dual core machine with, 3Gigs of RAM
Anyone else finding this?
What is you work-around?

Christopher,
True, but given the number of posts where users have had similar problems ingesting images into LR--where LR runs without crashes and further trouble once the images are in--the probative evidence points to some LR problem ingesting large numbers.
I may also be that users are attempting to use LR for editing during the ingestion of large numbers--I found that I simply could not do that without a crash occuring. When I limited it to 2k at a time--leaving my hands off the keyboard-- while the import occured, everything went without a hitch.
However, as previously pointed out, it shouldn't require that--none of my other DAMs using SQLite do that, and I can multitask while they are ingesting.
But, you are right--multiple single causes--and complexly interrated multiple causes--could account for it on a given configuration.

BRTOOLS with tape continuation for large database

Hello,
I have a R/3 database of 2TB which needs to be copied onto a new staging server for upgrade.
The problem I am facing is that I dont have any space in the SAN(storage) for taking the backup on disk.
So the only option I have is to take backups on tape.
Even tried backup with compress mode on tape but ended up with CPIO error for handling larger than 2GB files(note 20577).
And Since the database size is 2TB and the tapes I have hold 800GB I would have to use multiple tapes.
Since my some of the files in the database like *DATA_1 range between 6GB to 10GB CPIO cannot
handle files larger than 2GB as per note 20577.
So I had to change the parameter tape_copy_cmd = dd in init<sid>.sap.
But DD will end once the end of tape is reached with a error message thereby failing my backup.
Please help me get out of this situation.
Regards,
Guru

Hi,
Please check the 'Sequential backup' section in the backup guide. If its not possible to use a tape with a big capacity, you could use this method instead.
You would need to add/ modify the following parameter in init<sid>.ora :
1. volume_backup
2. tape_address
3.tape_address_rew
4.exec_parallel
You'll find more info about these parameters in www.help.sap.com and in the backup guide itself.
Br,
Javier

Working with large database with multiple large tables ( orcale 11)

Similar Messages

Maybe you are looking for