Design considerations while accessing data of a different web application

Question

I am database developer for a web application (*WebApp1*) which supports around *6 million* merchant data across various tables in database (*DB1*).
The users for the above mentioned application are less, around *50* who monitor the 6 million merchant data.
Now a proposal has come up to develop a second web application for the merchants themselves.
Thus the user base for this web application (*WebApp2*) is in millions.
Now the requirement for WebApp2 is :
On each login of individual merchants, they need to see their individual data (mostly read-only) which is stored in (*DB1*)
along with other data which is stored in its own database (*DB2*). WebApp2 will have other functionalities too but those
are independent of WebApp1.
Now I can think of three approaches how to display DB1 data in WebApp2 :
1. Creating a stored procedure and accessing DB1 data through dblink. (*Cons* - have seen access through dblink is slower as it serializes access).
2. Replicate DB1 data to DB2 through batch jobs running every night. (*Cons* - Will store the same data in 2 places and will have the headache of data sync.)
3. WebApp1 will expose webservice using DB2 data, so that on each login, WebApp2 users use the webservice to display DB1 data through webservice.
(Please answer if there are any cons. beacuse of high user base ?)
Need your suggestion and views. Also if there are any other way please let me know.
Will it depend on any other factor ? Let me know if you need any further information.
Cheers !!!

Answer 1

Ah - this is a much bigger question! And may be outside the scope of this forum to answer. I will have a go though.
The question really isn't can the database support 6,000,000 users. It is how many of the users will be making database requests at the same time, what is the average elapsed time of each request, and what do your users regard as an acceptable response time to each request?
To use a (very) simple analogy, my company employs 30,000 people but the IT Service Desk that they call if they have a problem with their PC is only staffed by 30 people. This is because the Service Desk Manager has estimated that at any one time only 1% of all employees will need to call the Service Desk. On average each call takes 6 minutes to resolve and the response time agreed with the business to a call is 1 hour. So within an hour one Service Desk technician can answer 10 calls in an hour, so 30 technician can answer 300 calls an hour.
Now you can apply this analogy to a computer system. The scalability of database systems is generally limited by the amount of available CPU. If we determine that the average database call uses 10ms of CPU, and the user will accept a response time of 1s, then each CPU can service 100 database calls within that response time. Multiply that by the number of available CPUs and you have a (very rough) idea of how far the database will scale. An server with 8 cores could support 800 concurrent requests. A cluster with 4 servers each with 8 cores could support 3200 concurrent requests.
I would stress that this is very crude. Much more information would be required to determine what the limits of database scalability were. My example does not take account of other source of potential latency, such as the network or the storage subsystem if requests involve significant physical I/O. Or indeed the latency in the application itself.
So, you need to answer these questions:
- How many concurrent database requests need to be supported
- What is the average amount of database CPU that each database request will consume
- What is the maximum acceptable elapsed time for each database request
On a more practical note, your application must use some form of connection pooling to limit the number of database connections required to service all the required database activity. You can scale the database horizontally using RAC and in extreme cases you can offload read activity entirely from the database using a middle-tier caching solution such a Coherence, GigaSpaces or GemFire.
That's probably enough for now. Any more and you'll probably need to get some consultancy!