Bulk email from the database

Question

Good morning,
Running Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production... (we'll be upgrading to 11g soon, but for now, stuck with 9i)
Our clients have requested an email subscription service to be implemented for our news web site. The anticipated load will be:
one module to support the following:
- 3000 emails, with up to 50 instant email notifications an hour... so 150,000 emails delivered per hour. Email is sent as soon as new article is posted. (articles are embargoed and go live once embargo has passed)
The other module is a open subscription form with double opt-in for the general public... with choices of instant email notification, daily digest or weekly digest or news article postings. the anticipated load is difficult to estimate for this one, but we could expect 50,000+ subscriptions... with up to 500 article posting per day. end users will have the ability to customize their subscription based on audience, department etc...
In summary, we're looking to implement a solution that can deliver hundred of thousands of emails a day.
As an Oracle developer, I always look first to the DB for solution. I know in 10g and up, utl_mail is available... not so in 9i. I do have a sample mail package from OTN that seems to do the trick. Emails will need to be sent as HTML, since they will contain a small image reference.
Some potential ideas I have considered so far:
for instant email notifications:
- setting up queue tables and DBMS_JOB to monitor every minute and send email out to recipients using PL/SQL mail procedure
- using Oracle AQ to manage queues, publish the payload with complete article information, subscriber will dequeue messages and send out using same PL/SQL email procedure. Messages in the queue will have delay set to article embargo date, to ensure articles are not emailed ahead of publishing.
I have never worked with Oracle AQ before, but it seems to offer some benefits and more intelligence than a custom solution. I have also considered setting up a new Oracle instance for sending out emails, to offload some of the work from the main instance feeding the news web site.
the daily and weekly digest emails are not as big a concern at this point, since they will be processed during off-peak hours, and will run once day/week... the Oracle AQ solution I thought would be an elegant and scalable solution for the instant notifications...
My major concern at this point is scalibility and performance... I will rely on bulk processing in SQL to collect data... looping through the arrays to send out emails, as well as building up the email objects and sending them out in time is a concern.
Given the potential volume we'll be dealing with, is a solution in Oracle the proper way to go? Our organization don't have an enterprise solution to this of course, so we have to build it from scratch. Other environments/tools at our disposal, Oracle 10g (10.1.3) application servers running Java ... could use JavaMail... our current set-up is 2 load-balanced application servers, with multiple OC4J containers running on each.
Thanks for any tips or advice you may have.
Stephane

Answer 1

Bill... thank you for taking the time to respond in such a detailed manner. This is greatly appreciated. I have passed this along to our DBAs and messaging experts for review.
I'm tasked with modeling the DB and optimizing it so it can achieve the target performance levels.
Billy  Verreynne  wrote:
Each SMTP conversation (from UTL_SMTP or UTL_MAIL) requires a socket handle. This is a server resource that needs to be allocated. The o/s has limits on the number of server resources a single process can use and that a single o/s account can use. Does not help that you have a scalable design, scalable code, and the server cannot supply the resources required to scale. So server capability and config are quite important.We've gotten our Unix specialists to look into this... we've been told our current platform is limited, and no further upgrades will be allocated, since we'll be moving to a newer platform in the "near future".
There's also the physical network itself - as that 100,000 mails will each have a specific size and needs to be transported from the client (Oracle db server) to the server (mail/smtp server). The network must have the capacity and b/w to support this in addition to the current loads it is handling.Our network analysts will be putting us on a segregated network (subnet) to avoid impacting the rest of the organization... although bandwitdh is shared at the end of the day, we'll be somewhat isolated, with perhaps even our own firewalls and load balancers.
You will need to look at a couple of things to decide on how to put all of this together. How are e-mails created in the database? Can they be immediately transmitted after being committed? Is the actual Mime body of the e-mail created by the transaction, or it is normal row and column data that afterward needs to be converted into a Mime body for mailing?The scheduling of emails are tricky.... articles are sometimes posted (comitted to the DB), but still under embargo... we plan to use this embargo date (date on which article goes live/public) as the "delay" in the job scheduling. At that point, article is emailed to thousands.
For instant notifications, all recipients get the same content, article content direct in email. There custom pieces to include, eg unsubscribe link with unique identifier, edit subscription etc... but these bits could be pre-generated and stored with the email subscriber info, and appended to the mail body. No images or other binary files are embedded or attached... so we're dealing with mostly text/html in the body.
Scalability is a direct function of design. So you will need to design each component for scalability. For example, do you create the Mime body dynamically as part of the e-mail transmission code unit? Or do you have a separate code unit that creates and persists Mime bodies that then are transmitted by the transmission code unit? Instant notification html email composition is a bit simpler. It gets tricky with daily, weekly, monthly digests. Here we have to assemble thousands of custom email bodies, based on subscription options (up to 5 custom fields, to configure subscription content)... from there assemble each individual mail body based on current subscriber options, then send the email.
Mail bodies are potentially different for each individual subscriber, given the various permutations of selections, so really don't see how a body can be persisted for re-use when emailing, each may be single use only. In this case, looking at assembling thousands of emails, then emailing each one in a loop.
Do you for example queue a single DBMS_JOB per e-mail? There are overheads in starting a job. So do you pay this overhead per mail, or do you schedule a job to transmit a 100 e-mails and pay this overhead once per 100 mails?for instant notifications, we'd be queuing a job for every article posted. From there, every email subscribed to instant notifications, and matching their subscription configuration to the article, will be retrieved and email sent.
So there are a number of factors to consider ito design, how to deal with Mime bodies, how to deal with exception processing, how to parallelise processing and so on. One factor will need to be on how to deal with catchup processing - as there will be a failure of some sorts at a stage that means processing is some hours behind. And this needs to be factored into the design.An email job that fails half-way through concerns us... how do we proceed where we left off etc... may have to keep track of job numbers etc...
The other option we're considering, is clustered Oracle 10gR3 application servers... to process and send the emails, using JavaMail... there is still an issue with Oracle handling the query volume required to assemble the customized emails for each subscriber (which could reach 50,000 within a year or two)...
I would not select that as an architecture. This moves the data and application away from one another - into different process boundaries and even across hardware boundaries.
When using PL/SQL, both data processing (SQL layer) and conditional processing and logic (PL layer) are integrated into a single server process. There is no faster way or more scalable way of combining code and data in Oracle. It does not matter how capable that Java platform/architecture is. For that Java code to get SQL data means shipping that data across a JDBC connection (and potentially between servers and across network infrastructure).
In PL/SQL, it means a simple context switch from PL to SQL to fetch the data.. and even that we consider "slow" and mitigate using bulk processing in PL/SQL in order to decrease context switching.
The fact the the data path for a Java app layer is a lot longer than for PL/SQL, automatically means that Java will be slower.totally agree with this. We're having a meeting this morning with all parties to review and discuss the points you have raised, and see if the required resources can be allocated on the Unix side to accommodate the potential load.
>
I'm looking at leveraging materialized views (to pre-assemble content), parallelism (query and procedural), Advanced Queuing (seems complex)... AQ is not that complex.. and perhaps not needed. You will however need a form of parallelism in order to run a number of e-mail transmission processes in parallel. The question is how do you tell each unique process what e-mails to transmit, without causing serialisation between parallel processes?
This can be home rolled parallelism as shown in {message:id=1534900} (from a technique posted by Tom on asktom that is now a 11.2 feature). You can also use parallel pipelined tables. Or use AQ. I'm pretty sure that a solid design will support any of these - modularising the parallel processing portion of it and allowing different methods to be used to test drive and even benchmark the parallel processing component.
If using AQ, we're considering a separate Oracle instance in a different AIX partition perhaps, which could manage the email function. Our main instance (which feeds our public web site, and stores all data), would push objects onto the queue, and items would be dequeued on the other end in the other Oracle instance.
It however sounds like a very interesting project. Crunching lots of data and dealing with high processing demands... that is the software engineer's definition of fun. :-)Indeed... I wouldn't consider myself as a software engineer at this point, but perhaps after this is done, I'll have earned my stripes. ;-)
Edited by: pl_sequel on Jul 13, 2010 9:57 AM