What type of monitoring/report will help me here? AWR/ASH/SQLTUNE?

Question

Hi All,
Oracle v11.2.0.2 Linux 64 bit.
In our application there is a service which keeps running on a Wondows machine and runs few (3 to 4) SQLs repeatedly.
Yesterday evening between 22:50 and 23:15 we had a performance issue with the service i.e. normally these SQLs which take between 3-6 seconds, were taking 35, 40 seconds. AFter 23:15, there was no problem because there was no data (or very less data) to process for next 2-3 hours. Later on it was running fine (back to the original timings).
Now I want to investigate that specific issue I had yesterday (about 12 - 14 hours ago). I have the SQL Ids, session id.
I don't have access to Enterprice manager or Grid control, only SQL prompt. However I can ask the DBAs to generate a report, but with all required details.
Problem is, what report do I run? what to use?
1) AWR is set to 1 hour interval. And my problem window is split in 2 different snapshots. If I look at 2 AWR reports ( 22:00 - 23:00 and 23:00 - 00:00), the numbers don't show much of an issue, as all the numbers are averaged out for the whole 1 hour (i.e. a SQL performed well 15 times and did badly 5 times, the overall average won't look bad)
2) ASH report is a complete mistry to me. I am trying to generate the report with ashrpt.sql / ashrpti.sql. Most of the sections say "No data exists for this section of the report". When using ashrpti.sql do I need to enter all the values it ask? Is there a Oracle's recomended way to give values and run this report? Documentation does not show a sample/example call which will work.
3) DBMS_SQLTUNE reports are also not showing any data. Almost blank reports. How can I find out if the data required for DBMS_SQLTUNE is there? or how long that data is retained?
We always get such delays randomly, ask a DBA, they say, "No problem". Not their fault, but they are looking at averaged out numbers.
Thoughts please.

Answer 1

Hi,
your problem is that you are trying to resolve a problem related to a specific statement by using instance-wide tools. You should look at performance stats for that particular statement.
1) Identify sql_id of the offending statement (e.g. select sql_id from v$sql where sql_fulltext like '%part_of_statement_text_here%', if it doesn't return results check case or look in
dba_hist_sqlstat)
2) Look for records with this sql_id in DBA_HIST_ACTIVE_SESS_HISTORY (use filter on sample_time). Compare performance indicators for the timeframe of the incident and outside it
3) You can also find performance stats for this statement in DBA_HIST_SQLSTAT, but you will have aggregates over snapshot duration
Given your symptoms, the most logical explanation is a plan flip. Check plan_hash_value in DBA_HIST_SQLSTAT -- if it changed around the time of the incident, that will confirm your suspicion.
Use dbms_stats.display_awr to display both "good" and "bad" plan. Try to understand the key difference, look at predicates for any hint that could explain it. If it doesn't give you results,
use format=>'advanced' to generate outlines for both plans, then you can set STATISTICS_LEVEL=ALL and run the statement with both plan, producing extended plan statistics. Use
dbms_xplan.display_cursor(..,..,format=>'allstats last') to display it. If having difficulty interpreting it, post it here to get the help from community.
Best regards,
Nikolay