Unity Connection 8.6.2 report error and issues

Question

Hello,
I found some other hits on this error and tried the workarounds (restarting tomcat, and report data harvester) but am still getting "Unable to find any report data based on parameter(s)" for a certain timeframe on multiple reports when I know activity occurred in that timeframe.
User Phone login report, Outcall Billing Detail Report, others.
CUC version is 8.6.2.20000-2
Problem is during this timeframe someone dialed into system and changed the transfer extensions to various LD numbers throughout an entire day and the reports aren't showing these outcalls. Call Manager CDR data show the calls being placed from Unity VM ports and the Unity Conversation Manager traces I grabbed for the timeframe show the calls to those numbers to.  I'm just trying to match up with a report and figure out if there is some other log or trace file I can look to determine for sure the caller logged in to the voicemail box and changed the transfer extension on it. (guessed the voicemaill password). The failed log in report doesn't even show the failed login attempt in output from traces below. The transfer extension dest addr was different on the transfers on this same mailbox.  Have locked down the system since but want to get better grip on this with the reports and traces.
I looked over the CUC 8.6.2 SU1 and SU2 release notes and not seeing any resolved bugs for these matters either.
Here are some samples lines from Conversation Manager traces:
00:32:46.399 HDR|02/04/2013 ,Significant
00:32:46.399 |9744,PhoneSystem-1-001,E68CA6DB3CAE4118BE9F9612BD3540A9,Arbiter,-1,Incoming Call [callerID='' callerName='' calledID='555123' redirectingID='555123' lastRedirectingID='555123' reason=4=FwdNoAnswer lastReason=8=FwdUncond] port=PhoneSystem-1-001 portsInUse=1 ansPortsFree=23 callGuid=E68CA6DB3CAE4118BE9F9612BD3540A9
00:33:55.006 |9744,PhoneSystem-1-001,E68CA6DB3CAE4118BE9F9612BD3540A9,MiuGeneral,25,Enter CAvMiuCall::TransferEx destAddr='715551231234' type=2=Release maxRings=4 mediaSwitch='00a499fa-a193-45b0-a153-564a64019eb8'
00:33:58.955 |9744,PhoneSystem-1-001,E68CA6DB3CAE4118BE9F9612BD3540A9,MiuGeneral,25,Exit CAvMiuCall::TransferEx=0x00000000=S_OK
00:34:37.090 |9832,PhoneSystem-1-001,9DB394E443A3419FA0020BC9D1A92806,Arbiter,-1,Incoming Call [callerID='' callerName='' calledID='555123' redirectingID='555123' lastRedirectingID='555123' reason=4=FwdNoAnswer lastReason=8=FwdUncond] port=PhoneSystem-1-001 portsInUse=1 ansPortsFree=23 callGuid=9DB394E443A3419FA0020BC9D1A92806
00:34:50.403 |9832,PhoneSystem-1-001,9DB394E443A3419FA0020BC9D1A92806,CDL,11,CCsCdlImsAccess::AuthenticateSubscriber: Cannot authenticate user: JOESMITH IMS Result Code: 1 on line 108 of file CdlIms/src/CsCdlImsAccess.cpp: Error: 0x80046505 Description: E_CDL_SP_EXEC_FAILURE
00:34:50.403 |9832,PhoneSystem-1-001,9DB394E443A3419FA0020BC9D1A92806,CDE,3,Authentication Failed for Subscriber  555123 (Src/CsCallSubscriber.cpp 2969)
00:34:50.404 |9832,PhoneSystem-1-001,9DB394E443A3419FA0020BC9D1A92806,-1,-1,An invalid password entered when trying to log into a user mailbox. Details -  [].
00:36:29.331 |9832,PhoneSystem-1-001,9DB394E443A3419FA0020BC9D1A92806,MiuGeneral,25,Enter CAvMiuCall::TransferEx destAddr='715551231234' type=2=Release maxRings=4 mediaSwitch='00a499fa-a193-45b0-a153-564a64019eb8'
00:36:33.484 |9832,PhoneSystem-1-001,9DB394E443A3419FA0020BC9D1A92806,MiuGeneral,25,Exit CAvMiuCall::TransferEx=0x00000000=S_OK
00:44:53.444 |9743,PhoneSystem-1-001,FEB6975B82284F37A161CD66ECBA61C6,Arbiter,-1,Incoming Call [callerID='' callerName='' calledID='555123' redirectingID='555123' lastRedirectingID='555123' reason=4=FwdNoAnswer lastReason=8=FwdUncond] port=PhoneSystem-1-001 portsInUse=1 ansPortsFree=23 callGuid=FEB6975B82284F37A161CD66ECBA61C6
00:45:20.019 |9743,PhoneSystem-1-001,FEB6975B82284F37A161CD66ECBA61C6,MiuGeneral,25,Enter CAvMiuCall::TransferEx destAddr='715551231234' type=2=Release maxRings=4 mediaSwitch='00a499fa-a193-45b0-a153-564a64019eb8'
00:45:24.121 |9743,PhoneSystem-1-001,FEB6975B82284F37A161CD66ECBA61C6,MiuGeneral,25,Exit CAvMiuCall::TransferEx=0x00000000=S_OK
Thanks

Answer 1

Hello ebergquist,
As advised we could try to run a report before an upgrade was performed to verify if it will pull the information, which in the event it does we would be matching the bug, therefore access the CUC CLI and type the following command:  file get install *
You will be required to have an SFTP available for the file transfer and port 22 open on the sftp client.
Once you receive the files you would need to check system-history.log, below is an example.
=======================================
Product Name -    Cisco Unity Connection
Product Version - 8.5.1.12900-7
Kernel Image -    2.6.9-89.0.20.EL
=======================================
02/24/2013 13:35:02 | root: Boot 8.5.1.12900-7 Start
02/24/2013 16:03:45 | root: Install 8.5.1.12900-7 Success
Once you check the  time stamp and date of the upgrade try to gather a report for a time range before the upgrade was performed, even though they might be overwritten, there still could be a possibility that it succeeds.
If this bug is not the issue then you might want to take into consideration:
1. Make sure that the service ReportDB is [STARTED] in the output of the command: utils service list.
2. If you have Digital Networking on your server make sure there is no stalled replication. Go to RTMT (Real Time Monitoring Tool)> Syslogs> Select on the dropdown your server> Doble click on Application logs> CiscoSyslog. You can then do a manual search or sort with the filter option:
Severity: Error
App ID: CuReplicator
Hit apply. Below is an example:
Date: Feb 19 08:01:15
Machine Name: LAB1111
Severity: Error
App ID: CuReplicator
Message: : 1151681: LAB1111.corpnet2.com: Feb 19 2013 08:01:11.662 UTC :  UC_UCEVNT-3-EvtReplicatorStalledReceiveReplication  %[ClusterID=][NodeID=LAB1111]: Detected stalled replication receiving from location LAB1111. Waiting for missing USN 2222. This situation may indicate network connectivity problems.
3.Make sure that replication is working properly in the cluster. You can check them via the following commands:  showcuc cluster status    and   utils dbreplcation runtimestate
For the TransferEx that you state changed to ceros ->  TransferEx=0x00000000=S_OK , this is just a hexadecimal value that is returned stating that the transfer that took place during the call was successful, in this case the actual transfer was to the following destination -> TransferEx destAddr='715551231234', and was during a call to ->calledID='555123' , this does not have to do with the actual change of a transfer number performed by a user.
Those traces won't be enough you will have to additionally set: ConvSub, level 3   and  CDE, level 16 so that you may see output in the traces as the example posted in the previous thread.
Traces would be the path to follow in order to verify the user that changed the transfer rule, as this is as i understand the goal your pursuing.
However if it is of any help, another way to get on hold of the numbers that were updated on the Transfer Rule is to use the User Data Dump tool, this information reflects changes regardless of whether it was done via TUI (Telephone User Interface) or GUI (Graphic User Interface). For more information on the tool go to:
http://ciscounitytools.com/Applications/CxN/UserDataDump/Help/UserDataDump.htm
Traces are extensive to be reviewed if done manually, however you can do a text search with a tool as Notepad ++ to look for keywords as "NewTransferNumber =" or "CDE,16,Transfer allowed", once you find this you check the call GUIDwhich is the reference number of the call and that exists during the entire length of the call, whether it is while a person is leaving a voicemail or the user is accessing his/her's inbox.  This would speed up your search.
So as explained in the first thread you would then do another text search returning all possible matches with that callGUID.
Now if you get to the very beginning of the call you would see something like:
Line 13712: 18:58:05.156 |24720,CCIE-1-001,0C9C97BDF5F444F0B535D231C32DDCB4,Arbiter,-1,Incoming Call [callerID='2199' callerName='' calledID='6789' redirectingID='' lastRedirectingID='' reason=1=DirectlastReason=1024=Unknown] port=CCIE-1-001 portsInUse=1 ansPortsFree=1callGuid=0C9C97BDF5F444F0B535D231C32DDCB4
In this example you see my test user callerID='2199'  calling the Pilot number 6789 which is the voicemail button
calledID='6789, and the what type of call is it whether Forward or Direct, when logging to voicemail you are going to see Direct which is going to match the Direct Routing Rules and send the call to the attempt sign in conversation.
The other approach is a rule out method between the Audit traces and the User Data Dump tool.
So let's say you have a baseline document that states all of the Transfer numbers for all users.
If the User Data Dump reflects a different number and the Audit logs do not show any change, starting from the time the baseline was created and the point of time where the number was changed, then you could deduce the change was done via TUI (Telephone User Interface). Unfortunately you wouldn't be able to tell who did the change in this case.
Nevertheless, remember that a user may have for example an Alternate Extension so the change might not necessarily be done from the office phone: "Alternate extensions can make calling Cisco Unity Connection from an alternate device—such as a mobile phone, a home phone, or a phone at another work site—more convenient. When you specify the phone number for an alternative extension, Connection handles all calls from that number in the same way that it handles calls from a primary extension (assuming that ANI or caller ID is passed along to Connection from the phone system). This means that Connection associates the alternate phone number with the user account, and when a call comes from that number, Connection prompts the user to enter a PIN and sign in."
Best regards,
David  Rojas
Cisco TAC Support Engineer, Unity
Email: [email protected]
Phone: [phone] ext 6406
Mon, Wed, and Fri 12:00 pm to 9:00 pm ET, Tue and Thu 8:00 am to 5:00pm ET
Cisco Worldwide Contact link is below for further reference.
http://www.cisco.com/en/US/support/tsd_cisco_worldwide_contacts.html