Anomaly detection using ODM

Question

I was asked the following question:
"My question is very simply, we are doing a monitoring system for a
website that helps the admin to mine on specific data (using ODM to
produce Web mining) so we want to apply the anomaly detection. We dont
know what we should do and what we should produce as a results."
A couple of suggestions come to mind:
1) For an overall discussion of intrusion detection in general using the Oracle RDBMS as an analytical platform the following paper might be useful:
http://www.oracle.com/technology/products/bi/odm/pdf/odm_based_intrusion_detection_paper_1205.pdf
2) A couple of things to think about and do:
(a) Define what is the "mining case", that is, the "object that defines what is the concept you want to mine. For example, in web mine you may want to detect anomalous session activity. This can be defined over the whole activity of a session or over time windows. In the first case each session will define a mining case (it will be a row in the training data). In the second case each section will generate many mining cases, one per time window. Let's assume for sake of discussion that the goal is to identify anomalous session activity. Then the training data will consist of the session activities (e.g., clicks, pages visited, and/or information from forms; or more generally, http requests). There will be one row per session in the training data. If we know beforehand that some of those sessions where intrusion or anomalous in some sense we can also capture this data as a target for supervised modeling.
(b) Decide what modeling to do. Two types of modeling can be performed (see the paper above for examples):
(i) Supervised modeling - case there are examples of anomalous cases as well as normal cases
This can be done by building a classifier on the training data. It is also possible to measure the quality of the classifier on a held aside sample.
(ii) Unsupervised modeling - this should be done as well even if we can create a supervised model
Unsupervised approaches don't provide a measure that indicates how good the model is at predicting anomalous events. These models are better at ranking cases by how anomalous the model believe they are.
Two common unsupervised techniques for anomaly detection are: Clustering and One-Class SVM. The latter is considered a state-of-art in many problem domains and it is the one implemented by ODM. ODM also has clustering but it does not return distance of a row to the center of cluster. This information is necessary for using it clustering for anomaly detection. If one wants to use clustering, the Oracle Data Mining blog has a post that can help compute distance from rows to centroids:
http://oracledmt.blogspot.com/2006/07/finding-most-typical-record-in-group.html
It is important to note that the method described in the post doesn't support nested column attributes.
When building unsupervised models, only the data for normal cases should be used to training the models. The unsupervised models can be seen as defining what is normal. It will recognize that something is anomalous when it does not match the definition of normality learned by the model.
(c) Use ODMR to help with modeling
(d) As new session information is gathered it is possible to score in real-time the session to detect anomalous behavior. One should score both supervised (if information was available) and unsupervised models to detect anomalous behavior. See the above paper for some discussion on this.
The supervised model will indicate if a case is anomalous or not based on known types of anomalous behavior. One should use ROC tuning in ODMR to find a good operating point for the model. This is necessary because the number of anomalous cases is usually small compared to normal ones.
The unsupervised model (one-class SVM) will provide a ranking. The higher the probability of belonging to class 1 the more normal. A 0.5 probability for class 1 indicates the boundary between normal and not normal. In reality it marks a boundary where normality dominates. There can be some anomalous cases with probability higher than 0.5 and some normal cases with probabilities less than 0.5. If working in batch mode we can rank the probabilities in ascending order and select the first K rows for investigation.
--Marcos

Answer 1

A suggestion to speed up the process: provide more information about your data (e.g., schema) and how you are invoking the algorithm (GUI, API, settings). Case you are using the APIs, have you tried the sample programs for anomaly detection?
Regarding the Apriori algorithm it does not support timestamps and dates columns. In fact, none of the algorithms in ODM does (see the documentation for Oracle Data Mining for the supported column data types). the dbms_predictive_analytics package does. Are you trying to do sequential association rules or just trying to do plain association rules using data from a date column? ODM does not support the former. The latter can be done by converting the date column to a VARCHAR or NUMBER column.
--Marcos