EDQP: knowledge studio - create data lens

Question

Hello Experts,
while creating data lens of a csv file, it recommends creating number of sample files with certain number of rows.
i have 90k rows in my csv file and it suggested 20 sample-files with 100 records each.
Little docuementation is available on managing the sample files, like delete, combine sample files etc. What i am looking is:
1. methodology of using these sample files for creating the data lens
2. with 20 sample-files and 100 records each, these are 2K records. How can I have the created data lens (rules) applied to my total # for rows (90k).
3. There are two other files created along with the sample files - 1.baseline.xml and 1.reserve.xml,,,,,what are these and how these are useful in creating the data lens.
Thanks a lot
aps

Answer 1

aps,
How and when to work with Sample Files depends on how you are implementing EDQP.  Standalone or with Oracle PIM.
If you are working with EDQP standalone, the number and type of sample files are first determined by the Item Definitions (Categories) that will be handled within each Data lens.  To determine which and how many item definitions will be handled by each data lens, you will need to have some sort of taxonomy defined.
Identifying the product taxonomy to use for EDQP is one of the first points of consideration when implementing EDQP, either as a standalone solution or with a non-PIM solution.
Generally it is ideal to have no more than 40 data lenses.  However, many businesses operate successfully with 10-15 data lenses.  Often a data lens may correspond to a Level 1 or Level 2 UNSPSC level.  The following factors should be considered when setting up the data lens taxonomy:
Number of categories per data lens ( recommended 250 item definitions maximum)
Mapping to categories that correspond to areas of responsibility within the business
Generally, the concept of product categories is familiar to most businesses. It often makes sense to use these categories as is, rather than identify a new taxonomy for the purposes of the EDQP solution.  However, if the taxonomy is not a product taxonomy (i.e. if the taxonomy is a display taxonomy), then it is advisable to adopt a product taxonomy. 
There are three points of consideration
The item definition structure must be a product hierarchy.  See discussion in chapter 2 for details about this.
Each leaf-node in the taxonomy should differ from other leaf nodes based on attributes (not attribute values). E.g.  Blue ballpoint pens and Black ballpoint pens should not be in different leaf-node categories, simply because the Color attribute value differs.
Once you have determined what categories from your taxonomy will be address in the Data lens you are working with, you are now ready for a strategy on sample files:  so now I will answer your questions in this context assuming you have done the above:
1. methodology of using these sample files for creating the data lens
     The methodology is to create enough sample files for the specific categories to be handled by the datalens.  We have empirical evidence that randomizing the data from a large corpus to data sample of 100 is a good way to test, refine, and augment the rules. the methodology of the randomized sample data is as follows:
     a. Create the recommended number of sample files for the set of categories to be handled by the data lens being refined.
     b. Open the first sample file of 100 items.  The lens (depending on how many rules or on the method or generating the lens (thru Autobuild from PIM Metadata or manually top down) will recognize between 0% to maybe 40% of the items in the sample.
     c.  Work on refining and augmenting the rules to recognize as close to 100% of the items in sample data file 1.  When done.  Open sample data file 2
     d.  What you will notice is that sample data file 2 will start at a much higher percent of recognition than sample data file 1.  This is because the randomized file will contain phrases that repeat from sample file one.
     e. Work on sample data file 2 until you reach close to 100% recognition. And now open sample data file 3
     f.  What you will notice is a pattern, sample data file 3 will open with a higher recognition percentage than 2 and much higher than 1.  -- you now get the methodology....as you continue to progress from the sample files, the random files will continue to increase in recognition percent until you reach a flattening of improved results.  At this point, the knowledge building has reached a point where you are no longer getting a good ROI in the lens building in terms of increased recognition.  This usually peaks between 95-100 percent due to various reasons including the variability or sparsity of the attributes in the data. 
2. with 20 sample-files and 100 records each, these are 2K records. How can I have the created data lens (rules) applied to my total # for rows (90k).
     a. If the 90K rows are really all related to the categories intended to be recognized by the datalens, then yes, starting with 20 sample files is recommended to build the rules.  Follow directions above on processing files.  If you don't reach the desired recognition percent, then point to the reserve file and create another 20 samples...this should be repeated until your recognition pattern flattens between sample files. 
     b. To test the theory that after you flatten out, the lens will recognize a larger population which was reflected in the sample files, the system created the additional two file types.  Baseline and Reserve.  See below for explanations on each:
3. There are two other files created along with the sample files - 1.baseline.xml and 1.reserve.xml,,,,,what are these and how these are useful in creating the data lens.
     Baseline:  Baseline file is usually a file of 1000 rows of randomized data from the larger set.  It is used to test the recognition percent of the Data Lens against a larger population of the data.  Use this to test if you are really flattened.  For example, if you are sample data file 8 and it opens up at 99% recognition.  So you then open up random sample file 9 and it opens up at 99% recognition.  This would be an indicator that you are converged in the rules creation.  To test this against a larger sample population, you open up Baseline file and in most cases it will open up at 96-98% for example.  If it does not, it means that maybe samples 8 and 9 were anomalies, so you should then go back and open up sample file 10 and continue building rules.
     Reserver:  The reserve file is a file that will store 10% of the total population that is not contained int he sample files.  It it used to create more sample files which you will know do not overlap with the original set.
I hope this helps.  let me know if you have additional questions.
Luis