BIA Delta Index Merge

Hi Gurus,
                 We have some cubes for BIA. In most of the cases, cubes get full and delta loads from different info sources and in some cases we have to delete the prior requests from the cubes.
Now I have following questions:-
1.     Which cube is best candidate for rebuilding the index and which for delta index?
2.     Which step in process chain we should call a program to merge the delta index?
3.     Should we create a separate process chain to merge delta index after all the loads are finished or we should call in the existing process chain.
Please help.
Warm Regards,
Anil

Similar Messages

How to track BIA delta index merge and rebuild

Hi Gurus,
             I have set up process chains which load the BIA delta index and full index rebuild in separate process chains.
Is there any way to check if BIA is behaving as desired? I know to check RSDDSTATTREX table.
Regards,
Anil

Not a reply from Guru..
But I put my thoughts in and here are few take aways for you ,which you may wish to have a look in to..
PS - BI Accelerator is quite a Virgin territory ..few documentations or persons worked with ..
You can execute some performance checks with tcode - RSRV to validate the same.
<b>BI Accelerator Performance Checks</b>
1)Size of Delta Index
If you have chosen delta mode for an index of a table, new data is not written to the main index but to the delta index. This can significantly improve performance during indexing. However, if the delta index is large, this can have a negative impact on performance when you execute queries. When the delta index reaches 10% of the main index, the system displays a warning.
The system performs a merge for the index in repair mode. The settings are retained.
2) Propose Delta Index for Indexes
It is useful to create a delta index for large indexes that are often updated with new data. New data is not written to the main index, but to the delta index. This can significantly improve the performance of indexing, since the system only performs the optimize step on the smaller set of data from the delta index. The data from the delta index is used at the runtime of the query.
The system determines proposals from the statistics data: Proposals are those indexes that received new data more than 10 times during the last 10 days. A prerequisite for these proposals is that the statistics for the InfoCube are switched on.
Data in the main index and delta index should be merged at regular intervals (see test Size of Delta Index).
In repair mode, the system sets the Has Delta Indexproperty for the proposed indexes. The delta index is created when the data is next loaded for this index.
3)Compare Size of Fact Tables with Fact Index
The system calculates the number of records in both fact tables (E and F tables) for the InfoCube and compares them with the number of records in the fact index of the BI accelerator index. If the number of records in the BI accelerator index is significantly greater than the number in the InfoCube (more than a 10% difference), you can improve query performance by rebuilding the BIA index.
The following circumstances can result in differences in the numbers of records:
○       The InfoCube was compressed after the BI accelerator index was built. Since the BI accelerator index is not compressed, it may contain more records than the InfoCube.
○       Requests were deleted from the InfoCube after the BI accelerator index was built. The requests are deleted from the BIA index in the package dimension only. The records in the fact index are therefore no longer referenced and no longer taken into account when the query is executed; however, they are not deleted.
http://help.sap.com/saphelp_nw2004s/helpdata/en/6b/cda64246c6c96ae10000000a155106/content.htm
Hope it Helps
Chetan
@CP..

When to set delta index flag for master data?

In transaction RSDDBIAMON2 option "Set Delta Flag" shows:
Table Name                     |Table Size          |Delta Index
/BIC/FTSGCSGMC          |10,155,000          |check mark in box
/BIC/DTSGCSGM1           |5,000,000           |check mark in box
/BI0/SVERSION               |700                    |check mark in box
When should I check the "Delta Index" column for fact, dimension, and master tables? I believe I need to check fact and dimension tables for delta BIA to occur during roll up so users can see the newly loaded requests but I am uncertain on when and why I should check this box for master data.
Thanks for your input.

Vitaliy,
Thanks for answering another of my question. Yes, we load master data a couple of times a day so based on your answer, I should also check box "Delta Index" for the master tables as well. Thanks for helping me understand this point.
If the below is our approach, #2 should be checked for all tables, correct?
1) Create/fill BIA index for cube "A"
2) Check the box "Delta Index" for all the tables which includes fact, dimension, and master tables for cube "A" .
3) Run process chain to roll up cube "A" daily
4) Run process chain to merge cube "A" weekly
Many thanks,
Thao

Talent Management - TM - Trex 7.1 - Delta Indexing

luke.....are you out there??
congrats on your blog being featured right on the SDN homepage... very nice...
http://scn.sap.com/community/erp/hcm/blog/2010/09/20/setting-up-netweaver-embedded-search-trex-for-sap-ehp4-talent-management
and congrats to Chris on his first two excellent blogs....
http://scn.sap.com/people/chris.mcnarney/blog/2010/08/26/whats-included-with-the-predefined-performance-management-process
http://scn.sap.com/people/chris.mcnarney/blog/2010/09/09/whats-included-with-the-predefined-performance-management-process--part-2
all 3 blogs are very helpful since there is little information available out there on these topics...
so we've been testing the delta indexing program very heavily - ESH_IX_PROCESS_CHANGE_POINTERS
we have found that it does not appear to be able to index new hires (new central_person index does not pick up the new CP). The only way this data can be indexed is by running a full re-index. That is problematic for us, b/c we were hoping to only do full re-indexing 1 a week or even 1 a month. our indexes take 2-3 hrs to complete and trex searching is pretty much inoperable during that time.
Have you tested new hires with delta indexing? it seems as a gap in the change_pointers program. We opened up OSS message to SAP.
some additional thoughts working with the program, relative to your blog. I would highly recommend that anyone testing this makes sure the following notes are implemented in your system. The first one is the most recent and we ourselves did not have that implemented.
1. 0.350 BC-EIM-ESH 1441190 Error in model assignments to logical system 24.02.2010
2. 0.230 BC-EIM-ESH 1408655 Alerts in syntax check due to unused indexes 18.11.2009
3. 0.260 PA-TM 1395703 Positions and termination are missing for indexing 30.10.2009
4. 0.330 BC-EIM-ESH 1249482 Functional corrections for Enterprise Search 7.01 SP03 26.01.2009
Another area which we found from testing: SAP by default does not delta index structural authority view. What this means is that if a new talent specialist is added with 741 relationship there access to see that structure will not be available in TREX search by default delta indexing. This would also impact existing organizations and positions shifted under existing 741 relationships.
The information in spro is actually quite helpful, you can find it in: talent management and talent development > basic settings > search > BAdI: Activate Delta Indexing for Authorization Data
Activtate the BADI, than the program to "create the change pointers" must run periodically in your system. This is done with RPTMC_CREATE_CHANGEPOINT_AUTH. Once that program creates the change pointers the next time your scheduled delta program runs (ESH_IX_PROCESS_CHANGE_POINTERS) all of the structural auths will be created and indexed correctly.
If we can just now get new hires to work with ESH_IX_PROCESS_CHANGE_POINTERS we will probably only run the full re-index once a week or once a month.
Let me know your thoughts,
Michael
Message was edited by: Luke Marson
Message was edited by: Luke Marson

so if anyone is interested, from my previous post I mentioned that new hires were not getting picked up by the change pointers program.
after further testing though, we found that when the authorization recon program identifies the new hires and appropriately creates a change pointer.
conclusion: to avoid full re-indexing of trex indexes, perform the following steps:
1. implement / activate the BADI - HRTMC_AUTHORITY_VIEW - talent management -> basic settings -> search
2. Schedule program RPTMC_CREATE_CHANGEPOINT_AUTH - every 30 mins, or 1 or 2 hr. This program will capture whenever anything changes from an authorization data perspective. This would include identifiying new central persons (new hires) or new TMC specialist being setup with 741 relationship. This program "creates change pointers" which than feeds the main delta program - ESH_IX_PROCESS_CHANGE_POINTERS.
3. Use Luke's blog on the instructions for the delta program - ESH_IX_PROCESS_CHANGE_POINTERS - this program you can have run every 5 mins, 10 mins, 15 mins, based on your needs. this processes all delta changes, the first time the program runs after the RPTMC_CREATE_CHANGEPOINT_AUTH program finishes it will also processes the data authorization changes, otherwise every other time the program is running it will catch non-data authorization for all the other indexes (employee name change, new performance rating, new potential, position name change, new profiiciency scale, new career profile data, etc).
hope this helps somebody!
michael
Edited by: Michael L Pappis on Oct 7, 2010 12:05 AM

Embedded Search - no delta Indexing

Hi Experts,
i want to configure Embedded Search in CRM 7.0. TREX is well configured and all Embedded Search configuration steps are done. I created all BO-Connectors in ESH_COCKPIT. But now I can`t start delta indexing (Actions-Schedule Indexing) because the Button is grey. The status of the connectors is new.
I also tried scheduling indexing with the report ESH_ADM_INDEX_ALL_SC, but it takes no effect.
What should i do?
Thanks,
Tobi

you mean on 7.0 and now you are on 7.1. Can you please be specific as i am also facing the same problem. i am on TREX 7.1 and tried report 'ESH_IX_PROCESS_CHANGE_POINTERS' but no any effact on trex search result. I created soem new data for job family and talent groups but its not coming up in HRTMC searches.
apprecite any help..
Mani

BIA physical indexes with no assignment to logical index (InfoCube)

Hi there.
In RSDDBIAMON2 I can see the following warning: one BIA physical indexes is not assigned to any logical index. They explain it as the following: when a BI InfoProvider is deleted from the BI accelerator, some physical indexes of this kind of index join may not be fully deleted and may remain on the hard disk.
Question: how to delete this physical index which is not assigned any more to any BIA index? If I even rebuild all BIA indexes this physical index may not be affected as I suppose.
Have anybody had such kind of issues? Please advise.
Thanks
Vadim

Zdravstvyite i vi Vitali
It's exactly my question. Yes, I know the physical index name but unfortunately I cannot find the way to delete it not in TREXADMIN transaction, not in TREX Admin stand alone tool...
Will try again if you think it's there
Thanks,
Vadim

Embedded Search delta indexing

Hello all,
i am trying to create delta indexing for EA-HRXXX component in SAP ECC 6 EH4 on TREX search 7.10 (standalone) but its not taking into account my changes.
I created some test data on ECC6 EH4 system related to talent group (new talent group) and job family (for talents) , now my HRTMC_Search_OBJECTS not retruning data related to those objects. i tried to run program 'ESH_IX_PROCESS_CHANGE_POINTERSESH_IX_PROCESS_CHANGE_POINTERS' as said in documentation (http://help.sap.com/saphelp_nw70ehp1/helpdata/en/47/fb4c38469e59e8e10000000a421937/frameset.htm) but its not showing results.
any reply will be highly appreciateed.
Mani

Hi Mani,
yes, there are moderators in SDN.
But in the end the forums are community froums - they are not a support system with guaranteed answering times or anything of the like. So, if you have questions about mission critical errors, please do use SAP support.
If you are looking for answers to "how to..:" questions, SDN forums are a great channel, but - again - without any answering guarantee.
By the way, I understood you correctly that you have found an answer to your query, right?
Even if that may have been "not possible"...
Best, Karsten

How to use the delta index in TREX?

We have been using TREX with EP KM for 2 years.
However we never used the delta index functionality.
Could you share your experience on how to use delta indexes?
Thanks!

Hai,
Please check the below link.....
http://help.sap.com/saphelp_nw70/helpdata/EN/d9/0418418291a854e10000000a1550b0/frameset.htm
Regards,
Yoganand.V

Delta Indexing

Hi All,
We have created a programme named RSRETDELTAINDEXING by refering SAP note 1003868 - KPRO Delta indexing report. Usually it indexes the delta indexing in search criteria. But presently it is not working.
If we do the full indexing through SAP standerd programme RSTIRIDX it is working. But the full indexing will take much time to index the same.
Can anyone help me out to solve this delta indexing issue..
Regards
Harshavardhan.G

If anyone has same problem in user search (who is who)...i found a solution, that works for me.
After a user changes his properties in his user profile i do the following
IIndexService indexService = (IIndexService)ResourceFactory.getInstance().getServiceFactory().getService(IServiceTypesConst.INDEX_SERVICE);
IIndex i = indexService.getIndex("ume");
IResourceFactory aResourceFactory = ResourceFactory.getInstance();
IResource rsrc = aResourceFactory.getResource(RID.getRID("/ume/users/" + user.getUniqueID() +".usr"), aResourceFactory.getServiceContext());
i.indexDocument(rsrc);
This way only the changed user is reindexed.
regards
Alex

Integrating a Delta Index into the Main Index Using the Python Schedule

Dear All
As per below link we are failed to merge the delta to main index in Trex 7.10.
http://help.sap.com/saphelp_banking60/helpdata/ru/89/65a740aa053a13e10000000a155106/content.htm
Kindly help us
Regards
Sriram

Hi Mikhail
1. I did find any error message
2 In the Corntab.ini file at the end typed **0 14 * * * "python mergeDeltaIndex.py --silent --allIndexes=1 ''** is it right or not ?
3. after that how we can find weather out setting are correct or not?
Kindly help us
Regards
Sriram

TREX - how to setup delta indexing?

Hi,
I have got TREX installed and I could search in Succession Planning.
However, when I changed some data (for e.g. assigning a new user id to a PERNR), that new ID couldn't perform a search on employee.
I have scheduled the program ESH_IX_PROCESS_CHANGE_POINTERS, and this program could be completed without error.
But still, I couldn't perform a search until I manually run program ESH_ADM_INDEX_ALL_SC.
FYI, I'm following Luke's guide to setup the TREX:
Setting up NetWeaver Embedded Search (TREX) for SAP HCM Talent Management EhP4, 5 and 6
Regards

The program ESH_IX_PROCESS_CHANGE_POINTERS is little bit different than regular background job.
It looks like a hang, but this program will keep calling SUBMIT of itself. It takes care of its own scheduling based on parameters you provide.
It will finish only after 'Exit processing after [hours]' you gave in selection screen.
Infact its not struck at 'Indexing object type: HRTMC_REL_S_JF_450'
It says that is the last index it worked on.
If it finds change pointers to any other object, it will index them and add a message at the end of job log saying 'Indexing object type: xxxxxx'. Make a change in org and monitor this job to validate this.
You see the job 'on hold', 'SLEEP' because it will wait for a while before it start checking for change pointers again. This duration is based on 'Delay[seconds]' parameter you specified in selection screen.
I thing your problem is that change pointers are not being generated properly which is the task of program RPTMC_CREATE_CHANGEPOINT_AUTH.
Thanks,
Nag

ABAP to set "Switched on/off BIA indexes for queries" flag?

Hi everyone,
Does anyone know if the "Switch on/off BIA Indexes for Queries" can be set for a specific cube by a delivered ABAP?
Here's my scenario. We go live with BIA next Monday 9/22. I've indexed our cubes in production BIA this past weekend and they're rolled up nightly. Until the go-live, I've manually set the "Switched Off for Queries" flag via RSDDBIAMON > BI Accelerator > Index Settings > Switch on/off BIA Indexes for Queries".
However, one indexed cube is deleted fully every night by the flag in the full load infopackage. In testing, I saw that the BIA fact table index was fully deleted when the E and F tables of the cube were truncated, and then reindexed when the full package is loaded and rolled up. This is all ok.
The issue is that after the delete, load and rollup, the "Switched Off for Queries" flag is not set. I have had to manually reset to off in the morning. This won't be an issue (hopefully) once we go live, but it does seem like a bug and I'll likely submit a customer message. I would think the delete, index and rollup process should not change the status of this flag.
Does anyone know if that flag can be programatically set?
Thanks for any advice,
Doug

Thanks, Vitaliy!
I think you're exactly right. Its seems to be a bug in processing of just this specific type of index reprocessing, e.g. dropping and reindexing. The indexing process properly retains INA status on the other delta indexed cubes.
I'm barely ABAP OO literate, but do understand enough to find, review and test the class and method code. Thanks for finding it!
Also, I reviewed table RSDDTREXDIR and it is correct/current. It does spawn another question, though. The field ITYPE (Type of BIA Index) is consistently ICB exept for 2 cubes/indexes, which are typed as PA2 and PA9. When I check possible values on the domain, RSDDTREX_TAGGR_TYPE, it only lists ICB and ICF (Infocube - flat).
Any idea what types PA2 and PA9 are, and why these 2 cubes would be different? From my perspective, they're basic infocubes, same as all the others.
Thanks again,
Doug

BIA Rev. 7.00.53.00, DBMAN 305, RSD_TREX 100, unloading temp indexes...

Hi BIA team,
we have several questions regarding some error situations which we get when we execute many parallel queries against the BIA.
On BI side we get this 2 errors:
DBMAN 305 Error reading the data of InfoProvider YREP_BC_1$X YREP_BC_1$X
RSD_TREX 100 Error executing physical plan: general error (no f 2999
On BIA side in the TrexRFCServerAlert trace at the same time we found this type of errors:
1. TREXRfcSvr   trexAggregate.cpp(01016) : TREX_RFC_AGGREGATE_SP5 rc: 2999 (Error executing physical plan: general error (no further information available);BwPopBuildIndexUnion pop29(setAliasIndex('pcs_yrep_bc_1-union_of_pcs_bi0_0805006369_pcs_bi0_0805006377en'),addIndexForUnion('pcs_bi0:0805006369en'),addIndexForUnion('pcs_bi0:0805006377en')),in executor::Executor in cube: pcs_yrep_bc_1).
2. TREXRfcSvr   trexAggregate.cpp(01016) : TREX_RFC_AGGREGATE_SP5 rc: 2007 (Error creating physical plan: Index does not exist;index=pcs_bi0:0805095572 in cube: pcs_yrep_bc_1).
We have observed that this types of errors are present if the BIA has the overloaded warning and unloading temp. indexes...
Now the Questions:
1. Why is BIA unloading indexes which are needed for the current query execution, why not deleting only old ones ?
2. After 10000 query executions we have about 22000 temp indexes and in the Reorg tab there is a warning "Standard Deviation of Join/OLAP Indexes 10147 MB, 1625 MB, 84 %, reorg yes", should we schedule a daily job for the cleanup ?
5. What exactly does "Standard Deviation of Join/OLAP Indexes" in BIA world mean ?
3. Were are the temp indexes stored, Shared Memory ?
4. Currently we have phys. 284 GB Ram and 95 MB shared memory assigned, what would be the ideal value for Shared Memory ?
Thanks a lot in advance!
Roman Doubrava

I believe you can dismiss the reorg message, it should take care of itself.
Make sure your BW statistics are running regularly, lack of it may contribute to this.
Looks like you need more memory ( by deleting data or adding blades). Before than I would do a reorg and re-start BWA to clean up temp indexes -there is one memory for data and temp indexes -.
I would check if you have APD queries and definitely exclude them from BWA.
If none works, I would check the statistics for the last 90 days to see what cubes and what queries went to BWA and how many times. Based on the facts that you can either eliminate the least used cubes, based on the nr of recordes being fetched you can exclude certain queries from BWA maybe schedule them overnight, you can exclude BATCH/Information Broadcasting queries, user aleremote, etc
Cheers
Tansu

Are BIA indexes transportable?

We are planning on implementing BIA for our BI system and are in the initial investigation stage. I wanted to know if BIA indexes created in Dev can be transported to remaining systems (QA and Prod), or do we have to create them again in each system (like agregates). What's the correct strategy.
Thanks

Hi Smitha,
The correct strategy would be to recreate the indexes again from scratch on your PROD system. Practically speaking, a BIA index is considered an aggregate on BW side and so, it needs more or less the same handling. Let me give you some more explanations:
From a technical point of view, it would be possible to exchange indexes between BIA installations by exporting/importing them. However, indexes copied this way would not be usable from the BW system as this BW system will neither have nor get the necessary information about the existence of those indexes! That means, the indexes would not be recognized as such. I cannot tell which steps would also have to be taken in order to manipulate the BW system to bypass the actual indexing process. Furthermore, there would also be the problem of the different namespaces under which the indexes are kept on BIA, even though it is a minor issue. Usually, you will use a distinct sid on the QA, DEV, and PROD system, e.g., BWQ, BWD, BWP.
Seen apart from the mentioned problems, can you really make sure that the data on the DEV BW system database is exactly the same as on the PROD BW system? If not you would run into severe inconsistencies anyway.
So, the question is if all the effort to be spent (exporting indexes on DEV BIA, importing them on PROD, and manipulating the BW PROD system in order to know about the existence of the indexes) and the danger of running into new unforeseen problems is really less than just reindexing all the cubes on PROD? How much time will the indexing process take? Roughly speaking, BIA can index about 200mio. records (with 15 to 20 attributes) per hour. With certain parameter adaptations on BW, it is also possible to increase this number.
Best regards,
Sascha.

Impact of logical partitioning on BIA

We are on BI release 701, SP 5. We are planning to create logical partition for some of the infocubes in our system. These cubes are already BIA enabled, so will creation of logical indexes have any impact on the BIA or improve the BIA rollup runtime

Hi Leonel,
Logical partitioning will have impact on BIA in terms of performance .
Current cube is already indexed on BIA. Now if you divide current cube data into different cubes and create multiprovider on it then each cube will have its own F table index on BIA.
you have to put new cubes to BIA and Execute Initial filling step on BIA and place rollups for respective cubes.
Point to be noted :
As Data will be deleted from Current cube and move to other cubes , Indexes will not get deleted
from corresponding F table indexes in BIA . There will be indexes for records which are not present in cube. Thus it is always a good practice to flush BIA which will remove all the indexes from BIA for current cube and create new Indexes on BIA.
In this case , we will have consistent indexes on BIA which will not hamper performance.
This will also improve rollup time as data will be less in each cube after logical partitioning. For rollup
improvement time , we can implement Delta indexing on BIA as well.
Question : Why do we want to create logical partitioning for cubes which are present on BIA as queries will never come to Cubes in BI system ?
Regards,
Kishanlal Kumawat.

BIA Delta Index Merge

Similar Messages

Maybe you are looking for