Performance problems encountered in CIF Post processing
Hello
We have an SCM 7.0 / ERP 6.06 system and had activated CIF Post processing.
However, due to very slow system performance, we had to deactivate CPP.
Subsequently, we have now decided to re-activate it.
I found one note 802980 for performance improvement.
We are also considering scheduling //cppr regularly to delete obsolete and processed CIFERRLOG records.
Apart from this are there any other best practices/checks that we can use before we go ahead with the re-activation?
Regards
Anupama
Hi Anupama,
Take a look of this note:
1095049 - CPP -> Performance during the data transfer from R/3 to
APO
Basically the best thing you can do is to Reorganize CIF PostProcessing in APO and R/3. Run /SAPAPO/CIF_POSTPROC_REORG and delete the obsolete and also the old records. Do this regularly.
Also try to send the error logs to the users and make a process to reduce the number of errors by correcting the cause of the problems.
Kind Regards,
Mariano
Similar Messages
-
In CIF post processing notable to see target system
Hi Gurus,
I am not able to see target system in CIF post processing.Can any one explain me where did i miss my settings? Please let me know it will be really great help
Thanks & Regards,
KumarHi,
Check in /SAPAPO/C2 whether Post Processing is active or not.
For CIF error handling to be active ,setting to be defined under error handling coloumn is
Post Processing, No splitting of LUW .
If Strict is defined CIF error handling is not active.( Standard setting )
This setting applies for the data transfer for APO to R3 and R3 to APO.
Thanks,
nandha -
Dear All,
In the CIF post processing, either via CIF Cockpit or via other post processing transactions, in APO or ECC, we need to go into each post processing record to find the reason for the post processing record being created and to check the status for Error / Warning.
Is there a way to view all the post processing records at APO or ECC system level?
Is there any method to analyze the reason for the post processing records, other than going into each one of them to find out?
I saw in the web of the existence of a program, where on adding some simple coding to the program, a report can be created to view the 'reason for post processing record' being created.
Regards,
Sridhar RHi Senthil,
Thanks for the reply.
The alerts system is already in place.
We are looking at a report where on execution, it would give the reason for the post processing record / error creation.
Based on the report corrective action could be taken, by appropriately notifying the respective teams / users in corresponding plants / regions.
Regards,
Sridhar R -
Hi Experts,
We have implemented CIF Post procesing functionality ( Standard), we are into SCM 5.1.
I am trying to check log in transaction /sapapo/cpp1 issue is :-
1) When I select log for one record (before processing ) It says u201CNo log was found for these selection criteriau201D
also observed that
2)After re-processing record It appears as error queue but it is just saying u201C1 Post processing recordu201D posted and processing sign will be red in color but no where is showing the exact error.
Can you please suggest what problem could be that I am not able to see exact log with error.
Thanks in advace.
SamirHi,
Please check in /SAPAPO/C2 have uou assined the both Logical sistems.
Here you have to give
1) R/3 Logical system, R/3 flag X, R/3 release (ex: 50 for ECC-5.0) Based on the size Inbound or Out bound Queue error Handling 1 Postprocessing for Errors, No Splitting of LUWs and role not specified.
2) APO logicasystem R/3 flag blank, APO release (ex: 41 for SCM-4.1) Based on the size Inbound or Out bound Queue error Handling 1 Postprocessing for Errors, No Splitting of LUWs and role not specified.
For selecting Inbound or Out bound Please read the bellow.
When you enter a logical target system in transaction /SAPAPO/C2, the system suggests the queue type Outbound Queues as standard. This setting means that queue processing is controlled by the sending system (SAP APO). Problems can occur if the receiving system (SAP R/3) does not have enough free work processes for processing the queue.
If you are using large amounts of integration data and expect that you may have insufficient free work processes, it may be useful to change to Inbound Queues. This setting means that queue processing is controlled by the receiving system (SAP R/3). If there are insufficient work processes available in the receiving system, the data is temporarily stored in SAP R/3 inbound and then processed gradually.
But we are using Inbound.
Hope this will help you.
Regards,
Kishore Reddy -
Problem caling ConcRequest with Post Processing from code using fnd_request
Gurus,
Background:
I am working on an XML Publihser report using Reports6i. I have defined a Report in Applications. This report generates XML data and as part of post processing I specify the template to pick under "Upon Completion" Section. The report works fine when I submit it from SRS window.
(This is possible because with XML Publisher installed, we can specify a template for any RDF report and its processed as part of Post processing)
Now I have to invoke the same report programmatically. I am using
fnd_request.submit_request (application => 'PO',
program => 'XXTPOREP',
description => '',
start_time => '',
sub_request => FALSE,
argument1 => 'R',
argument2 => CHR (0),
argument3 => CHR (0),
argument4 => CHR (0),
argument5 => CHR (0),
My query is how do I specify the parameters for post processing the request to ensure that the output is generated with the template .
Thanks in Advancebefore submitting the request add this line
v_request_status := Fnd_Request.add_layout('PO','XXTPOREP','en','US','PDF');
This is with the assumption that you need a pdf document as the output and the template language and territory is English and United States
Message was edited by:
user585821 on 19 July 2007 -
Performance problem while CPU is 80% Idel ?
Hi,
My end users are claim for performance problem during execution of batch process.
As you can see there are 1,745 statement executing each second.
Awr report shows 98.1% of the time , waits on CPU .
Also Awr report shows that Host CPU is :79.9% Idel.
The second wait event shows only 212 seconds waits on db file sequential read.
Yet , 4 minute in 1 hour period is seems not an issue.
Please advise
DB Name DB Id Instance Inst Num Startup Time Release RAC
QERP xxx erp 1 21-Jan-13 15:40 11.2.0.2.0 ; NO
Host Name Platform CPUs Cores Sockets Memory(GB)
erptst HP-UX IA (64-bit) 16 16 4 127.83
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 40066 22-Jan-13 20:00:52 207 9.6
End Snap: 40067 22-Jan-13 21:00:05 210 9.6
Elapsed: 59.21 (mins)
DB Time: 189.24 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 8,800M 8,800M Std Block Size: 8K
Shared Pool Size: 1,056M 1,056M Log Buffer: 49,344K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 3.2 ; 0.1 ; 0.00 ; 0.05
DB CPU(s): 3.1 ; 0.1 ; 0.00 ; 0.05
Redo size: 604,285.1 ; 27,271.3
Logical reads: 364,792.3 ; 16,463.0
Block changes: 3,629.5 ; 163.8
Physical reads: 21.5 ; 1.0
Physical writes: 95.3 ; 4.3
User calls: 68.7 ; 3.1
Parses: 212.9 ; 9.6
Hard parses: 0.3 ; 0.0
W/A MB processed: 1.2 ; 0.1
Logons: 0.3 ; 0.0
Executes: 1,745.2 ; 78.8
Rollbacks: 1.2 ; 0.1
Transactions: 22.2
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 ; Redo NoWait %: 100.00
Buffer Hit %: 99.99 ; In-memory Sort %: 100.00
Library Hit %: 99.95 ; Soft Parse %: 99.85
Execute to Parse %: 87.80 ; Latch Hit %: 99.99
Parse CPU to Parse Elapsd %: 74.76 ; % Non-Parse CPU: 99.89
Shared Pool Statistics Begin End
Memory Usage %: 75.37 ; 76.85
% SQL with executions>1: 95.31 ; 85.98
% Memory for SQL w/exec>1: 90.33 ; 82.84
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
DB CPU 11,144 98.1
db file sequential read 52,714 214 4 1.9 User I/O
SQL*Net break/reset to client 29,050 6 0 .1 Applicatio
log file sync 2,536 6 2 .0 Commit
buffer busy waits 4,338 2 1 .0 Concurrenc
Host CPU (CPUs: 16 Cores: 16 Sockets: 4)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
0.34 ; 0.33 ; 19.7 ; 0.4 ; 1.8 ; 79.9Nikolay Savvinov wrote:
if the users are complaining about performance of the batch process, then that's what you should be looking at, not the entire system.I find it strange to see "end users" and "the batch process" in the same sentence (as it was in the first post). "End users" gives me the feeling of a significant number of concurrent sessions with people waiting for results in real time at the far end, while "batch process" carries the image a small number of large scale processes running overnight to prepare the data for the following morning.
I mention this because my first view of the AWR output was: you've got 16 CPUs, only three in use, virtually no users, and doing very little work, how can the users complain. (One answer, of course, is that the 13 CPUs could be locked out of use as far as Oracle is concerned). On the second read I decided that the "users" had gone home, and the complaint was simply that the batch process wasn't completing in time.
In this case I think "the entire system" IS "the batch process"
Determine which stored procedures and/or SQL statements took longer than usual and then find out why. Most likely you'll be able to find
everything you need in AWR views (DBA_HIST_SQL%) and ASH archive (DBA_HIST_ACTIVE_SESS_HISTORY).
If the batch process has changed dramatically and recently, then a simple first step might be to look at the current AWR report, find the few most time-consuming SQL statements, and use the awrsqrpt.sql script to find their history of execution plans.
But I'd also just look at the expensive SQL - bearing in mind, particularly, that there are very few user calls per second, yet many hundred executions per second: it strikes me that there could be quite a lot of PL/SQL going on doing something a little bit expensive many times or some PL/SQL function that calls some SQL that used to be called rarely from an SQL statement but is now (due, perhaps to a change in plan) being called much more frequently - so check SQL Ordered by Executions.
Regards
Jonathan Lewis -
How can i retrieve purchased audiobooks that were not synced (problem encountered during sync process)? My account shows the purchase history, but the apps are no longer displayed in itunes or on my ipad or iphone. I'm not very savvy with using iTunes - I don't think is very intuitive (unlike me:)). Any help would be appreciated.
ThanksHello mcegirl4,
Thanks for using Apple Support Communities.
Please refer to the instructions in the following article to assist in downloading those previous purchases from iTunes and the App Store:
Download past purchases
http://support.apple.com/kb/HT2519
Take care,
Alex H. -
LR3 "Extra Processing in Develop" Performance Problem
I have been investigating a specific LR3 performance problem. It may explain a small subset of the problems people have reported in the "Why is LR3 So Slow?" thread. I'm starting this thread to focus on this particular problem. I hope others will confirm/refute/refine my findings.
The Problem
In Develop, when I make an adjustment, normally the following happens: The CPU usage (as shown in Activity Monitor's bar graph) jumps to between 50 and 75% for all four cores, the updated image appears, and the CPU usage settles back down. This all happens in less than half a second. Note: this is with the image at the Fit size. However, sometimes I instead get the following after an adjustment: the CPU usage jumps to 50 to 75% for all four cores and the updated image appears as usual, however, instead of settling back down, the CPU usage jumps up to 90 to 100% for all cores and stays there for 3 to 5 seconds before settling down. Thus it appears that LR is doing some kind of "extra processing" since a lot of computation is happening AFTER the updated image has already appeared. I will refer to this problem as "EP". Obviously, when you are getting EP, editing in Develop becomes very balky.
Dependency on ratio between image size and displayed size
It appears that EP only happens when the displayed size of the image (in Fit zoom level and perhaps also Fill zoom level) is above a certain percentage of the actual image size (as currently cropped). Evidence: When editing full 21MP 5D2 images, I don't experience EP. If I crop the 5D2 image fairly significantly, then I can get EP. When editing 10MP images from my Canon S90, I usually get EP for landscape orientation pictures but not for portrait orientation pictures (since in Fit mode, landscape images display at a higher zoom level than portrait images). If I am getting EP, I can eliminate it by sufficiently reducing the size that LR is displaying the image by resizing the LR window smaller, opening additional panels (I normally edit with only the right panel open), displaying the toolbar, etc. It appears that EP is enabled when the displayed image is about 50% or larger w.r.t. the actual image (as currently cropped). For example, EP becomes enabled when a 3648 pixel wide S90 image is displayed at least 17 and 7/8 inches wide on my 100 ppi monitor (i.e. about 1787 pixels).
Dependency on HOW an adjustment is invoked
Even when the displayed image size is large enough w.r.t. the actual image size to enable EP, whether you get it on a given adjustment depends on how you invoke it:
- If you CLICK (i.e. press the mouse button down and quickly release it) on the track of one of the sliders (a technique I use often to make big jumps), EP will happen.
- If you press the mouse button down on a slider handle, drag it to a new position, and quickly release the mouse button), EP will happen
- If you press the mouse button down on a slider handle, drag it to a new position, but continue to hold the mouse button down until the displayed image is updated, EP does NOT happen (either before or after you then release the mouse button).
- If you highlight the numeric field at the end of a slider and use the arrow keys (possibly along with Shift) to increment or decrement the value, EP does NOT happen.
- EP will happen if you resize the LR window such that the displayed image size is above the threshold. (In fact, I determined the threshold by making a series of window width increases until I saw EP indicated by the CPU bar graphs.)
- EP can happen with local adjustment brush applications, but as with the sliders, it depends on HOW you perform the brush stroke. Single click and drags with immediate mouse release cause EP, drags with delayed mouse button release don't.
- Clicking an earlier History state causes EP
- More exploration could be done. For example, I haven't looked at Graduated Filter and Spot Removal adjustments.
My theory of what's happening
With LR2, my understanding is that in Develop mode when the displayed image is below 1:1 zoom level, after an adjustment is invoked, LR calculates the new version of the image to display using a fast, simplified algorithm that doesn't include the more computationally intensive adjustments like Sharpening and Noise Reduction (and perhaps works on a lower rez version of the image with multiple sensels binned together?). It appears that in conditions described above, LR3 calculates the initial, fast image update and then goes on to do the full update of the image, including the computationally intensive adjustments. Evidence: setting Sharpening Amount and Luminance and Color Noise Reduction to zero eliminates EP (or reduces the amount of time it takes to be barely noticeable). I'm not sure whether the displayed image is updated with the results of the extra processing. I think the answer is Yes since when I tried an adjustment of changing Sharpening Amount from 0 to 90, the initial update of the displayed image showed sharpening but after the EP, the displayed image was updated again to show somewhat different sharpening. Perhaps Adobe felt that it would be useful to see the more accurate version of the image when it is at or above 50% zoom. Maybe the UI is supposed to cancel the EP if you start to make another adjustment before it has completed but the canceling doesn't happen unless you invoke the adjustment in one of the ways described above that doesn't cause EP.
Misc
- EP doesn't seem to happen for Process 2003
- As others have mentioned, I'm surprised that LR (both version 2 and 3) in 64bit mode doesn't use more available RAM. I don't think I've seen LR go above 4GB of virtual memory or above 3GB "Real Memory" (as reported by Activity Monitor) even though I have several GB free.
- It should be obvious from the above that if you experience EP, there are workarounds: reduce the size of the displayed image (e.g. by window resizing), invoke adjustments in ways that don't cause EP, turn off Sharpening and Noise Reduction until the end of editing an image.
System specs
First generation Intel Mac Pro with two dual-core CPUs at 2.66 Ghz
OS 10.5.8
21GB RAM
ACR cache on volume striped across 3 internal SATA drives
LR catalog and RAWs on an internal SATA drive
30" HP LP3065 monitor (2560 pixels wide)
NVIDIA GeForce 7300 GTI'm impressed by your thorough analysis.
Clearly, the programmers haven't figured out the best way to do intelligent caching and/or parallel rendering at a reduced size yet.
In my experience reducing the settings in the "Details" panel doesn't help.
What really bugs me is that the lag (or increasing lack of interactiveness) depends on the number of adjustments one has made.
This shouldn't be the case. If a cache is produced then every further adjustment should only cost the effort for that latest adjustment and not include adjustments before it. There are things that stand in the way of straightforward edit applications:
If you work below 1:1 preview, adjustments have to be shown in a reduced form. If you don't have a way to faithfully mimic the adjustments on the reduced size, you have to do them on the original image and then scale down. That's expensive.
To the best of my knowledge LR uses a fixed image pipeline. Hence, independently of the order in which you apply edits, they are always performed in the same fixed order. Say all spot removal operations are done first. If you have a lot of adjustment brush edits and then add a spot removal operation, it means that all the adjustment brush operations have to be replayed each time you do a little adjustment on your spot removal edit.
I believe what you are seeing is mostly related to 1.
I also believe that the way LR currently handles a moderate number of edits is unacceptable and incompatible with the notion that it is usable in a commercial setting for more than trivial edits. I suspect there is something else going on. If everyone saw the deterioration in performance after a number of edits that I see, I don't think LR would be as accepted as it is. Having said that, I've read that the problem of repeated applications of the adjustment brush slowing LR down has existed for a long time. I truly hope that this doesn't mean we'll have to live for it for the foreseeable future.
There are two ways I can see how 2. should be addressed:
combine the effects of a set of operations into one bitmap operation. Instead of replaying all adjustment brush strokes one after the other (speedwise it feels like this is happening), compute a single bitmap operation that combines all effects.
give up the idea that there is an image pipeline with a fixed execution order.
Some might argue that the second point is at odds with the whole idea of parametric editing, but I dispute that. Either edit operations are commutable in which case the order is immaterial, or they are not. If they are not, the user applies the edits in a way as he/she sees fit and will thus compensate for any effect of a changed ordering.
N.B., currently the doctrine of "fixed ordering of edit applications" results in the effect that even if you convert an image into B&W all your adjustment brush edits that applied colour tints will still show through. Reasoning: The user should be able to locally tint a B&W image. I agree with the latter but this could be achieved by only applying those tinting brush strokes that were created after the B&W conversion. All the ones that happened before should only be used to obtain the correct luminance values for the B&W conversion but obviously they shouldn't cause tinted areas.
The above example demonstrates to me that users naturally expect operations to occurr in the order they have been introduced, not in a fixed predefined order. If that principle were followed, I see no reason why the speed of a single edit should depend on the number of edits that were done to the image before.
I hope the programmers can (and the management wants to) address the performance issues. While I find LR usable for pretty modest edits, in no way the performance on my system approaches that would I would expect from an industrial strenght application.
P.S.: Your message reminded me of the following: When I experience serious lag with LR showing the strokes I make with an adjustment brush, it helps to pause a moment after the first click before one starts moving. This allows LR to catch up and then one can see the effect of the application pretty much interactively. Otherwise, there is terrible lag and the feedback where you have brushed an effect comes way too late. -
Hi,
We have a reqmnt to update an document on R/3, source - APO, with a parameter passed in the CIF. The SAP BAPI is not populating the field, so the fallback option is for us to update the doc manually.
Is anyone aware of any post processing exits, provided in the CIF, which we make use of.
TIA
Regards,
PramodHi,
Check in /SAPAPO/C2 whether Post Processing is active or not.
For CIF error handling to be active ,setting to be defined under error handling coloumn is
Post Processing, No splitting of LUW .
If Strict is defined CIF error handling is not active.( Standard setting )
This setting applies for the data transfer for APO to R3 and R3 to APO.
Thanks,
nandha -
Hi all,
I am facing an issue where i have done booking through MFBF and some parts got stored for post processing, now the problem is that when i am doing document specific reversal, parts which got stored for postprocessing is not getting reversed while other parts got cleared. Kindly help
Prashant.Pillai
SAP PP ConsultantHi,
There are 2 ways to solve the problem.
1. Clear the backflush error using MF47 and reverse both the docs.
2. Do the material document reversal and delete the post processing error in MF47 because the error there is MF47 is actually not posted and is under reserve for that Header material. This option is only good for testing but not in actual production scenario.
Regards,
Gaurav Mehra -
Red saturation problems and post-processing
I see problems with the color red saturating out quickly. This often causes detail to be lost. I have pictures of a red bird where the jpeg produced by the camera is usually better, showing more detail, than anything I can produce from the raw file. I have tried limiting saturation using the hue/saturation tool (PSE12) but that gives unacceptable results as the color gets lost when you reduce saturation.
My best workaround so far is to start with the jpeg file, open it as a raw file, and do post-processing that way. That has given pictures with more detail than starting with the original raw file. Is there anything else I can try? I find it hard to believe the camera processor does better than PSE12 in these cases.I think I'm missing something. Now I see the code and DC's inside the DTR. However, when I try to import into NWDS no DC's come in (the software components exist, but no DC's in them). Additionally, the CBS web ui reports that my software components do not contain any DC's even though I see them in the DTR. What things can I look at to determine what I'm missing here?
Thought I'd add some more info..after applying the support packs, we imported the new SAPBUILD, SAP_JTECH, and SAP_jee SCA'S into our track as we required some functionality from the newer build SCA. We also reimported our old archives back into the system by manually checking them in assuming this would fix the problem with us not seeing the source in the NWDS or the DTR. After the import, the CBS no longer sees our custom DC's, but the DTR does (both in active and inactive ws). When importing the dev configuration into the NWDS our custom DC's no longer appear, but SAP's standard SCA's do.
Message was edited by:
Eric Vota -
Performance problems with DFSN, ABE and SMB
Hello,
We have identified a problem with DFS-Namespace (DFSN), Access Based Enumeration (ABE) and SMB File Service.
Currently we have two Windows Server 2008 R2 servers providing the domain-based DFSN in functional level Windows Server 2008 R2 with activated ABE.
The DFSN servers have the most current hotfixes for DFSN and SMB installed, according to http://support.microsoft.com/kb/968429/en-us and http://support.microsoft.com/kb/2473205/en-us
We have only one AD-site and don't use DFS-Replication.
Servers have 2 Intel X5550 4 Core CPUs and 32 GB Ram.
Network is a LAN.
Our DFSN looks like this:
\\contoso.com\home
Contains 10.000 Links
Drive mapping on clients to subfolder \\contoso.com\home\username
\\contoso.com\group
Contains 2500 Links
Drive mapping on clients directly to \\contoso.com\group
On \\contoso.com\group we serve different folders for teams, projects and other groups with different access permissions based on AD groups.
We have to use ABE, so that users see only accessible Links (folders)
We encounter sometimes multiple times a day enterprise-wide performance problems for 30 seconds when accessing our Namespaces.
After six weeks of researching and analyzing we were able to identify the exact problem.
Administrators create a new DFS-Link in our Namespace \\contoso.com\group with correct permissions using the following command line:
dfsutil.exe link \\contoso.com\group\project123 \\fileserver1\share\project123
dfsutil.exe property sd grant \\contoso.com\group\project123 CONTOSO\group-project123:RX protect replace
This is done a few times a day.
There is no possibility to create the folder and set the permissions in one step.
DFSN process on our DFSN-servers create the new link and the corresponding folder in C:\DFSRoots.
At this time, we have for example 2000+ clients having an active session to the root of the namespace \\contoso.com\group.
Active session means a Windows Explorer opened to the mapped drive or to any subfolder.
The file server process (Lanmanserver) sends a change notification (SMB-Protocol) to each client with an active session \\contoso.com\group.
All the clients which were getting the notification now start to refresh the folder listing of \\contoso.com\group
This was identified by an network trace on our DFSN-servers and different clients.
Due to ABE the servers have to compute the folder listing for each request.
DFS-Service on the servers doen't respond for propably 30 seconds to any additional requests. CPU usage increases significantly over this period and went back to normal afterwards. On our hardware from about 5% to 50%.
Users can't access all DFS-Namespaces during this time and applications using data from DFS-Namespace stop responding.
Side effect: Windows reports on clients a slow-link detection for \\contoso.com\home, which can be offline available for users (described here for WAN-connections: http://blogs.technet.com/b/askds/archive/2011/12/14/slow-link-with-windows-7-and-dfs-namespaces.aspx)
Problem doesn't occure when creating a link in \\contoso.com\home, because users have only a mapping to subfolders.
Currently, the problem doesn't occure also for \\contoso.com\app, because users usually don't use Windows Explorer accessing this mapping.
Disabling ABE reduces the DFSN freeze time, but doesn't solve the problem.
Problem also occurs with Windows Server 2012 R2 as DFSN-server.
There is a registry key available for clients to avoid the reponse to the change notification (NoRemoteChangeNotify, see http://support.microsoft.com/kb/812669/en-us)
This might fix the problem with DFSN, but results in other problems for the users. For example, they have to press F5 for refreshing every remote directory on change.
Is there a possibility to disable the SMB change notification on server side ?
TIA and regards,
Ralf GaudesHi,
Thanks for posting in Microsoft Technet Forums.
I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.
Thank you for your understanding and support.
Regards.
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place. -
Performance problems with File Adapter and XI freeze
Hi NetWeaver XI geeks,
We are deploying a XI based product and encounter some huge performance problems. Here after the scenario and the issues:
- NetWeaver XI 2004
- SAP 4.6c
- Outbound Channel
- No mapping used and only the iDocs Adapter is involved in the pipeline processing
- File Adapter
- message file size < 2Ko
We have zeroed down the problem to Idoc adapters performance.
We are using a file channel and every 15 seconds a file in a valid Idoc format is placed in a folder, Idoc adapter picks up the file from this folder and sends it to the SAP R/3 instance.
For few minutes (approx 5 mins) it works (the CPU usage is less then 20% even if processing time seems huge : <b>5sec/msg</b>) but after this time the application gets blocked and the CPU gets overloaded at 100% (2 processes disp_worker.exe at 50% each).
If we inject several files in the source folder at the same time or if we decrease the time gap (from 15 seconds to 10 seconds) between creation of 2 Idoc files , the process blocks after posting 2-3 docs to SAP R/3.
Could you point us some reasons that could provoke that behavior?
Basically looking for some help in improving performance of the Idoc adapter.
Thanks in advance for your help and regards,
AdalbertHi Bhavesh,
Thanks for your suggestions. We will test...
We wonder if the hardware is not the problem of this extremely poor performance.
Our XI server is:
Windows 2003 Server
Processors: 2x3GHZ
RAM: 4GB (the memory do not soak)
The messages are well formed iDocs = single line INVOICES.
Some posts are talking 2000 messages processed in some seconds... whereas we got 5 sec per message.
Tnanks for your help.
Adalbert -
Post-Processing limit on number of files?
Hello,
When I export a set of pictures, I apply a post-processing Photoshop droplet to that applies the Imagenomic Portraiture filter. It works fine up to a certain point. I noticed that if the number of files is less than 21, it works fine. If the number is greater, it fails to perform the post-processing and won't even launch Photoshop to apply the droplet action.
Has anyone encountered a file limit problem with post-processing in export?
DelI'll assume you are talking about Windows and not Mac since I cannot speak about Mac's. In Windows, the export action is invoked by passing the filenames on a command line to the program/script that you specify. There are limits to how many characters can be passed on a command line, depending on which method the programmers used. I expect the limit is 2048 characters. When that limit is reached, then the action does nothing.
The only way to increase the number of photos you can process, is to change the export folder name to something very short, like C:\Temp, or even shorter. By default, it exports to a very long path name an putting that in front of each photo name eats up valuable characters.
There will still be a limit, but at least you can process more before failure.
Steve -
Performance problems with XMLTABLE and XMLQUERY involving relational data
Hello-
Is anyone out there using XMLTABLE or XMLQUERY with more than a toy set of data? I am running into serious performance problems tyring to do basic things such as:
* Combine records in 10 relational tables into a single table of XMLTYPE records using XMLTABLE. This hangs indefinitely for any more than 800 records. Oracle has confirmed that this is a problem and is working on a fix.
* Combine a single XMLTYPE record with several relational code tables into a single XMLTYPE record using XMLQUERY and ora:view() to insert code descriptions after each code. Performance is 10 seconds for 10 records (terrible) passing a batch of records , or 160 seconds for one record (unacceptable!). How can it take 10 times longer to process 1/10th the number of records? Ironically, the query plan says it will do a full table scan of records for the batch, but an index access for the one record passed to the XMLQUERY.
I am rapidly losing faith in XML DB, and desparately need some hints on how to work around these performance problems, or at least some assurance that others have been able to get this thing to perform.<Note>Long post, sorry.</Note>
First, thanks for the responses above. I'm impressed with the quality of thought put into them. (Do the forum rules allow me to offer rewards? :) One suggestion in particular made a big performance improvement, and I’m encouraged to hear of good performance in pure XML situations. Unfortunately, I think there is a real performance challenge in two use cases that are pertinent to the XML+relational subject of this post and probably increasingly common as XML DB usage increases:
• Converting legacy tabular data into XML records; and
• Performing code table lookups for coded values in XML records.
There are three things I want to accomplish with this post:
• Clarify what we are trying to accomplish, which might expose completely different approaches than I have tried
• Let you know what I tried so far and the rationale for my approach to help expose flaws in my thinking and share what I have learned
• Highlight remaining performance issues in hopes that we can solve them
What we are trying to accomplish:
• Receive a monthly feed of 10,000 XML records (batched together in text files), each containing information about an employee, including elements that repeat for every year of service. We may need to process an annual feed of 1,000,000 XML records in the future.
• Receive a one-time feed of 500,000 employee records stored in about 10 relational tables, with a maximum join depth of 2 or 3. This is inherently a relational-to-XML process. One record/second is minimally acceptable, but 10 records/sec would be better.
• Consolidate a few records (from different providers) for each employee into a single record. Given the data volume, we need to achieve a minimum rate of 10 records per second. This may be an XML-only process, or XML+relational if code lookups are done during consolidation.
• Allow the records to be viewed and edited, with codes resolved into user-friendly descriptions. Since a user is sitting there, code lookups done when a record is viewed (vs. during consolidation) should not take more than 3 seconds total. We have about 20 code tables averaging a few hundred rows each, though one has 450,000 rows.
As requested earlier, I have included code at the end of this post for example tables and queries that accurately (but simply) replicate our real system.
Why we did and why:
• Stored the source XML records as CLOBS: We did this to preserve the records exactly as they were certified and sent from providers. In addition, we always access the entire XML record as a whole (e.g., when viewing a record or consolidating employee records), so this storage model seemed like a good fit. We can copy them into another format if necessary.
• Stored the consolidated XML employee records as “binary XML”. We did this because we almost always access a single, entire record as a whole (for view/edit), but might want to create some summary statistics at some point. Binary XML seemed the best fit.
• Used ora:view() for both tabular source records and lookup tables. We are not aware of any alternatives at this time. If it made sense, most code tables could be pre-converted into XML documents, but this seemed risky from a performance standpoint because the lookups use both code and date range constraints (the meaning of codes changes over time).
• Stored records as XMLTYPE columns in a table with other key columns on the table, plus an XMLTYPE metadata column. We thought this would facilitate pulling a single record (or a few records for a given employee) quickly. We knew this might be unnecessary given XML indexes and virtual columns, but were not experienced with those and wanted the comfort of traditional keys. We did not used XMLTYPE tables or the XML Repository for documents.
• Used XMLTABLE to consolidate XML records by looping over each distinct employee ID in the source batch. We also tried XMLQUERY and it seems to perform about the same. We can achieve 10 to 20 records/second if we do not do any code lookups during consolidation, just meeting our performance requirement, but still much slower than expected.
• Used PL/SQL with XMLFOREST to convert tabular source records to XML by looping over distinct employee IDs. We tried this outside PL/SQL both with XMLFOREST and XMLTABLE+ora:view(), but it hangs in both cases for more than 800 records (a known/open issue). We were able to get it to work by using an explicit cursor to loop over distinct employee IDs (rather than processing all records at once within the query). The performance is one record/second, which is minimally acceptable and interferes with other database activity.
• Used XMLQUERY plus ora:view() plus XPATH constraints to perform code lookups. When passing a single employee record, the response time ranges from 1 sec to 160 sec depending on the length of the record (i.e., number of years of service). We achieved a 5-fold speedup using an XMLINDEX (thank you Marco!!). The result may be minimally acceptable, but I’m baffled why the index would be needed when processing a single XML record. Other things we tried: joining code tables in the FOR...WHERE clauses, joining code tables using LET with XPATH constraints and LET with WHERE clause constraints, and looking up codes individually via JDBC from the application code at presentation time. All those approaches were slower. Note: the difference I mentioned above in equality/inequality constraint performance was due to data record variations not query plan variations.
What issues remain?
We have a minimally acceptable solution from a performance standpoint with one very awkward PL/SQL workaround. The performance of a mixed XML+relational data query is still marginal IMHO, until we properly utilize available optimizations, fix known problems, and perhaps get some new query optimizations. On the last point, I think the query plan for tabular lookups of codes in XML records is falling short right now. I’m reminded of data warehousing in the days before hash joins and star join optimization. I would be happy to be wrong, and just as happy for viable workarounds if I am right!
Here are the details on our code lookup challenge. Additional suggestions would be greatly appreciated. I’ll try to post more detail on the legacy table conversion challenge later.
-- The main record table:
create table RECORDS (
SSN varchar2(20),
XMLREC sys.xmltype
xmltype column XMLREC store as binary xml;
create index records_ssn on records(ssn);
-- A dozen code tables represented by one like this:
create table CODES (
CODE varchar2(4),
DESCRIPTION varchar2(500)
create index codes_code on codes(code);
-- Some XML records with coded values (the real records are much more complex of course):
-- I think this took about a minute or two
DECLARE
ssn varchar2(20);
xmlrec xmltype;
i integer;
BEGIN
xmlrec := xmltype('<?xml version="1.0"?>
<Root>
<Id>123456789</Id>
<Element>
<Subelement1><Code>11</Code></Subelement1>
<Subelement2><Code>21</Code></Subelement2>
<Subelement3><Code>31</Code></Subelement3>
</Element>
<Element>
<Subelement1><Code>11</Code></Subelement1>
<Subelement2><Code>21</Code></Subelement2>
<Subelement3><Code>31</Code></Subelement3>
</Element>
<Element>
<Subelement1><Code>11</Code></Subelement1>
<Subelement2><Code>21</Code></Subelement2>
<Subelement3><Code>31</Code></Subelement3>
</Element>
</Root>
for i IN 1..100000 loop
insert into records(ssn, xmlrec) values (i, xmlrec);
end loop;
commit;
END;
-- Some code data like this (ignoring date ranges on codes):
DECLARE
description varchar2(100);
i integer;
BEGIN
description := 'This is the code description ';
for i IN 1..3000 loop
insert into codes(code, description) values (to_char(i), description);
end loop;
commit;
end;
-- Retrieve one record while performing code lookups. Takes about 5-6 seconds...pretty slow.
-- Each additional lookup (times 3 repeating elements in the data) adds about 1 second.
-- A typical real record has 5 Elements and 20 Subelements, meaning more than 20 seconds to display the record
-- Note we are accessing a single XML record based on SSN
-- Note also we are reusing the one test code table multiple times for convenience of this test
select xmlquery('
for $r in Root
return
<Root>
<Id>123456789</Id>
{for $e in $r/Element
return
<Element>
<Subelement1>
{$e/Subelement1/Code}
<Description>
{ora:view("disaac","codes")/ROW[CODE=$e/Subelement1/Code]/DESCRIPTION/text() }
</Description>
</Subelement1>
<Subelement2>
{$e/Subelement2/Code}
<Description>
{ora:view("disaac","codes")/ROW[CODE=$e/Subelement2/Code]/DESCRIPTION/text()}
</Description>
</Subelement2>
<Subelement3>
{$e/Subelement3/Code}
<Description>
{ora:view("disaac","codes")/ROW[CODE=$e/Subelement3/Code]/DESCRIPTION/text() }
</Description>
</Subelement3>
</Element>
</Root>
' passing xmlrec returning content)
from records
where ssn = '10000';
The plan shows the nested loop access that slows things down.
By contrast, a functionally-similar SQL query on relational data will use a hash join and perform 10x to 100x faster, even for a single record. There seems to be no way for the optimizer to see the regularity in the XML structure and perform a corresponding optimization in joining the code tables. Not sure if registering a schema would help. Using structured storage probably would. But should that be necessary given we’re working with a single record?
Operation Object
|SELECT STATEMENT ()
| SORT (AGGREGATE)
| NESTED LOOPS (SEMI)
| TABLE ACCESS (FULL) CODES
| XPATH EVALUATION ()
| SORT (AGGREGATE)
| NESTED LOOPS (SEMI)
| TABLE ACCESS (FULL) CODES
| XPATH EVALUATION ()
| SORT (AGGREGATE)
| NESTED LOOPS (SEMI)
| TABLE ACCESS (FULL) CODES
| XPATH EVALUATION ()
| SORT (AGGREGATE)
| XPATH EVALUATION ()
| SORT (AGGREGATE)
| XPATH EVALUATION ()
| TABLE ACCESS (BY INDEX ROWID) RECORDS
| INDEX (RANGE SCAN) RECORDS_SSN
With an xmlindex, the same query above runs in about 1 second, so is about 5x faster (0.2 sec/lookup), which is almost good enough. Is this the answer? Or is there a better way? I’m not sure why the optimizer wants to scan the code tables and index into the (one) XML record, rather than the other way around, but maybe that makes sense if the optimizer wants to use the same general plan as when the WHERE clause constraint is relaxed to multiple records.
-- Add an xmlindex. Takes about 2.5 minutes
create index records_record_xml ON records(xmlrec)
indextype IS xdb.xmlindex;
Operation Object
|SELECT STATEMENT ()
| SORT (GROUP BY)
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (AGGREGATE)
| FILTER ()
| TABLE ACCESS (FULL) CODES
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (GROUP BY)
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (AGGREGATE)
| FILTER ()
| TABLE ACCESS (FULL) CODES
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (GROUP BY)
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (AGGREGATE)
| FILTER ()
| TABLE ACCESS (FULL) CODES
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (AGGREGATE)
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (AGGREGATE)
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| TABLE ACCESS (BY INDEX ROWID) RECORDS
| INDEX (RANGE SCAN) RECORDS_SSN
Am I on the right path, or am I totally using the wrong approach? I thought about using XSLT but was unsure how to reference the code tables.
I’ve done the best I can constraining the main record to a single row passed to the XMLQUERY. Given Mark’s post (thanks!) should I be joining and constraining the code tables in the SQL WHERE clause too? That’s going to make the query much more complicated, but right now we’re more concerned about performance than complexity.
Maybe you are looking for
-
Xorg: blank screen, mouse cursor only - ATI Radeon HD3450
Last friday, my old Nvidia card died (overheated, I think). I got a replacement today: an ATI HD3450 card by Asus (R600 series). This is an old PC with an AGP slot, so my options were unfortunately limited. I was looking for a Nvidia replacement orig
-
Mac Mavericks has messed up my CS4
Since downloading Mavericks, Photoshop CS4 is not working correctly. Specifically, I have lost my Tools bar and many of my most useful Tools, including the main cursor. Anyone know how to fix this? Thanks.
-
AS3: Event.SOUND_COMPLETE event not firing
hey everyone have a look at the attached code. its not very complex but for some reason i can't get the onPlaybackComplete method to fire. any ideas why? SoundClick is an mp3 in the library set to export as an extension of the Sound class. it plays f
-
Hello, I wanted to send an entire URL as parameter to a Stored Procedure. How can I encode/decode this URL? Thanks Madhav
-
How to dinamically set parameters to HTMLCOMMANDBUTTON
Hi, I have the following code: FacesContext context = FacesContext.getCurrentInstance(); HtmlCommandButton button = new HtmlCommandButton(); button.setId("button_action_id"); button.setValue("mybutton"); button.setAction(context.getApplication().crea