Association rules - memory issue

Hi everyone,
I'm testing oracle data mining to generate a high number of association rules, starting from a table containing around 100 columns and 1800 lines. I transformed the table in a view as shown with the market basket demo, but I'm still having a memory error each time I'm trying to launch the data mining process.
I've tried with a table containing less columns and lines, and it worked fine.
Is there a way to set ODM to allow it to process this big table and generate millions of association rules without having some memory errors ?
Is it possible to find how exactly the apriori algorithm works, I mean to find if the sql process is only working with the RAM and writes in the database in the end, or if it commits into the database during the process (before exploding memory).
Merry christmas by the way !!!!!

Hi,
thanks again for your reply. I tried the tweak with the shared_pool_size, and I'm still having some memory issues.
Do you know how the model is build, and how it works technically ?
I can see that itemsets and rules are created when you build the model. Is it stored in a procedure (which applies the apriori algorithm), computes the model and then stores the rules generated into the database during commit ?
In this case all the process (model building) is stored in RAM, am I right ?
And that's why the process crash when there are too much data processed. Is it possible to avoid this, I mean commit the rules even during the process to free some RAM space in order to continue the rule generation ?

Similar Messages

Association rules - Memory problem?

I am running the associtaion rules program in Java and I get
the following error. Any idea what causes this error?
Thank u
K
arBuildTask_jdm is started, please wait. arBuildTask_jdm is failed.
Failure Description: ORA-40101: Data Mining System Error ORA-40101: Data Mining System Error ORA-40101: Data Mining System Error ODM_ASSOCIATION_MODEL-BUILD--20010
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "DMSYS.ODM_ASSOCIATION_RULE_MODEL", line 144
ORA-06512: at "DMSYS.ODM_ASSOCIATION_RULE_MODEL", line 1397
ORA-00600: internal error code, arguments: [15593], [kxhfFind: no partition with a full slot], [], [], [], [], [], []
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "DMSYS.ODM_ASSOCIATION_RULE_MODEL", line 144
ORA-06512: at "DMSYS.ODM_ASSOCIATION_RULE_MODEL", line 1397
ORA-06512: at "DMSYS.DBMS_DATA_MINING", line 312
ORA-06512: at "DMSYS.DBMS_JDM_INTERNAL", line 157
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86
ORA-06512: at "DMSYS.DBMS_JDM_INTERNAL", line 179
ORA-06512: at line 1
--

This is a known bug that has been fixed in the upcoming database release (11g).

Association Rules problem

Hello,
I have a problem with making association rules in Oracle 10g ODM. The problem is as follows:
I have my test dataset for testing assoc. rules. I used that dataset before for testing assoc. rules models with Oracle 9i ODM release (9.2.0.4). And I have found significant number of rules within that dataset when I used Oracle 9i ODM (9.2.0.4.).
But when I use Oracle 10g ODM to build an association rules model with the same set of parameters (min. support, min. confidence and max. rule length) I've got 0 rules.
I can't find where is the problem, so I think that there is some sort of bug within Oracle 10g ODM.
Does anyone have similar experience or have some explanation for this?
Thnx in advance,
Ivan

What version of Data Miner are you using?
If you are using the api directly, are you using the
java or pl/sql api.
Thanks, MarkI've been using Java API.
I've continued my investigation of the mentioned problem and I have found that DMS throws following exception:
ORA-38605: FI not enough memory (stringK) for candidate generation(stringK)
Cause: There was insufficient available memory for candidate generation.
Suggested action from Metalink is:
In workarea_size_policy='manual' mode, set ficarea_size to a reasonably larger value. Or, in workarea_size_policy='auto' mode, set pga_aggregate_target to a reasonably larger value
And I've increased the size of my SGA more than three times (from 128 to 512 MB), set work_area_policy=auto and set pga_aggregate_target to 20% of SGA .
After that I've got following exception:
ORA-40101: Data Mining System Error ODM_ASSOCIATION_MODEL-BUILD--20010
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "DMSYS.ODM_ASSOCIATION_RULE_MODEL", line 150
ORA-06512: at "DMSYS.ODM_ASSOCIATION_RULE_MODEL", line 1299
ORA-01452: cannot CREATE UNIQUE INDEX; duplicate keys found
ORA-06512: at "DMSYS.ODM_ASSOCIATION_RULE_MODEL", line 1364
ORA-06512: at line 1
DMS throws this exception after longer time of task execution.
Now, I don't know what to do further.
I repeat that with the same dataset and same model parameters (support, confidence and max. rule length) I've got significant number of rules with ODMS within Oracle 9i database (9.2.0.4).
Thnx,
Ivo

Association Rule Model Fails to Build

Hi all
I am running the association rules model (java api) and sometimes when I use attributes from lower hierarchies the model fails to complete and runs for
ever. In addition when I stop the process from netbeans the hard disk indicator remains permenantly on and my computer is really slow and I am unable to access anything. My attributes are quite long strings (include also gaps). Do you think that affects the model?
Thank u
K

Based on the number of rows and transactions, you are averaging 23.5 items per transaction. With only 80+ items and a few value combinations for each, my guess is that the baskets are quite dense. Many of the transactions will contain a decent-sized portion of the item name-value combinations.
Association rules has a parameter to limit the length of the rules that are generated. Have you set this to try limiting to shorter rules? Have you tried a maximum of length 2 to see if it completes? My guess is that the density of the data leads to exponentially larger combinations of rules as the length of the rule increases, and this requires more memory.
Please try to constrain the rule length to experiment. Once you have short rules (single antecedent item), take a look and see how many rules and itemsets are generated at your provided support and confidence levels (support will constrain itemsets, and support plus confidence will constrain rules). Try to increase the maximum rule length by one more, and see how fast the count of rules and itemsets expand.
This should help understand more about your data and the correct choice of algorithm parameters.
Ari

Application is not working due to memory issue

Hi Friesnds,
Kindly help me regarding settings in java. Scenario is like that. I have one server having two jboss versions (jboss-4.2.3 GA and jboss- 3.2.6). Each having java memory (JVM settings) 1 GB. The total memory of that server is 3 GB.
Problem is that every two days my application is not working due to memory issue. Once i freed the memory (through run the commands - (1). sync
(2). echo 3 > /proc/sys/vm/drop_caches
Application works fine.
I heared that the above command (echo 3 > /proc/sys/vm/drop_caches) can't run frequently because of server crash.
Kindly help me regarding this issue and provide the resolution.
Let me know if you need more information from my side.
Thanks
Ashish Shukla

All of the above. I tested on 4 different networks and had no luck, I also had friends test my network and the other networks with their iPhones and all had no issues using FaceTime.
Also after I wiped my phone (erased all content and settings) I was able to successfully initiate a FaceTime call with no changes to my network and was also able to initiate another call when connected to another previously tested network that didn't work before. Once I restored from backup though, FaceTime stopped working again.
I did find this discussion: https://discussions.apple.com/thread/5163024?start=0&tstart=0
and tried the suggestions found there too. That discussion describes my issue I am having as well.

How to correct internal memory issue

This is related to the standard SAP program SAPRCKM_MR11 in which there is one include RCKM_MR11F01 which contains a select statement :
SELECT *
     INTO CORRESPONDING FIELDS OF TABLE t_bhistory
     FROM v_ckmlgrir
     WHERE bukrs EQ p_bukrs
     AND   ebeln IN r_ebeln
     AND   ebelp IN r_ebelp
     AND   lifnr IN r_lifnr
     AND   ekorg IN r_ekorg
     AND   ekgrp IN r_ekgrp
     AND   werks IN r_werks
     AND   bedat IN r_bedat
     AND   bsart IN r_bsart
     AND   bstyp IN ht_bstyp
     AND   vgabe IN ('1', '2', '3') "Wareneingang/Rechnung/Nachbel.
     AND   loekz_k NE 'L'            "Loeschkz. Bestellkopf
     AND   loekz_p NOT IN ('L', 'S') "Loeschkz. Bestellposition
     AND ( frgrl   EQ space          "Freigabe unvollständig
     OR    frgrl   IS NULL )
     AND ( memory EQ space          "Bestellung noch nicht komplett
     OR    memory IS NULL )         "falls Feld nicht initialisiert
     AND ( xwoff   EQ space          "Wertbildung offen
     OR    xwoff   IS NULL )         "falls Feld nicht initialisiert
bei Dienstleistungsbestellungen werden mehrfach kontierte
Bestell-Pos. gegen das WE/RE-Konto gebucht, deshalb dürfen
mehrfach kontierte Pos. nicht generell überlesen werden.
      AND   vrtkz   EQ space          "keine Mehrfachkontierung
     AND   wepos   EQ 'X'            "Wareneingang wird erwartet
     AND   repos   EQ 'X'            "Rechnung wird erwartet
     AND   weunb   EQ space          "Wareneingang unbewertet
     AND   stapo   EQ space.         "Item is not statistical
There is one job FMT612P which contains this standard program and the job is going to abend
because of the select statement given above which is because of some memory issue with the internal table t_bhistory, please give us a suggestion how we can avoid the same.

Hi,
I am also facing the same issue. We had copied this report into a custom report and are trying to improve performance by breaking the view into separate queries on EKKO, EKPO and EKBE.
Can someone advise the best way to deal with this view v_ckmlgrir or a best approach to break down the view into separate queries.
Thanks
Edited by: Shreyas Shrikant Kulkarni on Jun 29, 2009 4:56 PM

Memory issue with Podcast !!!

Hi,
I have several issues with the Podcast App.
- i cannot sync podcast that i have downloaded on my laptop using itunes to the iphone,
- when i download some podcast directly on the iphone, i cannot play them, i still have to stream them,
- i have uge memory issues and i think that the downloaded version of the podcast might be somewhere on the iphone, but cannot access / delete them (over 16 GO, i have more than 4.5 GO used for 'other', that are not related to anything...)
i have tried to delete / reinstall the app, but this does not change anything...
Any help ?
Rgds

Quote
I have a MSI 975X Platinum rev 2 mb with a e6300 processor overclocked to 2.24ghz.
If your CPU operates @2.24 GHz, it means that you increased FSB frequency to about 348 MHz.
To make sure there is no misunderstanding here, let me point out the following:
Memory frequency is ALWAYS linked to FSB frequency by the FSB/DRAM Ratio you can select in BIOS. If you select 1:1 for example, the BIOS will show that memory frequency is 533 MHz. This value however, only applies to the default FSB clock speed of 266 MHz:
266 MHz x 1 = 266 MHz (or 533 MHz effective DDR2-frequency).
If your system is overclocked, what counts is only FSB/DRAM ratio, not the memory speed displayed in BIOS. That means, if you set FSB/DRAM ration to 1:1 and FSB frequency to 348 MHz, your RAM will operate @:
348 MHz x 1 = 348 MHz (or 696 MHz effective DDR2-frequency).
The main question is:
What are you talking about when you say, you can't make your RAM operate @800 MHz?
The BIOS does not offer a proper divider to get your RAM to 800 MHz @348 FSB clock speed to begin with.
You have the following choices if your overclock your system:
FSB=300 + FSB/DRAM ratio = 1.33
300 MHz x 1.33 ~ 400 MHz (or 800 MHz effective DDR2-frequency)
FSB=320 MHz + FSB/DRAM ratio = 1:1.25
320 MHz x 1.25 = 400 MHz (or 800 MHz effective DDR2-frequency)
FSB=400 MHz + FSB/DRAM ratio = 1:1
400 MHz x 1 = 400 MHz (or 800 MHz effective DDR2-frequency)
Use CPU-Z to monitor the DRAM frequency that is actually set if you are overclocking.

Memory issues

I have a 2008 Macpro...remarkable up-to -date for its age...I plan to use it for some time yet. Lately I have been a having what seems to be a memory issue. There are 12 gig of Ram installed...6x2 gig each in pairs. Sometimes on boot the screen only shows a flashing folder. I shut down and reboot, and it seems fine...but only 8gig of ram shows installed. I opened it up, unseated the boards that hod the ram, and unseated and reset all he ram, and replaced the boards. Great, all he ram shows, 12 gig again. This morning, a few days later, it did it again, showing 8. So, when I run Crucial's memory analyzer, it shows first four slots full, last four empty. A few questions...are the first four slots the top board, the last four the bottom board? This, BTW, is what it was also showing before I reseated the ram. If so, the ram in question s on a separate board. Opinions, is the ram bad, or the board, or neither? Should I reseat again and flush with contact/tuner cleaner/degreaser? Could this hurt? (it is made to clean electronics) the machine does seem ever so slightly unstable. FCP crashed once in a 5 hour edit, which it has done before but rarely. The machine itself did not crash, just the program. Can using the machine like this do more damage, or will it just not boot? I feel it is a seating or connection issue, as I have never seen ram come and go, once it is fried it's fried. The ram is still pretty pricy, but I was going to buy 4 more gigs just to fill it out. Are the two board that hold the ram identical, top and bottom? If so, is a good diagnostic tool to swap them and see what happens? Move all the ram and swap board positions, or do I risk blowing ram ? Remember, the ram did show up when I reseated, so it seems like it is I damaged. My goal would be to see if top ram stops appearing. Does ram need to be filled in order? I know in pairs, but can I just swap the boards and leave the ram, which should look like filling slots 1,2,5,6,7,8...as opposed to 1,2,3,4,5,6. Can the boards be swapped, or are they set somehow to be 1-4 and 5-8, or is that determined automatically by their upper or lower insertion? Any other advice is appreciated. Funds are low, so I need to keep this machine running. Other info, 3.06ghz (2 quad core) Xeon CPUs, 1tb system drive, 4 TB raid5' OSX 10.6.8 dual graphics display with 512MB (can't remember card model right now). Also, can I update graphic card in this machine? I looked at it a few years ago, but the I for I found was confusing and I never got to it. I figure I might be able to get a better card pretty cheap by now. Thanks all.

You have typed a huge list of ideas all as one paragraph with many questions in it without a single bit of white space. My eyes are not good enough to read a question, attempt to answer it, then dig through the sea of letters to find the next question.
Your writing needs white space, which gives it a peculiar overall shape, and allows a reader with poor eyesight to return to a particular spot in the text. So I will attempt to answer some parts of your query, but will be forced to do so from memory of what you wrote.
The Mac Pro uses Error Correcting Code (ECC) RAM. Eight extra bits are stored with each word, and checked when read back. Single-bit errors are detected, and the check bits are used to make a correction on the fly. More complicated errors are detected, and can cause your Mac to halt on a kernel panic to prevent bad data from propagating.
ECC RAM means your Mac can detect bad memory easily at Startup, and can declare pairs of DIMMs "absent" to avoid using them.
Contact corrosion, which was an issue with much older DIMMs, is rarely an issue with these FBDIMMs. Once re-Inserted ONCE, corrosion will not recur for a year or more.
The upper and lower Riser-cards are identical.
The FBDIMMs in these machines are subject to overheating and death. If Mac OS X or any program reports DIMMs as missing, at least one of the DIMMs in that pair has died (but since they are installed in pairs, both are marked absent).

Question for those with memory issues on k8n neo plat

I have been having issues with my memory on the k8n neo platinum board since i built the system last month and i just recently discovered a fact I had overlooked.
I Have discovered that if i test the system while it is cool (having sat turned off overnight for example), then memtest will pass the ram just fine and the system will be stable for a short period of time, generally longer if no intensive applications are used, ie. doom3 vs. websurfing).
This system, once it heats up ( seems mostly to be the nvidia 3 nb that needs to be "hot"- that is, a probe placed in contact with the module reaches about 58+ degrees c) then the system becomes unstable and memtest, if used, fails.
Prime95, if used on it when its cool, will usually take about an hour or so to fail, but if its used on the system when its been running a bit ( temps from nb at 58c+ and memory module hot to the touch) then it fails after about 20-30 seconds.
I have placed an active hs on the nb in an attempt to correct this, but temps still remain high with failures occuring.
I have spent the better part of three days testing the sytem this way and the above facts have held true in all these tests.
I have tested at stock ratings ( not using "auto" in bios, but using "approved ratings" of 10 x 200 for fsb and timings of 2-3-3-7 for my ram at 1:1 (which is recommended as well) since i am testing for base stability at this point;Voltages have been left at auto settings, however.
I was wondering if anyone else with memory issues could try this out and post if this seems to be the case for you as well.
Perhaps the memory issues on these boards are directly related to a problem with the nvidia 3 chipset running too hot?
Does anyone have a memory issue with these boards who is using water cooling on the NB ?
Lack of issues from people using 'problem" memory, but having no failures with water cooled nb's would support my assumption.

Yes i did mean chipset.
I am using a stock hs/fan from amd, which seems to run a tad hot but, considering that I wont overclock until I can get board stable, its not an issue. (runs 35-40 idle and up to 72 intensive). I need the warranty more then the extra mhz at present.
I removed the stock heatsink from the nvidia chipset and bought a generic chipset cooler (green, very little in the way of fins and seems to be aluminum) with a small fan.
I used a generic silver paste from Compusa to adhere both the cpu hs and the chipset.
I didnt lap either hs, just installed each as per instructions on artic silvers website (figured it had to go on the same way even generic).
I also bought a kingwin thermal center which i ran probes from to the cpu, case, and nb chipset.
I am not using it to adjust fans so those arent connected - its just a reference temp I can see at a glance, esp considering the funky bios reading this boards been known to give.
I assume from these responces that even a nvidia 3 250 chipset shouldnt get as hot as mine?
I asked about that from the retailer i bought this from but he seemed to think it was "ok" at 50ish celcius, but then again he didnt look too confident when he said it, lol.
Idle right now reads on Corecenter at 28 celcius for cpu and 33 celcius for the chipset, although, I have readings of 37 c and 43 c from the kingwin thermal center, which i believe to be more accurate.
My voltage reads at 11.67 for the 12v line at idle right now.
The psu i got is stick from the xblade case i bought in store. (http://www.xoxide.com/xblade2.html)
I have a dvd burner, cd drive, and two maxtor ide hd's attached to the system;The only card is the agp.
Is there a way to check and see if the cpu ispower starved? software monitor or even an external probe perhaps?
I havent put a probe on it, but the ram came in its own heatspreader, which gets very hot to the touch as well.
What temps should I be looking for on these items?

Firefox 3.6.8 and memory issues

After the last update to 3.6.6, I disabled plugin-container per the instructions given on the forum, and my RAM problems went away. However, after the automatic update to 3.6.8, the memory problems came back, with memory usage going up to 92%. I downgraded to 3.6.4, and turned plugin-container off again; while memory usage is currently at 84%, it is not as low as it was before the upgrade to 3.6.8. (79-80%) (There is very little difference in the websites being opened on the various tabs.)
== This happened ==
Every time Firefox opened
== I upgraded to 3.6.8

ffs guys. It's a free browser. It's cool. We love it. But can we do something about memory issues? Almost every release has huge memory leaks.
How about halving the release points and focus on memory issues for the other half of the time. I am a long time user since way back in the day and it seems to me that instead of trying to release so many versions, we should be focused more fixing the current release.
I say put a moratorium on new features until bloat can be fixed. These experiences are starting to give me a hardon for IE again.
Screw new features. I can deal with plain old tabs like they are. I don't need more bells and whistles. This browser has been kicking ass and taking names for a loonnnggggg time. We don't need more bloat. Keep the security fixes coming but everything else can wait. Take a breath. Stop and smell the roses for a few weeks.

Regarding Memory issues((error) while scheduling the job

Hi Friends
I am facing memory issue while rerunning BODS jobs in production.As we have rapid mart when jobs got failed i rerun the
job suddenly job got failed and in logs i found Memory issue
please help me what i have to do in steps
Thanks

I think there is no one solution to buffer pool issue. Buffer pool issue happens due to many reasons like how you design the data flow, requirement of push down or even insufficient memory in the running environment.
You can check by checking 'collect statistics for optimization' during running the job. Also in the data flow, try changing from pageable cache to memory cache by right clicking and selecting properties. The below link give some details on caching, might be helpful.
https://wiki.sdn.sap.com/wiki/display/BOBJ/CachinginDI
Arun

Memory issues while fetching content

HI all,
I am using SAP KM Apis. however i am facing some memory issues while using them.
I am using getContent() api on a resource. and then getInputStream on the Icontent object. I am using a ByteArrayOutputStream for writing the data. here is the code snippet:
byte[] buf= new byte[4096];
InputStream inputStream = content.getInputStream();
ByteArrayOutputStream os = new ByteArrayOutputStream(1024)
while ((count = inputStream.read(buf)) != -1)
    os.write(buf, 0, count);
byte[] arr=os.toByteArray();
THis code consuimes a lot of memory . we need to optimize this. we also tried removing BYteArrayOutputSteam and directly read the bytes in chunk :
int n;
do
     n = inputStream.read(buf, 0,(int)content.getContentLength());
while (n != -1);
however this approach has not helped us.
please suggest an approach where memory consumption is less and the entire content is fed.
thanks

Hi
This code worked for me for reading from a KM res...
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
        try {
            while ((line = reader.readLine()) != null) {
                sb.append(line + "\n");
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                is.close();
            } catch (IOException e) {
                e.printStackTrace();
return sb.toString();
Regards
BP

Memory issues - Bring demand paging to X6.

does anyone at nokia remember a few years ago a firmware update that added demand paging to N95 and enabled the phone to do multi-tasking properly without having always memory issues? Please bring it to the X6. It's frustating to run only one app (in this case, opera mobile) and having memory warnings always poping up... After a fresh start of the phone.
In the wise words of a famous Top Gear presenter: how hard can it be?

The numbers below were constantly moving as I copied them so they're not exact but here's what task manager reported. Funny thing is that it always seems to show around 3 GB available.
Physical:
Total - 12581952 Kb
Available - 3377056 Kb
System Cache - 3237124 Kb
Kernel Memory:
Total - 204288 Kb
Paged - 143704 Kb
Non-Paged - 60056 Kb
I just saw that OEM was advising "Increase the size of the SGA by setting the parameter "sga_target" to 10752 M"

IPod Nano 5th Gen memory issues?

Hello. My iPod Nano 5th gen is great, and I've been able to use it for 2 years now, and it serves me well. Perfect condition since the day I got it. However, there are some memory issues...
First off, I have only two recorded videos totaling 1 minute and 29 seconds, so it shouldn't take up much memory at all. And I don't have any other videos on my iPod either.
Well, when I go to "About" under "Settings", then it always says that there are only 902 MB left, and it's an 8 GB. What's weirder is that it says that 6.1 GB are being used my audio, and I only have 712 songs. When I select all of the songs that are on my iPod on my PC, then it tells me that it totals to only 3.53 GB. Kinda weird, and I really need this extra space now.
Secondly, I should tell you that I don't use iTunes to syncronize my iPod. I use a third-party program called CopyTrans Manager because it's simpler, easier to navigate, and faster in my opinion. Idk what it is, but it's bothering me. Hopefully some of you can tell me where I can go to eliminate these issues and finally get my iPod back to better use and full memory.
Help me please. I don't know where to go next...
Thankee.

I don't use iTunes to syncronize my iPod. I use a third-party program called CopyTrans Manager
Contact the manufacturer. Check with them to determine if the issue is with them.

Memory issues in java3d1.2(openGL ver) and 1.2.1-03 sdk(openGL ver)

hi ,
i am working in java 3d and handling memory issues .I have made following observations.
In java3d 1.2 version,when we close a primitive ,(transform group,point3d,transform 3d release) about 80% of memory used by the primitive when it was opened.but point4d releases only 2% of memory.
on a contrary when i use java3d1.2-03 sdk point4d releases about 75% of memory and others(transform3d,transform group,point3d ) releases about 40% of memory allocated to them.
can somebody tell me if there is some kind of difference in these two versions related to memory?or is there any memory bug in the later version?
thanks
sangeeta

hi,
We r using memory management tools like Jprobe and optimizer IT to count the rate of memory released.
Thanks
sangeeta

Association rules - memory issue

Similar Messages

Maybe you are looking for