Loading time into memory for a large datastore ?

Is there some analysis/statistics about what would be the loading time for a timesten data store according to the size of the data store.
We have a problem with one of our clients where loading of datastore into memory takes a long time. but only certain instances it takes this long.. maximum size for data store is set to be 8GB (64bit AIX with 45GB physical memory), is it something to do with transactions which are not committed?
Also is it advisable to have multiple smaller datastores or one single large datastore...

When a TimesTen datastore is loaded into memory it has to go through the following steps. If the datastore was shut down (unloaded from memory) cleanly, then the recovery steps essentially are no-ops; if not then they may take a considerable time:
1. Allocate appropriately sized shared memory segment from the O/S (on some O/S this can take a significant time if the segment is large)
2. Read the most recent checkpoint file into the shared memory segment from disk. The time for this step depends on the size of the checkpoint file and the sustained read performance of the storage subsystem; a large datastore, slow disks or a lot of I/O contention on the disks can all slow down this step.
3. Replay all outstanding transaction log files from the point corresposnding to the checkpoint until the end of the log stream is reached. Then rollback any still open transactions. If there is a very large amount of log data to replay then this can take quite some time. This step is skipped if the datastore was shut down cleanly.
4. Any indices that would have been modified during the log replay are dropped and rebuilt. If there are many indices, on large tables, that need to be rebuilt then this step can also take some time. This phase can be done in parallel (see the RecoveryThreads DSN attribute).
Once these 4 steps have been done the datastore is usable, but if recovery had to be done then we will immediately take a checkpoint which will happen in the background.
As you can see from the above there are several variables and so it is hard to give general metrics. For a clean restart (no recovery) then the time should be very close to size of datastore divided by disk sustained read rate.
The best ways to minimise restart times are to (a) ensure that checkpoints are occurring frequently enough and (b) ensure that the datastore(s) are always shutdown cleanly before e.g. stopping the TimesTen main daemon or rebooting the machine.
As to whether it is better to have multiple smaller stores or one large one - that depends on several factors.
- A single large datastore may be more convenient for the application (since all the data is in one place). If the data is split across multiple datastores then transactions cannot span the datastores and if cross-datastorestore queries/joins are needed they must be coded in the application.
- Smaller datastores can be loaded/unloaded/recovered faster than larger datastores but the increased number of datastores could make system management more complex and/or error prone.
- For very intensive workloads (especially write workloads) on large SMP machines overall better throughput and scalability will be seen from multiple small datastores compared to a single large datastore.
I hope that helps.
Chris

Similar Messages

Could not create the TimesTen database and load it into memory for the first time

Hi all,
I am making tutorial from http://www.oracle.com/webfolder/technetwork/tutorials/obe/db/timesten/tt1121/1_creating_imdbcache_win/1_creating_imdbcache_win.htm
I've got problem on step "Create the TimesTen Database and Configure It to be an In-Memory Cache Database" - step two.
after command:
ttisql myDatabaseDSN
I've got only:
connect "DSN=inmemoryDB";
The connection should be successful but it's not. What could be the problem of it ? I want to run it on my laptop with proccesor Intel 5 on Windows 7.
on

Per your paste, this is the error:
9:28:30.65 Info:    : 4412: 7208 0000000001EBFAD0: No such data store
19:28:30.65 Info:    : 4412: daDbConnect failed
19:28:30.65 Info:    : 4412: return 1 833 'no such data store!' arg1='c:\timesten\database\my_ttdb'
So it's trying to find or create a TimesTen checkpoint file with the prefix "my_ttdb" in the directory path c:\timesten\database. Does this directory path exist? Or alternatively have you unticked AutoCreate under the First Connection tab in the TimesTen ODBC Setup for this database? If you have unticked this, then TimesTen won't try to automatically create the database if it does not find any already existing checkpoint files, and will return an 833 error.

How to find out memory consumption for table in HANA without load it into memory

Hi,
To determine the memory consumption for a table in HANA, you can query table M_CS_TABLES, however, it requires load table into memory first, I just wonder if there has another table store memory consumption information for all HANA tables regardless it load into memory or not. Below is screenshot for one of table in my system, since that table is partially loaded into memory, "Total Memory Consumption (KB):" tell me the memory consumption of the portion load into memory, what I am looking for is something like "Estimated Maximun Memory Consumption (KB)" which provides me total memory consumption for that table including portion doesn't load into memory, of course I can use this Esitmated information, but consider I have close to thousand tables in my HANA system already, it's not pratical to check tables one by one.
Thanks,
Xiaogang.

Hi Xiaogang,
Estimated Memory Size that you see in the Table Run time Information - same is available in M_CS_TABLES also
If you don't get the size of any Table in M_CS_TABLES View, then the same will also not be available in Runtime information of the Table
Even if tables are not loaded into memory, you can get the Estimated Size, just try running the query with filter LOADED = 'NO'
Regards,
Vivek

How do I find out the exact path of each and every file that LabVIEW finds and loads into memory for a given top level vi?

How do I find out the exact path of each and every file that LabVIEW finds and loads into memory for a given top level vi? There is probably a trivial, easy way to get this info, but I have not yet found it! Thanks..

Or if you want to grab all the paths programatically, try the attached VI.
Open the top level that you want all the paths from and close all others, then open the
attached and run it. It will return an array of all the VIs that the VI
in question uses, including vi.lib VIs. You can filter these as well if
you like.
Ed
Message Edited by Ed Dickens on 08-01-2005 07:01 PM
Ed Dickens - Certified LabVIEW Architect - DISTek Integration, Inc. - NI Certified Alliance Partner
Using the Abort button to stop your VI is like using a tree to stop your car. It works, but there may be consequences.
Attachments:
Get all paths.vi ‏29 KB

Missing/empty symbols, "Could not load scene into memory. Your document may be damaged." CS6

Hey folks,
we've been getting this problem a lot. We had it in Flash CS5.5 and still have it CS6.
We're animating a series in Flash and this problem keeps cropping up way to often, but seems totally random.
Basically I'll be working on a .fla saving regular versions through out the day, no problems. I'll close the file down. Then I (or someone else on a different computer) will open it up and I'll get the error msg "Could not load scene into memory. Your document may be damaged." Everything will be fine except for a few missing graphic symbols. Sometimes just the one, sometimes a whole bunch. So where the symbol should be on the stage, instead you get a small white square, but it still contains the animation information. So it will still move about the stage. If I bring in the same symbol from an old scene I can swap them out and the problem is fixed, (until it does it again on a totally different graphic symbol).
If i select the symbol in the library the preview is white/blank and i can't go into it to edit. When I select the symbol on the stage, the "instance of:" gives me a blank "_ _ _" in the properties tab. How ever if i right click show in library it does show me the correct (although broken as in blank) symbol in the library.
I havent been able to reproduce this problem on purpose but I'd say it happens in roughly one scene out of 10. Its seems to happen at random, I know there must be something in common but i havent figured out what. Its going on in heavy scenes (large library multipule characters), lite scenes (just the one character). Its not like it happens to the same symbol, different ones each time. Everything is local to the scene, all elements are created in flash. We are working across a network but as everything is local to the scene i don't see how that would be a problem. Also we don't have duplicate named symbols. It also won't necessarily affect something that has been changed since the last version. could be a background element thats been there, untouched since the first version of the scene.
One point on how we set up our scenes which may be relevant. Start with empty scene, copy and paste symbols from other scenes to populate. Save.
The problem could appear on say version 4 or version 36. Older versions are usually fine.
So to sum up. Flash file is fine when we save it, open it up again later and a symbol will have disappeared. It's happening far too often.
I know how to fix the problem, thats not what I'm asking. I need to know WHY its doing it so we can stop this from happening in the first place.
Phew! Thanks for reading this far. Any help would be highly appreciated.
Sander/

Can you elaborate on how you use the network? Are you working from FLA/XFLs stored on a machine across a network (and saving them across the network)? I had plenty of those issues and Adobe has always warned not to work across a network. I just fell into the habit of copying over what I need to work on locally, then updating the file servers at the end of the day. Nothing was corrupt after that.
Also lately in CS5.5 (not CS6 yet) I had noticed that I could change some graphics assets, close the document in OS X and open in Windows only to find freshly updated graphics reverted back. What's even more odd is if the OS X machine that made the changes opens it the changes are still made. This happens vice versa as well. My only solution on that was not to work cross-platform with other machines on a network unless absolutely necessary.

Loading images into memory

I have an applet that I need to load images into memory. For instance, lets say I am in the first section of the applet. While that section is being showed, I want to load the background image of the second section into memory.
To load the initial image into momory, I am just using a Mediatracker. No problem there. But I don't want to load all 20 backgrounds into memory at the same time as they take a lot of time. My understanding is that if I create a new MediaTracker while the first chapter is running, it will potentially cause some chaos, as that will stop my thread from running while I have an image loading.
Somebody told me perhaps I could create a new thread and have that thread load the backgroudn into momory? Perhaps something like this?
public class TestClass extends JApplet {
     private TestClass thisClass;
     public void init() {
          thisClass = this;
          Runnable r = new Runnable() {
               public void run() {
                    MediaTracker tracker = new MediaTracker(thisClass);
                    Image nextImage = getImage( getDocumentBase(), getImagePath() +"img1.jpg");
                    tracker.addImage(nextImage,0);
                    try {
                         tracker.waitForID(0);
                    } catch (InterruptedException ie) {
                         System.out.println(ie.toString());
          Thread t = new Thread(r);
          t.setDaemon(false);
          t.start();
          while(t.isAlive()) {
               int i = 1;
          t.stop();
          t.destroy();
}No idea if I am on the right track or not? Another friend told me something about swing helpers but couldn't tell me much more?
Thanks in advance!

I use media tracker when I need information about how percent the image is loaded. you can use JLabel to load the images since it has own image observer in it.
hope you just want to deal with it. easiest way I can offer :)

Flash Pro CC saved fla project - "Could not load scene into memory. Your document may be damaged."

Hi all, I've been working on a project in Flash Professional CC in windows 7 that was started in a previous version of Flash Pro and occasionally after I've saved the project and tried to open it the next day I get the "Could not load scene into memory. Your document may be damaged." message and only a blank Scene 2 in the project. I've previously been reverting back to the old saved project file and starting over. Its wasted a lot of time so far.
I can still open the same exact project in CS6 on another older computer but not in CC. I can also save the fla in CS6, transfer it to the other computer and then it will work again (for now). This isn't going to be an option in the future as CS6 was a trial version that is on its last use today.
I've tried to create a new flash pro cc project and just copied and pasted the library and scene layout from the old project into the new project. This worked for a couple of saves/ opens but the error has happened again. I'm scared that now that I've reached the point in the project were my work in the animation/ visual side will start to be more intensive that I will continuosly have to worry about losing all of my work. I'm using a document class so the AS3 code hasn't been an issue thankfully. Any help would be appreciated.

Thanks for the reply Amy, that is essentially what I am doing already only I'm not using version control software I'm just managing it on my own since its a solo project, though I have been considerring using git anyway. It still doesn't explain why I can open and save the supposedly damaged project (according to CC) in CS6 and only then will it work again in CC. Also I'm not gettting a chance to save as a new version over and over for long. Its only a couple of days max, usually just over one day, and I've primarily been working on the code in Flash Builder with little functional change in the actual fla project. Its primarily changing movieclips instance names, which in this project is tedious and time consuming. Having to revert back to the older working version is exaclty what I'm trying to avoid. I shouldn't have to commit the project to a repository every little change that I make to the fla file.

Damaged Flash CC files ("Could not load scene into memory.")

I just installed Creative Cloud the other day, and had the following problem immediately.
I create a brand new .FLA and import a few assets from another person's Illustrator CC file, then write some ActionScript for basic interactivity. I can work for hours with that file. But when I save, quit and reopen the file (either 2 seconds later or the next day), I get a warning that says "Could not load scene into memory. Your file may be damaged." The library is empty, no instances of symbols on the stage, nothing. I can't open the file with Flash CS6 or anything: it's shot. I repeated the process several times with brand new Flash files, but always get the same results. I even deleted everything and reinstalled and still got the same results.
I switched to using Flash CS6, but when I imported artwork from Illustrator CC, I was getting corrupted artwork and periodically I'd get an error that said something to the effect of "You can't import artwork from a newer version of Illustrator into Flash CS 5.5. (WTF? I'm in Flash CS6.)
Any ideas? I'm on a Mac, OS X 10.8.4. Thanks very much.

Hi Gregir ,
Is it possible for you to share the file or the illustrator assets with us? We can take a look and get back on it.
Can you check the size of the PublishSettings.xml file which would be present after unzipping the zip created from the fla renaming it as .zip.
Thanks and Regards,
Sudeshna Sarkar
Adobe Flash Professional Team

"Could not load scene into memory. Your document may be damaged." Lost whole timeline.

Please help me - I have been working on a Flash CS6 file for a while now, and just got the message "Could not load scene into memory. Your document may be damaged." The file is not big, all it contains is 3 layers - one embedded video FLV file and 2 layers on top that only contain brushstrokes. The file is on my external hard-drive and now I get the message when I open it on either of my computers. I read feedback that I should re-link or re-import my assets, so I re-made the FLV file and imported it to replace the old one that it said couldn't be found. The problem is the timeline is completely empty. Even once I import the new video, the timline only has one "layer 1" with only one blank keyframe. Where is the rest of my work? I am good with backing up but was just away for a week and got a lot of work done so I couldn't back it up until I was back home... and obviously that isn't happening now.
Please help if you have any idea where my work has disappeared to.

Hi GregirArt,
Can you please check your PublishSettings.xml file whether is there any duplicate profile entries?
You can check the same by renaming your .fla to .zip and extract the zip file to open the PublishSettings.xml
If you find multiple duplicate profile entries then please remove all the duplicate entries from the file and rename the file back to .fla and see if that helps.
Thanks!
Mohan

Installing Flash CS5.5 corrupted my CS5 Fla - Could not load scene into memory. Your document may be

I have a FLA that worked fine in CS5, I have copies of the file going back 6 months. Every file I open gives me this error: Could not load scene into memory. Your document may be damaged.
How could every fla I've backed up give the same error? They all worked fine in CS5. Does anyone know of a fix? Or do I have to go back to CS5?
Thanks!

I reinstalled Flash Pro CS5 and it opens up all my FLA's (of the same project) perfectly. But Flash Pro CS5.5 gives me this error: Could not load scene into memory. Your document may be damaged.
So is CS5.5 unusable?
cheeres.

Flash CS5 Could not load scene into memory. Your document may be damaged.

Hello everyone!
Just bought the new CS5 Flash and was eager to begin my first document. I opened a new document Saved As to begin and then setup my basic page with only one imported graphic. Hit Save and received an error message: Could not load scene into memory. Your document may be damaged. When I closed and reopened the document this same error appears twice in my Output Menu.
I have noticed other people with several of simular issues and am not sure where to start to find an answer. Any help is greatly appreciated.
Thanks in Advance,
Biddler1

Thought I'd add to this since I have a file with a similar problem. I'm carrying the files home to work on in the evening. I've noticed at work that when I update an image in the library flash shows me a path going all the way out to the root of the network share and then back to the file. The file is just one folder away. Maybe Flash has recorded an absolute path instead of a relative path to the asset.
At home, Flash can't find the asset at all since it's recorded a non-portable path.
The question is, if the asset has been imported to the project file, why is flash checking the original in the first place?

Loading Time in TPVS for entire MOT

Hi ,
I have an issue in TPVS planning. I have maintained Loading time based on Means of Transport in Condition "LOAD".Say Loading time is 4:00hrs.
After the optimisation run,in the planned shipment System is taking Loading time 4:00hrs for each freight unit rather than for entire Vehicle.For example,if i have 4 freight units in the Shipment,Total Loading time 16hours is being captured by system.It is understandable that since Transportation planning is based on Freight units in TPVS hence it is taking for each item. Is there any way to capture only 4:00 hours for entire Truck(means of transport).
Regards
Ahmed
Edited by: Aktar Ahmed Shaik on Feb 5, 2008 2:01 PM

Hi Ahmed,
TP/VS calls APO-scheduling per freight unit that is loaded or unloaded, and then the loading and unloading times will be calculated separately.
Although it's possible to influence the individual scheduling result by a BAdI, I don't think it's possible to get the desired result as during the call you never know how many freight units will be loaded onto the truck in total. For this reason it's not possible to distribute the total loading time per truck to the number of freight units that is loaded or unloaded.
For the optimizer it would of course be possible to update the loading/unloading durations in the shipments after they have been created, but as this will change the total duration for loading and unloading you will most likely end up with shipments that can not be posted to LC due to some resource capacity conflicts. I would not recommend to do this...
For me it looks like there's no way to achieve the desired result, especially not in standard.
To get to a solution that's at least close to what you want you could create some stepfunctions based on the capacity of the resource and the freight unit. However, if the resource will not be fully utilized this will lead to too small loading and unloading durations...
Regards,
Sebastian

Read a file without loading it into memory

I have a VERY large file filled with words. I want to determine if a String in my program matches one of them. Loading the entire file into memory would take a long time and would use way too much memory. Is there some way to load just the part I need each time or make the comparison directly from the file on the hard disk?

The file is static, so your plan seems to have merit.
I've only used sequential files before, but I'm
assuming this would require a random access file since
you have to begin reading at a specified offset. Am I
correct?Yes.
In addition, if this file is fixed-length "records" so that you could programmatically determine the word offsets without having to keep a map of indexes in memory, you could just employ a search algorithm (I forget what this one is called) - you could start by defining 2 endpoints, initially at offset 0 and the offset of the last word.
Iteratively read the word halfway between the 2 endpoints, adjusting the endpoints until a match is found or the endpoints converge.

Process of initalization of DB into memory for timesten

Hi
Can i know the internal process of initialization of DB into memory in timesten , when a new connection is establishing ?
Will timesten create tables and indexes in RAM when first connection is established if the RAM policy is default?
want to know the internal functional flow of timesten when any command is fired against it.
Regards
Siva Kumar

TimesTen is a fully persistent database. The database contents, including tables, indexes and other objects along with the data exist both in memory (the operational database) and in the checkpoint files on disk (a persistent copy). When the database is started up (ramLoaded) the most recent checkpoint fikle is loaded into memory. There is no need to 'create' any structures etc. at this point since they are already present in the checkpoint image loaded from disk. Although it is possible to have a TimesTen database that gets loaded into memory when the first application connection occurs and gets unloaded from memory when the last application connection disconnects in general we recommend that you explicitly control when the database is ramLoaded/ramUnloaded using the 'manual' ramPolicy and the ttAdmin -ramLoad / -ramUnload commands. In this way you can avoid excessive loading and unloading of the database to/from memory.
TimesTen works similarly to other databases; when a SQL statement is prepared (parsed in Oracle database parlance) the TimesTen optimiser used the information in the data dictionary tables to generate a query plan. There is no concept of a data dictionary 'cache' since all data in TimesTen is permanently in memory. Active query plans are cached within TimesTen memory since they are transient data. When executing a query (using a plan) TimesTen accesses index and table data directly in memory. There is no 'buffer cache' since all data stored in TimesTen is permanently resident in memory. In general, for OLTP type queries, a query is parameterised, prepared just once and then executed many, many times. This is the way to achieve maximum performance with any SQL database.
As a result of these (and many other) optimisations/simplifications Timesten can achieve very high performance through simpler algorithms that require less CPU cycles to do the same work as in a more complex database.
Does that help?
Chris

How can I "profile" folio articles load time and memory use?

Hi folks,
I am beating my head against the wall since last week:
How can I figure out a problematic article (one or more), that causes both Content Viewer and custom viewer (.IPA) to crash on load?
The crash report states that "application took too long to start", in some cases it is "jettisoned".
The whole layout has 150+ arcicles, each about 30 pages long in average, with the total size of 705 Mb. We have about 30...40 3D images there, lots of navto links, embedded video/ audio clips - I think we used every MSO type available for iOS viewer.
All articles are of PDF type.
No custom JavaScript code, no HTML tweaking - just a transition from relatively complex inDesign layout.
At some point, while filling the folio with content, the Content Viewer started crashing on load.
We rolled back and startred re-assembling the whole package, checking once 30...60 more articles were added.
Our DTP guy was backing up intermediate results while adding articles, so I have several .IPA of the same layout, one with less than 30 articles, and several subsequent versions (60, 90 articles, etc). The smallest one works great even on 1st iPad - a bit slow (about 5 seconds to render a page when jumping between articles), but no crashes, no jerky pages, zooming and flipping is perfect.
All bigger versions (30+ articles) crash on both iPad1 and iPad3.
Adobe states there is no limitation in the number of articles and pages per article, but it seems to be a limit which we don't know. Or there is something in our design that causes crashes.
WHAT TO DO???
The idea to start from stratch again, making a backup before adding each article, uploading and testing is not good. It will take weeks, we don't have that much time.
Can there be a "profiling tool" for DPS, similar to what programmers use when the code is unacceptably slow?
Some switches to run the installed custom viewer app from command line, with verbose logging, debugging, whatever???
Is there any way to know, what exactly on our side causes the problems?
Need help ASAP.
Regards
Serge

Mike, I basically have to do the same. The idea to use DPS for something else than glossy magazine or corporate catalog, is my brainchild, and losing hopes feels like euthanazing a pet, to say the least.
With the only exception: I don't see DPS as an inferior product. Rather, it is a raw product, brought to the light before it is really ready, in order (as it is often, if not always, happens in hi-tech industry) to corner the territory before competitors catch up.
As you could see in other threads, there are no books or training courses available yet, and I believe Adobe team is not quite sure what can and cannot be done with DPS. A part of the minefield is Apple iOS, that enforces certain limitations to ensure system stability. And I am sure, there is a lot of other factors in play that we are not aware of, and probably never will be.
The point of saying what's been said is: there is a price to be paid for being on the bleeding edge of technology. This price, other than spending money for expensive equpment, other than investing time into learning and experimenting, is taking the risk of failure. The risk that it will not work out due to reasons beyond your control. So I don't think Adobe is much to blame here, they are playing the same game. Accept the rules or leave...
Just my 5 cents...

Loading time into memory for a large datastore ?

Similar Messages

Maybe you are looking for