Should I care about "Damaged Database" files in ~/Library/Syndication/ ?

Hi, friends —
I haven't used RSS much.
I'm curious: What is the purpose of the database files contained in the ~/Library/Syndication folder — and is there any reason to save the several "Damaged" Database files that are retained there?
At the moment, this folder contains a "Database3" and four "Damaged" Databases (~old), one of which is labeled "Broken." Each file is ~8–20 MB.
From the name and contents (I opened and perused a "Damaged Database" file using TextEdit), I'm guessing these are associated with RSS...?
Thanks!
Dean

Hi, macjack and Dr. Smoke —
Muchas gracias to you both for your thoughtful replies!
I'd just finished restarting my eMac after removing the Damaged Databases to my desktop per macjack's suggestion when I saw your message, Dr. Smoke. Thanks for your analysis and for the citation.
I had also searched the KnowledgeBase (9 results) and ADC (3) for resources on syndication, but hadn't had a chance yet to read the Safari RSS Developer Release Note. As you've noted, it comes to the point of my question —
Quoting from the Note for anyone who stumbles across this thread and is interested:
"Purging the Database
Safari RSS stores all news articles in a database. There is not yet any mechanism to control how long articles live, or to delete articles on demand, so the database will grow monotonically.
If the database is taking up too much disk space, you can use the following workaround to delete it. You won't lose any of your subscription data (that's stored in Safari's bookmarks) and the SyndicationAgent will re-fetch all current articles the next time it runs, but older articles will be lost.
    1.   Quit Safari.
    2.   Quit the SyndicationAgent process by entering "killall SyndicationAgent" in a Terminal window.
    3.   Move the directory ~/Library/Syndication to the Trash, or delete it."
[Emphasis added.]
Thanks for pointing out the potential for growth of this database; it may be worth keeping an eye on for some folks — depending on RSS usage, I guess. Seems an odd design concept to me.
My current Database3 file covers ~ the last 9 weeks — but this may not be comparable to others', since I haven't too big a collection of RSS feeds.
I was ~surprised to find so little documentation about SyndicationAgent on Apple's website (no Man page, etc.). Does that color me naïve?
I scanned through my (Text-archived) log files without finding any close correlations between SyndicationAgent or Safari entries and these databases (in terms of time). Since the latest "Damaged Database" file is dated 4-½ months ago, this issue isn't a big concern — as I said, just curious...
Thanks again!
Warm regards,
Dean
      I edited this message...   But probably not enough!

Similar Messages

  • Should I care about HD versus SD when buying a new camcorder?

    Should I care about HD versus SD when buying a new camcorder? I coach track, and need a good optical zoom and want to keep my budget around $300-400. Should I care about HD? Suggested camcorders?

    imho, there is no more reason to decide for SDef.
    speaking of Optical Zoom + price range: consider those 'still'-cams with video-ability. I like to recommend my (old) Lumix FZ38, a lense of 26-580mm is excellent for sports recordings. Price. >300€ ...

  • Should I care about BT not available? Or... Is Deep Sleep safe??

    As many others here from what I've seen, BT in not available from time to time. I would rather put my MB to sleep rather than shut it down completely. I've downloaded the Deep Sleep Widget and I'm thinking of using this every time BT becomes unavailable. It seems that BT is not available more and more these days.
    I don't use BT for anything and I'm wondering if it's safe to use Deep Sleep to put MB to bed each night.
    Is it OK to use DS on my MB when BT is MIA?
    Thanks

    So what does this mean? Does this mean that if I am
    going to spawn a thread which updates a Swing
    component, that any code which modifies that component
    (say setText, or setLocation, etc...), I should wrap
    those commands inside Runnable and pass them to
    SwingUtilities.invokeAndWait()???If the code that is updating the Swing component is not actually in the Swing thread, then you do have to do that. The Swing thread, by the way, is the one that calls your ActionListeners and other component listeners like them. So you can update Swing components directly from Swing listeners, but otherwise you have to do the invokeAndWait business.
    Do I only have to do this if I think there may be
    multiple threads accessing the Swing object at the
    same thing? I assume so...Yes. And remember that the Swing thread is one of those multiple threads; refer back to the previous comment.
    An example I'm dealing with is updating a
    JProgressBar. I create multiple threads at startup to
    get initial values. These commands take time and it is
    much faster to start them all at once rather than run
    them sequentially. However, each job then increments
    the JProgressBar when it completes. I figured this is
    as good example as any of violating this "Thead Safe"
    issue, since you have multiple threads trying to
    update a single Swing component.Yes, that's a good example.

  • Should I care about a dead pixel?

    I just got my replacement and I noticed a black dead pixel on the top right. I hate idea of waiting another month or being too picky. I thought I read someone had posted something you can try to fix pixels.

    Kappy wrote:
    You will find that many displays have one or two dead pixels, and this is considered normal. Apple will not replace a display with one dead pixel.
    Whether it's 'considered normal' by Apple or not is irrelevant. They only state this knowing that it will put some people off from doing anything about it. As it's perfectly possible for Apple to produce a Mac without dead pixels, there's no reason to accept anything less. Everyone pays their hard-earned money to buy a new Mac. It isn't a lottery as to whether you get a flawless display or not. If I get a perfect example and yours comes with dead pixels, wouldn't you expect yours to have the same level of quality as mine? Would you be happy knowing it was possible to have a dead-pixel-free display but got the next one off the production line? Whether you use your 14 day 'cooling off' period to return your Mac due to dead pixels, and here in the UK we pay no restocking fee, or whether you insist on a replacement instead is up to you. You are entitled to receive a fully working machine and that includes ALL the pixels.

  • Cpufrequtils: Why should I care, and acpi-cpufreq vs. p4_clockmod

    I'm a cpufrequtils newbie. In the years I've been running Arch on my laptop, I've never set it up.
    I wonder, why should I care about cpufrequtils? What should I know about the technology as a laptop user? Does the fact that I rarely unplug make a difference?
    I also wonder what module I should be using for frequency scaling. The cpufrequtils article recommends acpi-cpufreq, but the article for my specific laptop recommends p4_clockmod (for some unspecified reason).
    Any insight would be greatly appreciated. Thanks in advance!

    Just modprobe both acpi-cpufreq and p4-clockmod, only one of them should be able to load.
    the article wrote:
    For older Intel CPUs, the command above may return with:
    FATAL: Error inserting acpi_cpufreq ([...]/acpi-cpufreq.ko): No such device
    In this case, replace the kernel module acpi-cpufreq with speedstep-centrino, p4-clockmod or speedstep-ich.
    CPU frequency scaling saves power and keeps your processor cooler. It's worth using on a desktop or a server, not just a laptop.
    If you use the ondemand governor, and add:
    (sleep 5 && echo 11 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold)
    to /etc/rc.local, you'll have basically zero performance loss. Usually it scales up the CPU at 95%, with this it will do it at 11% (you can't go lower atm).
    You can load the  "cpufreq_stats" module and see what it's doing with cpufreq-info. You should notice your laptop fan spin down quite a bit once you set it up.
    If you have a watt meter, you can take out the battery and see the power use drop from something like 17w to 12w.
    Last edited by thestinger (2010-11-26 05:18:31)

  • Perf. tip (windows): defragment database files only

    Hi,
    since the database files of LR are large and are heavily accessed, it is good idea to defragment those files frequently. With the defrag built into XP and vista you can however only defragment a whole disk and this can take a considerable amount of time. Below is a link to a (free) command line tool that can defrag only a single file and does an excellent job on that!
    http://www.microsoft.com/technet/sysinternals/FileAndDisk/Contig.mspx
    One option is to put a link to contig.exe in the 'send to' folder and then manually defrag the LR database files ('Lightroom Library.aglib' and 'Lightroom Previews.lrdata\thumbnail-cache.db' ) or setup a special task in the windows task scheduler to do it automatically.
    After importing pics, my 500MB aglib file was spread all over the disk with more than 800 fragments and contig.exe reduced it to 1 fragment after ~30s. Speed improvement after the defrag was quite noticeable.
    Another good idea is to switch of windows file indexing when using LR, especially during a large import. You should set your anti virus software to ignore the LR library files. Doing all this did make quite a difference for me! Hope it helps!

    Ok, I think I found my problem.
    oracle-xe configuration runs a script called XE.sh to create the databse in
    oradata/XE. This script is failing because the init.ora and initXETemp.ora
    have two parameters which are not being set.
    I changed these:
    sga_target=%sga_target%
    pga_aggregate_target=%pga_aggregate_target%
    to:
    sga_target=209715200
    pga_aggregate_target=69206016
    in both .ora files. Erased /etc/default/oracle-xe and reconfigured.
    Now the database is created in oradata/XE.
    If anyone knows why these parameters are not being populated or if these
    numbers are not correct please let me know.
    Thanks
    -- Ramiro

  • I loaded iTunes on a new laptop. I copied and pasted my albums to a 4G stick and down loaded them to the new laptop. Placing them in, Libraries – Music – iTunes and than in Itunes used File – add file to library. So far so good. For what ever reason one o

    I loaded iTunes on a new laptop. I copied and pasted my albums to a 4G stick and down loaded them to the new laptop. Placing them in, Libraries – Music – iTunes and than in Itunes used File – add file to library. So far so good. For what ever reason one of the CDs would not load onto the stick. I got the orginal CD and loaded directly to Itunes. Now when I look in  Libraries – Music – iTunes it’s not there but it is in and plays in iTunes. What should I have done or should I be doing? Thanks.

    It's not a unique situation - happens all the time, smaller hard drives getting filled up.
    It's also easy to fix. I just moved my itunes content to a bigger hard drive.
    First I made a backup of the itunes database file - iTunes Library.itl. Just in case anything went wrong.
    Nothing did go wrong, but I'd have it in case anything DID go wrong. That file holds ratings, playcounts, Date Added, and other metadata.
    Second, in itunes change the preferences > advanced tab to tell itunes to put everything on G:
    Set the itunes folder location to something like G:Media or G:iTunes
    This will affect all FUTURE additions to itunes and not do anything to the existing locations.
    Make sure the checkbox for "Keep itunes organized" is NOT checked. I also left the "Copy files when adding" unchecked.
    Third, to move all the files to G: without itunes losing the paths, run the File > Library > Organize > Consolidate command. This will need to run a while, so be patient as it copies all the files to the new G: location. This does NOT delete the originals. You will have to do that.
    Fourth, make sure everything in itunes is OK and you can play stuff in your library. If you're really cautious you can run a script to check for missing/dead tracks.

  • How can I fix a damaged FileMaker database file?

    My Mac mini with a FileMaker Server crashed. One of the database files did not close correctly. How can I fix this file?

    Thank you very much for your good questions.
    I had forgotten about the repair function.
    I repaired the damaged file with my FM and reloaded it into the FM Server.
    It works fine again. My problem is solved
    Thank you very much.

  • Error message after 10.8.2 Install: SpeechSynthesis.app is damaged or incomplete. What should I do about this?

    I installed 10.8.2 and it seem to go well except at end an error message reading 'Error SpeechSynthesisServer.app damaged or incomplete'. What should I do about this and what apps would be impacted?

    Triple-click anywhere in the line below to select it:
    codesign -vv /S*/*/*/A*/*/A/*/S*/*/A/S*S*S*
    Copy the selected text to the Clipboard (command-C).
    Launch the Terminal application in any of the following ways:
    ☞ Enter the first few letters of its name into a Spotlight search. Select it in the results (it should be at the top.)
    ☞ In the Finder, select Go ▹ Utilities from the menu bar, or press the key combination shift-command-U. The application is in the folder that opens.
    ☞ Open LaunchPad. Click Utilities, then Terminal in the icon grid.
    Paste (command-V) into the Terminal window.
    Post any lines of output that appear below what you entered — the text, please, not a screenshot.

  • I cannot view folders and files in finder on macbook pro OS version 10.9.5. Why it happens and what should I do about it?

    I cannot view folders and files in finder on macbook pro OS version 10.9.5. Why it happens and what should I do about it?

    Please post a screenshot that shows what you mean. Be careful not to include any private information.
    Start a reply to this message. Drag the image file into the editing window to upload it. You can also include text in the reply.

  • HT201303 I get several messages each day about resetting my Apple ID password. Should I care?

    I have no idea where to put this security question. So I guess it could go here.
    I have accounts for iPhone, iPad and for program development.
    There is no clue if someone is attempting to hack my password or if they do believe that my username is theirs. Any way they seem not to give up trying.
    Should I care? Should I do something?

    Just what do the emails say?
    Could somebody be attempting to change the passwords for you account?
    Without knowing more all I can suggest is
    Apple - Support - iTunes - Contact Us

  • I just purchased MBP and it is my first Apple computer. How concerned should I be about virus software and what do you recommend for security software?

    I just purchased MBP and it is my first Apple computer. How concerned should I be about virus software and what do you recommend for security software?

    1. This comment applies to malicious software ("malware") that's installed unwittingly by the victim of a network attack. It does not apply to software, such as keystroke loggers, that may be installed deliberately by an intruder who has hands-on access to the victim's computer. That threat is in a different category, and there's no easy way to defend against it. If you have reason to suspect that you're the target of such an attack, you need expert help.
    2. All versions of OS X since 10.6.7 have been able to detect known Mac malware in downloaded files, and to block insecure web plugins. This feature is transparent to the user, but internally Apple calls it "XProtect." The malware recognition database is automatically checked for updates once a day; however, you shouldn't rely on it, because the attackers are always at least a day ahead of the defenders.
    The following caveats apply to XProtect:
    It can be bypassed by some third-party networking software, such as BitTorrent clients and Java applets (see below.)
    It only applies to software downloaded from the network. Software installed from a CD or other media is not checked.
    3. Starting with OS X 10.7.5, there has been another layer of built-in malware protection, designated "Gatekeeper" by Apple. By default, applications and Installer packages downloaded from the network will only run if they're digitally signed by a developer with a certificate issued by Apple. Software certified in this way hasn't actually been tested by Apple (unless it comes from the Mac App Store), but you can be reasonably sure that it hasn't been modified by anyone other than the developer. His identity is known to Apple, so he could be held legally responsible if he distributed malware. For most practical purposes, applications recognized by Gatekeeper as signed can be considered safe.
    Gatekeeper has, however, the same limitations as XProtect, and in addition the following:
    It can easily be disabled or overridden by the user.
    A malware attacker could get control of a code-signing certificate under false pretenses, or could find some other way to evade Apple's controls.
    For more information about Gatekeeper, see this Apple Support article.
    4. Beyond XProtect and Gatekeeper, there’s no benefit, in most cases, from any other automated protection against malware. The first and best line of defense is always your own intelligence. All known malware circulating on the Internet that affects a fully-updated installation of OS X 10.6 or later takes the form of so-called "trojan horses," which can only have an effect if the victim is duped into running them. The threat therefore amounts to a battle of wits between you and the malware attacker. If you're smarter than he thinks you are, you'll win.
    That means, in practice, that you never use software that comes from an untrustworthy source. How do you know whether a source is trustworthy?
    Any website that prompts you to install a “codec,” “plug-in,” "player," "extractor," or “certificate” that comes from that same site, or an unknown one, is untrustworthy.
    A web operator who tells you that you have a “virus,” or that anything else is wrong with your computer, or that you have won a prize in a contest you never entered, is trying to commit a crime with you as the victim. (Some reputable websites did legitimately warn visitors who were infected with the "DNSChanger" malware. That exception to this rule no longer applies.)
    Pirated copies or "cracks" of commercial software, no matter where they come from, are unsafe.
    Software of any kind downloaded from a BitTorrent or from a Usenet binary newsgroup is unsafe.
    Software with a corporate brand, such as Adobe Flash Player, must be downloaded directly from the developer’s website. If it comes from any other source, it's unsafe.
    5. Java on the Web (not to be confused with JavaScript, to which it's not related, despite the similarity of the names) is a weak point in the security of any system. Java is, among other things, a platform for running complex applications in a web page, on the client. That was never a good idea, and Java's developers have had a lot of trouble implementing it without also creating a portal for malware to enter. Past Java exploits are the closest thing there has ever been to a Windows-style "virus" affecting OS X. Merely loading a page with malicious Java content could be harmful. Fortunately, Java on the Web is mostly extinct. Only a few outmoded sites still use it. Try to hasten the process of extinction by avoiding those sites, if you have a choice.
    Java is not included in OS X 10.7 and later. Discrete Java installers are distributed by Apple and by Oracle (the developer of Java.) Don't use either one unless you need it. Most people don't. If Java is installed, disable it — not JavaScript — in your browsers. In Safari, this is done by unchecking the box marked Enable Java in the Security tab of the preferences dialog.
    Regardless of version, experience has shown that Java on the Web can't be trusted. If you must use a Java applet for a specific task, enable Java only when needed for the task and disable it immediately when done. Close all other browser windows and tabs, and don't visit any other sites while Java is active. Never enable Java on a public web page that carries third-party advertising. Use it only on well-known, password-protected, secure websites without ads. In Safari 6 or later, you'll see a lock icon in the address bar with the abbreviation "https" when visiting a secure site.
    Follow these guidelines, and you’ll be practically as safe from malware as you can be.
    6. Never install any commercial "anti-virus" or "Internet security" products for the Mac, as they all do more harm than good, if they do any good at all. If you need to be able to detect Windows malware in your files, use the free software ClamXav — nothing else.
    Why shouldn't you use commercial "anti-virus" products?
    Their design is predicated on the nonexistent threat that malware may be injected at any time, anywhere in the file system. Malware is downloaded from the network; it doesn't materialize from nowhere.
    In order to meet that nonexistent threat, the software modifies or duplicates low-level functions of the operating system, which is a waste of resources and a common cause of instability, bugs, and poor performance.
    By modifying the operating system, the software itself may create weaknesses that could be exploited by malware attackers.
    7. ClamXav doesn't have these drawbacks. That doesn't mean it's entirely safe. It may report email messages that have "phishing" links in the body, or Windows malware in attachments, as infected files, and offer to delete or move them. Doing so will corrupt the Mail database. The messages should be deleted from within the Mail application.
    ClamXav is not needed, and should not be relied upon, for protection against OS X malware. It's useful only for detecting Windows malware. Windows malware can't harm you directly (unless, of course, you use Windows.) Just don't pass it on to anyone else.
    A Windows malware attachment in email is usually easy to recognize. The file name will often be targeted at people who aren't very bright; for example:
    ♥♥♥♥♥♥♥♥♥♥♥♥♥♥!!!!!!!H0TBABEZ4U!!!!!!!.AVI♥♥♥♥♥♥♥♥♥♥♥♥♥♥.exe
    ClamXav may be able to tell you which particular virus or trojan it is, but do you care? In practice, there's seldom a reason to use ClamXav unless a network administrator requires you to run an anti-virus application.
    8. The greatest harm done by anti-virus software, in my opinion, is in its effect on human behavior. It does little or nothing to protect people from emerging threats, but they get a false sense of security from it, and then they may behave in ways that expose them to higher risk. Nothing can lessen the need for safe computing practices.
    9. It seems to be a common belief that the built-in Application Firewall acts as a barrier to infection, or prevents malware from functioning. It does neither. It blocks inbound connections to certain network services you're running, such as file sharing. It's disabled by default and you should leave it that way if you're behind a router on a private home or office network. Activate it only when you're on an untrusted network, for instance a public Wi-Fi hotspot, where you don't want to provide services. Disable any services you don't use in the Sharing preference pane. All are disabled by default.

  • About "arch1_31941_678636531.dbf" files

    Hi, All -
    Oracle 11g on RHEL5
    These physical arch files are filling up the space available resulting in archiver errors. Please let me know if there is a way to turn this off and should I turn this off. We do take regular backups and I am not sure the use of these. Please let me know... Thanks!
    Under $ORACLE_HOME/dbs
    -rw-r----- 1 oraedw oinstall 47807488 Jul 28 11:37 arch1_31941_678636531.dbf
    -rw-r----- 1 oraedw oinstall 48188928 Jul 28 11:37 arch1_31942_678636531.dbf
    -rw-r----- 1 oraedw oinstall 49212416 Jul 28 11:37 arch1_31943_678636531.dbf
    -rw-r----- 1 oraedw oinstall 47802368 Jul 28 11:37 arch1_31944_678636531.dbf
    -rw-r----- 1 oraedw oinstall 47801856 Jul 28 11:37 arch1_31945_678636531.dbf
    -rw-r----- 1 oraedw oinstall 47803392 Jul 28 11:37 arch1_31946_678636531.dbf
    -rw-r----- 1 oraedw oinstall 47804928 Jul 28 11:37 arch1_31947_678636531.dbf
    -rw-r----- 1 oraedw oinstall 47805440 Jul 28 11:38 arch1_31948_678636531.dbf
    -rw-r----- 1 oraedw oinstall 48787456 Jul 28 11:38 arch1_31949_678636531.dbf
    dallinux161 oraedw ccedw > pwd
    /orahome10/ccedwdb/11.1/dbs

    mbobak wrote:
    You say you're taking regular backups. How, exactly? RMAN? Conventional, non-RMAN? Hot? Cold?
    If you're taking a hot (aka "online") backup, the archive logs are aboslutely required for recovery. You MUST back them up, then remove them from the archive log area, to free space. This continual backup and removal of archive logs should be set up to run all day long, as needed, in between backups.Most people will allocate enough space to the archivelog destination to not require "continual" backup and removal. I find once a day, along with my database backup, to be sufficient. The most I've ever had to do was once every 6 hours.
    >
    If you're only taking cold (aka "offline") backups, then it depends on what your recovery point objective is. If you only care about recovering to the point in time that the backup was taken, then you don't need the archive logs, and can turn off archivelog mode on your database. If, however, you want to be able to recover up to the last committed transaction, then you need to remain in archivelog mode, and need to manage backups of the archive logs as I described in the previous paragraph. If that's the case, you should consider switching to hot backups, as there's no point in shutting down your database to take backups if you're in archivelog mode.
    Hope that helps,
    -Mark

  • Damaged iTunes File

    I have over 50gigs on my mac mini on my car and when it booted up today it said iTunes file was damaged and I clicked ok bc that was ONLY option then VIOLA NADA, no music etc... The music is still on iTunes folder but not showing in iTunes? How do I recover all this without importing it ALL OVER again?
    PLEASE SOMEONE HELP
    THANKS
    G
    http://homepage.mac.com/gwilchek/poweredbymac/PhotoAlbum37.html

    Not sure about playlists but I suspect they may be gone and will have to be recreated. There are two database files in the iTunes folder which contains that information. If you opened your library and it was empty, then those database files may have been rewritten without the playlists. With the actual music, it is still on the drive and you need to re-add it, but with the playlists, they may have been overwritten unfortunately.
    Which is why it is a good idea to back up those two database files occasionally somewhere outside the iTunes folder.
    Patrick

  • Confused about the log files

    I have written an application that has a Primary and Secondary database. The application creates tens-of-thousands of records in the Primary database, with a 1-to-1 relationship in the Secondary. On subsequent runs it will either update existing Primary records (which should not update the secondary as that element does not change) or it will create new records.
    The application actually works correctly, with the right data, the right updates and the right logical processing. The problem is the log files.
    The input data I am testing with is originally 2Mb as a CSV file and with a fresh database it creates almost 20Mb of data. This is about right for the way it splits the information up and indexes it. If I run the application again with exactly the same data, it should just update all the entries and create nothing new. My understanding is that the updated records will be written to the end of the logs, and the old ones in the earlier logs would be redundant and the cleaner thread would clean them up. I am explicitly cleaning as per the examples. The issue is that each run, the data just doubles in size! Logically it is fine, physically it is taking a ridiculous amount of space. RUnning DbSpace shows that the logs are mostly full (over 90%) where I would expect most to be empty, or sparsely occupied as the new updates are written to new files. cleanLog() does nothing. I am at a total loss!
    Generally the processing I am doing on the primary is looking up the key, if it is there updating the entry, if not creating one. I have been using a cursor to do this, and using the putCurrent() method for existing updates, and put() for new records. I have even tried using Database.delete() and the full put() in place of putCurrent() - but no difference (except it is slower).
    Please help - it is driving me nuts!

    Let me provide a little more context for the questions I was asking. If this doesn't lead us further into understanding your log situation, perhaps we should take this offline. When log cleaning doesn't occur, the basic questions are:
    a. is the application doing anything that prohibits log cleaning? (in your case, no)
    b. has the utilization level fallen to the point where log cleaning should occur? (not on the second run, but it should on following runs)
    c. does the log utilization level match what the application expects? (no, it doesn't match what you expect).
    1) Ran DbDump with and withour -r. I am expecting the
    data to stay consistent. So, after the first run it
    creates the data, and leaves 20mb in place, 3 log
    files near 100% used. After the second run it should
    update the records (which it does from the
    applications point of view) but I now have 40mb
    across 5 log files all near 100% usage.I think that it's accurate to say that both of us are not surprised that the second run (which updates data but does not change the number of records) creates a second 20MB of log, for a total of 40MB. What we do expect though, is that the utilization reported by DbSpace should fall to closer to 50%. Note that since JE's default minimum utilization level is 50%, we don't expect any automatic log cleaning even after the second run.
    Here's the sort of behavior we'd expect from JE if all the basics are taken care of (there are enough log files, there are no open txns, the application stays up long enough for the daemon to run, or the application does batch cleanLog calls itself, etc).
    run 1 - creates 20MB of log file, near 100% utilization, no log cleaning
    run 2 - updates every record, creates another 20MB of log file, utilization falls, maybe to around 60%. No log cleaning yet, because the utilization is still above the 50% threshold.
    run 3 - updates every record, creates another 20MB of log file, utilization falls below 50%, log cleaning starts running, either in the background by the daemon thread, or because the app calls Environment.cleanLog(), without any need to set je.cleaner.forceCleanFiles.
    So the question here is (c) from above -- you're saying that your DbSpace utilization level doesn't match what you believe your application is doing. There are three possible answers -- your application has a bug :-), or with secondaries and whatnot, JE is representing your data in a fashion you didn't expect, or JE's disk space utilization calculation is inaccurate.
    I suggested using DbDump -r as a first sanity check of what data your application holds. It will dump all the valid records in the environment (though not in key order, no -r is slower, but dumps in key order). Keys and data should up on different lines, so the number of lines in the dump files should be twice the number of records in the environment. You've done this already in your application, but this is an independent way of checking. It also makes it easier to see what portion of data is in primary versus secondary databases, because the data is dumped into per-database files. You could also load the data into a new, blank environment to look at it.
    I think asked you about the size of your records because a customer recently reported a JE disk utilization bug, which we are currently working on. It turns out that if your data records are very different in size (in this case, 4 orders of magnitude) and consistently only the larger or the smaller records are made obsolete, the utilization number gets out of whack. It doesn't really sound like your situation, because you're updating all your records, and they don't sound like they're that different in size. But nevertheless, here's a way of looking at what JE thinks your record sizes are. Run this command:
    java -jar je.jar DbPrintLog -h <envhome> -S
    and you'll see some output that talks about different types of log entries, and their sizes. Look at the lines that say LN and LN_TX at the top. These are data records. Do they match the sizes you expect? These lines do include JE's per-record headers. How large that is depends on whether your data is transactional or not. Non-transactional data records have a header of about 35 bytes, whereas transactional data records have 60 bytes added to them. If your data is small, that can be quite a large percentage. This is quite a lot more than for BDB (Core), partly because BDB (Core) doesn't have record level locking, and partly because we store a number of internal fields as 64 bit rather than 16 or 32 bit values.
    The line that's labelled "key/data" shows what portion JE thinks is the application's data. Note that DbPrintLog, unlike DbSpace, doesn't account for obsoleteness, so while you'll see a more detailed picture of what the records look like in the log, you may see more records than you expect.
    A last step we can take is to send you a development version of DbSpace that has a new feature to recalculate the utilization level. It runs more slowly than the vanilla DbSpace, but is a way of double checking the utilization level.
    In my first response, I suggested trying je.cleaner.forceCleanFiles just to make it clear that the cleaner will run, and to see if the problem is really around the question of what the utilization level should be. Setting that property lets the cleaner bypass the utilization trigger. If using it really reduced the size of your logs, it reinforces that your idea of what your application is doing is correct, and casts suspicion on the utilization calculation.
    So in summary, let's try these steps
    - use DbDump and DbPrintLog to double check the amount and size of your application data
    - make a table of runs, that shows the log size in bytes, number of log files, and the utilization level reported by DbSpace
    - run a je.cleaner.forceCleanFiles cleanLog loop on one of the logs that seems to have a high utilization level, and see how much it reduces to, and what the resulting utilization level is
    If it all points to JE, we'll probably take it offline, and ask for your test case.
    Regards,
    Linda

Maybe you are looking for