Does cleaning sort records and impact on max log size

Question

I'm looking for confirmation that the act of cleaning logs produces sorted records, equivalent to having inserted them in sorted order to begin with. If that is the case is the memory overhead of cleaning proportional to the maximum log file size, or will it spill to disk to sort? I ask since the max log file size is 1G, and if there is a memory component then it may be relevant to take into consideration.
On a similar note, is increasing the log file size from 10MB recommended if the database is large (say a TB)? Are there some linear operations (most likely in the admin/maintenance category) that have to iterate over each log file?
Thanks.

Answer 1

Hi,
I'm looking for confirmation that the act of cleaning logs produces sorted records, equivalent to having inserted them in sorted order to begin with.No, that's not exactly true. When records are written (migrated to the end of the log) during the log cleaning process, any clustering (writing of adjacent records in key order) occurs on a per BIN (bottom internal node) basis. So at most 128 records (or DatabaseConfig.setNodeMaxEntries) with adjacent key values are sorted and written out together in this manner. But normally not all records in a BIN will be migrated and I'll describe more about this below.
If that is the case is the memory overhead of cleaning proportional to the maximum log file size, or will it spill to disk to sort? I ask since the max log file size is 1G, and if there is a memory component then it may be relevant to take into consideration.No, the memory used for sorting in this case is the JE cache itself -- the memory that holds the Btree. The log cleaner thread moves records to be migrated into the Btree (in the cache), and flags them for migration. These records are written lazily the next time their parent Btree node (the BIN I mentioned) needs to be flushed. This can occur for two reasons: 1) when the BIN is evicted from the cache because the cache is full, or 2) during the next checkpoint, since a checkpoint will flush all dirty BINs. The most desirable situation is that the BIN is flushed during the checkpoint, not during eviction, to reduce the number of times that a single BIN is flushed to disk. If a BIN is evicted, it may have to be fetched again later in order to migrate more records as part of the log cleaning process, and this is counterproductive.
So during this process, the number of records that are written to disk in key order in a cluster, such that they are physically adjacent on disk, is variable. It depends on how many records are processed by cleaner threads, and inserted in their parent BIN for migration, between checkpoint intervals, and whether the BINs are evicted during this period.
A physical cluster of records includes the BIN itself as well as its child records that are migrated for log cleaning. The fact that the BIN is part of the cluster is beneficial to apps where not all BINs fit in the cache. If a record and its parent BIN are not in cache, and the app reads that record, the BIN and the record must both be fetched from disk. Since they will be in physical proximity to each other on disk, disk head movement is avoided or reduced.
However, not all of the child records in a BIN will necessarily be part of a physical cluster. If only some of the records in a BIN were processed by the log cleaner before the BIN is flushed by the checkpointer or evictor, then only a subset of the child records will be written as part of the cluster. In other words, even for a single BIN, all the child records in that BIN are not normally written out together in a cluster.
Clustering is a useful feature and something we would like to enhance in the future. What I've described above is as far as we've gotten with this feature in the currently released and supported product. However, there are two experimental configuration settings that cause additional clustering to occur. These cause additional records to be written to disk when their parent BIN is flushed. If you're interested in trying these, I will be happy to explain them and work with you. But you should be aware that they're not currently in the standard, supported set of features, and are not well tested.
I'm not sure whether I've given you less information than you need or more than you wanted. :-) If you describe in detail what you're trying to do, and more about your app and performance requirements, I may be able to offer more concrete advice.
On a similar note, is increasing the log file size from 10MB recommended if the database is large (say a TB)? Are there some linear operations (most likely in the admin/maintenance category) that have to iterate over each log file?Yes, there are such operations and per-log file overhead, and we recommend using using a 20M to 50M log file size for large data sets. Log files larger than 50MB could decrease log cleaner performance.
--mark