Obsolete jdb not being cleaned up

Question

Hi,
Setup:
* We are using Oracle NoSQL 1.2.123.
* We have 3 replication groups with 3 replication nodes each.
Problem:
* 2 of the slaves (in 2 different replication groups) occupy much more space in JDB files (10 times more) then all the others. As these are slaves, and writes always go through the master, and all nodes in a replication group have the same data (eventually), I assume that this is stale data that has not been cleaned up by the BDB garbage collection (cleaner threads). Unfortunately the logs do not show anything new (since Dec. last year) and the oldest JDB files are from February.
Questions:
* Any ideas what could have gone wrong?
* What can I do to trigger the cleaners to cleanup the old data? Is that safe to do in production environment and without downtime?
* Is it really safe to assume that the current data in within a replication groups is really the same?
Thank you in advance
Dimo
PS. A thread dump shows 2 cleaner threads that do nothing.

Answer 1

1) The simplest and fastest way to correct the replica node is to restore it from the master node. We will send you instructions for doing this later today.Here are directions for refreshing the data storage files (.jdb files) on a target node. NoSQL DB will automatically refresh the storage files from another node, after we manually stop the target node, delete its storage files, and finally restart it, as described below. Thanks to Linda Lee for these directions.
First, be sure to make a backup.
Suppose you want to remove the storage files from rg1-rn3 and make it refresh its files from rg1-rn1. First check where the storage files for the target replication node are located using the show topology command to the Admin CLI. Start the AdminCLI this way:
    java -jar KVHOME/lib/kvstore.jar runadmin -host <host> -port <port>Find the directory containing the target Replication Node's files.
    kv-> show topology -verbose
    store=mystore  numPartitions=100 sequence=108
      dc=[dc1] name=MyDC repFactor=3
      sn=[sn1]  dc=dc1 localhost:13100 capacity=1 RUNNING
        [rg1-rn1] RUNNING  c:/linda/work/smoke/KVRT1/dirB
                     single-op avg latency=0.0 ms   multi-op avg latency=0.67391676 ms
      sn=[sn2]  dc=dc1 localhost:13200 capacity=1 RUNNING
        [rg1-rn2] RUNNING  c:/linda/work/smoke/KVRT2/dirA
                  No performance info available
      sn=[sn3]  dc=dc1 localhost:13300 capacity=1 RUNNING
        [rg1-rn3] RUNNING  c:/linda/work/smoke/KVRT3/dirA
                     single-op avg latency=0.0 ms   multi-op avg latency=0.53694165 ms
      shard=[rg1] num partitions=100
        [rg1-rn1] sn=sn1 haPort=localhost:13111
        [rg1-rn2] sn=sn2 haPort=localhost:13210
        [rg1-rn3] sn=sn3 haPort=localhost:13310
        partitions=1-100In this example, rg1-rn3's storage is located in
    c:/linda/work/smoke/KVRT3/dirAStop the target service using the stop-service command
    kv-> plan stop-service -service rg1-rn3 -waitIn another command shell, remove the files for the target Replication Node
    rm c:/linda/work/smoke/KVRT3/dirA/rg1-rn3/env/*.jdbIn the Admin CLI, restart the service
     plan start-service -service rg1-rn3 -waitThe service will restart, and will populate its missing files from one of the other two nodes in the shard. You can use the "verify" or the "show topology" command to check on he status of the store.
--mark