Disk full event although disk is not full

Question

Hello,
we have a HANA database running with revision 62.
After the filesystem for the log backup was not available (fileserver restart and mount did not reconnect) the database log went full.
The nameserver now cannot start and tells:
[11282]{-1}[-1/-1] 2013-08-24 00:43:07.470082 i EventHandler     EventManagerImpl.cpp(00647) : handleEvents(1)
[11282]{-1}[-1/-1] 2013-08-24 00:43:07.470119 i EventHandler     EventManagerImpl.cpp(00650) : --handleEvent: DiskFullEvent[id= 1, path= /sapmnt/log/HA1/mnt00001/hdb00001/, state= NEW]
[11282]{-1}[-1/-1] 2013-08-24 00:43:07.470156 i EventHandler     LocalFileCallback.cpp(00437) : [DISKFULL] restarting queue with 1 requests
[11282]{-1}[-1/-1] 2013-08-24 00:43:07.471884 i EventHandler     EventManagerImpl.cpp(00653) : --event not handled: DiskFullEvent[id= 1, path= /sapmnt/log/HA1/mnt00001/hdb00001/, state= HANDLING]
But the filesystem is not full.
df tells me:
df -h /sapmnt/log/HA1/mnt00001/hdb00001/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sapmntdata       5.8T  947G  4.9T  17% /sapmnt
I already checked the note
Note 1679938 - Log Volume is full (log mode = legacy)
But this does not work for me, as the filesystem is not full.
Is there a chance to bring the system up ?
Do I need to change a parameter ?
Many thanks in advance.
Any help is really appreciated.
Andreas

Answer 1

Hi Domenico,
yes I found a solution.
The problem was that there was a disk full indeed.
But it could not be seen by standard unix df command as the HANA logs are on a GPFS fileset.
This GPFS fileset was full.
You can check this with the command "mmrepquota"
mmrepquota --block-size G -a
this gives an overview of all filesets
*** Report for USR GRP FILESET  quotas on sapmntdata
                         Block Limits                                    |
Name       type             GB      quota      limit   in_doubt    grace |
root       USR             203          0          0          1     none |
ha1adm     USR              77          0          0          1     none |
ha2adm     USR             137          0          0          1     none |
ha3adm     USR              47          0          0          1     none |
ha4adm     USR             295          0          0          1     none |
ha5adm     USR              86          0          0          1     none |
110        USR               1          0          0          0     none |
root       GRP             198          0          0          1     none |
sapinst    GRP               2          0          0          0     none |
sapsys     GRP             643          0          0          1     none |
root       FILESET         499          0          0          1     none |
hanadata   FILESET         150        768        768          1     none |
hanalog    FILESET         193        256        256          1     none |
In the column "GB" where actually the value is 193 was the value 256 which is exactly the quota of this fileset. So not the filesystem was full but the fileset in there.
As You can see we are running more than one HANA DB on this server. That might be the reason why the hanalog fileset was full very quickly.
The solution was to run the following statement on each of the DB's:
ALTER SYSTEM RECLAIM LOG;
After that the filling went down to 22 GB which is far away from the quota.
We now check regularly that we do not reach the quota.
To solve the problem final we are planning to patch SLES to SP2 or at least do an upgrade on GPFS because the shown behaviour is a bug in our GPFS revision.
hope that helps
Andreas