SAP CAL HANA errors

Question

SAP HANA, developer edition 1.0 SPS 09 created from SAP CAL running on AWS.
I am receiving the following errors when executing a query in HANA via jdbc.
The first execution runs for 6 minutes then produces this error:
Caused by: com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [2]: general error: Allocation failed $REASON$
Caused by: com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [2048]: column store error:  [9] Memory allocation failed
I re execute the query it runs for 25 minutes then produces the same error.
Then the instance is unresponsive.
If I look in HANA studio general overview there is plenty of memory and cpu available to the HANA instance.
There is also a crush dump listed in the log files.
I see the following errors in the indexserver_alert_vhcalhdbdb.trc on the HANA server:
[3742]{-1}[-1/-1] 2015-03-18 18:41:42.430797 e Licensing   
LicenseMeasurementPersistence.cpp(00161) : Error storing license measurement for application id '000670', value 26 at ' 2015-03-18 18:41:39.3910000', internal error
[3742]{-1}[-1/-1] 2015-03-18 18:41:42.430999 e Licensing   
LicensePluginRegistry.cpp(00394) : Error storing license measurement: internal error
[3732]{300601}[27/-1] 2015-03-18 18:30:16.406867 e Executor    
PlanExecutor.cpp(00825) : pop1245 (rc 9, Memory allocation failed)
pop = RowPlanOperator() # pop1245
pop.setDetails('')
[3732]{300601}[27/-1] 2015-03-18 18:30:16.406890 e Executor    
PlanExecutor.cpp(00825) : pop1243 (rc 9, Memory allocation failed)
[4437]{300601}[27/-1] 2015-03-18 18:28:13.921145 e Memory      
ReportMemoryProblems.cpp(00799) :  8: 0x00007f0b1ba7a8c5 in ptime::TrexPlanOp::executePtimeOp(ltt_adp::vector<Executor::PlanData*, ltt::integral_constant<bool, true> > const&, ltt_adp::vector<Executor::PlanData*, ltt::integral_constant<bool, true> > const&, TRexCommonObjects::TRexApiError&, Executor::ExecutionInfo const&)+0x41 at trex_plan.cc:340 (libhdbcswrapper.so)
The memory errors are repeated many times.
What is the recommendation to investigate this issue?
Thank you.

Answer 1

The file indexserver_alert_vhcalhdbdb.trc contains the following errors:
[4175]{300560}[23/-1] 2015-03-19 10:57:26.017346 e Executor    
X2.cpp(02961) :  1: 0x00007f049f94df0a in _ZN5ptime3pcc16MonitorAllocBase14throwExceptionEv.isra.53+0x22 at pcc_MonitorAlloc.h:61 (libhdbrskernel.so)
[4175]{300560}[23/-1] 2015-03-19 10:57:26.017346 e Executor    
X2.cpp(02961) :  2: 0x00007f049f9403f5 in ptime::Transaction::Transaction(ptime::Connection*, bool, bool, bool, bool)+0x18f1 at pcc_MonitorAlloc.h:114 (libhdbrskernel.so)
[4175]{300560}[23/-1] 2015-03-19 10:57:26.017346 e Executor    
X2.cpp(02961) :  3: 0x00007f049ca15949 in ptime::TrexTempIndex::clearIndex() const+0x25 at trex_itab.cc:3020 (libhdbcswrapper.so)
[4175]{300560}[23/-1] 2015-03-19 10:57:26.017346 e Executor    
X2.cpp(02961) :  4: 0x00007f049eb7bd74 in ptime::TrexObjectFactory::destroyTrexTempIndexObject(char const*, char const*, char const*)+0x40 at trex_object_factory.cc:1386 (libhdbrskernel.so)
[4175]{300560}[23/-1] 2015-03-19 10:57:26.017346 e Executor    
X2.cpp(02961) :  5: 0x00007f049ca4e706 in ptime::TrexPlanData::~TrexPlanData()+0x1d2 at trex_plan_data.cc:49 (libhdbcswrapper.so)
6: 0x00007f049ca4e849 in ptime::TrexPlanData::~TrexPlanData()+0x5 at trex_plan_data.cc:53 (libhdbcswrapper.so)
7: 0x00007f04c44739ed in Executor::X2ConnectorValue::deleteItsValue()+0x29 at X2.cpp:323 (libhdbexecutor.so)
8: 0x00007f04c4488101 in Executor::X2::failPop(int)+0xb0 at X2.h:176 (libhdbexecutor.so)
9: 0x00007f04c44879b4 in Executor::X2::finishValue(Executor::X2ConnectorValue&, int&, bool)+0x110 at X2.cpp:920 (libhdbexecutor.so)
10: 0x00007f04c4488163 in Executor::X2::failPop(int)+0x110 at X2.cpp:3571 (libhdbexecutor.so)
11: 0x00007f04c44879b4 in Executor::X2::finishValue(Executor::X2ConnectorValue&, int&, bool)+0x110 at X2.cpp:920 (libhdbexecutor.so)
12: 0x00007f04c4488163 in Executor::X2::failPop(int)+0x110 at X2.cpp:3571 (libhdbexecutor.so)
13: 0x00007f04c44938d0 in Executor::X2::runPopTask(Executor::X2::PopTaskInfo&, int&, ltt::allocator&, ltt::allocator&)+0x1290 at X2.cpp:3548 (libhdbexecutor.so)
14: 0x00007f04c4494e32 in Executor::X2::runPopJob(Executor::X2Job*)+0x80 at X2.cpp:2956 (libhdbexecutor.so)
15: 0x00007f04c44956b8 in Executor::X2Job::run(Execution::JobObject&)+0x224 at X2.cpp:5294 (libhdbexecutor.so)
16: 0x00007f04ab3ec1fe in Execution::JobObjectImpl::run(Execution::JobWorker*)+0x72a at JobExecutorImpl.cpp:822 (libhdbbasis.so)
17: 0x00007f04ab3fadd2 in Execution::JobWorker::runJob(ltt::smartptr_handle<Execution::JobObjectForHandle>&)+0x2e0 at JobExecutorThreads.cpp:205 (libhdbbasis.so)
18: 0x00007f04ab3fbd24 in Execution::JobWorker::run(void*&)+0x1a0 at JobExecutorThreads.cpp:370 (libhdbbasis.so)
19: 0x00007f04ab433759 in Execution::Thread::staticMainImp(void**)+0x875 at Thread.cpp:492 (libhdbbasis.so)
[4175]{300560}[23/-1] 2015-03-19 10:57:26.017346 e Executor    
X2.cpp(02961) : 20: 0x00007f04ab4343bd in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)
[4190]{300560}[23/-1] 2015-03-19 10:57:26.576080 e Memory      
ReportMemoryProblems.cpp(00799) : OUT OF MEMORY occurred.
Host: sid-hdb
Executable: hdbindexserver
PID: 3599
Failed to allocate 131088 byte.
Current callstack:
1: 0x00007f04ab5a07fd in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0x2a9 at PoolAllocator.cpp:1139 (libhdbbasis.so)
2: 0x00007f049dcfca79 in ptime::pcc::SingletonAlloc<ptime::pcc::MonitorAlloc<ptime::pcc::HeapAlloc>, ptime::pcc::GlobalScope>::allocate(unsigned long)+0x15 at pcc_MonitorAlloc.h:97 (libhdbrskernel.so)
3: 0x00007f049e095b50 in ptime::fastvector<void*, 16ul>::__allocate(unsigned long)+0x50 at fastvector.h:325 (libhdbrskernel.so)