System.InsufficientMemoryException: Failed to allocate a managed memory buffer of 268435456 bytes. The amount of available memory may be low. --- System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

Question

Appfabric 1.1 server setup on 3 Windows server 2008 R2 machines.
Client windows 7 64 bit.
Because there is no bulkupdate ,we are trying to persist around 4000 objects wrapped in a CLR object.
 [Serializable]
    public class CacheableCollection<T> : ICacheable, IEnumerable<T>
        where T : class, ICacheable
        [DataMember]
        private Dictionary<string, T> _items;
        public IEnumerator<T> GetEnumerator()
            return _items.Values.GetEnumerator();
        IEnumerator IEnumerable.GetEnumerator()
            return _items.Values.GetEnumerator();
        [DataMember]
        public string CacheKey { get; private set; }
        public T this[string cacheKey] { get { return _items[cacheKey]; } }
        public CacheableCollection(string cacheKey, T[] items)
            if (string.IsNullOrWhiteSpace(cacheKey))
                throw new ArgumentNullException("cacheKey", "Cache key not specified.");
            if (items == null || items.Length == 0)
                throw new ArgumentNullException("items", "Collection items not specified.");
            this.CacheKey = cacheKey;
            _items = items.ToDictionary(p => p.CacheKey, p => p);
We tried with the following options on server and client
Server:
 <advancedProperties>
            <partitionStoreConnectionSettings leadHostManagement="false" />
            <securityProperties mode="None" protectionLevel="None">
                <authorization>
                    <allow users="[email protected]" />
                    <allow users="[email protected]" />
                </authorization>
            </securityProperties>
            <transportProperties maxBufferSize="500000000" />
        </advancedProperties>
Client: 
 <transportProperties connectionBufferSize="131072" maxBufferPoolSize="500000000"
                       maxBufferSize="838860800" maxOutputDelay="2" channelInitializationTimeout="60000"
                       receiveTimeout="600000"/>
I see different people experiencing different memory size issues. What is the actual memory limit of an  object that can be pushed to Appfabric. 
Can some one please help ?
Stack trace:
Test method Anz.Cre.Pdc.Bootstrapper.Test.LoaderFuncCAOTest.AppFabPushAndRetrieveData threw exception: 
System.InsufficientMemoryException: Failed to allocate a managed memory buffer of 268435456 bytes. The amount of available memory may be low. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
System.Runtime.Fx.AllocateByteArray(Int32 size)
System.Runtime.Fx.AllocateByteArray(Int32 size)
System.Runtime.InternalBufferManager.PooledBufferManager.TakeBuffer(Int32 bufferSize)
System.Runtime.BufferedOutputStream.ToArray(Int32& bufferSize)
System.ServiceModel.Channels.BufferedMessageWriter.WriteMessage(Message message, BufferManager bufferManager, Int32 initialOffset, Int32 maxSizeQuota)
System.ServiceModel.Channels.BinaryMessageEncoderFactory.BinaryMessageEncoder.WriteMessage(Message message, Int32 maxMessageSize, BufferManager bufferManager, Int32 messageOffset)
System.ServiceModel.Channels.FramingDuplexSessionChannel.EncodeMessage(Message message)
System.ServiceModel.Channels.FramingDuplexSessionChannel.OnSendCore(Message message, TimeSpan timeout)
System.ServiceModel.Channels.TransportDuplexSessionChannel.OnSend(Message message, TimeSpan timeout)
System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout)
Microsoft.ApplicationServer.Caching.WcfClientChannel.SendMessage(EndpointID endpoint, Message message, TimeSpan timeout, WaitCallback callback, Object state, Boolean async)
Microsoft.ApplicationServer.Caching.WcfClientChannel.Send(EndpointID endpoint, Message message, TimeSpan timeout)
Microsoft.ApplicationServer.Caching.WcfClientChannel.Send(EndpointID endpoint, Message message)
Microsoft.ApplicationServer.Caching.DRM.SendRequest(EndpointID address, RequestBody request)
Microsoft.ApplicationServer.Caching.RequestBody.Send()
Microsoft.ApplicationServer.Caching.DRM.SendToDestination(RequestBody request, Boolean recordRequest)
Microsoft.ApplicationServer.Caching.DRM.ProcessRequest(RequestBody request, Boolean recordRequest)
Microsoft.ApplicationServer.Caching.DRM.ProcessRequest(RequestBody request, Object session)
Microsoft.ApplicationServer.Caching.RoutingClient.SendMsgAndWait(RequestBody reqMsg)
Microsoft.ApplicationServer.Caching.DataCache.SendReceive(RequestBody reqMsg)
Microsoft.ApplicationServer.Caching.DataCache.ExecuteAPI(RequestBody reqMsg)
Microsoft.ApplicationServer.Caching.DataCache.InternalPut(String key, Object value, DataCacheItemVersion oldVersion, TimeSpan timeout, DataCacheTag[] tags, String region)
Microsoft.ApplicationServer.Caching.DataCache.Put(String key, Object value, String region)
Anz.Cre.Pdc.DataCache.DataCacheAccess.Put[T](String cacheName, String regionName, T value) in C:\SVN\2.3_Drop3\app\Src\Anz.Cre.Pdc.DataCache\DataCacheAccess.cs: line 141
Anz.Cre.Pdc.DataCache.DataCacheAccess.Put[T](String cacheName, String regionName, Boolean flushRegion, T value) in C:\SVN\2.3_Drop3\app\Src\Anz.Cre.Pdc.DataCache\DataCacheAccess.cs: line 372
Anz.Cre.Pdc.Bootstrapper.Test.LoaderFuncCAOTest.AppFabPushAndRetrieveData() in C:\SVN\2.3_Drop3\app\Src\Anz.Cre.Pdc.Bootstrapper.Test\LoaderFuncCAOTest.cs: line 281

Answer 1

Essentially what we are trying to do is the following.
we have different kinds of objects in our baseline. Objects of type CAO, Exposures, Limits that change everyday after close of business day. 
We wanted to push these different objects in to respective named regions in the cache. 
Region Name     Objects
CAO                   ienumerable<caos>
Exposures           ienumerable<exposures>
Limits                ienumerable<limits>
we have a producer that pushes this data in to the cache and consumers of this data acting on the data when its available.
Now the issue we are facing is when we try to push around 4000 cao objects (roughly in the size of 300MB when serialized using xml) ,we are getting the above error. Increasing the size on the client and cache cluster didnt help.
The other alternative we were thinking about is chunking and pushing because appfabric doesnt support streaming. We might be able to push this data successfuly if we chunk. But how about the consumers ? wouldnt they face the same memory issue when we use
getallobjectsinregion ?
We thought if there was a way to figure out the keys in the region then probably the consumers can get one by one. However there is no such API. 
The only option i see is using Appfabric notifications which msdn says isnt a reliable way.
Please help.