SSIS / SQL Memory Contention

Question

Hi all,
I'm running SSIS and SQL server on a single PC. 
I have an ETL process consisting of several SSIS packages to load data from a source OLTP to a data warehouse.
OLTP DB resides on SQL 2012 instance.
Datawarehouse resides on SQL 2008R2.
My SSIS packages 'hangs' and reports it is failing to swap out memory buffers - they're all held by the SQL 2012 (source db) instance.
On my PC I've configured max memory on each SQL instance:
PC Memory = 16GB
SQL 2012 Max Memory = 6GB
SQL 2008r2 Max Memory = 4GB
My ETL package now runs smoothly. 
My question is - what is best practice for managing this memory contention in a production environment. I do not want to limit the available memory to my OLTP db. 
My ETL is ran once per day  - overnight.  Perhaps I could reconfigure max memory before and after each ETL processs? But I don't like this as it would require a server restart & cold cache for my OLTP apps.
Thanks for reading. Appreciate suggestions.
Clay

Answer 1

OK, It is more clear now :-)
This is  a developing server, and I assume that you have to hold several instances in order to re-create an environment that is like the production's environment  (I hope this is the reason, otherwise you need a very good reason for holding several
instances on one machine, which has resources problem, since each instance need lot of resources only to keep it running and most time it is better to have one instance with all the databases, than several instaces on the same machine). I actually have the
same problem on my developing environment by the way (I have about 8 instances of different database servers). Everything hat i write from now is only recommend for this situation for developing environment!
The following points should be taken into account and act according:
This is a small list that can take 2 books if we have time... this is just some point to think about.
* Remember that SQL Server is trying to use as much memory as you let him in order to execute more efficience, It dose not take into direct consideration of other instances or application. 
* In order to understand what need to be done you need to monitor what specificly use all the memory.
    ** look at sys.dm_os_memory_clerks
    ** look for DBCC MEMORYSTATUS
    ** look at this link: http://www.mssqltips.com/sqlservertip/2304/how-to-identify-microsoft-sql-server-memory-bottlenecks/
    ** It is good time for Google https://www.google.co.il/search?q=SQL+Server+monitoring+memory
* Once you found the cause and source for the memory use, you can focus on solving and start reduce memory usage before you start the ETL procces (and mabe during as well).
    **  look for DBCC FREEPROCCACHE 
    ** look for  DBCC DROPCLEANBUFFERS
    ** We can build a better query sometimes
    ** we can pass some of the operation to the disk instead of the RAM
    ** Setting the Memory Options (as you did control max server memory,
check this link for more info)
    ** look for SET AUTO_CLOSE ON (This is usually not recommended but in your case using several instances this can help a lot sometimes)
* The ETL proccess do not use only SQL Server but the application that execute it as well. The SQL Server will see the queries. While SQL Server is great in menaging multiple users and multiple paralel actions (or example using locks), some ETL are not so
good. You need to make sure if the SQL Server return an error or the failed come from the ETL proccess it self. Please post the full error message for mor specific information.
* Remember that some actions in SQL Server use memory out of the Buffer pool memory and some action are not part of the max server memory (for example some CLR used).
as I mentioned i can continue this documents for days :-)
I hope this is helpfull for you as it is :-), and if you have more information (like error messages from log files) we might focuse on other option.
[Personal Site] [Blog] [Facebook]