Change Management strategies?

Question

Would appreciate any shared experiences on how people manage the orderly running of developer scripts.
At the moment, I get told to check a script out of SVN and run it. Fine -except there's no record in the database of what was run, or what I had to do to get it to run successfully. For example, I might have a script which creates a new package, drops a table or two, adds an index then does five hundred inserts; one line of which might not mention a table in the full schema-dot-name notation, so it perhaps erros out with a 'object doesn't exist' problem. So I add the schema name to get the job finished. Now what I ran isn't 100% what's in SVN, and so on. Similarly, I add a header to the script so that it spools output and so on, but that's not in the SVN either, and maybe a different DBA will add a different header, spooling to a different location, and so on.
Another issue arose just today: which scripts have been run on a particular database? We can only tell by looking at the output spools -but who knows if one of the spool files was moved, spooled to the wrong place, deleted etc. SVN will only tell you that scripts 1 to 15 exist or were, at least, committed to the SVN repository; it won't tell you that 13 was run, but 14 was missed, etc.
It seems to me that if the developers use SVN when developing, that's fine. But as a DBA, I'd like what gets run to be stored in a database table, and the results of running it to be similarly captured in a CLOB somewhere... but how does everyone else manage this stuff?!

Answer 1

>
But I was asking about how DBAs deal with the fact that sometimes, because we live in an imperfect world, the DBA has to tweak things a bit to get them to run. You say that 'of necessity' there will be different scripts for the different environments; if I need to get to that state of affairs, that's something I'd take on board, for sure. But it sounds complicated to manage.
>
It can be complicated to manage.
And it is important to distinguish between at least two types of 'tweaks'. If a DBA needs to tweak something that is missing but should have been there (e.g. a privilege unrelated to the new release being installed) then tweak it. There isn't really anything more to do since it should not have been missing to begin with and, presumably, if the DB were rebuilt from scripts that privilege would be there. It may be that someone did a 'tweak' and removed that privilege at some earlier time.
The second kind of tweak is to DDL or DML that is part of the new release being installed. Again, that could be a privilege that isn't there. But in this case the privilege never was there; either the developer or a DBA missed discovering that the privilege was needed. Sometimes this occurs when the testing is done by someone with more privileges than the user that will actually be using the app.
For those 'tweaks' the DDL for the missing privileged definitely needs to be added to version control so that it will be available if the code for that release is needed again and to document what was really installed.
>
There isn't just ONE set of scripts that will work in ALL environments
So, you're saying that your devs will write three different scripts to be run in dev, testing and prod? Fine if so. If that's how you manage it, that's what I'm interested in knowing about. My devs don't do that, though. The assumption is that if it works in one, it will work in the other. Indeed, I believe one of my responsibilities is to make that true (by not granting permissions in one that I don't in the others, for example). But I can spot a losing battle when I see one.
>
Well a good example of that is the one I used regarding DB links to remote systems. In dev and even test environments you certainly aren't going to have DB links that point to a remote production system.
So someone has to modify the code to alter the DB links to point to the appropriate system. Sometimes you can mitigate that by using the same DB link name but in legacy systems you can't always change the link names that have existed for years.
In orgs I work with it is the developer responsible for the code that makes the required changes. After testing things in DEV and checking the code into the DEV branch of version control they check out the code again, modify the DB links and check it back in to the TEST/QA/PROD code branches in version control.
And you are right, that is where it get's sticky when you have more than one developer working on a package. They have to make sure that only ONE of them has the 'official' version of any given package checked out.
>
Again, I think the "semi-duplicate" word might be the all-important one.
>
The DB link example I just discussed is what I mean. A developer making changes to one DEV package function might be required to also update the DB Links in that DEV package to the appropriate values for TEST/QA/PROD also.
>
What I was suggesting was that what's in SVN should be considered the 'master idea'. I check it out, I then make it runnable, without altering any code "significantly" (and I know that's a weasel-word). I would like a record of what I actually ran. And I'd like it stored in the (or "a") database, together with its results. Or maybe not: you're saying I should check what I actually ran back into SVN?
>
Yes - what is in SVN IS CONSIDERED the 'master'. But SVN has a different code branch for DEV/TEST/QA/PROD so each of those branches will have the 'master' for its environment.
Again consider the example where the DB links are different for each of the four environments and a developer is adding a 'fix' to one package function:
1. exclusive check out of current package code from DEV branch in SVN
2. make and test fix in dev
3. check in updated code to DEV branch in SVN
4. exclusive check out of current package code from TEST branch in SVN
5. update new DEV package code to alter the DB links as needed.
6. test fix in TEST
7. check in updated code to TESTS branch in SVN
n. continue above as you migrate from DEV to TEST to QA to PROD
A modification of the above is along the lines of always using the current SVN code for PROD as a starting point and altering its DB links to what is needed in DEV.
>
And, again, I wasn't suggesting they could or should, but merely how other DBAs manage this sort of situation. It's honestly just a question about how DBAs manage change/code management in a multi-environment setting.
What I think you're saying is: check out the developer's code from SVN. Do what you have to do to make it work in three or four different environments, and commit each changed version back to SVN. Is that how you do it?
>
Yes - but that's not the worst of it. In this age of Agile and Scrum and all of that there are usually multiple release cycles going on at the same time and you need to use the code base (SVN code branch) for the release cycle PRIOR to the one you are working on.
If the PRIOR cycle code changes (hey -those people are still working on it and it isn't release yet) you have to sync up with it before you can finish yours.
This more or less sums up what you have to try to deal with:
1. SVN has multiple TRUNKS (e.g. a trunk for each release)
2. Each SVN trunk has multiple branches (at least one for each environment)
3. Some objects may have multiple developers working on them simultaneously for the same release. Two developers doing bug fixes on two different functions in the same package.
4. Different environments may be at a different release cycle.
A. PROD has what is being used now: release 3.4
B. The next PROD upgrade, 3.5 may currently be in the QA environment ungoing final testing.
C. The second release, 3.6 may be in TEST with some of those developers also still working in DEV to address any issues discovered in the TEST environment.
D. The third release, 3.7 will be in DEV and some of those developers willl sometimes have conflicts with developers trying to finalize the 3.6 release - the 3.6 developers have priority.
When a release reaches QA the 'tweaks' had better be few and far between and, if minor, are just tweaked so they work and the 'tweaks' updated to the appropriate SVN TRUNK and branch.