[Java and all else] Requirements tracking, testing, and versioning

Question

Hello,
as a semi-experimental subject, I am considering managing requirements and test documents in the same version control system as the code.
For some reasons, this versioning policy doesn't seem mainstream, so I'd like to have your opinion on that.
+N.B.: the prefix "java and all else" refers to the fact that I'm using this area to post a message that is related not only to Java but also to other topics (people, processes, tools,...) that help program in Java - so this post is not really off-topic either. For the record, I filed a suggestion to open a dedicated forums to discuss topics like that, although it hasn't received enthusiastic support so far :o(...+
Here is how I had imagined to proceed:
- requirements document gets edited and stored as text in the VCS
- test plan (basically a matrix of which tests cover which requirements) goes into the VCS as well, text or binary
- test instructions (detailed steps, actions, expected results) are stored as text in the VCS
- test results - I do'nt know yet, but I planned to store them as text (lists of testId=result) in the VCS (I'm not sure indeed, that those would go into the VCS)
The advantage of versioning those as text is that all VCS clients support easy-as-a-click history browsing and diff capabilities.
Diffs and logs are useful to trace when and why a requirement has appeared, evolved, obsolated,... Especially in long-lived systems whose maintenance crew rarely embeds "historical" pioneers who developed the original systems.
My questions:
1) Is it good practice to version such docs in the same repository as the code?
I've never seen this done.
- In half of the shops I've been these docs are not really versioned (the PM considered it enough that the latest applicable version of the docs was available)
- The other half has used versioning systems ranging from ad-hoc (shared folder acting as repository of all versions) to "document management systems". I found such tools to be developer-unfriendly (poor diff capability, poor text searching), and developers to be office-document-reluctant (mostly because of the shortcomings of the doc format and doc management systems).
- The third half <wink to aficionados of French cinema> used dedicated softwares (Test Director, RQM,...) to manage requiremetns, test plans, test cases, test results,... The drawback is that the developers were hardly involved in using such tools, and even when they were, were not trained to use the administrative and metadata capabilities (who modified the test steps, when and why?).
On the contrary, developers would easily and gladly browse histiry, check diffs, etc... on text documents versioned using their daily-used source control system.
2) Are there "standard", preferrably text-based, formats, for the documents listed above?
Each and every organization I've seen had its own format, but most items have equivalents in all organizations, even if disguised under different terminologies. I have no formal education or training in quality assurance, but I guess some ISO or CMM norms define those recurring items, and I expect some abstract XML format might exist
3) Are there tools already that cover edition and exploitation of such formats, or why not, of the versioning of such formats (naive example, graphing the pass rate of tests over time)?
A pet project of mine is to develop and open-source some Swing UI to do this, but existing tools would save me the trouble.
I'm looking forward to reading about your own experience.
Best regards,
Jerome.
Edited by: jduprez on Jun 4, 2010 10:33 PM
Fixed link.

Answer 1

I don't have data on comparative bug rates because we did not do studies focused on that. This stuff was mostly driven by convenience #) considerations.
Subjectively, yes it seemed to have lower bug rate compared to maintaining documents in, say, pdf format (like our doc writers did with User/Dev guides). BTW doc writers are another kind of guys who dislike plain text.
As for using test results history the way you describe - agree, that makes some sense. Now that you mention it I recall using it myself that way, in one of the projects where testers managed to establish really convenient way to assess test execution history. After I figured how easy it was, I began using it. Quite helpful indeed.
Regarding "tool or format that mapped tests to specific requirements" in your original post - did you consider Polarion? IIRC it supports mapping requirements to test descriptions. Telelogic DOORS are maybe also capable of that but I am not completely certain.
#) convenience -- BTW we maintained test descriptions not in plain text but in simple HTML (you know few tags one can use in javadocs). With limited line length, it is almost as VCS friendly as plain text, even with tables:
<!DOCTYPE "stuff to make HTML validator happy">
<html><title>test description</title>
<body>
<table>
  <tr>
    <td>test id</td>
    <td>synchronized.10988343</td>
  </tr>
  <tr>
    <td>instructions</td>
    <td>
      <ul>
        <li>sacrifice a goat
          <ul><li>under the full moon</li></ul>
        </li>
        <li>twirl counterclockwise
          <ul><li>until you barf</li></ul>
        </li>
    </td>
  </tr>
  <tr>
    <td>pass criteria</td>
    <td>
      <ul>
        <li><b>pass</b> if method is synchronized</li>
        <li><b>fail</b> otherwise</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>comments</td>
    <td>
       <a href="http://forums.sun.com/thread.jspa?messageID=10988343#10988343">
       click here for details</a>
    </td>
  </tr>
</table>
</body>
</html>
{code}
We were also using [HTML validator|http://validator.w3.org/|tool] tool to automatically catch simple syntax mistakes like missing tags.