Managing SunOS Development With TeamWare |
![]() |
Management recognized during the project's planning stages that developers would need parallel access to the most current source -- the classic one-developer-at-a-time access allowed by SCCS (the Source Code Control System) was too limiting for their needs. As a more capable alternative, they adopted Configuring, the TeamWare component that manages groups of files for groups of developers.
Configuring provides an isolated workspace for each developer and automatically records the change history of each source file. By judiciously setting up their hierarchy of Configuring workspaces, the group allows each developer access to the newest stable source code at any time. If two developers edit copies of the same file, Configuring detects the differences and presents them graphically in the Merging tool. One or both of the developers merge the changes into a single file that is then returned to the common source base.
Five integration engineers coordinate and maintain the Configuring workspace hierarchy for the project. Only a fraction of their time is spent on Configuring issues, however. Their additional duties include:
Workspace -- A directory (and its subdirectories) that has been designated for use by Configuring. Configuring identifies a workspace by the presence of a subdirectory named Codemgr_wsdata.
Workspace hierarchy -- An organization of parent and child workspaces, much like a standard directory hierarchy. Each workspace can have a parent and one or more children, in the same way that a directory has a parent directory and subdirectories.
Putback -- A transaction between a child workspace and its parent, in which changes to files from the child are copied ("put back") into the parent.
Bringover -- A transaction between a child workspace and its parent, in which changes to files from the parent are copied ("brought over") into the child.
Reparent -- An operation that changes the parent of a child workspace.
Beneath the top-level workspace, called train, is a group of sibling integration workspaces (see Figure 1). Developers bring files into their private workspaces from these integration workspaces. They also put back to the integration workspaces, never to train directly. Because the integration workspaces restrict access to the release-level workspace, the SunOS group calls them gates. In each group, a senior-level developer is designated as the gatekeeper. The gatekeeper is responsible for maintaining the group's gate workspace, making sure that the source it contains is without error before it is put back to the release-level workspace.
Figure 1 Basic Workspace Hierarchy
Building and Testing Executables
Each developer in the SunOS development group is expected to build (compile and link) and test the new code he writes. Custom File List Programs (FLPs) and hierarchical makefile structures help ensure that every development workspace builds with the most recent header files and libraries. After a successful build and test, the developer puts back his new source files into the integration workspace. If a conflict occurs because another developer's changes were put back while the first developer was working, the first developer resolves the conflicts with the Merging tool, and the resulting source files are once again compiled, linked, and tested. Isolating Test Workspaces
The SunOS development group creates identical copies ("clones") of integration workspaces in which to build and test. After a successful test of the software, reparenting is generally used to put back changes to the release workspace. Depending on the demands of the moment, testing is done in a clone of one gate or in a temporary composite workspace formed by cloning one gate, then reparenting the clone to a second gate and updating it with files from the second gate. This section describes both approaches. Testing in a Single Integration Workspace
Consider the gatekeeper who is responsible for maintaining the integration workspace gate_1. The workspace has accumulated enough changes to warrant putting them back into train. Before putting back, however, the gatekeeper must do three things:
Instead of locking the gate workspace, the SunOS group's gatekeepers use a cloning technique that allows developers access to the gate while they resolve conflicts and fix bugs in the source base. The following figure (Figure 2) shows the steps the gatekeeper would perform to integrate gate_1 with the release workspace train.
Note that in the day or two the gatekeeper spent building and testing in gate_1_clone, developers may have been putting new changes into gate_1. The best method for synchronizing gate_1 and gate_1_clone in that event involves three steps, as described in Figure 3.
The cloning technique just outlined allows development to move forward at the same time a large, complex workspace is being tested and integrated into the release level. It's disadvantage comes when a developer puts back a large functional module into the integration workspace while testing is going on in the clone. In such a case, the final bringover and putback (\xb2 and \xb3 , Figure 3) can result in many conflicts and a great deal of work for the gatekeeper. These cases can be avoided by polling developers about their work in progress and scheduling the build and test operations for lull times following a developer milestone putback event.
Testing in Merged Workspaces
With a few additional steps, the process discussed in the last section can be applied to merging several gates, building and testing, and then reparenting and putting back to the release workspace. If the gates do not contain a lot of new code, gatekeepers can save time by merging the gates before building and testing them: instead of building and testing a clone of each gate, the gatekeepers build and test once for all the merged workspaces.
Clearly, more than two integration workspaces can be merged in the manner just described in Figure 4. Projects of lesser complexity may extend workspace merging to include all integration workspaces in the project, periodically merging, building, testing, and putting back to the release level. See the TW Topic "Workspace Hierarchy Strategies for Software Development and Release on page 37" for a discussion of this approach.
Managing Release-Level Workspaces
Many of the considerations that apply to integration-level workspaces also apply to the release level -- providing transaction access for child workspaces during build and test cycles, for example. Additional issues at the release level involve providing for alpha, beta, and customer releases, multiple simultaneous customer releases, and maintenance (patch) releases. Posting Notifications
Configuring can send email messages when an interworkspace transaction occurs (bringover, putback) or when a workspace is deleted, moved, or reparented. Integrators at the release level set up these notifications so that they are alerted whenever an interesting transaction occurs in workspaces for which they have responsibility. Controlling Access to Source Files
Controlling access to workspaces is important, particularly at the release level. Configuring, through the access_control file in the Codemgr_wsdata directory, allows only specified users and net groups to perform transactions with a workspace (bringovers and putbacks) and operations on a workspace (reparentings, deletions, and so on). Building and Testing at the Release Level
At the release level, building and testing is performed in a clone workspace just as it is at the integration level. However, procedures at the release level are more formalized because risks to the code base are higher. Figure 5 illustrates the process. By cloning a workspace (\xac ) for test activity, the release workspace itself is free for transactions with integration workspaces.
Figure 5 Release Level Building and Testing
Because the SunOS operating system is compiled for more than one architecture (SPARC® and Intel® x86 architectures at this writing), a build clone is created for each architecture.
Managing Test Releases
If the train clone shown in Figure 6 is created for a test release, it can be named Alpha or Beta and maintained for the period of the test release. If it is created only to establish a stable release workspace, it can be deleted after testing is completed.
Managing Multiple Customer Releases
The SunOS group publishes an ongoing set of releases, handling at least two releases concurrently. As one release enters beta testing and First Customer Ship (FCS), development and alpha testing begins on the next release. A modified cascade workspace structure is used to support the releases (see the TW Topic "Workspace Hierarchy Strategies for Software Development and Release on page 37" for a more general discussion of workspace hierarchies).
The release-level workspace train is always the top of the current development hierarchy. For each release, train is stabilized and cloned. The old train workspace is renamed to designate its release number and the clone is renamed train. Active integration workspaces are reparented to the new train, and development continues on the next release. Figure 7 illustrates the process for Release 3.0.
By reparenting all the existing development and integration workspaces to the new train, any work in progress that was not finished for the last release (the development of a new feature, for example) will automatically be applied to the next release.2
The analogy to a train is apt when the workspace hierarchy is viewed over time. Each release begins as a snapshot of the current train, and a string of releases takes on the appearance of a series of boxcars left behind by train, which proceeds along the track. From a developer's perspective, "the train" is always the active development workspace with the most recent code, always moving on to the next release (Figure 8).
Figure 8 A String of Releases Left Behind by train
Managing Maintenance Releases
Of course, each release does not remain frozen in time. As bugs are reported against a particular release, child workspaces are created in which to fix the bug. Moreover, a bug fix applied to a shipping release may also be applied to later release workspaces, including train. This is the normal case for the SunOS project because each succeeding release is a superset of the previous release.
In a similar vein, a child of a late release is never allowed to reparent into the hierarchy of an earlier release because of the risk of putting back a late feature into an earlier release. Figure 9 illustrates what can go wrong.
The inadvertent contamination of earlier releases illustrated in Figure 9 is difficult to guard against and can be difficult to fix. The best defense is to strongly discourage such reparenting, as the SunOS group does. Instead, they fix the bug in the earliest release to which it applies and bring the changes forward along the release chain.
Bringing Changes Into a Recent Release
When changes are brought over from an early release workspace into a later release, they are almost always brought over into a clone of the later release workspace (not into the later release directly). For example, when the person responsible for release-level integration wants to bring changes from the Release 2.0 workspace into Release 3.0, she creates a clone of Release 3.0, reparents to Release 2.0, and brings over the changes into the clone (Figure 10). After resolving conflicts and thoroughly testing the changes in the Release 3.0 clone, she puts back to the Release 3.0 workspace.
If the build reveals a major problem, the bug is investigated in the clone and narrowed down to one area of functionality in one of the release workspaces. The bug is fixed in the integration workspace where the functionality originated, and the change is put back into the release workspace. The change is then brought over into the build clone, and build and test begins again.
By merging Releases 2.0 and 3.0 in a clone of Release 3.0, the release-level integrator leaves Release 3.0 free for putbacks from its child bug-fix workspaces. The penalty for allowing these putbacks is that the putback from the clone into Release 3.0 (\xae ) may be blocked. In that case, the integrator must bring over updates from Release 3.0 into the clone, resolve conflicts, build, test, and put back again.
Automating Builds Within Configuring
Maintaining logical, consistent source and makefile hierarchies is important to any large project, whether or not it uses Configuring. Like most large projects, the SunOS project organizes its source files into directories along functional lines. The makefile hierarchy follows the source file organization: each directory contains one default makefile, and each makefile may include other makefiles from parent directories. Organizing the Workspace Directory Structure
The root of each Configuring workspace is defined by the environment variable CODEMGR_WS. This variable contains the absolute path name of the workspace, and can be thought of as the relative root of the workspace. Beneath this relative root are three top-level directories:
Figure 11 shows part of the top-level source tree. Besides directories for commands, device drivers, header files, and so on, the top-level directory contains a makefile that is used to build the entire product, and a master makefile that is included in other makefiles in the source tree.
Figure 11 Partial Directory Tree for SunOS Source Hierarchy
Coordinating Makefiles
The SunOS group had the following goals in mind when it wrote its guidelines for writing makefiles:
The following example shows how the policy of including parent makefiles helps keep each makefile simple. The example shows the build makefile for the ls command. It includes two higher-level makefiles, Makefile.cmd and Makefile.targ.
Makefile.cmd contains variables and rules common to the builds of all commands in the $SRC/cmd directory. It includes the top-level makefile, Makefile.master. The second included makefile, Makefile.targ, specifies system-wide install targets.
Figure 12 Makefile for the ls Command
Using DMake
Large software projects typically consist of multiple independent modules that can be built in parallel. DMake supports concurrent processing of targets on a single machine; this concurrence can significantly reduce the time required to build a large project.
The ws Utility
The SunOS group wrote the ws utility to automatically initialize the build environment within a Configuring workspace. It configures an environment to build the SunOS executables, sets environment variables for the workspace, and spawns a shell for the environment that has been set up. Using ws
Consider the developer responsible for the ls command. Only a few source files are required to build the ls executable, and these are the files she brings over into her workspace when she begins work (Figure 13). CODEMGR_WS is the environment variable that contains the absolute path name of the current workspace.
Figure 13 Directory Structure for ls Development Workspace
Figure 14 Example Search Path for Files in proto Directory
If the necessary files cannot be found in any proto directory, the search can default to the native file system libraries, issue a warning message, or both. Whatever search sequence she decides on, ws provides an easy way to specify it.
Shielding Developers From Unstable Files
If a developer sets up his build search path to refer to a copy of a library or header file located in another workspace's proto directory, he is exposed to the changes other developers make to the file. Any change in the file will be immediately visible the next time he performs a build. However, there are times when a developer does not want to see such changes -- when he is diagnosing a bug, for example. A developer can do one of two things to shield himself from changes:
The sccsrm script initially renames a file to a .del* name in the current directory; another script is run periodically to move .del* files to an area in a directory hierarchy parallel to (and with the same structure as) the source hierarchy. See the TW Topic "Deleting Files From the Configuring Tool Workspaces on page 89" for details on this strategy.
Because users are not required to remember the path name of the directory that stores deleted files when they rename a file they want to delete, the sccsrm utility helps avoid mistakes. Figure 16 shows the entire script for sccsrm.
Figure 16 Script for the sccsrm Utility
sccsmv
Because moving SCCS-controlled files is so similar to deleting them in the Configuring environment, the utility sccsmv was written as a companion to sccsrm. Using File List Programs
When a developer brings over a file into her workspace, she explicitly specifies the file she wants to move. When she specifies a directory, a default File List Program (FLP) sees to it that all the files in the directory are brought over. The default FLP accomplishes this feat by listing the contents of the directory and writing them to stdout, from which the bringover and putback commands read them. Users can replace the default FLP with a custom FLP that writes any file names they want to stdout, enabling them to bring over (and put back) files without naming them explicitly. Default File List Program
Once again, consider the developer responsible for the ls command. With the default FLP (named def.dir.flp) in force, the developer would explicitly bring over the ls and proto directories, resulting in a development workspace directory structure like the one shown in Figure 17. (The Codemgr_wsdata directory is present in every workspace.)
Figure 17 Development Workspace for the ls Command (Default FLP)
A less obvious problem is that other files in the source base may need to be built before ls, as determined by the ls makefile. If such files exist, they must be brought over explicitly when using the default FLP.
Custom File List Program
The SunOS group has customized the default FLP so that it executes scripts that identify the files that are required for a specific build. The required files are listed by scripts named inc.flp and req.flp, which are stored in the same directory as the source files required for the build.
Figure 18 Development Workspace for the ls Command (Custom FLP)
Figure 19 The Custom FLP Searches Directories for req.flp and inc.flp
Registering Public Workspaces With NIS
The SunOS group takes advantage the automounter and the Network Information Service (NIS), both standard parts of the Solaris operating environment, to reduce the amount of information developers must remember daily.
server:/export/here/there/everywhere/sunos1
is normally accessed by the automounter with the command
cd /net/server/export/here/there/everywhere/sunos1
In contrast, after a map name for the workspace has been entered in the automounter map, SunOS developers can access the same workspace with the command
cd /workspace/sunos1
2 And, if it is not finished for the next release, it can be applied to the one after that.