End user or development-oriented build system?

I spent the past three weeks working on Xerces-C++ 3.0.0 which uses automake-based build system. Our own projects here at Code Synthesis all use the build system called build. The work on Xerces-C++ made me realize just how awkward the automake-based build systems are to develop with. It also made me realize that most build systems can be placed into one of the two categories: the ones that are optimized for the end user and the ones that are optimized for development (the Boost build system is a notable exception for it is a pain to use for both end users and, I suspect, the Boost developers).

The primary goal of an end user-oriented build system is to make once-off builds from scratch as straightforward as possible. Because the user can choose to build the software on any platform and installation of additional tools is an inconvenience, the following requirements are imposed on user-oriented build systems:

  • Support for a wide range of platforms
  • Least common denominators in the tools and features used

On the other hand, the primary goal of a development-oriented build system is to make the common development tasks as easy and fast as possible. This translates to the following requirements:

  • Ease of adding/removing files from the build
  • Complete dependency tracking for fast incremental builds

To realize how big a difference a development-oriented build system can make, let’s examine the fairly common development task of implementing a new feature in a library and adding a test for it. Assuming we already made the changes in the library source code as well as added the directory with the new test, here is the list of steps required in an automake-based project:

  1. Add the new test directory into higher-level Makefile.am
  2. Add the new test Makefile.am to configure.ac
  3. Run the bootstrapping script to generate configure, Makefile.in, etc.
  4. Run configure
  5. Run make in the library source directory to update the library
  6. Run make in the test directory

Instead of the last two steps one can run make in the top-level directory which will update the library, update (at least relink) all the tests and examples and finally run all the tests. In my experience, some people prefer this method because while taking longer it requires less manual work and ensures that everything that the test may depend on is up to date. In contrast, here is the list of steps required in a build-based project:

  1. Add the new test directory into higher-level Makefile
  2. Run make in the test directory

The last step automatically updates the library as well as any other parts of the project on which this test depends and which are out of date.

The steps in the build-based project take hardly one-tenth of the time required by the automake-based project. Someone may say that the task of adding a new test is not very frequent in most projects. Let’s then consider another common task: making a change in the library source code and running a specific test. For automake the list is as follows:

  1. Run make in the library source directory to update the library
  2. Run make in the test directory

As in the previous example, instead of these two steps some people prefer to just run make check from the top-level directory. The equivalent one step for the build-based project is:

  1. Run make in the test directory

The automake steps take at least several times longer to complete and can be much more than that if make is run from the top-level directory. In my experience these delays result in a much smaller number of development iterations I could do on a project as well as reluctance to make changes that are not absolutely necessary (e.g., code quality improvements).

It is clear that the constraints imposed by the two orientations are often incompatible: the development-oriented build system requires powerful tools while the user-oriented one requires us not to depend on anything but the bare minimum.

It is hard to say which build system a project should prefer if the goal is to be successful. On one hand, if the speed of development is restricted to a crawl by the build system, then you are unlikely to produce something worth using in a reasonable time. On the other hand, if potential users are bogged down with numerous build-time dependencies that your project imposes then they are less likely to try it.

Another alternative, which we are using in some of our projects, is to provide two parallel build systems. The obvious drawback of this approach is the need to maintain the two systems. In our case the second build system is only provided for a small sub-set of the project (examples) which helps minimize the negative impact of this approach.

The natural improvement of the two build systems idea is the development-oriented build system that can automatically generate makefiles for the end user build system. Note that this is not the same solution as offered by some build system generators (for example, CMake and MPC) since the overhead of running the generator every time a file is added or removed from the project makes them much less suitable for a development-oriented build system.

4 Responses to “End user or development-oriented build system?”

  1. John Snelson Says:

    I’m an automake lover, so I feel I have to jump in and defend it here. The way that Xerces-C uses automake is not really the best way to use it:

    http://www.gnu.org/software/automake/manual/automake.html#Alternative

    I use a single Makefile.am for building a number of my projects, and it’s just as easy to develop with as you describe the “build” system to be.

    Automake also handles build dependencies, parallel builds, multiple build directories, and distribution creation and testing - all on more platforms than you’ve ever heard of.

  2. boris Says:

    Hi John,

    I think right now the Xerces-C++ 3.0.0 code base strikes a nice balance between the two extremes: one makefile for the whole project and one makefile for every directory. Currently it has one makefiles for each of these components: the library itself, tests, and examples. I don’t think the one giant makefile approach will scale in the long run for any substantial project.

    I agree that automake provides support for more tasks though there is no reason a development-oriented build system can’t support them as well (in fact it can do a better job because it can rely on more powerful tools). The support for a wide range of platforms offered by automake, on the other, may be hard to replicate. Though I mention this tradeoff and the possible solution in the post.

    Boris

  3. John Snelson Says:

    A single Makefile can be split up with includes, so it can always be as maintainable as a multiple Makefile model.

    I guess I don’t believe there is such a thing as a developer or user oriented build system. Good build systems make both simple, and automake is the best I’ve found so far.

  4. boris Says:

    You can split a makefile into several parts (not sure you can do this with Makefile.am though) but you will still have to treat it as one makefile because of the shred variable scope. Here is an example. Let’s say you have two makefile parts, one for the library and one for the test. The library makefile has something along these lines:

    CPPFLAGS:= -DFOO
     
    libfoo.so: foo.o bar.o
      $(CXX) $(CPPFLAGS) -o $@ $^
    

    And the test makefile looks similar:

    CPPFLAGS:= -DTEST
     
    test: test.o libfoo.so
      $(CXX) $(CPPFLAGS) -o $@ $^
    

    Then you have the master makefile which includes the two. The problem is that the second makefile overwrites the CPPFLAGS variable defined in the first makefile. The common way to resolve this would be to prefix each variable in a makefile fragment with its directory name, e.g., libfoo_CPPFLAGS, test_CPPFLAGS. This quickly gets out of hand in large projects where there might be hundreds of tests with deep subdirectory structure. How do you like the tests_cxx_parser_built_in_CPPFLAGS and tests_cxx_parser_validation_built_in_CPPFLAGS names? These would be actual names taken from one of my projects.

    Build addresses this in a scalable way by implementing variable scoping (so that included makefiles don’t override includer’s variables with the same names) and target-specific variables. But this can only be done in the latest version of GNU make (3.81) so an automake-like build system that has to support every version of make down to the ten-year-old 3.79 can’t use this.