Parallel compilation from the command line
I often need to compile a bunch of generated C++ files from the command line to make sure everything compiles cleanly and to run a quick test. These are C++ data binding files generated by XSD and XSD/e from various XML schemas. There can be quite a few files (like a thousand) or there can be a handful of them but each is quite large and takes long to compile. I used to run g++ from the command line which compiles one file at a time:
g++ -c *.cxx
This was slow. Annoyingly, I also had three more CPU cores idling while they could have been used to speed things up. If it were a longer-term project, then I would have created a makefile for it and run GNU make in parallel (-j 4). But these were just quick tests and creating a makefile for each set of schemas that I test seemed like a chore (after the initial test the schemas are added to a test repository where a set of special scripts and makefiles are used to automatically check for regressions before each release).
What would then be nice is a single, generic makefile that I could use to compile a set of files in parallel. Ideally, it shouldn’t be much more typing than the simple g++ invocation above. It is quite easy to come up with such a makefile for GNU make:
ifeq ($(src),) src := *.cxx endif obj := $(patsubst %.cxx,%.o,$(wildcard $(src))) .PHONY: all clean all: $(obj) clean: rm -f $(obj) %.o: %.cxx $(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $< -o $@
Now I can compile all the C++ files in a directory in parallel:
make -f ~/makefile -j 4
I can also specify a custom set of C++ files using the src variable:
make -f ~/makefile -j 4 src="test*.cxx driver.cxx"
April 18th, 2009 at 6:07 am
The real solution would be to add parallel parsing/compiling support to GCC. I don’t see a reason why this can’t be done, except for the huge pile of legacy code. I think that the LLVM guys are working on implementing parallel compilation to separate modules.
April 19th, 2009 at 2:45 am
There are two things that can be done in the compiler. The first is to make the compiler compile several translation units (i.e., input files) in parallel. This could be fairly easy to implement but it will amount to pretty much the same thing as what’s already possible with GNU make. This is probably the reason it hasn’t been done in GCC.
The second is to make the compiler compile a single translation union in parallel, Now that will be very useful but I don’t think it is easily achievable. The C/C++ compilation process is inherently sequential and it is not clear how to parallelize it. Perhaps the optimization part can be done in parallel. That is, have multiple threads work on various parts of the (intermediate) code.