Parallel Build in Linux/Unix

As a software developer, lengthy build time is always an enemy. You want to do almost anything to shorten the build time. One of the way to do that is to make the build process runs in parallel. If you are using GNU Make, it's as easy as adding "-j" flag to your build script. This is a sample bash script to do that:

#!/bin/bash

_architectures="x86_64-w64-mingw32 i686-w64-mingw32"
CPU_CORES="$(nproc)"

build_exe () {
 local arch=$1
 local core_count=$2

 pushd build-$arch
 CMAKE_INCLUDE_PATH="/usr/"$arch"/include"
 echo "CMAKE_INCLUDE_PATH = "${CMAKE_INCLUDE_PATH}
 export CMAKE_INCLUDE_PATH
   $arch-cmake -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug ..
   ##$arch-cmake -DCMAKE_BUILD_TYPE=Release ..
 make VERBOSE=1 -j$core_count
 popd
}

for _arch in ${_architectures}; do
 case "$1" in 

  clean) if [ ! -d build-${_arch} ]; then
             echo "build directory does not exist. Terminating script.."
         else 
             rm -rvf build-${_arch}
         fi
   ;;
 
  rebuild) if [ -d build-${_arch} ]; then
              rm -rvf build-${_arch}
           fi

      mkdir -p -v build-${_arch}

      ## call build function 
      build_exe ${_arch} ${CPU_CORES}
   ;;
 
  *) if [ ! -d build-${_arch} ]; then
     echo "build directory does not exist. Creating directory.."
     mkdir -p -v build-${_arch}
     fi
   
     ## call build function 
     build_exe ${_arch} ${CPU_CORES}
   ;;
 esac

done 

The preceding script is probably rather intimidating. However, it's just a simple bash script. Just focus to the build_exe() function. That's where the core of the action happens: make is invoked with -j parameter, followed by the number of CPU cores in the system. FYI, the script above is a cross-compilation script which runs on Linux and creates Windows executables. But, the latter fact shouldn't deter you from trying to understand the big picture though ;)

The script shows how to obtain the number of CPU cores in bash, i.e. via the nproc command. nproc is part of coreutils in Linux. If you're using other kind of Unix, try to find an equivalent command. Once, the number of CPU cores is known, that number is used as parameter to make. Running make in parallel should cut down the build time quite a bit. In some projects, it could be quite a lot of saving in build time.

Not all projects can benefit from parallel build. However, it's worth it to try modifying your project to use parallel build before discounting it as useless ;-)

---

Post a Comment