Monday, February 27, 2017

Parallel Build in Linux/Unix

As a software developer, lengthy build time is always an enemy. You want to do almost anything to shorten the build time. One of the way to do that is to make the build process runs in parallel. If you are using GNU Make, it's as easy as adding "-j" flag to your build script. This is a sample bash script to do that:
#!/bin/bash

_architectures="x86_64-w64-mingw32 i686-w64-mingw32"
CPU_CORES="$(nproc)"

build_exe () {
 local arch=$1
 local core_count=$2

 pushd build-$arch
 CMAKE_INCLUDE_PATH="/usr/"$arch"/include"
 echo "CMAKE_INCLUDE_PATH = "${CMAKE_INCLUDE_PATH}
 export CMAKE_INCLUDE_PATH
   $arch-cmake -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug ..
   ##$arch-cmake -DCMAKE_BUILD_TYPE=Release ..
 make VERBOSE=1 -j$core_count
 popd
}

for _arch in ${_architectures}; do
 case "$1" in 

  clean) if [ ! -d build-${_arch} ]; then
             echo "build directory does not exist. Terminating script.."
         else 
             rm -rvf build-${_arch}
         fi
   ;;
 
  rebuild) if [ -d build-${_arch} ]; then
              rm -rvf build-${_arch}
           fi

      mkdir -p -v build-${_arch}

      ## call build function 
      build_exe ${_arch} ${CPU_CORES}
   ;;
 
  *) if [ ! -d build-${_arch} ]; then
     echo "build directory does not exist. Creating directory.."
     mkdir -p -v build-${_arch}
     fi
   
     ## call build function 
     build_exe ${_arch} ${CPU_CORES}
   ;;
 esac

done 
The preceding script is probably rather intimidating. However, it's just a simple bash script. Just focus to the build_exe() function. That's where the core of the action happens: make is invoked with -j parameter, followed by the number of CPU cores in the system. FYI, the script above is a cross-compilation script which runs on Linux and creates Windows executables. But, the latter fact shouldn't deter you from trying to understand the big picture though ;)

The script shows how to obtain the number of CPU cores in bash, i.e. via the nproc command. nproc is part of coreutils in Linux. If you're using other kind of Unix, try to find an equivalent command. Once, the number of CPU cores is known, that number is used as parameter to make. Running make in parallel should cut down the build time quite a bit. In some projects, it could be quite a lot of saving in build time.

Not all projects can benefit from parallel build. However, it's worth it to try modifying your project to use parallel build before discounting it as useless ;-)

Monday, February 13, 2017

Fix for systemd v232 build failure when using GNU gperf 3.1

You might encounter the build failure in this post title if you're the kind that roll your own Linux Systemd. I encountered it while building Systemd package for my Arch Linux SELinux variant.

The culprit is mismatch in lookup functions declaration--hash functions--generated by GNU gperf version 3.1 and the function declaration in Systemd version 232. I managed to complete the build after creating and using this patch:https://github.com/pinczakko/systemd-gperf-3.1-patch. As for whether the patch is working or not, well, it works without problems in my machine. Nonetheless, it's just a very minor patch.

UPDATE:
------------
This issue has been fixed just now in systemd v232. See: https://github.com/systemd/systemd/commit/c9f7b4d356a453a01aa77a6bb74ca7ef49732c08

UPDATE 2:
---------------
You can add the change as cherry-picked git change to the PKGBUILD to fix this issue in Arch Linux SELinux package. This is the diff (or patch):
diff --git a/PKGBUILD b/PKGBUILD
index 47d82d1..1e57ec7 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -61,6 +61,7 @@ _backports=(
   'cfed63f60dd7412c199652825ed172c319b02b3c'  # nspawn: fix exit code for --help and --version (#4609)
   '3099caf2b5bb9498b1d0227c40926435ca81f26f'  # journal: make sure to initially populate the space info cache (#4807)
   '3d4cf7de48a74726694abbaa09f9804b845ff3ba'  # build-sys: check for lz4 in the old and new numbering scheme (#4717)
+  'c9f7b4d356a453a01aa77a6bb74ca7ef49732c08'  # build-sys: add check for gperf lookup function signature (#5055)
 )
  _validate_tag() {
Hopefully, this temporary fix could help before the official fix is included in the main Arch Linux package.