Project

General

Profile

This page is work in progress.

Gwbuild

Preamble

I don't plan to replace the current autotool usage in AqBanking with gwenbuild, I just want to speedup my own development process and circumvent the current problem where I need a complete rebuild of AqBanking as soon as a single source files has changed because otherwise the resulting libaqbanking.so sometimes still contains older objects. The process of "./configure && make" simply takes an inacceptable amount of time to finish which slows down development considerably.

Introduction

GwenBuild is yet another build system.

autotools (automake, autoconf) are really cool and they work in almost every case as expected.
However, they are extremely slow which is a problem with larger projects containing many source files
and convenience libraries. Also, they are hard to debug if you encounter unusual problems, e.g. in AqBanking
we have the problem that sometimes not all sources/convenience libraries are correctly rebuilt after source code
changes and up to now I have no idea why that is.

A nice solution would be to switch to cmake, especially when using their build target ninja which is
astonishingly fast. cmake is quite easy to setup for simple projects, however, it is not that easy to change larger projects
to use it. Also, there is quite a steep learning curve.

meson is another good alternative since it also uses ninja and is therefore also extremely fast. But I wasn't able to
adapt my projects to use "meson" because some features I needed for e.g. AqFinance or AqBanking required a version of meson newer
than what is provided by the distribution I'm currently working with.

Out of curiosity gwenbuild was started. I wanted to see how fast a build system can be and how hard it would be to write one.
Gwenbuild is a tool inside the library gwenhywfar, which is a base library for all my projects. Therefore it is always available in
the required version for depending projects. Gwenhywfar itself will not be ported to use this build system to avoid the
hen-and-egg-problem.

The following requirements were identified before development started:
  • needs to be much faster than autotools/make
  • needs to provide library, header and function search functions
  • needs to be able to generate a "config.h" file
  • needs to be able to rewrite files by replacing placeholders in source files (like "sed" does in ./configure)
  • needs to be simple to setup
  • needs to have support for typemaker2 source files
  • needs to allow out-of-source build
  • should be easy to extend
  • should handle dependency trees
  • needs to allow for custom build commands to generate future or unusual file types

Quickstart

AqBanking is the first project using gwenbuild in parallel with autotools. AqBanking is quite a complex project using convenience libraries, shared loadable modules, shared libraries and applications. Therefore it is a good starting point to understand how gwenbuild is used.

To build AqBanking with gwenbuild try the following commands inside the source tree of AqBanking using 14 processes (takes about 9 secs on my build machine):


mkdir build
cd build
gwbuild ..
gwbuild -p
gwbuild -B tm2builder
gwbuild -j14

  • the first gwbuild is a replacement for what "./configure" does in autotool projects (e.g. looking for headers, libraries, functions, programs etc.)
  • the second invocation prepares typemaker2 files (only needed if typemaker2 is used, in AqBanking it is used)
  • the third call makes gwbuild selectively build the typemaker2 files. This is needed because the dependency tree of the files can only be build correctly if all compilable source files exist.
  • the last call builds all the rest and uses 14 processes in parallel for this (my build machine has 16 threads so 14 is a nice value here, but you mileage may vary). Only this line needs to be repeated if you change the source files. If 0BUILD files are changed they will be updated by internally calling the equivalent of "gwbuild .." with all the parameters you gave to that call.

Current Status

As of Apr 18th 2021 the following is implemented:
  • compiling and linking complex projects on Linux (tested with AqBanking and AqFinance)
  • transforming typemaker2's type description files into source code files and compiling them
  • user-defined commands, in AqBanking and AqFinance used to merge XML files and to generate translation catalog templates (e.g. aqfinance.pot)
  • cross-compiling simple projects for Windows on Linux
  • installing target files
  • cleaning the build environment (i.e. removing generated files from the build folder)

Format of Build Files

Gwenbuild uses XML-like files to define the project you want to build. Every folder of a project should contain a file named 0BUILD.

To be continued... (the following are fragments of a documentation for the various XML elements found in a 0BUILD file).

CheckProgs

Long Version

Look for a program/executable with the given name (or alterantive names) and write the result into a variable to be used later on in the build files.


 <checkProgs>
   <prog id="xmlmerge" cmd="xmlmerge" >
     <alternativeNames>
       xmlmerge
     </alternativeNames>

     <paths>
       /usr/local/bin
       /usr/bin
       /bin
     </paths>
   </prog>
 </checkProgs>

Short Version


 <checkProgs>
   <prog id="xmlmerge" cmd="xmlmerge" />
 </checkProgs>

BuildFiles

This element can be used to run custom commands to create files like in the following example.


    <buildFiles>
      <input>
        jobgetbalance.xml
        jobgettransactions.xml
        jobgettrans_camt.xml
        jobforeignxferwh.xml
        jobloadcellphone.xml
        jobsepadebitdatedsinglecreate.xml
        jobsepadebitdatedmulticreate.xml
        jobsepacor1datedmulticreate.xml
        jobsepaxfermulti.xml
        jobsepaxfersingle.xml
        jobsepadebitsingle.xml
        jobsepacor1datedsinglecreate.xml
        jobsepastandingordercreate.xml
        jobsepastandingorderget.xml
        jobsepastandingordermodify.xml
        jobsepastandingorderdelete.xml
        jobgetestatements.xml
        jobgetestatements2.xml
        jobgetdepot.xml
      </input>

      <output>
        accountjobs.xml
      </output>

      <cmd tool="$(xmlmerge)" checkDates="true" >
        -v --compact -o $(OUTPUT[]) $(INPUT[])
      </cmd>
      <buildMessage>
        Merging XML files into $(OUTPUT[0])
      </buildMessage>
    </buildFiles>

This example takes the files listed under INPUT and merges them into a destination file using the tool hbcixml from libgwenhywfar.