<![CDATA[Wild Duck Theories Australia Pty. Limited]]>https://blog.wildducktheories.com/Ghost 0.7Fri, 11 Jul 2025 10:37:41 GMT60<![CDATA[towards an automated git submodule tracking agent]]>If you ask a moderately experienced git user about submodules they are likely to turn to you, look you earnestly in the eye and say: "don't use them, life is too short". Or, if they are a more experienced and helpful git user, they might explain exactly why git submodules

]]>https://blog.wildducktheories.com/towards-an-automated-git-submodule-tracking-agent/dad4261a-3e0c-453a-88d5-6c85360c8e3fFri, 17 Jun 2016 11:56:25 GMTIf you ask a moderately experienced git user about submodules they are likely to turn to you, look you earnestly in the eye and say: "don't use them, life is too short". Or, if they are a more experienced and helpful git user, they might explain exactly why git submodules are painful. Before telling you: "don't use them, life is too short"

And, it is true, git submodules are painful particularly if you have to interact with them manually before or after every push to a shared remote repo - there are too many moving parts that can make life hell when they get out of sync.

git submodules are ok for tracking the precise version of a few rarely updated external dependencies because the manual book keeping overhead their use normally implies is small if done only occasionally, but they are terrible for tracking the precise versions of frequently updated internal dependencies, particularly in the absence of a robust automated solution.

why bother?

Given the hassles, why bother using them at all?

It turns out that git submodules do provide quite a neat solution to the problem of tracking the configuration of multi-component releases.

To explain, consider a system comprised of multiple components where the source for each component is tracked by its own git repository. To track the configuration of a particular release of the system as a whole requires some way to track git commits of each component in the system and tracking particular commits of a collection of components is something that git submodules can do quite well - it's just the user experience that is horrible.

So, what if you could provide a user experience that didn't suck and still take advantage of git submodules for configuration management? That is what this article is about.

the approach

This section sets forth the elements of the approach.

tracking agent

The approach assumes the existence of one or more tracking agents. Each tracking agent would be responsible for monitoring the tracking branches of one or more submodules and updating the tracking branch of a superproject as the submodules change.

dedicated workspaces

The tracking agents would operate with a dedicated workspace - in particular, the tracking agent should never be run in a developer's workspace which might have untracked, unstaged or uncommitted work in it.

centralisation

Most tools in the git suite operate in a distributed fashion without reference to a central repository. Submodule tracking agent should be centralised. The main reason is that we want remove the responsibility for maintaining the superproject from individual developers and also we want to simplify the solution to eliminate the possibility of multiple agents operating on the same superproject branch at the same time.

read-only superprojects

The intent is that developers never have to manually update a superproject, making them effectively read-only from the developer's point of view. Developers might choose to use superprojects to checkout all the components of a release, but would not be required to.

submodule-only superprojects

One way to avoid the need for developers to manually update a superproject is to not store any files in the superproject. In other words, a superproject should be a pure collection of submodules and contain no other files. If there are no other files, there is no need for a developer to edit them and if a developer doesn't need to edit them, then there is no need for a developer to update to the superproject directly and so there can be no possibility of merge conflicts between a developer's changes and the tracking agent's changes.

This is really more a guideline for superproject design than a strict rule that would be checked by the submodule tracking agents.

no requirement for developer superproject checkout

If a developer doesn't need to maintain the superproject, then there is no need for the developer to checkout the superproject.

That said, when used in a readonly fashion, superprojects can still be a handy way to initially checkout a group of related submodules or periodically resync them all with their origins at the latest revision levels.

This post has described the requirements for a git submodule tracking agent. A future article will discuss the in detailed design of such an agent.

]]><![CDATA[a technique for duplicate detection with overlapping imports of key-less records]]>My financial affairs are simple enough that I can keep track of them by exporting the transaction data from my bank accounts and annotating the exports in a Google sheet. The annotations allow me to classify each income or expenditure item with a category that allows me to generate reports

]]>https://blog.wildducktheories.com/a-technique-for-duplicate-detection-with-overlapping-data-imports/644a70f4-7384-4375-9d63-f5c09d97ce44Sun, 26 Oct 2014 05:52:18 GMTMy financial affairs are simple enough that I can keep track of them by exporting the transaction data from my bank accounts and annotating the exports in a Google sheet. The annotations allow me to classify each income or expenditure item with a category that allows me to generate reports that aggregate the data across each category for reporting and analysis purposes.

One of the tedious aspects of this process is the need to export the transaction data from the bank. I will eventually write a Chrome extension to help me automate this, but for now on a weekly basis I manually export the transaction data from each account as a CSV file and then run an import process over the resulting files.

One challenge with this is that so that I don't miss any data I need to be able to overlap the periodic exports. The overlap means that the export files from consecutive exports may contain duplicate records.

Obviously, from an accounting point of view, it is important to eliminate the duplicates from the import and, from a tedium mimimisastion perpective, to do this automatically.

This would be a trivial problem if the source transaction data had a natural or surrogate key for each record but unfortunately it does not. In fact, each source record has the following fields:

  • account
  • transaction date
  • narrative
  • credit amount
  • debit amount

No combination of these fields is guaranteed to produce a unique key for a record - it is possible that two records may specify the same account, transaction date, narrative, credit and debit amounts. And, in fact, I have observed that the bank occasionally alters the narrative for a given transaction record between subsequent exports so the narrative cannot be used as part of a record key since to do so would risk creating two imported records, with different keys, for the same transaction.

The solution I chose for this problem is to extend the imported record with two additional fields: repetition and md5 sum. So:

  • account
  • transaction date
  • narrative
  • credit amount
  • debit amount
  • repetition
  • md5 sum

Repetition is initially blank. md5 sum is calculated over the account, transaction date, credit and debit amount and repetition fields. If two records in the same import duplicate the same fields, then the md5 for those two records will be identical. In this case, the repetition for the current record is set to 1 and the md5 sum for that record is recalculated. The process is repeated, incrementing repetition on each occasion, until the record obtains an md5 sum that is unique within the same import.

The md5 sum so calculated is then effectively a surrogate key for the record which can be assumed to identify the same transaction across all previous (and future) imports under the additional assumptions that all imports always contain a full days worth of data, the bank never changes the date of a transaction and never withdraws (as opposed to reverses) a transaction previously published. Should those assumptions be violated, a manual intervention would still be required to delete downstream references to the altered or deleted records.

With a surrogate key for each record now established, I can then merge the newly imported data with previously imported data by matching the new and existing records on the basis of their surrogate keys.

The key assignment process is both stateless and robust - I don't need access to all the data I previously imported in order to generate unique keys for each record and it doesn't matter if different imports at different times contain overlapping data -in both cases, the same keys will be generated for the same set of same transactions.*

* It is technically true that if the order of transactions with identical dates and amounts is transposed between subsequent imports then the narratives for two records may be swapped on subsequent imports. This potentially becomes problematic if downstream processing depends on automated interpretation of the narrative. It might be necessary, in this case, to account for narrative changes during the merge process, if required.

To support this pattern, I have written some tools in golang that can generate additional fields to ensure that each record of a CSV input stream contains a unique natural key and another tool that can then generate a surrogate key from the MD5 sum of the unique natural key.

So, for example:

uniquify \
  --partial-key BankAccount,Date,CreditAmount,DebitAmount \
  --additonal-key Repetition |
surrogate-keys \
   --natural-key \
     BankAccount,Date,CreditAmount,DebitAmount,Repetition \
   --surrogate-key KeyMD5

For more information, see: https://godoc.org/github.com/wildducktheories/go-csv/uniquify, https://godoc.org/github.com/wildducktheories/go-csv/surrogate-keys and https://github.com/wildducktheories/go-csv .

]]><![CDATA[building VirtualBox on OSX 10.9]]>This post documents a guide to building VirtualBox on OSX 10.9. This document should be read in conjunction with the official Mac OS X build instructions [1]. Its intended audience are people relatively new to VirtualBox and/or OSX development who might find that the official instructions don't provide

]]>https://blog.wildducktheories.com/building-virtual-box-on-osx-10-9/7d3c883b-908a-49d4-a1d7-4226bed74330Fri, 20 Jun 2014 10:32:37 GMTThis post documents a guide to building VirtualBox on OSX 10.9. This document should be read in conjunction with the official Mac OS X build instructions [1]. Its intended audience are people relatively new to VirtualBox and/or OSX development who might find that the official instructions don't provide quite enough guidance to navigate the obstacles introduced by the need to use an older version of Xcode to successfully execute the build.

Most of the information documented here was originally provided by Klaus Espenlaub and Vadim Galitsyn in a thread "Sketch of how to build on OSX 10.9?" in the Virtual Box developers mailing list.[2] [3]

The reason these additional instructions may be helpful for novices is that the VirtualBox source tree doesn't support building against XCode 5.1 which is the current set of development tools that a OSX 10.9 user is likely to have installed. To build VirtualBox successfully, the 10.9 user needs to obtain an earlier version of XCode (I chose 4.1) and manually unpack parts of it to obtain the necessary SDKs and gcc compiler required by the VirtualBox source tree. Later versions of XCode symlink gcc to clang/LLVM which does not support all the gcc and g++ options used by the VirtualBox build scripts.

Unlike the official instructions, these instructions also assume that other dependencies (such as openssl and Qt) are installed with homebrew rather than MacPorts.

The build process, in overview:

  1. ensure you have enough disk space
  2. checkout the SVN source tree
  3. download the XCode 4.1 installer from Apple
  4. unpack and extract a portion of the XCode 4.1 installation
  5. build openssl from source (using brew)
  6. configure the build
  7. run the build
  8. fixup the permissions on the build results
  9. load the kernel extensions
  10. start VirtualBox
  11. (optionally) switch back to the official distribution.

Ensure you have enough disk space

To successfully build the VirtualBox tree you are going to need around 10GB of diskspace. 3GB for the XCode 4.1 image, 1.2GB for the partial XCode 4.1 extract and 2.2 GB for the VirtualBox source tree and build output plus some head room required during extraction from the XCode 4.1 disk image.

Checkout the SVN source tree

Run this command to checkout the source code from SVN.

svn co http://www.virtualbox.org/svn/vbox/trunk vbox
cd vbox

The source tree I built was http://www.virtualbox.org/svn/vbox/trunk@51648

The following instructions may need to be adapted if you want to build from a commit earlier than 51645 because that commit introduces support for the --with-xcode-dir option that these instructions use.

Download the XCode 4.1 installer from Apple

On an OSX 10.9 system, Apple wants you to use XCode 5.1. Unfortunately, XCode 5.1 currently can't be used to build the VirtualBox source tree because of two reasons: the source tree doesn't recognize the 10.9 SDK and Xcode 5.1 doesn't provide a true implementation of gcc 4.2 which contains support for compiler options that the Virtual Box build scripts rely on (for example, -fcheck-new).

Fortunately, you can still download earlier versions of XCode from Apple's developer site [4] [registration required].

The file you are looking for is called "Xcode 4.1 for Lion" and should download as a file called "installxcode_41_lion.dmg".

Unpack and extract a portion of the XCode 4.1 installation

Unfortunately the installer in the disk image prevents you from installing this version of XCode on an OSX 10.9 system. Instead, it is necessary to manually (or, at least, semi-automatically) unpack just the parts of XCode 4.1 that are actually needed to compile the VirtualBox source tree.

I've written a script called extract-xcode41.sh [5] that sequences the xar and cpio commands necessary to extract just those dependencies required to build the VirtualBox source tree.

Assuming you have downloaded installxcode_41_lion.dmg and extract-xcode41.sh script into the root of your VirtualBox source tree, then the following command will create a subdirectory called xcode41-partial containing just the necessary dependencies.

./extract-xcode41.sh

For the record, the following is a list of packages that are extracted from the XCode 4.1 installer.

  • DevSDK.pkg
  • MacOSX10.6.pkg
  • MacOSX10.7.pkg
  • XcodeTools.pkg
  • llvm-gcc4.2.pkg
  • gcc4.2.pkg
  • DeveloperTools.pkg

Build openssl from source (using brew)

VirtualBox needs a copy of the openssl source tree. The easiest way to obtain this on a system which uses homebrew is to run the brew installer with the --build-from-source option. So:

brew install openssl --build-from-source

By default, brew puts the openssl source tree into /usr/local/opt/openssl.

I didn't explicitly install Qt myself as part of this build but it appears that the configure script successfully located a copy of qt that had previously be installed by brew. If this doesn't apply to you, you might try:

brew install qt

Configure the build

This step uses a recently introduced option --with-xcode-dir to tell VirtualBox how to find the Xcode SDKs and tools it should use for the build. Per the official guide, I will also use the --disable-hardening option but note the warning in the official build notes against using this option with a build that you intend to distribute to others.

 ./configure --disable-hardening \
      --with-xcode-dir=$(pwd)/xcode41-partial \
      --with-openssl=/usr/local/opt/openssl

Run the build

The build should now be ready to run. You can build it with:

source env.sh
kmk

I found that the build took about 20 minutes to run on my Mac Book Air.

Fixup the permissions

For full functionality, some components of VirtualBox need to run as setuid root. In particular, the host-only network interface will only work correctly if these permissions are set correctly.

So, fixup the permissions as follows:

pushd out/darwin.amd64/release/dist/VirtualBox.app/Contents/MacOS/
for b in VBoxHeadless VBoxNetAdpCtl VBoxNetDHCP VBoxNetNAT VirtualBoxVM; 
do 
    sudo chown root $b; 
    sudo chmod u+s $b; 
done
popd

Load the kernel extensions

VirtualBox needs some kernel extensions to operate correctly. To load these, run the loadall.sh script provided for the purpose.

pushd out/darwin.amd64/release/dist
./loadall.sh
popd

If the load completes successfully, you should see messages like this:

...
load.sh: Successfully unloaded org.virtualbox.kext.VBoxDrv
load.sh: loading /Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxDrv.kext...
.sh: loading /Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxDrv.kext... (kextutil -t "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxDrv.kext")
Diagnostics for /Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxDrv.kext:
Code Signing Failure: not code signed
WARNING - Invalid signature -67062 0xFFFFFFFFFFFEFA0A for kext "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxDrv.kext"
  144    0 0xffffff7f82259000 0x46000    0x46000    org.virtualbox.kext.VBoxDrv (4.3.53) <7 5 4 3 1>
loadusb.sh: loading /Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxUSB.kext... (kextutil -t -d "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxDrv.kext" "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxUSB.kext")
Diagnostics for /Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxUSB.kext:
Code Signing Failure: not code signed
WARNING - Invalid signature -67062 0xFFFFFFFFFFFEFA0A for kext "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxUSB.kext"
  144    1 0xffffff7f82259000 0x46000    0x46000    org.virtualbox.kext.VBoxDrv (4.3.53) <7 5 4 3 1>
  146    0 0xffffff7f822a7000 0x8000     0x8000     org.virtualbox.kext.VBoxUSB (4.3.53) <144 54 34 7 5 4 3 1>
loadnetflt.sh: loading /Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxNetFlt.kext... (kextutil -t -d "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxDrv.kext" "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxNetFlt.kext")
Diagnostics for /Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxNetFlt.kext:
Code Signing Failure: not code signed
WARNING - Invalid signature -67062 0xFFFFFFFFFFFEFA0A for kext "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxNetFlt.kext"
  144    2 0xffffff7f82259000 0x46000    0x46000    org.virtualbox.kext.VBoxDrv (4.3.53) <7 5 4 3 1>
  146    0 0xffffff7f822a7000 0x8000     0x8000     org.virtualbox.kext.VBoxUSB (4.3.53) <144 54 34 7 5 4 3 1>
  147    0 0xffffff7f822af000 0x5000     0x5000     org.virtualbox.kext.VBoxNetFlt (4.3.53) <144 7 5 4 3 1>
loadnetadp.sh: loading /Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxNetAdp.kext... (kextutil -t -d "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxDrv.kext" "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxNetAdp.kext")
Diagnostics for /Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxNetAdp.kext:
Code Signing Failure: not code signed
WARNING - Invalid signature -67062 0xFFFFFFFFFFFEFA0A for kext "/Users/jonseymour/svn/vbox/out/darwin.amd64/release/dist/VBoxNetAdp.kext"
  144    3 0xffffff7f82259000 0x46000    0x46000    org.virtualbox.kext.VBoxDrv (4.3.53) <7 5 4 3 1>
  146    0 0xffffff7f822a7000 0x8000     0x8000     org.virtualbox.kext.VBoxUSB (4.3.53) <144 54 34 7 5 4 3 1>
  147    0 0xffffff7f822af000 0x5000     0x5000     org.virtualbox.kext.VBoxNetFlt (4.3.53) <144 7 5 4 3 1>
  148    0 0xffffff7f82327000 0x6000     0x6000     org.virtualbox.kext.VBoxNetAdp (4.3.53) <144 5 4 1>

The warnings about invalid signatures are expected and not indicative of a problem that should prevent local testing.

Start VirtualBox

To start VirtualBox, change into the directory containing the VirtualBox binary, and run it:

pushd out/darwin.amd64/release/dist/\
VirtualBox.app/Contents/MacOS/
./VirtualBox
popd

Switching back to the official distribution

The following instructions will help you unload the privately built version of VirtualBox (and its kernel extensions) and restore the official distribution.

Copy the following script [6] into a file called 'reload-vbox.sh' somewhere on your path.

#!/usr/bin/env bash
# based on https://gist.github.com/rtgibbons/2024307 with updates from comments of that gist for OSX 10.9

EXTDIR="/Library/Application Support/VirtualBox"

unload() {
    kextstat | grep "org.virtualbox.kext.VBoxUSB" > /dev/null 2>&1 && sudo kextunload -b org.virtualbox.kext.VBoxUSB
    kextstat | grep "org.virtualbox.kext.VBoxNetFlt" > /dev/null 2>&1 && sudo kextunload -b org.virtualbox.kext.VBoxNetFlt
    kextstat | grep "org.virtualbox.kext.VBoxNetAdp" > /dev/null 2>&1 && sudo kextunload -b org.virtualbox.kext.VBoxNetAdp
    kextstat | grep "org.virtualbox.kext.VBoxDrv" > /dev/null 2>&1 && sudo kextunload -b org.virtualbox.kext.VBoxDrv
}

load() {
    sudo kextload "$EXTDIR/VBoxDrv.kext" -r "$EXTDIR"
    sudo kextload "$EXTDIR/VBoxNetFlt.kext" -r "$EXTDIR"
    sudo kextload "$EXTDIR/VBoxNetAdp.kext" -r "$EXTDIR"
    sudo kextload "$EXTDIR/VBoxUSB.kext" -r "$EXTDIR"
}

case "$1" in
    unload|remove)
        unload
    ;;
    load)
        load
    ;;
    *|reload)
        unload
        load
    ;;
esac

Then run:

reload-vbox.sh reload

References

]]><![CDATA[welcome to technically quacking]]>a blog by wild duck theories

"wild duck theories" is the name of a company I established after I resigned from IBM in Februrary 2014 after a career spanning 24 years; this blog "technically quacking" gives voice to my inner "wild duck".

The name, "wild duck theories", was inspired in

]]>https://blog.wildducktheories.com/welcome-to-technically-quacking/925ea96a-c7c3-4f1e-bf89-5c489d08fee5Thu, 05 Jun 2014 18:08:34 GMTa blog by wild duck theories

"wild duck theories" is the name of a company I established after I resigned from IBM in Februrary 2014 after a career spanning 24 years; this blog "technically quacking" gives voice to my inner "wild duck".

The name, "wild duck theories", was inspired in part by a colleague's query, in response to news of my resignation, about whether IBM was losing "its last wild duck". The "theories" part of the name signifies the importance of the scientific method as the foundation of a good problem solving technique. "wild duck theories" will earn at least part of its income by exploiting my abilities to apply the scientific method to the solution of tricky customer problems.

My decision to resign from IBM was not the result of being unhappy with IBM. If anything, at the time I resigned I was having very good fun helping to solve a critical situation at an important customer. Rather, the decision to resign was motivated by an invigorating glimpse of a vision of life beyond IBM, a vision I had never had before. With this vision, I realised that by resigning I would be free to decide how I spent my time each day and free to decide to whom I would assign my intellectual property. Once glimpsed, this vision became irresistable, so my resignation became inevitable.

In the future, I intend to find a balance between doing things I enjoy, such as writing software, and things I need to do to earn income. Ideally, the software I write will help drive business towards a consultancy which in turn will help me earn the income I need to support my software development activities - a virtuous circle.

I intend to use this blog to explore topics that are of interest to me and might be of interest to potential clients. In future months, I will likely write about projects I push to github (such as jsh) , technologies such as Docker and logstash, and how such technologies might be applied to enterprise software stacks, such as IBM WebSphere. I also plan to discuss git workflows and techniques I have found useful in the past.

]]>