Help-Wanted: Role = build and shell cleanup


Your topic should be the title: “SIG Foo is looking for a release manager”

Sponsoring SIG: Probably sig-release and sig-arch crossover

Mentors/Point of Contact: @thockin


Our build scripts rely heavily on some bash code. Every time I look at it, I have to fight the urge to streamline or even rewrite big chunks. Some of the work would be easy, some would be very challenging. We have some less-than-pleasant shell code in there that seems like it could be reduced to less complex, more reliable forms. Maybe not even shell!

Note: I am not talking about the Makefiles (though that is an opportunity, too) but the scripts that get called to actually build things, e.g. hack.lib/

If you enjoy shell scripting (or understand it well enough to want to see it get reduced or replaced), this is a good opportunity to help reduce some debt. I won’t force anyone into OWNERship if they don’t want it but it’s a possibility. So you can consider this a short-term opportunity or a long-term one.

Time Commitment: a few weeks worth of somewhat high-intensity work, followed by (optional) perpetual vigilance and ownership.

Skills Required: shell scripting, build-tooling, good taste, docker. Less critically Go tooling (can be learned), Go, and make.

k8s Membership Required: n


Hi Tim,

I’d be interested in taking on this work.

What would be the best way of getting started?



For the audience, conversation started on slack :slight_smile:


Working on this today.

1 Like

I love shell scripting, be glad to help too. @RobKielty @thockin


I’m not sure if that counts as a point in your favor or against you :slight_smile:

I have a love-hate thing with shell. I see the value and the ease of it, but it can get out of hand quickly :slight_smile:

1 Like

I agree with Tim, bash scripts offer convenience but they can get out of hand.

On the positive side, the scripts in the hack folder are :

  1. consistent and relatively neat and tidy
  2. checked statically using shellcheck
  3. the are in place and working, so that has to be taken into account

The smells associated the scripts in the hack directory are

  1. there are too many of them
  2. the scripts are too long
  3. they are resource hungry, forky and execy via subshells and pipes

In addition, Tim has specific annoyances based on his experience with same.

I have found one inaccuracy in
in relation to shellcheck’s SC1090 warning

# disabled lints
  # this lint disallows non-constant source, which we use extensively without
  # any known bugs
  # this lint prefers command -v to which, they are not the same

have done some work on adding the shellcheck directive to eliminate the warning where I can. (This may or may not be suitable as a first PR)

shellcheck emits a warning when it cannot follow a path to a shell script when that path is only known at run time;

the reason for emitting this warning is that shellcheck would like to read and check the invoked “sub” script but cannot do so because it cannot determine the path at build time.

The warning relates to shellcheck’s operation rather than anything fundamentally wrong with dynamically invoking other shell scripts. A lot of the scripts define a KUBE_ROOT variable that is used in paths to either library scripts that are present in the working tree or sometime user supplied scripts (which I cannot find in the working tree) So where I can find a script in the working tree I can add the shellcheck source directive as follows to remove the warning

Here’s an example:

KUBE_ROOT=(dirname "{BASH_SOURCE[0]}")/…
# shellcheck source=…/build/
source “${KUBE_ROOT}/build/”

However before going down that rat hole I have questions for Tim.

  1. How, in general are the scripts invoked? Could you provide some sign posting as to how I could trace the invocation of some or one of the scripts? Are they invoked from jobs kicked off by PROW etc?

  2. What are the specific pain points for you when you look at and use these scripts?

  3. Why do the scripts have to exist at all? In general, I view bash script invocation to get work done in a CI/CD process as at best solving a corner case where a CI/CD orchestrator has a deficiency or a yet to be implemented piece of functionality. I suppose what I need here is an overview of PROW jobs and bazel and the Makefiles and how they all hang together. For sure, I can imagine if there’s a lack of community knowledge on PROW and bazel and someone said well here’s how you can add a bash script to get a step done then that’s what people went with. I suppose my question is, can we make better use of bazel to eliminate the use of bash scripts? Or can we look at implementing pipelining the CI steps in some other way?

1 Like

Invocation: Some are invoked via CI - e.g. verify-. Some are run by humans before sending PRs (e.g. update-). Some are part of the build (e.g. make-rules/* and lib/*).

Pain points, to name a few:

  • inconsistency and under-documented behaviors. functions that “accept” or “return” array-like data but don’t document the assumptions. e.g. are they whitespace safe? Do they produce one line per array record? How should I capture the results of such a function? E.g. mapfile vs the stuff that mtaifen just did vs Christoph’s stuff. Basically anything with arrays.
  • under-documented functions that do not explain what globals they use or produce
  • Abuse of default values in places where a default doesn’t make sense. e.g. {foo:-} where a value is required or "{array[@]:+${array[@]}}" stupidness
  • failures like lack of jq not failing update-vendor even though it calls kube::util::require-jq
  • use of :: is symbol names

Bazel: we have taken a position that we keep bazel working for CI but do not force users to use it. So we have bothe Makefile and bazel, and I am not 100% sure they are the same (e.g. for generated code Makefile will regen it all on the fly – I don’t know if Bazel is “finished” with that).

Wish list stuff: simplify the build. Maybe more to a docker-based build as the default. Maybe move it out of shell and into a Go program. It’s awful because of the docker data container, but that is because Mac sucks at docker, and some people insist on using Mac.

1 Like

I would as a pain point the inconsistency between bash versions , that´s even worse if you add Mac OSX to the picture that comes with Bash 3 and the rest of the universe is using Bash 4 :slight_smile:

1 Like

Can we simply require MacOS users install bash 4 via HomeBrew or something?



Would love to see that.

Not to get too off topic, but I am really curious how long apple will stick with 3.x At some point they’ll have to give or go some other route.

1 Like

It has to do with licensing. bash 3 is GPLv2 and bash 4+ is GPLv3.