Pulumi on despatches

Use `protoc` bin instead of building from source in `bazel`

Mon, 09 Jun 2025 20:57:53 +0000

I work on a project that uses pulumi automation api, which in turn uses protobuf. I think there are other bits in bazel that also uses it.

For a while, it was fine - except that some CI runs would take 15 minutes and we couldn’t quite figure out why.

I finally had to update bazel from 7.x to 8 and in that process (which honestly could have been easier, but oh well), I ran into a problem with compiling protobuf.

Namely, I kept running into this error:

``` error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory ```

I tried many things but was not able to get past this error. In the end, I was able to narrow it down to devbox/nix and how it messes with the environment that doesn’t quite agree with bazel.

What didn’t work

installing gcc from devbox
installing stdenv.cc.cc.lib
Setting LD_LIBRARY_PATH (to /.devbox/nix/profile/default/lib, which is where the devbo version of libstdc++.so.6 is installed by stdenv.cc.cc.lib)
Using the --action-env=
rules-nixpkgs

None of which really worked.

Disabling devbox did work, which was at least partly encouraging. I also found a bunch of evidence online around this issue (1, 2, 3, 4, 5). Bazel seems to generally dislike non-system gcc.

A ray of hope

I was just about ready to throw in the towel when ChatGPT, in passing suggested using the protobuf binary directly. I wasn’t even using this binary - it was being pulled in as a dependency and I just needed it to work. I did not need it to be built from source.

Furthermore, I found, from my research into trying to solve this that building protobuf was the likely culprit in the CI taking 15 minutes.

I also remembered that someone else on the team had issues trying to get it to build on a mac at some point.

All in all, it was a problematic piece of software and I was seriously considering switching to opentofu - but they might be using it too.

bin `protoc`

It was difficult to find decent documentation about how to achieve this though, apart from a partially answered on:

stackoverflow question,
google groups for bazel-discuss
- which also led me to a gitlab repo with some code

These resources gave me just enough to be able to cobble together a working solution, which needs:

Install `protoc`

The first thing we need is a locally installed protoc bin (I’ll used devbox to install it for consistency across dev environments)

```bash devbox add protobuf ```

You can add a version specifier for better reproducibility.

Runnable Target

We also need a shell target that will execute this correctly. The bazel flag to override protoc takes a local target, not a bin.

third_party/tools/BUILD.bazel

```python package(default_visibility = ["//visibility:public"]) sh_binary( name = "protoc", srcs = ["protoc.sh"], ) # https://github.com/protocolbuffers/protobuf/blob/b4b0e304be5a68de3d0ee1af9b286f958750f5e4/BUILD#L773 proto_lang_toolchain( name = "cc_toolchain", command_line = "--cpp_out=$(OUT)", runtime = ":protoc", visibility = ["//visibility:public"], ) ```

third_party/tools/protoc.sh

```bash #!/bin/env bash protoc "$@" ```

Override protoc

You should now be able to override protoc with:

```bash bazel build --proto_compiler=//third_party/tools:protoc ... ```

Having to pass in a flag each time is annoying though, and you can add it to your .bazelrc

``` build --proto_compiler=//third_party/tools:protoc ```

Sample code

You can find the sample code in the wordsonsand repo which uses this exact solution to override protoc and get the pulumi sample code to work ;)

PS: It also includes a fully migrated MODULES.bazel with support for golang. You will also want to check out BUILD in the root.

Build and push container image using bazel

Fri, 06 Jun 2025 13:06:14 +0000

I am building a small golang webapp, and I want to push a container up for it, which can eventually be used in Google Cloud Run, or elsewhere.

In this post, I want to describe how I got it to push images build locally up to googles artifact repository. It will include creating the artifac repository using infrastructure as code

The project is in a monorepo that uses bazel.

The executable to be packaged

```starlark # //products/example/cmd/server/BUILD.bazel load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", "go_test") go_library( name = "server_lib", srcs = ["main.go"], visibility = ["//visibility:private"], ) go_binary( name = "server", embed = [":server_lib"], visibility = ["//visibility:public"], ) ```

Build Container Image

add `rules_oci`

rules_oci is a good place to start to integrate container support into your bazel configuration. It’s also worth reading up a bit on distroless if you are not aware of it.

Adding rules_oci into bazel MODULES is straightforward:

```starlark bazel_dep(name = "rules_oci", version = "2.2.6") # For testing, we also recommend https://registry.bazel.build/modules/container_structure_test oci = use_extension("@rules_oci//oci:extensions.bzl", "oci") # Declare external images you need to pull, for example: oci.pull( name = "distroless_base", # 'latest' is not reproducible, but it's convenient. # During the build we print a WARNING message that includes recommended 'digest' and 'platforms' # values which you can use here in place of 'tag' to pin for reproducibility. tag = "latest", image = "gcr.io/distroless/base", platforms = ["linux/amd64"], ) # For each oci.pull call, repeat the "name" here to expose them as dependencies. use_repo(oci, "distroless_base") ```

If you use the WORKSPACE, or if you want the latest version, you can find details on their releases page.

`tar.bzl`

oci_rules uses tar files to build the image. To build tar images, you will want to use tar.bzl.

Add the following to you MODULES

```starlark bazel_dep(name = "tar.bzl", version = "0.3.0") ```

Latest version and instructions for WORKSPACE can be found on their releases page

`BUILD.bazel`

There are language specific sample build files that you can start from.

Starting from the example go BUILD.bazel, and simplifying it, I have:

```starlark # //products/example/deploy/BUILD.bazel load("@rules_go//go:def.bzl", "go_binary", "go_library", "go_test") load("@rules_oci//oci:defs.bzl", "oci_image", "oci_load", "oci_push") load("@tar.bzl", "mutate", "tar") # Put app go_binary into a tar layer. tar( name = "app_layer", srcs = ["//products/muster/cmd/server:server"], out = "app_layer.tar", mutate = mutate(strip_prefix = package_name() + "/app_"), ) oci_image( name = "image", # This is defined by an oci.pull() call in /MODULE.bazel base = "@distroless_base", entrypoint = ["/app"], # Link the resulting image back to the repository where the build is defined. labels = { "org.opencontainers.image.source": "https://github.com/aspect-build/bazel-examples", }, tars = [":app_layer"], ) ```

There is more information at the go doc page including details of migrating from rules_docker

```bash products/example/deploy $ bazel build ... INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured). INFO: Found 2 targets... INFO: Elapsed time: 0.444s, Critical Path: 0.07s INFO: 2 processes: 1 internal, 1 linux-sandbox. INFO: Build completed successfully, 2 total actions ```

Building of the images now completes

Push Image

Pushing the image can be pretty straightforward:

Enable the container repository in Google Cloud
Configure it in the BUILD.bazel file
bazel run to push

I would like to do the whole thing through IaC so that it is

Repeatable, and
more importantly documented

IaC Tool Options

terraform has no bazel support, and is a questionable choice from an ethics perspective. opentofu has

rules_tf. However, that does not support apply

Google Cloud Infrastucture Manager is vendor lock in.

That leaves us with pulumi (If there is another alternative, please let me know).

I like the stacks concept in pulumi, and while it doesn’t have bazel integration, what it does have is an automation api. While not he best documented, there is enough documentation out there to be able to piece it together.

With this, we can build a runnable unit, and then run it too. I’ve experimented with tagging specific runnables and then running them through the CI. I won’t get that far in this post. My focus here is to get it working locally.

Enable Artifact Repository

In pulumi, you can enable the artifact repository with:

```go func enableArtifactRepository(ctx *pulumi.Context) error { // Set your GCP project and region projectID := "" region := "europe-west1" repoName := "" // Create Artifact Registry repository repo, err := artifactregistry.NewRepository(ctx, repoName, &artifactregistry.RepositoryArgs{ Format: pulumi.String("DOCKER"), Location: pulumi.String(region), RepositoryId: pulumi.String(repoName), Description: pulumi.String("Repository for OCI container images"), Project: pulumi.String(projectID), }) if err != nil { return err } // Export the repository URL (used for pushes) ctx.Export("repositoryURL", pulumi.Sprintf("%s-docker.pkg.dev/%s/%s", region, projectID, repo.Name)) return nil } ```

Authenticating against the repo

You need to authenticate against the repository so that the oci_rule can pick it up

```bash gcloud auth configure-docker europe-west1-docker.pkg.dev ```

Define the push target

```starlark oci_push( name = "push", image = ":image", remote_tags = [ "latest", "1h", ], repository = "europe-west1-docker.pkg.dev///", ) ```

Pushing

Once all the above is set up, you can then push the image with one of the following:

```bash bazel run :push # if you in the directory bazel run //:push # fron anywhere in the repo ```

Side note

Please do not take my heavy usage of google and its products to be a fan letter or an endorsement. They might at best, be the lesser evil.

How to run your lambda code locally as its role (for testing)

Tue, 14 Nov 2023 14:23:37 +0000

While AWS Lambda is fantastic in providing a serverless platform with few worries about maintaining servers, it is not the easiest to test in an automated fashion with rapid feedback.

You could write end to end tests, but it means a deployment after each change and then checking the logs to see what failed. Even if you use iac (terraform/pulumi), the deployment will take seconds or a minute or two - not exact rapid test feedback.

What I have been doing is to set up a hook which is called from the lambda handler, which can also be called locally. Within the test, I then assume the role that runs the lambda and then test the hook.

This mechanism allows to me easily test that the permissions are set up correctly and that details are in place for the code to work.

For the full end to end test, I then have a simple smoke test or two.

The code samples are in golang(only because it happens to be my current language of choice), but the idea should be equally applicable in other languages.

Assuming The Role

```go roleToAssume := os.Getenv("AUTH_LAMBDA_ROLE_ARN") ctx := context.TODO() cfg, err := config.LoadDefaultConfig(ctx) if err != nil { logger.Fatal("error: ", err) } // Create the credentials from AssumeRoleProvider to assume the role // referenced by the "myRoleARN" ARN using the MFA token code provided. creds := stscreds.NewAssumeRoleProvider(sts.NewFromConfig(cfg), roleToAssume) logger.Debugf("creds: %v", creds) cfg.Credentials = aws.NewCredentialsCache(creds) ```

the cfg is then passed into the New method for the resource you are interested in. e.g.:

```go ssmClient := ssm.NewFromConfig(cfg) ```

Working Example

You can find a full, working example test in my github repo under post/2023/11/autolambdatest

NOTE: It WILL automatically try and deploy a role and a ssm parameter, and it will delete it after the test.

The BeforeSuite will deploy the minimum configuration to be able to run the test, and the AfterSuite will destroy the same stack.

You will likely need to log into pulumi to get this test to work.

If you run into permissions issue for AssumeRole, read on.

AssumeRole Permissions

For this to work, the user running the tests need to have permissions to AssumeRole.

There are two steps to this. The first part is to allow "anyone" to AssumeRole the relevant role:

```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::root" }, "Action": "sts:AssumeRole" } ] } ```

This will allow "any user" to assume the role, as long as they have the permission to do so.

arn:aws:iam::[your-account]:root is a special user the represents the account (and the non-IAM root user). Since IAM (user, roles etc.) exists under the "root" account, all calls are also authenticated by this account - i.e. all users, roles etc. in IAM is also this account. There is a post on reddit discussing what exactly the root iam principal is for more information

Finally, unless you have the AdministratorAccess policy set against your account, you will also need to attach a policy to the relevant group (or your user) that grants permissions to call sts:AssumeRole (or *)

```json { "Version": "2012-10-17", "Statement": [ { "Sid": "123", "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": ["arn:aws:iam::123456789012:role/desired-role"] } ] } ```

You can of course, also use * for Resource above to allow the user/group to Assume Any role. In practice, you might want to automate this as part of the creation of the relevant roles. (i.e. create the role, then give the relevant group permissions to Assume that role).

Streaming Progress With Pulumi Automation API

Wed, 08 Nov 2023 12:38:32 +0000

When using the pulumi automation API, you lose some of the niceties of the pulumi CLI, like having to set up command line args processing and the output is not as friendly or pretty as before. It also doesn't stream the output - though this one is easier to fix.

This is lifted straight out of their golang example code, so if you're working in another language - you should be able to find the relevant code in the same repo

```go // wire up our update to stream progress to stdout stdoutStreamer := optup.ProgressStreams(os.Stdout) // run the update to deploy our fargate web service res, err := stack.Up(ctx, stdoutStreamer) if err != nil { fmt.Printf("Failed to update stack: %v\n\n", err) } ```

Bazel + Pulumi Automation API. Deploying an inline stack along with Pulumi.[stack].yaml

Tue, 07 Nov 2023 15:57:52 +0000

I believe in CI/CD/CD as in Continuous, integration, delivery and deployment. As part of this, I am setting up a workflow where on merge to develop (or main/trunk), the deployment is triggered automatically. Pulumi deploys the current state of code and infrastructure through GitHub actions and OpenID Connect(OIDC) as part of the GitHub Action.

I used to configure Pulumi to be triggered directly from the build process, but bazel (as far I know), does not support Pulumi. When I used pants, there was a custom module, developed by one of the community members which did support pulumi (You might have to ask in the slack channel if you're interested), but they stopped maintaining it as they moved to the Pulumi Automation API.

I am using Automation API from the start, and configuring a "deployer" per product/project within the monorepo. The intention is for the deployer to be as smart as possible - eventually up-ing only the stacks that have changes since the last time- but that's a way down the line.

Another benefit from the Automation API is to pick up the stack outputs automatically when running integration/e2e tests, making the test configuration smoother.

The first step is to be able to define a stack within a product, hook it into the main iac executable and have it working. My directory structure is roughly as below: While I am using golang as my language of choice, it's probably not hugely different in other languages.

products
- productA
  - auth
    - iac_auth
      - BUILD
      - deploy.go
      - Pulumi.dev.yaml
      - Pulumi.yaml
    - OtherAuthModules
  - iac
    - BUILD
    - main.go

`iac_auth/BUILD`

```wp-block-syntaxhighlighter-code # load etc. go_library( name = "iac_auth", srcs = ["deploy.go"], data = glob([ "Pulumi.*.yaml", ]) + [ "Pulumi.yaml", ], visibility = ["//visibility:public"], deps = [ # dependencies automated with gazelle ], ) ```

`iac_auth/deploy.go`

```wp-block-syntaxhighlighter-code package iac_sync import ( "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) const moduleName = "auth" func DeployGithubLambda(ctx *pulumi.Context) error { fmt.Println("Deploy") conf := config.Require(ctx, "key") fmt.Printf("conf: %s\n", conf) // Will successfully pick up the value if key is in Pulumi..yaml // Actual deployment code here return nil } ```

`iac/BUILD`

```wp-block-syntaxhighlighter-code # other config including standard config of :iac_lib by gazelle go_binary( name = "iac", args = [ "-iac_auth", "$(location //products/productA/auth/iac_auth)", ], data = [ "//products/gitolink/productA/iac_auth", ], embed = [":iac_lib"], visibility = ["//visibility:public"], ) ```

Worth noting the args bit, which is what we use to identify the path where the Pulumi.yaml and Pulumi..yaml files are:

`iac/main.go`

```wp-block-syntaxhighlighter-code package main import ( "context" "flag" "fmt" iacAuth "github.com/drone-ah/monorepo/products/gitolink/auth/iac_auth" "github.com/pulumi/pulumi/sdk/v3/go/auto" "path/filepath" ) func main() { // Pick up the path from args var iacAuthPath = flag.String("iac_auth", "", "bin for iac_auth") flag.Parse() fmt.Printf("param iac: %s \n", *iacAuthPath) ctx := context.Background() projectName := "gitolink" stackName := "dev" stack, err := auto.NewStackInlineSource(ctx, stackName, projectName, iacAuth.DeployGithubLambda, // Define workdir for locatin of the Pulumi yaml files auto.WorkDir(filepath.Dir(*iacAuthPath))) preview, err := stack.Preview(ctx) fmt.Println(err) fmt.Println(preview) } ```

You might also have to use RLocation (see my previous post about including an artifact in another target for an example of this), though when I tried it, it was missing one of the yaml files and I didn't investigate further.

Including a built artifact in another target (Bazel, golang)

Wed, 01 Nov 2023 19:42:30 +0000

We use pulumi to do IaC and we use a monorepo with Bazel as the build tool. We have out modules set out as following

One of the requirements we have is to build a lambda module and then deploy it. The lambda module is a target being built by Bazel (golang, but shouldn't matter):

```go go_binary( name = "lambda_module", visibility = ["//visibility:public"], ) ```

We then have the iac module, which should get the built version of the above module, so that it can then upload it into lambda

```go go_binary( name = "iac", args = [ "-lambda_module", "$(location //products/productA/module/lambda_module)", ], data = ["//products/productA/module/lambda_module"], visibility = ["//visibility:public"], ) ```

There are two key parameters here to note:

args: We generate the path to the target module using //products/productA/module/lambda_module)
data: We use the data tag to ensure that the built output is included when building/running this target

We then need to use runfiles support within golang to be ablet to identify the correct location for the built binary. The reason this part is complex is to be able to support multiple operating systems. I should caveat that I have only got this working on Linux, but Mac/Win shouldn't be too different.

```go package main import ( _ "embed" "flag" "fmt" "github.com/bazelbuild/rules_go/go/runfiles" "path/filepath" ) func main() { var webhookAuth = flag.String("webhook_auth", "", "bin for webhook_auth") flag.Parse() fmt.Printf("param : %s \n", *webhookAuth) path, err := runfiles.Rlocation(fmt.Sprintf("workspace_name/%s", *webhookAuth)) fmt.Printf("rLoc path: %s, err: %v \n", path, err) symlinks, err := filepath.EvalSymlinks(path) fmt.Printf("evaluated path: %s, err: %v \n", symlinks, err) } ```

We use the flag module to retrieve the path passed in as a runtime parameter

We use runfiles.Rlocation to pick up the "real" path to the file, prepending the workspace name to the start. You can define the workspace name in the root WORKSPACE file with workspace(name = "workspace_name")

Finally, resolve the Symlink to get the actual file path

References

There are similar mechanisms to find the rLocation in other languages, a couple of which are described in its design document

There is some documentation in rules_go around accessing go_binary from go_test which I referenced and updated to get the above example

I found the above link from a stackoverflow post about feeding bazel output to another bazel rule