Golang on despatches

Projector: Keep YouTube Descriptions synced

Mon, 07 Jul 2025 20:08:21 +0100

In my previous post, I used hugo to generate correctly linked, always up to date descriptions for my YouTube Videos.

But if I’m generating the descriptions automatically… I’m hardly going to be excited about copying and pasting them into YouTube - right? right!

Automating this process brings up a few design choices.

Planning

Which language

There were a few contenders, and here’s how I thought them through:

Zig

I’m currently learning Zig, and I love using it for my game development. But it doesn’t yet have mature libraries for working with the YouTube Data API - and I don’t feel like writing one. So, sadly, Zig’s out for this one.

Python

I used Python for despatches and it was the right fit there - good libraries for BlueSky and Reddit.

However, I did not enjoy the experience:

bazel was a constant struggle
poetry is nice… but still a bit of a nightmare. It just makes the pain more structured
Worst of all: blusky failed after reddit succeeded caused a partial success, which broke the Git commit and silently caused a post to be repeated (embarrassing!)

That kind of problem can happen in Go (nil pointer), though it wouldn’t in Zig. But at least with Go, most handleable errors stay errors — they don’t crash the whole tool.

java

Sure, I could do this in Java - but I really don’t want to mess with the JVM. And more importantly, I’m doing this for fun. Java doesn’t feel like that anymore.

golang

Not quite my favourite any more, but still a close second. It’s fast, has YouTube libraries and it somehow seems fitting that Hugo is also a go baby.

Even though I’m not wiring the two directly, the ecosystem fit is nice.

Overall Plan

Let Hugo render the YouTube description as plain text
Traverse the youtube/*.md files in the source directory
- Skip videos that are too old to update (maybe older than 30 days?)
- Hash the rendered output (title, description, tags, etc.)
- Compare that hash with the one stored in the frontmatter
- If it doesn’t match,
  - Update the metadata on YouTube
  - Update the hash
- commit and push any updates (should be only hash changes)

Validation

One thing worth being careful about is whether the metadata is valid. We do not want the sync to fail during its scheduled run - when it won’t have many choices on how to resolve it.

In a bid to mitigate this, we’ll add a command to validate the source and rendered files.

The validation would expect the rendered files to be generated as well, which seems reasonable since Hugo is probably running as hugo serve while the content files are being updated.

```go func validate(sourcePath string, renderedPath string) error { targetSourceDir, err := getTargetDir(sourcePath) if err != nil { return err } targetRenderedDir, err := getTargetDir(renderedPath) if err != nil { return nil } videos, err := findRecentVideos(targetSourceDir) for _, video := range videos { _, err := video.getDescription(targetRenderedDir) if err != nil { slog.Warn("unable to find rendered file", "file", video.renderedPath) } } return nil } ```

The validate function will retrieve the relevant files and check that there is a corresponding rendered description.

If it errors in that process, we know that it would error out in the sync.

We can’t catch errors around the API though at this stage, and that’s unavoidable.

Sync

Hashing the Description

This part was surprisingly easy:

```go bdesc, err := video.getDescription(targetRenderedDir) if err != nil { slog.Warn("unable to find rendered file", "file", video.renderedPath) } // We want to hash the contents of description // Check with the hash in the metadata to see if it matches hash := md5.Sum(bdesc) strHash := hex.EncodeToString(hash[:]) ```

The challenge was trying to write the updated yaml frontmatter back. I was using the adrg/frontmatter library to read the frontmatter, but it does not support writing it back.

Detour: Write a small frontmatter Library

I took a little detour to build inscribe, a little frontmatter library that supports reading and writing back in yaml.

Auth

We need the YouTube Client to have an OAuth Token, which we can retrieve by:

Create a new OAuth Client - Type of desktop is probably the easiest
add your user account to test users:
go to https://accounts.google.com/o/oauth2/v2/auth?client_id=YOUR_CLIENT_ID&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code&scope=https://www.googleapis.com/auth/youtube
- Remember to substitute your actual client_id
- Add any other scopes you might want
- Go through the flow steps - it’ll warn you that the app is unreleased, which is expected
Take the code that it provides
Call the following curl command

```bash curl -X POST https://oauth2.googleapis.com/token \ -d client_id=YOUR_CLIENT_ID \ -d client_secret=YOUR_CLIENT_SECRET \ -d code=PASTE_THE_CODE_HERE \ -d grant_type=authorization_code \ -d redirect_uri=urn:ietf:wg:oauth:2.0:oob ```

You should finally get something like:

```json { "access_token": "ya29...", "expires_in": 3599, "refresh_token": "1//0g...", "scope": "https://www.googleapis.com/auth/youtube", "token_type": "Bearer" } ```

The refresh_token is what you want to save / use as the access_token will expire (after an hour in this example).

The authentication was a bit more involved with a refresh token, but the oauth2 library helps us out:

```go func NewYouTube(ClientId string, ClientSecret string, RefreshToken string) (YouTube, error) { conf := &oauth2.Config{ ClientID: ClientId, ClientSecret: ClientSecret, Endpoint: google.Endpoint, Scopes: []string{"https://www.googleapis.com/auth/youtube"}, } // Construct a token from just the refresh token token := &oauth2.Token{RefreshToken: RefreshToken} ctx := context.Background() // Create an authenticated client httpClient := conf.Client(ctx, token) ytService, err := youtube.NewService(ctx, option.WithHTTPClient(httpClient)) if err != nil { return YouTube{}, err } return YouTube{ service: ytService, }, nil } ```

Updating the description

Setting the description is a little more complicated because you can’t set just the description.

Everything defined in the VideoSnippet gets updated.

To support this, what we need to do is get the current snippet for the video, then update it:

```go vListCall := ytService.Videos.List([]string{"snippet"}) vListCall = vListCall.Id(videoId) res, err := vListCall.Do() if err != nil { return err } if len(res.Items) != 1 { return fmt.Errorf("wrong number of videos returned: %d", len(res.Items)) } ytVideo := res.Items[0] ytVideo.Snippet.Description = desc vUpdateCall := ytService.Videos.Update([]string{"snippet"}, ytVideo) _, err = vUpdateCall.Do() ```

GitHub Action

The GitHub Action is fairly straightforward, mostly a copy of the Hugo one, then:

Add Bazel
Run projector sync
Commit if changed

```yaml - uses: bazel-contrib/setup-bazel@0.15.0 with: # Avoid downloading Bazel every time. bazelisk-cache: true # Store build cache per workflow. disk-cache: ${{ github.workflow }} # Share repository cache between workflows. repository-cache: true - name: Run projector sync env: GOOGLE_CLIENT_ID: ${{ secrets.PROJECTOR_GOOGLE_CLIENT_ID }} GOOGLE_CLIENT_SECRET: ${{ secrets.PROJECTOR_GOOGLE_CLIENT_SECRET }} GOOGLE_REFRESH_TOKEN: ${{ secrets.PROJECTOR_GOOGLE_REFRESH_TOKEN }} continue-on-error: true run: bazel run //tools/projector:projector -- sync -source blog/content/youtube -rendered blog/public/youtube - name: Commit and push if changed run: | git config user.name "drone-ah bot" git config user.email "github.actions@drone-ah.com" if ! git diff --quiet; then git add -u git commit -m "auto: log youtube updates" git push else echo "No changes to commit" fi ```

We also needed to upgrade one permission - contents

```yaml permissions: contents: write ```

Conclusion

The Google/YouTube documentation was the hardest part here in that it was pretty obtuse and hard to understand.

Writing a little frontmatter library was unexpected, but while it took a little time, was straightforward.

Once I got a handle on that, the rest of it was pretty straightforward, partly because I was reusing parts from before.

Inscribe: Updating frontmatter in-place with Go and yaml.Node

Sat, 05 Jul 2025 10:44:02 +0100

While building a little tool to sync up YouTube video descriptions from hugo, I needed a library to read and write frontmatter in yaml.

I started with adrg/frontmatter before I realised that it didn’t have the ability to write back.

I considered contributing to that one, but writing back is a little more complex than reading - particularly because the frontmatter.Parse in that one is built to support partial reading.

Because adrg/frontmatter only unmarshals into a struct, and doesn’t store the original bytes, you can’t write back without losing untouched keys.

Looking around, I could not find another frontmatter library for Go. I know that python has a decent library which supports writing back to it (I used it in the depatcher but I don’t want to write python.

Multiple Formats

Let’s make it extendable by defining a Format that will allow us to add in other formats later:

```go // A Format knows how to (un)marshal a particular format of frontmatter. // e.g. yaml, toml etc. type Format struct { // TODO: We could add details like delimiter to support other formats // and auto detection of format Unmarshal UnmarshalFunc Marshal MarshalFunc } var yamlFormat = Format{ Unmarshal: yaml.Unmarshal, Marshal: yaml.Marshal, } ```

Writing Back

We could also do with a struct to hold the whole file contents so that we can write it back easier.

```go // A Scribed is a representation of a file that contains frontmatter and markdown content type Scribed struct { format Format frontmatter []byte Content string } ```

By storing the full frontmatter, we can later accept partial updates without losing other keys.

Naive Merging of Updates

We (the user) should be able to update just the keys we care about. All the other keys should be preserved.

The easiest way I could find to do this was to Marshal, Unmarshall and then merge with the raw Unmarshall:

Minimal merge strategy (loses order and formatting)

```go // Merge frontmatter var raw map[string]any // full unmarshalled frontmatter err := s.format.Unmarshal(s.frontmatter, &raw) updatedBytes, _ := yaml.Marshal(fm) // convert updated to yaml var updates map[string]any yaml.Unmarshal(updatedBytes, &updates) // get updated keys as map for k, v := range updates { raw[k] = v // overwrite only touched fields } // raw is now the preserved + updated keys data, err := s.format.Marshal(raw) if err != nil { return err } ```

⚠️ Warning: Key ordering is lost

Due to the way maps work, the key ordering is lost More accurately, the keys are sorted alphabetically during write.

It fully rewrites the frontmatter, which also means that double quotes might disappear etc.

In Place Merging of Updates

If it is important to keep the frontmatter formatting as much as possible, we need to bigger sledgehammer.

I explored an approach using yaml.Node with ChatGPT’s help.

I fitted it into the Format as well

```go type MergeFunc func(raw []byte, fm any) ([]byte, error) type Format struct { // TODO: We could add details like delimiter to support other formats // and auto detection of format Unmarshal UnmarshalFunc Marshal MarshalFunc Merge MergeFunc } func MergeYaml(raw []byte, fm any) ([]byte, error) { var node yaml.Node if err := yaml.Unmarshal(raw, &node); err != nil { return nil, err } // Expecting a document with a single mapping node if len(node.Content) == 0 || node.Content[0].Kind != yaml.MappingNode { return nil, errors.New("invalid frontmatter") } b, err := yaml.Marshal(fm) if err != nil { return nil, err } var updates map[string]string yaml.Unmarshal(b, &updates) m := node.Content[0] for key, value := range updates { // Search for the key and update it, or append a new one found := false for i := 0; i < len(m.Content); i += 2 { k := m.Content[i] if k.Value == key { m.Content[i+1].Value = value found = true break } } if !found { m.Content = append(m.Content, &yaml.Node{Kind: yaml.ScalarNode, Value: key}, &yaml.Node{Kind: yaml.ScalarNode, Value: value}, ) } } var buf bytes.Buffer enc := yaml.NewEncoder(&buf) if err := enc.Encode(&node); err != nil { return nil, err } return buf.Bytes(), nil } ```

This preserves frontmatter formatting well - but it only handles flat YAML. Nested maps require a bit more work.

Supporting maps etc. as values

We need to switch to map[string]any and tweak a bit of the loop

```go var updates map[string]any yaml.Unmarshal(b, &updates) m := node.Content[0] for key, value := range updates { // Encode value to a yaml.Node valNode := &yaml.Node{} if err := valNode.Encode(value); err != nil { return nil, err } // Search and replace or append found := false for i := 0; i < len(m.Content); i += 2 { if m.Content[i].Value == key { m.Content[i+1] = valNode found = true break } } if !found { m.Content = append(m.Content, &yaml.Node{Kind: yaml.ScalarNode, Value: key}, valNode, ) } } ```

It’s funny how things can be more complicated than it first seems.

Minor tweaks

I also wanted to add the delimiter into the format and use that instead of hardcoding, which was easy enough.

```go // A Format knows how to (un)marshal a particular format of frontmatter. // e.g. yaml, toml etc. type Format struct { Delimiter string Unmarshal UnmarshalFunc Marshal MarshalFunc Merge MergeFunc } var yamlFormat = Format{ Delimiter: "---", Unmarshal: yaml.Unmarshal, Marshal: yaml.Marshal, Merge: MergeYaml, } func (s *Scribed) Write(fm any, out io.Writer) error { // Merge frontmatter data, err := s.format.Merge(s.frontmatter, fm) if err != nil { return err } io.WriteString(out, s.format.Delimiter+"\n") out.Write(data) io.WriteString(out, s.format.Delimiter+"\n\n") io.WriteString(out, s.Content) return nil } // splitFrontmatter will split frontmatter from Content and store them func (s *Scribed) splitFrontmatter(r io.Reader) error { data, err := io.ReadAll(r) if err != nil { return err } parts := bytes.SplitN(data, []byte("\n"+s.format.Delimiter+"\n"), 2) if len(parts) != 2 { return errors.New("invalid frontmatter format") } // Remove the opening '---\n' from the first part s.frontmatter = bytes.TrimPrefix(parts[0], []byte("---\n")) s.Content = string(bytes.TrimLeft(parts[1], "\r\n")) return nil } ```

Conclusion

This is probably the main bits of functionality I’ll need to continue with the projector sync.

I had considered frontmatter to be a blackbox with complicated functionality, but cutting it up and working on it has demystified it and made it easier to work with. ChatGPT helped.

It should be fairly straightforward to add other frontmatter formats like TOML, and to autodetect the formats, but I don’t need it right now.

Build and push container image using bazel

Fri, 06 Jun 2025 13:06:14 +0000

I am building a small golang webapp, and I want to push a container up for it, which can eventually be used in Google Cloud Run, or elsewhere.

In this post, I want to describe how I got it to push images build locally up to googles artifact repository. It will include creating the artifac repository using infrastructure as code

The project is in a monorepo that uses bazel.

The executable to be packaged

```starlark # //products/example/cmd/server/BUILD.bazel load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", "go_test") go_library( name = "server_lib", srcs = ["main.go"], visibility = ["//visibility:private"], ) go_binary( name = "server", embed = [":server_lib"], visibility = ["//visibility:public"], ) ```

Build Container Image

add `rules_oci`

rules_oci is a good place to start to integrate container support into your bazel configuration. It’s also worth reading up a bit on distroless if you are not aware of it.

Adding rules_oci into bazel MODULES is straightforward:

```starlark bazel_dep(name = "rules_oci", version = "2.2.6") # For testing, we also recommend https://registry.bazel.build/modules/container_structure_test oci = use_extension("@rules_oci//oci:extensions.bzl", "oci") # Declare external images you need to pull, for example: oci.pull( name = "distroless_base", # 'latest' is not reproducible, but it's convenient. # During the build we print a WARNING message that includes recommended 'digest' and 'platforms' # values which you can use here in place of 'tag' to pin for reproducibility. tag = "latest", image = "gcr.io/distroless/base", platforms = ["linux/amd64"], ) # For each oci.pull call, repeat the "name" here to expose them as dependencies. use_repo(oci, "distroless_base") ```

If you use the WORKSPACE, or if you want the latest version, you can find details on their releases page.

`tar.bzl`

oci_rules uses tar files to build the image. To build tar images, you will want to use tar.bzl.

Add the following to you MODULES

```starlark bazel_dep(name = "tar.bzl", version = "0.3.0") ```

Latest version and instructions for WORKSPACE can be found on their releases page

`BUILD.bazel`

There are language specific sample build files that you can start from.

Starting from the example go BUILD.bazel, and simplifying it, I have:

```starlark # //products/example/deploy/BUILD.bazel load("@rules_go//go:def.bzl", "go_binary", "go_library", "go_test") load("@rules_oci//oci:defs.bzl", "oci_image", "oci_load", "oci_push") load("@tar.bzl", "mutate", "tar") # Put app go_binary into a tar layer. tar( name = "app_layer", srcs = ["//products/muster/cmd/server:server"], out = "app_layer.tar", mutate = mutate(strip_prefix = package_name() + "/app_"), ) oci_image( name = "image", # This is defined by an oci.pull() call in /MODULE.bazel base = "@distroless_base", entrypoint = ["/app"], # Link the resulting image back to the repository where the build is defined. labels = { "org.opencontainers.image.source": "https://github.com/aspect-build/bazel-examples", }, tars = [":app_layer"], ) ```

There is more information at the go doc page including details of migrating from rules_docker

```bash products/example/deploy $ bazel build ... INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured). INFO: Found 2 targets... INFO: Elapsed time: 0.444s, Critical Path: 0.07s INFO: 2 processes: 1 internal, 1 linux-sandbox. INFO: Build completed successfully, 2 total actions ```

Building of the images now completes

Push Image

Pushing the image can be pretty straightforward:

Enable the container repository in Google Cloud
Configure it in the BUILD.bazel file
bazel run to push

I would like to do the whole thing through IaC so that it is

Repeatable, and
more importantly documented

IaC Tool Options

terraform has no bazel support, and is a questionable choice from an ethics perspective. opentofu has

rules_tf. However, that does not support apply

Google Cloud Infrastucture Manager is vendor lock in.

That leaves us with pulumi (If there is another alternative, please let me know).

I like the stacks concept in pulumi, and while it doesn’t have bazel integration, what it does have is an automation api. While not he best documented, there is enough documentation out there to be able to piece it together.

With this, we can build a runnable unit, and then run it too. I’ve experimented with tagging specific runnables and then running them through the CI. I won’t get that far in this post. My focus here is to get it working locally.

Enable Artifact Repository

In pulumi, you can enable the artifact repository with:

```go func enableArtifactRepository(ctx *pulumi.Context) error { // Set your GCP project and region projectID := "" region := "europe-west1" repoName := "" // Create Artifact Registry repository repo, err := artifactregistry.NewRepository(ctx, repoName, &artifactregistry.RepositoryArgs{ Format: pulumi.String("DOCKER"), Location: pulumi.String(region), RepositoryId: pulumi.String(repoName), Description: pulumi.String("Repository for OCI container images"), Project: pulumi.String(projectID), }) if err != nil { return err } // Export the repository URL (used for pushes) ctx.Export("repositoryURL", pulumi.Sprintf("%s-docker.pkg.dev/%s/%s", region, projectID, repo.Name)) return nil } ```

Authenticating against the repo

You need to authenticate against the repository so that the oci_rule can pick it up

```bash gcloud auth configure-docker europe-west1-docker.pkg.dev ```

Define the push target

```starlark oci_push( name = "push", image = ":image", remote_tags = [ "latest", "1h", ], repository = "europe-west1-docker.pkg.dev///", ) ```

Pushing

Once all the above is set up, you can then push the image with one of the following:

```bash bazel run :push # if you in the directory bazel run //:push # fron anywhere in the repo ```

Side note

Please do not take my heavy usage of google and its products to be a fan letter or an endorsement. They might at best, be the lesser evil.

How to run your lambda code locally as its role (for testing)

Tue, 14 Nov 2023 14:23:37 +0000

While AWS Lambda is fantastic in providing a serverless platform with few worries about maintaining servers, it is not the easiest to test in an automated fashion with rapid feedback.

You could write end to end tests, but it means a deployment after each change and then checking the logs to see what failed. Even if you use iac (terraform/pulumi), the deployment will take seconds or a minute or two - not exact rapid test feedback.

What I have been doing is to set up a hook which is called from the lambda handler, which can also be called locally. Within the test, I then assume the role that runs the lambda and then test the hook.

This mechanism allows to me easily test that the permissions are set up correctly and that details are in place for the code to work.

For the full end to end test, I then have a simple smoke test or two.

The code samples are in golang(only because it happens to be my current language of choice), but the idea should be equally applicable in other languages.

Assuming The Role

```go roleToAssume := os.Getenv("AUTH_LAMBDA_ROLE_ARN") ctx := context.TODO() cfg, err := config.LoadDefaultConfig(ctx) if err != nil { logger.Fatal("error: ", err) } // Create the credentials from AssumeRoleProvider to assume the role // referenced by the "myRoleARN" ARN using the MFA token code provided. creds := stscreds.NewAssumeRoleProvider(sts.NewFromConfig(cfg), roleToAssume) logger.Debugf("creds: %v", creds) cfg.Credentials = aws.NewCredentialsCache(creds) ```

the cfg is then passed into the New method for the resource you are interested in. e.g.:

```go ssmClient := ssm.NewFromConfig(cfg) ```

Working Example

You can find a full, working example test in my github repo under post/2023/11/autolambdatest

NOTE: It WILL automatically try and deploy a role and a ssm parameter, and it will delete it after the test.

The BeforeSuite will deploy the minimum configuration to be able to run the test, and the AfterSuite will destroy the same stack.

You will likely need to log into pulumi to get this test to work.

If you run into permissions issue for AssumeRole, read on.

AssumeRole Permissions

For this to work, the user running the tests need to have permissions to AssumeRole.

There are two steps to this. The first part is to allow "anyone" to AssumeRole the relevant role:

```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::root" }, "Action": "sts:AssumeRole" } ] } ```

This will allow "any user" to assume the role, as long as they have the permission to do so.

arn:aws:iam::[your-account]:root is a special user the represents the account (and the non-IAM root user). Since IAM (user, roles etc.) exists under the "root" account, all calls are also authenticated by this account - i.e. all users, roles etc. in IAM is also this account. There is a post on reddit discussing what exactly the root iam principal is for more information

Finally, unless you have the AdministratorAccess policy set against your account, you will also need to attach a policy to the relevant group (or your user) that grants permissions to call sts:AssumeRole (or *)

```json { "Version": "2012-10-17", "Statement": [ { "Sid": "123", "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": ["arn:aws:iam::123456789012:role/desired-role"] } ] } ```

You can of course, also use * for Resource above to allow the user/group to Assume Any role. In practice, you might want to automate this as part of the creation of the relevant roles. (i.e. create the role, then give the relevant group permissions to Assume that role).

Separating out integration tests for golang in Bazel

Mon, 13 Nov 2023 13:47:53 +0000

There are many kinds of automated tests and two main kinds are integration tests and unit tests.

Unit tests are designed to run as fast as possible, so any slower processes like databases are mocked out. While super helpful and powerful in terms of providing confidence in the software, it should be only one part of the testing strategy.

Integration tests, as is implied runs tests of the different part of the software integrated together. Technically speaking, you can still mock out the database and other slower layers to keep it running quickly. However, there is value in including a database or other slower services in the process to test as them in an automated fashion.

What this does mean though, is that you want to be able to run only the unit tests or run the integration tests as well. You might also want to have smoke tests, which are run on your live production environment.

How

You could define a separate target in your BUILD file with the unit tests and let gazelle automatically build your default test target with all the tests. I found this frustrating to use as I had to keep tweaking the dependencies manually whenever anything changed (which happened often)

Tagging

The easiest way to achieve this for golang and bazel is to tag your source code files. You can do this by adding the following to the top of your integration test files

something_integration_test.go

```go //go:build integration_test package somepackage ```

You can pick any tag name you want instead of integration_test like integration, smoke_test etc.

IDE Support

You will likely need to add this source file into the IDEs build constraints to get the IDE to treat it as a source file. In IntelliJ (IDEA/Goland), you will be warned of this

If you click Edit settings, you can add the tag in

When running gazelle, you want to include the files with these tags

Gazelle

```bash bazel run //:gazelle -- -build_tags=integration_test ```

If you have multiple tags, you can separate them with commas. This command will generate a test target with all of the source files and its dependencies

Bazel Integration on test

To run only the unit tests, you test as normal:

```bash bazel test ... # or the specific target, and it'll run only the unit tests ```

To run the integration tests as well, include that tag

```bash bazel test ... --define gotags=integration_test # Will run unit & integration tests ```

Run only Unit Tests

This setup will currently not allow you to run ONLY the integration tests. To be able to do that you'll need to add a unit_test tag to the unit test files so that you can exclude them.

``` something_test.go ``` ```go //go:build integration_test package somepackage ```

You can then run only the unit tests with

```bash bazel test … --define gotags=unit_test # Will run unit & integration tests ```

Only the integration tests

```bash bazel test … --define gotags=integration_test # Will run unit & integration tests ```

Or both:

```bash bazel test … --define gotags=unit_test,integration_test # Will run unit & integration tests ```

Simpler gazelle command

You can enable the tags by default in the BUILD file so that you don't have to pass the tags into gazelle each time.

BUILD

```starlark # gazelle:build_tags unit_test,integration_test ```

You can then just run bazel run //:gazelle which will run with these tags enabled.

Sample Source

You can find sample source code demonstrating this in my github repo, under post/2023/11/separatetests

Streaming Progress With Pulumi Automation API

Wed, 08 Nov 2023 12:38:32 +0000

When using the pulumi automation API, you lose some of the niceties of the pulumi CLI, like having to set up command line args processing and the output is not as friendly or pretty as before. It also doesn't stream the output - though this one is easier to fix.

This is lifted straight out of their golang example code, so if you're working in another language - you should be able to find the relevant code in the same repo

```go // wire up our update to stream progress to stdout stdoutStreamer := optup.ProgressStreams(os.Stdout) // run the update to deploy our fargate web service res, err := stack.Up(ctx, stdoutStreamer) if err != nil { fmt.Printf("Failed to update stack: %v\n\n", err) } ```

How to get current Function URL (aws-lambda + golang)

Wed, 08 Nov 2023 12:10:16 +0000

When deploying a function lambda that needs details of its own function URL. It's an OAuth Callback, and needs to calculate the redirect. There are possible security issues doing it this way, so will switch to http gateway on launch. In the meantime, though, I ran into a bit of a chicken and egg problem.

In Pulumi, the function URL is created after the function (and even otherwise), I can't pass the output of the lambda (or lambdaFunctionUrl) back in as an environment variable. Fortunately, there is an easy way to pick up the Function URL (or the function name for that matter) - if you know how ;)

```wp-block-syntaxhighlighter-code domainName := request.RequestContext.DomainName funcName := os.Getenv("AWS_LAMBDA_FUNCTION_NAME") return fmt.Sprintf("Domain: %s, funcName: %s", domainName, funcName), nil ```

There are other defined lambda function environment variables as well that you can use.

Bazel + Pulumi Automation API. Deploying an inline stack along with Pulumi.[stack].yaml

Tue, 07 Nov 2023 15:57:52 +0000

I believe in CI/CD/CD as in Continuous, integration, delivery and deployment. As part of this, I am setting up a workflow where on merge to develop (or main/trunk), the deployment is triggered automatically. Pulumi deploys the current state of code and infrastructure through GitHub actions and OpenID Connect(OIDC) as part of the GitHub Action.

I used to configure Pulumi to be triggered directly from the build process, but bazel (as far I know), does not support Pulumi. When I used pants, there was a custom module, developed by one of the community members which did support pulumi (You might have to ask in the slack channel if you're interested), but they stopped maintaining it as they moved to the Pulumi Automation API.

I am using Automation API from the start, and configuring a "deployer" per product/project within the monorepo. The intention is for the deployer to be as smart as possible - eventually up-ing only the stacks that have changes since the last time- but that's a way down the line.

Another benefit from the Automation API is to pick up the stack outputs automatically when running integration/e2e tests, making the test configuration smoother.

The first step is to be able to define a stack within a product, hook it into the main iac executable and have it working. My directory structure is roughly as below: While I am using golang as my language of choice, it's probably not hugely different in other languages.

products
- productA
  - auth
    - iac_auth
      - BUILD
      - deploy.go
      - Pulumi.dev.yaml
      - Pulumi.yaml
    - OtherAuthModules
  - iac
    - BUILD
    - main.go

`iac_auth/BUILD`

```wp-block-syntaxhighlighter-code # load etc. go_library( name = "iac_auth", srcs = ["deploy.go"], data = glob([ "Pulumi.*.yaml", ]) + [ "Pulumi.yaml", ], visibility = ["//visibility:public"], deps = [ # dependencies automated with gazelle ], ) ```

`iac_auth/deploy.go`

```wp-block-syntaxhighlighter-code package iac_sync import ( "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) const moduleName = "auth" func DeployGithubLambda(ctx *pulumi.Context) error { fmt.Println("Deploy") conf := config.Require(ctx, "key") fmt.Printf("conf: %s\n", conf) // Will successfully pick up the value if key is in Pulumi..yaml // Actual deployment code here return nil } ```

`iac/BUILD`

```wp-block-syntaxhighlighter-code # other config including standard config of :iac_lib by gazelle go_binary( name = "iac", args = [ "-iac_auth", "$(location //products/productA/auth/iac_auth)", ], data = [ "//products/gitolink/productA/iac_auth", ], embed = [":iac_lib"], visibility = ["//visibility:public"], ) ```

Worth noting the args bit, which is what we use to identify the path where the Pulumi.yaml and Pulumi..yaml files are:

`iac/main.go`

```wp-block-syntaxhighlighter-code package main import ( "context" "flag" "fmt" iacAuth "github.com/drone-ah/monorepo/products/gitolink/auth/iac_auth" "github.com/pulumi/pulumi/sdk/v3/go/auto" "path/filepath" ) func main() { // Pick up the path from args var iacAuthPath = flag.String("iac_auth", "", "bin for iac_auth") flag.Parse() fmt.Printf("param iac: %s \n", *iacAuthPath) ctx := context.Background() projectName := "gitolink" stackName := "dev" stack, err := auto.NewStackInlineSource(ctx, stackName, projectName, iacAuth.DeployGithubLambda, // Define workdir for locatin of the Pulumi yaml files auto.WorkDir(filepath.Dir(*iacAuthPath))) preview, err := stack.Preview(ctx) fmt.Println(err) fmt.Println(preview) } ```

You might also have to use RLocation (see my previous post about including an artifact in another target for an example of this), though when I tried it, it was missing one of the yaml files and I didn't investigate further.

Including a built artifact in another target (Bazel, golang)

Wed, 01 Nov 2023 19:42:30 +0000

We use pulumi to do IaC and we use a monorepo with Bazel as the build tool. We have out modules set out as following

One of the requirements we have is to build a lambda module and then deploy it. The lambda module is a target being built by Bazel (golang, but shouldn't matter):

```go go_binary( name = "lambda_module", visibility = ["//visibility:public"], ) ```

We then have the iac module, which should get the built version of the above module, so that it can then upload it into lambda

```go go_binary( name = "iac", args = [ "-lambda_module", "$(location //products/productA/module/lambda_module)", ], data = ["//products/productA/module/lambda_module"], visibility = ["//visibility:public"], ) ```

There are two key parameters here to note:

args: We generate the path to the target module using //products/productA/module/lambda_module)
data: We use the data tag to ensure that the built output is included when building/running this target

We then need to use runfiles support within golang to be ablet to identify the correct location for the built binary. The reason this part is complex is to be able to support multiple operating systems. I should caveat that I have only got this working on Linux, but Mac/Win shouldn't be too different.

```go package main import ( _ "embed" "flag" "fmt" "github.com/bazelbuild/rules_go/go/runfiles" "path/filepath" ) func main() { var webhookAuth = flag.String("webhook_auth", "", "bin for webhook_auth") flag.Parse() fmt.Printf("param : %s \n", *webhookAuth) path, err := runfiles.Rlocation(fmt.Sprintf("workspace_name/%s", *webhookAuth)) fmt.Printf("rLoc path: %s, err: %v \n", path, err) symlinks, err := filepath.EvalSymlinks(path) fmt.Printf("evaluated path: %s, err: %v \n", symlinks, err) } ```

We use the flag module to retrieve the path passed in as a runtime parameter

We use runfiles.Rlocation to pick up the "real" path to the file, prepending the workspace name to the start. You can define the workspace name in the root WORKSPACE file with workspace(name = "workspace_name")

Finally, resolve the Symlink to get the actual file path

References

There are similar mechanisms to find the rLocation in other languages, a couple of which are described in its design document

There is some documentation in rules_go around accessing go_binary from go_test which I referenced and updated to get the above example

I found the above link from a stackoverflow post about feeding bazel output to another bazel rule

Golang on despatches

Projector: Keep YouTube Descriptions synced

Planning

Which language

Zig

Python

java

golang

Overall Plan

Validation

Sync

Hashing the Description

Detour: Write a small frontmatter Library

Auth

Updating the description

GitHub Action

Conclusion

Links

Inscribe: Updating frontmatter in-place with Go and yaml.Node

Multiple Formats

Writing Back

Naive Merging of Updates

Minimal merge strategy (loses order and formatting)

In Place Merging of Updates

Supporting maps etc. as values

Minor tweaks

Conclusion

Links

Build and push container image using bazel

The executable to be packaged

Build Container Image

add rules_oci

tar.bzl

BUILD.bazel

Push Image

IaC Tool Options

Enable Artifact Repository

Authenticating against the repo

Define the push target

Pushing

Side note

How to run your lambda code locally as its role (for testing)

Assuming The Role

Working Example

AssumeRole Permissions

Separating out integration tests for golang in Bazel

How

Tagging

IDE Support

Gazelle

Bazel Integration on test

Run only Unit Tests

Simpler gazelle command

Sample Source

Streaming Progress With Pulumi Automation API

How to get current Function URL (aws-lambda + golang)

Bazel + Pulumi Automation API. Deploying an inline stack along with Pulumi.[stack].yaml

iac_auth/BUILD

iac_auth/deploy.go

iac/BUILD

iac/main.go

Including a built artifact in another target (Bazel, golang)

References

add `rules_oci`

`tar.bzl`

`BUILD.bazel`

`iac_auth/BUILD`

`iac_auth/deploy.go`

`iac/BUILD`

`iac/main.go`