Writing UDFs in Golang

Seafowl has supported UDFs for some time now, and they continue to offer an exciting way for Seafowl users to significantly extend functionality in a fully sandboxed environment. Recently, in response to user request, we added a Golang example.

See the UDF docs.

To learn more about compiling Golang to WASM, read on.

Handling UDF requirements

There are certain requirements incumbent on the UDF implementer when it comes to running UDFs in Seafowl, and Golang is included. Today we cover a partial list:

• reading data in from the host through linear raw memory
• returning the computed result as a msgpack-encoded buffer
• working around Golang default syntax (e.g. TitleCase function exports)
• compiling Golang into WASM

Reading data in

As discussed previously, when you register a Seafowl UDF it's required to specify certain parameters including e.g. wasm_export (e.g. addi64), return_type (e.g. BIGINT), input_types (e.g. BIGINT).

This is important because raw memory is the medium for passing data between Seafowl and the UDF. By specifying the parameter types upfront at UDF registration time, both Seafowl host and UDF know how to properly encode/decode the data they share with each other. That's why the function addi64, expects a pointer to a buffer vs conventional Go function params.

Writing data out

Conversely, when we return the result to Seafowl, we need to return a buffer that includes the length of the result first, with the actual msgpack-encoded result appended afterwards.

This allows Seafowl's query engine to return the result of the UDF in the typical way.

TitleCase function export workaround

Golang conventionally uses TitleCase to indicate when a function should be exported from its module scope. Some may find it cute and this convention is presumably fine in traditional Golang, but in certain contexts like UDFs, we are required to provide alloc/dealloc functions in lower case - which poses a fun hurdle.

Fortunately, while researching how tinygo works, I learned it supports annotating functions regardless of casing via //export $functionName

For example:

//export alloc
func alloc(size uintptr) unsafe.Pointer {
    ...
}

And Seafowl will be able to call this function in the required way.

Compiling Golang into WASM

In August 2023 the Golang team shipped WASI/WASM support, so the following approach we used may no longer be strictly necessary these days.

When we tackled this project, Golang did not yet support WASM natively so looking elsewhere, we found Tinygo.

It turns out we can generate a WASM module ready for import into Seafowl via: tinygo build -o seafowl-udf-go.wasm -target=wasi

NOTE: it's necessary to include -target=wasi.

Recap

In this post we reviewed some of the steps taken to port the Rust-based UDF into Golang. This was my first Golang project and I appreciated the opportunity to work at the lower level, learn about byteslice manipulation, msgpack serialization, WASI and more.

If you build a cool UDF, please consider letting us know!

Image credits