WebAssembly meets Kubernetes with Krustlet

Linux container technology has changed the face of computing, but especially distributed computing in publicly rentable servers commonly said to be “the public cloud” (like Microsoft Azure). With containers came tooling – like Docker – and systems that orchestrate potentially millions of them – with Kubernetes becoming the most widely used. Together, they have enabled millions of developers and organizations to deliver software more rapidly, widely, and productively than ever before.  

Even so, there are limitations to containers and Kubernetes, so tightly bound as they are in their booming ecosystem. A couple areas stand out: the work required to lock container runtimes down and specialized environments either requiring more speed or having sharply constrained resources.  

Yesterday, we released v0.1.0 of Krustlet, a project which explores using WebAssembly modules in Kubernetes to address some of these scenarios. With Krustlet you can test-drive WebAssemblies (also called WASM) in Kubernetes alongside your containers, offering the possibility of new security and runtime capabilities. If you’re already hooked and want to experiment with Kubernetes and WASM, you can dive right in with the project, or read the introduction to Krustlet by Matt Fisher. 

Secure runtimes and constrained environments are hard

The container community, including Microsoft, has worked extremely hard for years on security issues and has made amazing progress. Container runtimes have restricted the ability of containers to use sensitive kernel calls and the Kubernetes ecosystem has enough new tools – meshes, access policy engines, thin hypervisors – that with work, a substantial degree of confidence in security–sensitive environments can be achieved. If you’re good.  

Despite this tremendous work, there remain environments for which Kubernetes and containers are not yet well suited. For example, containers should be small, but often aren’t. Environments with constrained resources – memory and disk space, primarily – cannot use large containers and the runtimes they need. Even if they could, some organizations remain unready to deploy a runtime when the default configuration must be dramatically tightened down to provide confidence.  

What if we had another sandboxed process type, one that was by default completely sandboxed and for which the runtimes could be both extremely fast – faster than launching containers – and also much smaller? Could it be possible for Kubernetes to run those?  

The answer is yes, it is possible.  

One way to do the above is to use a very small hypervisor; in fact, there are several that have been built, among them gVisor. Start-up time is reduced, and security aspects are addressed as well, so this approach is a promising area for the current workloads in Kubernetes. Even so, there is another sandboxed process type that has a vibrant ecosystem, a secure-by-default runtime model, and is very fast and very small. WebAssembly. 

WASM and WASI enter the picture

Get ready for a lot of acronyms. WebAssembly – commonly called WASM –  is a compact bytecode format optimized for fast download and maximum execution speed and has been an open web standard for a few years. Up until now, it has been used mainly in browsers for fully sandboxed, but highly performant, computing. For one recent example, Microsoft’s Blazor Framework can use WASM to enable very performant and complex visualizations, among other things.

One of the hindrances of using WebAssembly outside browser environments has been the lack of a single compile target. But last year Mozilla announced the WebAssembly System Interface effort, or WASI, to provide a system interface so that compilers can target the interface and not the underlying operating system – and that could make WebAssembly a developer target at scale. If you want to know a lot more about WASI, Lin Clark’s original announcement is a great explainer.

If you’re unclear about the potential of WASI to bring WASM to the masses, perhaps Docker founder Solomon Hykes can convince you: 

Importantly, at a very high level WASM has two main features that the Kubernetes ecosystem might be able to use to its advantage:

• WebAssemblies and their runtimes can execute fast and be very small compared to containers
• WebAssemblies are by default unable to do anything; only with explicit permissions can they execute at all

These two features hit our sweet spot, which suggested to us that we might profitably use WASM with Kubernetes to work in constrained and security-conscious environments – places where containers have a harder time.

The Krustlet experiment

For several months our team has been experimenting with a Kubelet implementation written in Rust that runs WebAssemblies to understand how to run WASM in Kubernetes. Today we are announcing that this project, Krustlet, has reached a 0.1.0 release. Krustlet is early work, of course. Many of the things you would expect to be in a conformant Kubernetes system are not running yet, such as init “containers” and volumes; those will come soon.

What you can do with Krustlet v0.1.0 and the wasm-to-oci tool:

• Push and pull WASMs to Azure Container Registry or any OCI distribution greater than v2.7.0
• Use the YAML manifests you know and LOVE to schedule a WebAssembly module to a node running Krustlet – because it’s just Kubernetes.
• Use Minikube, KinD, and Azure Kubernetes Service (with instructions for more distributions coming)
• Use either wasmtime or waSCC, with other runtime support coming

Is that a lot? Well – not really. It’s the very beginning. But it means quite a bit about the future.

WASI is the way forward

Krustlet can run multiple runtimes because there are so many and not all of them support WASI, as it’s still a work in progress. As a result, in this initial release, however, two providers are supported:

• Wasmtime
• WaSCC

Both are great ways to get working with WASM in Kubernetes (and if you want to create a Krustlet provider for your favorite runtime, jump in with an issue). WaSCC is an interesting platform to try out WASMs in Kubernetes because it has a robust capabilities-based security model. You can build capability providers either as native binaries or as WASI modules, and support for these WASI capability providers will mature as the WASI specification and community progresses.

While Microsoft has been involved with WebAssembly (the Edge browser supports it), many organizations have realized that WASM with WASI could be the process model that helps get hard jobs done. In particular, the work of the Mozilla Foundation to launch the WASI specification process was an inspiration and, we think, will pay dividends sooner rather than later, though much hard work remains to be done. Much of that work continues under the Bytecode Alliance, formed among other things to help shepherd WASI.

We’ve also been collaborating with CapitalOne – the creators of waSCC – as well as taking notice of all the work in the WASM ecosystem, among them:

• Solo.io‘ s creation of the WebAssembly Hub for WASM-based Envoy proxy filters
• CloudFlare, who has introduced WASM-based Workers
• Fastly, who created Lucet, a runtime optimized for speed and who is working hard on WASI as well

Why Krustlet is a Kubelet

We didn’t set out to build a Kubelet. But trying to create a Container Runtime Interface implementation taught us that CRI v1 is really meant for container runtimes. Really meant for container runtimes. Though this work may pave the way for an elaboration of CRI v2, we learned that a Kubelet was really the best way to extend Kubernetes, much like the Virtual Kubelet project, which has many provider implementations.

We also learned that using Rust to build Kubernetes components was a wonderful experience compared to Go, and it should be much more widely adopted for Kubernetes work. We’ll elaborate on this point in the near future, but the use of Rust is the reason we did not rely on Virtual Kubelet for the project.

Both WebAssemblies and containers are needed

Despite the excitement about Wasm and WASI, it should be clear that containers are the major workload in K8s, and for good reason. No one will be replacing the vast majority of their workloads with WebAssembly. Our usage so far makes this clear. Do not be confused by having more tools at your disposal.

For example, WebAssembly modules are binaries and not OS environments, so you can’t simply bring your app code and compile it into a WASM like you can a container. Instead, you’re going to build one binary, which in good cloud-native style should do one thing, and well. This means, however, that WASM “pods” in Kubernetes are going to be brand new work; they likely didn’t exist before. Containers clearly remain the vast bulk of Kubernetes work.

WASMs can be packed very, very densely, however, so using WebAssembly might maximize the processing throughput for large public cloud servers as well as more constrained environments. They’re unable to perform any work unless granted the permissions to do so, which means organizations that do not yet have confidence in container runtimes will have a great possibility to explore. And memory or otherwise constrained environments such as ARM32 or other system-on-a-chips (SOCs) and microcontroller units (MCUs) may be now attachable to and schedulable from larger clusters and managed using the same or similar tooling to that which Kubernetes uses right now.

Get going with Krustlet

Get up and running with Krustlet by jumping straight into the quickstart. If you’re the sort who wants to understand WASM and WASI, start with Lin Clark’s WASI announcement, but for a larger tour, it’s really worth watching Fastly’s CTO Tyler McMullen’s presentation at CodeMesh LDN this past year.

Give it a run, and let us know what you think.

Ralph Squillace

Principal Product Manager

Principal Product Manager, Microsoft Azure Core Upstream. Currently, he is Microsoft's representative on the board of the Bytecode Alliance Foundation and is responsible for WebAssembly outside of the browser at the company. Ralph has worked with Linux for several decades and had worked on many Azure services through the years. His team is also responsible for the ContainerD project Runwasi, part of the SpinKube project.

See more articles from this author