Experimenting With .NET & WebAssembly - Running .NET Based Slight Application On WASM/WASI Node Pool in AKS
A year ago I wrote about running a .NET based Spin application on a WASI node pool in AKS. Since then the support for WebAssembly in AKS hasn't changed. We still have the same preview, supporting the same versions of two ContainerD shims: Spin and Slight (a.k.a. SpiderLightning). So why am I coming back to the subject? The broader context has evolved.
With WASI preview 2, the ecosystem is embracing the component model and standardized APIs. When I was experimenting with Spin, I leveraged WAGI (WebAssembly Gateway Interface) which allowed me to be ignorant about the Wasm runtime context. Now I want to change that, cross the barrier and dig into direct interoperability between .NET and Wasm.
Also, regarding the mentioned APIs, one of the emerging ones is wasi-cloud-core which aims to provide a generic way for WASI applications to interact with services. This proposal is not yet standardized but it has an experimental host implementation which happens to be Slight. By running a .NET based Slight application I want to get a taste of what that API might bring.
Last but not least, .NET 8 has brought a new way of building .NET based Wasm applications with a wasi-experimental workload. I want to build something with it and see where the .NET support for WASI is heading.
So, this "experiment" has multiple angles and brings together a bunch of different things. How am I going to start? In the usual way, by creating a project.
Creating a .NET 8 wasi-wasm Project
The wasi-experimental workload is optional, so we need to install it before we can create a project.
dotnet workload install wasi-experimental
It also doesn't bundle the WASI SDK (the WASI SDK for .NET 7 did), so we have to install it ourselves. The releases of WASI SDK available on GitHub contain binaries for different platforms. All you need to do is download the one appropriate for yours, extract it, and create the WASI_SDK_PATH environment variable pointing to the output. The version I'll be using here is 20.0.
With the prerequisites in place, we can create the project.
dotnet new wasiconsole -o Demo.Wasm.Slight
Now, if you run dotnet build and inspect the output folder, you can notice that it contains Demo.Wasm.Slight.dll, other managed DLLs, and dotnet.wasm. This is the default output, where the dotnet.wasm is responsible for loading the Mono runtime and then loading the functionality from DLLs. This is not what we want. We want a single file. To achieve that we need to modify the project file by adding the WasmSingleFileBundle property (in my opinion this should be the default).
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
...
<WasmSingleFileBundle>true</WasmSingleFileBundle>
</PropertyGroup>
</Project>
If you run dotnet build after this modification, you will find Demo.Wasm.Slight.wasm in the output. Exactly what we want.
But, before we can start implementing the actual application, we need to work on the glue between the .NET and WebAssembly so we can interact with the APIs provided by Slight from C#.
From WebAssembly IDL To C#
The imports and exports in WASI APIs are described in Wasm Interface Type (WIT) format. This format is an IDL which is a foundation behind tooling for the WebAssembly Component Model.
WIT aims at being a developer-friendly format, but writing bindings by hand is not something that developers expect. This is where wit-bindgen comes into the picture. It's a (still young) binding generator for languages that are compiled into WebAssembly. It currently supports languages like Rust, C/C++, or Java. The C# support is being actively worked on and we can expect that at some point getting C# bindings will be as easy as running a single command (or even simpler as Steve Sanderson is already experimenting with making it part of the toolchain) but for now, it's too limited and we will have to approach things differently. What we can use is the C/C++ support.
There is one more challenge on our way. The current version of wit-bindgen is meant for WASI preview 2. Meanwhile, a lot of existing WIT definitions and WASI tooling around native languages are using WASI preview 1. This is exactly the case when it comes to the Slight implementation available in the AKS preview. To handle that we need an old (and I mean old) version of wit-bindgen. I'm using version v0.2.0. Once you install it, you can generate C/C++ bindings for the desired imports and exports. Slight in version 0.1.0 provides a couple of capabilities, but I've decided to start with just one, the HTTP Server. That means we need imports for http.wit and exports for http-handler.wit.
wit-bindgen c --import ./wit/http.wit --out-dir ./native/
wit-bindgen c --export ./wit/http-handler.wit --out-dir ./native/
Once we have the C/C++ bindings, we can configure the build to include those files in the arguments passed to Clang. For this purpose, we can use the .targets file.
<Project>
<ItemGroup>
<_WasmNativeFileForLinking Include="$(MSBuildThisFileDirectory)\..\native\*.c" />
</ItemGroup>
</Project>
We also need to implement the C# part of the interop. For the main part, it's like any other interop you might have ever done. You can use generators or ask for help from a friendly AI assistant and it will get you through the code for types and functions. However, there is one specific catch. The DllImportAttribute requires passing a library name for the P/Invoke generator, but we have no library. The solution is as simple as surprising (at least to me), we can provide the name of any library that the P/Invoke generator knows about.
internal static class HttpServer
{
// Any library name that P/Invoke generator knows
private const string LIBRARY_NAME = "libSystem.Native";
[DllImport(LIBRARY_NAME)]
internal static extern unsafe void http_server_serve(WasiString address,
uint httpRouterIndex, out WasiExpected<uint> ret0);
[DllImport(LIBRARY_NAME)]
internal static extern unsafe void http_server_stop(uint httpServerIndex,
out WasiExpected<uint> ret0);
}
With all the glue in place, we can start implementating the HTTP server capability.
Implementing the Slight HTTP Server Capability
In order for a Slight application to start accepting HTTP requests, the application needs to call the http_server_serve function to which it must provide the address it wants to listen on and the index of a router that defines the supported routes. I've decided to roll out to simplest implementation I could think of to start testing things out - the HttpRouter and HttpServer classes which only allow for calling the serve function (no support for routing).
internal class HttpRouter
{
private uint _index;
public uint Index => _index;
private HttpRouter(uint index)
{
_index = index;
}
public static HttpRouter Create()
{
http_router_new(out WasiExpected<uint> expected);
if (expected.IsError)
{
throw new Exception(expected.Error?.ErrorWithDescription.ToString());
}
return new HttpRouter(expected.Result.Value);
}
}
internal static class HttpServer
{
private static uint? _index;
public static void Serve(string address)
{
if (_index.HasValue)
{
throw new Exception("The server is already running!");
}
HttpRouter router = HttpRouter.Create();
http_server_serve(
WasiString.FromString(address),
router.Index,
out WasiExpected<uint> expected
);
if (expected.IsError)
{
throw new Exception(expected.Error?.ErrorWithDescription.ToString());
}
_index = expected.Result;
}
}
This allowed me to implement a simplistic application, a one-liner.
HttpServer.Serve("0.0.0.0:80");
It worked! Every request results in 404, because there is no routing, but it worked. So, how to add support for routing?
The http_router_* functions for defining routes expect two strings - one for the route and one for the handler. This suggests that the handler should be an exported symbol that Slight will be able to call. I went through the bindings exported for http-handler.wit and I've found a function that is being exported as handle-http. That function seems to be what we are looking for. It performs transformations to/from the request and response objects and calls a http_handler_handle_http function which has only a definition. So it looks like the http_handler_handle_http implementation is a place for the application logic. To test this theory, I've started by implementing a simple route registration method.
internal class HttpRouter
{
...
private static readonly WasiString REQUEST_HANDLER = WasiString.FromString("handle-http");
...
public HttpRouter RegisterRoute(HttpMethod method, string route)
{
WasiExpected<uint> expected;
switch (method)
{
case HttpMethod.GET:
http_router_get(_index, WasiString.FromString(route), REQUEST_HANDLER,
out expected);
break;
case HttpMethod.PUT:
http_router_put(_index, WasiString.FromString(route), REQUEST_HANDLER,
out expected);
break;
case HttpMethod.POST:
http_router_post(_index, WasiString.FromString(route), REQUEST_HANDLER,
out expected);
break;
case HttpMethod.DELETE:
http_router_delete(_index, WasiString.FromString(route), REQUEST_HANDLER,
out expected);
break;
default:
throw new NotSupportedException($"Method {method} is not supported.");
}
if (expected.IsError)
{
throw new Exception(expected.Error?.ErrorWithDescription.ToString());
}
return new HttpRouter(expected.Result.Value)
}
...
}
Next I've registered some catch-all routes and implemented the HandleRequest method as the .NET handler. It would still return a 404, but it would be mine 404.
internal static class HttpServer
{
...
public static void Serve(string address)
{
...
HttpRouter router = HttpRouter.Create()
.RegisterRoute(HttpMethod.GET, "/")
.RegisterRoute(HttpMethod.GET, "/*");
http_server_serve(
WasiString.FromString(address),
router.Index,
out WasiExpected<uint> expected
);
...
}
private static unsafe void HandleRequest(ref HttpRequest request,
out WasiExpected<HttpResponse> result)
{
HttpResponse response = new HttpResponse(404);
response.SetBody($"Handler Not Found ({request.Method} {request.Uri.AbsolutePath})");
result = new WasiExpected<HttpResponse>(response);
}
...
}
It's time for some C. I went through the Mono WASI C driver and found two functions that looked the right tools for the job: lookup_dotnet_method and mono_wasm_invoke_method_ref. The implementation didn't seem overly complicated.
#include <string.h>
#include <wasm/driver.h>
#include "http-handler.h"
MonoMethod* handle_request_method;
void mono_wasm_invoke_method_ref(MonoMethod* method, MonoObject** this_arg_in,
void* params[], MonoObject** _out_exc, MonoObject** out_result);
void http_handler_handle_http(http_handler_request_t* req,
http_handler_expected_response_error_t* ret0)
{
if (!handle_request_method)
{
handle_request_method = lookup_dotnet_method(
"Demo.Wasm.Slight",
"Demo.Wasm.Slight",
"HttpServer",
"HandleRequest",
-1);
}
void* method_params[] = { req, ret0 };
MonoObject* exception;
MonoObject* result;
mono_wasm_invoke_method_ref(handle_request_method, NULL, method_params, &exception, &result);
}
But it didn't work. The thrown exception suggested that the Mono runtime wasn't loaded. I went back to studying Mono to learn how it is being loaded. What I've learned is that during compilation a _start() function is being generated. This function performs the steps necessary to load the Mono runtime and wraps the entry point to the .NET code. I could call it, but this would mean going through the Main method and retriggering HttpServer.Serve, which was doomed to fail. I needed to go a level lower. By reading the code of the _start() function I've learned that it calls the mono_wasm_load_runtime function. Maybe I could as well?
...
int mono_runtime_loaded = 0;
...
void http_handler_handle_http(http_handler_request_t* req,
http_handler_expected_response_error_t* ret0)
{
if (!mono_runtime_loaded) {
mono_wasm_load_runtime("", 0);
mono_runtime_loaded = 1;
}
...
}
Now it worked. But I wasn't out of the woods yet. What I've just learned meant that to provide dedicated handlers for routes I couldn't rely on registering dedicated methods as part of the Main method flow. I could only register the routes and the handlers needed to be discoverable later, in a new context, with static HandleRequest as the entry point. My thoughts went in the direction of a poor man's attribute-based routing, so I've started with an attribute for decorating handlers.
internal class HttpHandlerAttribute: Attribute
{
public HttpMethod Method { get; }
public string Route { get; }
public HttpHandlerAttribute(HttpMethod method, string route)
{
Method = method;
Route = route;
}
...
}
A poor man's implementation of an attribute-based routing must have an ugly part and it is reflection. To register the routes (and later match them with handlers), the types must be scanned for methods with the attributes. In a production solution, it would be necessary to narrow the scan scope but as this is just a small demo I've decided to keep it simple and scan the whole assembly for static, public and non-public, methods decorated with the attribute. The code supports adding multiple attributes to a single method just because it's simpler than putting proper protections in place. As you can probably guess, I did override the Equals and GetHashCode implementations in the attribute to ensure it behaves nicely as a dictionary key.
internal class HttpRouter
{
...
private static readonly Type HTTP_HANDLER_ATTRIBUTE_TYPE = typeof(HttpHandlerAttribute);
private static Dictionary<HttpHandlerAttribute, MethodInfo>? _routes;
...
private static void DiscoverRoutes()
{
if (_routes is null)
{
_routes = new Dictionary<HttpHandlerAttribute, MethodInfo>();
foreach (Type type in Assembly.GetExecutingAssembly().GetTypes())
{
foreach(MethodInfo method in type.GetMethods(BindingFlags.Static |
BindingFlags.Public |
BindingFlags.NonPublic))
{
foreach (object attribute in method.GetCustomAttributes(
HTTP_HANDLER_ATTRIBUTE_TYPE, false))
{
_routes.Add((HttpHandlerAttribute)attribute, method);
}
}
}
}
}
...
}
With the reflection stuff (mostly) out of the way, I could implement a method that can be called to register all discovered routes and a method to invoke a handler for a route. This implementation is not "safe". I don't do any checks on the reflected MethodInfo to ensure that the method has a proper signature. After all, I can only hurt myself here.
internal class HttpRouter
{
...
internal HttpRouter RegisterRoutes()
{
DiscoverRoutes();
HttpRouter router = this;
foreach (KeyValuePair<HttpHandlerAttribute, MethodInfo> route in _routes)
{
router = router.RegisterRoute(route.Key.Method, route.Key.Route);
}
return router;
}
internal static HttpResponse? InvokeRouteHandler(HttpRequest request)
{
DiscoverRoutes();
HttpHandlerAttribute attribute = new HttpHandlerAttribute(request.Method,
request.Uri.AbsolutePath);
MethodInfo handler = _routes.GetValueOrDefault(attribute);
return (handler is null) ? null : (HttpResponse)handler.Invoke(null,
new object[] { request });
}
...
}
What remained was small modifications to the HttpServer to use the new methods.
internal static class HttpServer
{
...
public static void Serve(string address)
{
...
HttpRouter router = HttpRouter.Create()
.RegisterRoutes();
http_server_serve(
WasiString.FromString(address),
router.Index,
out WasiExpected<uint> expected
);
...
}
private static unsafe void HandleRequest(ref HttpRequest request,
out WasiExpected<HttpResponse> result)
{
HttpResponse? response = HttpRouter.InvokeRouteHandler(request);
if (!response.HasValue)
{
response = new HttpResponse(404);
response.Value.SetBody(
$"Handler Not Found ({request.Method} {request.Uri.AbsolutePath})"
);
}
result = new WasiExpected<HttpResponse>(response.Value);
}
...
}
To test this out, I've created two simple handlers.
[HttpHandler(HttpMethod.GET, "/hello")]
internal static HttpResponse HandleHello(HttpRequest request)
{
HttpResponse response = new HttpResponse(200);
response.SetHeaders(new[] { KeyValuePair.Create("Content-Type", "text/plain") });
response.SetBody($"Hello from Demo.Wasm.Slight!");
return response;
}
[HttpHandler(HttpMethod.GET, "/goodbye")]
internal static HttpResponse HandleGoodbye(HttpRequest request)
{
HttpResponse response = new HttpResponse(200);
response.SetHeaders(new[] { KeyValuePair.Create("Content-Type", "text/plain") });
response.SetBody($"Goodbye from Demo.Wasm.Slight!");
return response;
}
That's it. Certainly not complete, certainly not optimal, most likely buggy, and potentially leaking memory. But it works and can be deployed to the cloud.
Running a Slight Application in WASM/WASI Node Pool
To deploy our Slight application to the cloud, we need an AKS Cluster with a WASM/WASI node pool. The process of setting it up hasn't changed since my previous post and you can find all the necessary steps there. Here we can start with dockerizing our application.
As we are dockerizing a Wasm application, the final image should be from scratch. In the case of a Slight application, it should contain two elements: app.wasm and slightfile.toml. The slightfile.toml is a configuration file and its main purpose is to define and provide options for the capabilities needed by the application. In our case, that's just the HTTP Server capability.
specversion = "0.1"
[[capability]]
name = "http"
The app.wasm file is our application. It should have this exact name and be placed at the root of the image. To be able to publish the application, the build stage in our Dockerfile must install the same prerequisites as we did for local development (WASI SDK and wasi-experimental workload).
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
RUN curl https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-20/wasi-sdk-20.0-linux.tar.gz -L --output wasi-sdk-20.0-linux.tar.gz
RUN tar -C /usr/local/lib -xvf wasi-sdk-20.0-linux.tar.gz
ENV WASI_SDK_PATH=/usr/local/lib/wasi-sdk-20.0
RUN dotnet workload install wasi-experimental
WORKDIR /src
COPY . .
RUN dotnet publish --configuration Release
FROM scratch
COPY --from=build /src/bin/Release/net8.0/wasi-wasm/AppBundle/Demo.Wasm.Slight.wasm ./app.wasm
COPY --from=build /src/slightfile.toml .
With the infrastructure in place and the image pushed to the container registry, all that is needed is a deployment manifest for Kubernetes resources. It is the same as it was for a Spin application, the only difference is the kubernetes.azure.com/wasmtime-slight-v1 node selector.
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: "wasmtime-slight-v1"
handler: "slight"
scheduling:
nodeSelector:
"kubernetes.azure.com/wasmtime-slight-v1": "true"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: slight-with-dotnet-8
spec:
replicas: 1
selector:
matchLabels:
app: slight-with-dotnet-8
template:
metadata:
labels:
app: slight-with-dotnet-8
spec:
runtimeClassName: wasmtime-slight-v1
containers:
- name: slight-with-dotnet-8
image: crdotnetwasi.azurecr.io/slight-with-dotnet-8:latest
command: ["/"]
---
apiVersion: v1
kind: Service
metadata:
name: slight-with-dotnet-8
spec:
ports:
- protocol: TCP
port: 80
targetPort: 80
selector:
app: slight-with-dotnet-8
type: LoadBalancer
After applying the above manifest, you can get the service IP with kubectl get svc or from the Azure Portal and execute some requests.
WebAssembly, Cloud, .NET, and Future?
Everything I toyed with here is either a preview or experimental, but it's a glimpse into where WebAssembly in the Cloud is heading.
If I dare to make a prediction, I think that when the wasi-cloud-core proposal reaches a mature enough phase, we will see it supported on AKS (probably it will replace the Spin and Slight available in the current preview). The support for WASI in .NET will also continue to evolve and we will see a non-experimental SDK once the specification gets stable.
For now, we can keep experimenting and exploring. If you want to kick-start your own exploration with what I've described in this post, the source code is here.
Postscript
Yes, I know that this post is already a little bit long, but I wanted to mention one more thing.
When I was wrapping this post, I read the "Extending WebAssembly to the Cloud with .NET" and learned that Steve Sanderson has also built some Slight samples. Despite that, I've decided to publish this post. I had two main reasons for that. First,I dare to think that there is valuable knowledge in this dump of thoughts of mine. Second, those samples take a slightly different direction than mine, and I believe that the fact that you can arrive at different destinations from the same origin is one of the beautiful aspects of software development - something worth sharing.