feat: Readiness endpoint

[sergerad]

Relates to #2176.

WIP.

Creates a new endpoint for loadbalancers to get gRPC 0 or 14 status codes from.

[Mirko-von-Leipzig]

Something that bothers me here though, is that the load-balancer must also serve this endpoint 🤔

But I suppose that they would just forward it, and if anything is available it returns Ok, and if nothing is available they wouldn't have a service. So maybe it just works :)

[Mirko-von-Leipzig]

After some digging, it looks like gRPC has its own health check service defined. We should probably consider that instead since it probably is supported by load-balancers out the box (..maybe).

https://github.com/grpc/grpc/blob/master/doc/health-checking.md

[sergerad]

After some digging, it looks like gRPC has its own health check service defined. We should probably consider that instead since it probably is supported by load-balancers out the box (..maybe).

https://github.com/grpc/grpc/blob/master/doc/health-checking.md

I think this approach adds some complexity that we would never leverage (service name, additional status codes). Happy to do it if we think we will use it. But if not, prefer to keep simple with just 0 and 14 codes and no service names.

[Mirko-von-Leipzig]

I think this approach adds some complexity that we would never leverage (service name, additional status codes). Happy to do it if we think we will use it. But if not, prefer to keep simple with just 0 and 14 codes and no service names.

The way I understand it, its still just the same except we return an enum instead of the status code.

And since each server effectively only has a single gRPC service (+ health service I guess), we only need to support the empty, health (always returns healthy), and the specific gRPC service the server implements? If we do this only for nodes then its always:

"" - overall server health
"rpc.Api"
"Health"