Skip to content

fix: Ensure the LambdaRuntimeAPI server is up before launching INIT phase #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gregfurman merged 2 commits into from
Oct 22, 2025
Merged

fix: Ensure the LambdaRuntimeAPI server is up before launching INIT phase #42

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
66e356c
fix: Ensure the LambdaRuntimeAPI server is up before launching INIT p…
gregfurman Oct 20, 2025
da02e82
Increase frequency of polling
gregfurman Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions cmd/localstack/custom_interop.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,18 @@ import (
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strconv"
"strings"
"time"

"github.com/go-chi/chi"
log "github.com/sirupsen/logrus"
"go.amzn.com/lambda/core/statejson"
"go.amzn.com/lambda/interop"
"go.amzn.com/lambda/rapidcore"
"go.amzn.com/lambda/rapidcore/standalone"
"io"
"net/http"
"strconv"
"strings"
"time"
)

type CustomInteropServer struct {
Expand Down
22 changes: 19 additions & 3 deletions cmd/localstack/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,15 @@ package main

import (
"context"
log "github.com/sirupsen/logrus"
"go.amzn.com/lambda/interop"
"go.amzn.com/lambda/rapidcore"
"os"
"runtime/debug"
"strconv"
"strings"
"time"

log "github.com/sirupsen/logrus"
"go.amzn.com/lambda/interop"
"go.amzn.com/lambda/rapidcore"
)

type LsOpts struct {
Expand Down Expand Up @@ -188,6 +190,12 @@ func main() {
SetLogsEgressAPI(localStackLogsEgressApi).
SetTracer(tracer)

// Corresponds to the 'AWS_LAMBDA_RUNTIME_API' environment variable.
// We need to ensure the runtime server is up before the INIT phase,
// but this envar is only set after the InitHandler is called.
runtimeAPIAddress := "127.0.0.1:9001"
sandbox.SetRuntimeAPIAddress(runtimeAPIAddress)
Comment thread
gregfurman marked this conversation as resolved.

// xray daemon
endpoint := "http://" + lsOpts.LocalstackIP + ":" + lsOpts.EdgePort
xrayConfig := initConfig(endpoint, xRayLogLevel)
Expand Down Expand Up @@ -225,6 +233,14 @@ func main() {
}
go RunHotReloadingListener(interopServer, lsOpts.HotReloadingPaths, fileWatcherContext, lsOpts.FileWatcherStrategy)

log.Debugf("Awaiting initialization of runtime api at %s.", runtimeAPIAddress)
// Fixes https://github.com/localstack/localstack/issues/12680
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
if err := waitForRuntimeAPI(ctx, runtimeAPIAddress); err != nil {
log.Fatalf("Lambda Runtime API server at %s did not come up in 30s, with error %s", runtimeAPIAddress, err.Error())
}
cancel()

// start runtime init. It is important to start `InitHandler` synchronously because we need to ensure the
// notification channels and status fields are properly initialized before `AwaitInitialized`
log.Debugln("Starting runtime init.")
Expand Down
48 changes: 48 additions & 0 deletions cmd/localstack/runtime.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
package main

import (
"context"
"fmt"
"net/http"
"net/url"
"strings"
"time"
)

func waitForRuntimeAPI(ctx context.Context, targetAddress string) error {
if !strings.HasPrefix(targetAddress, "http://") {
targetAddress = fmt.Sprintf("http://%s", targetAddress)
}

healthEndpoint, err := url.JoinPath(targetAddress, "2018-06-01", "ping")
if err != nil {
return err
}

req, err := http.NewRequestWithContext(ctx, http.MethodGet, healthEndpoint, nil)
if err != nil {
return err
}
client := &http.Client{
Timeout: 5 * time.Second,
}

ticker := time.NewTicker(500 * time.Millisecond)
Copy link
Copy Markdown
Member

@dfangl dfangl Oct 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you per chance test how many - on average - retries we get here, and how this affects the average startup times? Does it make sense to reduce this to 10 - 50ms, for example? More pings, but less delay? The startup time should be significantly less than 10ms ideally anyway, right?

Copy link
Copy Markdown
Author

@gregfurman gregfurman Oct 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dfangl Fair points. I saw these in the order of milliseconds based on my local runs (keeping in mind that I'm on ARM64 and don't have issues with this).

{"file":"lambda-runtime-init/lambda/rapi/server.go:108","func":"go.amzn.com/lambda/rapi.(*Server).Listen","level":"debug","msg":"Runtime API Server listening on 127.0.0.1:9001","time":"2025-10-17T19:35:58+02:00"}
{"file":"lambda-runtime-init/lambda/rapi/middleware/middleware.go:76","func":"go.amzn.com/lambda/rapi/middleware.AccessLogMiddleware.func1.1","level":"debug","msg":"API request - GET /2018-06-01/ping, Headers:map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]","time":"2025-10-17T19:35:59+02:00"}
{"file":"lambda-runtime-init/lambda/rapi/middleware/middleware.go:76","func":"go.amzn.com/lambda/rapi/middleware.AccessLogMiddleware.func1.1","level":"debug","msg":"API request - GET /2018-06-01/ping, Headers:map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]","time":"2025-10-17T19:35:59+02:00"}
{"file":"ambda-runtime-init/cmd/localstack/main.go:245","func":"main.main","level":"debug","msg":"Starting runtime init.","time":"2025-10-17T19:36:04+02:00"}

In anycase, these checks are probably more conservative than what is necessary so increasing frequency and decreasing timeout duration seems logical.

Otherwise, if we're trying to make this as fast as possible, so as to not delay startup times, we can also do a single check to see if the port is open (similar to LocalStack's is_port_open()) with some timeout of 5 seconds (or something equivalently short).

Copy link
Copy Markdown
Member

@dfangl dfangl Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine either way - I just think 500ms for delay is too much, especially since the first one will for quite some systems fail.

Copy link
Copy Markdown
Author

@gregfurman gregfurman Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK changed to 50ms! Happy to 🚢 ?

defer ticker.Stop()

for {
resp, err := client.Do(req)
if err == nil {
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK {
return nil
}
}

select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
}
}
}