Skip to content

Use log.Printf for AWS poll errors to avoid mangled log lines - #27

Draft
saltzm wants to merge 1 commit into
mainfrom
devin/1784840260-aws-poll-log-fix
Draft

Use log.Printf for AWS poll errors to avoid mangled log lines#27
saltzm wants to merge 1 commit into
mainfrom
devin/1784840260-aws-poll-log-fix

Conversation

@saltzm

@saltzm saltzm commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

The AWS IMDS poll loop logged failures with fmt.Printf("error during aws poll: %v", err)no trailing newline and no timestamp, unlike every other log call in the codebase (which use log.Printf). The missing \n causes the error to run into the next log line, e.g.:

...connect: connection refused2026/07/18 13:12:51 http: TLS handshake error from ...

This is what made ~15 of these errors get mis-parsed in Datadog. Fix is a one-liner:

-  fmt.Printf("error during aws poll: %v", err)
+  log.Printf("error during aws poll: %v", err)

Context — this error is benign transient noise

Datadog (source:vprox, 7d): 228 hits across 148 distinct hosts, ≤6/host (~0.005% of polls); 170× IMDS 503 (throttling), 43× connection refused, clustered on fleet-wide blips. A failed poll originates in awsClient.GetAddresses(), which returns before any sm.Start/sm.Stop, so pollAws returns the previous IP set unchanged and just retries on the next 5s tick — it cannot drop a live tunnel (error starting new ip = 0 occurrences in 30d).

Unaddressed follow-ups (intentionally left out — judgment calls)

  • Retry/backoff: IMDS client uses SDK default (3 attempts); could raise MaxAttempts / add a backoff retryer to trim 503 noise.
  • Initial-poll tolerance: the startup poll aborts server launch on a transient IMDS blip (fail-fast); could retry instead.
  • Metric: an aws_poll error counter would allow alerting on rates rather than eyeballing logs.

Link to Devin session: https://modal.devinenterprise.com/sessions/456280ed46e24e64a4e317cdb1c51dc9
Requested by: @saltzm

Co-Authored-By: matthew@modal.com <saltzm@gmail.com>
@saltzm saltzm self-assigned this Jul 23, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant