Interaction-based idle culling for VS Code (code-server)

[tylerpotts]

Closes #208

Problem

An open VS Code tab holds a websocket whose keepalives flow through jupyter-server-proxy and count as Jupyter activity, so pods with an idle VS Code tab are never culled. The heartbeat itself is a local file touch; the proxied connection traffic is what defeats culling. The env var proposed in the issue (CODE_SERVER_IDLE_TIMEOUT_SECONDS) only fires after all browser connections close (code-server's idle timer gates on getConnections() > 0), so on its own it does not cover the open-tab case — and it doesn't exist in the previously pinned code-server 4.104.3 (added in 4.106.0 via coder/code-server#7539).

What this PR does

Two composing mechanisms:

1. Interaction-based activity (open-but-idle tabs). The vscode jupyter-server-proxy entry is now registered by this repo (the jupyter-vscode-proxy package is removed) with update_last_activity: False, so VS Code keepalives stop refreshing the in-pod api_last_activity. A bundled server-side VS Code extension (nebari-activity-reporter, plain JS, packaged to a vsix by a stdlib script at image build, installed per-user by a postStart hook) reports real interaction — typing, scrolling, terminal use, focus — and treats a running terminal command as busy (mirrors cullBusy: false). Idle-tab pods are then culled by the in-pod singleuserCuller.server.shutdownNoActivityTimeout (900s default). Note: the hub-level jupyterhub.cull culler still sees CHP-level websocket activity while a tab is connected, so the in-pod culler is the load-bearing mechanism — documented in docs/src/content/docs/configuration.md.
2. The issue's mechanism (disconnected sessions). code-server is bumped 4.104.3 → 4.133.0 (install.sh is byte-identical between tags; hash pin unchanged) and CODE_SERVER_IDLE_TIMEOUT_SECONDS is derived from jupyterhub.cull.timeout (skipped when culling is disabled or timeout ≤ 60, which code-server rejects), so lingering code-server processes exit on the culler's schedule after the last connection drops.

Escape hatch: vscodeActivity.enabled: false reverts to counting raw proxied traffic. The polarity is fail-safe: an image running without the new chart plumbing defaults to the old behavior (over-spending) rather than culling active users who have no reporter installed.

Testing

• 147 unit tests pass (19 new: spawner env wiring incl. boundary/non-numeric cases, chart-derived plumbing, vscode proxy registration, vsix packager incl. XML escaping, postStart hook shell semantics).
• 4 new e2e tests (tests/e2e/test_vscode_idle_culling.py): proxied /vscode/ traffic does not advance last_activity; a contents-API ping does; the extension is installed; the idle-timeout env matches cull.timeout.
• Follow-up commit incoming on this PR: the e2e job pulls the singleuser image tag pinned in values.yaml, so once CI publishes this branch's image I'll run scripts/bump_image_tags.py so the two image-dependent e2e tests exercise the new image.
• Outstanding manual check before merge: browser soak of extension activation (idle tab culls at shutdownNoActivityTimeout; active typing does not).

[github-actions]

Docs preview for fix/vscode-idle-culling:
https://fix-vscode-idle-culling.data-science-pack.pages.dev

[tylerpotts]

CI status note: all checks are green except integration / rbac-bootstrap, which fails identically on latest main (https://github.com/nebari-dev/data-science-pack/actions/runs/32128283826, base commit a1e49ab) with configuration validation failed: repository field is required — pre-existing breakage unrelated to this PR.

The four new e2e tests pass against this branch's built image (sha-7a07d8e): the activity-reporter extension installs via postStart, CODE_SERVER_IDLE_TIMEOUT_SECONDS reaches the pod, proxied /vscode/ traffic no longer advances last_activity, and a contents-API ping does.

[tylerpotts]

Culling behavior reference: what keeps a VS Code pod alive, and where it can cull against user intent

For reviewers (and future docs): the exhaustive list, derived from the implementation in this PR (extension.js + images/nebi/jupyter_server_config.py + singleuserCuller).

What keeps the pod alive (resets the in-pod idle clock)

The extension reports activity (throttled to one ping / 60s) on:

1. Typing / document edits (onDidChangeTextDocument)
2. Cursor or selection movement (onDidChangeTextEditorSelection)
3. Scrolling (onDidChangeTextEditorVisibleRanges)
4. Window focus changes — switching to or away from the tab (onDidChangeWindowState)
5. Opening or closing a terminal
6. Starting or finishing a terminal command (shell-integration execution events)
7. A foreground terminal command still running — while any shell execution is in flight, the extension pings every 60s ("busy = active", mirroring cullBusy: false for kernels). Terminal output is irrelevant either way: a silent 3-hour training run in a bash/zsh terminal keeps the pod alive; detection is via shell-integration execution events, not output (VS Code's output-data event is proposed-API and unavailable to stable extensions).
8. Opening/reconnecting VS Code (activation ping)

Independent of the extension, the pod also stays alive through the pre-existing Jupyter-side signals: a busy notebook kernel (never culled, cullBusy: false), any authenticated Jupyter API traffic (JupyterLab UI use, JupyterLab terminals), and traffic through the default /proxy/<port> routes (those still count as activity — only the /vscode/ route is opted out).

Where it can cull against user intent (false-idle cases)

1. Long-running command in a shell without VS Code shell integration — fish/xonsh/custom shells, or bash/zsh where injection failed or was disabled (terminal.integrated.shellIntegration.enabled: false). Execution events never fire, so the busy-hold never engages. This is the closest real case to the "silent long-running process" example — the trigger is missing shell integration, not missing output.
2. Backgrounded or detached work — ./job &, nohup/disown, or jobs inside tmux/screen after detaching. The foreground execution ends (or the tmux client exits), and the background process is invisible to the extension.
3. Long debug sessions — the extension does not subscribe to debug-session events. A program running under the debugger via the default internalConsole with no user interaction for 15 min gets culled. (A debuggee launched in the integrated terminal with shell integration IS covered via the busy-hold.)
4. Passive reading — staring at code for >15 min without a single scroll, cursor move, or focus change. Any input event resets the clock, so this requires literally zero interaction.
5. Interacting only with a port forwarded through VS Code's Ports tab — that traffic rides the /vscode/ proxy route, which no longer counts as activity. (Using the JupyterLab-side /proxy/<port> route instead does count. And if the server behind the port was started as a foreground terminal command, the busy-hold covers it anyway.)
6. Extension delivery failure — if the vsix install or activation silently breaks, the whole keep-alive channel disappears and active users cull at shutdownNoActivityTimeout (900s). Guards: the e2e install tripwire in this PR, activation logging, and the vscodeActivity.enabled: false escape hatch.

Knobs / workarounds

• Run long jobs in a foreground bash/zsh terminal (busy-hold) or a notebook kernel (cullBusy: false) rather than backgrounded.
• Deployers can raise singleuserCuller.server.shutdownNoActivityTimeout, or set vscodeActivity.enabled: false to restore the old any-open-tab-keeps-alive behavior.
• jupyterhub.cull.maxAge remains available as a hard cap in the other direction.

[viniciusdc]

Late to this and most of what I had is already in your reference post and the description, so just the bits I don't think are covered.

Your item 6, extension-delivery failure. The guards listed are the e2e install tripwire and activation logging, and I think both sit at the wrong layer for the failure that'll actually happen. The chart sets VSCODE_PROXY_UPDATE_LAST_ACTIVITY=false from a chart value, reliably, while the reporter goes in via timeout 60 code-server --install-extension ... || true. Separate failure domains, so a single user can land with proxy activity disabled and no reporter, and cull at 900s while actively working. Wedged extensions dir, stale vsix on the home PVC, the timeout tripping on a slow mount. The e2e test catches build regressions, the logging is after the fact, and vscodeActivity.enabled: false is a deploy-time switch for a per-pod runtime failure.

Could the two share a fate? postStart writing a marker on success and jupyter_server_config.py requiring both the env var and the marker would make a failed install degrade to over-spending, which is the polarity you argue for everywhere else.

CODE_SERVER_IDLE_TIMEOUT_SECONDS derives from cull.timeout and is gated on cull.enabled, but you also establish that cull is the culler CHP defeats. Should it key off singleuserCuller.server.shutdownNoActivityTimeout instead? As written, a deployer disabling the hub culler silently turns the code-server timer off too.

For the soak: Heart.isActive is getConnections() > 0 behind a 5s timeout that resolves false, with a comment above it saying "getConnections appears to not call the callback when there are no more connections. Feels like it must be a bug?"

https://redirect.github.com/coder/code-server/blob/v4.133.0/src/node/routes/index.ts#L35-L53

If that trips with a browser attached the state goes expired and code-server exits 1800s later under someone. heart.beat() on any HTTP request clears it so it needs a quiet-HTTP window, but worth watching rather than assuming.

Last one, minor: /api/contents/ works because _track_activity isn't set on that handler upstream, and that flag has trended toward covering more read-only endpoints. If contents ever joins them the ping stops silently. last_activity() folds in any settings key ending _last_activity and calls it a hook for extensions, so a small server extension setting vscode_last_activity would sit on the documented seam instead.

Hey @krassowski , pinging you here because you might have some idea around idle culling or code-server on the Jupyter side. Short version is that an open but idle VS Code tab keeps the pod alive, since jupyter-server-proxy counts the websocket keepalives as API activity, and this PR opts the /vscode/ route out of activity reporting and feeds real interaction back in from a small VS Code extension. Mostly curious in your perspectives around this... or similar type of problems (in case code-server was a problem in the past or not)

[tylerpotts]

@viniciusdc all four points were on target. What happened with each:

Extension-delivery shared fate: implemented (f656491). Went with a variant of your marker idea: instead of a postStart-written marker, jupyter_server_config.py gates the opt-out on the installed artifact itself, globbing the extensions dir for nebari.nebari-activity-reporter-* before setting update_last_activity: False. A failed install now degrades to over-spending, matching the polarity everywhere else. One caveat baked into the code comment: postStart runs concurrently with the entrypoint, so on a user's first-ever spawn the install can lose the race against config evaluation and that one session over-spends; the PVC-backed extensions dir makes every later spawn see the artifact. (CI's e2e proxy-traffic test implicitly exercises the gate and passed, meaning the install won the race there; if that test ever flakes red, this race is why, and the failure direction is the safe one.)

CODE_SERVER_IDLE_TIMEOUT_SECONDS keying: implemented (f656491). Now derives from singleuserCuller.server.shutdownNoActivityTimeout via the chart-derived plumbing, and the cull.enabled gate is gone, so disabling the hub culler no longer silently kills the code-server timer. 0 and anything <= 60 (code-server rejects those) skip the env var. The two schedules also no longer disagree by default (both 900 now, vs 1800/900 before).

Heart.isActive / getConnections: watched on a live cluster. Soak config drives the timer down to 120s, so one 7-minute session of websocket-only editor use covered multiple potential expiry cycles with a connected client. code-server logged Idle timeout set to 120 seconds at startup and showed zero exit/restart markers; last_activity advanced every 60-62s from the reporter the whole time. Not proof the callback bug can never trip, but it survived the tightest window we can give it. If it ever does trip in the field, the fix is dropping the env var (one line in the spawner); the in-pod culler is unaffected.

/api/contents/ vs the *_last_activity settings seam: agreed, deferred. Moving to a vscode_last_activity settings key means shipping a jupyter server extension plus retargeting the reporter's ping, which is real scope for a hardening concern. The e2e test pins the current contract (contents ping advances last_activity) and fails loudly on the image bump that breaks it. Will open a follow-up issue referencing your comment so it doesn't get lost.

Full soak on the review-fix image (sha-ce941be) also re-verified the headline behaviors: busy-hold kept a hands-off sleep 300 pod alive with pings every 60s, and the pod culled ~2 minutes after the sleep ended with the tab still open.

[viniciusdc]

Nice work — dropping the betatim/vscode-binder git dependency is the most valuable change in here.

Needs fixing before merge: the PR description describes a different design than the diff. It says CODE_SERVER_IDLE_TIMEOUT_SECONDS "is derived from jupyterhub.cull.timeout", but the code derives it from singleuserCuller.server.shutdownNoActivityTimeout and is emphatic about it — hub-config.yaml:63 never reads cull.timeout, 01-spawner.py:322 literally says "NOT the hub-level cull.timeout", there's a test called test_idle_timeout_independent_of_hub_culler, and the e2e asserts "900" while cull.timeout is 1800. The code is right and the choice is right. It's the description that's wrong, and anyone approving from it approves a different change.

That same sentence appears as a code contract in three places the diff itself falsifies — jupyter_server_config.py:87, extension.js:4, and test_vscode_server_registration.py's docstring, whose own assertions at :58/:82/:107 all say is True. update_last_activity is computed and defaults to True. Test counts are off too: 132 on main -> 155 on head, so +23 new, not 19.

Same values.yaml staleness behind the first comment also drops KeyCloakOAuthenticator.manage_groups: true (d10da09) and moves the nine image refs to sha-ce941be, which git cat-file doesn't recognise — it's a synthetic PR-merge SHA from 43cae85, so the quay tag exists but maps to no commit. The git checkout fixes all of it.

Nothing lints, type-checks or executes extension.js. Ruff is scoped to config/, docs.yml only triggers on docs/**, and the file sits outside docs/tsconfig.json and npm test — so 123 lines that the comment itself calls the load-bearing half have no coverage at all. One line in lint.yaml:

      - name: Check extension syntax
        run: node --check images/jupyterlab/vscode-activity-reporter/extension.js

Related, and the reason I'd want that: _vscode_reporter_installed() globs for the extension directory, which only proves something got unzipped. A broken extension.js installs perfectly cleanly — build-vsix.py just zips bytes and --install-extension never runs the module — so the directory appears, the gate opens, update_last_activity goes False, and an actively-working user gets culled. Same for a stale 0.1.0 dir when a --force install fails, since the glob matches any version. Having activate() write a marker keyed to the extension version or vsix hash would make the gate mean "the reporter actually ran" — a plain marker on the PVC goes sticky and won't survive a later broken upgrade.

CODE_EXTENSIONSDIR and CODE_WORKINGDIR are set nowhere in the repo — no values.yaml, no extraEnv, no Dockerfile, no spawner env, no docs. Only readers and test monkeypatches. They're betatim/vscode-binder's env contract and that package is gone, so the os.path.expanduser(...) fallback is the only branch that ever runs. Deleting them takes jupyter_server_config.py:109-112 and :128-130, the ${CODE_EXTENSIONSDIR:+...} expansion plus its comment at 01-spawner.py:1050-1058, 5 docstring lines at test_nss_wrapper_shared_dir.py:170-174, and two tests — and removes the reason the "keep the install location in sync" comment exists.

Docs: values-reference.md has no vscodeActivity entry, and per convention that's where field detail lives. Also configuration.md:74 — "set it to 0 to disable this feature" reads like it's about VS Code culling when 0 disables in-pod self-termination for notebooks too. The next bullet already glosses zero correctly, so it's really just the antecedent of "this feature" that's ambiguous.

A few smaller things, not worth blocking:

• Don't drop mechanism 2. It looks like ~118 redundant lines next to mechanism 1, but it's what bounds the terminal leak above once the tab closes — without it a leaked counter keeps the ext host pinging after disconnect and the pod never culls at all. It's the backstop for mechanism 1's failure mode.
• install-code-server.sh:8,14 write the version twice and the URL doesn't interpolate it, so a future bump touching one verifies tag A's installer while installing version B — and expected_sum binds to no version, so it keeps passing silently. wget --quiet ".../v${CODE_SERVER_VERSION}/install.sh" fixes it. The unchanged expected_sum itself is correct, by the way — both tags' install.sh are byte-identical and both match the pin, so nothing is broken there.
• Nothing logs which mode a pod landed in, and the two failure modes are "bills 24/7" and "kills an active user's session". A single print() of the effective mode is the cheapest thing in this PR.
• test_code_server_idle_timeout_env_matches_inpod_culler and test_activity_reporter_extension_installed are both "spawn a pod, assert one static fact" and each burns a full kind cluster. Merging them drops a leg for free; the env-var one is arguably a unit test since test_chart_derived.py already renders the chart and pins "900".
• Untested input worth adding: the empty string for shutdown-no-activity-timeout. The double-gate matrix is otherwise complete.

CI is green apart from files_are_visible_and_writable_to_groupmates, which is a network flake — helm dependency update got a connection reset from hub.jupyter.org in the unrelated shared-storage test. All four new e2e tests pass.

Worth a separate issue: the checksum pins a 15 KB script that then downloads a ~100 MB tarball with zero verification (curl -> mv -> tar -xzf, no checksum anywhere in the file). Pre-existing, and coder publishes no checksum or signature asset, so closing it means pinning a self-computed sha256 and recomputing on every bump — not this PR's job.

_{Parts of this review were drafted using AI generative tools.}

[tylerpotts]

@viniciusdc Looks like it was a network error that caused the CI to fail. It's green now

[viniciusdc]

Currently testing this on a live cluster

[viniciusdc]

Hey @tylerpotts, I ran this end to end on a local NIC (kind) cluster against 613a1f0 with the CI built images, keycloak auth through the gateway, and singleuserCuller.server.shutdownNoActivityTimeout: 180 so each run takes minutes instead of 15+ (just for quick iteration on testing). I watched the in-pod last_activity, the code-server extension host log and the reporter's output channel on every run.

The #208 fix itself holds up. To guarantee the control was working as expected, with vscodeActivity.enabled: false an idle background VS Code tab bumped last_activity 49 times in 8 minutes and the pod never culled, so the bug reproduces here. Now, with it on, the same idle tab culled on schedule (No kernels for 186 seconds; shutting down), both with the reporter active and without it. And once the reporter is running it does keep a working user alive: pings every 60 to 80s while typing, and the startup ping lands ~90ms after activation. So all working as expected to address the original issue 🚀

One small caveat found while testing, and it blocks this for me: the reporter never activates in VS Code's default state. For new users/sessions code-server opens /home/jovyan in Restricted Mode, and the extension doesn't declare untrustedWorkspaces support, so VS Code disables it until the user clicks Trust. Nothing on the pack side notices, because _vscode_reporter_installed() only checks that the directory exists, so the proxy opt-out still applies. Net effect: a user who never trusts the folder gets culled at shutdownNoActivityTimeout while actively typing, with nothing in any log.

Repro from my run, Restricted Mode, typing continuously from ~21:43:

21:42:18  last_activity (VS Code opening)
21:43 ->  typing; last_activity never moves again
21:45:27  server self-terminates

The extension host log for that session lists the usual vscode.debug-auto-launch and vscode.merge-conflict activating on onStartupFinished, but never nebari.nebari-activity-reporter, and no "Nebari Activity Reporter" output channel gets created. After trusting the folder, same image and same user, it shows up right away (_doActivateExtension nebari.nebari-activity-reporter, alongside vscode.git, which also only activates in trusted workspaces). One-line fix inline on package.json.

This is also the concrete case for the activation marker from my earlier review. The directory gate passed here while the reporter was never running, which is exactly the "installed but not actually reporting" gap. If activate() wrote the marker and jupyter_server_config.py gated on it, this would have degraded to over-spending instead of culling.

Smaller things, not blocking:

• onDidChangeWindowState counts every state change, including blur and going inactive (inline comment).
• After a cull, VS Code just loops "Attempting to reconnect" against a 302 to the hub, so the user never learns the server was stopped. Pre-existing, but this PR makes culling a VS Code session much more common, so maybe worth a line in the docs.
• code-server checks open-vsx.org/vscode/gallery/vscode/nebari/nebari-activity-reporter/latest for updates (404 today). The installed copy is "pinned": true so I don't think it would auto-update, but should we claim the nebari namespace on Open VSX anyway?
• The items from my previous review body are still open: the description still says the timeout derives from jupyterhub.cull.timeout, the update_last_activity=False wording in extension.js and the test docstring, node --check in lint, the unused CODE_EXTENSIONSDIR / CODE_WORKINGDIR handling, and the missing vscodeActivity entry in values-reference.md.

What I didn't cover in the browser: killing a terminal mid-command (the Set fix) and the tab-closed path for CODE_SERVER_IDLE_TIMEOUT_SECONDS.

[viniciusdc]

issue (blocking): without this the reporter is disabled in Restricted Mode, which is what code-server opens /home/jovyan in by default, so an untrusted user gets culled while typing (repro in the review body).

The extension only listens to editor events and makes one local HTTP request, so declaring untrusted support is safe:

Suggested change

	"contributes": {}
	"capabilities": {
	"untrustedWorkspaces": { "supported": true }
	},
	"contributes": {}

[viniciusdc]

suggestion (non-blocking): this fires on every window state change, so losing focus and going inactive also count as activity. In my run a quick alt-tab through the VS Code window produced one ping on focus and a second one two minutes later on the way out, which pushed the cull back by a full timeout.

Counting only transitions into focused/active keeps the intent:

context.subscriptions.push(
  vscode.window.onDidChangeWindowState((s) => {
    if (s.focused || s.active) {
      recordActivity("focus");
    }
  }),
);

[viniciusdc]

going to approve this one since the main issue is addressed with this PR, and open a follow up issue for the extension for untrustedWorkspaces

[viniciusdc]

Moving this to approval, we can have the necessary untrusted changes in a follow-up