feat(managed): mint agent API token with instance-ops permissions - #92
Merged
Merged
Conversation
- plugin/plugin.go: aggregate agent-side plugins (logs_processor, elastic logging/metric, agent enterprise) plus the framework enterprise data-processing processors (dissect, json, date, mutate, field_standardize, otel_normalize, drop_fields, drop_event, geoip) and the otlp_export processor, so their init() registrations fire on boot; fixes the broken _ "infini.sh/agent/plugin" import in main.go - agent.yml: document the opt-in logs_otlp_egress pipeline (consumer(logs) -> for_each(dissect, field_standardize) -> otlp_export -> gateway OTLP intake) - deps: maxminddb-golang, otlp/proto (via framework plugins)
…s it Ports internal/managed from pr/agent-managed-token-flow-20260524 (adapted to the current CreateAPIToken signature and post-register hook shape): at startup the agent creates an access token via the framework's token manager (persisted in its keystore), and after registering to the config manager exchanges it so the manager can call back into the agent (pipeline tasks, stats — the LogPilot ingestion detail panel).
…KEN]]) Follows the established MANAGED/REMOTE_CONFIG_SERVERS pattern so the one-time registration pass can be passed via -e ENROLLMENT_TOKEN=et-... without editing the yaml.
Dials the config manager's /ws endpoint (derived from configs.servers, carrying the instance ID and manager credential), handshakes via HELLO, and serves the manager's proxied HTTP requests as loopback calls against the agent's own web port — the manager can now reach agents that sit behind NAT/firewalls and are only one-way reachable. Reconnects every 5s; module priority 101 (after managed bootstrap so the manager token is already in the keystore).
…tly dropped
The websocket hub frames every message as "<MsgType> <payload>": the
proxied requests arrive as "PRIVATE reverse_request {json}". The
client dispatched on the FIRST space-delimited token ("PRIVATE"), so
every request fell through the switch unanswered and the manager's
ProxyRequest hit its 30s deadline.
Strip the type prefix (CONFIG/PRIVATE), then dispatch on the command
(reverse_request). Matches the console agent's reference dispatch.
Requests also execute in a goroutine now, so a slow call no longer
blocks the read loop.
logs config gains queue_type: "kafka" routes tailed log envelopes into a kafka-type bus queue instead of the local disk queue (ship_direct path stays for direct OTLP). ship emitter tags envelopes with the source queue type; the kafka_queue plugin is imported in plugin.go/main.go so the backend is available purely via configuration. Includes the reverseclient import switch and go.mod/go.sum refresh that the current tree builds on.
Importing modules/queue exposes /queue/stats on the agent API, so the logpilot sampler can pull local staging queue depth/consumer offsets via the reverse channel (previously 404 — agent-side backlog was invisible).
…reverseclient The agent-side reverse websocket/HTTP channel now lives in the framework (modules/configs/reverseclient), shared with gateway — the binary already blank-imports it from main.go. Drop the local copy.
…tance - harvester: detect .gz paths and read the decompressed stream from the start (immutable archives — offsets in it are not comparable across runs, so state tracking stays mtime-based) - file detector: track files seen in the last walk; a re-created path carrying the same file identity (rename/move) inherits the old offset instead of re-reading from the beginning
- kafka_queue: local bus backend for transport=kafka streams (logs processor writes, gateway consumes) - transfer.receiver_port_range: 30000-30100 for receiver listeners - enable web.embedding_api for the aggregation UI
/elasticsearch/logs/_list and /elasticsearch/logs/_read accepted any caller-supplied logs_path (e.g. /etc) and served whatever the agent process could read. Reads are now confined to: - elasticsearch_logs.allowed_paths from the agent config, the escape hatch for layouts discovery cannot see - log directories the local search nodes report themselves (settings path.logs, path.home/logs fallback), discovered via the same process scan the console derives its paths from (60s TTL cache, stale cache survives refresh failures, no source at all means deny) System paths are never readable (util.IsSystemReadPath), traversal and symlink escapes are rejected by util.ReadGuard, non-regular files are skipped (a fifo no longer hangs row counting), and denied requests get 403 with a remediation hint. Console/cloud flows keep working unchanged since the paths they pass come from the agent's own discovery. 🤖 Generated with ZCode
…out blocking The whitelist missed directories the console derives from the search processes' command lines, so GC logs placed outside settings path.logs (-Xlog:...:file=) or a -Des.path.logs override returned 403. The agent now mirrors that derivation (path.logs override, path.home/logs, gc file dir) on top of the node settings scan, and logs_path may be a subdirectory of a whitelisted root via ReadGuard.ContainsUnder. Refreshes no longer hold request handling hostage: once a guard exists it serves even when stale while a single background refresh runs; only the first caller builds synchronously. 🤖 Generated with ZCode
The framework's operational routes (stats, queue browsing, config list/runtime, keystore read/write, pipeline task ops) are moving to the web port behind RequireLogin + RequirePermission, and the permission filter denies by default. The self-minted AGENT_API_ACCESS_TOKEN therefore carries security.InstanceOpsPermissionKeys() so manager- originated calls (reverse channel loopback, direct endpoint access) keep passing. Existing tokens were minted with no permissions; they are now re-minted at startup when the stored record no longer covers the required keys, and the exchange flow re-registers the fresh token with the manager. Depends on infinilabs/framework instance-ops-routes-rbac.
medcl
marked this pull request as draft
September 10, 2026 14:53
…ssions # Conflicts: # go.mod # go.sum
Replace the remaining Chinese comments in the PR diff (agent.yml, main.go, plugin/logs) with English equivalents and apply the gofmt alignment fix in plugin/logs/logs.go required by the CI format check.
- agent.yml: keep WEB_BINDING on port 23000, default MANAGED to false, restore the previous memory-limit comment, and remove the active kafka_queue section (opt-in via the documented example instead) - plugin/plugin.go: rely on make update-plugins discovery for enterprise plugins instead of importing them from source, so public builds resolve without the private checkouts - go.mod/go.sum: go mod tidy against the public tree (matches the convention on main)
medcl
marked this pull request as ready for review
September 17, 2026 11:58
The CI checkout of framework main does not ship util.ReadGuard yet (it lands with framework#420, still unmerged), which broke the plugin/api build and with it unit_test and code_lint. Vendor the framework's saferead (ReadGuard, NewReadGuard, IsSystemReadPath, canonicalPath) plus its tests into lib/util and switch plugin/api/log_whitelist.go to the local copy. Drop the vendored files and go back to infini.sh/framework/core/util once a framework release carrying #420 becomes a hard dependency.
- logs processor: derive scanCtx from the pipeline context instead of forwarding cancellation through a watcher goroutine; the embedded stdlib cancelCtx makes context.WithCancel link the cancellation via the parentCancelCtx fast path, so no extra goroutine is spawned and in-flight walks still abort when the pipeline stops - es logs whitelist: the CAS-winning request refreshes the stale cache inline while concurrent requests keep serving the cached guard; removes the background refresh goroutine (and with it the unrecovered-panic risk on a bare goroutine)
3 tasks
SteveLauC
approved these changes
Sep 18, 2026
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Mint the self
AGENT_API_ACCESS_TOKENwithsecurity.InstanceOpsPermissionKeys()so manager-originated calls keep passing after the framework moves its operational routes (stats, queue browsing, config list/runtime, keystore read/write, pipeline task ops, log tail) onto the web port behindRequireLogin+RequirePermission.The permission filter denies by default, and existing deployed tokens were minted with no permissions — so
getOrCreateAgentAPITokennow:Dependencies
instance-ops-routes-rbac— providesInstanceOpsPermissionKeys, merged)instance-log-tail-api—util.ReadGuard): the saferead implementation is vendored intolib/utilso the build no longer waits on it; drop the vendored copy once a framework release carrying #420 becomes a hard dependencyfeat/log-pipeline-aggregatorowns the exchange flow being patched); merge after both.