Skip to content

Latest commit

 

History

History
119 lines (104 loc) · 8.9 KB

limitations.md

File metadata and controls

119 lines (104 loc) · 8.9 KB

Limitations

Written to be read by whoever inherits this repo, including future me. Nothing here is hypothetical — each item is either observed behaviour or an explicit decision with a cost.

Not implemented

Missing Consequence Where to fix
Queue position A message waiting behind maxConcurrency shows the working indicator at once, but not its place in line Report the position after N seconds of waiting
Fact extraction for memory/ Notes are kept by the agent following AGENTS.md, and the adapter asks for a tidy-up when MEMORY.md grows or repeats; nothing extracts facts after a conversation on its own A curator pass on a schedule over recent sessions
A custom DSH_HOME The services, the sandbox runner and the health probe read ~/.dsh; the installer refuses another harness home rather than half-support it Pass DSH_HOME through every plist and read it everywhere
Desktop control The agent cannot see the screen, or use the mouse and keys in another app. A page it reaches with the browse tool; an app only through a command line or an MCP server of its own. Its commands cannot script or start another app, take the screen, or touch the clipboard Decided against for now (PLAN.md). The harness has no tool for it, and an MCP desktop server would run outside the sandbox with Accessibility and Screen Recording, for every conversation
Semantic recall A message that points at the past has the conversation's model choose among the channel's latest passages (recallPickFrom, 40) and relate words to it; older passages are matched by words alone (rare words in common, Korean particles stripped), so one about an older past put in other words may still miss. The two model calls cost up to recallExpandMs (4 s) of the turn's time, and the choice up to about 8,000 characters of passages (a few thousand tokens) at the default 40. Trigger words match inside other words (전에 in "배포 전에", again in "against"), so some messages ask for no reason Embed and index externally; a new runtime dependency is a decision, not a detail. recallExpandMs: 0 turns the call off

Verified partially, or not at all

  • Recall is the adapter's, not the agent's. recall/deepbot-recall.ts indexes past sessions into $DSH_HOME/recall/<home>/recall-index.jsonl on a schedule; the adapter ranks passages from the same channel and injects them before a turn. Sandboxed commands cannot read the index. The harness's own FTS index held 0 rows when this was built and is not used.

  • Duplicate-event suppression. The rule is implemented and the code path was exercised, but no real Slack redelivery was observed, so the end-to-end path is untested.

  • Reconnect behaviour. Backoff and disconnect handling are implemented from the Socket Mode protocol; no live network interruption was induced.

  • Attachment/thread edge cases. Only plain threaded text has been exercised.

  • Reads are confined by policy, on two surfaces. Commands through the sandbox runner, file tools through a tools/pre-execute read scope. It is a policy boundary rather than a kernel one: the tool list has to grow with the composition, and a new tool that reads files another way would not be covered. A separate OS user per instance remains the stronger answer, and is not built. A file tool's read outside the workspace is asked for in the conversation; the agent's commands read an ordinary file outside without asking (they are refused only the user's credentials and what DEEPBOT_DENY_READ names), so the consent is the file tools', not a wall around the user's documents. Keychain items in an unlocked login keychain reach commands through security(1); only the keychain files are refused. The present tool's nested files[].path is not checked by the read scope (it is not composed in a deepbot profile today).

Design risks

  • Non-public harness APIs. Session driving, message construction, and the event fold are recovered from the shipped implementation. There are no type declarations to confirm they are public contract. A harness upgrade can break this plugin without a deprecation notice. Pin your harness version, and treat docs/harness-api-notes.md as the list of things to re-check on upgrade.
  • installModelSelection is not called. The shipped runners pin the model selection inside setup; this plugin passes agentOptions only, because the helper is not importable from a profile-installed bundle. It works in the configuration tested, but this is the most likely source of a subtle model-selection failure.
  • The harness is a release candidate (0.2.0-rc.2 at the time of writing). Bundles, patch grammar, and service names can change.
  • An MCP server is outside every gate. A server added to the profile runs as the gateway's child: not under the sandbox runner, able to read the bot's credential files. Its tools pass the approval gate (unless askTools names them), the read scope and the fetch allowlist, are shared by every conversation (tenants included), and what they return is not marked as instruction-like text. Add only servers you trust (docs/setup.md).
  • One FTS index owner per path. Running two gateways against the same index file is a documented constraint of that component, not something this plugin guards against.
  • One Socket Mode connection per Slack app. Two clients on one app token make Slack split events across them and both sides start dropping messages. Never run a migration in parallel on the same app unless you intend the cutover.

Operational caveats

  • launchd cannot read ~/Documents (macOS TCC). A job whose wrapper lives there dies with getcwd: Operation not permitted. service/install-service.sh copies the wrapper under $HOME for exactly this reason — do not "simplify" it away.
  • Session resume is bound to the working directory. Moving the agent home orphans every existing thread's session. Threads keep accepting messages, but each one starts fresh, which looks like amnesia rather than an error.
  • targetChannels: ['*'] answers in every channel the bot has joined. The plugin refuses to start with an empty list, but '*' is a deliberate act: an invited bot plus '*' plus replyMode: all will answer every human message in those channels.
  • Token handling. Credentials are read through the harness credential service; they are never written into plugin state. The plugin redacts xox…/xapp… patterns from its own log lines, but the log is still a plain file under $DSH_HOME/slack-state/. Treat it as sensitive.
  • Session logs are the memory substrate. Deleting a session log deletes the memory of that conversation; the FTS index is derived and will simply follow.

Verifying recall

INDEX=$(ls -d "$DSH_HOME"/recall/*/ | head -1)recall-index.jsonl
wc -l "$INDEX"                              # rows: grows as conversations are indexed
grep -c 'some phrase you said' "$INDEX"     # the phrase reached the index
node test/run-scenario.ts test/scenarios/recall-immediate.json

If the row count stays flat after conversations, the refresh job is not running: launchctl print gui/$(id -u)/ai.deepbot.recall-index.

  • A browser, signed in only where you signed it in. web_fetch runs no JavaScript; for such a page the agent runs dist/tools/browse.js, a headless browser that types, clicks and screenshots one page per run. A site that needs a sign-in is reached only through a session you keep for it (--login, then --state; docs/setup.md), and a site that refuses sign-in in a browser driven by automation stays out of reach. The session file holds the account's cookies, and the agent's commands can read it: whoever can talk to the bot in that workspace can browse as you there. A page the agent reads can tell it to send the file out, and commandNetwork does not stop that while public addresses are reachable. It needs Chromium's headless shell installed (the installer does it), and --login a browser with a window. It prints the page's own text, not that of frames inside it. Text given to --type is on the command line, where the process list and the session log show it: not for a password. Its proxy carries what the browser sends over TCP; WebRTC's UDP does not go through it. web_fetch is bounded by the tool gate (scheme, metadata service, optional allowlist) and opens public addresses only; the agent's own commands (curl) are not bounded by that gate, and reach anything unless commandNetwork is public or local (docs/setup.md). Bounded, a command that ignores a proxy (ssh, a raw socket) reaches nothing; a program compiled to send Apple Events to another app is not stopped, and MCP servers and the harness's own tools are not bounded by it. Its output is not run through the injection guard — that covers channel history, attachments and material arriving in a prompt, not a tool result.