feat(telemetry): adopt shared PostHog schema v2 (server events, features, flush)

TELEMETRY.md

@@ -0,0 +1,99 @@
+# flAPI Telemetry
+
+flAPI collects **anonymous, aggregate usage telemetry** to help us understand
+which capabilities are used and where the product breaks. It is designed to be
+privacy-preserving by construction: only bounded enumerations, route
+*templates*, and numbers ever leave the machine — **never** your data, queries,
+URLs, credentials, or configuration.
+
+Telemetry is emitted against the shared DataZoo telemetry schema
+(`telemetry_schema: 2`) via the [`posthog-telemetry`](https://github.com/DataZooDE/posthog-telemetry)
+library, into PostHog's **EU** ingestion endpoint (`eu.i.posthog.com`). Because
+flAPI is a long-running server (not a DuckDB extension), events carry
+`install_kind = "server"` and a single session id per server uptime.
+
+## How to turn it off
+
+Any **one** of the following disables all telemetry — a single guard
+short-circuits every event, and nothing leaves the machine:
+
+- **Environment variable:** `DATAZOO_DISABLE_TELEMETRY=1` (also `true` / `yes`),
+  or `FLAPI_NO_TELEMETRY=1`.
+- **Config file** (`flapi.yaml`):
+  ```yaml
+  telemetry:
+    enabled: false
+  ```
+- **CLI flag:** `flapi --no-telemetry`.
+
+For very high-QPS deployments you can down-sample the per-request/per-tool
+events (lifecycle events are always sent in full):
+
+```yaml
+telemetry:
+  sample_rate: 0.1   # emit 10% of rest_endpoint_served / mcp_tool_called; events carry sample_rate
+```
+
+## What is collected
+
+Every event carries a common envelope from the library: `product` (`flapi`),
+`product_version`, `product_edition` (`oss`/`enterprise`), `telemetry_schema`,
+`os`, `arch`, `platform`, `is_ci`, `is_container`, a per-uptime `$session_id`,
+a pseudonymous per-machine `distinct_id` (salted SHA-256 of the OS machine id;
+identifies a *machine/install*, not a person), and `$groups` once associated.
+flAPI additionally stamps `install_kind = "server"` on every event.
+
+### Events
+
+| Event | When | Properties (beyond the envelope) |
+|---|---|---|
+| `server_started` | server boot | `endpoint_count` (number), `auth_kind` ∈ `none`\|`basic`\|`bearer`\|`oidc` |
+| `rest_endpoint_served` | a REST endpoint is served | `method` (`GET`/`POST`/…), `route_template` (e.g. `/customers/:id` — **the template, never the filled path**), `status_class` ∈ `2xx`\|`3xx`\|`4xx`\|`5xx`, `duration_ms` (number), `cache_hit` (bool) |
+| `mcp_tool_called` | an MCP tool call completes | `tool` (registered tool name), `status_class`, `duration_ms` (number) |
+| `auth_enforced` | auth is enforced on a request | `auth_kind` ∈ `basic`\|`bearer`\|`oidc`, `outcome` ∈ `allow`\|`deny` |
+| `$exception` | a request fails | `error_class` ∈ `server_error`\|`bad_request` (REST), `feature`, `route_template` (template) |
+
+Sampled events additionally carry `sample_rate` (number) so aggregate counts
+scale back up.
+
+### Groups
+
+- **`deployment`** — always associated at boot; key is the pseudonymous
+  per-machine `distinct_id`. Powers active-deployment and retention analytics.
+- **`account`** — associated only when a license id is present
+  (`FLAPI_LICENSE_ID`); key is `sha256(license_id)`. The raw license id is never
+  sent.
+
+## What is **never** collected
+
+By design, the following never appear in any property — call sites only pass
+enums/templates/numbers, and the library additionally clamps every string
+property to 512 bytes as a backstop:
+
+- Filled request paths or query strings (only the **route template**).
+- SQL text, rendered templates, or query plans.
+- Request or response **bodies**, or any **row data**.
+- HTTP **headers**, tokens, passwords, or connection strings.
+- Table names, file paths, hostnames, or user names.
+- Free-form error messages (only an enumerated `error_class`).
+
+## Notes on specific fields
+
+- **`cache_hit`**: flAPI's cache is a materialized DuckLake table that an
+  endpoint's SQL queries against — there is no per-request "served-from-cache"
+  flag. This field therefore reports whether the **endpoint is cache-backed**
+  (`cache.enabled` with a cache table), not a true per-request hit/miss.
+- **`$session_id`**: one id per **server uptime**, so PostHog Paths reflect what
+  a deployment served during a single run. It is never tied to any end-user
+  identity.
+
+## Where this lives in the code
+
+- Facade + gating: `src/flapi_telemetry.{hpp,cpp}` (`flapi::GlobalTelemetry()`).
+- Boot / shutdown (server_started, groups, `Flush()` on SIGINT/SIGTERM):
+  `src/main.cpp`.
+- `rest_endpoint_served` + errors: `src/api_server.cpp` (`handleDynamicRequest`).
+- `mcp_tool_called`: `src/mcp_tool_handler.cpp` (`executeTool`).
+- `auth_enforced`: `src/auth_middleware.cpp` (`before_handle`).
+- Tests (incl. a no-leak assertion against the real transport):
+  `test/cpp/test_flapi_telemetry.cpp`.

src/api_server.cpp

@@ -3,6 +3,9 @@
 #include "api_server.hpp"
 #include "auth_middleware.hpp"
 #include "database_manager.hpp"
+#include "flapi_telemetry.hpp"
+
+#include <chrono>
 #include "config_service.hpp"
 #include "config_tool_adapter.hpp"
 #include "open_api_doc_generator.hpp"
@@ -198,29 +201,48 @@ void APIServer::setupHeartbeat() {
     heartbeatWorker->start();
 }
 
-void APIServer::handleDynamicRequest(const crow::request& req, crow::response& res) 
+void APIServer::handleDynamicRequest(const crow::request& req, crow::response& res)
 {
+    const auto t0 = std::chrono::steady_clock::now();
     std::string path = req.url;
     // Match endpoint by both path and HTTP method
     std::string method = crow::method_name(req.method);
     const auto& endpoint = configManager->getEndpointForPathAndMethod(path, method);
 
+    // Emit one rest_endpoint_served with the ROUTE TEMPLATE (never the filled
+    // path), status class, duration, and whether the endpoint is cache-backed.
+    // Errors are emitted with an enumerated class only (no message/SQL/body).
+    auto emit_telemetry = [&](const std::string& route_template, bool cache_hit) {
+        const double duration_ms =
+            std::chrono::duration<double, std::milli>(
+                std::chrono::steady_clock::now() - t0).count();
+        auto& telemetry = flapi::GlobalTelemetry();
+        telemetry.restEndpointServed(method, route_template, res.code, duration_ms, cache_hit);
+        if (res.code >= 500) {
+            telemetry.error("server_error", "rest_endpoint_served", route_template);
+        } else if (res.code == 400) {
+            telemetry.error("bad_request", "rest_endpoint_served", route_template);
+        }
+    };
+
     if (!endpoint) {
         res.code = 404;
         res.body = "Not Found";
         res.end();
+        emit_telemetry("<unmatched>", false);
         return;
     }
 
     std::vector<std::string> paramNames;
     std::map<std::string, std::string> pathParams;
-    
+
     bool matched = RouteTranslator::matchAndExtractParams(endpoint->urlPath, path, paramNames, pathParams);
 
     if (!matched) {
         res.code = 404;
         res.body = "Not Found";
         res.end();
+        emit_telemetry("<unmatched>", false);
         return;
     }
 
@@ -243,6 +265,11 @@ void APIServer::handleDynamicRequest(const crow::request& req, crow::response& r
     }
 
     requestHandler.handleRequest(req, res, *endpoint, pathParams, auth_params);
+
+    // flapi has no per-request cache hit/miss signal (the "cache" is a
+    // materialized DuckLake table the template queries); the honest bounded
+    // value is whether this endpoint is cache-backed. See TELEMETRY.md.
+    emit_telemetry(endpoint->urlPath, dbManager->isCacheEnabled(*endpoint));
 }
 
 crow::response APIServer::getConfig() {

src/auth_middleware.cpp

@@ -18,6 +18,7 @@
 #include "auth_middleware.hpp"
 #include "password_hasher.hpp"
 #include "database_manager.hpp"
+#include "flapi_telemetry.hpp"
 #include "duckdb.hpp"
 #include "duckdb/common/types/blob.hpp"
 #include "oidc_provider_presets.hpp"
@@ -153,12 +154,16 @@ void AuthMiddleware::before_handle(crow::request& req, crow::response& res, cont
 
     CROW_LOG_DEBUG << "Auth enabled for endpoint: " << req.url;
 
+    // auth_kind is the endpoint's configured scheme (enum, never a secret).
+    const std::string auth_kind = endpoint->auth.type;
+
     auto auth_header = req.get_header_value("Authorization");
     if (auth_header.empty()) {
         CROW_LOG_DEBUG << "No Authorization header found";
         res.code = 401;
         res.set_header("WWW-Authenticate", "Basic realm=\"flAPI\"");
         res.end();
+        flapi::GlobalTelemetry().authEnforced(auth_kind, /*allow=*/false);
         return;
     }
 
@@ -180,6 +185,7 @@ void AuthMiddleware::before_handle(crow::request& req, crow::response& res, cont
     } else {
         CROW_LOG_DEBUG << "Authentication successful for user: " << ctx.username;
     }
+    flapi::GlobalTelemetry().authEnforced(auth_kind, ctx.authenticated);
 }
 
 bool AuthMiddleware::authenticateBasic(const std::string& auth_header, const EndpointConfig& endpoint, context& ctx) {

src/config_manager.cpp

@@ -136,7 +136,9 @@ void ConfigManager::parseMainConfig() {
 
         if (config["telemetry"]) {
             telemetry_enabled = safeGet<bool>(config["telemetry"], "enabled", "telemetry.enabled", true);
-            CROW_LOG_DEBUG << "Telemetry Enabled: " << telemetry_enabled;
+            telemetry_sample_rate = safeGet<double>(config["telemetry"], "sample_rate", "telemetry.sample_rate", 1.0);
+            CROW_LOG_DEBUG << "Telemetry Enabled: " << telemetry_enabled
+                           << ", sample_rate: " << telemetry_sample_rate;
         }
     } catch (const std::exception& e) {
         CROW_LOG_ERROR << "Error in parseMainConfig: " << e.what();