feat: organization-scoped event custom fields with management UI and filter integration#1815
Draft
ejsmith wants to merge 131 commits into
Draft
feat: organization-scoped event custom fields with management UI and filter integration#1815ejsmith wants to merge 131 commits into
ejsmith wants to merge 131 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds custom field support for events by updating the event model and repository logic, removing legacy query visitor code, and updating test infrastructure to support the new behavior.
- Removed obsolete EventFieldsQueryVisitor usage and file.
- Introduced automatic custom field creation in the event repository.
- Updated the PersistentEvent model to implement virtual custom fields and modified test configurations accordingly.
Reviewed Changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/Exceptionless.Tests/Search/PersistentEventQueryValidatorTests.cs | Removed legacy query visitor call in tests. |
| tests/Exceptionless.Tests/Search/EventIndexTests.cs | Updated repository query to include organization filtering. |
| tests/Exceptionless.Tests/Migrations/FixDuplicateStacksMigrationTests.cs | Set log level to Trace for migration tests. |
| tests/Exceptionless.Tests/AppWebHostFactory.cs | Added Kibana container configuration. |
| src/Exceptionless.Core/Repositories/Queries/Visitors/EventFieldsQueryVisitor.cs | Removed unused visitor implementation. |
| src/Exceptionless.Core/Repositories/EventRepository.cs | Added auto-creation of custom fields and related tenant logic. |
| src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs | Removed legacy dynamic mapping for event index custom fields. |
| src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs | Added index creation for custom fields. |
| src/Exceptionless.Core/Models/PersistentEvent.cs | Updated Idx property type and implemented IHaveVirtualCustomFields. |
Files not reviewed (1)
- src/Exceptionless.Core/Exceptionless.Core.csproj: Language not supported
| public IDictionary<string, object?> Idx { get; set; } = new DataDictionary(); | ||
|
|
||
| object? IHaveVirtualCustomFields.GetCustomField(string name) => Data?[name]; | ||
| IDictionary<string, object?> IHaveVirtualCustomFields.GetCustomFields() => Data ?? []; |
There was a problem hiding this comment.
Using an empty array literal '[]' as a fallback for an IDictionary<string, object?> is invalid. Replace '[]' with an appropriate empty dictionary initializer (e.g. 'new DataDictionary()').
Suggested change
| IDictionary<string, object?> IHaveVirtualCustomFields.GetCustomFields() => Data ?? []; | |
| IDictionary<string, object?> IHaveVirtualCustomFields.GetCustomFields() => Data ?? new DataDictionary(); |
Updates various NuGet packages, including .NET extensions to 10.0.4, Aspire to 13.1.2, and several testing and instrumentation libraries.
* Add project notification settings maintenance cleanup * Fix organization cleanup review feedback * Revert cleanup notes from agents * Refine notification cleanup logic and update agent skills Adjusts the processing loop in the notification settings cleanup job to improve batch handling and timing. Also updates AI agent skill documentation to provide clearer usage instructions and normalize code formatting across the skill library.
Fixes broken relative links across the skill library and updates backend testing snippets to use System.Text.Json-based serializers. Additionally, adds a strict consistency rule for frontend development requiring the use of shared formatter components.
Introduces specialized personas for engineering, triaging, and code review alongside a formal "observe-inspect-amend" cycle for self-improving agent skills. This establishes a structured, automated workflow for the development lifecycle, including security-first PR reviews and root-cause-driven triage.
Introduces a smart build task that detects the active file extension and automatically executes the appropriate build command (npm or dotnet). This streamlines the development workflow when switching between frontend and backend files.
Updates Foundatio and Microsoft library versions. This includes migrating from `AddPropertyRequiredForRemove` to `AddRequiredField` and optimizing cache invalidation logic by checking `PatchAsync` results.
…Text.Json Removes default initializers for dictionary properties in core models to reduce unnecessary allocations and handles resulting nullability in the event pipeline. This also completes the migration of the test suite from Newtonsoft.Json to System.Text.Json, updating serialization logic and helpers.
Switches from Include to Update for appsettings.yml files to avoid potential item duplication, as these files are typically already included by the .NET SDK's default globbing patterns.
Changes the response status code for marking stacks as fixed or snoozed from 202 Accepted to 200 OK to accurately reflect that the operations are completed. This also updates the corresponding XML documentation and OpenAPI specification.
Updates agent definitions to include preferred model assignments (Sonnet and Opus) and adds a new pre-planning step for the engineer agent to fetch PR comments and CI status via the GitHub CLI. This also restructures the review loop to be non-blocking, allowing the agent to address existing feedback while waiting for CI and automated reviews to complete.
* refactor: migrate from AutoMapper to Mapperly - Replace AutoMapper (runtime reflection) with Mapperly (compile-time source generation) - Add Riok.Mapperly 4.2.1 to Exceptionless.Web - Remove AutoMapper 14.0.0 from Exceptionless.Core Breaking changes: - Controllers now use abstract mapping methods instead of generic MapAsync<T> - Base controllers require derived classes to implement MapToModel, MapToViewModel, MapToViewModels Mapping structure: - Created dedicated mapper files per type in src/Exceptionless.Web/Mapping/ - OrganizationMapper: NewOrganization -> Organization, Organization -> ViewOrganization - ProjectMapper: NewProject -> Project, Project -> ViewProject - TokenMapper: NewToken -> Token, Token -> ViewToken - UserMapper: User -> ViewUser - WebHookMapper: NewWebHook -> WebHook - InvoiceMapper: Stripe.Invoice -> InvoiceGridModel - ApiMapper facade delegates to individual mappers Testing: - Added comprehensive unit tests for all mappers (29 tests) - Tests follow backend-testing skill patterns Benefits: - Compile-time type safety for mappings - Better performance (no runtime reflection) - Cleaner separation of concerns with per-type mappers refactor: improve Mapperly mapper safety and configuration - Add [MapperIgnoreTarget] for computed/populated-later properties on OrganizationMapper (IsOverRequestLimit, IsThrottled, ProjectCount, StackCount, EventCount) and ProjectMapper (HasPremiumFeatures, OrganizationName, StackCount, EventCount) - Upgrade UserMapper to RequiredMappingStrategy.Target since it only maps Model→ViewModel; new ViewUser properties will produce compile warnings unless explicitly mapped or ignored - Add [MapperIgnoreTarget] for ViewUser.IsInvite (manually constructed) - Let Mapperly generate collection methods (MapToViewTokens, MapToViewUsers) natively instead of manual .Select().ToList() - Add SuspensionCode enum→string mapping tests - Fix StackService ValueTuple serialization bug (restore StackUsageKey) - Update Mapperly 4.2.1 → 4.3.1 * PR feedback * fix: align serialization input with localhost domain and update stale AutoMapper comments The event-serialization-input.json still used test.example.com while the response baseline was updated to test.localhost, causing CI failure. Also updated stale AutoMapper references in test comments to reflect Mapperly migration. * Apply suggestion from @niemyjski * fix: deep-copy Roles and OrganizationIds in UserMapper Mapperly does shallow reference assignments for collection properties, which means mutating ViewUser.Roles (e.g. removing GlobalAdmin in UserController) would also mutate the source User model. Fixed by using a core mapper plus manual deep-copy via HashSet constructor. Added regression test. * refactor: clarify TokenMapper Type test name and comment NewToken has no Type property, so the test was really just asserting the C# enum default (Authentication=0). Rename the test and update the comment to explain: NewToken lacks Type, [MapperIgnoreTarget] makes the intent explicit, and the controller sets Type=Access in AddModelAsync.
* System Admin Page * fix: address PR review feedback - Fix typo: 'Highest completed versioned' → 'Highest completed version' - Fix Migrations sidebar icon: Play → DatabaseZap to match routes.svelte.ts - Fix nullable dereference warnings in AdminControllerTests: split Terms()?.Buckets into separate assertions - Fix generic catch clause in AdminController: catch Exception, log error before returning Problem * pr feedback * fix: replace all underscores in snapshot status Update the snapshot status cell to use `replaceAll` instead of `replace`, ensuring that status values with multiple underscores are correctly formatted with spaces in the UI. * PR Feedback * PR Feedback * PR feedback * PR Feedback
Enhance agent metadata in both the individual agent definitions and AGENTS.md to include specific "Use when" scenarios and keyword triggers. This helps improve agent selection and intent matching for tasks like feature implementation, PR reviews, and issue triage.
* feat(intercom): add svelte-intercom provider, hidden launcher, reactive boot options, and logout shutdown * fix(intercom): consolidate update calls and fix context scoping in snippet - Consolidate redundant intercom.update() calls into single effect tracking route and visibility - Extract getIntercom() into separate snippet to avoid context scoping issues - Both changes prevent duplicate updates and ensure context is properly available * Keep Intercom support flows on the message list Root cause: the sidebar support action was opening a fresh composer instead of the existing message history, and the new auth test needed immediate consistency so the seeded user was visible before login. The current staged set also carries the Intercom boot/auth wiring that this branch has been validating. * triage template * feat(intercom): implement token refresh and support link fallback - Adds JWT expiration (1 hour) and issued-at claims to the Intercom token on the backend. - Configures the frontend query to automatically refresh the token every 55 minutes to ensure continuous authentication. - Centralizes help and documentation links and provides a fallback to GitHub issues when the Intercom messenger is unavailable or disabled. * Tighten Intercom token refresh and contracts Root cause: the Intercom token query reused a static cache key across auth-session changes, and the new auth/intercom contract updates were missing their HTTP and OpenAPI baselines. * Minimize Intercom lockfile changes Root cause: the initial package-lock regeneration pulled in unrelated transitive resolver updates instead of only recording the new Intercom packages required by the feature. * Refactor Intercom setup into a dedicated shell component - Encapsulates Intercom provider, initializer, and fallback logic into a reusable `IntercomShell` component. - Simplifies the main app layout by removing nested provider logic. - Updates the backend auth controller to correctly return 401 for unauthenticated Intercom token requests and aligns the OpenAPI documentation. * Cleanup and format Intercom shell components - Standardize file line endings across Intercom-related files. - Reorder imports and alphabetize mock keys in the test suite. - Refine property formatting for consistency in the Intercom shell component and main app layout. * fixed linting
- Bump actions/cache from v4 to v5 - Update Node.js version from 22 to 24 in copilot-setup-steps.yml to align with build.yaml
* Polish saved views and filters * Fix faceted filter formatting * Fix client lint issues * Update OpenAPI saved view sort baseline * Address saved view PR feedback
* feat: Add stacks dashboard page route and API query - Add getStacksQuery function for listing stacks with filtering - Create /stacks route with basic stacks list page - Support filtering by status, tags, date ranges, and custom Lucene expressions - Display stack information: title, tags, event count, first/last occurrence, status Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build: Simplify stacks page to avoid table library issues - Remove complex TanStack table setup that was causing build errors - Use semantic HTML table with Tailwind styling for simplicity - Implement row selection with checkboxes and bulk actions UI - Support filter input, limit selector, and pagination controls - Display stack properties: title, tags, event count, last occurrence, status - Add loading and empty state handling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Add stacks bulk actions button with all operations - Create StacksBulkActionsButton component for simplified bulk operations - Support mark open/fixed/snoozed/ignored/discarded and delete actions - Integrate with RemoveStackDialog and status change dialogs - Wire bulk actions into stacks page with selection tracking - All dialogs and confirmations included Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Add stacks dashboard to navigation menu - Add stacks navigation item to Dashboards group - Use layers icon for visual consistency - Place between issues and event stream - Accessible from sidebar navigation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: Add stacks filter and bulk operation examples to HTTP tests - Add filter query examples (status:discarded, tags:production, type:error) - Add bulk operation examples (change status, mark fixed, mark snoozed, delete) - Document filtering syntax for stacks dashboard feature - Help users understand query capabilities Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: rewrite stacks dashboard using proper patterns - Rewrote stacks page using useFetchClient, createTable, getSharedTableOptions, and DataTable components matching the issues/sessions page patterns - Created proper table column definitions with NumberFormatter, TimeAgo, and badge components for status/tags - Created StacksDataTable component mirroring EventsDataTable - Uses existing TableStacksBulkActionsDropdownMenu (deleted the duplicate stacks-bulk-actions-button.svelte) - Moved navigation from Dashboards to Reports group - Removed broken getStacksQuery and list query key from api.svelte.ts - Uses FacetedFilter system for filtering (status, project, date, etc.) - WebSocket StackChanged event handling for real-time updates - Offset-based pagination with page size selector - Fixed organization abbreviation (never use org) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: lint errors in stacks page Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: clear stale organization selection state - Clear persisted organization id when no organizations remain - Re-select first valid organization when persisted id is stale - Hide manage/billing links when current organization is invalid - Prevent organization settings layout from redirecting to undefined ids Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: preserve impersonation and stabilize stack refresh handling - Keep impersonated organization selections from being overwritten by org sync - Refresh stacks table on non-removed StackChanged events with throttling - Correct stacks HTTP samples to match ids path-segment routes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Add stacks dashboard page route and API query - Add getStacksQuery function for listing stacks with filtering - Create /stacks route with basic stacks list page - Support filtering by status, tags, date ranges, and custom Lucene expressions - Display stack information: title, tags, event count, first/last occurrence, status Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build: Simplify stacks page to avoid table library issues - Remove complex TanStack table setup that was causing build errors - Use semantic HTML table with Tailwind styling for simplicity - Implement row selection with checkboxes and bulk actions UI - Support filter input, limit selector, and pagination controls - Display stack properties: title, tags, event count, last occurrence, status - Add loading and empty state handling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Add stacks bulk actions button with all operations - Create StacksBulkActionsButton component for simplified bulk operations - Support mark open/fixed/snoozed/ignored/discarded and delete actions - Integrate with RemoveStackDialog and status change dialogs - Wire bulk actions into stacks page with selection tracking - All dialogs and confirmations included Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Add stacks dashboard to navigation menu - Add stacks navigation item to Dashboards group - Use layers icon for visual consistency - Place between issues and event stream - Accessible from sidebar navigation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: Add stacks filter and bulk operation examples to HTTP tests - Add filter query examples (status:discarded, tags:production, type:error) - Add bulk operation examples (change status, mark fixed, mark snoozed, delete) - Document filtering syntax for stacks dashboard feature - Help users understand query capabilities Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: rewrite stacks dashboard using proper patterns - Rewrote stacks page using useFetchClient, createTable, getSharedTableOptions, and DataTable components matching the issues/sessions page patterns - Created proper table column definitions with NumberFormatter, TimeAgo, and badge components for status/tags - Created StacksDataTable component mirroring EventsDataTable - Uses existing TableStacksBulkActionsDropdownMenu (deleted the duplicate stacks-bulk-actions-button.svelte) - Moved navigation from Dashboards to Reports group - Removed broken getStacksQuery and list query key from api.svelte.ts - Uses FacetedFilter system for filtering (status, project, date, etc.) - WebSocket StackChanged event handling for real-time updates - Offset-based pagination with page size selector - Fixed organization abbreviation (never use org) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: lint errors in stacks page Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: clear stale organization selection state - Clear persisted organization id when no organizations remain - Re-select first valid organization when persisted id is stale - Hide manage/billing links when current organization is invalid - Prevent organization settings layout from redirecting to undefined ids Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: preserve impersonation and stabilize stack refresh handling - Keep impersonated organization selections from being overwritten by org sync - Refresh stacks table on non-removed StackChanged events with throttling - Correct stacks HTTP samples to match ids path-segment routes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: align issue management and settings navigation UX Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: polish row interactions and issue detail links Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: clean up settings sidebar context Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: nest project settings under projects nav Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: show project submenu hierarchy in settings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: render project settings as nested sidebar tree Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add settings separator after organizations Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Polish sidebar hierarchy and issue management framing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move sessions into dashboards navigation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add collapsed sidebar hover flyouts for child nav Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert out-of-scope changes per PR review Reverted: Startup.cs (next build path), SavedView model/tests, organization-features extraction, events-overview pager, saved-view 422 error handling hack, issues page filter changes, project manage page link change. * Remove stacks page and navigation link per PR review The stacks dashboard was not ready for inclusion. Removed the route and its sidebar link from the layout. * Improve stacks table: rename severity to critical, fix tags cell UX - Renamed stack-severity-cell to stack-critical-cell (matches column name) - Added tooltip with Kbd for platform-agnostic shortcut hints on tags - Made overflow tags clickable in tooltip dropdown - Added getProjectStacksQuery tanstack wrapper in stacks API - Moved defaultColumnVisibility and getTableOptions to options file * Refactor project issues page to use TanStack Query Replaced manual useFetchClient calls with getProjectStacksQuery and getTableOptions from the options file. Removed unused showStackPager and onEventChange props from stack/event detail pages. * Cosmetic fixes: remove double newlines, simplify margin, extract \ - Removed extra blank lines in events and stream pages - Simplified account layout margin from mx-6 my-6 to m-6 - Extracted detailsHref into a \ in event-detail-sheet * Apply code formatting (npm run format) * Revert stacks view type additions from saved view models The stacks page was removed per review. These model changes adding 'stacks' as a valid ViewType are no longer needed. * Fix data-table click handling and stacks.http filter syntax - Don't intercept clicks on links/buttons in table cells (was breaking link navigation when rowHref not set) - Added keyboard accessibility (Enter/Space) to table cells - Fixed stacks.http filter: tags: → tag: (correct filter term) * Make org settings routes consistent, fix stacks viewType doc - Settings group now has all the same entries as Organization Settings (General, Usage, Billing, Features) for navigation consistency - Removed stale 'stacks' from SavedViewController viewType doc comment to match the reverted model validation * Revert tag filter from AND back to OR semantics Multi-tag filter should use OR (show events with any selected tag) not AND (require all tags). AND was an accidental breaking change. * Fix data-table keyboard a11y and stacks.http date format - Move tabindex from individual cells to row level (reduces tab stops) - Remove per-cell keydown handler, use row-level Enter/Space instead - Fix snoozeUntilUtc to use ISO-8601 format (2030-12-31T00:00:00Z) * Fix CI: prettier formatting and remove obsolete feature-gate tests - Fix prettier formatting in data-table-body.svelte (multiline onkeydown) - Remove 3 WhenFeatureDisabled tests since feature gate was removed from SavedViewController (saved views are now available to all orgs) * Fix ESLint curly rule: wrap if body in braces * Refactor feature flag documentation: remove specific feature references and clean up comments --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Polish next app navigation and settings UI * Fix next app client lint * Update Aspire and .NET build versions * Make event serialization test user agent deterministic
* Polish next UI saved views and sample data * Fix saved view hydration reset
* Polish dashboards and saved views * Fix PR build failures * Fix saved view patch schema metadata
…ests (#2250) * Add comprehensive UserController integration tests Tests cover all 12 UserController endpoints: - GetCurrentUser (me), GetById, GetByOrganization - Patch/Put user updates - DeleteCurrentUser, Delete (admin) - UpdateEmailAddress, VerifyEmail, ResendVerification - UnverifyEmailAddress (admin) - AddAdminRole, DeleteAdminRole (admin) Each endpoint tested for happy path, unauthorized access, and forbidden access (non-admin on admin endpoints). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add comprehensive API controller test coverage and serializer model tests Controller tests: - UserControllerTests: 22 tests covering all 12 endpoints - UtilityControllerTests: 5 tests for search/validate endpoint - StripeControllerTests: 3 tests for webhook error cases - StackControllerTests: ~20 new tests (ChangeStatus, Delete, GetAll, GetAsync, GetByOrganization, GetByProject, MarkCritical, MarkNotCritical, RemoveLink, Snooze) - StatusControllerTests: ~9 new tests (GetAbout, QueueStats, SystemNotification CRUD) Serializer model tests (ported from feature/system-text-json-v2): - TokenSerializerTests: roundtrip and snake_case deserialization - OrganizationSerializerTests: core properties, invites, suspension - LocationSerializerTests: full, partial, unicode roundtrip - MethodSerializerTests: all properties, minimal, snake_case - InnerErrorSerializerTests: nested errors, stack traces - ManualStackingInfoSerializerTests: signature data, special chars - SubmissionClientSerializerTests: IPv4, IPv6, minimal - UserDescriptionSerializerTests: data dictionary, minimal - SavedViewSerializerTests: columns, filters, minimal All tests use 3-part naming (Method_Scenario_Expected) with Arrange/Act/Assert pattern, inserted alphabetically. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OrganizationController PATCH tests and core utility tests - Add PatchAsync tests to OrganizationControllerTests: UpdateName, NonExistent, Anonymous, EmptyName validation - Add SemanticVersionParserTests: standard semver, pre-release, wildcard, cache - Add EnumerableExtensionsTests: CollectionEquals order/equality, GetCollectionHashCode - Add DictionaryExtensionsTests: CollectionEquals, AddRange, ContainsKeyWithValue - Fix inverted logic in EnumerableExtensions.CollectionEquals (returned false for equal elements) - Fix inverted logic in DictionaryExtensions.CollectionEquals (returned false for equal values, early-returned true on first null pair) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add comprehensive test coverage for ProjectController and AdminController ProjectController - new tests: - DeleteAsync: existing project removal, non-existent returns 404 - DeleteDataAsync: removes data key, invalid key returns 400, not found - GetAllAsync: returns user projects, respects limit parameter - GetByOrganizationAsync: returns org projects, invalid org returns 404 - GetConfigAsync: by project ID returns config - GetV2ConfigAsync: client auth returns config - GetNotificationSettingsAsync: admin gets all, user gets own - IsNameAvailableAsync: taken name returns 204, new name returns 201, org-scoped - PatchAsync: name change persists and preserves other fields, not found, extra props - PostDataAsync: valid persists, empty/dash key returns 400, not found - ResetDataAsync: clears stacks and events - SetNotificationSettingsAsync: persists settings, null removes settings AdminController - new tests: - GetSettings: returns app options, forbidden for non-admin, unauthorized - EchoRequest: returns headers/IP, forbidden, unauthorized - GetAssemblies: returns assembly list, forbidden, unauthorized - ChangePlanAsync: valid plan changes org, invalid plan fails, forbidden - SetBonusAsync: applies bonus, with expiration, invalid org returns 422, forbidden Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Token, WebHook controller tests and core extension unit tests - Add 14 new TokenController tests (delete, get by org/project, patch, auth) - Add 10 new WebHookController tests (CRUD, unsubscribe, auth) - Add StringExtensions unit tests (28 tests) - Add HashExtensions unit tests (11 tests) - Add JsonExtensions unit tests (11 tests) - Fix double blank lines in serializer test files - Clean up whitespace in Status/Stripe/User controller tests - Add content-unchanged assertions on error responses Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback: fix test patterns, DTO types, TimeProvider usage, and IDisposable - AdminControllerTests: remove System.Text.Json import, add private DTOs (AppSettingsDto, EchoResult, AssemblyInfo, ChangePlanResult), fix DateTime.UtcNow -> TimeProvider.GetUtcNow() in SetBonusAsync_WithExpiration - ProjectControllerTests: use TryGetValue instead of ContainsKey+indexer - StackControllerTests: rename now->utcNow/snoozeUntil->snoozeUntilUtc, add content-unchanged assertion after bad request - StatusControllerTests: restore System.Net import for HttpStatusCode - StripeControllerTests: using var StringContent, String.Empty, remove import - UtilityControllerTests: fix premium query to use tags:error (free field fix) - WebHookControllerTests: remove spurious RefreshDataAsync before GetByIdAsync - DictionaryExtensionsTests: fix 2-part names to 3-part - SemanticVersionParserTests: implement IDisposable for LoggerFactory Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix remaining test failures: status codes, index refresh, and private-setter deserialization - Fix IsNameAvailable tests: 204=available, 201=taken (not reversed) - Remove Assert.Single(Workers) from DeleteAsync test (project delete has no work items) - Fix PatchAsync_EmptyName: expects 400 BadRequest (not 422 UnprocessableEntity) - Fix GetCurrentUserAsync: remove IsEmailAddressVerified assertion (not set in sample data) - Fix GetByOrganizationAsync invalid org: GlobalAdmin gets 200 empty list (not 404) - Fix WebHook GetByProject: add RefreshDataAsync() before search-based GET - Fix WebHook Unsubscribe: add RefreshDataAsync() before unsubscribe (search-dependent) - Fix GetAssemblies: parse JSON directly (AssemblyDetail has private setters, STJ can't deserialize) - Add StatusCodeShouldBeNoContent() extension method Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address api coverage review feedback * test: expand web api coverage * test: cover stripe billing api paths * test: exclude generated code from coverage * test: expand web api coverage * test: address stripe billing review feedback * test: use fixture time in stripe service tests * fix: reuse stripe billing client * test: address token and pagination review feedback * test: harden api controller coverage * test: assert serializer wire contracts * Make new token project ID required The NewToken model's ProjectId property is now non-nullable, and the API schema is updated to reflect `project_id` as a required field. This change enforces explicit project association for all new tokens. When both `ProjectId` and `DefaultProjectId` are provided during token creation, `DefaultProjectId` will now be cleared. Attempts to create a token without a `ProjectId` will result in a validation error. * test: use no-content assertion helper * test: improve project patch API tests for contract adherence Refactor `ProjectController.PatchAsync` tests to send raw JSON payloads using `snake_case` property names. New assertions verify: - Read-only fields are ignored during patch updates. - Updatable fields are correctly applied while preserving others. - The API response consistently uses `snake_case` property naming, preventing potential serialization drift back to `PascalCase`. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nfrastructure Enables organizations to define indexed custom fields on events using Elasticsearch slot-based dynamic templates. Fields are managed through new CRUD endpoints on OrganizationController and automatically indexed during event ingestion via EventCustomFieldService. Key design decisions: - Remove base framework DocumentsChanging/BeforeQuery handlers in EventRepository to preserve fast delete-by-query path for RemoveAll - Subscribe to DocumentsAdding (not DocumentsChanging) so custom field processing only runs during ingestion, not removes - 8 slot-based dynamic templates support keyword, boolean, date, numeric, ip, object, text, and wildcard field types
Unit tests for field name validation, sensitive detection, type inference. Unit tests for PersistentEvent IHaveVirtualCustomFields implementation. Integration tests for Organization controller custom field CRUD endpoints. Integration tests for custom field indexing through the event pipeline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace CreatedAtAction with StatusCode(201) to avoid JSON serializer issues with Foundatio's CustomFieldDefinition record type. The GET endpoint correctly handles serialization for listing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Backend changes: - Remove dynamic templates from EventIndex (base handles via AddStandardCustomFieldTypes) - Remove DocumentsChanging.RemoveHandler from EventRepository (handled by service) - Rewrite EventCustomFieldService: simplified reserved field check (starts with @), expanded wildcard-based sensitive patterns, robust ConvertValue with strict parsing, removed InferIndexType (user must configure explicitly) - Add DataAnnotation validation to CustomFieldModels (IValidatableObject pattern) - Implement safe delete flow: check saved view filter usage (409 if in use), soft-delete, enqueue RemoveCustomFieldWorkItem for eventual hard deletion - Fix DI: RemoveCustomFieldWorkItemHandler uses IServiceProvider for lazy resolution - Register work item handler in Bootstrapper - Never abbreviate organization as org Frontend (Svelte 5): - Custom fields API client with TanStack Query (CRUD operations) - Management page at /organization/[id]/custom-fields with create/delete dialogs - Custom field filter builders component for dynamic filter integration - Index as Custom Field action in event extended-data-item - Navigation link in organization settings - All 243 frontend unit tests pass Tests: - 115 custom field tests pass (service, API, indexing, controller) - Updated OpenAPI baseline for new endpoints - Updated validation tests (400 → 422 for DataAnnotations) - Replaced InferIndexType tests with ConvertValue tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Simplify and polish org custom-fields management page: - Skeleton loaders instead of spinner during load - Empty state with icon, description, and CTA button - Type badge with tooltip descriptions in table - Added createdUtc (DateTime) column - Inline edit dialog for field description - Table wrapped in bordered container (matches billing page) - Pending-deletion rows shown with dimming and 'Pending deletion' badge - Real-time field name validation feedback in create dialog - Richer type selector with per-type descriptions in dropdown - Better error parsing from ProblemDetails (reads errors.general[0]) - Spinner on delete button during pending state - type="button" on Cancel buttons to prevent accidental form submit - Add INDEX_TYPE_DESCRIPTIONS to models for richer UI labels - Update index-as-custom-field-action to show per-type descriptions in dropdown - Fix error parsing in index-as-custom-field-action to read errors.general[0] - Remove fallback regex in IsCustomFieldUsedInFilter that could false-positive on built-in event fields (type:error, level:error); all custom field filter references use the idx. prefix so only that pattern is needed - Fix ConvertToLong double/float boundary overflow: use < (double)Int64.MaxValue instead of <= Int64.MaxValue to avoid the double rounding issue where (double)Int64.MaxValue > Int64.MaxValue - Remove legacy dynamic templates from EventIndex (handled by AddStandardCustomFieldTypes) - Remove sensitive field patterns and ValidIndexTypes from validation - Move frontend module from custom-fields/ to organizations/custom-fields/ Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove dead _appOptions field from EventCustomFieldService
- Remove IServiceProvider from RemoveCustomFieldWorkItemHandler; inject
ICustomFieldDefinitionRepository directly (simpler, testable)
- Fix PATCH: empty string description now clears the stored value (null)
instead of being silently ignored — previously impossible to clear
- Fix POST/DELETE BadRequest and Conflict to return proper ProblemDetails
(Problem() with statusCode) so the frontend error chain reliably surfaces
the message via error.detail
FindByTenantAsync already excludes soft-deleted definitions
- Compile the field-name regex once with RegexOptions.Compiled and a
1-second timeout; pass to per-document helper instead of reconstructing
it on every call
- Fix frontend isValidFieldName regex: [a-zA-Z0-9_.\\-] was matching
literal backslashes (double-escape in JS char class) — corrected to
[a-zA-Z0-9_.\-]
- Fix frontend handleEdit: always send description as a string (never
undefined) so clearing the textarea sends "" and triggers the clear path
- Add 6 new API tests:
- PostField_RejectDuplicateName
- PostField_RejectDuplicateName_CaseInsensitive
- DeleteField_ReturnsNotFound_WhenAlreadyQueuedForDeletion (documents
correct REST semantics — no double work-item enqueue possible)
- PatchField_ClearsDescription_WhenEmptyStringProvided
- DeleteField_ReturnsConflict_WithProblemDetails (verifies error.detail
contains field name)
- Retained DeleteField_ReturnsConflict_WhenFieldUsedInSavedViewFilter
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ents - ProcessEventCustomFields: iterate fieldMapping (≤20 entries) instead of building an intermediate GetCustomFields() dict from all event Data keys. DataDictionary is OrdinalIgnoreCase so direct Data.TryGetValue lookup preserves case-insensitive field name matching. Avoids O(Data.Count) allocation on the hot event-ingestion path for every event in a batch. - BuildFieldNameRegex: remove RegexOptions.Compiled from a per-request one-shot regex. Compiled triggers JIT upfront without amortization and is strictly slower for single-use patterns. - Tests: remove unused _organizationRepository field; rename misleading DeleteField_RemovesField → DeleteField_SoftDeletesField_AndHidesFromList (field is soft-deleted, not removed); add 3 cross-org IDOR tests verifying that a user in TEST_ORG gets 404 (not data) for FREE_ORG's custom fields on GET, POST, and DELETE endpoints. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The five legacy dynamic templates (idx_bool/*-b, idx_date/*-d, idx_number/*-n, idx_reference/*-r, idx_string/*-s) are STILL required. EventFieldsQueryVisitor translates 'data.field:value' queries to 'idx.field-s/n/b/d/r' (legacy suffix scheme), and CopySimpleDataToIdxAction writes to the same suffix keys. AddStandardCustomFieldTypes() registers templates for the NEW slot-based scheme (keyword-*, double-*, bool-*, etc.) — a completely different naming convention. These two systems coexist and neither replaces the other. Removing the legacy templates broke all data.* searches for events indexed with the legacy pipeline (GetByCustomDataAsync tests: 0 results instead of 1). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t coverage
- Normalize IndexType to lowercase before AddFieldAsync (PATCH/DELETE returned 404 for
soft-deleted via PATCH — was already returning correct 200/409/404)
- Add PATCH + double-DELETE return 404 tests; add data.{field} saved-view conflict test;
add uppercase IndexType normalization test; add unicode field name rejection test
- Fix TOCTOU: paginate through all FindByTenantAsync pages for POST quota check
- Fix soft-deleted definitions mutable via PATCH/DELETE (IsDeleted guard)
- Fix EnsureSystemFieldsAsync to check ALL system fields, not just sessionend
- Fix ConvertToFloat missing long case (range: -16777216 to 16777216)
- Fix ConvertToDate: use AssumeUniversal|AdjustToUniversal (tz-less strings → UTC)
- Fix IsValidFieldName: restrict to ASCII-only (Char.IsAsciiLetterOrDigit)
- Fix BuildFieldNameRegex to match both idx.{field} and data.{field} filter refs
- Fix RemoveCustomFieldWorkItemHandler: correct GetByIdAsync behavior comment;
add normal-path logging; slot intentionally NOT freed (reuse contamination risk)
- Remove unused MaxCandidateFields from CustomFieldOptions
- Fix bits-ui body scroll-lock not released after programmatic dialog close —
releases pointer-events:none/overflow:hidden when all dialogs are closed
- Fix TypeScript null error in handleDelete: capture fieldName before mutation
- Update docs/custom-fields.md: slot churn analysis, FAQ expansion, accurate lifecycle
- 118 backend tests pass; 243 frontend unit tests pass; 0 svelte-check errors
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ure/custom-fields # Conflicts: # src/Exceptionless.Core/Models/PersistentEvent.cs # src/Exceptionless.Core/Repositories/EventRepository.cs # tests/Exceptionless.Tests/Migrations/FixDuplicateStacksMigrationTests.cs # tests/Exceptionless.Tests/Search/EventIndexTests.cs # tests/Exceptionless.Tests/Search/PersistentEventQueryValidatorTests.cs
…ed log API - Remove duplicate Foundatio.Repositories.Elasticsearch PackageReference introduced by merge - Fix Log.DefaultMinimumLevel → Log.DefaultLogLevel (renamed in xunit.v3 upgrade) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+65
to
+70
| foreach (var kvp in Data) | ||
| { | ||
| if (String.IsNullOrEmpty(kvp.Key) || kvp.Key.StartsWith('@')) continue; | ||
| if (kvp.Value is string or bool or int or long or float or double or decimal or DateTime or DateTimeOffset) | ||
| result[kvp.Key] = kvp.Value; | ||
| } |
…end lint, address PR code quality feedback - Remove System.Text.Json PackageReference (NU1510 warning-as-error in .NET 10 CI) - Fix ESLint errors: perfectionist import/export ordering, curly braces, escape chars, svelte/require-each-key across custom-fields and event components - Fix prettier formatting in custom-fields Svelte/TS files - PersistentEvent.GetCustomFields(): use LINQ Where, return DataDictionary (Copilot feedback) - EventCustomFieldService: filter before foreach loop (CodeQL Where suggestion), narrow catch clauses to non-cancellation exceptions and typed conversion errors - OrganizationMaintenanceWorkItemHandler: narrow catch to non-cancellation exceptions - Fix ref.session query resolution: system fields use deterministic slots without tenant context - Regenerate OpenAPI baseline for CustomFieldDefinitionResponse DTO changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…PATCH system field guard
- Quota race condition: move field creation into EventCustomFieldService.CreateFieldAsync()
behind a distributed lock (ILockProvider, key: custom-field-create:{orgId}) so concurrent
requests from the same org cannot race past the 20-field quota check.
- haserror never written: add hasError parameter to PersistentEvent.UpdateSessionStart()
and wire it through EventRepository.UpdateSessionStartLastActivityAsync(). Session plugin
now computes hasError = session has any 'error'-type event and passes it to both call sites.
- Inbound Idx injection: ClearCustomFieldSlots now removes all new-format managed slot keys
(keyword-N, date-N, bool-N etc.) before re-populating from ev.Data, preventing clients
from injecting pre-computed slot values. Legacy keys (e.g. sessionend-d) are preserved
for backward-compatibility with pre-PR Elasticsearch data.
- PATCH system field guard: add IsSystemField check in PatchEventCustomFieldAsync (was
already guarded in DELETE but missing in PATCH).
Tests added:
- PatchField_RejectsSystemField
- PostField_ConcurrentCreation_AtQuotaLimit_OnlyOneSucceeds
- UpdateSessionStart_SetsHasError_WhenTrue
- Event_WithPrePopulatedIdx_IsStripped_AndRecomputedFromData
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d views - EventController and StackController return 426 when all organizations in scope are on free plans and the filter references premium features (idx.*) - SavedView.UsesPremiumFeatures is computed at save/update time via PersistentEventQueryValidator and exposed in ViewSavedView - Frontend shows upgrade dialog on 426 search responses and badges saved views that require a paid plan in the sidebar
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds opt-in, organization-scoped custom fields for event data — letting users promote arbitrary
Datadictionary keys into Elasticsearch-indexed fields for use in search filters, saved views, and analytics dashboards.This is a fully additive change with no breaking API or schema changes.
What Changed
Backend
PersistentEventIHaveVirtualCustomFieldsto connect eventDatato Foundatio's pooled-slot indexing infrastructureEventIndexAddStandardCustomFieldTypes()— removes dynamic templates (no longer needed)EventCustomFieldServiceIStartupActionthat wires the document-changing pipeline hook; provisions system fields (sessionend,haserror) per org; validates field names (ASCII, no@prefix, max length); converts and type-checks values before slot writeOrganizationController{id}/event-custom-fields: GET list, POST create, PATCH update, DELETE soft-delete, GET candidates from eventRemoveCustomFieldWorkItemHandlerCustomFieldOptionsMaxFieldsPerOrganization = 20,MaxSlotChurn = 100CopySimpleDataToIdxActionandEventFieldsQueryVisitor— replaced by first-class field definitionsAPI Endpoints (new)
Plan gating: Free-plan organizations receive
426 Upgrade Requiredon POST.Reserved names: Any field starting with
@is rejected (reserved for system use).Active quota: Maximum 20 active fields per organization (soft-deleted fields do not count toward the quota; they do occupy their slot).
Deletion Policy
Deletion is deliberately two-phase to prevent slot reuse corruption:
IsDeleted = true, removes the field from the UI, blocks new events from indexing into this slot. Enqueues a cleanup work item.Saved-view protection: DELETE returns
409 Conflictwith field name if any saved view filter references the field (checks bothidx.{field}anddata.{field}patterns).Frontend (Svelte 5)
/organization/{id}/custom-fieldsUpgradeDialog)keyword→ equals/not-equals/exists/missing;string→ contains/exists/missing;numeric/date→ range/gt/gte/lt/lte/exists/missing;bool→ true/false/exists/missing)Architecture Docs
See
docs/custom-fields.mdfor the full architecture, slot system internals, lifecycle, deletion policy, operator support, FAQ, and churn analysis.Test Evidence
CustomFieldApiTestsEventCustomFieldServiceTestsCustomFieldIndexingTestsPersistentEventCustomFieldsTestsKey test scenarios covered
@-prefix, too long, non-ASCII)IsDeleted=true→ 404)idx.{field}anddata.{field}patterns)sessionend,haserror) on first createlong→float(range guard), date → UTC, bool coercionKnown Limitations
Breaking Changes
None. All changes are additive. Existing events, queries, saved views, and API clients are unaffected.
Original Requirements
Custom Fields System for Exceptionless
Allow organizations to define indexed fields from event data properties, enabling rich search, filter, and analytics capabilities for user-defined data.
Core Requirements
keyword,string,int,long,double,float,bool,dateManagement UI Requirements
Filter Operator Requirements
keyword: equals, not-equals, exists, missingstring/full text: contains/search, exists, missingnumeric/date: equals, range, gt, gte, lt, lte, exists, missingbool: true, false, exists, missingDeletion Policy
System Fields
sessionend(date) andhaserror(bool) provisioned per org automatically