Redis backed Caching of Internal Embeddings by ajtiwari07 · Pull Request #3532 · Azure/data-api-builder

ajtiwari07 · 2026-05-11T20:00:12Z

1. Embeddings-Specific Cache Configuration

New EmbeddingsCacheOptions class for embeddings-specific cache settings
Configurable TTL (Time-To-Live) with default of 24 hours (optimized for deterministic embeddings)
Independent cache configuration from global runtime cache
Support for enabling/disabling caching at the embeddings level

2. Level 2 (L2) Distributed Redis Cache Support

New EmbeddingsCacheLevel2Options class for Redis-specific configuration
Dedicated Redis connection for embeddings cache
Independent from global runtime L2 cache (separate connection string)
FusionCache integration with Redis backplane for distributed cache invalidation
Graceful fallback to L1-only if L2 is not configured or fails

3. Enhanced Configuration Schema

Updated JSON schema with new cache section under embeddings
Support for environment variable substitution in Redis connection strings
Clear documentation and examples in schema
Validation for required fields when L2 is enabled

4. Dedicated FusionCache Instance for Embeddings

Separate FusionCache instance named "EmbeddingsCache"
Independent cache lifecycle from global DAB cache
Optimized entry options for embedding workloads
Error resilience with exception suppression for distributed operations

5. Comprehensive Test Coverage

Unit tests for EmbeddingsCacheOptions class (205 lines)
Unit tests for EmbeddingsCacheLevel2Options class (64 lines)
Serialization tests for cache options (233 lines)
Runtime cache L2 options tests (241 lines)
Redis connection startup tests (138 lines)

  "$schema": "https://github.com/Azure/data-api-builder/releases/latest/download/dab.draft.schema.json",
  "data-source": {
    "database-type": "mssql",
    "connection-string": "@env('DATABASE_CONNECTION_STRING')"
  },
  "runtime": {
    "embeddings": {
      "enabled": true,
      "provider": "azure-openai",
      "base-url": "https://my-openai.openai.azure.com",
      "api-key": "@env('AZURE_OPENAI_API_KEY')",
      "model": "text-embedding-ada-002",
      "dimensions": 1536,
      "cache": {
        "enabled": true,
        "ttl-hours": 72,
        "level-2": {
          "enabled": true,
          "connection-string": "@env('EMBEDDINGS_REDIS_CONNECTION_STRING')"
        }
      }
    }
  }```

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

…configuration Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

… and telemetry integration Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

…ate empty embeddings Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

…oint/health sub-objects Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

…orization Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

…lthCheckConfig in converter

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…dding controller tests, switching the dab schema for the embedding system to default to false

…icitly set

embeddings endpoint is now permanently fixed to /embed with no user-configurable path option. This removes unnecessary configuration surface since the feature has not been released yet, eliminating the need for backward compatibility. Changes: - Remove path property from dab.draft.schema.json - Remove Path, UserProvidedPath, and EffectivePath from EmbeddingsEndpointOptions - Remove EffectiveEndpointPath from EmbeddingsOptions - Remove path deserialization from EmbeddingsOptionsConverterFactory - Remove --runtime.embeddings.endpoint.path CLI option - Remove path configuration logic from ConfigGenerator - Remove endpoint path validation from RuntimeConfigValidator - Update Startup.cs logging to use DEFAULT_PATH constant - Update all tests to remove path references

…e keys

…onal

Copilot

Pull request overview

This PR extends the embeddings feature set (“phase2”) by adding embeddings-specific cache configuration (L1 in-memory + optional L2 Redis), wiring a dedicated FusionCache instance for embeddings in the Service startup path, and updating config serialization/schema and tests to cover the new options and related behaviors.

Changes:

Add runtime.embeddings.cache configuration surface (object model + JSON converters + JSON schema).
Configure a named FusionCache instance (EmbeddingsCache) and optionally attach Redis distributed cache/backplane for embeddings.
Add/adjust unit tests and update configuration snapshots to reflect the new runtime config shape.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/Service/Startup.cs	Adds embeddings cache setup and overrides embedding service registration to use a named FusionCache (and optional Redis).
src/Service/HealthCheck/HealthCheckHelper.cs	Adds fields intended to track incoming role/token for health checks.
src/Service/Controllers/EmbeddingController.cs	Minor formatting change (blank line).
src/Service.Tests/UnitTests/StartupRedisConnectionTests.cs	New unit tests for Redis Entra-auth detection logic in `Startup`.
src/Service.Tests/UnitTests/EmbeddingsCacheOptionsTests.cs	New unit tests for embeddings cache option defaults/behavior.
src/Service.Tests/UnitTests/EmbeddingsCacheOptionsSerializationTests.cs	New unit tests validating (de)serialization of embeddings cache options with runtime config serializer options.
src/Service.Tests/UnitTests/EmbeddingsCacheLevel2OptionsTests.cs	New unit tests for embeddings L2 cache option record/config behavior.
src/Service.Tests/UnitTests/EmbeddingControllerTests.cs	Extends controller tests around service availability; currently introduces duplicate local declarations.
src/Service.Tests/UnitTests/ConfigValidationUnitTests.cs	Adds additional embeddings validation tests; currently duplicates existing test methods in the same class.
src/Service.Tests/Snapshots/ConfigurationTests.TestReadingRuntimeConfigForPostgreSql.verified.txt	Updates expected runtime-config snapshot output to match new serialization/config shape.
src/Service.Tests/Snapshots/ConfigurationTests.TestReadingRuntimeConfigForMySql.verified.txt	Updates expected runtime-config snapshot output to match new serialization/config shape.
src/Service.Tests/Snapshots/ConfigurationTests.TestReadingRuntimeConfigForCosmos.verified.txt	Updates expected runtime-config snapshot output to match new serialization/config shape.
src/Service.Tests/Configuration/ConfigurationTests.cs	Fixes JSON request body formatting in an existing strict-mode test.
src/Core/Services/Embeddings/EmbeddingTelemetryHelper.cs	Adds OpenTelemetry namespace import for tracing-related extensions.
src/Config/RuntimeConfigLoader.cs	Registers the new `EmbeddingsCacheOptionsConverterFactory` in serializer options.
src/Config/ObjectModel/Embeddings/EmbeddingsOptions.cs	Adds `cache` property to embeddings runtime options and helper booleans for cache enablement.
src/Config/ObjectModel/Embeddings/EmbeddingsCacheOptions.cs	Introduces embeddings cache options model (enabled/ttl/level2 + “user provided TTL” semantics).
src/Config/ObjectModel/Embeddings/EmbeddingsCacheLevel2Options.cs	Introduces embeddings L2 cache options model (Redis connection string).
src/Config/Converters/EmbeddingsOptionsConverterFactory.cs	Extends embeddings options converter to read/write the new `cache` section.
src/Config/Converters/EmbeddingsCacheOptionsConverterFactory.cs	New custom converter to control embeddings cache options serialization (avoid writing default TTL unless explicitly provided).
src/Cli/Commands/ConfigureOptions.cs	Adds CLI flags for embeddings runtime configuration; currently contains duplicated option/property blocks.
schemas/dab.draft.schema.json	Adds schema definition for `runtime.embeddings.cache` (including L2 settings).

Comments suppressed due to low confidence (1)

src/Cli/Commands/ConfigureOptions.cs:474

Duplicate option properties were introduced for embeddings (e.g., RuntimeEmbeddingsEnabled/Provider/BaseUrl/etc. are declared twice). This will not compile due to duplicate member definitions and duplicate [Option] attributes; remove the duplicated block and keep a single set of embeddings options (including endpoint.path/chunking options as needed).

        [Option("runtime.embeddings.enabled", Required = false, HelpText = "Enable/disable the embedding service. Default: true")]
        public CliBool? RuntimeEmbeddingsEnabled { get; }

        [Option("runtime.embeddings.provider", Required = false, HelpText = "Configure embedding provider type. Allowed values: azure-openai, openai.")]
        public EmbeddingProviderType? RuntimeEmbeddingsProvider { get; }

Co-authored-by: Copilot <copilot@github.com>

ajtiwari07 · 2026-05-12T19:12:12Z

/azp run

azure-pipelines · 2026-05-12T19:12:40Z

Azure Pipelines successfully started running 6 pipeline(s).

ajtiwari07 · 2026-05-12T19:25:47Z

/azp run

azure-pipelines · 2026-05-12T19:26:12Z

Azure Pipelines successfully started running 6 pipeline(s).

ajtiwari07 · 2026-05-13T05:32:56Z

/azp run

azure-pipelines · 2026-05-13T05:33:24Z

Azure Pipelines successfully started running 6 pipeline(s).

souvikghosh04

Posting some comments which needs to be fixed. Will have another review after that.

souvikghosh04 · 2026-05-13T11:38:17Z

+                    services.AddSingleton<IEmbeddingService>(serviceProvider =>
+                    {
+                        IFusionCache cache = embeddingsOptions.IsCachingEnabled
+                            ? serviceProvider.GetRequiredService<IFusionCacheProvider>().GetCache("EmbeddingsCache")
+                            : serviceProvider.GetRequiredService<IFusionCache>(); // Fallback to default cache
+
+                        ILogger<EmbeddingService> logger = serviceProvider.GetRequiredService<ILogger<EmbeddingService>>();
+                        IHttpClientFactory httpClientFactory = serviceProvider.GetRequiredService<IHttpClientFactory>();
+                        HttpClient httpClient = httpClientFactory.CreateClient(typeof(EmbeddingService).Name);
+
+                        return new EmbeddingService(httpClient, embeddingsOptions, logger, cache);
+                    });


is this a replacement for the existing AddHttpClient<IEmbeddingService> added above previously? please check and validate

Yes, also added comments around it.

currently at line 464, you have services.AddHttpClient(nameof(EmbeddingService));. you can may be change to services.AddHttpClient(nameof(EmbeddingService)); as you have the same below

souvikghosh04 · 2026-05-13T11:43:57Z

+                        IFusionCache cache = embeddingsOptions.IsCachingEnabled
+                            ? serviceProvider.GetRequiredService<IFusionCacheProvider>().GetCache("EmbeddingsCache")
+                            : serviceProvider.GetRequiredService<IFusionCache>(); // Fallback to default cache


is this behaviour correct? currently its falling back to IFusionCache if caching is disabled. shouldn't it completely disable caching if that's what the IsCachingEnabled means?

No, we default to L1 cache always unless L2 is configured.

I understand but the naming IsCachingEnabled is confusing. it reads as enabled or disabled but doesn't tell anything about falling back to L1. may be use meaningful naming?

Co-authored-by: Copilot <copilot@github.com>

souvikghosh04

added additional comments. mainly about Redis connection failure -> fallback, retry, loggings. additional tests.

souvikghosh04 · 2026-05-18T13:22:58Z

+                // Create connection multiplexer for embeddings Azure Managed Redis cache
+                Task<IConnectionMultiplexer> connectionMultiplexerTask =
+                    CreateConnectionMultiplexerAsync(level2Options.ConnectionString);


if Redis connection fails here where is the fallback? also, can we know why connection failed? do you have tests covering this? some failures could be transient so may be retry can succeed? can you refactor and address these gaps?

souvikghosh04 · 2026-05-18T13:26:28Z

+        if (TtlHours is not null)
+        {
+            this.TtlHours = TtlHours;
+            UserProvidedTtlHours = true;
+        }


you need additional validations like negative or zero ttl in Runtime. although JSON schema enforces it but runtime properties should have validations/checks.

souvikghosh04 · 2026-05-18T13:28:39Z

+    [NonAction]
    [Consumes("text/plain", "application/json")]
    [Produces("application/json", "text/plain")]


if its marked NonAction then what is the use of Consumes and Produces?

souvikghosh04 · 2026-05-18T13:41:24Z

+                    services.AddSingleton<IEmbeddingService>(serviceProvider =>
+                    {
+                        IFusionCache cache = embeddingsOptions.IsCachingEnabled
+                            ? serviceProvider.GetRequiredService<IFusionCacheProvider>().GetCache("EmbeddingsCache")
+                            : serviceProvider.GetRequiredService<IFusionCache>(); // Fallback to default cache
+
+                        ILogger<EmbeddingService> logger = serviceProvider.GetRequiredService<ILogger<EmbeddingService>>();
+                        IHttpClientFactory httpClientFactory = serviceProvider.GetRequiredService<IHttpClientFactory>();
+                        HttpClient httpClient = httpClientFactory.CreateClient(typeof(EmbeddingService).Name);
+
+                        return new EmbeddingService(httpClient, embeddingsOptions, logger, cache);
+                    });


currently at line 464, you have services.AddHttpClient(nameof(EmbeddingService));. you can may be change to services.AddHttpClient(nameof(EmbeddingService)); as you have the same below

Copilot AI and others added 30 commits February 3, 2026 21:06

Initial plan

0dc589e

Add EmbeddingsOptions and EmbeddingProviderType configuration models

d3187bd

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Add CLI configure options for embeddings and register embedding service

6064826

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Add unit tests for embeddings and update JSON schema with embeddings …

0653f15

…configuration Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Simplify HttpClient registration for embedding service

21e81b9

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Plan for embedding service enhancements

0cd8e53

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Refactor embedding code into dedicated namespaces

7fa1c49

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Fix property renames and update tests

c3f6937

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Add EmbeddingsOptionsConverter and fix all tests

1e18c25

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Address code review feedback

857203a

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Address PR feedback: Add Azure OpenAI validation, cache key security,…

89cb2d9

… and telemetry integration Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Optimize ProviderName to avoid repeated string allocations

64b592e

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Fix schema mismatch, remove unused field, add enabled handling, valid…

e8d7238

…ate empty embeddings Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Add embedding health check execution and update JSON schema with endp…

d9c8a29

…oint/health sub-objects Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Add EmbeddingController for /embed REST endpoint with role-based auth…

3e02c0f

…orization Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

Address code review feedback for EmbeddingController

5c05464

Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com>

fix: manually deserialize EmbeddingsEndpointOptions and EmbeddingsHea…

c9eba20

…lthCheckConfig in converter

feat: add embeddings config validation and unit tests

d3a5209

Update src/Core/Services/Embeddings/EmbeddingService.cs

203e232

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Config/Converters/EmbeddingsOptionsConverterFactory.cs

2cb3999

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Service/Startup.cs

b432e4f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Taking care of feedback

a8442f4

Merge branch 'main' into add-internal-text-embedding-system

a301290

Taking care of copilot feedback, adding embedding service tests, embe…

5f12542

…dding controller tests, switching the dab schema for the embedding system to default to false

Adding default return of application/json and only text/plain if expl…

bd84c84

…icitly set

feat(embeddings): require L2 cache and include provider/model in cach…

7100b8a

…e keys

fix(embeddings): keep L1 allow distributed cache when configured/opti…

73658de

…onal

Adding max text-count validation for embedding requests

ece97ae

Phase 1 Added Embedding Support with Chunking

d065f4b

Copilot started reviewing on behalf of ajtiwari07 May 11, 2026 20:01 View session

Copilot AI reviewed May 11, 2026

View reviewed changes

ajtiwari07 and others added 2 commits May 11, 2026 13:58

Remove merge side effects

63f6cac

Co-authored-by: Copilot <copilot@github.com>

Revert merge side effecgs

b49a090

ajtiwari07 changed the title ~~Embedding phase2~~ Redis backed Caching of Internal Embeddings May 11, 2026

ajtiwari07 and others added 3 commits May 11, 2026 14:31

Formatting Fixes

7eda610

Fix failing UTs

e46f5e7

Entra Auth Support

2c51096

Co-authored-by: Copilot <copilot@github.com>

ajtiwari07 requested review from JimRoberts-MS, prshri-msft and sayalikudale May 12, 2026 17:36

ajtiwari07 and others added 2 commits May 12, 2026 23:07

Merge branch 'main' into embedding-phase2

37e376f

Fix Embed API Routing

405d3cd

Co-authored-by: Copilot <copilot@github.com>

souvikghosh04 requested changes May 13, 2026

View reviewed changes

Post review commit

818253c

JimRoberts-MS reviewed May 13, 2026

View reviewed changes

Comment thread src/Service.Tests/UnitTests/EmbeddingControllerTests.cs

Update Controller UTs

5bb067f

ajtiwari07 requested a review from souvikghosh04 May 13, 2026 16:44

JimRoberts-MS reviewed May 14, 2026

View reviewed changes

Comment thread src/Service.Tests/UnitTests/EmbeddingControllerTests.cs

More asserts

937af67

Co-authored-by: Copilot <copilot@github.com>

JimRoberts-MS approved these changes May 15, 2026

View reviewed changes

Merge branch 'main' into embedding-phase2

5891c84

prshri-msft approved these changes May 15, 2026

View reviewed changes

souvikghosh04 requested changes May 18, 2026

View reviewed changes

Conversation

ajtiwari07 commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Embeddings-Specific Cache Configuration

2. Level 2 (L2) Distributed Redis Cache Support

3. Enhanced Configuration Schema

4. Dedicated FusionCache Instance for Embeddings

5. Comprehensive Test Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajtiwari07 commented May 12, 2026

Uh oh!

azure-pipelines Bot commented May 12, 2026

Uh oh!

ajtiwari07 commented May 12, 2026

Uh oh!

azure-pipelines Bot commented May 12, 2026

Uh oh!

ajtiwari07 commented May 13, 2026

Uh oh!

azure-pipelines Bot commented May 13, 2026

Uh oh!

souvikghosh04 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

souvikghosh04 May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

souvikghosh04 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ajtiwari07 commented May 11, 2026 •

edited

Loading

souvikghosh04 May 18, 2026 •

edited

Loading