Nullable object must have a value thrown for a query with DefaultIfEmpty()

Hi,
I am getting error "Nullable object must have a value" using the following scenario:

var categories = from c in dbContext.Set<AMOS_ADDRESSCATEGORY>().AsNoTracking()
                                 select new
                                 {
                                     Id = c.ADDRESSCATEGORYID,
                                     Name = c.DESCRIPTION
                                 };

var addresses = (from address in dbContext.Set<AMOS_ADDRESS>().AsNoTracking()
                                 join c in categories on address.ADDRESSCATEGORYID equals c.Id into aC
                                 from category in aC.DefaultIfEmpty()
                                 select new
                                 {
                                     Id = address.ADDRESSID,
                                     Category = category
                                 }).ToArray();

The above scenarion works pefectly using EF Standard and works in EF Core if I change to:

var addresses = (from address in dbContext.Set<AMOS_ADDRESS>().AsNoTracking()
                                 **join c in dbContext.Set<AMOS_ADDRESSCATEGORY>().AsNoTracking()** 
                                on address.ADDRESSCATEGORYID equals c.ADDRESSCATEGORYID into aC
                                 from category in aC.DefaultIfEmpty()
                                 select new
                                 {
                                     Id = address.ADDRESSID,
                                     Category = category
                                 }).ToArray();

Can you help me please?
Thanks in advance
Luigi

EF Core version: 8.0.0-preview.4.23259.3
Database provider: Microsoft.EntityFrameworkCore.SqlServer
Target framework: NET 6.0
Operating system:
IDE: Visual Studio 2022 17.4

This issue is lacking enough information for us to be able to fully understand what is happening. Please attach a small, runnable project or post a small, runnable code listing that reproduces what you are seeing so that we can investigate.

@ajcvickers

Hi @ajcvickers
Please find attached a project for reproducing issue.

ReproducibleNullObject.zip

Note for triage: we don't handle assigning null to the anonymous type after calling DefaultIfEmpty, but it seems like this should work, and it works in LINQ-to-Objects.

@ajcvickers

Hi @ajcvickers
I believe it has same issue with non anonymous type.
Example:

var categories = from c in dbContext.Set<AMOS_ADDRESSCATEGORY>().AsNoTracking()
                         select new Category
                         {
                             Id = c.ADDRESSCATEGORYID,
                             Name = c.DESCRIPTION
                         };

 var addresses = (from address in dbContext.Set<AMOS_ADDRESS>().AsNoTracking()
                                 join c in categories
                                on address.ADDRESSCATEGORYID equals c.Id into aC
                                 from category in aC.DefaultIfEmpty()
                                 select new
                                 {
                                     Id = address.ADDRESSID,
                                     Category = category
                                 }).ToArray();

Hi, Has been solved in version 8 ?
I tested and seems still present..
Thanks in advance

This issue is in the Backlog milestone. This means that it is not fixed in EF Core 8.0. We will re-assess the backlog and consider this item in the upcoming planning. However, keep in mind that there are many other high priority features with which it will be competing for resources. Make sure to vote (👍) for this issue if it is important to you.

Hi, thanks for prompt reply. I believe that this issue should be with high priority because fails on a simple left join operation.
Anyway thanks for suggestion

I have the same exception using a simple group by statement, I think the root cause might be the same. You can run this in roslyn pad to reproduce.

#r "nuget: Microsoft.EntityFrameworkCore.Sqlite,8.0.0"
#r "nuget: Microsoft.EntityFrameworkCore,8.0.0"

using System.ComponentModel.DataAnnotations;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging;

await using var ctx = new Context();
await ctx.Database.EnsureDeletedAsync();
await ctx.Database.EnsureCreatedAsync();

ctx.Steps.Add(new Step
{
    Id = 5
});
await ctx.SaveChangesAsync();

await ctx.Steps
    .GroupBy(e => new
    {
        e.Planned
    }).ToListAsync();

public class Context : DbContext
{
    public DbSet<Step> Steps { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        => optionsBuilder
            .UseSqlite(Environment.ExpandEnvironmentVariables("DataSource=%USERPROFILE%/Desktop/test_db.sqlite"))
            .LogTo(Console.WriteLine, LogLevel.Information)
            .EnableSensitiveDataLogging();
}

public class Step
{
    [Key]
    public int Id { get; set; }
    public DateTime? Planned { get; set; }
}

Address review: guard non-grouping subquery clone, encapsulate marker…

Implement Seed30915 required by EF Core preview 7 specification tests

Upgrade to EF Core 11 preview 7 and sync tests with upstream

Upgrade to .NET 11 and EF Core 11 (preview 7), and xunit v3 (#294)

Status update on the remaining-work checklist above. Since that comment:

Second join after the left join: Make the #30915 nullability marker survive a subsequent join #38499
GroupBy after the left join (to-one lowering): Make the nullability marker survive GroupBy-element to-one subquery lowering #38577
Value-type struct / record struct (member-init): Default a value-type non-entity left-join projection to default(T) on no-match #38555 (this also resolves the Nullable<T> item)
ValueTuple / Tuple whole-object projection: Fold ValueTuple/Tuple member access through constructor-bound projections #38560
Union / set operations: [release/11.0] Translate set operations over null-checked subqueries #38847
RightJoin with the nullable outer: Materialize a right/full-joined outer non-entity projection as null on no-match (#30915) #39556 (open). Still throws when the outer needs no pushdown, is a plain Select with only Where/Take, or is itself a join result.
Plain inner with no pushdown: tracked as Query: whole non-entity object from a plain (non-aggregated) LeftJoin/DefaultIfEmpty inner is not nulled on no-match #38608, which needs the navigation-expansion rework (Consider removing nav expansion as a way to tackle the pending selector problem #32957 / Rearchitect navigation expansion to happen in translation, not preprocessing #37859).
Server-side obj == null / obj != null in Where / OrderBy: a complete Result of left joining a nullable nominal type should be null (i.e. no instance) and not an instance with all null property values #22517 would probably enable this.
~~InMemory parity: not started (EFCore.InMemory: "System.InvalidOperationException : Nullable object must have a value." when doing second left join via object #23425 was closed as not planned).~~

Constructor-bound DTO / positional record struct: this is not a null-handling problem and doesn't belong on this checklist. Member access on any constructor-bound (non-anonymous, non-ValueTuple/Tuple) projection fails to translate, even without a join. For example, Where(c => c.Count > 1) over a GroupBy projected into a DTO with a constructor, or an inner Join keyed on a DTO member. That is #28248. The fold in ReplacingExpressionVisitor was deliberately limited to ValueTuple/Tuple in #38560, because an ordinary type can redeclare a same-named property with a transforming initializer and silently produce the wrong value. Widening it needs a sound way to map constructor parameters to members, which is a separate design question. I've left that item out of the #30915 work.

Final status for this issue. I re-ran the scenarios from this thread against current main (SQLite; the failure is in client-side materialization, so it is provider independent).

Reported scenarios

Scenario	Result on `main`
Original report: `join c in <Select into anonymous type> ... DefaultIfEmpty()`, projecting `Category = category`	Still throws. Plain inner with no pushdown: the marker column is inlined as the constant `1`, so it never becomes `NULL`. Tracked as #38608.
Same with a `new Category { ... }` DTO (comment from 2023-05-19)	Still throws, same cause. Reading `category.Name` instead works.
Joining to the entity set directly (the workaround in the report)	Works.
Non-nullable `int` member of a null left-joined entity (`c.DetailCount`)	Throws. I believe this is expected: LINQ-to-Objects would throw a `NullReferenceException` here. Adding a null check works.
`GroupBy(new { e.Planned })` over a nullable column (comment from 2023-12-20)	Works; it does not reproduce on `main`.

Fixed (whole non-entity object now materializes as null on a no-match row, when the inner needs a subquery): second join after the left join #38499, GroupBy after the left join #38577, struct / record struct and Nullable<T> #38555, ValueTuple / Tuple #38560, Union / set operations #38847.

In review: RightJoin / FullJoin where the outer side is the nullable one: #39556. It covers outers that are a GroupBy, Distinct, Take or MemberInit DTO, plus FullJoin (both sides). Still throws when the outer is a plain Select that needs no pushdown (same cause as #38608), a plain Select with only Where/Take, or is itself a join result. Those are pinned by characterization tests marked #30915 TODO.

Remaining, and why I think they should not block closing this issue

Plain inner / plain outer with no pushdown: blocked on the navigation-expansion rework (Query: whole non-entity object from a plain (non-aggregated) LeftJoin/DefaultIfEmpty inner is not nulled on no-match #38608, Consider removing nav expansion as a way to tackle the pending selector problem #32957, Rearchitect navigation expansion to happen in translation, not preprocessing #37859). This is the shape in the original report.
Anonymous type holding a whole entity on the nullable side (e.g. Select(r => new { r, r.CategoryId })), for both LeftJoin and RightJoin: still throws. Navigation expansion rewrites the user's reference to the object into a fresh member-wise copy (new { r = ti.Inner.r, CategoryId = ti.Inner.CategoryId }), which no longer matches the node the marker was recorded against, so the null gate never fires. A scalar-only anonymous type is kept as a plain reference and works. Same family as Query: whole non-entity object from a plain (non-aggregated) LeftJoin/DefaultIfEmpty inner is not nulled on no-match #38608. Pinned by tests (branch test/30915-entity-in-anonymous-nullable-side, not yet a PR).
Server-side obj == null / obj != null in Where / OrderBy: translation failure. A complete Result of left joining a nullable nominal type should be null (i.e. no instance) and not an instance with all null property values #22517 would likely enable it.
Distinct over a join result, then OrderBy on a member of the nullable side: fails to translate for LeftJoin on main already, and now also for RightJoin / FullJoin (previously it failed at materialization). Not specific to this issue.
GroupBy followed by a client-evaluated Select: throws for both LeftJoin and RightJoin; works with no join. Predates this work.
Constructor-bound DTO / positional record struct: not a null-handling problem. Member access on any constructor-bound projection fails to translate even without a join. That is GroupJoin throws when applied on an anonymous return type output of another Linq #28248.
InMemory: not planned (EFCore.InMemory: "System.InvalidOperationException : Nullable object must have a value." when doing second left join via object #23425 was closed as not planned, and InMemory is not updated in this way).

Suggested disposition: the reported scenario cannot be fixed until #38608 is resolved, so I would keep this issue open and mark it blocked on #38608. The remaining items above are either blocked on the same rework, tracked in their own issues, or pre-existing limitations, so I do not think any of them need a separate fix before then.

Follow-up to my status comment above, with the result of triaging the remaining RightJoin shapes and the server-side null checks.

The remaining RightJoin shapes are all the same problem as #38608. I traced each one:

Plain Select outer with only Where / Take, and plain outer with no pushdown: navigation expansion keeps the Select as a pending selector, so at the join the outer shaper is the bare entity and no marker is injected. The anonymous object is then rebuilt after the join as a fresh new { PickupStatusId = ti.Outer.PickupStatusId, Count = 1 }. The Where / Take is not the cause.
Outer that is itself a join result: the marker is injected and recorded, but navigation expansion then rebuilds the first join's result after the second join as a member-wise copy (new { ... = ti0.Outer.Inner..., countInfo = ti0.Outer.Outer }), which no longer matches the node the marker was recorded against. With no pushdown the marker would also be inlined as a constant.
Anonymous type holding a whole entity on the nullable side (LeftJoin and RightJoin): same rewrite, as described in my earlier comment.

In each case the user's reference to the nullable-side object is replaced by a fresh copy before translation, so the null gate (keyed on the original node) never fires. Fixing these needs the navigation-expansion rework (#38608 / #32957 / #37859), not more marker plumbing.

Server-side null checks. Where(x == null), Where(x != null) and OrderBy(x == null) against a whole non-entity object from the nullable side fail to translate with a clean "could not be translated" error (no wrong results); the same for a RightJoin outer. Select(x == null), Select(new { Id, IsNull = x == null }) and Where(x.N > 0) work. All of this is pinned by tests. A narrow fix may exist (translate x == null to marker IS NULL), but it would be a new feature rather than a bug fix, so I have not attempted it; a complete #22517 would likely cover it.

The pinning tests are on the fork: ajcvickers/efcore branches fix/30915-rightjoin-triage and test/30915-entity-in-anonymous-nullable-side, both stacked on #39556. I will open PRs for them once #39556 merges.

@AndriySvyryd

@AndriySvyryd There is a current PR and one more PR for this, then the issue should be marked blocked on #38608.

Navigation Menu

Repository navigation

Nullable object must have a value thrown for a query with DefaultIfEmpty() #30915

55 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Nullable object must have a value thrown for a query with DefaultIfEmpty() #30915

Description

Activity

ajcvickers commented on May 17, 2023

Luigi6821 commented on May 17, 2023

ajcvickers commented on May 18, 2023

Luigi6821 commented on May 19, 2023

Luigi6821 commented on Nov 15, 2023

ajcvickers commented on Nov 15, 2023

Luigi6821 commented on Nov 15, 2023

UliPlabst commented on Dec 20, 2023

Luigi6821 commented on Feb 14, 2024

roji commented on Feb 14, 2024

Luigi6821 commented on Feb 15, 2024

55 remaining items

ajcvickers commented on Nov 6, 2026

ajcvickers commented on Nov 7, 2026

ajcvickers commented on Nov 7, 2026

ajcvickers commented on Nov 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions