Shadows on forward renderer

[ada-tv]

shadows and msaa 🎉

Deferred	Forward

---

unfortunately #2102 also affects the shadows setting, but that's unrelated to this pr

---

• Shadow pass running before doing forward (and passing the shadow frame to the deferred transparent pass)
• Shadows rendering (currently reuses the fade params buffer slot)
• Invalid pipeline pointer crash
• The rest of the shaders (simple, voxel, etc, model is the prototyping shader since it's the most commonly used one)
• Figure out what to do with the forward-only ifdef for Android in RenderViewTask

  for now it'll still do the shadow pass so it compiles, idk how i'd dummy it out to never run

• Update the settings QML to make the shadow toggle visible on both deferred and forward
• Fix broken resolution scale setting on forward
• Fix mirrors using deferred instead of forward
• Figure out why opaque forward simple doesn't receive shadows but transparent does (maybe something with instancing?)

[ada-tv]

weird crash that only sometimes happens when shadows are enabled

Thread 55 "RenderThread" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffeb67fc6c0 (LWP 123354)]
gpu::Batch::setPipeline (this=0x7ffea07ace70, 
    pipeline=<error reading variable: Cannot access memory at address 0x8>)
    at /home/ada/src/overte/libraries/gpu/src/gpu/Batch.cpp:400
400	    _currentPipeline = pipeline;
(gdb) bt
#0  gpu::Batch::setPipeline
    (this=0x7ffea07ace70, pipeline=<error reading variable: Cannot access memory at address 0x8>) at /home/ada/src/overte/libraries/gpu/src/gpu/Batch.cpp:400
#1  0x00007ffff58a0412 in render::ShapePlumber::pickPipeline
    (this=<optimized out>, args=0x7ffeb67fb090, key=<optimized out>)
    at /home/ada/src/overte/libraries/render/src/render/ShapePipeline.cpp:166
#2  0x00007ffff58a06c1 in render::ShapePlumber::pickPipeline
    (this=0x5555797d3080, args=args@entry=0x7ffeb67fb090, key=...)
    at /home/ada/src/overte/libraries/render/src/render/ShapePipeline.cpp:153
#3  0x00007ffff586ae9d in renderShape
    (args=args@entry=0x7ffeb67fb090, shapeContext=std::shared_ptr<render::ShapePlumber> (use count 4145, weak count 0) = {...}, item=..., globalKey=...)
    at /usr/include/c++/16/bits/shared_ptr_base.h:1751
#4  0x00007ffff586b1cc in render::renderShapes
    (renderContext=std::shared_ptr<render::RenderContext> (use count 1, weak count 0) = {...}, shapeContext=std::shared_ptr<render::ShapePlumber> (use count 4145, weak count 0) = {...}, inItems=std::vector of length 1, capacity 1 = {...}, maxDrawnItems=maxDrawnItems@entry=-1, globalKey=...)
    at /home/ada/src/overte/libraries/render/src/render/DrawTask.cpp:83
#5  0x00007ffff61a240c in sortAndRenderZPassShapes
    (shapePlumber=<optimized out>, renderContext=<optimized out>, inShapes=std::unordered_map with 7 elements = {...}, itemBounds=std::vector of length 4, capacity 4 = {...})
    at /home/ada/src/overte/build/libraries/render-utils/src/RenderPipelinesInit.cpp:2037
#6  0x00007ffff613e46c in operator() (__closure=0x7ffea10f6170, batch=...)
    at /home/ada/src/overte/libraries/render-utils/src/RenderShadowTask.cpp:255
#7  std::__invoke_impl<void, RenderShadowMap::run(const render::RenderContextPointer&, const Inputs&)::<lambda(gpu::Batch&)>&, gpu::Batch&> (__f=...)
    at /usr/include/c++/16/bits/invoke.h:63
#8  std::__invoke_r<void, RenderShadowMap::run(const render::RenderContextPointer&, const Inputs&)::<lambda(gpu::Batch&)>&, gpu::Batch&> (__fn=...)
    at /usr/include/c++/16/bits/invoke.h:113
#9  std::_Function_handler<void(gpu::Batch&), RenderShadowMap::run(const render::RenderContextPointer&, const Inputs&)::<lambda(gpu::Batch&)> >::_M_invoke(const std::_Any_data &, gpu::Batch &) (__functor=<optimized out>, __args#0=...)
    at /usr/include/c++/16/bits/std_function.h:295
#10 0x00007ffff475e81a in std::function<void(gpu::Batch&)>::operator()
    (this=0x7ffeb67fa1b0, __args#0=<optimized out>)
    at /usr/include/c++/16/bits/std_function.h:581
#11 gpu::doInBatch
    (name=name@entry=0x7ffff61aaf60 "RenderShadowMap::run", context=std::shared_ptr<gpu::Context> (use count 7, weak count 0) = {...}, f=...)
    at /home/ada/src/overte/libraries/gpu/src/gpu/Context.cpp:367
#12 0x00007ffff6140cf2 in RenderShadowMap::run
    (this=0x555583c52860, renderContext=std::shared_ptr<render::RenderContext> (use count 1, weak count 0) = {...}, inputs=<optimized out>)
    at /home/ada/src/overte/libraries/render-utils/src/RenderShadowTask.cpp:234

[JulianGro]

On OpenGL this looks good. On Vulkan, my GPU crashes when running this PR. Master works fine on Vulkan. I tried with and without shadows and forward and deferred.
While the GPU shouldn't crash, I feel like this probably still indicates a potential bug.

[  740.169934] amdgpu 0000:06:00.0: GPU fault detected: 147 0x00004402
[  740.169943] amdgpu 0000:06:00.0:  Process AppRun.wrapped pid 5989 thread PresentThread pid 6052
[  740.169947] amdgpu 0000:06:00.0:   VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x00080000
[  740.169950] amdgpu 0000:06:00.0:   VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0E044002
[  740.169952] amdgpu 0000:06:00.0: VM fault (0x02, vmid 7, pasid 61) at page 524288, read from 'TC5' (0x54433500) (68)
[  740.186490] amdgpu 0000:06:00.0: GPU fault detected: 147 0x00004402
[  740.186498] amdgpu 0000:06:00.0:  Process AppRun.wrapped pid 5989 thread PresentThread pid 6052
[  740.186501] amdgpu 0000:06:00.0:   VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x00080000
[  740.186504] amdgpu 0000:06:00.0:   VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0E044002
[  740.186507] amdgpu 0000:06:00.0: VM fault (0x02, vmid 7, pasid 61) at page 524288, read from 'TC5' (0x54433500) (68)
[  740.202569] amdgpu 0000:06:00.0: GPU fault detected: 147 0x00004402
[  740.202575] amdgpu 0000:06:00.0:  Process AppRun.wrapped pid 5989 thread PresentThread pid 6052
[  740.202579] amdgpu 0000:06:00.0:   VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x00080000
[  740.202582] amdgpu 0000:06:00.0:   VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0A044002
[  740.202584] amdgpu 0000:06:00.0: VM fault (0x02, vmid 5, pasid 61) at page 524288, read from 'TC5' (0x54433500) (68)
[  742.266401] amdgpu 0000:06:00.0: Dumping IP State
[  742.266409] amdgpu 0000:06:00.0: Dumping IP State Completed
[  742.266469] amdgpu 0000:06:00.0: [drm] AMDGPU device coredump file has been created
[  742.266472] amdgpu 0000:06:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
[  742.276479] amdgpu 0000:06:00.0: ring gfx timeout, signaled seq=120477, emitted seq=120478
[  742.276485] amdgpu 0000:06:00.0:  Process AppRun.wrapped pid 5989 thread PresentThread pid 6052
[  742.276489] amdgpu 0000:06:00.0: GPU reset begin!. Source:  1
[  746.278083] amdgpu 0000:06:00.0: failed to suspend display audio
[  746.658020] amdgpu 0000:06:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_0.2.1.0 test failed (-110)
[  746.658247] [drm:gfx_v8_0_hw_fini [amdgpu]] *ERROR* KCQ disable failed
[  746.938108] amdgpu: cp is busy, skip halt cp
[  747.247604] amdgpu: rlc is busy, skip halt rlc
[  747.248631] amdgpu 0000:06:00.0: BACO reset
[  747.426371] amdgpu 0000:06:00.0: GPU reset succeeded, trying to resume
[  747.429180] amdgpu 0000:06:00.0: [drm] PCIE GART of 256M enabled (table at 0x000000F400800000).
[  747.429202] amdgpu 0000:06:00.0: VRAM is lost due to GPU reset!
[  747.551763] amdgpu 0000:06:00.0: [drm] UVD and UVD ENC initialized successfully.
[  747.652831] amdgpu 0000:06:00.0: [drm] VCE initialized successfully.
[  747.699061] amdgpu 0000:06:00.0: GPU reset(1) succeeded!
[  747.699110] amdgpu 0000:06:00.0: [drm] device wedged, but recovered through reset
[  748.257741] snd_hda_intel 0000:06:00.1: azx_get_response timeout, switching to polling mode: last cmd=0x00672400
[  748.257749] snd_hda_intel 0000:06:00.1: spurious response 0x0:0x0, last cmd=0x672400
[  748.257754] snd_hda_intel 0000:06:00.1: spurious response 0x0:0x0, last cmd=0x672400
[  748.257757] snd_hda_intel 0000:06:00.1: spurious response 0x0:0x0, last cmd=0x672400
[  748.257760] snd_hda_intel 0000:06:00.1: spurious response 0x0:0x0, last cmd=0x672400
[  748.257763] snd_hda_intel 0000:06:00.1: spurious response 0x0:0x0, last cmd=0x672400
[  748.257766] snd_hda_intel 0000:06:00.1: spurious response 0x0:0x0, last cmd=0x672400
[  748.257768] snd_hda_intel 0000:06:00.1: spurious response 0x0:0x0, last cmd=0x672400
[  748.257771] snd_hda_intel 0000:06:00.1: spurious response 0x0:0x0, last cmd=0x672400
[  748.257774] snd_hda_intel 0000:06:00.1: spurious response 0x0:0x0, last cmd=0x672400
[  748.257777] snd_hda_intel 0000:06:00.1: spurious response 0x0:0x0, last cmd=0x672400
[  748.670992] ------------[ cut here ]------------
[  748.670998] atomic remove_fb failed with -22
[  748.671000] WARNING: drivers/gpu/drm/drm_framebuffer.c:1176 at drm_framebuffer_remove+0x21c/0x230, CPU#9: kworker/9:2/1065
[  748.671008] Modules linked in: xt_conntrack xt_MASQUERADE bridge stp llc xfrm_user xfrm_algo xt_set ip_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype nft_compat nf_tables vxlan ip6_udp_tunnel udp_tunnel ccm rfcomm snd_seq_dummy snd_hrtimer overlay qrtr cmac algif_hash algif_skcipher af_alg bnep binfmt_misc amdgpu snd_hda_codec_alc882 snd_hda_codec_realtek_lib snd_hda_codec_generic snd_hda_codec_atihdmi snd_hda_codec_hdmi snd_hda_intel rtw89_8851be snd_hda_codec rtw89_8851b snd_hda_core rtw89_pci snd_intel_dspcfg snd_intel_sdw_acpi amdxcp drm_panel_backlight_quirks snd_hwdep gpu_sched drm_buddy rtw89_core snd_pcm drm_ttm_helper ttm btusb drm_exec drm_suballoc_helper btmtk snd_seq_midi btrtl intel_rapl_msr snd_seq_midi_event drm_display_helper btbcm snd_rawmidi btintel cec mac80211 snd_seq amd_atl snd_seq_device nls_iso8859_1 intel_rapl_common snd_timer rc_core amd64_edac i2c_piix4 edac_mce_amd i2c_algo_bit cfg80211 snd rapl serio_raw gigabyte_wmi wmi_bmof k10temp i2c_smbus video ccp
[  748.671119]  soundcore bluetooth libarc4 input_leds gpio_amdpt joydev mac_hid sch_fq_codel msr parport_pc ppdev lp parport efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs libblake2b dm_crypt raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 linear dm_mirror dm_region_hash dm_log hid_logitech_hidpp nvme hid_logitech_dj uas hid_generic usbhid hid usb_storage ghash_clmulni_intel nvme_core psmouse r8169 nvme_keyring ahci realtek nvme_auth libahci hkdf wmi aesni_intel
[  748.671191] CPU: 9 UID: 0 PID: 1065 Comm: kworker/9:2 Not tainted 7.0.0-30-generic #30~24.04.1-Ubuntu PREEMPT(lazy) 
[  748.671195] Hardware name: Gigabyte Technology Co., Ltd. B550M GAMING X WIFI6/B550M GAMING X WIFI6, BIOS F2 09/04/2025
[  748.671197] Workqueue: events drm_mode_rmfb_work_fn
[  748.671201] RIP: 0010:drm_framebuffer_remove+0x21e/0x230
[  748.671205] Code: c6 48 8d 3d 54 28 e2 01 48 c7 c1 50 16 f7 af 4c 89 ea 67 48 0f b9 3a e9 26 fe ff ff 4c 8b 2f eb d8 48 8d 3d 44 28 e2 01 89 c6 <67> 48 0f b9 3a e9 4e fe ff ff 31 ff eb b8 0f 1f 40 00 90 90 90 90
[  748.671208] RSP: 0018:ffffd449059d7de8 EFLAGS: 00010282
[  748.671211] RAX: 00000000ffffffea RBX: ffff8c214fae3400 RCX: 0000000000000000
[  748.671213] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: ffffffffb0efc5a0
[  748.671215] RBP: ffffd449059d7e08 R08: 0000000000000000 R09: 0000000000000000
[  748.671217] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8c216c200010
[  748.671219] R13: ffff8c214fae3400 R14: ffffd449026e7928 R15: ffffd449026e7908
[  748.671221] FS:  0000000000000000(0000) GS:ffff8c308da63000(0000) knlGS:0000000000000000
[  748.671223] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  748.671225] CR2: 00005855a3ffa8d0 CR3: 00000001038dd000 CR4: 0000000000f50ef0
[  748.671227] PKRU: 55555554
[  748.671229] Call Trace:
[  748.671232]  <TASK>
[  748.671236]  drm_mode_rmfb_work_fn+0x7b/0xb0
[  748.671240]  process_one_work+0x1af/0x430
[  748.671246]  worker_thread+0x1bf/0x350
[  748.671249]  ? srso_alias_return_thunk+0x5/0xfbef5
[  748.671253]  ? __pfx_worker_thread+0x10/0x10
[  748.671257]  kthread+0xfb/0x140
[  748.671260]  ? __pfx_kthread+0x10/0x10
[  748.671264]  ret_from_fork+0x1f8/0x2b0
[  748.671269]  ? __pfx_kthread+0x10/0x10
[  748.671272]  ret_from_fork_asm+0x1a/0x30
[  748.671280]  </TASK>
[  748.671281] ---[ end trace 0000000000000000 ]---

I don't think I can get a better backtrace, since my session closes when the GPU gets reset.

                                                       Stack trace of thread 6000:
                                                       #0  0x00007ee86269ec0c __pthread_kill_implementation (libc.so.6 + 0x9ec0c)
                                                       #1  0x00007ee86264527e __GI_raise (libc.so.6 + 0x4527e)
                                                       #2  0x00007ee8626288ff __GI_abort (libc.so.6 + 0x288ff)
                                                       #3  0x00007ee849e64b90 n/a (libgallium-25.2.8-0ubuntu0.24.04.2.so + 0xa64b90)
                                                       #4  0x00007ee849e68e13 n/a (libgallium-25.2.8-0ubuntu0.24.04.2.so + 0xa68e13)
                                                       #5  0x00007ee849966091 n/a (libgallium-25.2.8-0ubuntu0.24.04.2.so + 0x566091)
                                                       #6  0x00007ee849999e0c n/a (libgallium-25.2.8-0ubuntu0.24.04.2.so + 0x599e0c)
                                                       #7  0x00007ee86269cb84 start_thread (libc.so.6 + 0x9cb84)
                                                       #8  0x00007ee862729d6c __clone3 (libc.so.6 + 0x129d6c)
                                                       
                                                       Stack trace of thread 6034:
                                                       #0  0x00007ee86271b5fd __GI___poll (libc.so.6 + 0x11b5fd)
                                                       #1  0x00007ee882ef76f7 n/a (/tmp/.mount_OverteLBeocc/usr/lib/libQt5Core.so.5 + 0x6f76f7)
                                                       ELF object binary architecture: AMD x86-64

Since you are also on AMD Mesa, maybe you can give it a test on Vulkan too? Your drivers (and graphics card) are probably way newer.

I wonder if the crash is related to #2076 though.

[ada-tv]

On Mesa 26.2.1 and an RX 7800 XT I don't get a crash. The shadows don't render (as expected on Vulkan atm) but my GPU doesn't outright crash.