[GMCM & SpaceCore] Stardew Access Integration

[khanshoaib3]

This PR attempts to make GMCM and SpaceCore screen reader and keyboard accessible via Stardew Access.

It does so by adding a dummy ClickableComponent to Element which is used for gamepad/keyboard navigation only, and implementing IScreenReadable to Element.

Changes

• In Element.cs:
  • Implement IScreenReadable interface.
  • Added DummyClickableComponent and CreateDummyClickableComponent properties.
  • Added two events MouseEntered and MouseHovered, because the ui components are in SpaceShared, I couldn't figure out how to get StardewAccess' API and directly speak the element here, so instead I added these events with their listener registered while initializing StardewAccess' API.
  • StardewAccess has mouse key simulation keys which are used for invoking receiveLeftClick() and so on of the active menu, because SpaceCore doesn't handle mouse clicks like that, I hardcoded the keybind for now as I couldn't figure out how to access StardewAccess' API which has the mouse sim keybindings exposed.
• In Table.cs:
  • Initialize DummyClickableComponent while assigning proper ids for navigation.
• In ModConfigMenu, SpecificModConfigMenu and KeybindsOverlay:
  • Initialize the screen reader properties and add the dummy clickable components to allClickableComponents.
  • Gamepad/Keyboard navigation is suppressed when a textbox is selected/active.
  • While a slider or dropdown is hovered and left/right direction keys are pressed, it changes the slider's/dorpdown's value.
• In ModConfigMenu:
  • Snap to last snapped element when navigating back to this menu.
  • For this I modified OpenModMenu action and added snapTo parameter to the constructor.
• NewSkillPage overrides the vanilla skill page so I modified it similar to how the vanilla page is patched for accessibility, look here.
• Disabled snappy cursor movement ban in SkillLevelUpMenu and made it accessible. Also modified the bounds of clickable components for better navigation.

Issues

• GMCM allows free cursor movement with left joystick but these changes allows vanilla gamepad navigation so free cursor movement is interfered and buggy. Maybe we can have free cursor movement with the right joystick?

These changes include some major changes (like mouse events) so I'm awaiting your input on those and also on how to access StardewAccess' API in Element for mouse click.

And I've missed some things which I'm listing below:

• I18n in NewSkillPage

This PR should be compatible with v1.7.0-beta.1 of stardew access but I've used the builds from my fork personally.

[spacechase0]

I am very interested in this - the clickablecomponent stuff is something I've meant to do for a while, so that controllers could have snappy movement. The free cursor movement was just a hack until I got that done, so that can be removed entirely (unless vanilla menus have a way to start using it even with snappy menus on?)

I somewhat want to work on this as part of GMCM 2.0 that I've been contemplating, since a lot will be changing for both GMCM and the UI library for that. On the other hand. accessibility is important enough that I should go ahead with this anyways, especially for a mod as widely used as GMCM...

So yeah, I think I do want to go ahead with merging this. Before I fully review this (since it's a larger PR than most I get), I want to address this:

These changes include some major changes (like mouse events) so I'm awaiting your input on those and also on how to access StardewAccess' API in Element for mouse click.
I took a look at this specifically, and I think the solution you made isn't feasible since it's a static event. The reason being, For mods using the UI library through SpaceCore, the event would be shared for all of them. So, if a hypothetical SuperFunCatMod subscribes to the event, it will also get the handler invoked for things from a hypothetical SuperFunMagicMod who also uses the UI.

I think a better approach would be a non-static event on RootElement, and Element would invoke the event field stored on the Element.GetRoot(). This way, it's still local to each mod (or I suppose each screen, actually) without having to subscribe to every element. I've only looked at a tiny portion of this PR though - do you think that would still be feasible with what you've done so far?

[khanshoaib3]

I think a better approach would be a non-static event on RootElement, and Element would invoke the event field stored on the Element.GetRoot(). This way, it's still local to each mod (or I suppose each screen, actually) without having to subscribe to every element. I've only looked at a tiny portion of this PR though - do you think that would still be feasible with what you've done so far?

Definitely. I'll try to implement it. And just to understand it correctly, how would we subscribe to the non-static events, won't that require subscribing for every object we create? I haven't tested this so I really don't have a clue, and my event implementation was kind of just brute force, I found it easiest to implement although being not ideal. So I'm open to any other way to implement this, if any.

One more way that I could think of is having overridable methods for hover and others with default a11y implementation.
So mods extending elements can override and provide their own method of a11y.

And I do agree this being in GMCM 2.0, it would be best if we have a11y at the core of the library, makes it easier for devs.
So incase you go this route, I'll just continue providing this version to users on my discord server.
Or maybe you can publish this build as experimental or something so it's atleast somewhat official.

[spacechase0]

You would only need to subscribe to the RootElement instances, which by nature only have one per screen. This would also be the only class with the event field - any children call their RootElement's version, since they aren't intended for use without one.

This would mean that instead of subscribing once in Entry (or where ever you put it), you subscribe each time a RootElement is created (typically once per screen, I think?)

---

The problem with overridable methods is that then you have to subclass those methods to change the implementation, which isn't exactly feasible from an external mod.

Are there any other accessibility mods that exist that would find the functionality useful? If so, maybe instead of GMCM integrating with Stardew Access directly, GMCM could provide an API that gives hooks where Stardew Access needs, and then any mod could use that API (and I could add more later if those mods need more). But if there aren't actually any alternate accessibility mods of this type already existing, that might be more work than it's worth on your end.

---

I didn't mean this would wait until 2.0, I actually meant the opposite - I don't want to wait until 2.0, since that likely won't come out until SDV 1.7 - and it will likely be quite a while before it comes out - in case I have to make any compatibility breaking changes for general mod use to make that happen.

Also, I'll probably need to design the new API slightly differently to allow custom widgets to interface with Stardew Access properly without needing to do it directly, which would be hard to add with how things work currently (though I could make a small new function or two for that if necessary for a sooner release)

[khanshoaib3]

Oh so we would subscribe for every screen like maybe at their constructors. Will this have the possibility to have a default implementation to stardew access, like so, so other mods would only need to add ScreenReaderText and others to make screens somewhat accessible without them needing to integrate stardew access.

---

No, there aren't any screen reader mods like stardew access that I can think of, there are tts mods but they only read certain things like dialogs afaik.

---

That's great! Folks have been asking GMCM accessibility for a long time.

[spacechase0]

Nobody really uses the UI stuff besides me, and most of those mods are either unreleased or not very popular, so I don't think a default implementation is necessary. I do want to eventually rework my UI system entirely, so it'll be something I keep in mind then, but for now, GMCM should be fine.

It's a little complicated anyways, becauee GMCM doesn't actually use the copy of the UI that SpaceCore has - it has its own copy that is completely separate. and theoretically any of my other mods could do that, too. (Actually, every mod has a copy right now whether they use it or not, which is something I need to fix.)

[spacechase0]

Hey, just wanted to check on this since it's been a bit. I know many people are busy during the holiday season, so no rush, but if it's going to be a while longer I want to go ahead and get the release that's currently pending out.

[khanshoaib3]

Hey, just wanted to check on this since it's been a bit. I know many people are busy during the holiday season, so no rush, but if it's going to be a while longer I want to go ahead and get the release that's currently pending out.

Oh yeah, go ahead, I'm kind of busy these days with my new job (need to get used to working in corporate and all lol), so I don't know when I'll get back to modding, but I'll try to make time from now on.