bthome-writable

[yerpj]

bthome-writable initially started in this HA integration topic. Now it has a dedicated topic, to avoid mixing things up.

Please see the GitHub repo at https://github.com/yerpj/bthome-writable for Getting Started info

[yerpj]

@gfwilliams ,

please could you put the length after 0xFF, and then I'll roll this out in my actual home (I don't generally want to fiddle too much unless I break something I use!). I don't use the Pi's bluetooth, but instead use a Docker container on a NAS and a few ESPHome devices at BT proxies, so that should be another interesting test point :)

Done — spec is at 2.0-draft.4, both sides implemented, 588 tests green.

• Length after 0xFF. In. FF <ids…>, self-delimiting like everything else. It still has to be last, but that rule is now written down as what it really is — the reference parser stops at an ID it doesn't know — so it can go once 0xFF is assigned.
• MTU. Dropped, you're right. §4.4 now just says only text and raw are ever constrained, and by whatever MTU got negotiated.
• Rotation. Went simpler than either of us: a device has one declaration. No frequency rule, no consistency rule, nothing to verify. The cost is that all writable entries fit in one payload, which seems fair.
• Brightness/light. Agreed, out of scope. Same call I'd made about clamping: we carry a command and say whether it was delivered.

Counters — mostly already there. The device persists its write counter, and Home Assistant persists its own; a freshly created config entry seeds from the clock, which fixed a nasty one where re-adding a device left every sealed write silently refused.

One thing to separate though: skipping persistence is the device's own risk for writes, but the device also seals its advertising and reads, and a counter restarting at zero there reuses a nonce under the same key. Found that in my own firmware last week.

Your overflow window is right about wrap-around — one snag: it bounds forward jumps too, and receivers need those. §5.3 asks them to offer a resynchronisation, and mine seeds from wall-clock seconds, well outside LAST+1000. Worth an exception for an explicit resync?

On letting the stack split long writes — I'd like that, but the one time I tried a payload above MTU-3 it was refused outright rather than prepared. I'll measure before promising it.

Length byte is in if you still want to try it. Your NAS-plus-proxies setup is exactly the second bench this has been missing :-)

[gfwilliams]

All agreed - that sounds great!

Counters.. Worth an exception for an explicit resync?

Hmm - it's tricky. Doesn't allowing a resync potentially void the whole defence against a replay attack?

Or - maybe instead of 1000, we allow last+1 ... last+0x80000000. That's super broad but:

• To replay, you'd need to have a packet from 2 billion writes before, which isn't going to happen - so we're still secure
• If there were an error writing, you could resync just by adding 0x80000000 to the counter and trying again?

[yerpj]

Taking your window — it's better than the exception I was asking for, and it dissolves the problem: both jumps I care about land inside LAST+1 … LAST+0x80000000, and it handles wrap-around properly, which mine didn't.

Before we settle §5.3 though, I went looking at how other protocols handle the lost-state case, and they converge somewhere neither of us proposed: ask the device, and authenticate the answer.

• Zigbee R23 makes it mandatory (§4.6.3.8): the receiver sends a random 8-byte challenge, the peer replies with its current frame counter MIC'd under the link key, and the receiver adopts it even if it is lower. That is safe precisely because freshness comes from the challenge, not from the counter.
• Matter does the same thing (MCSP, §4.18). KNX Data Secure seeds its sequence from the clock like I do — and still ships a sync handshake on top. SwitchBot and Nuki just hand out their current IV/nonce before each command.
• What nobody does any more is accept a low counter after a reboot. LoRaWAN 1.0.x is the documented replay hole, removed in 1.1; Matter's spec warns about its own trust-first mode. Worth knowing: bthome-ble currently accepts any counter below 100 for exactly that reason, and it is already the subject of a "BTHome is not secure at all" issue upstream. Fine for uplink sensors, not something to inherit on the downlink.

So: would you consider a readable characteristic carrying the device's current write counter? A receiver reads it once at setup, or after writes start failing, and never has to guess.

The nice part is it costs us almost nothing — reads are already sealed under the read direction byte, so the answer is authenticated for free, which is the one real objection (a bare read would let anyone in range push a hub's counter forward). And the counter isn't secret: Zigbee's response is explicitly not encrypted, Matter returns it plainly, BTHome already puts it in the clear on every advertisement, and NIST SP 800-38C doesn't ask for nonce secrecy.

My clock seeding would then become what KNX treats it as — a sensible default, not a substitute for asking.

[yerpj]

Correction to myself: I said the sealed read authenticates the answer "for free". True against forgery — nobody without the key can produce one — but not against replay. Something impersonating the device could serve an old sealed counter, and a receiver that has just lost its state has no way to tell it's stale. It would then seed too low and have its writes silently refused: a denial of service rather than a compromise, but exactly the hole Zigbee and Matter close by binding the answer to a challenge.

So the shape should be theirs: the receiver writes a random value first, the device seals it alongside the counter, and a replayed response doesn't carry the right one. One extra write before the read, and it's closed properly.

[gfwilliams]

readable characteristic carrying the device's current write counter

Sounds good - just make it 2FAA1000-.... or something like that?

Write before read sounds ok too. Shame about the complexity but it's good to properly fix these niggles.

I just hope BTHome takes it on after this :)

[yerpj]

Agreed for 2FAA1000 . It is better than mine: FFFF was a dead end, yours opens a block, so I'll write the split into §4.1 while I'm there: 0x0001–0x0FFF for entries, 0x1000+ reserved for protocol control. Entries cap at 255 anyway, so there's plenty of room.

On the complexity: it's all opt-in. Off by default on both sides, so a device that doesn't want the resync adds no characteristic and pays nothing.

Implementing now, I will notify once done.

[MaBecker]

Does that mean we can have also actors and not only sensor's.

[gfwilliams]

That's the idea, yes. They can be both.

[yerpj]

Hopefully, if bthome is kind enough to accept the proposal ;-) .

[yerpj]

§5.6
is written and pushed — spec at
2.0-draft.7,
both sides implementing the counter report at 2FAA1000, validated on hardware,
and optional throughout.

The receiver asks by default, which I had not expected.
§5.3
has a device resume strictly above anything it accepted, so it is ahead of its
receiver after every restart with nothing to say so — a reflash was enough to
reproduce it, and the resynchronise button turned a silent 100.8 → 99.5 lux into
101.2 → 596.5 lux.

What it does not close is a device restarting while the receiver keeps running,
since the receiver only asks at set-up. I do not think a receiver can detect
that at all, so it needs something in the protocol:

• a bit the device sets for its first few packets after a restart, or
• a line in §5.3 saying a receiver SHOULD ask before the first write of a
  session rather than once at start-up.

I have deliberately implemented neither. Which appeals to you?

Separately: an audit against the real bthome-ble found I was building the CCM
nonce from the in-packet MAC as transmitted rather than reversed, so any device
setting the MAC-included flag was unusable. Fixed, and now checked against
BTHomeData.get_nonce().

[gfwilliams]

Thanks!

The receiver asks by default,

So it now asks before every write, or just after a reboot?

and the resynchronise button turned a silent 100.8 → 99.5 lux into 101.2 → 596.5 lux.

Why does the value change depending on if it is resynced or not? I'm not sure I really understand this - do we need to be validating writes better?

"The receiver"

I think the language may be a bit confusing here. For BTHome the 'receiver' (Home Assistant?) makes sense, but when we're talking about writes in bthome-writable, the receiver could be considered to be the device we're writing to :)

5.3 ... "and MUST persist it across restarts."

Because we've got this resynchronisation now, do we really need to enforce that it is persisted?

Which appeals to you?

Well, maybe get the receiver to ask? But what do we class as a 'session'? Part of me feels that to make this robust we should have some way to tell if a write failed, so that the receiver can then automatically re-request the counter?

I know Espruino doesn't allow it right now, but that's probably not a reason to have 'write with ack' fail if the write counter isn't correct. I could always tweak the firmware.

Or I guess either include 0x26 (problem) at the end of the 0xFF bthome-writable list, or maybe even include 0xFF in the bthome-writable list (because we know that's not going to collide with any writable entity) if there was an error? then bthome-writable can detect it and re-request the counter

edit: Or maybe we say that if the receive counter is wrong (which would indicate a restarted BTHome device) we automatically request a write counter on the next write?

[yerpj]

I know Espruino doesn't allow it right now, but that's probably not a reason
to have 'write with ack' fail if the write counter isn't correct

That is the answer, and it needs nothing new —
§4.2
already says a device SHOULD reject a write with an ATT error where its platform
allows it, and names Espruino as the exception. Lift that and the SHOULD simply
starts being honoured. The receiver side needs no change either: a failed GATT
write already surfaces as an error in Home Assistant.

if the receive counter is wrong (which would indicate a restarted BTHome device)
we automatically request a write counter on the next write?

Yes — and that follows from the above rather than replacing it: without a
failure signal the receiver has nothing to trigger on. Together they make
§5.6
cheaper than it is today. To answer your first question: the receiver currently
asks once at set-up, never before a write. On a refused write it would ask on
demand instead, which is better and one connection less.

include 0x26 (problem) at the end of the 0xFF bthome-writable list

Worth keeping as a fallback for platforms that cannot fail the write, but not as
the primary: it is device-wide rather than bound to the write that failed, it
costs advertising bytes, and a receiver can miss the packet.

maybe even include 0xFF in the bthome-writable list

This one I would avoid. The list is positional — entries are numbered from 1 in
order and map to characteristic numbers — so a marker that is not an entry
breaks that mapping.

On persistence: I would keep the MUST, since the counter report is optional and
a device may not offer it. With ATT errors it becomes an optimisation rather
than a necessity, but only for devices that do.

Fair point on "receiver" being ambiguous here — I will add a line to §1 rather
than rename, since it is BTHome's own word for the consumer.

And the lux figures were my sloppy writing, not a validation problem: two runs
of the same loop. The first had a desynchronised counter, so the write was
refused and the LED never lit — 100.8 → 99.5 is just the room. The second, after
resynchronising, actually lit it: 101.2 → 596.5.

[yerpj]

A short summary for anyone arriving here, since the thread has got long.

bthome-writable adds a downlink to BTHome. A device lists in its ordinary
BTHome advertising which object types it accepts writes for — object 0xFF,
last in the service data, so existing receivers skip it. Home Assistant connects
briefly, writes one value in BTHome's own object format to that entry's own GATT
characteristic, and disconnects. The write response is the acknowledgement, and
a writable value is never advertised.

So a BTHome device can have switches, lights, setpoints, text and buttons
alongside its sensors — on the same device card, with the same bindkey, sealed
with BTHome's own AES-CCM in both directions. Nothing to configure: the
declaration is the configuration.

Trying it takes about ten minutes and a Puck.js:
docs/try-it.md.
The example closes the loop with a measurement rather than an echo — Home
Assistant switches the green LED, and the Puck's own light sensor reports it
back as illuminance. Put the Puck under a mug first, or the room swamps it.

Protocol:
spec/PROTOCOL.md.
Everything so far has run on one bench, so a second pair of hands is the thing
this most needs.

[gfwilliams]

Just to say I tried this on a Puck and it works great! It doesn't seem to be keeping the connection open any more than required which is really good, and I haven't seen it miss a write yet (although I'm not running encrypted atm)

I appreciate the '...-standalone' examples in https://github.com/yerpj/bthome-writable/tree/main/espruino/dist so there's no messing about with modules too. As soon as you're happy it'd be great to pull the modules into EspruinoDocs so folks can start experimenting easily.

[enaon]

My setup is: RPI4b with openwrt, latest HA image running on podman, 6 espHome proxies and the local bluez BT.

I tested it a bit, it seems to be working fine, both using local BT and the proxies. Something puzzles me though. The BTH writable extension does not get the battery arvertized, I have to use the normal BTH extension to get it. Is this intended?

an one more thing that is a bit strange is that every so often, I can see that once I call changed(), the HA connects more than once, 2 or 3 times, it gets the reading on the first try, but connects again.

var bw = require("ew_m_btHwr");
bw.setup({
  advertise: [
    { type: "battery", interval: 300000, get: function () { return ew.sys.batt(1); } },
    {
      type: "light",
      get: function () { return ew.apps.lamp.state.isOn; },
      set: function (v) {
        if (v) ew.apps.lamp.on();
        else  ew.apps.lamp.off();
      },
    },
  ],
  interval: 1000,
  maxServiceData: 20,
  showName: false,
  onError: function (error) {
    console.log("write rejected:", error.code, error.message);
  },
});

[yerpj]

@enaon — thank you, and not politely: this is the first time it has run anywhere
but my own bench, and you found a real bug in an afternoon.

every so often, I can see that once I call changed(), the HA connects more
than once, 2 or 3 times

Two faults, both mine. The re-read loop continued whenever any advertisement
arrived during a read — and since a device advertises every second or so while a
read takes a connection, that was almost always. Worse, the guard against
starting a second read was re-entrant: async_create_task starts the coroutine
eagerly, so the task variable still pointed at the previous, finished one, and an
advertisement handled in that window found no read in progress and started
another. Your six proxies make advertisements arrive often enough to hit it; one
adapter and one proxy here did not. Fixed, with a test that pushes an
advertisement from inside the read — the old one pushed after it had settled,
which is why it never saw this.

The battery is working as intended: this integration adds controls only and
your sensors keep coming from the core BTHome integration — you want both on the
same device. That was in the docs as an aside rather than as the first thing it
says, which is my fault and now fixed.

If you keep it running, the thing I would most like to know is whether writes
ever come back acknowledged without the device acting on them.

[yerpj]

As soon as you're happy it'd be great to pull the modules into EspruinoDocs so folks can start experimenting easily.

Thanks for testing it — and yes to EspruinoDocs. I fixed two things in the
module first, both found by review rather than by anyone hitting them:

• When the radio refuses an advertising packet the module publishes a minimal
  one instead, so the device stays connectable. On a device with a bindkey that
  packet still said "encrypted" while carrying plaintext, so receivers decided
  the key was wrong.
• The connect and disconnect handlers were re-registered on every setup(),
  so a device being developed on reacted twice to each connection, and a
  failure inside one of them escaped into the stack's callback.

@gfwilliams @enaon :One gap worth naming: both reports so far have been unencrypted, so the
sealed path has still only ever run on my own bench. If anyone fancies it, that
is the part I would most like a second pair of eyes on — it is where most of the
protocol machinery lives.

Switching an existing device over is two steps:

1. On the device, add a bindkey to setup() — 32 hex characters.
   espruino/examples/encrypted-light.js
   is the same light example with one already set.
2. In Home Assistant, it asks for that key by itself once it notices the
   advertising no longer decodes. Give the same key to the core BTHome
   integration as well — the two keep separate copies, which is a seam worth
   seeing for yourself.

Nothing else changes: same declaration, same characteristics, same writes —
just sealed in both directions.

[enaon]

BTHome supports rotating multiple packets I believe, so yes, you could have:

• Packet 1: lists all the stuff you want to be able to write - ~16 items at once IIRC
• Packet 2/3/4/5: all the state you want to advertise

That would make it more powerful I believe, I tried to understand how this could be done in the existing code but I think it is not very easy. so maybe @yerpj can take a look at it at a later time, thank you.

[yerpj]

@enaon — all three fixed on main, thanks.

• The reboot. A device resumes its write counter ahead of the receiver, so
  every write after E.reboot() was refused after being acknowledged —
  invisible. The counter report existed but only ran at HA startup; a restart
  announces itself with a fresh settings revision now, and HA asks before the
  next command.
• The counter warnings. Equal, not smaller: one advertisement delivered by
  several of your six proxies. Quiet now — bthome-ble relaxed that same rule
  between 3.22.1 and 3.24.0.
• One you had not hit, found checking your third point: after a write, a
  readable characteristic held the write instead of the value.

The UI state is what stays: a refused write is acknowledged before it is
validated (§4.2),
so HA believes it, and the fix is removing the cause rather than detecting it.
The ON→OFF→ON does not reproduce here — which commit are you on? Yes please to
issues on the repo. All of it in
D-092.

Multiple packets: already allowed — a device may rotate payloads, the
declaration only has to be in the packet that carries it
(§2.2).
The module publishes one for now; splitting may come after the BTHome submission.
Today, measured: your five settable parameters cost 7 bytes in total, since
writable values are never advertised. Your full list is 22 bytes — one packet in
clear (24 available), not sealed (16).

[yerpj]

@gfwilliams

My thoughts: [...]

Thank you, that framing is better than mine, and the use-case list
is a good addition.

Two things since: the sealed path got its first test outside this bench — twenty
lights behind six ESPHome proxies — and it found a real fault. A device that
reboots resumes its write counter ahead of the receiver, and the fix for that was
wired to the wrong event, so every command after a reboot was refused in silence.
Fixed, along with a characteristic that held the last write instead of the entry's
current value.

I will prepare the communication for BTHome in line with your proposal. Thanks again!

[yerpj]

@enaon — a correction, and it is owed.

The ON→OFF→ON does not reproduce here — which commit are you on?

It reproduces. It was in my own recorder the same day, four times over, and I
had not looked. Caught live since: a command takes a few seconds to catch the
device, you press again while it is still out there, and when the first one
finally lands the entity treats that completion as its own — it drops the value
it is showing and falls back on the one that write just stored. Here the wrong
state stood for three seconds.

So it is neither your setup nor your checkout. Completions were reported per
entry rather than per write, and an entity could not tell its own from an older
one's. Fixed on main
(D-094).

The test is worth a line, because it is why this survived: asserting the state
after both commands passes with the fault still in place — in a test harness
both writes finish in microseconds and the wrong value is shown for no time at
all. It now records every state the entity passes through.

Nothing for you to do; it is there if you pull.

[enaon]

Caught live since: a command takes a few seconds to catch the
device, you press again while it is still out there, and when the first one
finally lands the entity treats that completion as its own

This is not the case I think. I uploaded a video in the issues on your GitHub, in that video I also have serial connection to the light open so I can see the BT writes. The previous wrte was done, yet the on-off-on occured. I only mention it because I understand it should not happen, otherwise it is not a problem.

I have seen it happen on the first press too, I do not think it is related to the sequence of commands. But I will test your new code, it is not happening anymore, case closed :)

edit: you are correct, I just watched again the video I uploaded, indeed the previous write command was not done, nice.

[yerpj]

@gfwilliams — the modules are in, as two pull requests:
EspruinoDocs#761 for AESCCM ,
and #762 for BTHomeWritable.

Separate, because AESCCM stands on its own: BANGLEJS2.py builds AES_CCM and not AES, so a Bangle.js 2 needs it whatever BTHome decides — while the writable module carries an object ID BTHome has not assigned yet. Both pages say so above the usage section.

[yerpj]

when you do submit it to BTHome, please let us know where, so we can show some support.

Done so far:

• PR on bthome.io
• PR on bthome-ble
• A short description of what has been done in an open issue

[enaon]

• Remove the device in Home Assistant and add it again, so the stored write counter restarts, then restart Home Assistant.

Will do

• Install the module and AESCCM from the same source, and tell us which version ew_m_btHwr holds.

I am always using your GitHub, I have renaimed the modules just so to be sure I do not get the ones from ww.espruino.com when I do changes on the scripts that calls the modules> So I am using your latest versions. I will not update for some time, I will try to reproduce the issue reliably.

Take the setTimeout(() => bw.changed(), 250) back out. If the wrong reads stay away without it, the cause really was on the receiver side.

I did, and cannot reproduce it now :) I will try again at~8-9 am, maybe environmental noice was a factor.

I will try to reproduce, and do a better job at logging the problem this time. Thanks.

Found along the way, and worth knowing: we had added a read back after every write, to prove the write landed. It does not work on a device like yours. A GATT read returns what the device sealed the last time it published, and the module republishes on the tick after the write, so a fade publishes before the fade is over. Every turn_off came back as "the command did not reach the device" while the lamp turned off. It is now a log line instead of an error.

Nice, Thanks! This is how it all started, I noticed that, I kept pressing the button to understand why, I realized it was the way my fading works, but I got the issue we are talking about in the process. Maybe I was pressing the button too fast, I will try button storming too :)

[enaon]

@enaon, we reproduced one of your three faults on the bench and fixed it.

I was not able to reproduce the other two. I was indeed changing the config, adding and removing entities before the issue, so it seems the cached table, along with the "worth knowing", is the whole story. Thank you for you time.

I did manage to get to get the BT stuck, by pressing the buttons on the lamp rapidly. The lamp was saying

NRF.getSecurityStatus().connected
=true

but no connection was open from HA. The hci0 adapter was working fine on other lamps. I disconnected and connected the USB cable, so I got the console back.

>process.memory()
={ free: 8523, usage: 4477, total: 13000, history: 15,
  gc: 363, gctime: 27.83203125, blocksize: 14,
  rx: { used: 0, total: 2048 },
  tx: { used: 255, total: 256 },
  stackEndAddress: 537077272, stackFree: 54368, flash_start: 0, flash_binary_end: 473572, flash_code_start: 487424,
  flash_length: 1048576 }
>E.getErrorFlags()
=[
  "LOW_MEMORY"
 ]
 
 >print(E.internal.BLE_ADV_D)
new Uint8Array([2, 1, 6, 3, 255, 144, 5, 23, 22, 210, 252, 65, 169, 86, 124, 76, 231, 147, 70, 149, 76, 12, 252, 45, 236, 33, 1, 239, 144, 115, 150]).buffer
=undefined
> 

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/commands.py:336
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 6:07:57 AM (7 occurrences)
Last logged: 6:44:10 AM

[546606191664] Error during service call to switch.turn_off: Light 1 could not be reached: the command was not delivered. E6:2F:28:C4:37:82 - E6:2F:28:C4:37:82: Failed to connect after 4 attempt(s): TimeoutError
[546606191664] Error during service call to switch.turn_on: Light 1 could not be reached: the command was not delivered. E6:2F:28:C4:37:82 - E6:2F:28:C4:37:82: Failed to connect after 3 attempt(s): TimeoutError
[546606188944] Error during service call to switch.turn_off: Light 1 could not be reached: the command was not delivered. E6:2F:28:C4:37:82 - E6:2F:28:C4:37:82: Failed to connect after 4 attempt(s): [org.bluez.Error.InProgress] In Progress
[546604518864] Error during service call to switch.turn_off: Light 1 could not be reached: the command was not delivered. E6:2F:28:C4:37:82 - E6:2F:28:C4:37:82: Failed to connect after 4 attempt(s): TimeoutError
[546604518864] Error during service call to switch.turn_on: Light 1 could not be reached: the command was not delivered. E6:2F:28:C4:37:82 - E6:2F:28:C4:37:82: Failed to connect after 3 attempt(s): TimeoutError

I then did a
NRF.disconnect()
and everything was working again

I now have a function that makes sure bw.changed() cannot be called too often, so it is not really a problem with the module.

[yerpj]

Thank you very much for your deep dive into those bugs, very helpful! I am on it, I will provide an updated version ASAP

[yerpj]

@enaon
The wrong reads staying away is the answer I needed, and the
buttons you pressed found the next defect.

bw.changed() was the most expensive call in the module and it ran once per
press, 375 ms of the single thread on a sealed device. Twenty presses in a
second asked for seven and a half seconds of work, which is why your lamp's own
fade never got a step in edgeways. It now folds a burst into one rebuild 250 ms
later: measured on a nice!nano, twenty presses were twenty rebuilds before and
are three now. So you can take your rate limiter out, and the
setTimeout(() => bw.changed(), 250) with it. Call changed() on every local
change and let the module group them. The specification says so now too (§3.2).

What I did not reproduce: LOW_MEMORY. My boards kept 8000 JsVars free at the
worst of the same burst. The thread saturation is the cause and it is fixed; the
step from there to a stack that still believes it is connected is your board's,
and I am not going to pretend otherwise.

Same branch, both sides from the same commit:
https://github.com/yerpj/bthome-writable/tree/candidate/enaon-retest, module
at espruino/BTHomeWritable.js.

Still open from last time, if you have the time: the device removed and added
again so the stored write counter restarts, and which version ew_m_btHwr
holds.

[enaon]

Still open from last time, if you have the time: the device removed and added
again so the stored write counter restarts, and which version ew_m_btHwr
holds.

I tested device removed and added again yesterday, forgot to mentioned it, I think it works as should. On the BTHomeWritable.js version, I was on the one you made the PR for that same day. It is just that I was using it few hours before the PR, manually downloaded it from you GitHub.

I now moved to this BTHomeWritable.js (https://github.com/yerpj/bthome-writable/blob/candidate/enaon-retest/espruino/BTHomeWritable.js), and the custom component from that branch.

I run some manual stress tests, not very thoroughly, all looks good at my side.

[enaon]

one little thing that I want to verify the behavior of, against your bench, is the following: (not related to latest changes, it was this was from the start)

The first command I send over the the lights after a period of inactivity, say 3-5 minutes, takes a lot of time in my case. I tested the hci0 alone, the ESPproxy alone, and both. The first command can take anywhere from 5-10 seconds. The next command is fast in the lasted versions, maybe 1-3 sec. This seems not light/Espruino BT sleep mode related, meaning that if I send one command to one light that takes 5-10 seconds, I can then send a command to a different light and it will arrive fast**. This looks like a HA assistant thing to me, is this the case in your bench too?

** I maybe wrong about this, I will test some more.

[yerpj]

It may actually come from bthome-writable integration itself. Not sure yet, I am looking at it right now...

[enaon]

It may actually come from bthome-writable integration itself. Not sure yet, I am looking at it right now...

It does not look like an integration issue. It looks like sometimes espHome is loosing a lot of advertisements, like 7-9 at 1 sec interval.

2026-10-23 11:30:15.026 DEBUG (MainThread) [habluetooth.wrappers] E6:2F:28:C4:37:82: Looking for backend to connect
2026-10-23 11:30:15.027 INFO (MainThread) [habluetooth.wrappers] E6:2F:28:C4:37:82 - eT-Black-4: Found 2 connection path(s), preferred order: pr-4 (30:AE:A4:5D:AF:AA) (RSSI=-63) (failures=0) (in_progress=0) (slots=3/3 free) (score=-63.0), hci0 (B8:27:EB:11:22:33) (RSSI=-65) (failures=0) (in_progress=0) (slots=5/5 free) (score=-65.0)
2026-10-23 11:30:15.027 DEBUG (MainThread) [habluetooth.wrappers] E6:2F:28:C4:37:82 - eT-Black-4 -> 30:AE:A4:5D:AF:AA: Connecting via pr-4 (30:AE:A4:5D:AF:AA) [partial] (last rssi: -63)
2026-10-23 11:30:15.110 INFO (MainThread) [homeassistant.components.esphome.manager] pr-4: [I][bluetooth_proxy:122]: [0] [E6:2F:28:C4:37:82] Connecting v3 with cache
2026-10-23 11:30:15.112 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp32_ble_tracker:145]: connecting: 0, discovered: 1, disconnecting: 0, active: 0
2026-10-23 11:30:15.113 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp32_ble_tracker:594]: Stopping scan to make connection
2026-10-23 11:30:15.113 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp32_ble_tracker:602]: Promoting client to connect
2026-10-23 11:30:15.114 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp32_ble_tracker:640]: Setting coexistence to Bluetooth to make connection.
2026-10-23 11:30:15.114 INFO (MainThread) [homeassistant.components.esphome.manager] pr-4: [I][bluetooth_connection:137]: [0] 0x01 Connecting
2026-10-23 11:30:15.124 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp32_ble_tracker:145]: connecting: 1, discovered: 0, disconnecting: 0, active: 0
2026-10-23 11:30:22.932 DEBUG (MainThread) [habluetooth.base_scanner] pr-4 (30:AE:A4:5D:AF:AA): Scanner watchdog time_since_last_detection: 7.899887561798096
2026-10-23 11:30:24.635 INFO (MainThread) [homeassistant.components.esphome.manager] pr-4: [I][bluetooth_connection:584]: [0] Connection open
2026-10-23 11:30:24.636 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][bluetooth_connection:138]: [0] [E6:2F:28:C4:37:82] Connected with cached services, sending connected (mtu=23)
2026-10-23 11:30:24.636 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][bluetooth_connection:356]: [0] custom conn params
2026-10-23 11:30:24.638 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp32_ble_tracker:145]: connecting: 0, discovered: 0, disconnecting: 0, active: 1
2026-10-23 11:30:24.642 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp-idf:000][BTU_TASK]: W (92641318) BT_L2CAP: L2CA_UpdateBleConnParams connection parameter already set
2026-10-23 11:30:24.644 DEBUG (MainThread) [habluetooth.wrappers] E6:2F:28:C4:37:82 - eT-Black-4 -> 30:AE:A4:5D:AF:AA: Connected via pr-4 (30:AE:A4:5D:AF:AA) [partial] (last rssi: -63)
2026-10-23 11:30:25.079 INFO (MainThread) [homeassistant.components.esphome.manager] pr-4: [I][bluetooth_connection:193]: [0] Disconnecting (conn_id: 0)
2026-10-23 11:30:25.079 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp32_ble_tracker:145]: connecting: 0, discovered: 0, disconnecting: 1, active: 0
2026-10-23 11:30:25.079 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp-idf:000][BTU_TASK]: W (92641776) BT_HCI: hci cmd send: disconnect: hdl 0x0, rsn:0x13
2026-10-23 11:30:25.080 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][bluetooth_connection:168]: [0] [E6:2F:28:C4:37:82] Disconnected, reason=0x00, freeing slot
2026-10-23 11:30:25.085 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][bluetooth_connection:614]: [0] DISCONNECT_EVT reason=0x16
2026-10-23 11:30:25.085 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][bluetooth_connection:614]: [1] DISCONNECT_EVT reason=0x16
2026-10-23 11:30:25.086 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp32_ble_tracker:145]: connecting: 0, discovered: 0, disconnecting: 0, active: 0
2026-10-23 11:30:25.153 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp32_ble_tracker:644]: Setting coexistence preference to balanced.
2026-10-23 11:30:25.650 DEBUG (MainThread) [homeassistant.components.esphome.manager] pr-4: [D][esp-idf:000][BTU_TASK]: W (92642321) BT_HCI: hcif disc complete: hdl 0x0, rsn 0x16 dev_find 1

[yerpj]

Your logs settled it: the GATT exchange is 0.44 s in the 11:30 trace, and the 9.5 s goes before the link exists.

I think your lost advertisements and the slow connect are one event, not two. At 11:30:15.112, esp32_ble_tracker stops the scan to promote the client, so the proxy is blind for the whole attempt: 7.9 s without a detection inside a 9.5 s window, which is your 7 to 9 missing advertisements at a 1 s interval. So the question I can act on is not why ESPHome loses them, it is why I was asking for so many connections.

Two were mine, and they are gone. The counter question of §5.6 now rides the connection a command already holds, which is what made your first command after a restart slow, and a state read that a moved settings revision has called for rides it too. On top of that, every device now gets its own turn: an Espruino serves one central, but Home Assistant's connection cap counts host slots, so a command and a read could each take one and then fight over the same board.

Nine samples per arm here: worst command 17.2 s to 6.6 s, median 5.9 s to 3.1 s, and a knob turned locally reaching Home Assistant 9 times out of 9, against 4 of 9.

candidate/enaon-retest is updated if you want another round. bthome_writable_write now splits the time into queued_ms, connect_ms, counter_ms, write_ms.

And your other question: the read is keyed to the settings revision rather than to the write, so calling changed() once the dimmer settles is the right move. It just costs less now, since the next command carries that read.

Code is ready to be tested if you want to :-)

[enaon]

Two were mine, and they are gone. The counter question of §5.6 now rides the connection a command already holds, which is what made your first command after a restart slow, and a state read that a moved settings revision has called for rides it too.

I had not realized that every call on changed() was causing the next HA action to ask for the counter. So it was allways 2 connections in that case? This explains why "sometimes" I was getting delays.

I tested it, it is a lot faster, very usable. Well done, I think this was maybe the most important improvement usability wise. Thanks!

[yerpj]

I'm glad to hear that!
I still need to test one or two things in encrypted mode, and then I think the module should be ready enough to be released.

Thanks again for all the testing you've done!