feat: add openHiTLS backend support by tlhc · Pull Request #2058 · obgm/libcoap

tlhc · 2026-06-16T05:00:56Z

Add openHiTLS DTLS/TLS backend
Integrate openHiTLS crypto into libcoap (D)TLS and OSCORE paths.
Extend TLS tests and add interop validation script.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds openHiTLS as a supported DTLS/TLS (and OSCORE-only) backend, with build-system integration and expanded TLS/crypto test coverage.

Changes:

Introduce a new openHiTLS backend implementation (coap_openhitls.c) and expose it via the TLS library enum and debug/version string reporting.
Integrate openHiTLS into Autotools + CMake builds (including OSCORE-only mode) and update licensing notes.
Expand TLS/DTLS/OSCORE unit tests and add a backend smoke-test script for coap-client/coap-server.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/test_tls.c	Adds openHiTLS detection plus a large set of TLS/DTLS/OSCORE capability and behavior tests.
src/coap_sha1.c	Avoids compiling internal SHA1 fallback when openHiTLS (or openHiTLS OSCORE) is present.
src/coap_oscore.c	Consolidates recipient sequence APIs into a new function signature.
src/coap_openhitls.c	New openHiTLS DTLS/TLS + OSCORE crypto backend implementation.
src/coap_notls.c	Updates NOTLS compilation guards to exclude openHiTLS builds.
src/coap_debug.c	Adds openHiTLS to `coap_string_tls_version()` formatting.
include/coap3/coap_dtls.h	Adds `COAP_TLS_LIBRARY_OPENHITLS` enum value.
examples/tls_backend_testcases.sh	Adds a comprehensive local smoke test script for TLS backends (focused on openHiTLS).
examples/coap-server.c	Guards OSCORE helper functions behind `COAP_OSCORE_SUPPORT` to match build options.
examples/Makefile.am	Installs/distributes the new TLS backend test script.
configure.ac	Adds Autotools options and link checks for openHiTLS (DTLS/TLS + OSCORE-only).
cmake_coap_defines.h.in	Adds CMake config defines for openHiTLS and openHiTLS OSCORE-only.
cmake/FindopenHiTLS.cmake	New CMake find-module providing `openHiTLS::openhitls` target.
Makefile.am	Adds the new CMake find-module and includes `src/coap_openhitls.c` in sources.
LICENSE	Documents openHiTLS license implications.
CMakeLists.txt	Adds openHiTLS to selectable DTLS/OSCORE backends and links it via `openHiTLS::openhitls`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tlhc · 2026-06-16T05:24:39Z

currently can build libcoap with openhitls support use this scripts

https://gist.github.com/tlhc/16b4ad85967c9039d006e3d4e621b698

tlhc · 2026-06-16T10:42:27Z

Hi @mrdeep1, should I add GitHub actions coverage for the new openHiTLS SSL backend in this PR?

mrdeep1 · 2026-06-16T11:10:37Z

should I add GitHub actions coverage for the new openHiTLS SSL backend in this PR?

Yes, this makes sense.

As an FYI (have not looked into why yet), building and installing the openHiTLS libraries (into /usr/local/lib), and then (in the libcoap directory) running

./configure --with-openhitls

fails with

....
checking for compatible openHiTLS version (>= 0.2.0)... no
configure: error: ==> openHiTLS headers too old. openHiTLS >= 0.2.0 required for suitable DTLS/TLS support build.

tlhc · 2026-06-16T11:47:22Z

should I add GitHub actions coverage for the new openHiTLS SSL backend in this PR?

Yes, this makes sense.

As an FYI (have not looked into why yet), building and installing the openHiTLS libraries (into /usr/local/lib), and then (in the libcoap directory) running
./configure --with-openhitls
fails with
....
checking for compatible openHiTLS version (>= 0.2.0)... no
configure: error: ==> openHiTLS headers too old. openHiTLS >= 0.2.0 required for suitable DTLS/TLS support build.

Thanks, I reproduced this, the current autotools detection handles the openHiTLS source-tree layout, but not the installed layout.
configure --with-openhitls fails at the version check and reports "headers too old", but the actual config log is the install tree layout issue, I will fix it

mrdeep1 · 2026-06-16T14:56:28Z

I suspect you may need he equivalent to m4/ax_pkg_check_mbedtls.m4 as openHiTLS does not appear to make / have a pkginfo *.pc file for getting the installed library information from.

tlhc · 2026-06-16T15:12:13Z

I suspect you may need he equivalent to m4/ax_pkg_check_mbedtls.m4 as openHiTLS does not appear to make / have a pkginfo *.pc file for getting the installed library information from.

make sense, I will fix, it's much clear for autotools

mrdeep1 · 2026-06-16T15:17:41Z

A very quick test shows that src/coap_openhitls.c is not handling / recovering error conditions (getting lots of writes into freed memory). I suggest that you build libcoap with ./configure --enable-sanitizer or cmake -DENABLE_SANITIZER=ON when doing any testing.

tlhc · 2026-06-16T16:11:36Z

A very quick test shows that src/coap_openhitls.c is not handling / recovering error conditions (getting lots of writes into freed memory). I suggest that you build libcoap with ./configure --enable-sanitizer or cmake -DENABLE_SANITIZER=ON when doing any testing.

OK, thanks for the guidance.I also used gcov to check the coverage for coap_openhitls.c:

lines:     69.2%  (1021 / 1476)
functions: 96.0%  (97 / 101)
branches:  49.9%  (638 / 1279)

I will continue improving the test coverage.

tlhc · 2026-06-17T06:10:29Z

I suspect you may need he equivalent to m4/ax_pkg_check_mbedtls.m4 as openHiTLS does not appear to make / have a pkginfo *.pc file for getting the installed library information from.

make sense, I will fix, it's much clear for autotools

fixed in https://github.com/obgm/libcoap/compare/9b119abed6f6741680053755c5d10cdd60ba3250..13062c45530b79a307e730b932e4f5226d39068f

tlhc · 2026-06-17T06:23:59Z

A very quick test shows that src/coap_openhitls.c is not handling / recovering error conditions (getting lots of writes into freed memory). I suggest that you build libcoap with ./configure --enable-sanitizer or cmake -DENABLE_SANITIZER=ON when doing any testing.

The ASan issue is fixed by validating the backend env

env helper

libcoap/src/coap_openhitls.c

Lines 1539 to 1548 in e4b7b1b

    
           static coap_hitls_env_t * 
        
           coap_hitls_live_env(coap_session_t *session, coap_hitls_env_t *env) { 
        
             return session && session->tls == (void *)env ? env : NULL; 
        
           } 
        
           static void 
        
           coap_hitls_clear_pdu(coap_hitls_env_t *env) { 
        
             if (env) { 
        
               env->pdu = NULL; 
        
               env->pdu_len = 0;

DTLS receive path

libcoap/src/coap_openhitls.c

Lines 2390 to 2429 in e4b7b1b

    
               int32_t hret = HITLS_Read(env->ctx, pdu, sizeof(pdu), &read_len); 
        
               if (hret == HITLS_SUCCESS && read_len > 0) { 
        
                 coap_log_debug("*  %s: dtls:  recv %4d bytes\n", 
        
                                coap_session_str(session), (int)read_len); 
        
                 ret = coap_handle_dgram(session->context, session, pdu, read_len); 
        
                 goto finish; 
        
               } 
        
               if (hret == HITLS_SUCCESS || coap_hitls_is_retry(hret)) { 
        
                 ret = -1; 
        
                 goto finish; 
        
               } 
        
               session->dtls_event = coap_hitls_is_closed(hret) ? 
        
                                     COAP_EVENT_DTLS_CLOSED : COAP_EVENT_DTLS_ERROR; 
        
               if (session->dtls_event == COAP_EVENT_DTLS_ERROR) { 
        
                 coap_log_warn("coap_dtls_receive: returned 0x%x (length %" PRIdS ")\n", 
        
                               (unsigned int)hret, data_len); 
        
               } 
        
             } else { 
        
               /* 
        
                * On completion coap_hitls_set_connected() drives l_establish(); on error 
        
                * session->dtls_event is set and handled below, so the handshake return 
        
                * value is not needed here. 
        
                */ 
        
               (void)coap_hitls_handshake(session, env); 
        
               ret = -1; 
        
             } 
        
             if (session->dtls_event >= 0) { 
        
               coap_handle_event_lkd(session->context, session->dtls_event, session); 
        
               if (session->dtls_event == COAP_EVENT_DTLS_ERROR || 
        
                   session->dtls_event == COAP_EVENT_DTLS_CLOSED) { 
        
                 coap_session_disconnected_lkd(session, COAP_NACK_TLS_FAILED); 
        
                 ret = -1; 
        
               } 
        
             } 
        
           finish: 
        
             coap_hitls_clear_pdu(coap_hitls_live_env(session, env)); 
        
             return ret;

DTLS hello path

libcoap/src/coap_openhitls.c

Lines 2460 to 2483 in e4b7b1b

    
             if (coap_hitls_is_retry(ret)) { 
        
               /* HITLS_Listen emits HelloVerifyRequest; Accept flushes it. */ 
        
               ret = coap_hitls_handshake(session, env); 
        
               if (ret < 0) 
        
                 return -1; 
        
               env = coap_hitls_live_env(session, env); 
        
               if (!env) 
        
                 return -1; 
        
               env->hello_verify_sent = 1; 
        
               return 0; 
        
             } 
        
             coap_log_warn("coap_dtls_hello: returned 0x%x\n", (unsigned int)ret); 
        
             session->dtls_event = COAP_EVENT_DTLS_ERROR; 
        
             return -1; 
        
           } 
        
           cookie_valid = coap_hitls_client_hello_cookie_valid(session, data, data_len); 
        
           ret = coap_hitls_handshake(session, env); 
        
           env = coap_hitls_live_env(session, env); 
        
           coap_hitls_clear_pdu(env); 
        
           if (ret < 0 || !env) 
        
             return -1;

tlhc · 2026-06-17T10:38:51Z

@mrdeep1 Add some tests, pls take a look thanks

mrdeep1

Thanks for doing all this work.

I have done an initial review of the code.

In general all looks to be fine, but I need to do some general interop between different TLS libraries type tests.

My real concern is the use of =DIR (or providing directory hints) to a openHiTLS build environment that has not had the libraries / include files installed like the other TLS library backends. I think this needs to be changed so things are consistent with the other backends.

It would be nice if openHiTLS provided a pkg-config .pc file at some point.

mrdeep1 · 2026-06-17T11:26:42Z

+    "${OPENHITLS_INCLUDE_DIR}"
+    "${OPENHITLS_INCLUDE_DIR}/bsl"
+    "${OPENHITLS_INCLUDE_DIR}/crypto")


There is no need for the additional /tls, /bsl, /crypto and /pki suffixes as src/coap_openhitls.c has them in the #include statements.

Same for all the instances that follow.

libcoap sources already use the prefix
The per-module dirs are still listed because the installed openHiTLS headers include sibling-module headers by bare name（without prefix）
when openHiTLS upstream fixed install path, we remove this submodule include

mrdeep1 · 2026-06-17T12:23:41Z

      else
          AC_MSG_ERROR([==> Option '--enable-dtls' is set but none of the needed cryptography libraries GnuTLS, OpenSSL, wolfSSL, Mbed TLS or TinyDTLS could be found!
-                        Install at least one of the package(s) that contains the development files for GnuTLS (>= $gnutls_version_required), OpenSSL(>= $openssl_version_required), wolfSSL(>= $wolfssl_version_required), Mbed TLS(>= $mbedtls_version_required), or TinyDTLS(>= $tinydtls_version_required)
+                        Install at least one of the package(s) that contains the development files for GnuTLS (>= $gnutls_version_required), OpenSSL(>= $openssl_version_required), wolfSSL(>= $wolfssl_version_required), Mbed TLS(>= $mbedtls_version_required), or TinyDTLS(>= $tinydtls_version_required), explicitly select openHiTLS using '--with-openhitls',


I would like to see openHiTLS hving the develop environment as well.

add auto fallback for dtls backend detection

mrdeep1 · 2026-06-17T13:40:29Z

+t_tls3(void) {
+  const coap_tls_version_t *v = coap_get_tls_library_version();
+  t_tls_support_t support = t_tls_expected_support();
+
+  ReturnIf_CU_ASSERT_PTR_NOT_NULL(v);
+  CU_ASSERT(v->type == support.type);
+  CU_ASSERT(coap_dtls_is_supported() == support.dtls);
+  CU_ASSERT(coap_tls_is_supported() == support.tls);
+  CU_ASSERT(coap_dtls_psk_is_supported() == support.psk);
+  CU_ASSERT(coap_dtls_pki_is_supported() == support.pki);
+  CU_ASSERT(coap_dtls_pkcs11_is_supported() == support.pkcs11);
+  CU_ASSERT(coap_dtls_rpk_is_supported() == support.rpk);
+  CU_ASSERT(coap_dtls_cid_is_supported() == support.cid);


Depending on the version of the TLS library, not all the options are available. For example, OpenSSL 4.0 does not (yet) have PKCS11 support, but OpenSSL 3.0 does which has ENGINE support.

Not sure we should be doing this set of testing unless there are wrappers as needed for the different library versions.

removed this cycle tests

mrdeep1 · 2026-06-17T15:55:47Z

These changes in
patch3.txt
need to be applied to allow the test suite to work across different TLS backends with different build options.

tlhc · 2026-06-18T04:30:49Z

My real concern is the use of =DIR (or providing directory hints) to a openHiTLS build environment that has not had the libraries / include files installed like the other TLS library backends. I think this needs to be changed so things are consistent with the other backends.

@mrdeep1 Thank you for your review. I’ve checked the issues and will fix them one by one.

mrdeep1 · 2026-06-22T11:58:36Z

Thanks for doing all this work.

I would like to see
patch4.txt added as this then makes it work across all the TLS libraries.

I am continuing to check things.

tlhc · 2026-06-23T03:26:12Z

Thanks for doing all this work.

I would like to see patch4.txt added as this then makes it work across all the TLS libraries.

I am continuing to check things.

OK, done, applied this patch

tlhc · 2026-06-23T04:08:30Z

These changes in patch3.txt need to be applied to allow the test suite to work across different TLS backends with different build options.

applied

mrdeep1

Thanks for doing all this work.

A couple of other changes required.

I'm still doing ongoing testing.

mrdeep1 · 2026-06-24T13:43:21Z

+#if defined(_WIN32)
+      *(int32_t *)parg = -1;
+#else /* ! _WIN32 */
+      *(int32_t *)parg = (int32_t)env->session->sock.fd;


Unfortunately, this needs to be

+#if COAP_SERVER_SUPPORT + *(int32_t *)parg = COAP_PROTO_NOT_RELIABLE(env->session->proto) ? env->session->type != + COAP_SESSION_TYPE_CLIENT ? env->session->endpoint->sock.fd : + env->session->sock.fd : env->session->sock.fd; +#else /* ! COAP_SERVER_SUPPORT */ *(int32_t *)parg = (int32_t)env->session->sock.fd; +#endif /* ! COAP_SERVER_SUPPORT */

as server UDP sockets hang off the endpoints, not the sessions.

Done as suggested

mrdeep1 · 2026-06-24T14:31:25Z

+  version.version = BSL_LOG_GetVersionNum();
+  version.built_version = OPENHITLS_VERSION_I;


openHiTLS CMakeLists.txt does a

set(OPENHITLS_VERSION_I "4194304" CACHE STRING "openHiTLS version integer")

which overrides the definition in include/bsl/bsl_log.h when building the code, but when compiling libcoap, this hand coded definition override is not there. This should be in one place. openHiTLS issue raised.

This is a note for now.

Resolved in openHiTLS 0a7d5a37

mrdeep1 · 2026-06-25T09:43:28Z

+  if((OSCORE_BACKEND STREQUAL "openhitls") OR ((DTLS_BACKEND STREQUAL "default") AND (NOT DTLS_FOUND)))
+    # libopenhitls
+    find_package(openHiTLS ${DTLS_REQUIRED})
+    if(openHiTLS_FOUND)
+      set(WITH_OPENHITLS ON)
+      message(STATUS "compiling with openHiTLS support")
+      set(COAP_WITH_LIBOPENHITLS 1)
+      set(DTLS_FOUND ON)
+    endif()
+  endif()
+


With the latest changes in the libcoap develop branch, this is no longer needed.

syncd with libcoap upstream dev, removed

mrdeep1 · 2026-06-25T09:44:33Z

+  if((OSCORE_BACKEND STREQUAL "openhitls") AND (NOT DTLS_FOUND))
+    find_package(openHiTLS REQUIRED)
+    set(WITH_OPENHITLS_OSCORE ON)
+    message(STATUS "compiling with openHiTLS OSCORE support")
+    set(COAP_WITH_LIBOPENHITLS_OSCORE 1)
+    set(DTLS_FOUND ON)
+  endif()
+


This needs to be

if((OSCORE_BACKEND STREQUAL "openhitls") AND (NOT DTLS_FOUND)) # libopenhitls find_package(openHiTLS REQUIRED) if(openHiTLS_FOUND) set(WITH_OPENHITLS_OSCORE ON) message(STATUS "compiling with openHiTLS OSCORE support") set(COAP_WITH_LIBOPENHITLS_OSCORE 1) set(DTLS_FOUND ON) endif() endif()

OK, add checks

mrdeep1 · 2026-06-25T10:43:15Z

The latest update to the openHiTLS main branch replaces BSL_LOG_GetVersionNum() with HITLS_VersionNum(), along with a new file bsl/bls_version.h so code will need to be updated to reflect this.

tlhc · 2026-06-25T11:43:29Z

The latest update to the openHiTLS main branch replaces BSL_LOG_GetVersionNum() with HITLS_VersionNum(), along with a new file bsl/bls_version.h so code will need to be updated to reflect this.

Update with the new version check API

https://github.com/obgm/libcoap/compare/396cc553353b27b3d30d3e949e17d10783a481dc..86126804816bf3e6f5493020c56c0564c70c718c

mrdeep1 · 2026-06-25T14:27:27Z

Thanks for all this. I will run some overnight regression tests to check that all is OK before merging.

tlhc · 2026-06-25T14:54:02Z

Thanks for all this. I will run some overnight regression tests to check that all is OK before merging.

Sounds good, and thank you for taking the time to verify everything.

mrdeep1 · 2026-06-26T08:41:15Z

It would be good to add this change,
patch5.txt
to remove a lot of the unnecessary -I include paths.

tlhc · 2026-06-26T12:25:46Z

It would be good to add this change, patch5.txt to remove a lot of the unnecessary -I include paths.

Done, Applied this patch, looks good and clean

- Add openHiTLS DTLS/TLS backend - Integrate openHiTLS crypto into libcoap (D)TLS and OSCORE paths. - Extend TLS tests and add interop validation script.

Copilot AI review requested due to automatic review settings June 16, 2026 05:00

Copilot AI reviewed Jun 16, 2026

View reviewed changes

Comment thread src/coap_oscore.c

Comment thread tests/test_tls.c Outdated

Comment thread tests/test_tls.c

Comment thread src/coap_openhitls.c Outdated

Comment thread configure.ac Outdated

Comment thread examples/tls_backend_testcases.sh Outdated

tlhc force-pushed the develop branch from 089d0f8 to 6357cdc Compare June 16, 2026 06:12

tlhc closed this Jun 16, 2026

tlhc reopened this Jun 16, 2026

tlhc force-pushed the develop branch 3 times, most recently from 7407365 to 140aa5f Compare June 16, 2026 09:41

tlhc force-pushed the develop branch from 140aa5f to 040038f Compare June 16, 2026 14:57

tlhc force-pushed the develop branch 3 times, most recently from 13062c4 to e4b7b1b Compare June 17, 2026 05:17

tlhc force-pushed the develop branch from e4b7b1b to 43e3f71 Compare June 17, 2026 10:33

mrdeep1 requested changes Jun 17, 2026

View reviewed changes

mrdeep1 reviewed Jun 18, 2026

View reviewed changes

Comment thread src/coap_openhitls.c Outdated

mrdeep1 reviewed Jun 18, 2026

View reviewed changes

Comment thread src/coap_openhitls.c Outdated

tlhc force-pushed the develop branch from 43e3f71 to 8880cd1 Compare June 19, 2026 11:22

tlhc force-pushed the develop branch from 8880cd1 to 99ed067 Compare June 23, 2026 03:24

tlhc force-pushed the develop branch from 99ed067 to 3c52f9f Compare June 23, 2026 04:06

tlhc force-pushed the develop branch 7 times, most recently from e25ad8f to 396cc55 Compare June 25, 2026 03:59

mrdeep1 requested changes Jun 25, 2026

View reviewed changes

tlhc force-pushed the develop branch from 396cc55 to 8612680 Compare June 25, 2026 11:41

tlhc force-pushed the develop branch 2 times, most recently from b30698b to f780fba Compare June 25, 2026 13:52

tlhc force-pushed the develop branch 2 times, most recently from 53176cf to 37fd3e4 Compare June 26, 2026 07:14

tlhc force-pushed the develop branch from 37fd3e4 to 77527f7 Compare June 26, 2026 12:24

tlhc force-pushed the develop branch from 77527f7 to dc31b65 Compare June 26, 2026 12:32

mrdeep1 requested changes Jun 26, 2026

View reviewed changes

Comment thread src/coap_openhitls.c Outdated

feat: add openHiTLS backend support

3c9084b

- Add openHiTLS DTLS/TLS backend - Integrate openHiTLS crypto into libcoap (D)TLS and OSCORE paths. - Extend TLS tests and add interop validation script.

tlhc force-pushed the develop branch from dc31b65 to 3c9084b Compare June 27, 2026 03:36

mrdeep1 merged commit aa68d5f into obgm:develop Jun 27, 2026
53 checks passed

		version.version = BSL_LOG_GetVersionNum();
		version.built_version = OPENHITLS_VERSION_I;

Conversation

tlhc commented Jun 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tlhc commented Jun 16, 2026

Uh oh!

tlhc commented Jun 16, 2026

Uh oh!

mrdeep1 commented Jun 16, 2026

Uh oh!

tlhc commented Jun 16, 2026

Uh oh!

mrdeep1 commented Jun 16, 2026

Uh oh!

tlhc commented Jun 16, 2026

Uh oh!

mrdeep1 commented Jun 16, 2026

Uh oh!

tlhc commented Jun 16, 2026

Uh oh!

tlhc commented Jun 17, 2026

Uh oh!

tlhc commented Jun 17, 2026

Uh oh!

tlhc commented Jun 17, 2026

Uh oh!

mrdeep1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mrdeep1 commented Jun 17, 2026

Uh oh!

tlhc commented Jun 18, 2026

Uh oh!

Uh oh!

Uh oh!

mrdeep1 commented Jun 22, 2026

Uh oh!

tlhc commented Jun 23, 2026

Uh oh!

tlhc commented Jun 23, 2026

Uh oh!

mrdeep1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

tlhc commented Jun 25, 2026 •

edited

Loading