Mongod hard-crashes exactly every 30 seconds (SIGSEGV)

[Borketh]

Genuinely lost for debug steps and couldn't find another place to post. Been having connection issues with mongodb and it turned out to be because the whole damn thing is crashing horribly without logging anything. I'm not sure where I could find debug symbols, so I can't find anything further. The crash happens after exactly 30 seconds of uptime, then the container is restarted due to my docker settings.

Happens on 8.0 and 8.2

I have read #485 and it doesn't appear to be related.

Hardware info if relevant:

OS: CachyOS x86_64
Host: GF63 Thin 11UC (REV:1.0)
Kernel: Linux 6.19.2-2-cachyos
Uptime: 20 hours, 28 mins
Packages: 1628 (pacman)
Shell: fish 4.5.0
Terminal: konsole 25.12.2
CPU: 11th Gen Intel(R) Core(TM) i5-11400H (12) @ 4.50 GHz
GPU 1: NVIDIA GeForce RTX 3050 Mobile [Discrete]
GPU 2: Intel UHD Graphics @ 1.45 GHz [Integrated]
Memory: 10.08 GiB / 31.05 GiB (32%)

Versions used:

floof@toast/v/l/s/coredump $ sudo docker exec -it mongodb /usr/bin/mongod --version
db version v8.0.19
Build Info: {
    "version": "8.0.19",
    "gitVersion": "cc1adb6b0875cc3003854ac2489818e459686f0b",
    "openSSLVersion": "OpenSSL 3.0.13 30 Jan 2024",
    "modules": [],
    "allocator": "tcmalloc-google",
    "environment": {
        "distmod": "ubuntu2404",
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}
floof@toast/v/l/s/coredump $ sudo docker exec -it mongodb /usr/bin/mongod --version
db version v8.2.5
Build Info: {
    "version": "8.2.5",
    "gitVersion": "a471a13094434666c48a1f75451f2efa49f8f5df",
    "openSSLVersion": "OpenSSL 3.0.13 30 Jan 2024",
    "modules": [],
    "allocator": "tcmalloc-google",
    "environment": {
        "distmod": "ubuntu2404",
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}

[tianon]

This is a wild one, especially that it seems to be exactly 30 seconds. Have you tried running with something like strace on mongod to narrow it down?

In the image, we don't do much more than install upstream's pre-built binaries and run them, so I don't think this failure is specific to the image, but it's possible!

[MS-BRT]

I encountered the same issue and did some research:

• It only affects the new Linux kernel, version 6.19.
• It only affects mongo Docker images 8.0.5 and later.

While reviewing the changes from 8.0.4 to 8.0.5, I noticed that RSEQ was disabled for glibc / enabled for Google TCMalloc in 7bf2228. It was also disabled from the beginning in 8.2 versions, starting with 9306458.
Since Linux 6.19 includes an overhaul of RSEQ, I figured it may be the cause of the issue and re-enabled RSEQ for glibc by changing the GLIBC_TUNABLES environment variable to glibc.pthread.rseq=1 when running a container with an affected version. The container has been running for four hours now without any issues.

It would be helpful if someone could try to reproduce this issue with MongoDB running outside of a Docker container.

For now, a temporary fix is to run the mongo Docker container with: -e GLIBC_TUNABLES=glibc.pthread.rseq=1.

[yosifkit]

I'm glad you figured out a workaround for the crashing. This sounds like a bug in TCMalloc, MongoDB, or perhaps glibc or the kernel (in order of likelihood). I don't think there is anything we should do in the image as the GLIBC_TUNABLES setting was added based on MongoDB documentation. Since it does cause it to crash, I'd suggest filing an issue with MongoDB and let them figure out what they need to change/update.

[kralos]

Hi everyone, I started an issue on the mongod tracker: https://jira.mongodb.org/browse/SERVER-120238

Another workaround: --env GLIBC_TUNABLES="glibc.cpu.hwcaps=-SHSTK"

[Lippiece]

I also have this problem, but I have Intel's i5-9300H. The workaround fixes it, so it must be unrelated to AMD CPUs.

[kralos]

a bit of an update - https://jira.mongodb.org/browse/SERVER-120238?focusedCommentId=8199578&focusedId=8199578&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-8199578

it looks like --env GLIBC_TUNABLES="glibc.cpu.hwcaps=-SHSTK" is simply wiping out GLIBC_TUNABLES=glibc.pthread.rseq=0 which is baked into the docker image. See:

kralos@strix:~$ docker image inspect mongo | jq '.[].Config.Env'
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "GOSU_VERSION=1.19",
  "JSYAML_VERSION=3.13.1",
  "JSYAML_CHECKSUM=662e32319bdd378e91f67578e56a34954b0a2e33aca11d70ab9f4826af24b941",
  "MONGO_PACKAGE=mongodb-org",
  "MONGO_REPO=repo.mongodb.org",
  "MONGO_MAJOR=8.2",
  "MONGO_VERSION=8.2.5",
  "HOME=/data/db",
  "GLIBC_TUNABLES=glibc.pthread.rseq=0"
]
kralos@strix:~$ docker image inspect mongodb/mongodb-community-server | jq '.[].Config.Env'
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "HOME=/data/db",
  "GLIBC_TUNABLES=glibc.pthread.rseq=0"
]
kralos@strix:~$

By setting docker run --env GLIBC_TUNABLES="glibc.cpu.hwcaps=-SHSTK"

I've effectively unset glibc.pthread.rseq=0, which defaults to 1 - https://sourceware.org/glibc/manual/2.43/html_mono/libc.html#index-glibc_002epthread_002erseq

I've checked the effective env inside the container and as you might expect, setting -SHSTK implicitly unsets the docker image's rseq=0:

kralos@strix:~$ docker run --rm mongo env | grep GLIBC
GLIBC_TUNABLES=glibc.pthread.rseq=0
kralos@strix:~$ docker run --rm --env GLIBC_TUNABLES="glibc.cpu.hwcaps=-SHSTK" mongo env | grep GLIBC
GLIBC_TUNABLES=glibc.cpu.hwcaps=-SHSTK
kralos@strix:~$ docker run --rm mongodb/mongodb-community-server env | grep GLIBC
GLIBC_TUNABLES=glibc.pthread.rseq=0
kralos@strix:~$ docker run --rm --env GLIBC_TUNABLES="glibc.cpu.hwcaps=-SHSTK" mongodb/mongodb-community-server env | grep GLIBC
GLIBC_TUNABLES=glibc.cpu.hwcaps=-SHSTK
kralos@strix:~$  

So I re-ran some quick tests:

• GLIBC_TUNABLES="glibc.cpu.hwcaps=-SHSTK:glibc.pthread.rseq=0" crashes 8.0.0 onwards
• GLIBC_TUNABLES="glibc.pthread.rseq=1" allows all versions to run forever
• GLIBC_TUNABLES="glibc.pthread.rseq=0" (or don't specify --env to docker) crashes 8.0.5 onwards

kralos@strix:~$ for VER in "8.2" "8.0.9" "8.0.8" "8.0.6" "8.0.5" "8.0.4" "8.0.3" "8.0.1" "8.0.0" "7.0" "6.0" "5.0"; do
    docker run -d --env GLIBC_TUNABLES="glibc.cpu.hwcaps=-SHSTK:glibc.pthread.rseq=0" --name="mongo-$VER" "mongo:$VER"
done
67200a16ded3929084bc7a5bb4a97d562481ae0218a92fa1fce1f591979ad5a2
c450605876fceb91e516e77c4905fcacb5660ad3bca241bd7aaa9aca574d3e7a
519ca78fce6c92ff826e192769241eeb464c5df419fb3f674a5822e4dc6ad9f5
b85d4c6aa590fe606431cd2a444043c62b02d1421c1d93df4b0008dc18d074f8
9dbc543d68d96bad91d115ef02f8c3283250e5bc70994434d090c4cb2bccfa64
acb79d20a89d3c63c7fc25159ba41fe0c9ffcf5837d6efeb05830cb4427752a0
f6f28c21a2b1d186e0c3c1f2828b64b29dd5d715bb30e96086930b9b03dab4b6
4853e623ecc30593639f8324946433041cc88d76999f0cbc2278994c98467cb8
280a061c4ebe9c21fc1b06db35bd8a46467a3636d6eb3c7743adcf6b1053a81b
5d59261680b6958a875b45fb1051b5856ca7080f104d90be07f1f5d24f7b7c6c
79ec3e08ed1dc8bd06c910eaea56001a85b5fc7c06c2fb7ba52213055b91f737
e8eced5d7fa8ea04e0915d1f1aa0a85448ba75c58af798329445d586460aa11e
kralos@strix:~$ docker ps -a
CONTAINER ID   IMAGE                 COMMAND                  CREATED          STATUS                        PORTS       NAMES
e8eced5d7fa8   mongo:5.0             "docker-entrypoint.s…"   57 seconds ago   Up 56 seconds                 27017/tcp   mongo-5.0
79ec3e08ed1d   mongo:6.0             "docker-entrypoint.s…"   57 seconds ago   Up 56 seconds                 27017/tcp   mongo-6.0
5d59261680b6   mongo:7.0             "docker-entrypoint.s…"   57 seconds ago   Up 56 seconds                 27017/tcp   mongo-7.0
280a061c4ebe   mongo:8.0.0           "docker-entrypoint.s…"   57 seconds ago   Exited (139) 27 seconds ago               mongo-8.0.0
4853e623ecc3   mongo:8.0.1           "docker-entrypoint.s…"   57 seconds ago   Exited (139) 27 seconds ago               mongo-8.0.1
f6f28c21a2b1   mongo:8.0.3           "docker-entrypoint.s…"   57 seconds ago   Exited (139) 27 seconds ago               mongo-8.0.3
acb79d20a89d   mongo:8.0.4           "docker-entrypoint.s…"   57 seconds ago   Exited (139) 27 seconds ago               mongo-8.0.4
9dbc543d68d9   mongo:8.0.5           "docker-entrypoint.s…"   58 seconds ago   Exited (139) 27 seconds ago               mongo-8.0.5
b85d4c6aa590   mongo:8.0.6           "docker-entrypoint.s…"   58 seconds ago   Exited (139) 27 seconds ago               mongo-8.0.6
519ca78fce6c   mongo:8.0.8           "docker-entrypoint.s…"   58 seconds ago   Exited (139) 28 seconds ago               mongo-8.0.8
c450605876fc   mongo:8.0.9           "docker-entrypoint.s…"   58 seconds ago   Exited (139) 28 seconds ago               mongo-8.0.9
67200a16ded3   mongo:8.2             "docker-entrypoint.s…"   58 seconds ago   Exited (139) 28 seconds ago               mongo-8.2
kralos@strix:~$ 
for VER in "8.2" "8.0.9" "8.0.8" "8.0.6" "8.0.5" "8.0.4" "8.0.3" "8.0.1" "8.0.0" "7.0" "6.0" "5.0"; do
    docker stop "mongo-$VER"
    docker rm -v "mongo-$VER"
done
mongo-8.2
mongo-8.2
mongo-8.0.9
mongo-8.0.9
mongo-8.0.8
mongo-8.0.8
mongo-8.0.6
mongo-8.0.6
mongo-8.0.5
mongo-8.0.5
mongo-8.0.4
mongo-8.0.4
mongo-8.0.3
mongo-8.0.3
mongo-8.0.1
mongo-8.0.1
mongo-8.0.0
mongo-8.0.0
mongo-7.0
mongo-7.0
mongo-6.0
mongo-6.0
mongo-5.0
mongo-5.0
kralos@strix:~$ docker run --rm --env GLIBC_TUNABLES="glibc.cpu.hwcaps=-SHSTK:glibc.pthread.rseq=0" mongo env | grep GLIBC
GLIBC_TUNABLES=glibc.cpu.hwcaps=-SHSTK:glibc.pthread.rseq=0
kralos@strix:~$ for VER in "8.2" "8.0.9" "8.0.8" "8.0.6" "8.0.5" "8.0.4" "8.0.3" "8.0.1" "8.0.0" "7.0" "6.0" "5.0"; do
    docker run -d --env GLIBC_TUNABLES="glibc.pthread.rseq=1" --name="mongo-$VER" "mongo:$VER"
done
7601dc8221beac404d5711616cbf2bce721f15e3220f32ea9ad4e3ca61bb7168
77f4900396e8a9f03a9ea1579b3243cff961e9fda8ee71e7aeca50c9b05522eb
fd90d1eac456b01a80b0f51dcae8a1d591282236764d8ec5833e9d5126672d04
ac8226046102a58f82969257d787cdf1a1581e56bbb52c0677e3b539a54e40ed
7f68a10f52571611c3e41c7feeb2bc75d67e492e23c157cd67f0e633b73ac863
04c1f4136b80711eec47b110d233a2890007efb0056323fc68d9477b38e63e30
8a2a4fbe679863b1d8a63dba56e6b598fabed23c99165416c2a7b0386bfcc1aa
3f61f08869093af5db5a4d253c89396238e066a52121712288ebc84c39fdc603
15a8051d9e3f65c5023913b107faa76f774ec79fcf680744ba2a329e5203b387
9a428715f5c5e379f559703cf114b969eb4d96ae5b573c0f63dbd8b7abbbaf69
d5000f8a8f54cfb730a360eb50a6a17418372c44e88567e0005b94a009af0fb0
71161661b3819a716218c5ef7d69be6996cf1a1fdc0a2b625b56d68cd2257d39
kralos@strix:~$ docker ps -a
CONTAINER ID   IMAGE                 COMMAND                  CREATED          STATUS          PORTS       NAMES
71161661b381   mongo:5.0             "docker-entrypoint.s…"   43 seconds ago   Up 42 seconds   27017/tcp   mongo-5.0
d5000f8a8f54   mongo:6.0             "docker-entrypoint.s…"   43 seconds ago   Up 42 seconds   27017/tcp   mongo-6.0
9a428715f5c5   mongo:7.0             "docker-entrypoint.s…"   43 seconds ago   Up 43 seconds   27017/tcp   mongo-7.0
15a8051d9e3f   mongo:8.0.0           "docker-entrypoint.s…"   43 seconds ago   Up 43 seconds   27017/tcp   mongo-8.0.0
3f61f0886909   mongo:8.0.1           "docker-entrypoint.s…"   43 seconds ago   Up 43 seconds   27017/tcp   mongo-8.0.1
8a2a4fbe6798   mongo:8.0.3           "docker-entrypoint.s…"   43 seconds ago   Up 43 seconds   27017/tcp   mongo-8.0.3
04c1f4136b80   mongo:8.0.4           "docker-entrypoint.s…"   44 seconds ago   Up 43 seconds   27017/tcp   mongo-8.0.4
7f68a10f5257   mongo:8.0.5           "docker-entrypoint.s…"   44 seconds ago   Up 43 seconds   27017/tcp   mongo-8.0.5
ac8226046102   mongo:8.0.6           "docker-entrypoint.s…"   44 seconds ago   Up 43 seconds   27017/tcp   mongo-8.0.6
fd90d1eac456   mongo:8.0.8           "docker-entrypoint.s…"   44 seconds ago   Up 43 seconds   27017/tcp   mongo-8.0.8
77f4900396e8   mongo:8.0.9           "docker-entrypoint.s…"   44 seconds ago   Up 44 seconds   27017/tcp   mongo-8.0.9
7601dc8221be   mongo:8.2             "docker-entrypoint.s…"   44 seconds ago   Up 44 seconds   27017/tcp   mongo-8.2
kralos@strix:~$ 
for VER in "8.2" "8.0.9" "8.0.8" "8.0.6" "8.0.5" "8.0.4" "8.0.3" "8.0.1" "8.0.0" "7.0" "6.0" "5.0"; do
    docker stop "mongo-$VER"
    docker rm -v "mongo-$VER"
done
mongo-8.2
mongo-8.2
mongo-8.0.9
mongo-8.0.9
mongo-8.0.8
mongo-8.0.8
mongo-8.0.6
mongo-8.0.6
mongo-8.0.5
mongo-8.0.5
mongo-8.0.4
mongo-8.0.4
mongo-8.0.3
mongo-8.0.3
mongo-8.0.1
mongo-8.0.1
mongo-8.0.0
mongo-8.0.0
mongo-7.0
mongo-7.0
mongo-6.0
mongo-6.0
mongo-5.0
mongo-5.0
kralos@strix:~$ for VER in "8.2" "8.0.9" "8.0.8" "8.0.6" "8.0.5" "8.0.4" "8.0.3" "8.0.1" "8.0.0" "7.0" "6.0" "5.0"; do
    docker run -d --name="mongo-$VER" "mongo:$VER"
done
53a45bd4b4d146cdf0c4de023bede57b8c558af9ae68495fcbdec699b3a24668
e01c953032c0e2aa3cb01303da29c21b67bb32a4d5b651abcffff67dc67e38ff
d5ec85d0c35f5f3f7ab8662b811b230649faeccfefa1c8bd6c4b810938d713bf
190c962ed8e881b3a1b69d517fd7b87ff2cb031da592866713580a15fc8c09ba
c5ddb1c6be1e5f1097f71f472fdbb1b84c1daadf55e95cda6ee2ec42fa3a5f08
91e42d65cbd358ae2e7bd63a073119295dcd040b6585a4f107d0b2355d2dcf9e
cd68137a44147dfed0a05a8d3d41e183f6b69b52ee6c0085f7645a571dadf8ab
143d40f8089bdf459ea8042d69c61c6779a0fcff125c679728ef6448da49b9fa
f92e9bc7946d97cb18b7e166ef9e94294f0d05d0e23d13ff5af40eb7d7efa650
e3c0c0e9d6bcfa6bfdebdaa7581a73f7db2d3a0f29f9ebab8d010e5a08c3ad5b
95db71dbf27441d61169e73996a8563ac15af8e77388d1cb20a9aa6068e7e075
6a2a778429c9359c95c0552d9afe425a27ba2159a320fdc630534afddb869774
kralos@strix:~$ docker ps -a
CONTAINER ID   IMAGE                 COMMAND                  CREATED          STATUS                       PORTS       NAMES
6a2a778429c9   mongo:5.0             "docker-entrypoint.s…"   37 seconds ago   Up 36 seconds                27017/tcp   mongo-5.0
95db71dbf274   mongo:6.0             "docker-entrypoint.s…"   37 seconds ago   Up 36 seconds                27017/tcp   mongo-6.0
e3c0c0e9d6bc   mongo:7.0             "docker-entrypoint.s…"   37 seconds ago   Up 36 seconds                27017/tcp   mongo-7.0
f92e9bc7946d   mongo:8.0.0           "docker-entrypoint.s…"   37 seconds ago   Up 36 seconds                27017/tcp   mongo-8.0.0
143d40f8089b   mongo:8.0.1           "docker-entrypoint.s…"   37 seconds ago   Up 36 seconds                27017/tcp   mongo-8.0.1
cd68137a4414   mongo:8.0.3           "docker-entrypoint.s…"   37 seconds ago   Up 36 seconds                27017/tcp   mongo-8.0.3
91e42d65cbd3   mongo:8.0.4           "docker-entrypoint.s…"   37 seconds ago   Up 36 seconds                27017/tcp   mongo-8.0.4
c5ddb1c6be1e   mongo:8.0.5           "docker-entrypoint.s…"   37 seconds ago   Exited (139) 7 seconds ago               mongo-8.0.5
190c962ed8e8   mongo:8.0.6           "docker-entrypoint.s…"   38 seconds ago   Exited (139) 7 seconds ago               mongo-8.0.6
d5ec85d0c35f   mongo:8.0.8           "docker-entrypoint.s…"   38 seconds ago   Exited (139) 7 seconds ago               mongo-8.0.8
e01c953032c0   mongo:8.0.9           "docker-entrypoint.s…"   38 seconds ago   Exited (139) 8 seconds ago               mongo-8.0.9
53a45bd4b4d1   mongo:8.2             "docker-entrypoint.s…"   38 seconds ago   Exited (139) 8 seconds ago               mongo-8.2
kralos@strix:~$ 
for VER in "8.2" "8.0.9" "8.0.8" "8.0.6" "8.0.5" "8.0.4" "8.0.3" "8.0.1" "8.0.0" "7.0" "6.0" "5.0"; do
    docker stop "mongo-$VER"
    docker rm -v "mongo-$VER"
done
mongo-8.2
mongo-8.2
mongo-8.0.9
mongo-8.0.9
mongo-8.0.8
mongo-8.0.8
mongo-8.0.6
mongo-8.0.6
mongo-8.0.5
mongo-8.0.5
mongo-8.0.4
mongo-8.0.4
mongo-8.0.3
mongo-8.0.3
mongo-8.0.1
mongo-8.0.1
mongo-8.0.0
mongo-8.0.0
mongo-7.0
mongo-7.0
mongo-6.0
mongo-6.0
mongo-5.0
mongo-5.0
kralos@strix:~$