Permission issues on docker swarm with NFS (Operation not permitted)

[vhaudiquet]

Hello!

I have an issue with recent mongodb images.

I have a setup that might be a bit weird: docker swarm deploys mongo with a volume that uses the local driver, that points to a folder that is actually a mounted NFS share. The stack file is as follows:

...
  mongo:
    hostname: mongo
    image: mongo:8.0.15
    restart: always
    user: 2000:2000 # this is recent, I tried with no user, user 999:999, root:root and 2000:2000
    volumes:
      - mongo_data:/data/db:Z
      - mongo_data:/data/configdb:Z
    env_file: .env
...

volumes:
  mongo_data:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/app/buildpath/data/_data'

/app is a mounted NFS share, to my NAS.

With mongo:8.0.15, the container starts and is healthy, but all write transactions fail. For example:

{"t":{"$date":"2025-12-06T13:16:20.040+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"conn28","msg":"createCollection","attr":{"namespace":"matches.15.24.1","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"76316150-f940-49f8-b319-a6290ce27a38"}},"options":{}}}
{"t":{"$date":"2025-12-06T13:16:20.047+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"conn28","msg":"WiredTiger error message","attr":{"error":1,"message":{"ts_sec":1765026980,"ts_usec":47630,"thread":"1:0x7f31565356c0","session_name":"WT_SESSION.create","category":"WT_VERB_DEFAULT","log_id":1000000,"category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:924:/data/db/collection-1-4269577862602222102.wt: handle-open: open","error_str":"Operation not permitted","error_code":1}}}

With the more recent image mongo:8.2.2, the container does not even start, as opening the WiredTiger database file fails with 'Operation not permitted', even in read mode.

I tried setting the user to root:root, 999:999 and 2000:2000. I tried changing the owner and permissions of /app/buildpath/data recursively to any of those, it does not seem to change anything.
My NFS share has the following options (server-side):

"/mnt/fast_app_data/docker-homeprod"\
        *(sec=sys,rw,anonuid=0,anongid=0,all_squash,insecure,no_subtree_check)

I'm using a mapall to root, so that might be a problem for mongo as well.

What I don't understand is that it used to work fine not too long ago, and those recent images seem to have changed something. I tried to look through existing issues but could not find anything recent.

Did something change recently? Is my use case not supported at all? Is that an already fixed problem (Sorry then! I could not find the solution through issues and forums)?

Thanks!

[vhaudiquet]

I guess one of the recent change could be d9efcb7 ?

I also noted that some of the files are being reset to root:root with my current config of having mongo run with 2000:2000:

...
-rwxrwxrwx 1 2000 2000 430182400 Nov 24 00:16 collection-0-1197853415329889252.wt
-rwxrwxrwx 1 2000 2000    348160 Nov 24 00:16 collection-0-12554627696672054276.wt
-rw------- 1 root root         0 Dec  6 13:43 collection-0-13988773120443256580.wt
-rwxrwxrwx 1 2000 2000 392589312 Nov 24 00:16 collection-0-14202213368659583766.wt
-rwxrwxrwx 1 2000 2000    196608 Nov 24 00:16 collection-0-1429850162317927748.wt
-rw------- 1 root root         0 Dec  6 11:12 collection-0-15014294931737174972.wt
-rwxrwxrwx 1 2000 2000 814022656 Nov 24 00:16 collection-0-15156747466538672499.wt
...

[vhaudiquet]

I guess the error with 8.0.15 could be that because of the mappall to root, new files created by the mongo container are given root:root owner. But still, I don't understand why then the mapall doesn't apply to allow mongo 2000:2000 to write that file as well...