From unshare man page:

since Linux 3.19 a further restriction applies: the kernel gives permission to call
setgroups(2) only after the GID map (/proc/pid*/gid_map*) has been set. The GID map is
writable by root when setgroups(2) is enabled (i.e., allow, the default), and the GID map
becomes writable by unprivileged processes when setgroups(2) is permanently disabled
(with deny).

This means that "deny" in setgroups is required by the kernel for unprivileged users to write to gid_map.

See also man user_namespaces to get more details.

Note that unshare can also init the maps for your:

$ unshare --user --map-root-user -- cat /proc/self/{uid_map,gid_map,setgroups}
         0       1000          1
         0       1000          1
deny

 $ unshare --user --map-current-user -- cat /proc/self/{uid_map,gid_map,setgroups}
      1000       1000          1
      1000       1000          1
deny

$ unshare --user --map-auto -- cat /proc/self/{uid_map,gid_map,setgroups}
         0     100000      65536
         0     100000      65536
allow
'''

This means that "deny" in setgroups is required by the kernel for unprivileged users to write to gid_map.

Yes, I know unprivileged users need to write "deny" to /proc/<PID>/setgroups to write /proc/<PID>/gid_map. But nsenter use setgroups() to drop extra gid before and/or after enter the user namespace. I mean, we could make a exception that, if there is only one gid map, and outer gid is user's gid itself, then we don't use setgroups() to drop extra gid (there is no need to do that), thus nsenter don't need SetGID permission, such that it is not related whether /proc/<PID>/setgroups is "deny" or "allow".

Note that unshare can also init the maps for your:

For the unshare part, unshare use newgidmap as external helper program, since most system will set CAP_SETGID on it, so that newgidmap can change /proc/<PID>/setgroups back to allow after change /proc/<PID>/gid_map. But I think there might missing one logic in source code of newgidmap that it forget to set setgroups back to allow, when there is only one and user self gid map exist. (As the related issue I point to shadow repo)