I started to configure Package.swift for libgit2 yesterday.

It seems that this approach is simpler than Xcode project, because ( at least ) generated Xcode project has some troubles with address sanitizer ( even if CMake said that they are fixed this issue ). Also it can't work with .xcframework format.

This file can be generated from CMakeLists ( easily ), because it contains all directives ( headerSearchPaths, defines, linker flags ) and also it contains exclude files list.
So, instead of removing ( not adding ) files, you simply tell to it to ignore some files.

Seems promising ( and readable ) approach. All settings are in the one file.
Yes, it may be very "complicated" for conditions, but it can be generated by CMake.

I wish that CMake will go this route, but maybe it is better to add an example Package.swift to this project? No matter how simple it would be ( only macOS 11.4 and higher, for example ).