PEP 621 specifically left room in the license key, so we don’t need to deprecate it. We only need to deprecate license = {text = ...} and license = {file = ...}, and define meaning to license = "MIT". No new [project] keys are needed.

To allow multiple licenses, we add

license = { names = ... }

Also, I think we should always glob to avoid confusion. So we define

license = { files = ... }

where each string is a glob pattern, and if someone really really need stars in the path (extremely unlikely), they can escape them manually.

PEP 621 specifically left room in the license key, so we don’t need to deprecate it. We only need to deprecate license = {text = ...} and license = {file = ...}, and define meaning to license = "MIT". No new [project] keys are needed.

This was exactly the original approach I took here (see 4871854 ), later revised to split the license-files out into a top-level key while keeping the License-Expression field mapped to the license key, and further revised to split them both (see 88cba0c ). In practice, when considered more carefully, the first, and to a somewhat lesser extent the second approach provided to be distinctly suboptimal for a number of reasons, and quite possibly unworkable (due to the interaction with the dynamic key, depending on exactly how one interprets a rather imprecisely specified passage in PEP 621), while the only downside of the present approach was 1-2 extra top-level keys.

I spent a substantial amount of time carefully considering the alternatives here, and thoroughly address both of these in great detail in the Rejected Ideas section; see the Add expression and files subkeys to table and Define license expression as string value items under the Source metadata license key subsection. If you still have articulatable concerns with this approach, or can find ways around the various issues I've cited with the previous one, it might be better to discuss this on the Discourse thread, where we can all come to a clear consensus before I spend several more hours of my free time changing everything back (and us all having to bikeshed over exactly what to change it back to, heh).

To allow multiple licenses, we add license = { names = ... }

Perhaps I'm grossly misunderstanding your suggestion, but this proposal appears to contradict the underlying goal and basic objective of this PEP—adding SPDX license expressions to Python package metadata—which has been true its conception, has not been (at least as far as I could find) brought up throughout the course of the years-long relevant discussions, and is intrinsically incompatible with the core metadata field License-Expression that it is supposed to map to. Could you explain a little more about what you're suggesting here? Given it seems to be a fairly substantial shift in what this PEP has been proposing to do from the start, and which seemed to have a fairly strong consensus behind it, if you're really sure about it, you might want to bring it up on Discourse first.

Also, I think we should always glob to avoid confusion.

I think I'm the one that's a little confused, heh—could you explain a little more how users having to implicitly know that license.files is a glob is less potentially confusing than them explicitly specifying license-files.glob (or license-files.paths)? Furthermore, per the spec, for the latter tools will unambiguously know if the user mistakenly specify a glob instead of a literal path under the paths key, because it won't match an actual file.

In any case, like the previous, this was the approach I initially implemented here but after a substantial amount of careful thought and experimentation, this ran into significant conceptual and practical issues with, for a few extra characters, the more explicit alternative neatly avoids. I address this issue at length under the Only accept glob patterns and Infer whether paths or globs items in the Source metadata license-files key subsection of the Rejected Ideas section.

if someone really really need stars in the path (extremely unlikely), they can escape them manually.

Actually, this is not the main motivation for this choice, or even one of the primary drivers. The Only accept glob patterns and Infer whether paths or globs items in the Source metadata license-files key subsection of the Rejected Ideas section describes the rationale in detail.

Perhaps I'm grossly misunderstanding your suggestion, but this proposal appears to contradict the underlying goal and basic objective of this PEP—adding SPDX license expressions to Python package metadata—which has been true its conception, has not been (at least as far as I could find) brought up throughout the course of the years-long relevant discussions, and is intrinsically incompatible with the core metadata field License-Expression that it is supposed to map to.

No, you’re right, I was misunderstanding how License-Expression works (I thought it could be specified multiple times to automatically mean “pick one of these”).

I think I'm the one that's a little confused, heh—could you explain a little more how users having to implicitly know that license.files is a glob is less potentially confusing than them explicitly specifying license-files.glob (or license-files.paths)?

It’s not, but there’s no harm for that :) It is extremely uncommon for license files to be put in paths that resemble a glob pattern, so the user does not actually care if this triggers globbing when they only want to specify one file. And if they want globbing, now they have only one key to remember instead of two.

Perhaps this would be a better approach?

1. Add license-files to do glob (no nested keys).
2. Use license = "<spdx-expression>" and deprecate nested keys.

No, you’re right,

No worries, sorry for the confusion. Seems like the mixup might have been over a license identifier vs. a license expression; I do define these in the Terminology section, but is there anything in the PEP that I should clarify here?

It is extremely uncommon for license files to be put in paths that resemble a glob pattern, so the user does not actually care if this triggers globbing when they only want to specify one file.

Indeed; however, as discussed this isn't the (or even a) primary reason for this; sorry for not linking it when I referred to it previously, but I discuss the rationale in detail in the Only accept glob patterns and Infer whether paths or globs items in the Rejected Ideas, and see next item for a summary.

Add license-files to do glob (no nested keys).

Actually, as mentioned, this was also my initial approach, as "flat is better than nested", but it ended up conflicting with several of the behaviors specified so that "errors should never pass silently", especially if "in the face of ambiguity we must resist the temptation to guess", and in the end, I decided "explicit is better than implicit" :)

Use license = "<spdx-expression>" and deprecate nested keys.

In fact, following that which you previously proposed, this was my second approach here too :) However, upon implementing it, it still ran into less serious versions of some of the same issues as the previous, including mapping two distinct metadata types to the same key, inability to specify which was marked as dynamic, and creating additional inconsistency between core metadata, PEP 621 and setup.cfg/other tool formats. On top of that, it creates potential for serious user confusion and ambiguity, since it means license maps to completely syntactically and semantically different fields depending on the config format, and is much less explicit to authors, readers and consumers that it is intended to be a SPDX identifier/expression. See Define license expression as string value in the rejected ideas for more details.

Doesn't PEP 621 mandate that the field be marked as dynamic to be backfilled?

Only when a field is marked as dynamic may a tool provide a "new" value.

• https://www.python.org/dev/peps/pep-0621/#specification
• https://www.python.org/dev/peps/pep-0621/#have-the-dynamic-field-only-require-specifying-missing-required-fields

Or am I misunderstanding what you're saying here?

Potentially, depending on how strictly it is interpreted, but I address this explicitly in some detail in this very section. Given it is simply using the verbatim value of another specific, mutually-exclusive key as defined by this specification, such that any tool reading the pyproject.toml can "trivially, deterministically and unambiguously determine this using only the static data in the pyproject.toml file itself," it meets the spirit and intent, if (arguably) not the exact letter of PEP 621.

While I'd normally argue for explicitness, marking the field as dynamic has serious practical danger, as it means that any tool (including those not yet unaware of this PEP) could fill it with anything it wanted, directly violating the specification in this PEP, which is why the specification here explicitly prohibits marking license as dynamic if license-expression is also defined.

Also, as a side note, requiring it to be dynamic is unavoidably mutually exclusive with @uranusjr and @ofek 's proposal to move license-expression back under the license key, since it would be impossible to mark one one as dynamic without the other, and thus prohibited to define a license expression value there in the first place.

In any case, as this PEP updates the specification originally defined in PEP 621, it does not irrevocably constrain this PEP (in fact, for license-files, I also had to make what was arguably another mild exception in order to maintain safe, sane and backward-compatible default behavior, as also discussed here), and this section of the canonical project metadata spec will be updated accordingly to clarify these edge cases.

That said, I'm not completely wed to the necessity of allowing back-filling in the first place, as I haven't yet seen a 100% compelling rationale either way other than general backward compat and people on Discourse asking for it.

Aside from LicenseRef-Proprietary there is no real way to specific a custom (proprietary) license type. That isn't an issue for most public packages, but since Python is also used in corporate environments with custom package indexes, I could imaging they might want some more flexibility. Especially as we deprecate the license field.

Does it make sense to add something like LicenseRef-Custom:<some custom license name>?

Hmm, good point. SPDX already specifies the use of LicenseRef-<CUSTOM-LICENSE-NAME>, which is what we are taking advantage of here, so if we did allow other custom identifiers, they would presumably want to follow that pattern rather than making up something bespoke. For niche cases that need very specific, proprietary custom licenses, making up an arbitrary bespoke license identifier is not terribly useful, I'd think, since people and tools won't know recognize it unless they are already aware of it anyway, and would need to reference the License-Files anyway for the actual terms, which License Ref-Proprietary makes explicit. If they do have their own custom private infra, that can still allow it, and they can also still use it in their individual files and in other standardized SPDX interchange formats. I'm just not sure it adds value allowing arbitrary license IDs for all Python packages given the room it creates for ambiguity and mistakes relative to the seemingly small benefit for such specific cases.

We could add an explicit LicenseRef-Custom for theoretical cases where a license is FSF- or OSI-approved and thus not technically LicenseRef-Proprietary but doesn't yet have a SPDX identifier, but as these sources are generally kept closely in sync (I've been a small part of that), this would only be true for a tiny number of edge cases packages, as far as I'm aware. There could also hypothetically be cases where the author has a strong good faith belief that their bespoke license is not proprietary, but it hasn't (yet) been reviewed by FSF or OSI...but not sure if that does more harm than good to allow specifying that, given the increased ambiguity. And, in any case, in terms of actively maintained modern packages it seems rather unlikely this is more than a handful of cases. As such, my inclination would be to stick with @pombredanne 's original language here (which this PR essentially leaves unchanged) unless there is a clear consensus otherwise.

Didn't though of LicenseRef-<CUSTOM-LICENSE-NAME>. You are right that LicenseRef-Proprietary should cover it. I'm just wondering though if we should at least allow the former for build tools. (Maybe it would even make sense if only PyPI would reject them.)

When processing the ``License-Expression`` field to determine if it contains
 a valid license expression, build and publishing tools:
 - SHOULD halt execution and raise an error if:
   - The field does not contain a valid license expression
   - One or more license identifiers are not valid (as defined above)   # <-- !

--
I don't plan on using this personally, it's just something I though about that could be an issue if the license field is eventually removed. This could lower the barrier in that area.

Hmm, that's worth thinking about. Currently, given its a SHOULD, custom build and publishing tools and those oriented toward this sort of use case are allowed to not raise an error if the license identifier is unknown, if they have reason to do so (as they would, if there was a need for this). We could carve out an exception "...unless they begin with LicenseRef-, in which case build tools SHOULD issue a warning instead.".

However, that does complicate the spec a bit, and means that build tools won't catch misspellings of LicenseRef-Proprietary or LicenseRef-Public-Domain, or users who prepend LicenseRef to their SPDX identifiers (as some mistakenly do), or users who go for a custom license without carefully considering the implications...unless users see the warning, which at present could be quite buried within the very verbose build output and be very difficult to spot, even for Python core devs, a point made on the Discourse thread. At least anecdotally, this seems more likely than the number of authors who genuinely need to make up their own bespoke license identifiers and use them in the packaging metadata with open source tools.

Since custom and enterprise-oriented build tooling is already free to allow such, which seems likely to be the case if custom internal-only IDs are actually going to be useful for other tools to consume, and the other ways that custom IDs can still be used (SPDX manifest, individual file SPDX tags, REUSE project standard, DEP5 standard, etc, plus actual license file detection, which this PEP makes much easier), on top of the fact that such organizations would have to be using the License field for that very specific purpose already for it to be a regression, I'm still not sure its necessary in this PEP.

To note, when NPM implemented SPDX identifiers, they initially had a (very poorly named) UNLICENSED identifier and one that was See license in FILE_PATH (akin to our License-File),, otherwise it didn't allow any LicenseRef custom ones at all, including LicenseRef-Proprietary and LicenseRef-Public-Domain, nor allow a package to not have a license expression unless private: true was manually set. They later removed even that, and only then did users complain in droves, after which they later re-implemented UNLICENSED following a lot of complaints (see discussion in npm/npm#8918 ). So we'd still be offering clearer and more diverse tags than they do, and there didn't appear to be demand for custom LicenseRefs as opposed to what this PEP has, LicenseRef-Proprietary.

My feeling is given the deprecation process will presumably last several years, if there are actual necessary use cases where specifying a custom license more specific than License Ref-Proprietary in the License-Expression field packaging metadata, using open source tools, for proprietary projects, currently using license for this, and for which other options aren't sufficient, we'll hear about it at some point before license is actually removed, and we can always loosen the spec later but can't tighten it without breaking backward compat.

Some good points you mentioned. Especially "we can change it later if necessary" is quite convincing.

Thanks for taking the time to discuss these issues. I appreciate that!

Indeed, we could; the guidance as it stands now is based on yours, and I didn't really touch that section. I just worry about catching unintentional mispellings of the two canonically defined ones, or users thinking they have to prepend LicenseRef in front of license identifiers, as there unfortunately still seems to be some confusion over. That would also create the need for more cross-field validation, and if license-files.globs was used or the default was relied upon, there would be no guarantee that the package would actually contain the named license files unless tools checked the source tree/sdist contents. We'd also have to consider how we'd want PyPI to handle that. I'm not sure the relatively niche use cases where something more specific than LicenseRef-Proprietary was needed outweighs that, but I'd ultimately trust your judgement, assuming others are okay with requiring and cross-validating that at least one License-File is matched (and additionally, the package may just have a AUTHORS.md which it would match, which would not really be a license, so its not necessarily reliable even then).

@pombredanne As I understand it, the main reason against allowing LicenseRef-xyz is that it will make implementation and validation of License-Expressions much more difficult.

Maybe it's worth to consider a middle ground here. I think we agree that theoretically there might be instances where LicenseRef-Proprietary, LicenseRef-Public-Domain, and the other valid SPDX identifiers aren't enough. What do you guys think about allowing LicenseRef-Custom for these cases?

I would not recommend to enforce that a License-File must be provided, though. It would just get too complicated. For instance, what do we do if someone chooses a different name? Instead, anyone who encounters LicenseRef-Custom should assume it's equivalent to LicenseRef-Proprietary unless stated otherwise in a License-File. That's consistent with how it would have been handled anyway if the License-Expression was missing.

I was actually initially attracted to the LicenseRef-Custom approach as well, having initially suggested something like that. However, I'm not sure how much it really adds over LicenseRef-Proprietary, since AFAIK (correct me if I'm wrong, @pombredanne ) all of the (FSF-recognized) Free and (OSI-recognized) Open Source licenses (i.e., non-proprietary) have SPDX identifiers. Thus, other licenses would thus implicitly qualify as both non-free and non-open, thus are proprietary, aside from the potential for extremely niche cases of licenses that appear to meet the OSS definition and/or the Free Software definition, but are not yet officially recognized by those organizations, and don't otherwise already have a SPDX identifier. So I'm not really sure if there's a clear use case for LicenseRef-Custom that wouldn't be covered by a license expression including LicenseRef-Proprietary, LicenseRef-Public-Domain and the allowed license identifiers, whereas while allowing custom LicenseRefs comes with more downsides, it also has more practical benefits. If there's interest, though, we could think about allowing them in build tools, warning in publishing tools and not allowing on PyPI?

I also thought about proposing distinguishing All Rights Reserved from more permissive but still proprietary licensing, but if a package was really explicitly licensed All Rights Reserved (which is essentially equivalent to not having a license at all), technically speaking outside of any rights granted by PyPI's ToS, no one would be allowed to download or use it, so there doesn't seem to be a clear motivation to package and share it in the first place. But perhaps LicenseRef-All-Rights-Reserved could be allowed by packaging tools, but not by publishing tools/PyPI?

@pombredanne @cdce8p Regardless of what we decide, I think the outcome of our fruitful and insightful discussion so far should be reflected in the PEP. Should I add it as an open issue, or should I document the current approach more completely in the "rejected ideas" section, leaving the door open for further revision on that point?

As mentioned, for now, I added an Open Issue that summarized the points made thus far in our thread above, so they can be included in the PEP and can serve as a basis for further discussion on the Discourse thread, as needed.

I'd really strongly prefer that we use the existing license key instead.

The plain text form for the license key was intentionally kept unused for this specific usecase. From https://python.github.io/peps/pep-0621/#license:

A practical string value for the license key has been purposefully left out to allow for a future PEP to specify support for SPDX [6] expressions (the same logic applies to any sort of “type” field specifying what license the file or text represents).

There's absolutely no reason to introduce a new key here; and certainly no reason to mirror key names under project with what's in the METADATA file.

I agree - this is what the license key was intended for, and we should use it as defined.

But this discussion should happen on Discourse, as noted below.

The plain text form for the license key was intentionally kept unused for this specific usecase. From https://python.github.io/peps/pep-0621/#license:

Just to note once again, I'm well aware of this and it was my initial preference too, and what I initially implemented in the PEP, as I discuss in detail under the Rejected Ideas section.

There's absolutely no reason to introduce a new key here; and certainly no reason to mirror key names under project with what's in the METADATA file.

Actually, there are, as I took the time to describe in detail under the Rejected Ideas section, and as recently discussed at length on this PEP's Discourse thread (the most important one being the hard conflict with the use of the dynamic key, and reader and author confusion and ambiguity with other tool formats). I revised it now to address the points raised more explicitly and focus more on those core issues while minimizing the rest.

I'm happy to discuss this further and hear your ideas for other ways around these issues on the Discourse thread.

I agree - this is what the license key was intended for, and we should use it as defined.

While reserving the string value for potential not-yet-specified future use is normative, the multiple suggestions for how more precise license metadata might be implemented does not appear to be, being contained in an explanatory note, specified as an envisioned future use, and with multiple options specified; and it does not consider the practical implementation complexities and interactions with the dynamic field that I've spent much time analyzing in detail here, and which we can discuss further on the thread.

But this discussion should happen on Discourse, as noted below.

Agreed; for the record, I tried to politely steer previous substantive discussion to the Discourse thread the first time this was brought up, but didn't entirely succeed, sorry.

Yea, here's the thread: https://discuss.python.org/t/12622

"flat array of strings value" sounds a little awkward, but I see what you mean.

I agree, this is not only awkward but potentially confusing; I really struggled with wording this in a way that was both precise, clear and concise. I thought about it some more and rephrased it as "allowing a flat array of strings as the value for the license-files key has been left out for now", which isn't perfect but somewhat better, I hope.

Oh, the irony 😆 Funny enough, when I was doing a quick proofread, I actually caught this same word mispelled [sic] somewhere else, and thought about how embarrassing it would be if that had made it through...heh. Fixed, thanks.

Ah, another one of those...I really should have a regex for that. Fixed, thanks.