Rust: re-enable attribute macro expansion in library mode #19588
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR restores attribute macro expansion for library crates by removing the early-return guard that skipped expansion in library mode.
- Removed the
if self.source_kind == SourceKind::Libraryblock and accompanying TODO. - Now
emit_item_expansionruns uniformly for both binary and library sources. - Ensures that previously disabled macro expansions are once again performed in library mode.
Comments suppressed due to low confidence (3)
rust/extractor/src/translate/base.rs:734
- [nitpick] Add or update the doc comment for
emit_item_expansionto explain that attribute macro expansion is now always performed for library crates and note any implications for performance or configuration.
pub(crate) fn emit_item_expansion(&mut self, node: &ast::Item, label: Label<generated::Item>) {
rust/extractor/src/translate/base.rs:737
- Re-enabling expansion for libraries may reintroduce exponential expansion issues (as noted by the removed TODO). Consider adding a configurable expansion depth limit or heuristic to prevent performance regressions on large library crates.
(|| {
rust/extractor/src/translate/base.rs:737
- Add unit tests covering
emit_item_expansionbehavior in library mode to ensure macro expansions are handled correctly and to catch performance or correctness regressions early.
(|| {
|
Where might we have expected to see a failure if this had been re-enabled without the underlying fix? |
We ran into likely exponential performance problem for some nested macro expansions causing the extractor to use an enormous amount of memory (100GB+). An example library that caused issues was: https://github.com/ferrilab/ferrilab/blob/4105b12e8d6c213156149bb6ba170a821c434e14/funty/src/lib.rs#L1466-L1524 |
There was a problem hiding this comment.
We ran into likely exponential performance problem for some nested macro expansions causing the extractor to use an enormous amount of memory (100GB+).
I see no such regression in the DCA run, so 👍.
There is a decrease in "Lines of user-written code in the database", is that expected?
update: ah, that's probably because one of the projects failed (timed out). It would be good to be confident this is not a problem.
The new version has much fewer extracted files. Comparing the list of files of both variants I see that the missing files are all in |
Now #19572 is merged it should be possible to remove the workaround and re-enable attribute macro expansion on library files. Reverts df99e06