Skip to content

All exceptions are now modified with extend_exception! macro #2974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Sep 15, 2021

Conversation

sobolevn
Copy link
Contributor

@sobolevn sobolevn commented Aug 26, 2021

So, I've decided to go with the simplest extend_exception! I can think of.

Why?

  1. It solves the problem
  2. It is flexible enough to write any code we want
  3. We already had something similar in pub fn extend(ctx: &PyContext)
  4. We don't actually need exception structs
  5. We cannot write fancy things with module level declarations, because we don't have ctx yet

Update: it was really hard to make it work with subclasses ad custom __new__ and __init__ in mind.
So, I've switched to a more complex struct-based macro.

Closes #2973
Refs #2960
Refs #2897

@sobolevn
Copy link
Contributor Author

@youknowone quick question: these lines https://github.com/RustPython/RustPython/pull/2974/files#diff-d19484f26ba252a3b3ff27867db9682f948572dd37e83e811d6f073bfc2f07c2R547-R548 are not technically correct. We don't need to always call PyBaseException::tp_new / init. We need to call supertype's new / init. What's the easiest way to do that?

@youknowone
Copy link
Member

youknowone commented Aug 27, 2021

Here is a rough idea:

  • object.__new__ must call the type's designated tp_new
  • other types __new__ call need to track up to object to call object.__new__
    Then they always correct tp_new will be called.

And other stuffs:
It will require additional mro traverse but will be compatible to CPython.
When I discussed about this, I've heard CPython has different slot management policy. So they copy the slot of base class when they create a new class. I think we finally need to follow the same way to remove mro tracking.

@sobolevn
Copy link
Contributor Author

@youknowone it is hard to understand what do you mean without a code-snippet 😞

@youknowone
Copy link
Member

I think the part is not directly related to this issue. You can keep it as current behaviour for now. We need to fix it for entier __new__, not just for exceptions.

@eldpswp99
Copy link
Contributor

Here is a rough idea:

  • object.__new__ must call the type's designated tp_new
  • other types __new__ call need to track up to object to call object.__new__
    Then they always correct tp_new will be called.

And other stuffs:
It will require additional mro traverse but will be compatible to CPython.
When I discussed about this, I've heard CPython has different slot management policy. So they copy the slot of base class when they create a new class. I think we finally need to follow the same way to remove mro tracking.

can i fix this issue?

@sobolevn
Copy link
Contributor Author

sobolevn commented Aug 28, 2021

Ok, new concept. I was not able to fix this problem with __new__ slot. So, I wrote a new macro that has __new__ as a parameter.

Now, this works:

define_exception! {
    PyPermissionError,
    PyOSError,
    permission_error,
    "Not enough permissions.",
    os_error_custom_new,
}

fn os_error_custom_new(
        cls: PyTypeRef,
        args: FuncArgs,
        vm: &VirtualMachine,
) -> PyResult<PyRef<PyBaseException>> {
        println!("custom new");
        PyBaseException::new(args.args, vm).into_ref_with_type(vm, cls)
}

But, for some reason I experience [1] 2494 segmentation fault RUST_BACKTRACE=1 cargo run locally. And cannot find out why 🙂

@sobolevn
Copy link
Contributor Author

I am trying to debug what is going on with valgrind. Not really helpful for now.

» cargo valgrind run

    Finished dev [unoptimized + debuginfo] target(s) in 0.58s
     Running `/Users/sobolev/.cargo/bin/cargo-valgrind target/debug/rustpython`
error: invalid valgrind usage: --20007-- run: /usr/bin/dsymutil "target/debug/rustpython"
--20007-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option

valgrind: m_signals.c:1106 (void handle_SCSS_change(Bool)): Assertion 'ksa_old.sa_flags == skss_old.skss_per_sig[sig].skss_flags' failed.

host stacktrace:
==20007==    at 0x25804212B: ???
==20007==    by 0x2580424A1: ???
==20007==    by 0x258042484: ???
==20007==    by 0x258052F7E: ???
==20007==    by 0x258052C2A: ???
==20007==    by 0x2580CFF19: ???
==20007==    by 0x2580BC6A1: ???
==20007==    by 0x2580BAFD4: ???
==20007==    by 0x2580B8E4E: ???
==20007==    by 0x2580CA28B: ???

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable syscall unix:46 (lwpid 771)
  <stack>
    <frame>
      <ip>0x10439FE26</ip>
      <obj>/usr/lib/system/libsystem_kernel.dylib</obj>
      <fn>__sigaction</fn>
    </frame>
    <frame>
      <ip>0x1043E352A</ip>
      <obj>/usr/lib/system/libsystem_platform.dylib</obj>
      <fn>__platform_sigaction</fn>
    </frame>
    <frame>
      <ip>0x1041BE6C3</ip>
      <obj>/usr/lib/system/libsystem_c.dylib</obj>
      <fn>signal__</fn>
    </frame>
    <frame>
      <ip>0x1003EE566</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::stdlib::signal::make_module</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src/stdlib</dir>
      <file>signal.rs</file>
      <line>277</line>
    </frame>
    <frame>
      <ip>0x100191C28</ip>
      <obj>target/debug/rustpython</obj>
      <fn>core::ops::function::Fn::call</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops</dir>
      <file>function.rs</file>
      <line>70</line>
    </frame>
    <frame>
      <ip>0x1003C5A03</ip>
      <obj>target/debug/rustpython</obj>
      <fn>&lt;alloc::boxed::Box&lt;F,A&gt; as core::ops::function::Fn&lt;Args&gt;&gt;::call</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/alloc/src</dir>
      <file>boxed.rs</file>
      <line>1560</line>
    </frame>
    <frame>
      <ip>0x100845DBD</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::import::import_builtin::{{closure}}</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src</dir>
      <file>import.rs</file>
      <line>103</line>
    </frame>
    <frame>
      <ip>0x100D6B3FB</ip>
      <obj>target/debug/rustpython</obj>
      <fn>core::result::Result&lt;T,E&gt;::and_then</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src</dir>
      <file>result.rs</file>
      <line>704</line>
    </frame>
    <frame>
      <ip>0x100845C71</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::import::import_builtin</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src</dir>
      <file>import.rs</file>
      <line>93</line>
    </frame>
    <frame>
      <ip>0x100350DDD</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::vm::VirtualMachine::initialize::{{closure}}</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src</dir>
      <file>vm.rs</file>
      <line>330</line>
    </frame>
    <frame>
      <ip>0x1003507C1</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::vm::VirtualMachine::initialize</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src</dir>
      <file>vm.rs</file>
      <line>378</line>
    </frame>
    <frame>
      <ip>0x10000290B</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::vm::Interpreter::new_with_init</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src</dir>
      <file>vm.rs</file>
      <line>1972</line>
    </frame>
    <frame>
      <ip>0x10000166E</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython::run</fn>
      <dir>/Users/sobolev/Desktop/rustpython/src</dir>
      <file>lib.rs</file>
      <line>95</line>
    </frame>
    <frame>
      <ip>0x100002368</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython::main</fn>
      <dir>/Users/sobolev/Desktop/rustpython/src</dir>
      <file>main.rs</file>
      <line>2</line>
    </frame>
    <frame>
      <ip>0x100001F5D</ip>
      <obj>target/debug/rustpython</obj>
      <fn>core::ops::function::FnOnce::call_once</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops</dir>
      <file>function.rs</file>
      <line>227</line>
    </frame>
    <frame>
      <ip>0x100003330</ip>
      <obj>target/debug/rustpython</obj>
      <fn>std::sys_common::backtrace::__rust_begin_short_backtrace</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src/sys_common</dir>
      <file>backtrace.rs</file>
      <line>125</line>
    </frame>
    <frame>
      <ip>0x100002773</ip>
      <obj>target/debug/rustpython</obj>
      <fn>std::rt::lang_start::{{closure}}</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src</dir>
      <file>rt.rs</file>
      <line>49</line>
    </frame>
    <frame>
      <ip>0x101269560</ip>
      <obj>target/debug/rustpython</obj>
      <fn>std::rt::lang_start_internal</fn>
      <dir>/rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/ops</dir>
      <file>function.rs</file>
      <line>259</line>
    </frame>
    <frame>
      <ip>0x10000274D</ip>
      <obj>target/debug/rustpython</obj>
      <fn>std::rt::lang_start</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src</dir>
      <file>rt.rs</file>
      <line>48</line>
    </frame>
    <frame>
      <ip>0x1000023A5</ip>
      <obj>target/debug/rustpython</obj>
      <fn>main</fn>
    </frame>
  </stack>
client stack range: [0x1076C8000 0x107EC6FFF] client SP: 0x107EC4298
valgrind stack range: [0x700001BA0000 0x700001C9FFFF] top usage: 9768 of 1048576


Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.

If that doesn't help, please report this bug to: www.valgrind.org

In the bug report, send all the above text, the valgrind
version, and what OS and version you are using.  Thanks.

@sobolevn
Copy link
Contributor Author

@sobolevn
Copy link
Contributor Author

It should be good now 🎉
Finally!

@sobolevn
Copy link
Contributor Author

sobolevn commented Aug 29, 2021

@youknowone I cannot reproduce this failure (https://github.com/RustPython/RustPython/pull/2974/checks?check_run_id=3454164130) locally. Is it related?
But, I don't have the latest macos.

@@ -161,37 +162,45 @@ class SubError(MyError):
assert BaseException.__new__.__qualname__ == 'BaseException.__new__'
assert BaseException.__init__.__qualname__ == 'BaseException.__init__'
assert BaseException().__dict__ == {}
assert BaseException.__doc__
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these lines deleted?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +586 to +583
define_exception! {
PyStopIteration,
PyException,
stop_iteration,
"Signal the end from iterator.__next__()."
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a weak suggestion. If this one would be not a maro_rules but a proc-macro like now, I think we can keep it more structural way for readability and future extension.

/// Signal the end from iterator.__next__().
#[pyexception(name="stop_iteration", base = "PyException")]
struct PyStopIteration {}

This code have a little bit more code but understandable like other common #[pyclass]es

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is problematic to pass __init__ and tp_new this way. Since proc-macro only allows key=string pairs, not key=ident
Writting

#[pyexception(base = "PyException", ctx_name = "stop_iteration", init="import_error_init")]
struct PyImportError {}

fn import_error_init() { ... }

seems a bit off.

Maybe something like:

#[pyexception(stop_iteration, base_exception_new, import_error_init)]
#[pyclass(base = "PyException")]
struct PyImportError {}

?

But, this does not look way better than define_exception! macro right now.

Copy link
Contributor Author

@sobolevn sobolevn Aug 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is even more verbose, because pyclass requires name= and module= to be set:

/// Some doc
#[pyexception(stop_iteration, base_exception_new, import_error_init)]
#[pyclass(name = "ImportError", module = false, base = "PyException")]
struct PyImportError {}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • new and init:tp_new and init can be added in pyimpl.
  • #[pyexception] can trigger #[pyclass] inside the macro. So they shouldn't be stacked.
  • I don't think string and ident matter. We already use string literal for pyclass

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new and init:tp_new and init can be added in pyimpl.

Do you mean like manually? Or do you mean with some extra macro syntax?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't expect extra macro there. We already have pyimpl, so I thought it could work just like pyclass.

Copy link
Member

@youknowone youknowone Sep 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I expect finally it looks like:

#[pyexception(name = ...)]
struct ImportError {}

#[pyimpl]
impl importError {
    #[pyslot]
    fn new(...) {}

    #[pymethod(magic)]
    fn init(...) {}
}

I don't think declaring a free function import_error_init and use the name in the macro have advantage to the traditional way.

@fanninpm
Copy link
Contributor

What about an exception in, e.g., vm/src/stdlib/termios.rs?

@youknowone
Copy link
Member

@coolreader18 looks busy.

I left a few suggestions but I think this version is good enough because it is solving problems.
@sobolevn do you have plan for more work? otherwise, could you rebase this PR to merge current version?

@sobolevn
Copy link
Contributor Author

@sobolevn do you have plan for more work? otherwise, could you rebase this PR to merge current version?

Yes, I do! But, I don't have the time right now. This requires quite a lot of physical labor 😆
I will just rebase it for now.

@sobolevn sobolevn force-pushed the extends-exception-macro branch from a25f47e to 63db6d7 Compare September 15, 2021 07:35
@sobolevn
Copy link
Contributor Author

Rebased! Please, squash it on merge

@youknowone youknowone merged commit b723bbf into RustPython:main Sep 15, 2021
@youknowone
Copy link
Member

thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CPython3.8+ does not have TargetScopeError exception
4 participants