Replies: 8 comments 6 replies
-
|
hi bruce, thanks for the feedback! to get a general sense of the problem, can you please indicate the size of the program, whether or not it uses a recursive algorithm (this is sometimes a problem for shedskin), and whether or not you are sure there are no actual dynamic types in there (which can also make type inference a lot harder)..? or even better, would it be possible to provide the code you are trying to compile (in private if necessary)? I'd be happy to see if I can get it to work. one thing you might try is to start with a smaller version of the code. if it compiles, increase its size. if it doesn't or there are actual dynamic types, fix/work around those and increase its size. I'm actually going to try and improve type inference a bit for 0.9.9 (which should be released over the coming weeks) because it also fails (~half of the time) for a new example (examples/pycscg), so you might also wait for that. ideally though, while looking into this, I could also have a look at your program (or better, a minimized version that still cannot be analyzed). thanks again, cheers, |
Beta Was this translation helpful? Give feedback.
-
|
I started to simplify the core class, and it got past the The simplified code transpiles to 6373 lines in .cpp and .hpp files (without the Also, in one class a common pattern is for a method to call the same method of another instance of the same class (Recursion). These instances form a chain, so recusion is convenient. However it is still getting lots of types wrong (It is outputting a lot of types as "pyobj"). It is also trying to cast one of my classes to an "__ss_float" for no apparent reason: Everything seemed to work fine when I tried compiling other helper classes alone (including the one with recursion). Unfortunately I can't provide the code, and it seems coming up with a minimal reproduction will be challenging. Thank you very much for your help so far. |
Beta Was this translation helpful? Give feedback.
-
|
hi bruce, 6373 lines of .?pp code is not too much, but it's getting on the large side.. there are several larger examples, but it really depends more on the complexity of the code how hard it is to analyze. I assume you still get a lot of dynamic typing warnings even if shedskin finishes..? it's really worth looking at those first, and try to pinpoint where they originate. hence my suggestion to gradually build up the code so you can see when they start occurring. or could you share all warnings perhaps? another suggestion could be to not try and analyze all your code, but just part of it as an extension module, and import and use that. for the new pycsg example, you can see that we only translate csg/geom.py. so we get a huge performance boost, without sacrificing flexibility (~keyword args) in the user facing API in csg/core.py. without actually looking at the code/testing it myself, I'm not sure I can do much more.. again, you could send it to me privately (happy to sign an NDA even :P) so I can have a better look. thanks, good luck, |
Beta Was this translation helpful? Give feedback.
-
|
Hi Mark, Thank you very much! I'm stepping away from this for a little while, but will look into dynamic typing warnings if/when I come back to it. We need to change everything to C++ so it can be integrated with a C++ project. Unfortunately an NDA won't work. |
Beta Was this translation helpful? Give feedback.
-
|
(note that geom.py also doesn't always analyze. I hope to fix this for the upcoming release. which might also help other programs of course..) |
Beta Was this translation helpful? Give feedback.
-
|
I managed to analyze the pyscg example, by lowering INCREMENTAL_ALLOCATIONS in infer.py to 1. it makes the analysis slower especially on larger programs, but it is more incremental so in the end more scalable. you might try this setting to see how far it gets, or play with the other settings next to it. I also found a bug while playing with CPA_LIMIT, which results in 'dynamic types' being incorrectly reported. I will try to look into that for 0.9.10. but I'm still not convinced you do not have actual dynamic types..? :) for example, are you sure there are no tuples with more than 2 types of different elements somewhere..? (this is a documented limation, though I'm sure you read through the limitations). I made INCREMENTAL_ALLOCATIONS==1 the default for now, as that seems fundamentally better. I did not start out with an incremental type analysis, but it is become more and more incremental over time as that makes the analysis much more managable.. thanks, cheers! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. I pulled in some of the latest changes with INCREMENTAL_ALLOCATIONS==1 and tried again, and unfortunately got the same error. |
Beta Was this translation helpful? Give feedback.
-
|
Shedskin is great for performance.
I've never tried shedskin for conversion from python to human readable
c++. Maybe it does that, I have no experience with it.
Do whatever is best for you and your team, but there are ways of getting
CPython and C++ to play nicely together - like boost or cython.
https://www.boost.org/doc/libs/1_85_0/libs/python/doc/html/index.html
https://stackoverflow.com/questions/8933263/how-do-i-wrap-a-c-class-with-cython
…On Wed, Jun 19, 2024 at 7:46 PM Bruce ***@***.***> wrote:
Hi Mark,
Thank you very much! I'm stepping away from this for a little while, but
will look into dynamic typing warnings if/when I come back to it. We need
to change everything to C++ so it can be integrated with a C++ project.
Unfortunately an NDA won't work.
—
Reply to this email directly, view it on GitHub
<#484 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAGTEGQI7H7O2C7YRFYZHX3ZII66VAVCNFSM6AAAAABJMB5FRWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TQMRTGYZDQ>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
I do find the C++ produced a bit challenging to read. Hopefully we wouldn't need to edit it much! |
Beta Was this translation helpful? Give feedback.
-
|
thanks for trying again! and too bad it still won't work at this point. I'm planning some further type inference improvements for 0.9.10, perhaps those will help.. I'm always looking for interesting new example programs in this regard, as I don't want to try and fix for something I cannot see/test. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! I can share some of my learning from trying Shedskin on our project: Here are some things that Shedskin doesn't support (from my experience) but also doesn't give a good error message for. Generally I'd get an error when the C++ was compiled, and it was challenging to figure out the cause. If shedskin pointed these out in the python code, it would have made progress quicker:
|
Beta Was this translation helpful? Give feedback.
-
|
thanks bruce! I created two issues for 0.9.10. one for assert with a non-str, and one for class representation (should be '<class ..>' rather than 'class ..'. as for the type hints, shedskin 0.9.9 should actually skip any type annotations (one of the last changes for the release), and also skip any import of the 'typing' module. I played a bit with finally, isinstance and varargs ('*args') result in warnings/errors before generating C++. I don't see how the segfault could occur.. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Mark, There could have been special contexts, which I've now lost. For the "type hints for properties at the start of a class" I mean something like the class foo:
_x: int
def __init__(x: int):
self._x = xFor the "*" In the parameter list of a function, I mean the def my_func(arg1, arg2, *, kwarg1, kwarg2):
pass |
Beta Was this translation helpful? Give feedback.
-
|
thank you! I will create a new issue for the '*' argument. weirdly it works when adding a name, as in '*args'. the other example seems to work fine, even with shedskin 0.9.8. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Any help with this would be appreciated:
I'm trying to transpile the class that performs the core of the algorithm (importing several other classes).
It takes a very long time (about 2 min.), during which I can see a python process is using large amounts of memory (often 8GB, although if fluctuates rapidly).
It outputs
reached maximum number of iterations3 times while transpiling to C++, and outputs lots of warnings (see below). Noteably,XYZ has dynamic (sub) type: {SIDCModifier, str}. Even when the type should be dead simple to infer asset[str].Not surprisingly, the C++ is getting the wrong type as a result (i.e.
set<pyobj *>when it should beset<str>), and doesn't compile.I can see that
reached maximum number of iterationsis output from theiterative_dataflow_analysismethod of the shedskin "infer.py" file whengx.iterations == MAXITERS, so I tried increasing MAXITERS from 30 to 130, but that didn't help..Here's part of the terminal output:
Beta Was this translation helpful? Give feedback.
All reactions