Skip to content

Core implementation of OrderedDict #1162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

pfalcon
Copy link
Contributor

@pfalcon pfalcon commented Mar 17, 2015

No description provided.

@pfalcon
Copy link
Contributor Author

pfalcon commented Mar 18, 2015

I initially wanted to subclass dict and override setter methods, but then found this opportunity to (ab)use table_is_fixed_array flag. Note that I didn't even bother to introduce another flag. Semantics of that flag might have had connotation of "stored in ROM", but then I'd want to redefine it right away. Because "R/O data structure" is a bigger problem (we once touched it), and solving it just for maps doesn't make sense. That's also the reason why I didn't ifdef new additions to map.c. Another reason is that they might be useful even w/o explicit exposing of OrderedDict (for example, there was talk on python-dev on preserving kwarg order).

Note that subclassing native types on its own doesn't work, #1159

@pfalcon
Copy link
Contributor Author

pfalcon commented Mar 18, 2015

As can be seen, this breaks cmd_showbc & cmd_verbose. What can I say - I understand the motives why they were added - to get coverage to a magic number of 90%. But they surely are fragile. So, I didn't look into them, hoping that you, @dpgeorge, already worked around bunch of cases, and can spot what went wrong again more easily than I ;-). At the very least, .out file should be written with masked fields, so it was easy to diff real, unmasked differences.

@dpgeorge
Copy link
Member

Semantics of that flag might have had connotation of "stored in ROM", but then I'd want to redefine it right away. Because "R/O data structure" is a bigger problem (we once touched it), and solving it just for maps doesn't make sense.

It's neat that you can reuse this flag.

BUT, it is very important that this flag means "sored in ROM" and that you can't change the dict. Eg with your patch the following now crashes:

list.a = 1

All types and modules use a constant map/dict to implement their lookup tables. It's nothing to do with const data from a bytecode point of view (ie #573).

@pfalcon
Copy link
Contributor Author

pfalcon commented Mar 18, 2015

"stored in ROM" and that you can't change the dict

Well, but the same may apply to e.g. lists. I can't give immediate example (sys.path = 1 somehow doesn't crash), but that's still the issue. I proposed to have is_ro(void_) or is_rw(void_) as generic solution (would need good thought which is better - they become non-trivial in the presence of loadable modules). Of course, if you want to still postpone it and rather have me still another bit from map used size (or maybe start to steal bits from alloc size to keep it balanced), ok.

@dpgeorge
Copy link
Member

Well, but the same may apply to e.g. lists.

A constant list is a tuple. There are no lists that are in ROM (as far as I know).

@pfalcon
Copy link
Contributor Author

pfalcon commented Mar 19, 2015

Just to clarify motivation for this patch: uctypes uses dict as the most memory-efficient structure for the data. But calculating structure field offsets "on the fly" (well, not by user) requires ordered dict. And without calculating offsets by uctypes itself, almost any uctypes description for Unix is arch-dependent, which renders it unusable in real life.

Paul Sokolovsky added 3 commits March 19, 2015 18:07
Given that there's already support for "fixed table" maps, which are
essentially ordered maps, the implementation of OrderedDict just extends
"fixed table" maps with add/remove operations, and reuses 95% of objdict
code, just making methods tolerant to both dict and OrderedDict.

Some things are missing so far, like CPython-compatible repr and comparison.
Mostly to have coverage of newly added code in map.c.
@pfalcon
Copy link
Contributor Author

pfalcon commented Mar 19, 2015

Ok, added separate flag for ordered maps. Left it in the same word as fixed to optimize || testing. all_keys_are_qstrs would be better candidate for moving, but left for later.

@dpgeorge
Copy link
Member

Merged in 0ef01d0. @pfalcon I hope you don't mind that I squished and tidied up some things. Also fixed a couple of bugs.

@dpgeorge dpgeorge closed this Mar 20, 2015
@pfalcon
Copy link
Contributor Author

pfalcon commented Mar 20, 2015

Sure, I added separate commit to make changes more visible and easier to review, intended to squash it either once ok'ed.

@pfalcon
Copy link
Contributor Author

pfalcon commented Mar 20, 2015

And, now the biggy question - to make ordered dicts really efficient way to deal with data, it would be nice to support literal ordered dicts. Do you think you could change compiler to recognize OrderedDict "keyword" and create an ordered dict? ;-)

@pfalcon
Copy link
Contributor Author

pfalcon commented Mar 20, 2015

Oh my, no, you couldn't, because we don't support dict literals at all :-(.

@dpgeorge
Copy link
Member

we don't support dict literals

That's because they are mutable. We need frozendict first, then we can have FrozenOrderedDict.

Maybe upstream (ie CPython) can be convinced to add frozendict?? :)

@pfalcon
Copy link
Contributor Author

pfalcon commented Mar 20, 2015

There's kind of read-only dict type in CPython:

>>> list.__dict__
mappingproxy({...
>>> list.__dict__["a"]=1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'mappingproxy' object does not support item assignment

But what we really need, as we discussed once, a way to have dict literals, to either serve as a template to create a new dict instance, or if it can be proven that it can be used only once, to be used as object directly. But that still has bytecode persistence on critical path to be done right.

@pfalcon pfalcon deleted the ordereddict branch March 20, 2015 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants