A parenting story from almost 30 years ago.
My wife told me about something her dad did when she was young: in the car,
knowing they were approaching an exit on the highway, he’d say to himself, but
loud enough for his daughters in the back to hear, “If only I could find exit 10
to Westchester...” The girls would look out the window and soon spot the very
sign he needed! “There it is Dad, we found it!” I liked it, it was clever and
sweet.
When my son Max was six or so, we were headed into
Boston to visit the big FAO Schwarz toy store that used to be on Boylston St.
They had a large bronze statue of a teddy bear on the corner in front of the
store. It must have been 10 or 12 feet tall. I wanted to try my father-in-law’s
technique with it.
Max has always been observant, competent and confident. The kind who could
quickly tell you if a piece was missing from a Lego kit. I figured he’d be the
perfect target for this.
We got off the T (the subway if you aren’t from Boston), and had to walk a
bit. When we were a half block from the store, I could clearly see the bear
up ahead. I said, “If only I could find that bear statue...”
Max responded, “Oh Dad, I knew you didn’t know where it was!”
• • •
The store closed in 2004 and the bear was removed. I thought it was gone for
good. But on a walk a few weeks ago, I happened upon it outside the Tufts
Children’s Hospital.
Now I definitely know where it is:
Well, Anthropic and I were not a good fit, though
as predicted it was an experience. I’ve
started a new job on the Python language team at Netflix. It feels like a much
better match in a number of ways.
When sorting strings, you’d often like the order to make sense to a person.
That means numbers need to be treated numerically even if they are in a larger
string.
For example, sorting Python versions with the default sort() would give
you:
Python 3.10
Python 3.11
Python 3.9
when you want it to be:
Python 3.9
Python 3.10
Python 3.11
I wrote about this long ago (Human sorting), but have
continued to tweak the code and needed to add it to a
project recently. Here’s the latest:
import re
def human_key(s: str) -> tuple[list[str | int], str]:
"""Turn a string into a sortable value that works how humans expect.
"z23A" -> (["z", 23, "a"], "z23A")
The original string is appended as a last value to ensure the
key is unique enough so that "x1y" and "x001y" can be distinguished.
"""
def try_int(s: str) -> str | int:
"""If `s` is a number, return an int, else `s` unchanged."""
try:
return int(s)
except ValueError:
return s
return ([try_int(c) for c in re.split(r"(\d+)", s.casefold())], s)
def human_sort(strings: list[str]) -> None:
"""Sort a list of strings how humans expect."""
strings.sort(key=human_key)
The central idea here is to turn a string like "Python 3.9" into the
key ["Python ", 3, ".", 9] so that numeric components will be sorted by
their numeric value. The re.split() function gives us interleaved words and
numbers, and try_int() turns the numbers into actual numbers, giving us sortable
key lists.
There are two improvements from the original:
- The sort is made case-insensitive by using casefold() to lower-case the
string.
- The key returned is now a two-element tuple: the first element is the list
of intermixed strings and integers that gives us the ordering we want. The
second element is the original string unchanged to ensure that unique strings
will always result in distinct keys. Without it,
"x1y" and
"x001Y" would both produce the same key. This solves a
problem that actually happened when sorting the items of
a dictionary.
# Without the tuple: different strings, same key!!
human_key("x1y") -> ["x", 1, "y"]
human_key("x001Y") -> ["x", 1, "y"]
# With the tuple: different strings, different keys.
human_key("x1y") -> (["x", 1, "y"], "x1y")
human_key("x001Y") -> (["x", 1, "y"], "x001Y")
If you are interested, there are many different ways to split the string into
the word/number mix. The comments on the old post
have many alternatives, and there are certainly more.
This still makes some assumptions about what is wanted, and doesn’t cover all
possible options (floats? negative/positive? full file paths?). For those, you
probably want the full-featured natsort (natural sort)
package.
AI is everywhere these days, and everyone has opinions and thoughts. These
are some of mine.
Full disclosure: for a time I worked for Anthropic, the makers of
Claude.ai. I no longer do, and nothing in
this post (or elsewhere on this site) is their opinion or is proprietary to
them.
How to use AI
My advice about using AI is simple: use AI as an assistant, not an expert,
and use it judiciously. Some people will object, “but AI can be wrong!” Yes,
and so can the internet in general, but no one now recommends avoiding online
resources because they can be wrong. They recommend taking it all with a grain
of salt and being careful. That’s what you should do with AI help as well.
We are all learning how to use AI well. Prompt engineering is a new
discipline. It surprises me that large language models (LLMs) give better
answers if you include phrases like “think step-by-step” or “check your answer
before you reply” in your prompt, but they do improve the result. LLMs are not
search engines, but like search engines, you have to approach them as unique
tools that will do better if you know how to ask the right questions.
If you approach AI thinking that it will hallucinate and be wrong, and then
discard it as soon as it does, you are falling victim to
confirmation bias. Yes, AI will be wrong sometimes. That
doesn’t mean it is useless. It means you have to use it carefully.
I’ve used AI to help me write code when I didn’t know how to get started
because it needed more research than I could afford at the moment. The AI
didn’t produce finished code, but it got me going in the right direction, and
iterating with it got me to working code.
One thing it seemed to do well was to write more tests given a few examples
to start from. Your workflow probably has steps where AI can help you. It’s
not a magic bullet, it’s a tool that you have to learn how to use.
The future of coding
In beginner-coding spaces like Python Discord,
anxious learners ask if there is any point in learning to code, since won’t AI
take all the jobs soon anyway?
Simon Willison seems to be our best guide to the
head-spinning pace of AI development these days (if you can keep up with the
head-spinning pace of his blog!) I like what he said
recently about how AI will affect new programmers:
There has never been a better time to learn to code — the
learning curve is being shaved down by these new LLM-based tools, and the
amount of value people with programming literacy can produce is going up by
an order of magnitude.
People who know both coding and LLMs will be a whole lot more attractive
to hire to build software than people who just know LLMs for many years to
come.
Simon has also emphasized in his writing what I have found: AI lets me write
code that I wouldn’t have undertaken without its help. It doesn’t produce the
finished code, but it’s a helpful pair-programming assistant.
Can LLMs think?
Another objection I see often: “but LLMs can’t think, they just predict the
next word!” I’m not sure we have a consensus understanding of what “think” means
in this context. Airplanes don’t fly in the same way that birds do.
Automobiles don’t run in the same way that horses do. The important thing is
that they accomplish many of the same tasks.
OK, so AI doesn’t think the same way that people do. I’m fine with that.
What’s important to me is that it can do some work for me, work that could also
be done by people thinking. Cars (“horseless carriages”) do work that used to
be done by horses running. No one now complains that cars work differently than
horses.
If “just predict the next word” is an accurate description of what LLMs are
doing, it’s a demonstration of how surprisingly powerful predicting the next
word can be.
Harms
I am concerned about the harms that AI can cause. Some people and
organizations are focused on Asimov-style harms (will society collapse, will
millions die?) and I am glad they are. But I’m more concerned with
Dickens-style harms: people losing jobs not because AI can do their work, but
because people in charge will think AI can do other people’s work. Harms due to
people misunderstanding what AI does and doesn’t do well and misusing it.
I don’t see easy solutions to these problems. To go back to the car analogy:
we’ve been a car society for about 120 years. For most of that time we’ve been
leaning more and more towards cars. We are still trying to find the right
balance, the right way to reduce the harm they cause while keeping the benefits
they give us.
AI will be similar. The technology is not going to go away. We will not turn
our back on it and put it back into the bottle. We’ll continue to work on
improving how it works and how we work with it. There will be good and bad.
The balance will depend on how well we collectively use it and educate each
other, and how well we pay attention to what is happening.
Future
The pro-AI hype in the industry now is at a fever pitch, it’s completely
overblown. But the anti-AI crowd also seems to be railing against it without a
clear understanding of the current capabilities or the useful approaches.
I’m going to be using AI more, and learning where it works well and where it
doesn’t.
After nearly two years, I think this is finally ready: coverage.py can use
sys.monitoring to more efficiently measure branch
coverage.
I would love for people to try it, but it’s a little involved at the
moment:
Once you have both of those things, set the environment variable
COVERAGE_CORE=sysmon and run coverage as you usually do. If all goes
well, it should be faster. Please let me know!
Feedback is welcome in GitHub issues or in the
#coverage-py channel in the Python Discord server.
This has been a long journey, starting when I first
commented on PEP 669 that underpins this work. Mark Shannon and I have had
many back and forths about the behavior of sys.monitoring, finally landing on
something that would work for us both.
For the curious: traditionally coverage.py relied on
sys.settrace. Python calls my recording function for
every line of Python executed. It’s simple and effective, but inefficient.
After I’ve been told a line was executed once, I don’t need to be told again,
but settrace keeps calling my function. The new
sys.monitoring that arrived in Python 3.12 lets me
disable an event once it’s fired, so after the first ping there’s no overhead to
running that same code multiple times.
It took a while to iron out the event behavior that lets us measure branches
as well as lines, but Python 3.14.0 after alpha 5 has it, so we’re finally able
to announce coverage.py support for people to try out.
Someone asked recently, “is there any reason to use a generator if I need to
store all the values anyway?” As it happens, I did just that in the code for
this blog’s sidebar because I found it the most readable way to do it. Maybe it
was a good idea, maybe not. Let me show you.
If you look at the sidebar on the left, “More blog” lists tags and years
interleaved in an unusual way: two tags, a year, a tag, a year. That pattern of
five repeats:
python
coverage
‘25
my code
‘24
math
beginners
‘23
git
‘22
github
testing
‘21
audio
‘20
(etc)
I chose this pattern because it seemed to fill the space nicely and simpler
schemes didn’t look as good. But how to implement it in a convenient way?
Generators are a good way to express iteration (a sequence of values)
separately from the code that will consume the values. A simplified version of
my sidebar code looks something like this:
def gen_sidebar_links():
# Get list of commonly used tags.
tags = iter(list_most_common_tags())
# Get all the years we've published.
years = iter(list_all_years())
while True:
yield next(tags)
yield next(tags)
yield next(years)
yield next(tags)
yield next(years)
This nicely expresses the “2/1/1/1 forever” idea, except it doesn’t work:
when we are done with the years, next(years) will raise a
StopIteration exception, and generators aren’t allowed to raise those so we have
to deal with it. And, I wanted to fill out the sidebar with some more tags once
the years were done, so it’s actually more like this:
def gen_sidebar_links():
# Get list of commonly used tags.
tags = iter(list_most_common_tags())
# Get all the years we've published.
years = iter(list_all_years())
try:
while True:
yield next(tags)
yield next(tags)
yield next(years)
yield next(tags)
yield next(years)
except StopIteration: # no more years
pass
# A few more tags:
for _ in range(8):
yield next(tags)
This relates to the original question because I only use this generator
once to create a cached list of the sidebar links:
@functools.cache
def sidebar_links():
return list(gen_sidebar_links)
This is strange: a generator that’s only called once, and is used to make a
list. I find the generator the best way to express the idea. Other ways to
write the function feel more awkward to me. I could have built a list directly
and the function would be more like:
def sidebar_links():
# ... Get the tags and years ...
links = []
try:
while True:
links.append(next(tags))
links.append(next(tags))
links.append(next(years))
links.append(next(tags))
links.append(next(years))
except StopIteration: # no more years
pass
for _ in range(8):
links.append(next(tags))
return links
Now the meat of the function is cluttered with “links.append”, obscuring the
pattern, but could be OK. We could be tricky and make a short helper, but that
might be too clever:
def sidebar_links():
# ... Get the tags and years ...
use = (links := []).append
try:
while True:
use(next(tags))
use(next(tags))
use(next(years))
use(next(tags))
use(next(years))
except StopIteration: # no more years
pass
for _ in range(8):
use(next(tags))
return links
Probably there’s a way to use the itertools
treasure chest to create the interleaved sequence I want, but I haven’t tried
too hard to figure it out.
I’m a fan of generators, so I still like the yield approach. I like that it
focuses solely on what values should appear in what order without mixing in what
to do with those values. Your taste may differ.
Older:
