Skip to content

.python_history file causes considerable slowdown #85735

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
bytecookie mannequin opened this issue Aug 16, 2020 · 9 comments
Open

.python_history file causes considerable slowdown #85735

bytecookie mannequin opened this issue Aug 16, 2020 · 9 comments
Labels
3.12 only security fixes build The build process and cross-build performance Performance or resource usage

Comments

@bytecookie
Copy link
Mannequin

bytecookie mannequin commented Aug 16, 2020

BPO 41563
Nosy @stevendaprano

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-08-16.11:42:32.757>
labels = ['build', '3.8', 'performance']
title = '.python_history file causes considerable slowdown'
updated_at = <Date 2020-08-17.10:33:46.799>
user = 'https://bugs.python.org/bytecookie'

bugs.python.org fields:

activity = <Date 2020-08-17.10:33:46.799>
actor = 'bytecookie'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2020-08-16.11:42:32.757>
creator = 'bytecookie'
dependencies = []
files = []
hgrepos = []
issue_num = 41563
keywords = []
message_count = 5.0
messages = ['375505', '375506', '375538', '375539', '375540']
nosy_count = 2.0
nosy_names = ['steven.daprano', 'bytecookie']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue41563'
versions = ['Python 3.8']
@bytecookie
Copy link
Mannequin Author

bytecookie mannequin commented Aug 16, 2020

The seemingly unlimited growth of the ".python_history"-file seems to cause a massive slowdown when starting or exiting python, the same after running commands.
On one machine python took about 3 minutes to start and I found out that while it is seemingly doing nothing, it was constantly aceesing the ".python_history"-file, which was 130 Mb large. After deleting the file, anything was back to normal.

@bytecookie bytecookie mannequin added 3.8 (EOL) end of life build The build process and cross-build performance Performance or resource usage labels Aug 16, 2020
@bytecookie
Copy link
Mannequin Author

bytecookie mannequin commented Aug 16, 2020

Using Python 3.8.3 on Windows 10 1803

@stevendaprano
Copy link
Member

How very odd. I use the Python interactive interpreter extensively, and have done so for years. My history file is only 500 lines.

Did you happen to inspect the file before deleting it to see if it contained something odd?

What does this print for you?

    import readline
    readline.get_history_length()

@stevendaprano
Copy link
Member

My history file is only 500 lines.

*slaps forehead*

Of course it is, I'm running a customer history hook that has a limit of 500 lines in the history file.

It looks to me that by default the history feature is set to unlimited lines, so I guess that implies that this isn't a bug and it is your responsibility to set a maximum history length.

You can put these two lines in your Python startup file:

    import readline
    readline.set_history_length(1000)  # or any number you like

Personally, I don't think that having a default setting that allows the history file to grow to 130MB is a good idea.

@bytecookie
Copy link
Mannequin Author

bytecookie mannequin commented Aug 17, 2020

it is your responsibility to set a maximum history length.

No, sorry. The problem is not the history file, is the massive slow down it causes. And if you have to utilize a process monitoring tool to find out that the history file is the cause, its not a matter of responsibility .
You can only be responsible for something you know of.

Still I wonder why this isn't an already widely known problem, or is this a new feature? (I am an seasoned developer but very new to python)

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
@iritkatriel iritkatriel added 3.12 only security fixes and removed 3.8 (EOL) end of life labels Sep 12, 2022
@Kentzo
Copy link
Contributor

Kentzo commented Sep 22, 2022

Would be nice to have a set of environment variables to control the size and location of .python_history. Something like PYTHONHISTFILE / PYTHONHISTSIZE.

Kentzo added a commit to GreatFruitOmsk/cpython that referenced this issue Sep 22, 2022
@Kentzo
pythongh-85735: Add PYTHONHISTFILE and PYTHONHISTSIZE to control Pyth…
526db25 
…on history file
@bedevere-bot bedevere-bot mentioned this issue Sep 22, 2022
Closed
1 task
@sdavids
Copy link

sdavids commented Feb 2, 2024 •
edited
Loading

Maybe support the XDG directory structure out of the box?

${XDG_STATE_HOME}/python/history

@sdavids
Copy link

sdavids commented Feb 2, 2024

Related discussions:

https://unix.stackexchange.com/questions/630642/change-location-of-python-history
https://unix.stackexchange.com/questions/121377/how-can-i-disable-the-new-history-feature-in-python-3-4/297834#297834

@sdavids
Copy link

sdavids commented Feb 2, 2024

related #88405

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.12 only security fixes build The build process and cross-build performance Performance or resource usage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Kentzo @sdavids @iritkatriel @stevendaprano