Skip to content

csv.writer with QUOTE_NONE still requires non-emtpy quotechar and escapechar #132813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
vadimkantorov opened this issue Apr 22, 2025 · 11 comments
Open

csv.writer with QUOTE_NONE still requires non-emtpy quotechar and escapechar #132813

vadimkantorov opened this issue Apr 22, 2025 · 11 comments
Assignees
@serhiy-storchaka
Labels
extension-modules C modules in the Modules dir stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@vadimkantorov
Copy link

vadimkantorov commented Apr 22, 2025 •
edited by bedevere-app bot
Loading

Bug report

Bug description:

I'd like to output all quotes verbatim, without any escaping/quoting, but this fails:

import csv
w = csv.writer(open('test.tsv', 'w'), delimiter = '\t', quoting = csv.QUOTE_NONE)
w.writerow(('"hello"', '"world"'))
# _csv.Error: need to escape, but no escapechar set

When I set escapechar='' or quotechar='', it also fails:

import csv
w = csv.writer(open('test.tsv', 'w'), delimiter = '\t', quoting = csv.QUOTE_NONE, escapechar = '')
w.writerow(('"hello"', '"world"'))
# TypeError: "escapechar" must be a 1-character string

import csv
w = csv.writer(open('test.tsv', 'w'), delimiter = '\t', quoting = csv.QUOTE_NONE, quotechar = '')
w.writerow(('"hello"', '"world"'))
# TypeError: "quotechar" must be a 1-character string

I would suggest that under QUOTE_NONE, escaping of quotes should not be performed at all, or at least allow empty escapechar / quotechar

The workaround / hack I found:

import csv
w = csv.writer(open('test.tsv', 'w'), delimiter = '\t', quoting = csv.QUOTE_NONE, quotechar = '\t')
w.writerow(('"hello"', '"world"'))

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs
@vadimkantorov vadimkantorov added the type-bug An unexpected behavior, bug, or error label Apr 22, 2025
@picnixz picnixz added stdlib Python modules in the Lib dir extension-modules C modules in the Modules dir labels Apr 24, 2025
@picnixz picnixz changed the title csv.witer with QUOTE_NONE still requires non-emtpy quotechar and escapechar csv.writer with QUOTE_NONE still requires non-emtpy quotechar and escapechar Apr 24, 2025
@picnixz picnixz added type-feature A feature request or enhancement and removed type-bug An unexpected behavior, bug, or error labels Apr 24, 2025
@picnixz
Copy link
Member

picnixz commented Apr 24, 2025

The docs say:

Instructs writer objects to never quote fields. When the current delimiter occurs in output data it is preceded by the current escapechar character. If escapechar is not set, the writer will raise Error if any characters that require escaping are encountered.

So, it aligns with the expected behavior, namely:

If escapechar is not set, the writer will raise Error if any characters that require escaping are encountered.

I'm converting this into a feature request but since 3.14 is almost feature-frozen, it will need to wait 3.15, unless we consider this as a bug, which I don't as it's documented as such.

I don't think we need to change the meaning of QUOTE_NONE. However, we should allow a verbatim output, namely something that considers the content of a field as being whatever between two delimiters. If someone needs to use a delimiter inside a field, they must already have quoted it inside the .csv.

@vadimkantorov
Copy link
Author

Yeah, I expected that QUOTE_NONE means "do nothing about quoting / attempt to escape quotes", but apparently it's not so...

@picnixz
Copy link
Member

picnixz commented Apr 24, 2025

For that to happen, I think we need a new quoting strategy, something like QUOTE_IGNORE but this name is a bit confusing =/

@vadimkantorov
Copy link
Author

new QUOTE_IGNORE might work! :)

@picnixz
Copy link
Member

picnixz commented Apr 24, 2025 •
edited
Loading

Yes, but the name is very ambiguous as it's too close to QUOTE_NONE. We need a name that convene the fact that we don't do anything but that is different from QUOTE_NONE. We need to think of a better name or think of an alternative (e.g., we could keep QUOTE_NONE and ignore quotes that required escapes with another option). I mainly used the csv module with the default settings, so I didn't encounter this and I don't know if it's not too niche in this case (and if it doesn't excess the scope of the module itself).

If we want to reduce the number of options, we could simply use flags (namely QUOTE_NONE | QUOTE_IGNORE_ERRORS) something like that?

@serhiy-storchaka
Copy link
Member

Use quotechar=None.

The documentation is not accurate. It only mentions the current delimiter, but the quoting is also needed for quotechar, escapechar and the line separator.

@serhiy-storchaka serhiy-storchaka self-assigned this Apr 30, 2025
@serhiy-storchaka
Copy link
Member

On other hand, there may be a bug in the code too. In other places, quotechar is ignored when quoting is QUOTE_NONE, but not here.

I'm working on this.

@vadimkantorov
Copy link
Author

The documentation is not accurate. It only mentions the current delimiter, but the quoting is also needed for quotechar, escapechar and the line separator.

Yep, I think it would be good to add this to the docs at the very least

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Apr 30, 2025
@serhiy-storchaka
pythongh-132813: Fix the csv documentation for quoting and escaping
a08b1ef 
quotechar and new-line characters are always quoted or escaped.
escapechar is always escaped.
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Apr 30, 2025
@serhiy-storchaka
pythongh-132813: Fix the csv documentation for quoting and escaping
84e3b7b 
quotechar and new-line characters are always quoted or escaped.
escapechar is always escaped.
@bedevere-app bedevere-app bot mentioned this issue Apr 30, 2025
Open
@serhiy-storchaka serhiy-storchaka moved this to In Progress in CSV issues Apr 30, 2025
@serhiy-storchaka
Copy link
Member

I left the code unchanged. There may be a use case for escaping " when quoting is disabled.

But quotechar=None is a "stronger" version of quoting=QUOTE_NONE as it outputs " literally. Escaping is still needed if the fields contain the delimiter character or new-line characters.

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue May 1, 2025
@serhiy-storchaka
pythongh-132813: Improve error messages for incorrect types and value…
a43c931 
…s of csv.Dialog attributes

Make them similar to PyArg_Parse error messages, mention None as
a possible value, show a wrong type and the string length.
@bedevere-app bedevere-app bot mentioned this issue May 1, 2025
Merged
@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented May 1, 2025 •
edited
Loading

The part of the problem is that None is not mentioned as valid quotechar value in the error message for empty string. #133241 improves error messages.

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue May 1, 2025
@serhiy-storchaka
pythongh-132813: Improve error messages for incorrect types and value…
a0ca9e7 
…s of csv.Dialect attributes

Make them similar to PyArg_Parse error messages, mention None as
a possible value, show a wrong type and the string length.
@vadimkantorov
Copy link
Author

in the error message

The docs also don't mention that quotechar=None is accepted. And empty quotechar is not allowed makes think that quotechar=None is not allowed as well, as wording empty is a bit ambiguous

https://docs.python.org/3/library/csv.html#csv.Dialect.quotechar:

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters. It defaults to '"'.

Changed in version 3.11: An empty quotechar is not allowed.

serhiy-storchaka added a commit that referenced this issue Jun 2, 2025
@serhiy-storchaka
gh-132813: Improve error messages for incorrect types and values of c…
df98a47 
…sv.Dialog attributes (GH-133241)

Make them similar to PyArg_Parse error messages, mention None as
a possible value, show a wrong type and the string length.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 2, 2025
@serhiy-storchaka @miss-islington
pythongh-132813: Improve error messages for incorrect types and value…
aa490f0 
…s of csv.Dialog attributes (pythonGH-133241)

Make them similar to PyArg_Parse error messages, mention None as
a possible value, show a wrong type and the string length.
(cherry picked from commit df98a47)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-app bedevere-app bot mentioned this issue Jun 2, 2025
Merged
serhiy-storchaka added a commit that referenced this issue Jun 2, 2025
@miss-islington @serhiy-storchaka
[3.14] gh-132813: Improve error messages for incorrect types and valu…
94306f7 
…es of csv.Dialog attributes (GH-133241) (GH-135050)

Make them similar to PyArg_Parse error messages, mention None as
a possible value, show a wrong type and the string length.
(cherry picked from commit df98a47)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@serhiy-storchaka serhiy-storchaka

Labels
extension-modules C modules in the Modules dir stdlib Python modules in the Lib dir type-feature A feature request or enhancement
Projects
CSV issues
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vadimkantorov @serhiy-storchaka @picnixz