Skip to content

bpo-29240: Fix locale encodings in UTF-8 Mode - #5170

Merged
vstinner merged 1 commit into
python:masterfrom
vstinner:decode_locale_ex
Jan 15, 2018
Merged

bpo-29240: Fix locale encodings in UTF-8 Mode#5170
vstinner merged 1 commit into
python:masterfrom
vstinner:decode_locale_ex

Conversation

@vstinner

@vstinner vstinner commented Jan 13, 2018 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Member

Modify locale.localeconv(), time.tzname, os.strerror() and other
functions to ignore the UTF-8 Mode: always use the current locale
encoding.

Changes:

  • Add _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx(). On decoding or
    encoding error, they return the position of the error and an error
    message which are used to raise Unicode errors in
    PyUnicode_DecodeLocale() and PyUnicode_EncodeLocale().
  • Replace _Py_DecodeCurrentLocale() with _Py_DecodeLocaleEx().
  • PyUnicode_DecodeLocale() now uses _Py_DecodeLocaleEx() for all
    cases, especially for the strict error handler.
  • Add _Py_DecodeUTF8Ex(): return more information on decoding error
    and supports the strict error handler.
  • Rename _Py_EncodeUTF8_surrogateescape() to _Py_EncodeUTF8Ex().
  • Replace _Py_EncodeCurrentLocale() with _Py_EncodeLocaleEx().
  • Ignore the UTF-8 mode to encode/decode localeconv(), strerror()
    and time zone name.
  • Remove PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize()
    and PyUnicode_EncodeLocale() now ignore the UTF-8 mode: always use
    the "current" locale.
  • Remove _PyUnicode_DecodeCurrentLocale(),
    _PyUnicode_DecodeCurrentLocaleAndSize() and
    _PyUnicode_EncodeCurrentLocale().

https://bugs.python.org/issue29240
@vstinner
bpo-29240: Fix locale encodings in UTF-8 Mode
59e6f9e 
Modify locale.localeconv(), time.tzname, os.strerror() and other
functions to ignore the UTF-8 Mode: always use the current locale
encoding.

Changes:

* Add _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx(). On decoding or
  encoding error, they return the position of the error and an error
  message which are used to raise Unicode errors in
  PyUnicode_DecodeLocale() and PyUnicode_EncodeLocale().
* Replace _Py_DecodeCurrentLocale() with _Py_DecodeLocaleEx().
* PyUnicode_DecodeLocale() now uses _Py_DecodeLocaleEx() for all
  cases, especially for the strict error handler.
* Add _Py_DecodeUTF8Ex(): return more information on decoding error
  and supports the strict error handler.
* Rename _Py_EncodeUTF8_surrogateescape() to _Py_EncodeUTF8Ex().
* Replace _Py_EncodeCurrentLocale() with _Py_EncodeLocaleEx().
* Ignore the UTF-8 mode to encode/decode localeconv(), strerror()
  and time zone name.
* Remove PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize()
  and PyUnicode_EncodeLocale() now ignore the UTF-8 mode: always use
  the "current" locale.
* Remove _PyUnicode_DecodeCurrentLocale(),
  _PyUnicode_DecodeCurrentLocaleAndSize() and
  _PyUnicode_EncodeCurrentLocale().
@vstinner

vstinner commented Jan 15, 2018

Copy link
Copy Markdown
Member Author

I tested this PR using test_all_locales.py attached to https://bugs.python.org/issue29240 on Fedora 27, FreeBSD 11.0 and macOS 10.13.2. It works as expected.
@vstinner
vstinner merged commit 7ed7aea into python:master Jan 15, 2018
@vstinner
vstinner deleted the decode_locale_ex branch January 15, 2018 09:45
jeethu pushed a commit to jeethu/cpython that referenced this pull request Jan 16, 2018
@vstinner Jeethu Rao
bpo-29240: Fix locale encodings in UTF-8 Mode (python#5170)
438385b 
Modify locale.localeconv(), time.tzname, os.strerror() and other
functions to ignore the UTF-8 Mode: always use the current locale
encoding.

Changes:

* Add _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx(). On decoding or
  encoding error, they return the position of the error and an error
  message which are used to raise Unicode errors in
  PyUnicode_DecodeLocale() and PyUnicode_EncodeLocale().
* Replace _Py_DecodeCurrentLocale() with _Py_DecodeLocaleEx().
* PyUnicode_DecodeLocale() now uses _Py_DecodeLocaleEx() for all
  cases, especially for the strict error handler.
* Add _Py_DecodeUTF8Ex(): return more information on decoding error
  and supports the strict error handler.
* Rename _Py_EncodeUTF8_surrogateescape() to _Py_EncodeUTF8Ex().
* Replace _Py_EncodeCurrentLocale() with _Py_EncodeLocaleEx().
* Ignore the UTF-8 mode to encode/decode localeconv(), strerror()
  and time zone name.
* Remove PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize()
  and PyUnicode_EncodeLocale() now ignore the UTF-8 mode: always use
  the "current" locale.
* Remove _PyUnicode_DecodeCurrentLocale(),
  _PyUnicode_DecodeCurrentLocaleAndSize() and
  _PyUnicode_EncodeCurrentLocale().
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip news

3 participants

@vstinner @the-knights-who-say-ni @bedevere-bot