Merge branch 'main' into pyio_bytesio_threading

Author: cmaloney

Doc/c-api/object.rst

@@ -197,6 +197,13 @@ Object Protocol
    in favour of using :c:func:`PyObject_DelAttr`, but there are currently no
    plans to remove it.
 
+   The function must not be called with ``NULL`` *v* and an an exception set.
+   This case can arise from forgetting ``NULL`` checks and would delete the
+   attribute.
+
+   .. versionchanged:: next
+      Must not be called with NULL value if an exception is set.
+
 
 .. c:function:: int PyObject_SetAttrString(PyObject *o, const char *attr_name, PyObject *v)
 
@@ -207,6 +214,10 @@ Object Protocol
    If *v* is ``NULL``, the attribute is deleted, but this feature is
    deprecated in favour of using :c:func:`PyObject_DelAttrString`.
 
+   The function must not be called with ``NULL`` *v* and an an exception set.
+   This case can arise from forgetting ``NULL`` checks and would delete the
+   attribute.
+
    The number of different attribute names passed to this function
    should be kept small, usually by using a statically allocated string
    as *attr_name*.
@@ -215,6 +226,10 @@ Object Protocol
    For more details, see :c:func:`PyUnicode_InternFromString`, which may be
    used internally to create a key object.
 
+   .. versionchanged:: next
+      Must not be called with NULL value if an exception is set.
+
+
 .. c:function:: int PyObject_GenericSetAttr(PyObject *o, PyObject *name, PyObject *value)
 
    Generic attribute setter and deleter function that is meant

Doc/howto/functional.rst

@@ -1217,7 +1217,7 @@ flow inside a program.  The book uses Scheme for its examples, but many of the
 design approaches described in these chapters are applicable to functional-style
 Python code.
 
-https://www.defmacro.org/ramblings/fp.html: A general introduction to functional
+https://defmacro.org/2006/06/19/fp.html: A general introduction to functional
 programming that uses Java examples and has a lengthy historical introduction.
 
 https://en.wikipedia.org/wiki/Functional_programming: General Wikipedia entry

Doc/installing/index.rst

@@ -188,7 +188,7 @@ switch::
    Once the Development & Deployment part of PPUG is fleshed out, some of
    those sections should be linked from new questions here (most notably,
    we should have a question about avoiding depending on PyPI that links to
-   https://packaging.python.org/en/latest/mirrors/)
+   https://packaging.python.org/en/latest/guides/index-mirrors-and-caches/)
 
 
 Common installation issues

Doc/library/compression.zstd.rst

@@ -523,8 +523,14 @@ Advanced parameter control
    .. attribute:: compression_level
 
       A high-level means of setting other compression parameters that affect
-      the speed and ratio of compressing data. Setting the level to zero uses
-      :attr:`COMPRESSION_LEVEL_DEFAULT`.
+      the speed and ratio of compressing data.
+
+      Regular compression levels are greater than ``0``. Values greater than
+      ``20`` are considered "ultra" compression and require more memory than
+      other levels. Negative values can be used to trade off faster compression
+      for worse compression ratios.
+
+      Setting the level to zero uses :attr:`COMPRESSION_LEVEL_DEFAULT`.
 
    .. attribute:: window_log
 

Doc/library/shutil.rst

@@ -618,7 +618,8 @@ provided.  They rely on the :mod:`zipfile` and :mod:`tarfile` modules.
    *format* is the archive format: one of
    "zip" (if the :mod:`zlib` module is available), "tar", "gztar" (if the
    :mod:`zlib` module is available), "bztar" (if the :mod:`bz2` module is
-   available), or "xztar" (if the :mod:`lzma` module is available).
+   available), "xztar" (if the :mod:`lzma` module is available), or "zstdtar"
+   (if the :mod:`compression.zstd` module is available).
 
    *root_dir* is a directory that will be the root directory of the
    archive, all paths in the archive will be relative to it; for example,
@@ -673,6 +674,8 @@ provided.  They rely on the :mod:`zipfile` and :mod:`tarfile` modules.
    - *gztar*: gzip'ed tar-file (if the :mod:`zlib` module is available).
    - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available).
    - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available).
+   - *zstdtar*: Zstandard compressed tar-file (if the :mod:`compression.zstd`
+     module is available).
 
    You can register new formats or provide your own archiver for any existing
    formats, by using :func:`register_archive_format`.
@@ -716,8 +719,8 @@ provided.  They rely on the :mod:`zipfile` and :mod:`tarfile` modules.
    *extract_dir* is the name of the target directory where the archive is
    unpacked. If not provided, the current working directory is used.
 
-   *format* is the archive format: one of "zip", "tar", "gztar", "bztar", or
-   "xztar".  Or any other format registered with
+   *format* is the archive format: one of "zip", "tar", "gztar", "bztar",
+   "xztar", or "zstdtar".  Or any other format registered with
    :func:`register_unpack_format`.  If not provided, :func:`unpack_archive`
    will use the archive file name extension and see if an unpacker was
    registered for that extension.  In case none is found,
@@ -789,6 +792,8 @@ provided.  They rely on the :mod:`zipfile` and :mod:`tarfile` modules.
    - *gztar*: gzip'ed tar-file (if the :mod:`zlib` module is available).
    - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available).
    - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available).
+   - *zstdtar*: Zstandard compressed tar-file (if the :mod:`compression.zstd`
+     module is available).
 
    You can register new formats or provide your own unpacker for any existing
    formats, by using :func:`register_unpack_format`.

Doc/library/tarfile.rst

@@ -18,8 +18,8 @@ higher-level functions in :ref:`shutil <archiving-operations>`.
 
 Some facts and figures:
 
-* reads and writes :mod:`gzip`, :mod:`bz2` and :mod:`lzma` compressed archives
-  if the respective modules are available.
+* reads and writes :mod:`gzip`, :mod:`bz2`, :mod:`compression.zstd`, and
+  :mod:`lzma` compressed archives if the respective modules are available.
 
 * read/write support for the POSIX.1-1988 (ustar) format.
 
@@ -47,6 +47,10 @@ Some facts and figures:
    or paths outside of the destination. Previously, the filter strategy
    was equivalent to :func:`fully_trusted <fully_trusted_filter>`.
 
+.. versionchanged:: 3.14
+
+   Added support for Zstandard compression using :mod:`compression.zstd`.
+
 .. function:: open(name=None, mode='r', fileobj=None, bufsize=10240, **kwargs)
 
    Return a :class:`TarFile` object for the pathname *name*. For detailed
@@ -71,6 +75,8 @@ Some facts and figures:
    +------------------+---------------------------------------------+
    | ``'r:xz'``       | Open for reading with lzma compression.     |
    +------------------+---------------------------------------------+
+   | ``'r:zst'``      | Open for reading with Zstandard compression.|
+   +------------------+---------------------------------------------+
    | ``'x'`` or       | Create a tarfile exclusively without        |
    | ``'x:'``         | compression.                                |
    |                  | Raise a :exc:`FileExistsError` exception    |
@@ -88,6 +94,10 @@ Some facts and figures:
    |                  | Raise a :exc:`FileExistsError` exception    |
    |                  | if it already exists.                       |
    +------------------+---------------------------------------------+
+   | ``'x:zst'``      | Create a tarfile with Zstandard compression.|
+   |                  | Raise a :exc:`FileExistsError` exception    |
+   |                  | if it already exists.                       |
+   +------------------+---------------------------------------------+
    | ``'a' or 'a:'``  | Open for appending with no compression. The |
    |                  | file is created if it does not exist.       |
    +------------------+---------------------------------------------+
@@ -99,6 +109,8 @@ Some facts and figures:
    +------------------+---------------------------------------------+
    | ``'w:xz'``       | Open for lzma compressed writing.           |
    +------------------+---------------------------------------------+
+   | ``'w:zst'``      | Open for Zstandard compressed writing.      |
+   +------------------+---------------------------------------------+
 
    Note that ``'a:gz'``, ``'a:bz2'`` or ``'a:xz'`` is not possible. If *mode*
    is not suitable to open a certain (compressed) file for reading,
@@ -115,6 +127,15 @@ Some facts and figures:
    For modes ``'w:xz'``, ``'x:xz'`` and ``'w|xz'``, :func:`tarfile.open` accepts the
    keyword argument *preset* to specify the compression level of the file.
 
+   For modes ``'w:zst'``, ``'x:zst'`` and ``'w|zst'``, :func:`tarfile.open`
+   accepts the keyword argument *level* to specify the compression level of
+   the file. The keyword argument *options* may also be passed, providing
+   advanced Zstandard compression parameters described by
+   :class:`~compression.zstd.CompressionParameter`. The keyword argument
+   *zstd_dict* can be passed to provide a :class:`~compression.zstd.ZstdDict`,
+   a Zstandard dictionary used to improve compression of smaller amounts of
+   data.
+
    For special purposes, there is a second format for *mode*:
    ``'filemode|[compression]'``.  :func:`tarfile.open` will return a :class:`TarFile`
    object that processes its data as a stream of blocks.  No random seeking will
@@ -146,6 +167,9 @@ Some facts and figures:
    | ``'r|xz'``  | Open an lzma compressed *stream* for       |
    |             | reading.                                   |
    +-------------+--------------------------------------------+
+   | ``'r|zst'`` | Open a Zstandard compressed *stream* for   |
+   |             | reading.                                   |
+   +-------------+--------------------------------------------+
    | ``'w|'``    | Open an uncompressed *stream* for writing. |
    +-------------+--------------------------------------------+
    | ``'w|gz'``  | Open a gzip compressed *stream* for        |
@@ -157,6 +181,9 @@ Some facts and figures:
    | ``'w|xz'``  | Open an lzma compressed *stream* for       |
    |             | writing.                                   |
    +-------------+--------------------------------------------+
+   | ``'w|zst'`` | Open a Zstandard compressed *stream* for   |
+   |             | writing.                                   |
+   +-------------+--------------------------------------------+
 
    .. versionchanged:: 3.5
       The ``'x'`` (exclusive creation) mode was added.

Doc/library/zipfile.rst

@@ -129,14 +129,28 @@ The module defines the following items:
 
    .. versionadded:: 3.3
 
+.. data:: ZIP_ZSTANDARD
+
+   The numeric constant for Zstandard compression. This requires the
+   :mod:`compression.zstd` module.
+
    .. note::
 
-      The ZIP file format specification has included support for bzip2 compression
-      since 2001, and for LZMA compression since 2006. However, some tools
-      (including older Python releases) do not support these compression
-      methods, and may either refuse to process the ZIP file altogether,
-      or fail to extract individual files.
+      In APPNOTE 6.3.7, the method ID ``20`` was assigned to Zstandard
+      compression. This was changed in APPNOTE 6.3.8 to method ID ``93`` to
+      avoid conflicts, with method ID ``20`` being deprecated. For
+      compatibility, the :mod:`!zipfile` module reads both method IDs but will
+      only write data with method ID ``93``.
+
+   .. versionadded:: 3.14
+
+.. note::
 
+   The ZIP file format specification has included support for bzip2 compression
+   since 2001, for LZMA compression since 2006, and Zstandard compression since
+   2020. However, some tools (including older Python releases) do not support
+   these compression methods, and may either refuse to process the ZIP file
+   altogether, or fail to extract individual files.
 
 .. seealso::
 
@@ -176,10 +190,11 @@ ZipFile Objects
 
    *compression* is the ZIP compression method to use when writing the archive,
    and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`,
-   :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized
-   values will cause :exc:`NotImplementedError` to be raised.  If
-   :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified
-   but the corresponding module (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not
+   :const:`ZIP_BZIP2`, :const:`ZIP_LZMA`, or :const:`ZIP_ZSTANDARD`;
+   unrecognized values will cause :exc:`NotImplementedError` to be raised.  If
+   :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2`, :const:`ZIP_LZMA`, or
+   :const:`ZIP_ZSTANDARD` is specified but the corresponding module
+   (:mod:`zlib`, :mod:`bz2`, :mod:`lzma`, or :mod:`compression.zstd`) is not
    available, :exc:`RuntimeError` is raised. The default is :const:`ZIP_STORED`.
 
    If *allowZip64* is ``True`` (the default) zipfile will create ZIP files that
@@ -194,6 +209,10 @@ ZipFile Objects
    (see :class:`zlib <zlib.compressobj>` for more information).
    When using :const:`ZIP_BZIP2` integers ``1`` through ``9`` are accepted
    (see :class:`bz2 <bz2.BZ2File>` for more information).
+   When using :const:`ZIP_ZSTANDARD` integers ``-131072`` through ``22`` are
+   commonly accepted (see
+   :attr:`CompressionParameter.compression_level <compression.zstd.CompressionParameter.compression_level>`
+   for more on retrieving valid values and their meaning).
 
    The *strict_timestamps* argument, when set to ``False``, allows to
    zip files older than 1980-01-01 at the cost of setting the
@@ -415,9 +434,10 @@ ZipFile Objects
    read or append. *pwd* is the password used for encrypted files as a :class:`bytes`
    object and, if specified, overrides the default password set with :meth:`setpassword`.
    Calling :meth:`read` on a ZipFile that uses a compression method other than
-   :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or
-   :const:`ZIP_LZMA` will raise a :exc:`NotImplementedError`. An error will also
-   be raised if the corresponding compression module is not available.
+   :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2`,
+   :const:`ZIP_LZMA`, or :const:`ZIP_ZSTANDARD` will raise a
+   :exc:`NotImplementedError`. An error will also be raised if the
+   corresponding compression module is not available.
 
    .. versionchanged:: 3.6
       Calling :meth:`read` on a closed ZipFile will raise a :exc:`ValueError`.

Doc/using/configure.rst

@@ -29,6 +29,9 @@ Features and minimum versions required to build CPython:
 
 * Tcl/Tk 8.5.12 for the :mod:`tkinter` module.
 
+* `libmpdec <https://www.bytereef.org/mpdecimal/doc/libmpdec/>`_ 2.5.0
+  for the :mod:`decimal` module.
+
 * Autoconf 2.72 and aclocal 1.16.5 are required to regenerate the
   :file:`configure` script.
 

Lib/asyncio/base_events.py

@@ -1016,38 +1016,43 @@ async def _connect_sock(self, exceptions, addr_info, local_addr_infos=None):
         family, type_, proto, _, address = addr_info
         sock = None
         try:
-            sock = socket.socket(family=family, type=type_, proto=proto)
-            sock.setblocking(False)
-            if local_addr_infos is not None:
-                for lfamily, _, _, _, laddr in local_addr_infos:
-                    # skip local addresses of different family
-                    if lfamily != family:
-                        continue
-                    try:
-                        sock.bind(laddr)
-                        break
-                    except OSError as exc:
-                        msg = (
-                            f'error while attempting to bind on '
-                            f'address {laddr!r}: {str(exc).lower()}'
-                        )
-                        exc = OSError(exc.errno, msg)
-                        my_exceptions.append(exc)
-                else:  # all bind attempts failed
-                    if my_exceptions:
-                        raise my_exceptions.pop()
-                    else:
-                        raise OSError(f"no matching local address with {family=} found")
-            await self.sock_connect(sock, address)
-            return sock
-        except OSError as exc:
-            my_exceptions.append(exc)
-            if sock is not None:
-                sock.close()
-            raise
+            try:
+                sock = socket.socket(family=family, type=type_, proto=proto)
+                sock.setblocking(False)
+                if local_addr_infos is not None:
+                    for lfamily, _, _, _, laddr in local_addr_infos:
+                        # skip local addresses of different family
+                        if lfamily != family:
+                            continue
+                        try:
+                            sock.bind(laddr)
+                            break
+                        except OSError as exc:
+                            msg = (
+                                f'error while attempting to bind on '
+                                f'address {laddr!r}: {str(exc).lower()}'
+                            )
+                            exc = OSError(exc.errno, msg)
+                            my_exceptions.append(exc)
+                    else:  # all bind attempts failed
+                        if my_exceptions:
+                            raise my_exceptions.pop()
+                        else:
+                            raise OSError(f"no matching local address with {family=} found")
+                await self.sock_connect(sock, address)
+                return sock
+            except OSError as exc:
+                my_exceptions.append(exc)
+                raise
         except:
             if sock is not None:
-                sock.close()
+                try:
+                    sock.close()
+                except OSError:
+                    # An error when closing a newly created socket is
+                    # not important, but it can overwrite more important
+                    # non-OSError error. So ignore it.
+                    pass
             raise
         finally:
             exceptions = my_exceptions = None

Lib/encodings/idna.py

@@ -316,7 +316,7 @@ def _buffer_encode(self, input, errors, final):
 class IncrementalDecoder(codecs.BufferedIncrementalDecoder):
     def _buffer_decode(self, input, errors, final):
         if errors != 'strict':
-            raise UnicodeError("Unsupported error handling: {errors}")
+            raise UnicodeError(f"Unsupported error handling: {errors}")
 
         if not input:
             return ("", 0)