🌐 US-Proxy
class="logged-out env-production page-responsive" style="word-wrap: break-word;" >
Skip to content

Support pickling slots in subclasses of common classes #70766

Closed
Closed
Support pickling slots in subclasses of common classes#70766
Labels
3.11only security fixesextension-modulesC modules in the Modules dirstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@serhiy-storchaka
serhiy-storchaka
opened on Mar 17, 2016
BPO 26579
Nosy @rhettinger, @pitrou, @vstinner, @avassalotti, @bitdancer, @berkerpeksag, @serhiy-storchaka
PRs
  • bpo-26579: Add object.__getstate__(). #2821
    		•
    Files
  • copyreg_getstate.patch
  • copyreg_getstate2.patch
  • object_getstate.patch: Add object.getstate
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2022-04-06.17:02:02.310>
created_at = <Date 2016-03-17.08:43:06.421>
labels = ['extension-modules', 'type-feature', 'library', '3.11']
title = 'Support pickling slots in subclasses of common classes'
updated_at = <Date 2022-04-07.14:57:28.953>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2022-04-07.14:57:28.953>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-04-06.17:02:02.310>
closer = 'serhiy.storchaka'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2016-03-17.08:43:06.421>
creator = 'serhiy.storchaka'
dependencies = []
files = ['42183', '43462', '43716']
hgrepos = []
issue_num = 26579
keywords = ['patch']
message_count = 5.0
messages = ['261903', '268830', '270403', '416888', '416931']
nosy_count = 7.0
nosy_names = ['rhettinger', 'pitrou', 'vstinner', 'alexandre.vassalotti', 'r.david.murray', 'berker.peksag', 'serhiy.storchaka']
pr_nums = ['2821']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26579'
versions = ['Python 3.11']

    Linked PRs

    Activity

    1. serhiy-storchaka commented on Mar 17, 2016

      @serhiy-storchaka
      serhiy-storchaka
      on Mar 17, 2016
      MemberAuthor

      Pickling and copying instances of subclasses of some basic classes pickles and copies instance attributes. Example:

      >>> class BA(bytearray):
...     pass
... 
>>> b = BA(b'abc')
>>> b.x = 10
>>> c = copy.copy(b)
>>> c.x
10
>>> c = pickle.loads(pickle.dumps(b))
>>> c.x
10

      But this doesn't work if attributes are saved not in instance dictionary, but in slots.

      >>> class BA(bytearray):
...     __slots__ = ('x',)
... 
>>> b = BA(b'abc')
>>> b.x = 10
>>> c = copy.copy(b)
>>> c.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: x
>>> c = pickle.loads(pickle.dumps(b))
>>> c.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: x

      Since using __slots__ is implementation detail, this failure can be considered as a bug.

      Proposed patch adds support of pickling and copying slots in subclasses of all classes that already support pickling and copying non-slot attributes. It is backward compatible, classes with slots can be unpickled on older Python versions without slots. Affected classes: bytearray, set, frozenset, weakref.WeakSet, collections.OrderedDict, collections.deque, datetime.tzinfo.

      The patch adds the copyreg._getstate() function for Python classes and exposes the _PyObject_GetState() function for extension classes. An alternative (and simpler for end user) solution would be to add default implementation as object.__getstate__(). But this is not easy to reject non-pickleable classes (bpo-22995) in this case, since __getstate__ is looked up as instance attribute, not as other special methods.

    2. serhiy-storchaka
      added
      extension-modulesC modules in the Modules dir
      stdlibStandard Library Python modules in the Lib/ directory
      type-featureA feature request or enhancement
      on Mar 17, 2016

    3. serhiy-storchaka commented on Jun 18, 2016

      @serhiy-storchaka
      serhiy-storchaka
      on Jun 18, 2016
      MemberAuthor

      Synchronized with current sources.

    4. serhiy-storchaka commented on Jul 14, 2016

      @serhiy-storchaka
      serhiy-storchaka
      on Jul 14, 2016
      MemberAuthor

      An alternative way is to expose a default state as object.__getstate__(). It is more efficient since it is implemented in C. Following patch implements this approach.

    5. serhiy-storchaka
      added
      3.7 (EOL)end of life
      on Jul 23, 2017

    6. serhiy-storchaka commented on Apr 6, 2022

      @serhiy-storchaka
      serhiy-storchaka
      on Apr 6, 2022
      MemberAuthor

      New changeset 884eba3 by Serhiy Storchaka in branch 'main':
      bpo-26579: Add object.__getstate__(). (GH-2821)
      884eba3

    7. serhiy-storchaka
      added
      3.11only security fixes
      and removed
      3.7 (EOL)end of life
      on Apr 6, 2022
    8. serhiy-storchaka
      closed this as completedon Apr 6, 2022

    9. vstinner commented on Apr 7, 2022

      @vstinner
      vstinner
      on Apr 7, 2022
      Member

      bpo-26579: Add object.__getstate__(). (GH-2821)
      884eba3

      This change introduced reference leaks: see bpo-47250.

    10. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    11. oscarbenjamin
      added 3 commits that reference this issue on May 16, 2022

      fix(polys): fix pickling of poly objects for Python 3.11

      b5dca62

      fix(polys): fix pickling of poly objects for Python 3.11

      17208bb

      fix(polys): fix pickling of poly objects for Python 3.11

      5e22f0b
    12. BenjaminBossan
      added a commit that references this issue on Dec 13, 2022

      Bugfix: BaseEstimator __getstate__ in Python 3.11

      d5d91ab

    13. gpshead commented on Jul 13, 2023

      @gpshead
      gpshead
      on Jul 13, 2023
      Member

      This caused object.__getstate__ to exist where it hadn't in the past, causing trouble for CLIF generated extension modules.
      See https://github.com/google/clif/pull/64/files#diff-de65cd2a0ef58411075450cf769b38dd6d53803108d536309289cda5810a379aR23 for an example of what we're unfortunately likely to have to do with CLIF for its generated extension modules to not be broken by this behavior change.

      It'd be nice if the 3.11 What's New documentation covered this API change. https://docs.python.org/3/whatsnew/3.11.html#other-language-changes only extolls the change's virtues and does not mention anything about the behavior change consequences and workarounds at the Python and C API levels that people may need to adopt to retain compatibility.

    14. serhiy-storchaka commented on Jul 14, 2023

      @serhiy-storchaka
      serhiy-storchaka
      on Jul 14, 2023
      MemberAuthor

      I don't know what can be suggested as a general workaround. Each problem requires an individual solution, depending on why that code was broken by that change.

    15. lwinkler commented on Aug 23, 2023

      @lwinkler
      lwinkler
      on Aug 23, 2023

      This also creates a bug in pyyaml and ruamel.yaml. It seems that for an empty class, the new getstate method returns None. This causes the serialization to crash.

      @serhiy-storchaka : Is there a good reason why getstate returns None for an empty class instead of an empty directory ?

      Minimal code:

      class Empty:
    pass

empty = Empty()
print("getstate", empty.__getstate__())

      prints: getstate None

    16. serhiy-storchaka commented on Aug 23, 2023

      @serhiy-storchaka
      serhiy-storchaka
      on Aug 23, 2023
      MemberAuthor

      @serhiy-storchaka : Is there a good reason why getstate returns None for an empty class instead of an empty directory ?

      Because pickle.dumps() creates more compact pickle if __getstate__() returns None. Also, it saves time in __getstate__() for creating an empty dict and saves time in pickle.loads() for calling __setstate__().

      Add def __getstate__(self): return {} to an empty class and compare the result of pickle.dumps().

    17. nitzmahone commented on Aug 23, 2023

      @nitzmahone
      nitzmahone
      on Aug 23, 2023

      Accounting for this in PyYAML shouldn't be a big deal- IIUC None has always been a documented valid response from __getstate__() anyway, so we need to be able to handle and preserve that properly.

    18. gpshead
      added 2 commits that reference this issue on Aug 23, 2023

      pythongh-70766: Mention the obj getstate caveat in 3.11 What's new.

      9fe47d5

      gh-70766: Mention the object getstate caveat in 3.11 What's new. (#10…

      b6be188
    19. miss-islington
      added 2 commits that reference this issue on Aug 23, 2023

      pythongh-70766: Mention the object getstate caveat in 3.11 What's new. (

      666b503

      pythongh-70766: Mention the object getstate caveat in 3.11 What's new. (

      f4fc465
    20. gpshead
      added a commit that references this issue on Aug 23, 2023

      [3.11] gh-70766: Mention the object getstate caveat in 3.11 What's ne…

      869dc14
    21. Yhg1s
      added a commit that references this issue on Aug 23, 2023

      [3.12] gh-70766: Mention the object getstate caveat in 3.11 What's ne…

      b312d4f
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.11only security fixesextension-modulesC modules in the Modules dirstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions