Every Dunder Method in Python

(pythonmorsels.com)

154 points | by 8organicbits 11 days ago

16 comments

  • neoncontrails 11 days ago
    I have such a love/hate relationship with dunders. On the one hand, I get warm fuzzies from simplifying a module's interface with `__call__`. More than once I've seized an opportunity to rewrite some ugly conditional logic with a partial ordering by leveraging `__gt__`, `__lte__` and friends. `__setattr__` remains an interesting alternative to using decorators to execute some pre- or post-call behavior to a class method. (IIRC it can also be more efficient than wrapping a method with a decorator, since the `__setattr__` override gets precompiled into Python bytecode along with the rest of the class def.) Probably some other sleights of hand along these lines that I'm forgetting. I'm sure they were fun to write and I probably felt clever doing it.

    Unfortunately in almost every case, dunder methods made it harder for me to collaborate with other people on the same codebase. And I get it. Tinkering with `__setattr__` leads to recursion errors if you're not careful, `__call__` introduces state in an unexpected place (wait, foo is a class instance? But but it behaves just like a function...). It's one of the only "stupid Python tricks" I can think of that lacks a clear analog in other programming languages, so polyglots without a strong Python background tend to hate them. I've tried to make the case on two separate occasions that dunder methods represent a more object-oriented approach to dealing with systemic complexity than type dispatch, and I stand by this. But I concede that the benefits of making the codebase nicer and more ergonomic in some places invariably requires writing class definitions that look horrendous in other places. So it's not so much that dunder methods reduce that complexity, so much as try to contain it and hopefully prevent it from spilling out into the main execution logic.

    P.S. - Trey, if you're reading this, hello from a former TA!

    • th 10 days ago
      > Trey, if you're reading this, hello from a former TA!

      Hi former TA! It's been a long time.

      > Unfortunately in almost every case, dunder methods made it harder for me to collaborate with other people on the same codebase.

      I see dunder methods as methods that should usually (with the exception of __init__, __repr__, and __eq__) be used much more often than they're read.

      I typically use dunder methods when implementing a library or framework that's meant to be used by others who would be looking at its documentation but not necessarily its code.

      For example the dramatic library I published over the weekend relies on __enter__, __exit__, and __del__ (which is a very uncommon one typically): https://pypi.org/project/dramatic/

      Users of that code shouldn't need to look at the code.

      For code that's modified by newer Python users often, using dunder methods might be more difficult to justify. Though you might be able to get away with making a library which that could could use.

    • parpfish 11 days ago
      There’s also a risk that you could spend a long time making some nice behavior possible with dunders, but then people using your object don’t know about it because a lot of the new behaviors don’t get surfaced in typical IDEs.
    • packetlost 11 days ago
      Python has a bunch of facilities for poking holes in the "normal" execution flow (most "dunder" methods), but true enlightenment comes from understanding that it's all just dictionaries, type descriptors, functions, and a particularly gnarly execution order all the way down. The CPython interpreter has some pretty interesting C code too.
  • parpfish 11 days ago
    It took me far too long to realize that “dunder” just means “double underscore” and it wasn't some academic term from grad-level CS.

    Knowing that made hem less scary

    • mardifoufs 11 days ago
      I realized most python related terms usually don't have a very serious origin. I remember when I learned that the name for Python's packaging format ("wheels") is short for "cheese wheels"... because PyPi used to be called "The Cheese Shop"!

      I always thought it was some sort of reference to the structure of the package or something lol. I like it though!

    • userbinator 11 days ago
      I wonder when it became common, since I did some Python stuff many years ago and never encountered the term in that context.

      On the other hand... https://en.wikipedia.org/wiki/Dunder

      • maxnoe 11 days ago
        According to Wikipedia, the term dates back to at least 2002:

        > The frequent use of a double underscores in internal identifiers in Python gave rise to the abbreviation dunder; this was coined by Mark Jackson[3] and independently by Tim Hochberg,[4] within minutes of each other, both in reply to the same question in 2002.[5][6]

    • rigid 11 days ago
      ...at least you weren't looking for any references to "mifflin" functions.
      • parpfish 11 days ago
        The __mifflin__ method is used to define a way for the object to waste cpu-cycles so it looks busy
  • pixelmonkey 11 days ago
    This is a great guide. Awhile back, I wrote a blog post[1], "Python double-under, double-wonder," trying to dispel one common misconception I find among new Python programmers. That is, they often think that these methods are "magical" and thus shouldn't be implemented in their own classes (by "mere mortals"). That is true for a small subset of dunder methods (e.g. `__new__`), but not true for many of them (e.g. `__init__`, `__iter__`, `__call__`, `__len__`).

    A better way to think about it is that the double-underscore naming convention simply means "reserved by the core Python team" -- they offer ways to hook into widely-implemented protocols in the Python runtime.

    That also means you should never invent your own dunder methods or dunder attributes. This is another mistake new Python programmers -- and even some experienced Python programmers, even some Python framework writers! -- tend to make, "imitating" this naming convention to mark parts of their own code as "magic." Don't do that: the whole idea is that the core Python team reserved this namespace (`__*__`) for Python-wide protocols.

    [1]: https://amontalenti.com/2013/04/11/python-double-under-doubl...

  • unwind 11 days ago
    As a native Swedish speaker (and someone quite familiar with the "Bamse" [1] comic), this usage in Python always looks funny ("dunder" in Swedish means "thunder").

    [1]: https://en.wikipedia.org/wiki/Bamse

  • jihadjihad 11 days ago
    The context manager ones (__enter__ and __exit__) are really useful, especially if you're coming from something like Ruby where you're used to having blocks.

    Blocks are what I miss most when moving to a Python project from Ruby, and context managers aren't some sort of replacement for them, but sometimes it is nice to construct your own context manager with custom setup/teardown logic.

  • ddavis 11 days ago
    > Keep in mind that you're not meant to invent your own dunder methods. Sometimes you'll see third-party libraries that do invent their own dunder method, but this isn't encouraged and it can be quite confusing for users who run across such methods and assume they're "real" dunder methods.

    I don't think this is good advice. Example: the dunder methods implemented and used by the Scientific Python/PyData community (__array__, __array_ufunc__, __array_func__, etc.) are so, so, so important to that ecosystem.

    • ayhanfuat 11 days ago
      I think it is good advice. These are reserved for future use. If you define `__data__` and then Python decides to use it, it will cause problems. See "Reserved classes of identifiers" in the Python docs: https://docs.python.org/3/reference/lexical_analysis.html#re...

      > System-defined names, informally known as “dunder” names. These names are defined by the interpreter and its implementation (including the standard library). Current system names are discussed in the Special method names section and elsewhere. More will likely be defined in future versions of Python. Any use of __*__ names, in any context, that does not follow explicitly documented use, is subject to breakage without warning.

      • ddavis 11 days ago
        Yea there's some nuance here. But one of the beautiful things about Python is the existence of PEPs. One would have to write a PEP to get a new dunder method added to CPython. During the review/discussion of the PEP it would come up that one of the most popular ecosystems in all of Python would be impacted by adding a builtin __array__ dunder method, for example. It just wouldn't happen. It makes sense to me that the biggest packages associated with a language can have some impact on the way the language moves forward, even if they are not part of the core implementation. For example, the impact of PEP 563 on Pydantic (another wildly popular package outside of core) caused it to be rolled back.
        • pietro72ohboy 11 days ago
          I don’t think that’s a good argument against the original point. You’re essentially relying on PEPs to protect you against something you shouldn’t have done and which was eminently avoidable in the first place. The namespace __*__ should be considered reserved and alternatives are simple to implement.
    • jameshart 11 days ago
      I propose that we refer to the practice of modules declaring their own dunder methods as “Dunder Miffling”.

      Such declarations MUST be preceded by the comment:

         # I DECLARE
      • darkflame91 11 days ago
        Alternatively, a Boolean variable 'bankruptcy' can be declared in the class to indicate the use of dunder-miffling.
    • kodablah 11 days ago
      They didn't have to be dunder methods though, they could have used any other naming scheme (e.g. just `__` at the beginning, EDIT: or something else) and remained just as important in the ecosystem.
    • shadowgovt 11 days ago
      Those are the exception that proves the rule. And they basically make those scientific computation libraries Python's equivalent to Boost... Not part of the standard library, but it might as well be.

      But in general, Joe Random Developer's library is not that important to justify nibbling away at a function namespace that has no mechanism for keeping two libraries from choosing the same name with different semantic meaning.

    • jonathrg 11 days ago
      Those libraries should not have named the methods like that. They could have chosen any naming scheme and they went with something that collides with the only naming scheme available to Python itself
    • jimbokun 11 days ago
      Do those tie into Python syntax conventions? That seems to be the over riding theme of the dunder methods in the article.

      If not, dunder methods do not seem appropriate.

    • anamexis 11 days ago
      Why is it important that those functions use the double underscore naming convention?
    • akvadrako 9 days ago
      You get the same benefit without the chance of collision with runtime-level names if you use single underscores like _array, _array_func, etc.
    • hobs 11 days ago
      This is definitely bad advice and goes against the python data model. As far as I am concerned, dunder methods are key to idiomatic python.
      • TomBombadildoze 11 days ago
        No, it is bad advice, and it is absolutely not idiomatic Python. The language specification reserves them to ensure new ones can be added in the future.

        https://docs.python.org/3/reference/lexical_analysis.html#re...

        This has been in the specification since, at latest, Python 2.0, and perhaps earlier.

        Any you see defined outside the Python data model are entirely historical, written in contradiction to the spec. Many of them also can't be changed because there is so much code in the wild that depends on them.

        *Do not define your own dunder methods!*

    • agency 11 days ago
      Yeah it seems like a common pattern for custom protocols, like the proposed DataFrame interchange protocol[1] which uses a __dataframe__ method

      [1] https://data-apis.org/dataframe-protocol/latest/

  • bananapub 11 days ago
    • pknerd 11 days ago
      41 upvotes is a solid proof that people found it easier to navigate thru the OP's site than official docs.
      • a2800276 11 days ago
        Such a tragedy.. while the Python docs are excellent, they just aren't _nice_. They always feel like Microsoft chm files from the 90ies.

        And their SEO is terrible, I've started to preface all python searches with 'site:python.org' to not end up on crappy monetization blogs..m

        • zelphirkalt 11 days ago
          A good search engine should be able to surface the official docs of a programming language, when you search for things that are clearly part of those docs. I don't see why the official Python docs have to get into the business of SEO. If they set their sites' metadata attributes in a responsible way, there is nothing more they should be obligated to do.
        • scrlk 11 days ago
          > I've started to preface all python searches with 'site:python.org'

          You might find DevDocs to be useful: https://devdocs.io/

        • uneekname 11 days ago
          Or, open the Python docs in your favorite browser, bring the omnibar into focus, and click the little button at the bottom of it to add the Python documentation to your list of available search engines.

          In your browser's settings, you can set the shortcut for this search engine as "@python"

          Now whenever you're searching, just start typing @python, hit tab to complete, and then type your search term.

          I use this all the time in Firefox, and I'm 90% sure it works in Chrome, too.

        • Ancapistani 11 days ago
          > 'site:python.org'

          If you use DuckDuckGo as your primary search engine, you can use `!py` for the Python 2 docs, and `!py3` for Python 3.

          • hansvm 11 days ago
            Most sites are more easily searched via an external engine restricted to the appropriate domain. E.g., if I search "python object model" I expect to find this [0] lovely reference. I misremembered the exact title, but ddg, bing, and google all have it as one of the first results, depending on your search personalization profile. It's #17 in the docs' built-in search engine accessed through !py3.

            More importantly, in that custom search, it's behind the datetime module (which "models" some time-related behavior), the ssl module (which has a multiplexing "model" for wrapping socket "objects"), and many other results that are just stdlib modules with no consideration for the fact that the query doesn't seem to be asking about specific APIs or modules.

            It's not a great search phrase, and writing a proper search engine is hard, but when I use other tools (e.g., binding !py in qutebrowser's search customization to a ddg search with the site:docs.python.org restriction) I don't have to spend any time crafting a perfect query and still get exactly the results I want.

            [0] https://docs.python.org/3/reference/datamodel.html

            • Ancapistani 11 days ago
              > when I use other tools (e.g., binding !py in qutebrowser's search customization to a ddg search with the site:docs.python.org restriction)

              I think we have bit of a miscommunication here.

              I'm well aware of the advanced capabilities of various search engines - in fact, Google's removal of some of those around the time Google+ came out is what caused me to start looking for other search engines.

              Anyhow, while DuckDuckGo supports the `site:<url>` syntax, bangs are different, as they take you to the third-party site's search itself instead of providing a list of results on DDG.

              Here's the list of all supported bang commands: https://duckduckgo.com/bangs

              • hansvm 11 days ago
                I agree.

                I was noting that 1st-party search is often lacking, and you can build that same bang-syntax idea into your browser as a configuration option. In QuteBrowser it's literally a bang, and in other options usually it's triggered by a space after the identifier.

        • globular-toast 11 days ago
          > And their SEO is terrible, I've started to preface all python searches with 'site:python.org' to not end up on crappy monetization blogs..m

          That's a problem with your tools, not the Python docs. Use a better search engine.

      • hobs 11 days ago
        Then those people would probably enjoy reading Fluent Python: Clear, Concise, and Effective Programming 2nd Edition, as it explains the Python Data Model in depth and will make you understand the dunder methods and the official docs!
      • globular-toast 11 days ago
        Not necessarily. Tons of people using Python aren't even aware of the Python docs. The data model page is my favourite page of the whole Python docs and every dev should at least know of its existence. The OP does not link to the original page so doesn't help in this respect.
    • xyst 11 days ago
      But python docs doesn’t have emojis :(
  • qsort 11 days ago
    Completely unhinged idea, but I have always found the traditional way of doing operator overloading (that is as special methods of a class, like Python, C++ etc.) to be a bit strange.

    It would be more natural to declare the algebraic structure of one or more related classes.

    For example, this is something you would do to override addition:

      class C:
          ...
    
          def __add__(self, other: C):
              ...
    
    
    The way that would feel more natural to me is something like:

      class C:
          ...
    
      algebraic_structure Monoid<C>:
         def +(fst: C, snd: C) -> C:
             ...
    
    
    Which would also work for more complicated structure, for example:

      class Datetime:
          ...
    
      class Timedelta:
          ...
    
      algebraic_structure AffineSpace<Datetime, Timedelta>:
          def point_minus(fst: Datetime, snd: Datetime) -> Timedelta:
              ...
     
          def vector_add(fst: Timedelta, snd: Timedelta) -> Timedelta:
              ...
    
          def vector_inverse(vec: Timedelta) -> Timedelta:
              ...
    
    Where you could automatically "deduce" unspecified methods.
    • blowski 11 days ago
      It might be more obvious for you, but I have no idea what I'm looking at in the algebraic_structure. Perhaps the traditional method is easier for the average numpty like myself.
    • masklinn 11 days ago
      I’m not sure if that’s what you’re driving at, but this looks like type classes?
  • ingen0s 11 days ago
    Im going to go and say that you can make your own Tunder methods if you dont want to mess with confusing the Dunder core.

    ___tunder___

    • qsi 11 days ago
      Funder? Finder? Sunder? Just generalize it to Nunder already!
    • th 10 days ago
      I've never seen a ___tunder___ method, but I've definitely seen a _sunder_ method floating around in some third-party library years back.

      A simple _wondur method is probably a bit more common though.

  • gavi 11 days ago
    This is used to build micrograd and essentially extensively used in PyTorch.

    https://www.youtube.com/watch?v=VMj-3S1tku0

    Excellent intro video btw

  • nickdrozd 11 days ago
    A fun use of dunder hacking is to modify SomeClass.__new__ to return something other than an instance of SomeClass. Believe it or not, this is occasionally a sensible thing to do!

    But it can cause problems. For example, the Mypy type checker refuses to acknowledge that this is possible: https://github.com/python/mypy/issues/1020

  • jen20 11 days ago
    I still don’t understand how people claim Python is “elegant” given the number of files with four underscores in the name get sprinkled around repositories.
    • BurningFrog 11 days ago
      I've worked in Python for 6 years, and I don't think I've even heard it called "elegant".

      To me, it's a dirty blue collar language that gets work done.

      I think Ruby is far ahead of Python in elegance. I often have to use 7 Python lines for what Ruby can do in 3.

      I'm open to being told I'm wrong :)

    • Doxin 9 days ago
      IMO pythons elegance lies in how you can poke at basically all the bits of the object model. Dunder methods are part of that. Everything is an object and you can poke anything on any object.

      It's by far not a perfect language, but you have to appreciate how far it goes to avoid special cases.

    • zelphirkalt 11 days ago
      Elegance is in the eye of the beholder. If one hasn't seen more elegant languages, one might think of Python as elegant. It certainly can appear to be elegant, when comparing with some other languages.
  • panarchy 11 days ago
    No Dunder-Mifflin Method?
  • brauhaus 11 days ago
    There's no __mifflin__ method. Such a missed opportunity
  • testbf 11 days ago
    If you’re reading this, please don’t submit your coworkers to __post_init__ just write normal code, nobody cares is you have some sort of crazy justification, nobody extending that shit.
    • shadowgovt 11 days ago
      It's basically there because of dataclasses, we use it all the time.

      Python painted itself into a corner when they came up with a clever way to auto generate __init__ implementations but then also sometimes you need to modify the auto generated implementation. I kind of like their solution relative to alternatives I've seen... C++ is the worst offender to my mind in this case, because it has extremely complicated auto generation rules and the moment you want to do something special, you are now writing two dozen lines of code to make up for the fact that you got in the way of the auto-gen logic.