I have such a love/hate relationship with dunders. On the one hand, I get warm fuzzies from simplifying a module's interface with `__call__`. More than once I've seized an opportunity to rewrite some ugly conditional logic with a partial ordering by leveraging `__gt__`, `__lte__` and friends. `__setattr__` remains an interesting alternative to using decorators to execute some pre- or post-call behavior to a class method. (IIRC it can also be more efficient than wrapping a method with a decorator, since the `__setattr__` override gets precompiled into Python bytecode along with the rest of the class def.) Probably some other sleights of hand along these lines that I'm forgetting. I'm sure they were fun to write and I probably felt clever doing it.
Unfortunately in almost every case, dunder methods made it harder for me to collaborate with other people on the same codebase. And I get it. Tinkering with `__setattr__` leads to recursion errors if you're not careful, `__call__` introduces state in an unexpected place (wait, foo is a class instance? But but it behaves just like a function...). It's one of the only "stupid Python tricks" I can think of that lacks a clear analog in other programming languages, so polyglots without a strong Python background tend to hate them. I've tried to make the case on two separate occasions that dunder methods represent a more object-oriented approach to dealing with systemic complexity than type dispatch, and I stand by this. But I concede that the benefits of making the codebase nicer and more ergonomic in some places invariably requires writing class definitions that look horrendous in other places. So it's not so much that dunder methods reduce that complexity, so much as try to contain it and hopefully prevent it from spilling out into the main execution logic.
P.S. - Trey, if you're reading this, hello from a former TA!
> Trey, if you're reading this, hello from a former TA!
Hi former TA! It's been a long time.
> Unfortunately in almost every case, dunder methods made it harder for me to collaborate with other people on the same codebase.
I see dunder methods as methods that should usually (with the exception of __init__, __repr__, and __eq__) be used much more often than they're read.
I typically use dunder methods when implementing a library or framework that's meant to be used by others who would be looking at its documentation but not necessarily its code.
For example the dramatic library I published over the weekend relies on __enter__, __exit__, and __del__ (which is a very uncommon one typically): https://pypi.org/project/dramatic/
Users of that code shouldn't need to look at the code.
For code that's modified by newer Python users often, using dunder methods might be more difficult to justify. Though you might be able to get away with making a library which that could could use.
There’s also a risk that you could spend a long time making some nice behavior possible with dunders, but then people using your object don’t know about it because a lot of the new behaviors don’t get surfaced in typical IDEs.
Python has a bunch of facilities for poking holes in the "normal" execution flow (most "dunder" methods), but true enlightenment comes from understanding that it's all just dictionaries, type descriptors, functions, and a particularly gnarly execution order all the way down. The CPython interpreter has some pretty interesting C code too.
I realized most python related terms usually don't have a very serious origin. I remember when I learned that the name for Python's packaging format ("wheels") is short for "cheese wheels"... because PyPi used to be called "The Cheese Shop"!
I always thought it was some sort of reference to the structure of the package or something lol. I like it though!
According to Wikipedia, the term dates back to at least 2002:
> The frequent use of a double underscores in internal identifiers in Python gave rise to the abbreviation dunder; this was coined by Mark Jackson[3] and independently by Tim Hochberg,[4] within minutes of each other, both in reply to the same question in 2002.[5][6]
This is a great guide. Awhile back, I wrote a blog post[1], "Python double-under, double-wonder," trying to dispel one common misconception I find among new Python programmers. That is, they often think that these methods are "magical" and thus shouldn't be implemented in their own classes (by "mere mortals"). That is true for a small subset of dunder methods (e.g. `__new__`), but not true for many of them (e.g. `__init__`, `__iter__`, `__call__`, `__len__`).
A better way to think about it is that the double-underscore naming convention simply means "reserved by the core Python team" -- they offer ways to hook into widely-implemented protocols in the Python runtime.
That also means you should never invent your own dunder methods or dunder attributes. This is another mistake new Python programmers -- and even some experienced Python programmers, even some Python framework writers! -- tend to make, "imitating" this naming convention to mark parts of their own code as "magic." Don't do that: the whole idea is that the core Python team reserved this namespace (`__*__`) for Python-wide protocols.
As a native Swedish speaker (and someone quite familiar with the "Bamse" [1] comic), this usage in Python always looks funny ("dunder" in Swedish means "thunder").
The context manager ones (__enter__ and __exit__) are really useful, especially if you're coming from something like Ruby where you're used to having blocks.
Blocks are what I miss most when moving to a Python project from Ruby, and context managers aren't some sort of replacement for them, but sometimes it is nice to construct your own context manager with custom setup/teardown logic.
> The context manager ones (__enter__ and __exit__) are really useful, especially if you're coming from something like Ruby where you're used to having blocks.
See also @contextlib.contextmanager [1], which makes simple cases less verbose.
> Keep in mind that you're not meant to invent your own dunder methods. Sometimes you'll see third-party libraries that do invent their own dunder method, but this isn't encouraged and it can be quite confusing for users who run across such methods and assume they're "real" dunder methods.
I don't think this is good advice. Example: the dunder methods implemented and used by the Scientific Python/PyData community (__array__, __array_ufunc__, __array_func__, etc.) are so, so, so important to that ecosystem.
I think it is good advice. These are reserved for future use. If you define `__data__` and then Python decides to use it, it will cause problems. See "Reserved classes of identifiers" in the Python docs: https://docs.python.org/3/reference/lexical_analysis.html#re...
> System-defined names, informally known as “dunder” names. These names are defined by the interpreter and its implementation (including the standard library). Current system names are discussed in the Special method names section and elsewhere. More will likely be defined in future versions of Python. Any use of __*__ names, in any context, that does not follow explicitly documented use, is subject to breakage without warning.
Yea there's some nuance here. But one of the beautiful things about Python is the existence of PEPs. One would have to write a PEP to get a new dunder method added to CPython. During the review/discussion of the PEP it would come up that one of the most popular ecosystems in all of Python would be impacted by adding a builtin __array__ dunder method, for example. It just wouldn't happen. It makes sense to me that the biggest packages associated with a language can have some impact on the way the language moves forward, even if they are not part of the core implementation. For example, the impact of PEP 563 on Pydantic (another wildly popular package outside of core) caused it to be rolled back.
I don’t think that’s a good argument against the original point. You’re essentially relying on PEPs to protect you against something you shouldn’t have done and which was eminently avoidable in the first place. The namespace __*__ should be considered reserved and alternatives are simple to implement.
They didn't have to be dunder methods though, they could have used any other naming scheme (e.g. just `__` at the beginning, EDIT: or something else) and remained just as important in the ecosystem.
Double leading underscore has a special meaning and causes the name to be mangled in order to avoid collisions with other classes e.g. `__foo` -> `_ClassName_foo`
Those are the exception that proves the rule. And they basically make those scientific computation libraries Python's equivalent to Boost... Not part of the standard library, but it might as well be.
But in general, Joe Random Developer's library is not that important to justify nibbling away at a function namespace that has no mechanism for keeping two libraries from choosing the same name with different semantic meaning.
Those libraries should not have named the methods like that. They could have chosen any naming scheme and they went with something that collides with the only naming scheme available to Python itself
No, it is bad advice, and it is absolutely not idiomatic Python. The language specification reserves them to ensure new ones can be added in the future.
This has been in the specification since, at latest, Python 2.0, and perhaps earlier.
Any you see defined outside the Python data model are entirely historical, written in contradiction to the spec. Many of them also can't be changed because there is so much code in the wild that depends on them.
A good search engine should be able to surface the official docs of a programming language, when you search for things that are clearly part of those docs. I don't see why the official Python docs have to get into the business of SEO. If they set their sites' metadata attributes in a responsible way, there is nothing more they should be obligated to do.
Or, open the Python docs in your favorite browser, bring the omnibar into focus, and click the little button at the bottom of it to add the Python documentation to your list of available search engines.
In your browser's settings, you can set the shortcut for this search engine as "@python"
Now whenever you're searching, just start typing @python, hit tab to complete, and then type your search term.
I use this all the time in Firefox, and I'm 90% sure it works in Chrome, too.
Most sites are more easily searched via an external engine restricted to the appropriate domain. E.g., if I search "python object model" I expect to find this [0] lovely reference. I misremembered the exact title, but ddg, bing, and google all have it as one of the first results, depending on your search personalization profile. It's #17 in the docs' built-in search engine accessed through !py3.
More importantly, in that custom search, it's behind the datetime module (which "models" some time-related behavior), the ssl module (which has a multiplexing "model" for wrapping socket "objects"), and many other results that are just stdlib modules with no consideration for the fact that the query doesn't seem to be asking about specific APIs or modules.
It's not a great search phrase, and writing a proper search engine is hard, but when I use other tools (e.g., binding !py in qutebrowser's search customization to a ddg search with the site:docs.python.org restriction) I don't have to spend any time crafting a perfect query and still get exactly the results I want.
> when I use other tools (e.g., binding !py in qutebrowser's search customization to a ddg search with the site:docs.python.org restriction)
I think we have bit of a miscommunication here.
I'm well aware of the advanced capabilities of various search engines - in fact, Google's removal of some of those around the time Google+ came out is what caused me to start looking for other search engines.
Anyhow, while DuckDuckGo supports the `site:<url>` syntax, bangs are different, as they take you to the third-party site's search itself instead of providing a list of results on DDG.
I was noting that 1st-party search is often lacking, and you can build that same bang-syntax idea into your browser as a configuration option. In QuteBrowser it's literally a bang, and in other options usually it's triggered by a space after the identifier.
Then those people would probably enjoy reading Fluent Python: Clear, Concise, and Effective Programming 2nd Edition, as it explains the Python Data Model in depth and will make you understand the dunder methods and the official docs!
Not necessarily. Tons of people using Python aren't even aware of the Python docs. The data model page is my favourite page of the whole Python docs and every dev should at least know of its existence. The OP does not link to the original page so doesn't help in this respect.
Completely unhinged idea, but I have always found the traditional way of doing operator overloading (that is as special methods of a class, like Python, C++ etc.) to be a bit strange.
It would be more natural to declare the algebraic structure of one or more related classes.
For example, this is something you would do to override addition:
class C:
...
def __add__(self, other: C):
...
The way that would feel more natural to me is something like:
class C:
...
algebraic_structure Monoid<C>:
def +(fst: C, snd: C) -> C:
...
Which would also work for more complicated structure, for example:
It might be more obvious for you, but I have no idea what I'm looking at in the algebraic_structure. Perhaps the traditional method is easier for the average numpty like myself.
A fun use of dunder hacking is to modify SomeClass.__new__ to return something other than an instance of SomeClass. Believe it or not, this is occasionally a sensible thing to do!
I still don’t understand how people claim Python is “elegant” given the number of files with four underscores in the name get sprinkled around repositories.
IMO pythons elegance lies in how you can poke at basically all the bits of the object model. Dunder methods are part of that. Everything is an object and you can poke anything on any object.
It's by far not a perfect language, but you have to appreciate how far it goes to avoid special cases.
Elegance is in the eye of the beholder. If one hasn't seen more elegant languages, one might think of Python as elegant. It certainly can appear to be elegant, when comparing with some other languages.
If you’re reading this, please don’t submit your coworkers to __post_init__ just write normal code, nobody cares is you have some sort of crazy justification, nobody extending that shit.
It's basically there because of dataclasses, we use it all the time.
Python painted itself into a corner when they came up with a clever way to auto generate __init__ implementations but then also sometimes you need to modify the auto generated implementation. I kind of like their solution relative to alternatives I've seen... C++ is the worst offender to my mind in this case, because it has extremely complicated auto generation rules and the moment you want to do something special, you are now writing two dozen lines of code to make up for the fact that you got in the way of the auto-gen logic.
Unfortunately in almost every case, dunder methods made it harder for me to collaborate with other people on the same codebase. And I get it. Tinkering with `__setattr__` leads to recursion errors if you're not careful, `__call__` introduces state in an unexpected place (wait, foo is a class instance? But but it behaves just like a function...). It's one of the only "stupid Python tricks" I can think of that lacks a clear analog in other programming languages, so polyglots without a strong Python background tend to hate them. I've tried to make the case on two separate occasions that dunder methods represent a more object-oriented approach to dealing with systemic complexity than type dispatch, and I stand by this. But I concede that the benefits of making the codebase nicer and more ergonomic in some places invariably requires writing class definitions that look horrendous in other places. So it's not so much that dunder methods reduce that complexity, so much as try to contain it and hopefully prevent it from spilling out into the main execution logic.
P.S. - Trey, if you're reading this, hello from a former TA!
Hi former TA! It's been a long time.
> Unfortunately in almost every case, dunder methods made it harder for me to collaborate with other people on the same codebase.
I see dunder methods as methods that should usually (with the exception of __init__, __repr__, and __eq__) be used much more often than they're read.
I typically use dunder methods when implementing a library or framework that's meant to be used by others who would be looking at its documentation but not necessarily its code.
For example the dramatic library I published over the weekend relies on __enter__, __exit__, and __del__ (which is a very uncommon one typically): https://pypi.org/project/dramatic/
Users of that code shouldn't need to look at the code.
For code that's modified by newer Python users often, using dunder methods might be more difficult to justify. Though you might be able to get away with making a library which that could could use.
Knowing that made hem less scary
I always thought it was some sort of reference to the structure of the package or something lol. I like it though!
On the other hand... https://en.wikipedia.org/wiki/Dunder
> The frequent use of a double underscores in internal identifiers in Python gave rise to the abbreviation dunder; this was coined by Mark Jackson[3] and independently by Tim Hochberg,[4] within minutes of each other, both in reply to the same question in 2002.[5][6]
A better way to think about it is that the double-underscore naming convention simply means "reserved by the core Python team" -- they offer ways to hook into widely-implemented protocols in the Python runtime.
That also means you should never invent your own dunder methods or dunder attributes. This is another mistake new Python programmers -- and even some experienced Python programmers, even some Python framework writers! -- tend to make, "imitating" this naming convention to mark parts of their own code as "magic." Don't do that: the whole idea is that the core Python team reserved this namespace (`__*__`) for Python-wide protocols.
[1]: https://amontalenti.com/2013/04/11/python-double-under-doubl...
[1]: https://en.wikipedia.org/wiki/Bamse
Blocks are what I miss most when moving to a Python project from Ruby, and context managers aren't some sort of replacement for them, but sometimes it is nice to construct your own context manager with custom setup/teardown logic.
See also @contextlib.contextmanager [1], which makes simple cases less verbose.
[1]: https://docs.python.org/3/library/contextlib.html#contextlib...
I don't think this is good advice. Example: the dunder methods implemented and used by the Scientific Python/PyData community (__array__, __array_ufunc__, __array_func__, etc.) are so, so, so important to that ecosystem.
> System-defined names, informally known as “dunder” names. These names are defined by the interpreter and its implementation (including the standard library). Current system names are discussed in the Special method names section and elsewhere. More will likely be defined in future versions of Python. Any use of __*__ names, in any context, that does not follow explicitly documented use, is subject to breakage without warning.
Such declarations MUST be preceded by the comment:
https://peps.python.org/pep-0008/#descriptive-naming-styles
But in general, Joe Random Developer's library is not that important to justify nibbling away at a function namespace that has no mechanism for keeping two libraries from choosing the same name with different semantic meaning.
If not, dunder methods do not seem appropriate.
https://docs.python.org/3/reference/lexical_analysis.html#re...
This has been in the specification since, at latest, Python 2.0, and perhaps earlier.
Any you see defined outside the Python data model are entirely historical, written in contradiction to the spec. Many of them also can't be changed because there is so much code in the wild that depends on them.
*Do not define your own dunder methods!*
[1] https://data-apis.org/dataframe-protocol/latest/
And their SEO is terrible, I've started to preface all python searches with 'site:python.org' to not end up on crappy monetization blogs..m
You might find DevDocs to be useful: https://devdocs.io/
In your browser's settings, you can set the shortcut for this search engine as "@python"
Now whenever you're searching, just start typing @python, hit tab to complete, and then type your search term.
I use this all the time in Firefox, and I'm 90% sure it works in Chrome, too.
If you use DuckDuckGo as your primary search engine, you can use `!py` for the Python 2 docs, and `!py3` for Python 3.
More importantly, in that custom search, it's behind the datetime module (which "models" some time-related behavior), the ssl module (which has a multiplexing "model" for wrapping socket "objects"), and many other results that are just stdlib modules with no consideration for the fact that the query doesn't seem to be asking about specific APIs or modules.
It's not a great search phrase, and writing a proper search engine is hard, but when I use other tools (e.g., binding !py in qutebrowser's search customization to a ddg search with the site:docs.python.org restriction) I don't have to spend any time crafting a perfect query and still get exactly the results I want.
[0] https://docs.python.org/3/reference/datamodel.html
I think we have bit of a miscommunication here.
I'm well aware of the advanced capabilities of various search engines - in fact, Google's removal of some of those around the time Google+ came out is what caused me to start looking for other search engines.
Anyhow, while DuckDuckGo supports the `site:<url>` syntax, bangs are different, as they take you to the third-party site's search itself instead of providing a list of results on DDG.
Here's the list of all supported bang commands: https://duckduckgo.com/bangs
I was noting that 1st-party search is often lacking, and you can build that same bang-syntax idea into your browser as a configuration option. In QuteBrowser it's literally a bang, and in other options usually it's triggered by a space after the identifier.
That's a problem with your tools, not the Python docs. Use a better search engine.
It would be more natural to declare the algebraic structure of one or more related classes.
For example, this is something you would do to override addition:
The way that would feel more natural to me is something like: Which would also work for more complicated structure, for example: Where you could automatically "deduce" unspecified methods.___tunder___
A simple _wondur method is probably a bit more common though.
https://www.youtube.com/watch?v=VMj-3S1tku0
Excellent intro video btw
But it can cause problems. For example, the Mypy type checker refuses to acknowledge that this is possible: https://github.com/python/mypy/issues/1020
To me, it's a dirty blue collar language that gets work done.
I think Ruby is far ahead of Python in elegance. I often have to use 7 Python lines for what Ruby can do in 3.
I'm open to being told I'm wrong :)
It's by far not a perfect language, but you have to appreciate how far it goes to avoid special cases.
Python painted itself into a corner when they came up with a clever way to auto generate __init__ implementations but then also sometimes you need to modify the auto generated implementation. I kind of like their solution relative to alternatives I've seen... C++ is the worst offender to my mind in this case, because it has extremely complicated auto generation rules and the moment you want to do something special, you are now writing two dozen lines of code to make up for the fact that you got in the way of the auto-gen logic.