Python – Where Operators are Mapped to Magic Methods

python

I've been reading about magic methods in python, and I've found a lot of info about overriding them and what purpose they serve, but I haven't been able to find where in the language specific operators and actions are mapped to those methods (+ looks for __add__, += looks for __iadd__, creating a new object from a class might call __new__ and __init__, etc.) Is there somewhere I can see what happens when the python interpreter (or whatever lower level mechanism) encounters a plus sign?

Best Answer

Your question is a bit generic. There is a comprehensive list of "special methods", even though it misses some stdlib specific methods(e.g. __setstate__ and __getstate__ used by pickle etc. But it's a protocol of the module pickle not a language protocol).

If you want to know exactly what the interpreter does you can use the dis module to disassemble the bytecode:

>>> import dis
>>> def my_func(a):
...     return a + 2
... 
>>> dis.dis(my_func)
  2           0 LOAD_FAST                0 (a)
              3 LOAD_CONST               1 (2)
              6 BINARY_ADD          
              7 RETURN_VALUE

You can see that the intereper executes a BINARY_ADD byte code when doing addition. If you want to see exactly the operations that BINARY_ADD does you can download Python's source code and check the ceval.c file:

    case BINARY_ADD:
        w = POP();
        v = TOP();
        if (PyInt_CheckExact(v) && PyInt_CheckExact(w)) {
            /* INLINE: int + int */
            register long a, b, i;
            a = PyInt_AS_LONG(v);
            b = PyInt_AS_LONG(w);
            /* cast to avoid undefined behaviour
               on overflow */
            i = (long)((unsigned long)a + b);
            if ((i^a) < 0 && (i^b) < 0)
                goto slow_add;
            x = PyInt_FromLong(i);
        }
        else if (PyString_CheckExact(v) &&
                 PyString_CheckExact(w)) {
            x = string_concatenate(v, w, f, next_instr);
            /* string_concatenate consumed the ref to v */
            goto skip_decref_vx;
        }
        else {
          slow_add:
            x = PyNumber_Add(v, w);
        }
        Py_DECREF(v);
      skip_decref_vx:
        Py_DECREF(w);
        SET_TOP(x);
        if (x != NULL) continue;
        break;

So here we can see that python special cases int and string additions, and eventually falls back to PyNumber_Add, which checks if the first operand implements __add__ and calls it, eventually it tries __radd__ of the right hand side and if nothing works raises a TypeError.

Note that the byte codes are version-specific, so dis will show different results on different versions:

# python2.7
>>> def my_func():
...     return map((lambda x: x+1), range(5))
... 
>>> dis.dis(my_func)
  2           0 LOAD_GLOBAL              0 (map)
              3 LOAD_CONST               1 (<code object <lambda> at 0x16f8c30, file "<stdin>", line 2>)
              6 MAKE_FUNCTION            0
              9 LOAD_GLOBAL              1 (range)
             12 LOAD_CONST               2 (5)
             15 CALL_FUNCTION            1
             18 CALL_FUNCTION            2
             21 RETURN_VALUE        
# python3
>>> dis.dis(my_func)
  2           0 LOAD_GLOBAL              0 (map) 
              3 LOAD_CONST               1 (<code object <lambda> at 0x7f1161a76930, file "<stdin>", line 2>) 
              6 LOAD_CONST               2 ('my_func.<locals>.<lambda>') 
              9 MAKE_FUNCTION            0 
             12 LOAD_GLOBAL              1 (range) 
             15 LOAD_CONST               3 (5) 
             18 CALL_FUNCTION            1 (1 positional, 0 keyword pair) 
             21 CALL_FUNCTION            2 (2 positional, 0 keyword pair) 
             24 RETURN_VALUE

Also the same byte code may be optimized in future versions, so even if the byte code is the same different versions of python will actually perform different instructions.

If you're interested in learning how python works behind the scenes I'd advise you to write some C extensions, following the tutorials and documentation that you can find on the official python's website.

Related Question