5.2.1 Types and subtypes

Conceptually, a type is a set of values (i.e., possible states or objects) and a set of operations defined on the values in that set.

Similarly, a type S is (a behavioral) subtype of type T if the set of values of type S is a “subset” of the values in set T an set of operations of type S is a “superset” of the operations of type T. That is, we can safely substitute elements of subtype S for elements of type T because S’s operations behave the “same” as T’s operations.

This is known as the Liskov Substitution Principle [24,39].

Consider a type representing all furniture and a type representing all chairs. In general, we consider the set of chairs to be a subset of the set of furniture. A chair should have all the general characteristics of furniture, but it may have additional characteristics specific to chairs.

If we can perform an operation on furniture in general, we should be able to perform the same operation on a chair under the same circumstances and get the same result. Of course, there may be additional operations we can perform on chairs that are not applicable to furniture in general.

Thus the type of all chairs is a subtype of the type of all furniture according to the Liskov Substitution Principle.

5.2.2 Constants, variables, and expressions

Now consider the types of the basic program elements.

A constant has whatever types it is defined to have in the context in which it is used. For example, the constant symbol 1 might represent an integer, a real number, a complex number, a single bit, etc., depending upon the context.

A variable has whatever types its value has in a particular context and at a particular time during execution. The type may be constrained by a declaration of the variable.

An expression has whatever types its evaluation yields based on the types of the variables, constants, and operations from which it is constructed.

5.2.3 Static and dynamic

In a statically typed language, the types of a variable or expression can be determined from the program source code and checked at “compile time” (i.e., during the syntactic and semantic processing in the front-end of a language processor). Such languages may require at least some of the types of variables or expressions to be declared explicitly, while others may be inferred implicitly from the context.

Java, Scala, and Haskell are examples of statically typed languages.

In a dynamically typed language, the specific types of a variable or expression cannot be determined at “compile time” but can be checked at runtime.

Lisp, Python, JavaScript, and Lua are examples of dynamically typed languages.

Of course, most languages use a mixture of static and dynamic typing. For example, Java objects defined within an inheritance hierarchy must be bound dynamically to the appropriate operations at runtime. Also Java objects declared of type Object (the root class of all user-defined classes) often require explicit runtime checks or coercions.

5.2.4 Nominal and structural

In a language with nominal typing, the type of value is based on the type name assigned when the value is created. Two values have the same type if they have the same type name. A type S is a subtype of type T only if S is explicitly declared to be a subtype of T.

For example, Java is primarily a nominally typed language. It assigns types to an object based on the name of the class from which the object is instantiated and the superclasses extended and interfaces implemented by that class.

However, Java does not guarantee that subtypes satisfy the Liskov Substitution Principle. For example, a subclass might not implement an operation in a manner that is compatible with the superclass. (The behavior of subclass objects are this different from the behavior of superclass objects.) Ensuring that Java subclasses preserve the Substitution Principle is considered good programming practice in most circumstances.

In a language with structural typing, the type of a value is based on the structure of the value. Two values have the same type if they have the “same” structure; that is, they have the same public data attributes and operations and these are themselves of compatible types.

In structurally typed languages, a type S is a subtype of type T only if S has all the public data values and operations of type T and the data values and operations are themselves of compatible types. Subtype S may have additional data values and operations not in T.

Haskell is an example of a primarily structurally typed language.

5.2.5 Polymorphic operations

Polymorphism refers to the property of having “many shapes”. In programming languages, we are primarily interested in how polymorphic function names (or operator symbols) are associated with implementations of the functions (or operations).

In general, two primary kinds of polymorphic operations exist in programming languages:

1. Ad hoc polymorphism, in which the same function name (or operator symbol) can denote different implementations depending upon how it is used in an expression. That is, the implementation invoked depends upon the types of function’s arguments and return value.

   There are two subkinds of ad hoc polymorphism.

   1. Overloading refers to ad hoc polymorphism in which the language’s compiler or interpreter determines the appropriate implementation to invoke using information from the context. In statically typed languages, overloaded names and symbols can usually be bound to the intended implementation at compile time based on the declared types of the entities. They exhibit early binding.

      Consider the language Java. It overloads a few operator symbols, such as using the + symbol for both addition of numbers and concatenation of strings. Java also overloads calls of functions defined with the same name but different signatures (patterns of parameter types and return value). Java does not support user-defined operator overloading; C++ does.

      Haskell’s type class mechanism implements overloading polymorphism in Haskell. There are similar mechanisms in other languages such as Scala and Rust.

   2. Subtyping (also known as subtype polymorphism or inclusion polymorphism) refers to ad hoc polymorphism in which the appropriate implementation is determined by searching a hierarchy of types. The function may be defined in a supertype and redefined (overridden) in subtypes. Beginning with the actual types of the data involved, the program searches up the type hierarchy to find the appropriate implementation to invoke. This usually occurs at runtime, so this exhibits late binding.

      The object-oriented programming community often refers to inheritance-based subtype polymorphism as simply polymorphism. This is the polymorphism associated with the class structure in Java.

      Haskell does not support subtyping. Its type classes do support class extension, which enables one class to inherit the properties of another. However, Haskell’s classes are not types.

2. Parametric polymorphism, in which the same implementation can be used for many different types. In most cases, the function (or class) implementation is stated in terms of one or more type parameters. In statically typed languages, this binding can usually be done at compile time (i.e., exhibiting early binding).

   The object-oriented programming (e.g., Java) community often calls this type of polymorphism generics or generic programming.

   The functional programming (e.g., Haskell) community often calls this simply polymorphism.

5.2.6 Polymorphic variables

A polymorphic variable is a variable that can “hold” values of different types during program execution.

For example, a variable in a dynamically typed language (e.g., Python) is polymorphic. It can potentially “hold” any value. The variable takes on the type of whatever value it “holds” at a particular point during execution.

Also, a variable in a nominally and statically typed, object-oriented language (e.g., Java) is polymorphic. It can “hold” a value its declared type or of any of the subtypes of that type. The variable is declared with a static type; its value has a dynamic type.

A variable that is a parameter of a (parametrically) polymorphic function is polymorphic. It may be bound to different types on different calls of the function.

5.3.1 Objects

Python is object-based [13, Ch. 3]; it treats all data as objects.

A Python object has the following essential characteristics of objects [13, Ch. 3]:

1. a state (value) drawn from a set of possible values

   The state may consist of several distinct data attributes. In this case, the set of possible values is the Cartesian product of the sets of possible values of each attribute.

2. a set of operations that access and/or mutate the state

3. a unique identity (e.g., address in memory)

A Python pbject has one of the two important but nonessential characteristics of objects [13, Ch. 3]. Python does:

4. not enforce encapsulation of the state within the object, instead relying upon programming conventions and name obfuscation to hide private information

5. exhibit an independent lifecycle (i.e., has a different lifetime than the code that created it)

As we see in Chapter 7, each object has a distinct dictionary, the directory, that maps the local names to the data attributes and operations.

Python typically uses dot notation to access an object’s data attributes and operations:

• obj.data accesses the attribute data of obj

• obj.op accesses operation op of obj

• obj.op() invokes operation op of obj, passing any arguments in a comma-separated list between the parentheses

Some objects are immutable and others are mutable. The states (i.e., values) of:

• immutable objects (e.g., numbers, booleans, strings, and tuples) cannot be changed after creation

• mutable objects (e.g., lists, dictionaries, and sets) can be changed in place after creation

Caveat: We cannot modify a Python tuple’s structure (i.e., length) after its creation. However, if the components of a tuple are themselves mutable objects, they can be changed in-place.

All Python objects have a type.

5.3.2 Types

In terms of the discussion in Section {#sec:type-system-concepts}, all Python objects can be considered as having one or more conceptual types at a particular point in time. The types may change over time because the program can change the possible set of data attributes and operations associated with the object.

A Python variable is bound to an object by an assignment statement or its equivalent. Python variables are thus dynamically typed, as are Python expressions.

Although a Python program usually constructs an object within a particular nominal type hierarchy (e.g., as an instance of a class), this may not fully describe the type of the object, even initially. And the ability to dynamically add, remove, and modify attributes (both data fields and operations) means the type can change as the program executes.

The type of a Python object is determined by what it can do—what data it can hold and what operations it can perform on that data—rather than how it was created. We sometimes call this dynamic, structural typing approach duck typing. (If it walks like a duck and quacks like a duck, then it is a duck, even if is declared as a snake.)

For example, we can say that any object is of an iterable type if it implements an __iter__ operation that returns a valid iterator object. An iterator object must implement a __next__ operation that retrieves the next element of the “collection” and must raise a StopIteration exception if no more elements are available.

In Python, we sometimes refer to a type like iterable as a protocol. That is, it is an, perhaps informal, interface that objects are expected to satisfy in certain circumstances.

5.4.1 Singleton types

Python has single-element types used for special purposes. These include None and NotImplemented.

5.4.2 Number types

Core Python supports four types of numbers:

• integers
• real numbers
• complex numbers
• Booleans

    >>> type(1) 
    <class 'int'>
    >>> type(-14) 
    <class 'int'>
    >>> x = 2
    >>> type(x) 
    <class 'int'>
    >>> type(99999999999999999999999999999999)
    <class 'int'>

    >>> type(1.01) 
    <class 'float'>
    >>> type(-14.3) 
    <class 'float'>
    >>> x = 2 
    >>> type(x) 
    <class 'int'>
    >>> y = 2.0
    >>> type(y)
    <class 'float'>
    >>> x == y  # Note result of equality comparison
    True

    >>> type(complex('1+2j')) # real part 1, imaginary part 2
    <class 'complex'>
    >>> complex('1') == 1.0   # Note result of comparison
    True 
    >>> complex('1') == 1     # Note result of comparison
    True 

    >>> type(False) 
    <class 'bool'>
    >>> type(True) 
    <class 'bool'>
    >>> True == 1
    True

5.4.3 Sequence types

A sequence denotes a serially ordered collection of zero or more objects. An object may occur more than once in a sequence.

Python supports a number of core sequence types. Some sequences have immutable structures and some have mutable.

    >>> type('Hello world')
    <class 'str'>
    >>> type("Hi Earth")
    <class 'str'>
    >>> type('''
    ... Can have embedded newlines
    ... ''')
    <class 'str'>

    >>> type(())    # empty tuple
    <class 'tuple'>
    >>> type((1,))  # one-element tuple, note comma
    <class 'tuple'>
    >>> x = (1,'Ole Miss')  # mixed element types
    >>> type(x) 
    <class 'tuple'>
    >>> x[0]        # access element with index 0
    1
    >>> x[1]        # access element with index 1
    'Ole Miss'

    >>> list(range(5))
    [0, 1, 2, 3, 4]
    >>> list(range(1, 5))
    [1, 2, 3, 4]
    >>> list(range(0, 9, 3))
    [0, 3, 6]
    >>> list(range(0, 10, 3))
    [0, 3, 6, 9]
    >>> list(range(5, 0, -1))
    [5, 4, 3, 2, 1]
    >>> list(range(0))
    []
    >>> list(range(1, 0)) 
    []
    >>> list(range(0, 0)) 
    []

    >>> type(b'Hello\n   World!') 
    <class 'bytes>

    >>> type([])
    <class 'list'>
    >>> type([3])
    <class 'list'>
    >>> x = [1,2,3] + ['four','five'] # concatenation
    >>> x
    [1, 2, 3, 'four', 'five']
    >>> type(x)
    <class 'list'>
    >>> y = x[1:3] # get slice of list
    >>> y
    [2, 3]
    >>> y[0] = 3   # assign to list index 0
    [3, 3]

    >>> type(bytearray(b'Hello\n   World!')) 
    <class 'bytes>

5.4.4 Mapping types

Type dict (dictionary) denotes mutable finite sets of key-value pairs, where the key is an index into the set for the value with which it is paired.

The key can be any hashable object. That is, the key can be any immutable object or an object that always gives the same hash value. However, the associated value objects may be mutable and the membership in the set may change.

We can express dictionaries syntactically in various ways such as comma-separated lists of key-value pairs with braces.

    >>> x = { 1 : "one" }
    >>> x
    {1: 'one'}
    >>> type(x)
    <class 'dict'>
    >>> x[1]
    'one'
    >>> x.update({ 2 : "two" }) # add to dictionary
    >>> x
    {1: 'one', 2: 'two'}
    >>> type(x)
    <class 'dict'>
    >>> del x[1]  # delete element with key
    >>> x
    {2: 'two'}

5.4.5 Set Types

A set is an unordered collection of distinct hashable objects.

There are two built-in set types—set and frozenset.

    >>> sx = { 'Dijkstra', 'Hoare', 'Knuth' }
    >>> sx
    {'Knuth', 'Hoare', 'Dijkstra'}
    >>> sy = set(['Knuth', 'Dijkstra', 'Hoare'])
    >>> sy
<   {'Knuth', 'Hoare', 'Dijkstra'}      
    >>> sx == sy
    True
    >>> sx.add('Turing')  # add element to mutable set
    >>> sx
    {'Turing', 'Knuth', 'Hoare', 'Dijkstra'}

    >>> fx = frozenset(['Dijkstra', 'Hoare', 'Knuth'])
    >>> fx
    frozenset({'Knuth', 'Hoare', 'Dijkstra'})       
    >>> fy = frozenset(sy)
    >>> fy
    frozenset({'Knuth', 'Hoare', 'Dijkstra'})
    >>> fx.add('Turing')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'frozenset' object has no attribute 'add'

5.4.6 Other object types

We discuss callable objects (e.g., functions), class objects, module objects, and user-defined types (classes) in later chapters.

TODO: Perhaps be more specific about later chapters.

6.2.1 Simple statements

A simple statment is a statement that does not contain other statements. This subsection examines four simple statements. We discuss other simple statements later in this textbook.

TODO: Make last sentence above more explicit.

    pass

    expression1, expression2, ... 

   target1, target2, ... = expression1, expression2, ...

    >>> x, y = 1, 2 
    >>> x
    1
    >>> y
    2
    >>> x = 1, 2, 3
    >>> x
    (1, 2, 3)
    >>> x, y = 1, 2, 3
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: too many values to unpack (expected 2)

    >>> x, *y, z = 1, 2
    >>> y
    []
    >>> x, *y, z = 1, 2, 3
    >>> y
    [2]
    >>> x, *y, z = 1, 2, 3, 4, 5
    >>> y
    [2, 3, 4]

    del target1, target2, ...

6.2.2 Compound Statements

A compound statment is a construct that contains other statements. This subsection examines three compound statements. We discuss other compound statements later in this textbook.

TODO: Make last sentence above more explicit.

6.2.3 if statement

The if statement is a conditional statement typical of imperative languages. It is a compound statement with the form

    if cond1:
        statement_list1
    elif cond2:    # else with nested if
        statement_list2 
    elif cond3: 
        statement_list3 
    ...
    else:
        statement_listZ

where the elif and else clauses are optional.

When executed, the conditional statement evaluates the cond expressions from top to bottom and executes the corresponding statement_list for the first condition evaluating to true. If none evaluate to true, then the compound statement executes statement_listZ, if it is present.

Note colon terminates each clause header and that the statement_list must be indented. Most compound statements are structured in this manner.

6.2.4 while statement

The while statement is a looping construct with the form

    while cond:
        statement_list1
    else:   # executed after normal exit, not on break
        statement_list2

where the else clause is optional.

When executed, the while repeatedly evaluates the expression cond, and, if the condition is true, then the while executes statement_list1. If the condition is false, then the while executes statement_list2 and exits the loop.

A break statement executed in statement_list1 causes an exit from the loop that does not execute the else clause.

A continue statement executed in statement_list1 skips the rest of statement_list1 and continues with the next condition test.

6.2.5 for statement

The for statement is a looping construct that iterates over the elements of a sequence. It has the form:

    for target_list in expression_list:
        statement_list1
    else:   # executed after normal exit, not on break
        statement_list2

where the else clause is optional.

The interpreter:

• evaluates the expression_list once to get an iterable object (i.e., a sequence)

• creates an iterator to step through the elements of the sequence

• executes statement_list1 once for each element of the sequence in the order yielded by the iterator

  It assigns each element to the target_list (as described above for the assignment statement) before executing statement_list1. The variables in the target_list may appear as free variables in statement_list1.

  The for assigns the target_list zero or more times as ordinary variables in the local scope. These override any existing values. Any final values are available after the loop.

  If the expression_list evaluates to an empty sequence, the the body of the loop is not executed and the target_list variables are not changed.

• executes statement_list2 in the optional else, if present, after exhausting the elements of the sequence

The break and continue statement work as described above for the while statement.

It is best to avoid modifying the iteration sequence in the body of the loop. Modification can result in difficult to predict results.

6.5.1 Using import

Suppose we have the following Python code in a file named testmod.py.

    # This is module "testmod" in file "testmod.py"
    testvar = -1
    
    def test(x):
        return x

We can execute this code in a Python REPL session as follows.

    >>> import testmod   # import module in file "testmod.py"
    >>> testmod.testvar  # access module's variable "testvar"
    -1
    >>> testmod.testvar = -2  # set variable to new value
    >>> testmod.testvar
    -2
    >>> testmod.test(23) # call module's function "test"
    23
    >>> test(2)          # must use module prefix "test"
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: 'module' object is not callable
    >>> testmod          # below PATH = directory path 
    <module 'testmod' from 'PATH/testmod.py'>
    >>> type(testmod)
    <class 'module'>
    >>> testmod.__name__
    'testmod' 
    >>> type(type(testmod))
    <class 'type'>

The import statement causes the interpreter to execute all the top-level statements from the module file and makes the namespace available for use in the script or another module. In the above, the imported namespace includes the variable testvar and the function definition test.

A name from one module (e.g., testmod) can be directly accessed from an imported module by prefixing the name by the module name using the dot notation. For example, testmod.testvar accesses variable testvar in module testmod and testmod.test() calls function test in module testmod.

We also see that the imported module testmod is an object of type (class) module.

6.5.2 Using from import

We can also import names selectively. In this case, the definitions of the selected features are copied into the module.

Consider the module testimp below.

    # This is module "testimp" in file "testimp.py"
    from testmod import testvar, test

    myvar = 10

    def myfun(x, y, z):
        mylocvar = myvar + testvar
        return mylocvar
    
    class P:
        def __init__(self): 
            self.my_loc_var = None 

        def meth1(self, arg): 
            return test(arg)

        def meth2(self, arg): 
            if arg == None:
                return None
            else:
                my_loc_var= arg
                return arg

The definitions of variable testvar and function test are copied from module testmod into module testimp’s namespace. Module testimp can thus access these without prefix testmod.

Module testimp could import all of the definitions from module testmod by using the wildcard * instead of the explicit list.

We can execute the above code in a Python REPL session as follows.

    >>> import testimp
    >>> testimp.myvar
    10
    >>> testimp.myfun(1,2,3)
    9
    >>> pp = testimp.P()
    >>> pp.meth1(23)
    23
    >>> pp.meth2(14)
    14
    >>> type(pp)
    <class 'testimp.P'>
    >>> type(testimp.testmod) 
    Traceback (most recent call last): 
      File "<stdin>", line 1, in <module>
    NameError: name 'testmod' is not defined

Note that the from testmod import statement does not create an object testmod.

6.5.3 Programming conventions

Python programs typically observe the following conventions:

• All module import and import from statements should appear at the beginning of the importing module.

• All from import statements should specify the imported names explicitly rather than using the wildcard * to import all names. This avoids polluting the importing module’s namespace with unneeded names. It also makes the dependencies explicit.

• Any definition whose name begins with an _ (underscore) should be kept private to a module and thus should not be imported into or accessed directly from other modules.

6.5.4 Using importlib directly

TODO: Perhaps move the discussion below of the importlib a metaprogramming feature, to a later chapter that deals with metaprogramming?

The Python core module importlib exposes the functionality underlying the import statement to Python programs. In particular, we can use the function call

    importlib.import_module('modname') # argument is string

to find and import a module from the file named modname.py. Below we see that this works like an explicit import.

    >>> from importlib import import_module
    >>> tm = import_module('testmod')
    >>> tm          # below PATH = directory path 
    <module 'testmod' from 'PATH/testmod.py'>
    >>> type(tm)
    <class 'module'>
    >>> type(type(tm))
    <class 'type'>

7.5.1 Inheritance

In programming languages in general, inheritance involves defining hierarchical relationships among classes. From a pure perspective, a class C inherits from class P if C’s objects form a subset of P’s objects in the following sense:

• Class C’s objects must support all of class P’s operations (but perhaps are carried out in a special way).

  We can say that a C object is a P object or that a class C object can be substituted for a class P object whenever the latter is required.

• Class C may support additional operations and an extended state (i.e., more data attributes fields).

We use the following terminology.

• Class C is called a subclass or a child or derived class.

• Class P is called a superclass or a parent or base class.

• Class P is sometimes called a generalization of class C; class C is a specialization of class P.

In terms of the discussion in the Type System Concepts section, the parent class P defines a conceptual type and child class C defines a behavioral subtype of P’s type. The subtype satisfies the Liskov Substitution Principle [24,39].

Even in a statically typed language like Java, the language does not enforce this subtype relationship. It is possible to create subclasses that are not subtypes. However, using inheritance to define subtype relationships is considered good object-oriented programming practice in most circumstances.

In a dynamically typed like Python, there are fewer supports than in statically typed languages. But using classes to define subtype relationships is still a good practice.

The importance of inheritance is that it encourages sharing and reuse of both design information and program code. The shared state and operations can be described and implemented in parent classes and shared among the child classes.

The following code fragment shows how to define a single inheritance relationship among classes in Python. Instance method process is defined in the parent class P and overridden (i.e., redefined) in child class C but not overriden in child class D. In turn, C ’s instance method process is overridden in its child class G.

    class P:
        def __init__(self,name=None): 
            self.name = name 
        def process(self):
            return f'Process at parent P level'

    class C(P):   # class C inherits from class P
        def process(self):
            result = f'Process at child C level'
            # Call method in parent class
            return f'{result} \n  {super().process()}'
            
    class D(P):   # class D inherits from class P
        pass

    class G(C):   # class G inherits from class C
        def process(self):
            return f'Process at grandchild G level'

Now consider a (lengthy) Python REPL session with the above class definition.

    >>> p1 = P()
    >>> c1 = C()
    >>> d1 = D()
    >>> g1 = G()
    >>> p1.process()
    'Process at parent P level'
    >>> c1.process()
    'Process at child C level'
    'Process at parent P level'
    >>> d1.process()
    'Process at parent P level' 
    >>> g1.process()
    'Process at grandchild G level'
    #
    >>> type(P) 
    <class 'type'>
    >>> type(C) 
    <class 'type'>
    >>> type(G) 
    <class 'type'>
    >>> issubclass(P,object)
    True
    >>> issubclass(C,P)
    True
    >>> issubclass(G,C)
    True
    >>> issubclass(G,P)
    True
    >>> issubclass(G,object)
    True
    >>> issubclass(C,G)
    False
    >>> issubclass(G,D)
    False
    >>> issubclass(P,type)
    False
    >>> isinstance(P,type)
    True
    >>> isinstance(C,type)
    True
    >>> isinstance(G,type) 
    True
    >>> type(type)
    <class 'type'>
    >>> issubclass(type,object)
    True
    >>> isinstance(type,type) 
    True 
    >>> type(object)
    <class 'type'>
    >>> isinstance(object,type) 
    True 
    >>> issubclass(object,type)
    False

Note: The Python 3.7+ source code for the above version of the P class hierarchy is available in file inherit1.py.

7.5.1.1 Understanding relationships among classes

By examining the REPL session above, we can observe the following:

• Top-level user-defined classes like P implicitly inherit from the object root class. They have the issubclass relationship with object.

• A user-defined subclass like C inherits explicitly from its superclass P, which inherits implicitly from root class object. Class C thus has issubclass relationships with both P and object.

• By default, all Python classes (including subclasses) are instances of the root metaclass type (or one of its subtypes as we see later). But non-class objects are not instances of type.

As we noted in Chapter 6, we call class objects metaobjects; they are constructors for ordinary objects [1,14].

Also as we noted in Chapter 6, we call special class objects like type metaclasses; they are constructors for metaobjects (i.e., class objects) [1,14].

Note that classes object and type have special – almost “magical” – relationships with one another [31:593–595].

• Class object is an instance of class type (i.e., it is a Python class object).

• Class type is an instance of itself (i.e., it is a Python class object) and a subclass of class object.

The diagram in Figure 7.1 shows the relationships among user-definfed class P and built-in classes int, object, and type. Solid lines denote subclass relationships; dashed lines denote “instance of” relationships.

7.5.2 Subtype polymorphism

The concept of polymorphism (literally “many forms”) means the ability to hide different implementations behind a common interface. As we saw in Chapter 5, polymorphism appears in several forms in programming languages. Here we examine one form.

In the Python class hierarchy example above, the method process forms part of the common interface for this hierarchy. Parent class P defines the method, child class C overrides P ’s definition by refinement, and grandchild class G overrides C ’s definition by replacement. However, child class D does not override P ’s definition.

Subtype polymorphism (sometimes called polymorphism by inheritance, inclusion polymorphism, or subtyping) means the association of an operation invocation (e.g., method call) with the appropriate operation implementation in an inheritance (i.e., subtype) hierarchy.

This form of polymorphism is usually carried out at run time. Such an implementation is called dynamic binding.

In general, given an object (i.e., class instance) to which an operation is applied, the runtime system first searches for an implementation of the operation associated with the object’s class. If no implementation is found, the system checks the parent class, and so forth up the hierarchy until it finds an implementation and then invokes it. Implementations of the operation may appear at several levels of the hierarchy.

The combination of dynamic binding with a well-chosen inheritance hierarchy allows the possibility of an instance of one subclass being substituted for an instance of a different subclass during execution. Of course, this can only be done when none of the extended operations of the subclass are being used.

In a statically typed language like Java, we declare a variable of some ancestor class type. We can then store any descendant class instance in that variable. Polymorphism allows the program to apply any of the ancestor class operations to the instance.

Because of dynamically typed variables, polymorphism is even more flexible in Python than in Java.

In Python, an instance object may also have its own implementation of a method, so the runtime system searches the instance before searching upward in the class hierarchy.

Also (as we noted in an earlier section) Python uses duck typing. Objects can have a common interface even if they do not have common ancestors in a class hierarchy. If the runtime system can find an compatible operation associated with an instance, it can execute it.

Thus Python’s approach to subtype polymorphism gives considerable flexibility in structuring programs. However, unlike statically typed languages, the compiler provides little help in ensuring the compatibility of method implementations.

Again consider the simple inheritance hierarchy above in the following Python REPL session.

    >>> d1 = D()
    >>> g1 = G()
    >>> obj = d1     # variables support polymorphism
    >>> obj.process()
    'Process at parent P level' 
    >>> obj = g1     # variables support polymorphism
    >>> obj.process()
    'Process at grandchild G level' 

7.5.3 Multiple Inheritance

TODO: Discuss multiple inheritance. Issues include the diamond problem, Python syntax and semantics, and method resolution order.

9.1.1 Motivating example

Suppose we have a Python function add:

    def add(x, y):
        'Add x and y' 
        return x + y

A simple way we can approach debugging is to insert a print statement into the function to trace execution, as follows:

    def add(x, y):
        'Add x and y' 
        print('add')
        return x + y

However, suppose we need to debug several similar functions simultaneously. Following the above approach, we might have code similar to that in the example below.

    def add(x, y):
        'Add x and y' 
        print('add')
        return x + y

    def sub(x, y):
        'Subtract y from x' 
        print('sub')
        return x - y

    def mul(x, y):
        'Multiply x and y' 
        print('mul')
        return x * y

    def div(x, y):
        'Divide x by y' 
        print('div')
        return x / y

We insert basically the same code into every function.

This code is unpleasant because it violates the Abstraction Principle.

9.1.2 Abstraction Principle, staying DRY

The Abstraction Principle states, “Each significant piece of functionality in a program should be implemented in just one place in the source code.” [29:339]. If similar functionality is needed in several places, then the common parts of the functionality should be separated from the variable parts.

The common parts become a new programming abstraction (e.g., a function, class, abstract data type, design pattern, etc.) and the variable parts become different ways in which the abstraction can be customized (e.g.,, its parameters).

The approach encourages reuse of both design and code. Perhaps more importantly, it can make it easier to keep the similar parts consistent as the program evolves.

Andy Hunt and Dave Thomas [20:26–33] articulate a more general software development principle Don’t Repeat Yourself, known by the acronym DRY.

In an interview [38], Thomas states, “DRY says that every piece of system knowledge should have one authoritative, unambiguous representation. … A system’s knowledge is far broader than just its code. It refers to database schemas, test plans, the build system, even documentation.”

Our goal is to keep our Python code DRY, not let it get WET (“Write Everything Twice” or “Wasting Everyone’s Time” or “We Enjoy Typing” [47].)

9.1.3 Function decorators

To introduce an appropriate abstraction into the previous set of functions, we can use a Python function decorator.

A function decorator is a higher-order function that takes a function as its argument, wraps another function around the argument, and returns the wrapper function.

The wrapper function has the same parameters and same return value as the function it wraps, except it does extra processing when it is called. That is, it “decorates” the original function.

TODO: Show Wikipedia citation because of link.

Remember that Python functions are objects. Python’s decorator function concept is thus a special case of the Decorator design pattern, one of the classic Gang of Four patterns for object-oriented programming [16]. The idea of this pattern is to wrap one object with another object, the decorator, that has the same interface but enhanced behavior. The decoration is usually done at runtime even in a statically typed language like Java.

TODO: Perhaps expand on the Decorator design pattern and give a diagram.

9.1.4 Constructing a debug decorator

In the motivating example above, we want to decorate a function like add(x,y) by wrapping it with another function that prints the function name add before doing the addition operation. The wrapped function can then take the place of the original add in the program.

Let’s construct an appropriate decorator in steps.

In general, suppose we want to decorate a function named func that takes some number of positional and/or keyword arguments. That is, the function has the general signature:

    func(*args, **kwargs)

Note: For more information on the above function calling syntax, see the discussion on Function Calling Conventions in Chapter 6.

In addition, suppose we want to print the content of the variable msg before we execute func.

As our first step, we define function wrapper as follows:

    def wrapper(*args, **kwargs): 
        print(msg) 
        return func(*args, **kwargs)

As our second step, we define a decorator function debug that takes a function func as its argument, sets local variable msg to func’s name, and then creates and returns the function wrapper.

Function debug can retrieve the function name by accessing the __qualname__ attribute of the func object. Attribute __qualname__ holds the fully qualified name.

    def debug(func): 
        msg = func.__qualname__ 
        def wrapper(*args, **kwargs): 
            print(msg) 
            return func(*args, **kwargs)
        return wrapper 

Function debug returns a closure that consists of the function wrapper plus the the local environment in which wrapper is defined. The local environment includes the argument func and the variable msg and their values.

Note: For more information about the concepts and techniques used above, see the discussion of Nested Function Definitions, Lexical Scope, and Closures in Chapter 6.

It seems sufficient to assign the closure returned by debug to the name of func as shown below for add.

    def add(x, y):
        'Add x and y' # docstring (documentation) 
        return x + y
    add = debug(add)

But this does not work as expected as shown in the following REPL session.

    >>> add(2,5)
    add
    7
    >>> add.__qualname__
    debug.<locals>.wrapper
    >>> add.__doc__
    None

The closure returned by debug computes the correct result. However, it does not have the correct metadata, as illustrated above by the display of the name (__qualname__) and the docstring (__doc__) metadata.

To make the use of the decorator debug transparent to the user, we can apply the function decorator @wraps defined in the standard module functools as follows.

    def debug(func): 
        msg = func.__qualname__ 
        @wraps(func) 
        def wrapper(*args, **kwargs): 
            print(msg) 
            return func(*args, **kwargs)
        return wrapper

    def add(x, y):
        'Add x and y' # docstring (documentation) 
        return x + y
    add = debug(add)

The @wraps(func) decorator call above sets function wrapper’s metadata — it’s attributes __module__, __name__, __qualname__, __annotations__, and __doc__ — to the same values as func’s metadata.

With this new version of the debug decorator, the decoration of add now works transparently.

    >>> add(2,5)
    add
    7
    >>> add.__qualname__    add
    >>> add.__doc__
    Add x and y

Finally, because the definition of a function like add and the application of the debug decorator function usually occur together, we can use the decorator syntactic sugar @debug to conveniently designate the definition of a decorated function. The debug decorator function can be defined in a separate module.

    @debug
    def add(x, y):
        'Add x and y'
        return x + y

9.1.5 Using the debug decorator

Given the debug decorator as defined in the previous subsection, we can now simplify the motivating example.

We decorate each function with @debug but give no other details of the implementation here. The debug facility is implemented in one place but used in many places. The implementation supports the DRY principle.

    @debug 
    def add(x, y):
        'Add x and y' 
        return x + y

    @debug 
    def sub(x, y):
        'Subtract y from x' 
        return x - y

    @debug 
    def mul(x, y):
        'Multiply x and y' 
        return x * y

    @debug
    def div(x, y):
        'Divide x by y' 
        return x / y

Note: The Python 3.7+ source code for the above version of debug is available in linked file debug4.py.

9.1.6 Case study review

So far in this case study, we have implemented a simple debugging facility that:

• is implemented once in a place separate from its use

• is thus easy to modify or disable totally

• can be used without knowing its implementation details

9.1.7 Variations

Now let’s consider a couple of variations of the debugging decorator implementation.

      from functools import wraps
      import logging  # (1) logging module

      def debug(func):
          # (2) get the Logger for func's module
          log = logging.getLogger(func.__module__)
          msg = func.__qualname__
          @wraps(func)
          def wrapper(*args, **kwargs):
              log.debug(msg)  # (3) log msg
              return func(*args, **kwargs)
          return wrapper

    from functools import wraps
    import os  # (1) import os interface
    
    def debug(func):
        # (2) debug only if environment variable set
        if 'DEBUG' not in os.environ:
            return func
        msg = func.__qualname__
        @wraps(func)
        def wrapper(*args, **kwargs):
            print(msg)
            return func(*args, **kwargs)
        return wrapper

9.2.1 Motivating example

Suppose, for whatever reason, we want to add a prefix string to the debugging message that may differ from one use of @debug to another. Again consider the set of arithmetic functions.

    def add(x, y):
        'Add x and y' 
        print('***add')
        return x + y

    def sub(x, y):
        'Subtract y from x' 
        print('@@@sub') 
        return x - y

    def mul(x, y):
        'Multiply x and y' 
        print('***sub') 
        return x * y

    def div(x, y):
        'Divide x by y' 
        print('div')
        return x / y

We implement the needed capability by using function decorators with arguments.

9.2.2 Decorators with arguments

We can construct decorators that take arguments other than the function to be decorated.

Consider the following use of decorator deco:

    @deco(args)
    def func():
        # some body code

The above translates into the following decorator call and assignment:

    func = deco(args)(func)

The right-hand side denotes the chaining of two function calls. The system first calls function deco passing it the first argument list (args). This call returns a function, which is in turn called with the second argument list, variable (func).

The outer function call establishes a local environment in which the variables in args are defined. In this environment, we define a normal decorator as we did before.

9.2.3 Prefix decorator

We can thus define the outer layer of a prefix decorator with a function with parameter prefix that defaults to the empty string.

    def debug(prefix=''):
        def deco(func):
            # normal debug decorator body
        return deco

The full definition of the prefix decorator is shown below. If no argument is given to debug, the behavior is (almost) the same as the previous debug decorator function.

    from functools import wraps

    def debug(prefix=''):
        def deco(func):
            msg = prefix + func.__qualname__
            @wraps(func)
            def wrapper(*args, **kwargs):
                print(msg)
                return func(*args, **kwargs)
            return wrapper
        return deco

In this formulation, prefix can be given as either a positional or keyword argument.

We can apply the new prefix debug decorator to our motivating example functions as follows. Note that the prefix strings vary among the different occurrences.

    @debug(prefix='***') 
    def add(x,y):
        'Add x and y' 
        return x+y

    @debug(prefix='@@@') 
    def sub(x, y):
        'Subtract y from x' 
        return x - y

    @debug('***') 
    def mul(x, y):
        'Multiply x and y' 
        return x * y

    @debug()  # parentheses needed!
    def div(x, y):
        'Divide x by y' 
        return x / y

Note: The Python 3.7+ source code for the above version of debugprefix is available in linked file debugprefix1.py.

9.2.4 Reformulated prefix decorator

By a clever use of default arguments and partial application of a function to its arguments, we can transform the definition of the prefix decorator above to one that does not involve a nested definition.

    from functools import wraps, partial

    def debug(func = None, *, prefix = ''): 
        if func is None:
            return partial(debug, prefix=prefix)
        msg = prefix + func.__qualname__
        @wraps(func)
        def wrapper(*args, **kwargs):
            print(msg)
            return func(*args, **kwargs)
        return wrapper

If we call the debug decorator function with the single keyword argument prefix, then the func argument defaults to None. In this case, the if statement causes debug to call itself with that prefix argument and the decorated function (that follows the @debug annotation in the user-level code or occurs in a second argument list) as the func argument.

Note: The functools.partial function takes a function (object) and a group of positional and/or keyword arguments, partially applies the function to those arguments, then returns the resulting function (object). The returned function behaves like the original function except that it has the argument values supplied to partial as its default parameter values.

If we call the debug decorator function with no keyword arguments, then parameter prefix defaults to the empty string and func is the decorated function (e.g., that follows the @debug annotation).

If we call debug with both func and prefix arguments, then it works as we expect. This case is not used with the @debug annotation.

    @debug(prefix='***')
    def add(x,y):
        'Add x and y' 
        return x+y

    @debug(prefix='@@@') 
    def sub(x, y):
        'Subtract y from x'
        return x - y

    @debug(prefix='***') 
    def mul(x, y):
        'Multiply x and y' 
        return x * y

    @debug  # no parentheses required, but okay if given
    def div(x, y):
        'Divide x by y' 
        return x / y

Unlike the previous formulation of the prefix decorator, the prefix string must be supplied as a prefix argument.

Note: The Python 3.7+ source code for the above version of debugprefix is available in linked file debugprefix2.py.

9.3.1 Motivating example

Consider the class Account below for a simple bank account.

Suppose we want to debug all the methods using the simple debugging package we developed above.

    class Account:
        def __init__(self):
            self._bal = 0

        @debug 
        def deposit(self,amt):
            self._bal += amt

        @debug 
        def withdraw(self,amt):
            if amt <= self._bal:
                self._bal -= amt
            else:
                print(f'Insufficient funds for withdrawal of {amt}')

        @debug 
        def get_balance(self):
            return self._bal

        def __str__(self):
            return f'Account with balance {self._bal}'

Note: The Python 3.7+ source code for the above version of Account is available in linked file account2.py.

9.3.2 Class-level debugger

The Account example above is repetitive (not DRY). Can we do the decoration all at once?

Yes, we can define a class decorator debugmethods as shown below (where debug is the function-level prefix decorator defined above). A class decorator is a higher-order function that takes a class as its argument, modifies the class in some way, and then returns the modified class.

    def debugmethods(cls):                     # Notes
        for name, val in vars(cls).items():    # (1) (2) (3)
            if callable(val):                  # (4)
                setattr(cls, name, debug(val)) # (5) (6)
        return cls                             # (7)

The idea here is that the program walks through the class dictionary, identifies callable objects (e.g., methods), and wraps each with a function decorator.

Consider the numbered comments in the above code.

1. The built-in function call vars(cls) returns the dictionary (i.e., __dict__) associated with the (class) object cls.

2. The dictionary method call items() returns the list of key-value pairs in the dictionary.

3. The “for name, val in” statement loops through the pairs in the list, successively binding each key to name and value to val.

4. The built-in function call callable(val) returns True if val appears callable, False if not. (These are likely instance methods.)

5. The call debug(val) applies the function-level prefix debugger we defined above to the method val. That is, it wraps the method with function decorator debug.

6. The built-in function call setattr(cls, name, debug(val)) sets the name attribute of object cls (i.e., in its dictionary) to the value debug(val).

7. The decorator function debugmethods returns the modified class object cls in place of the original class.

The code below shows the application of this new decorator to the Account class.

    @debugmethods
    class Account:
        def __init__(self):
            self._bal = 0

        def deposit(self,amt):
            self._bal += amt

        def withdraw(self,amt):
            if amt <= self._bal:
                self._bal -= amt
            else:
                print(f'Insufficient funds for withdrawal of {amt}')

        def get_balance(self):
            return self._bal

        def __str__(self):
            return f'Account with balance {self._bal}'

Note: The Python 3.7+ source code for the above version of Account is available in linked file account3.py.

A single decorator application handles all the method definitions within the class.

Well, not quite!

It does not decorate class or static methods, such as the following which can be added to class Account.

    class Account:
    ... 
        @classmethod
        def classname(cls):
            return cls.__name__

        @staticmethod
        def warn(msg):
            print(f'Warning: {msg}')

Note: The Python 3.7+ source code for the extended version of Account is available in linked file account4.py.

TODO: Explain why this does not work.

9.3.3 Variation: Attribute access debugging

Suppose instead of printing a message on every call of a method, we do so for each access to an attribute.

We can do this by rewriting part of the class as shown below. In particular, we give a new implementation for the special method __getattribute__.

    def debugattr(cls):
        orig_getattribute = cls.__getattribute__

        def __getattribute__(self, name):
            print(f'Get: {name}')
            return orig_getattribute(self, name)

        cls.__getattribute__ = __getattribute__
        return cls

The special method __getattribute__ is called to implement accesses to “regular” attributes of the class. It is not called on accesses to other special methods such as __init__ and __str__.

In the above, we save the original implementation of the method and then call it to complete the access once we have printed an appropriate debugging message.

In the example below, we decorate the Account class with @debugattr.

    @debugattr
    class Account:
        def __init__(self):
            self._bal = 0

        def deposit(self,amt):
            self._bal += amt

        def withdraw(self,amt):
            if amt <= self._bal:
                self._bal -= amt
            else:
                print(f'Insufficient funds for withdrawal of {amt}')

        def get_balance(self):
            return self._bal

        def __str__(self):
            return f'Account with balance {self._bal}'

Note: The Python 3.7+ source code for the above version of Account is available in linked file account5.py.

We can see the effects of the decorator in the following REPL session.

    >>> acct = Account()
    >>> str(acct)
    Get: _bal
    'Account with balance 0' 
    >>> acct.deposit(100)
    Get: deposit
    Get: _bal 
    >>> str(acct)   
    Get: _bal 
    'Account with balance 100' 
    >>> acct.withdraw(60)
    Get: withdraw
    Get: _bal
    Get: _bal
    >>> str(acct)   
    Get: _bal 
    'Account with balance 40' 
    >>> acct.get_balance()
    Get: get_balance
    Get: _bal
    40
    >>> str(acct)
    'Account with balance 40' 

Note that both calls to the methods and the accesses to the “private” data attribute _bal are shown. (If we want to exclude accesses to the private instance variables, we can modify debugattr to exclude attributes whose names begin with a single underscore.)

9.4.1 Motivating example

Now let’s set up class-level debugging on the inheritance hierarchy P example from Chapter 7.

    @debugmethods 
    class P: 
        def __init__(self,name=None): 
            self.name = name 
        def process(self): 
            return f'Process at parent P level' 

    @debugmethods 
    class C(P):   # class C inherits from class P
        def process(self):
            result = f'Process at child C level'
            # Call method in parent class
            return f'{result} \n  {super().process()}'
            
    @debugmethods 
    class D(P):   # class D inherits from class P
        pass

    @debugmethods 
    class G(C):   # class G inherits from class C
        def process(self): 
            return f'Process at grandchild G level'

Note: The Python 3.7+ source code for the above version of the P class hierarchy is available in linked file inherit2.py.

So, we have another occurrence of code redundancy that we saw at the class level in the previous section. Let’s see if we can DRY out the code more.

To do this, the program needs to process the whole class hierarchy rooted at class P. Let’s review the nature of the Python object model to see how to do this.

9.4.2 Review of objects and types

In Chapters 5-7 of these notes, we learned:

• All Python values are objects.

• All objects have types.

• A class defines a new type.

• A class is a callable (i.e., function) that creates instances; the class is the type of the instances it creates. Hence, in some sense, a class is a type consisting of its potential instances and the operations it defines.

• A class itself is an object. It is an instance of other classes. Thus it has a type.

• The built-in class type is the root class (i.e., top-level metaclass) for all other classes (i.e., types). When a program invokes type as a constructor, it creates a new type (i.e., class) object.

• Classes may inherit (i.e., be a subclass of) other classes.

• The built-in class object is the root class for all other top-level user-defined and built-in classes.

TODO: Maybe repeat diagram below here.

Note: See the diagram in Figure 7-1 from Chapter 7.

The following Python REPL session illustrates these concepts.

    >>> class PP:
    ...     pass
    ... 
    >>> class CC(PP):
    ...     pass
    ...
    >>> PP
    <class '__main__.PP'>
    >>> type(PP) 
    <class 'type'>
    >>> issubclass(P,object) 
    True 
    >>> CC
    <class '__main__.CC'>
    >>> type(CC) 
    <class 'type'>
    >>> issubclass(CC,PP)
    True
    >>> x = PP()
    >>> x
    <__main__.PP object at 0x10cd3d048>
    >>> isinstance(x,PP)
    True
    >>> type(x)
    <class '__main__.PP'>
    >>> type
    <class 'type'>
    >>> type(type)
    <class 'type'>
    >>> issubclass(type,object) 
    True 
    >>> object
    <class 'object'>
    >>> type(object) 
    <class 'type'>

9.4.3 Class definition process

Now let’s examine how the Python interpreter elaborates class definitions at runtime. Consider the class MyClass defined as follows:

    class MyClass(Parent):
        def __init__(self, id): 
            self.id = id 
        def hello(self): 
            print(f'Hello from MyClass.hello, id = {self.id}')"

This class definition has three components.

• Name: "MyClass"

• Base classes: (Parent,)

• Functions: (___init___, hello)

The interpreter takes the following steps during class definition.

1. It isolates the body of the class. (Note the multiline string below.)

       body = '''
       def __init__(self, myid): 
           self.myid = myid 
       def hello(self): 
           print(f'Hello from MyClass.hello, myid = {self.myid}')
       '''

2. It creates the class dictionary.

       clsdict = type.__prepare__('MyClass', (Parent,))

   Method type.__prepare__ is a class method on the root metaclass type. In the process of creating the new class object for a class, the interpreter calls the __prepare__ method before it calls the __new__ method on type [31:701–3].

   In addition to metaclass argument (i.e., type), the __prepare__ class method takes two additional arguments:

   • the name of the class being created (e.g.,'MyClass' above)

   • a tuple of the one or more base classes (e.g., (Parent,) above)

   Method __prepare__ returns a dictionary that can be subsequently passed to the __new__ and __init__ methods. This dictionary serves as the local namespace for the statements in the class body.

3. It executes the body dynamically (using exec) in the current global namespace (returned by the call globals()) and the local namespace defined by the class dictionary clsdict.

       exec(body, globals(), clsdict)

   This step populates clsdict.

       >>> clsdict
       {'__init__': <function __init__ at 0x10cc21ea0>, 
        'hello': <function hello at 0x10d2b9bf8>}

4. It constructs the class object using its name, its base classes, and the dictionary populated in the previous step.

       >>> MyClass = type('MyClass', (Parent,), clsdict)
       >>> MyClass
       <class '__main__.MyClass'>
       >>> mc = MyClass('Conrad')
       <__main__.MyClass object at 0x100f96c50>
       >>> mc.myid
       Conrad
       >>> mc.hello()
       Hello from MyClass.hello, myid = Conrad

   The call type('MyClass', (Parent,), clsdict) constructs an instance of metaclass type with name MyClass, superclass Parent, and object dictionary clsdict. This is the class object for MyClass.

Note: The Python 3.7+ source code for the above creation of class MyClass is available in linked file MyClass1.py.

9.4.4 Changing the metaclass

A Python class definition has a keyword parameter named metaclass whose default value is type. So the parent class P from the motivating example for this section is equivalent to the following.

    class P(metaclass=type):
        def __init__(self,name=None): 
            self.name = name 
        def process(self): 
            return f'Process at parent P level' 

This keyword parameter sets the class for creating the new type for the class. Although the default is type, we can change it to some other metaclass.

To define a new metaclass, we typically define a type that inherits from type and gives a new definition for one or both of the special methods __new__ and __init__.

    class mytype(type):
        def __new__(cls, name, bases, clsdict):
            # possible preprocessing of arguments
            clsobj = super().__new__(cls, name, bases, clsdict)
            # possible postprocessing of object
            return clsobj

The special method __new__ allocates memory, constructs a new instance (i.e., object), and then returns it. The interpreter passes this new instance to special method __init__, which initializes the new instance variables.

We do not normally override __new__, but in a metaclass we may want to do some additional work either before or after the basic construction processing.

A metaclass can access information about a class definition at the time the class is defined. It can inspect the data and, if needed, modify the data.

Given the above definition, we can use the new metaclass as follows:

    class P(metaclass=mytype):
        ...

9.4.5 Debugging using a metaclass

Now we have the tools we need to remove the code redundancy from the motivating example. We can introduce the metaclass shown in the example below.

    class debugmeta(type):
        def __new__(cls, clsname, bases, clsdict):
            clsobj = super().__new__(cls,clsname,bases,clsdict)  #1
            clsobj = debugmethods(clsobj)                        #2
            return clsobj                                        #3

The approach above:

1. creates the class normally (using super().__new__)

2. immediately wraps the class object with the class-level debug decorator debugmethods we developed previously

3. then returns the wrapped class object

Given the above metaclass definition, we can apply it to the inheritance example as sketched below.

    class P(metaclass = debugmeta):
        ...
    class C(P):
        ...
    class D(P): 
        ... 
    class G(C):
        ... 

Note: The Python 3.7+ source code for the above version of the P class hierarchy is available in linked file inherit3.py.

Now consider a Python REPL session using the above code with the custom metaclass.

    >>> from inherit3 import *
    >>> type(P) 
    <class 'inherit3.debugmeta'>
    >>> issubclass(P,object) 
    True 
    >>> type(C)
    <class 'inherit3.debugmeta'>
    >>> issubclass(C,P) 
    True 
    >> type(G) 
    <class 'inherit3.debugmeta'>
    >>> issubclass(G,C) 
    True 
    >>> issubclass(G,P) 
    True 
    >>> p1 = P()
    P.__init__
    >>> type(p1) 
    <class 'inherit3.P'>
    >>> c1 = C() 
    P.__init__
    >>> type(c1)
    <class 'inherit3.C'>
    >>> g1 = G()
    P.__init__
    >>> type(g1)
    <class 'inherit3.C'>
    >>> p1.process()
    P.process
    'Process at parent P level'
    >>> c1.process()
    C.process
    P.process
    'Process at child C level \n  Process at parent P level'
    >>> g1.process()
    G.process
    'Process at grandchild G level'

9.4.6 Why metaclasses?

As we have seen, we can transform a class in similar ways using either a class decorator or a metaclass.

Given that a class decorator is easier to set up and apply, when and why should we use a metaclass?

One advantage to metaclasses is that they can propagate down class <hierarchies. Consider our motivating example again.

    class P(metaclass = debugmeta):
        ...
    class C(P):   # metaclass = debugmeta
        ...
    class D(P):   # metaclass = debugmeta
        ... 
    class G(C):   # metaclass = debugmeta
        ... 

As we can see in the REPL session output in the previous subsection, use of the metaclass in parent class P is passed down automatically to all its descendants. No changes are needed to the descendant classes.

In some sense, the metaclass mutates the DNA of the parent class and that mutation is passed on to the children. In this example, debugging is applied across the entire hierarchy. The code is kept DRY.