29 April 2018
Advisory: The HTML version of this document requires use of a browser that supports the display of MathML. A good choice as of April 2018 is a recent version of Firefox from Mozilla.
Basically, metaprogramming is writing code that writes code.
We often do metaprogramming in our routine programming tasks but do not call it that.
Our web applications may generate HTML, JavaScript, and CSS code to enable development of a browser-based user interface.
Our Java programs may use instanceof to check the type of objects or otherwise manipulate itself with the Java reflection package.
Our C programs may use macros to define new features in terms of existing features.
Under the above definition, much of our study of domain-specific languages uses metaprogramming.
The pic little language processor takes a program expressed in an external textual language that describes a picture and generates output expressed in another language that gives instructions to a display program [Kernighan 1984].
Several of the State Machine DSL processors read and parse a program written in a special-purpose textual language, represent the program internally in a semantic model, and then “execute” the model on inputs.
Other of the State Machine DSL processors use the semantic model to generate a “program” in another language such as C or the Graphviz dot language for graphs.
The Computer Configuration and Email Message internal DSLs use the host language itself to encode special-purpose languages. The processors can then read and parse descriptions written in these special-purpose languages and manipulate the resulting data structures similarly to the external DSLs.
The Lua- and Python-hosted Lair Configuration and the Ruby-hosted Survey internal DSLs manipulate the structure of the processing programs themselves to implement the special-purpose language.
The internal Survey DSL and Lair Configuration DSL are examples of reflexive (or reflective) metaprogramming.
This manipulation may be at compile time, involving a phase of transformations in the code before the final program is generated. Or it may be at runtime, involving manipulation of the program’s metamodel or generation of new code that is dynamically executed within the program.
The Survey DSL is a Ruby internal DSL. It takes advantage of Ruby’s metaprogramming facilities such as the abilities to trap calls to undefined methods, add methods or variables dynamically to existing objects at runtime, and execute dynamically generated strings as Ruby code. It also uses Ruby’s first-class functions (closures) and flexible syntax – although these are not technically metaprogramming features of Ruby.
The Lair Configuration DSL programs use the metaprogramming features of Lua and Python in similar ways.
Consider relatively common languages and their metaprogramming features.
Java is a statically typed, compiled language. What are metaprogramming features available in Java?
It has dynamic class loaders, a reflection API, annotation processing, dynamic method invocation (a JVM feature), JVM bytecode manipulation (mostly with external tools), etc. Java 8+ also has first-class functions and other features useful in metaprogramming.
Lua is a dynamically typed, interpreted language. What are the metaprogramming features available in Lua?
It has metatables, metamethods, manipulation of environments, a debug library (introspection/reflection features), loadfile and loadstring functions to dynamically execute code, extensions in C, etc.
What about Python 3?
The reflexive metaprogramming features of Python 3.6 and beyond is the primary topic of this set of lecture notes.
In everyday application programming, we often use the products developed by metaprogrammers, but we seldom use the techniques directly.
In everyday programming, use of reflexive metaprogramming techniques should not be one of our first approaches to a problem. We first should explore techniques supported by core language, its standard libraries, and stable extension packages.
If no acceptable solution can be found, then we can consider solutions that use reflexive metaprogramming techniques. We should approach metaprogramming with great care because these techniques can make programs difficult to understand and can introduce vulnerabilities into our programs. We should design, implement, test, and document the programs rigorously.
However, reflexive metaprogramming can be an important tool in a master programmer’s toolbox. If our jobs are to develop software frameworks, libraries, APIs, or domain-specific languages, we can use these techniques and features to develop powerful products that hide the complexity from the application programmer.
Even when our jobs are primarily application programming, understanding reflexive metaprogramming techniques can improve our abilities to use software frameworks, libraries, and APIs effectively.
TODO: Update this to better reflect what the final notes cover and include forward references as appropriate.
The reflexive metaprogramming features of Python 3 include:
exec, eval)We have already used the final two in our implementation of domain-specific languages. We will look at some of the others in these notes. In particular, Chapter 3 looks at use of decorators and metaclasses.
But, in the next chapter, let’s first examine the basic features of Python 3 upon which the reflexive metaprogramming features build.
TODO: Decide if any are appropriate.
I developed these notes in Spring 2018 for use in CSci 658 Software Language Engineering. The Spring 2018 version used Python 3.6.
Teaching a special topics course on “Ruby and Software Development” in Fall 2006 kindled my interests in domain-specific languages and metaprogramming. Building on these interests, I taught another special topics course on “Software Language Engineering” in which I focused on Martin Fowler’s work on domain-specific languages [Fowler 2011]; I subsequently formalized this as CSci 658. (I have collected some overall ideas on Domain-Specific Languages in a separate set of notes [Cunningham 2018c].)
The overall set of notes on Python 3 Reflexive Metaprogramming is inspired by David Beazley’s Python 3 Metaprogramming tutorial from PyCon’2013 [Beazley 2013a]. In particular, some chapters adapt Beazley’s examples. Beazley’s tutorial draws on material from his and Brian K. Jones’ book Python Cookbook [Beazley 2013b].
Chapter 1 of the notes subsumes my previous notes on Metaprogramming [Cunningham 2018b].
I maintain these notes as text in Pandoc’s dialect of Markdown using embedded LaTeX markup for the mathematical formulas and then translate the notes to HTML, PDF, and other forms as needed.
TODO: Update
Metaprogramming, reflexive metaprogramming.