Meta-programming vs. the mental gap

Boris Kolar wrote favorably about meta-programming in his comments to my previous post.

Meta-programming is, by definition, writing code that writes code. The very need to write code that writes more code is something that should be avoided - pre-empted - at the time of programming language design. Show me an example where you need to use meta-programming, and I will show you a language that lacks an essential feature for which your meta-programming is a workaround.

The essential problem of software development is not that programmers cannot churn out code, and so they need tools to help them churn out more code automatically. The essential crisis of programming today is that all code written nowadays has severe deficiencies in security and reliability. The problem is not that programmers are unable to create software. The problem is they are unable to create software that really does what they wanted it to do.

You see, all programs are perfect. All programs are complete, and they do exactly what you have written them to do. They implement the design you wrote down. The problem is, you don't know what you wrote down.

There is a wide gap between the programmer's expectations of what the code will do, and what the code actually does in reality. This gap is the source of most errors in software, and it comes about as a result of the whole programming environment being non-intuitive and too difficult to use. The languages and tools require the programmer to be 100% conscious of what he's doing at all times; whenever you are 99% or less conscious, you create bugs. And bugs are a problem when you ship to a bit more than just the machines in your living room. They are a problem when your customers depend on your software and cannot use it because of a freak bug. And bugs are a nightmare when they result in security vulnerabilities. It is untenable that a security researcher can easily find a 0-day exploitable vulnerability in any browser, any day. Such software cannot be trusted. We don't have desktop software today that can be trusted - and it's all because of the mental gap.

Now, getting rid of the gap between the programmer's mental model and the code does not call for giving the programmer more freedom. This is exactly the problem of C: no guidance and too much freedom to pursue any path. Closing the gap calls for simplifying the programming language and tools - reducing complexity, closing off the unwise paths - so that code can be reasoned about easily. Doing this in a way that retains all the necessary flexibility and still performs well is a challenge.

.NET and Java go a good way towards solving the mental-gap problem as far as memory management and single-threaded execution are concerned. The problem is, they don't solve the problem for parallel and asynchronous programming. That's what Flow is about. And additionally, it so happens that closing this gap in a way like Flow does also makes applications easier to design in an orthogonal, reusable an extensible manner, improving programmer productivity that way as well.


boris kolar said…
I agree with everything you said and remain convinced that metaprogramming is essential feature that, unless abused, will improve, simplify code and make it easier to reason about. Metaprogramming is not something you need often. It's something you use in <1% to make the rest >99% of the code cleaner.

You are absolutely right that everytime you need metaprogramming, you've just found a feature that is missing in programming language. Without metaprogramming, you have these choices:
1) Do it without metaprogramming, which often require error-prone or repetitive programming (like manually balancing lock/unlock, or saving object state to database after every change)
2) Hope that the next version of programming language will solve your problem (of course, your problem is not the only one, so language will quickly become bloated)
3) Choose another language that is better suited for your problem (but possibly worse for other problems you also need to solve)

Examples, where you need metaprogramming:
- optimize regular expressions at compile time
- convert .SVG graphics to binary format at compile time
- compile critical code to hardware code for programmable CPUs
- support automatic persistence and remote objects
- enable automatic logging for every function in specified namespace
- domain specific languages (language transformations, templating languages,...)
- make compiler complain about missing resources (for example,"images/leftarrow.png"))
- define which objects are accessible from specific namespace ...

Yes, for most of examples there are workarounds (for example, by using Makefile, precompiler or another build tool, all of which is essentially metaprogramming as well). In fact, every large program I've seen lately uses some form of metaprogramming already. (Almost) everytime you type "configure, make, make install", you depend on metaprogramming.

BTW: C# has a half-baked metaprogramming (System.Reflection.Emit), Ruby on Rails heavily depends metaprogramming, Python also has quite powerful metaprogramming. In almost every case, metaprogramming is used to reduce complexity, enhance reliability and improve readability.

Modern development process makes another possibility for securiy failure: a single "evil" team member of otherwise "trusted" company can break security. For example, the guy at Microsoft that wrote calc.exe can destroy your personal data. Metaprogramming could prevent that by denying access to HardDisk for all objects in Windows.Accessories namespace, so if the guy tried everything, compiler would catch the attempt.

Popular posts from this blog

When monospace fonts aren't: The Unicode character width nightmare

Circumcision as an adult, part 1

Circumcision as an adult, part 2