Conceptual Integrity at Scale

The central argument of the Mythical Man Month from Fred Brooks is that conceptual integrity is the most important consideration in system design, and that conceptual integrity will only be achieved if the design comes from one, or a few resonant minds.

I will contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas.


Conceptual integrity in turn dictates that the design must proceed from one mind, or from a very small number of agreeing resonant minds.

If you’ve been the creative force in a group work, you will have experienced these challenges. Core ideas are misunderstood, insoncistencies start to pop up, and the result is a patchwork.

For my part, I can confirm that consistency erodes quickly if you don’t pay close attention. Maintaining conceptual integriy is hard work.

This doesn’t happen because people are dumb, neglecting or malevolent. It happens because as soon as you specialize, you lose sight of the whole. Someone does a change here, someone a change there, and both changes end up not being fully consistent with each other.

Unfortunaltely, unlike Brooks suggests, doing all the design work alone is usually not realisitc.

With a good review culture you can scale your design team from one head to a few: let people design parts of the system even if their understanding of the whole system is lacunary, and have one central person review how well the contributions fit it.

It’s like having mutliple authors for an article but having one person in charge of doing a complete pass on the article at the end to ensure consistency.

But if you want to tackle bigger challenges, you will have to scale your design team even more.

Ensuring conceptual integrity at scale is hard because it requires not only scaling knowledge but also standardizing the decision making process.

This is what guidelines try to achieve. Guidelines encode the principles, maxims, constraints, and goals of the system in a way that different people reach similar decisions. It’s evidently impossible to encode the complete decision making process in guidelines, given that so much subjective, but they help achieve a basic overall consistency.

As for the subjectivity: just take one of your colleague and ask yourself “what would he decide?” You might have a hunch at his decision, but chances are, you don’t know enough about all the thinking that went in his previous decisions to predict this one accurately. If you do, well, you’re two “reasonant” minds, as Brooks would say.

If you know lots of people will be involved in the design process, you will need more than guidelines and reviews. You will have to decompose the problem in parts that can be solved individually. Each part can be assigned one “mind”. The whole might not be fully consistent, but the solution at each level of abstraction will at least be consistent.

Following the newspaper analogy, a newspapers has an editor in chief that sets the tone of the writings and the overall orientation (these are guidelines). He or she will review the topics of the individual articles to make sure they fit in the issue of the newspaper, but he or she won’t edit every article himself (the parts).

No large system will be fully consistent (think of Microsoft Office, that our dear journalists might be using), but it doesn’t hurt too much, because no user will ever use all of the system.

Evolution will also bring some inconsistencies in the system. Moving from one system paradigm to the next is like moving from one local maxima to the next one. In between things will be worse, that is, less consistent. But if you think there’s a superior design paradigm for the whole system, it’s worth challenging the current one and see if there’s a path.

Fred Brooks is right that conceptual integrity is the most important aspect in system design. He’s also right that the more designers there are, the harder it is to ensure concistency. But for large systems that evolve, some inconsistencies are inevitable. Address them like other risks in your project.

In Defense of Design Before Coding

Software design as a separate activity from implementation — “up front” design — got a bad press with agile methods.

Agile advocates say the design should be emergent. They say, design without coding is waterfall. It’s a waste of time.

I understand that you don’t want to design the whole system up front. But at the feature level, a bit of thinking before coding does miracles, I say.

My first argument is visible design. Looking at the code doesn’t reveal the whole design because code only shows the static structure. The design is more than that. To understand how the system works you must run it, but even then the sequencing of events is still invisible. If you want effective feedback on the design, you must make it visible. People that jump directly to code still end up sketching or drawing things for their colleagues to explain their design and get feedback. Designing up front makes the design visible up front.

My second argument is speed of iteration. Even with higher-level programming languages, there is a gap between the concepts and the implementation. There is some work needed to implement the thing for real and take care of all the details. Running the system in your your head, or on paper, to challenge the design enables faster iterations on the design. CRC sessions are for instance a nice way to explore the design space effectively, without coding yet.

My third argument is better reasoning. The code level is just one of the many abstraction levels you can use to reason about the system. When you’re trying to identify the main abstractions, what their responsibilities are, and how they play together, this abstraction level is often too low. One abstraction with a clear responsibility might map to several classes. There might be for instance a “scheduler” that will be implemented using several concurrency primitives. These are implementation details (althrough interesting ones!) irrelevant for now. Working at the code level forces you to think at one specific abstraction level. Working on paper enables you to choose the optimal abstraction level to work out the design.

And finally, my fourth argument is tracking rationale. The code defines how the system works but gives in itself little clues as to why it was designed so. Design is all about trade offs: what were the they? If you never learn to design up front on paper you will never learn to document software understandably, too. And without documentation later, the rationale will be lost.

You should design up front so far you can. Then switch to a computer.

How far you can design up front depends on your intellect and your knowledge of the problem domain. Learn to assess the confidence in your up front design correctly, and identify when to stop, since there lies the danger: too much time spent designing on paper something that doesn’t work. But some design up front has its place.


Why a Calendar App is a Great Design Exercise

To check if a salesman is good, one classic is the “Sell me this pen” test. To check if a software designer is good, I propose the “Design me a calendar app” test.

That was one of the topic we chose for the software engineering lab, and I loved the results.

There are several reasons why it works well as a design exercise:

Everybody can relate − The domain is easily understood and everybody can relate. Who hasn’t used a calendar app?

It’s easy but not so easy − Managing events that occur once and are short is easy. But it gets more interesting as soon as events are recurring (series), span multiple days, are entered in different time zones, or have rooms associated with them. The design becomes more complex not because independent features pile up, but because the complexity of the core model increases.

Time is messy − A lot of complexity in business software comes from the fact that business rules are “arbitrary”. They make sense at the business level because of processes, domain knowledge, etc. but it’s hard to capture some clear “logic” behind them in software. Introducing such a business domain for an exercise is possible, but takes time. On the other hand, every body knows the idiosyncrasies of the Gregorian calendar already. There is little “logic” behind February having only 28 days and occasionally 29. But, yes, it means a month might sometimes overlap exactly 4 weeks and not 5. Deal with it in you UI.

It’s not just the server − This design exercise raises interesting questions not only in the backend, but also the frontend. What’s the right model? How can we display it fast? What’s the expectation of the user when the start time of a meeting is changed: to shorten it or to move it? These questions don’t have to do with the technology stack. They are inherent to the product. For questions like the last one, I recommend reading The Math of Easy-to-Use from Terry Crowley, former head of development for Microsoft Office, including Microsoft Outlook. He knows about calendar apps.

In the The Mythical Man Month, Fred Brooks explains that one of his favorite interview question is “Where is next November?”.

I have long enjoyed asking candidate programmers, “Where is next November?” If the question is too cryptic, then, “Tell me about your mental model of the calendar.” The really good programmers have strong spatial senses; they usually have geometric models of time; and they quite often understand the first question without elaboration.

Mental models of time are cultural. In western societies, time flows from left to right; the past is behind us and the future in front of us. In other societies, it’s the way around. So I wouldn’t quite expect a specific answer to this question. But I would agree with Fred Brooks that a good ability to model time is a predictor of good design skills in general.

If you don’t get much from this exercise, it will at least make you more aware of the problems that exist dealing with time in computer programs and to use libraries properly. This is a valuable programming skill on its own. The system I’m working on (a train dispatching system) doesn’t work correctly during the night of the daylight saving time (DST) change in autumn, since time jumps back if the DST offsets aren’t accounted for. If you’ve designed a calendar app once in your life, you are aware of such pitfalls.

So, please, don’t design todo apps as exercise. Design calendar apps. It develops real design skills and will make real-world software less buggy.


Things You Can’t Abstract

The art of programming is to a large extent the art of devising abstractions. Some might be very general and reusable in many contexts, some will be more specialized and applicable only in some domains.

The purpose of abstraction is to hide complexity so that we don’t need to care about details. Using abstractions, we can “raise the abstraction level”.

Data structures, relational databases, file systems or garbage collection are all examples of common programming abstractions. There are of course many more.

Abstracting is not unique to programming. For instance, the DNA, the cell, the organ and the organism are different abstraction levels in biology.

An abstraction defines a contract between a user and a provider. The less constraints there are in the contract the more freedom there is in the implementation possibilities. It’s tempting to abstract away all non functional aspects, but it’s actually a bad idea: you will need to understand them to use the abstraction correctly.

First, you can not abstract performance. Wether an operation takes O(1) or O(n) is not something you can ignore. Eventually, at some point you will have to care about the implementation of the abstraction to understand its performance characteristics. Abstracting performance and letting the runtime figure out the best optimization strategy look nice on paper but is the source of many headache. You will need to know how your data structure performs, how your database fetches data, how many files can reasonably exist in a folder, and when your garbage collection kicks in.

Second, you can not abstract failure modes. If something can fail, you can not ignore it. This is especially true of the network: if something is remote it can be inaccessible. Attempts to abstract the network as if everything were local simply do not work. An abstraction can have few failure modes, but there is no abstraction that never fails. You will need to understand how your data structure reacts when it can’t expand, how you database reacts when your commit is so big that its transaction log is full, when your file system is not reachable, and when the garbage collector can’t reclaim space.

And third, you can not abstract the consumption of shared resources. Since shared resources are finite and common to the whole system, every component is indirectly related to every other components. You will need to understand how much memory you data structure takes, how much of your data fit in the database cache, how much disk space your system consumes, how much clock cycles are eaten by garbage collection runs.

That makes a lot of aspects we can’t abstract. Joel Spolsky was right. All non-trivial abstractions eventually “leak“. Barbara Liskov was wrong. In practice, two abstractions with the same functionality cannot be “substituted“, unless they also have the same non-functional characteristics.

It is discouraging to realize we can’t abstract as much as we want, but it doesn’t mean abstraction doesn’t work. You will need to know a bit more about data structure, database, file systems, and garbage collection than you thought to use them correctly, but you can still ignore a lot of the internal details. The goal of hiding some complexity is achieved, but not of hiding all complexity.


Become a Domain Expert

It is only possible to design a feature correctly if you have enough understanding of the underlying business domain. Large software systems may have complex domain models with many special situations that aren’t immediately obvious. Taking the time to understand the domain is crucial for making the right assumptions during development.

Failing to understand the domain correctly will result in features that work “most of the time” but break in special occasions. The detection of the these problems will happen late in the project, cost time to analyse and debug (see the famous estimate), und may in the worst case require a redesign of the whole feature if the domain model was used inappropriately.

A little example of domain knowledge in the project I’m working with is the following: we have train timetable modelled as sequences of stops at trains stations. Using the identifier of the station to uniquely identify a stop in the timetable is not enough since a train might pass the same train station twice in some case. Instead, it is needed to work with pairs of train stations to correctly identify the stops. It’s easy to do if you know it, but if don’t consider this correctly early on, you’re good to rewrite your code later on.

Without enough domain knowledge, the intent of the existing code remain obscure. Sure, you will understand what the goal was technically, but the problem it solves on the business side will escape you. This is unfortunate, because it limits your range of action. If you understand the problem on the business side as well you can put things in context and come up with more technical solutions.

Each minute spent better understanding the domain is worth it. If you want to become a better engineer, become a domain expert.

The Ambitions of Scala

In the object paradigm, a system consists of objects with mutable state, whereas in the functional paradigm, it consists of functions and immutable values. At first, these two worlds seem incompatible.

But not so for Odersky. In 2004 he released the first version of Scala, a language that combines both.

Scala’s roots are object-oriented, sharing the same basic constructs as Java, with whom it is fully compatible. Its functional flavor comes from several features borrowed or transposed from concepts in functional languages like Haskell.  This includes first-class and higher-order functions, including currying, but also pattern matching with case classes, and the support for monads and tail recursions.

The mariage is suprisingly elegant. Maybe the two worlds are compatible after all.

But the ambitions of Scala do not stop here. It also aims at beeing scaleable, both in terms of modularity and in terms of expressivity. Scala should support the modularisation of small and large components, and help reduce the gap between the code and the domain concepts.

The many features of the Scala’s type system enables scalability along both axes. Traits enable for instance a fined grained modularisation of object behaviors. Implicit conversions on the other hand enable existing types in libraries to be extended to express code more clearly.

But more importantly, features of the language create synergies. Abstract type members combined with type nesting enable the cake pattern, a form of dependency injection, or family polymorphism, a way to type check constellation of multiple related classes. The support of call-by-name combined with implicits enable the definition of domain specific languages.

You can’t but be amazed by how features sometimes combine. It is for instance possible to map a collection and convert its type at the same time using the special breakout object. You can even pattern match regular expressions!

Such synergies are possible because the foundations of Scala are principled.

  • First, everything is an object. There is no primitive types. Instead, the type hierarchy has two main roots, one for mutable objects (with reference semantics) and one for immutable objects (with value semantics).
  • Second, you can abstract over types, values, and functions using parametrization or abstract members. The three constructs support both forms of abstractions consistently.
  • Third, any object that defines an apply() function can be used as a function. This closes the gap between functions and objects. The inverse of apply() is unapply(). Any object that defines unapply() can be used as an extractor for pattern matching.

Take the expression “val l = List(1,2,3)”. This is not native syntax for list construction, but actually the evaluation of the function “apply” on the singleton object “List” with the arguments “1,2,3”. Or take the expression “val (x,y) = (1,2)”. This is not native syntax for multiple assignments, but tuple unpacking using extractors. These principles enable nice extensions of the language.

The flexibility of Scala has a price though: it is easy to learn Scala on the surface, but mastering its intricacies is challenging.

Also, Scala comes with many additional features that seem to exist more for convenience than necessity, making it even harder to master. It is for instance questionable wether structural typing or default parameter values, to name a few, should really have made it into the language. Clearly they are usefull and alleviate some pain points of Java, but they also distract from the essence of the language. Scala might at times appear to lack focus.

The richness of the language is acknowledged by the Scala community itself. To quote Odersky, “Scala is a bit of a chameleon. It makes many programming tasks refreshingly easy and at the same time contains some pretty intricate constructs that allow experts to design truly advanced typesafe libraries.”

Scala is a language with many very powerful features and with many ways to do things. It’s up to the developers to use the features well and enforce a consistent programming style. For corporations, these two aspects could be a barrier to adoption. In comparison, a language like Kotlin offers the same basic ingredients but is a lot more simple.

The long bet of Odersky seems to pay off though. Scala has found its audience and made its way to the industry, including top players like Twitter or LinkedIn. It has established itself as a viable alternative.

Scala is a source of innovation and inspiration. While functions were already in object-oriented languages like Smalltalk in the 80s, Scala showed that object-orientation doesn’t mean mutability. The resulting programming style “OO in the large, FP in the small” is gaining traction. Having shown that the combination works, other languages will certainly follow this path.

Ten years after its inception, Scala has a mature and vivid community of users. To gain further adoption, it must now consolidate its foundation and keep it stable across releases. Fortunately, we can still count on Odersky to continue to innovate at the same time. At the recent ScalaDays 2015, he unveiled his plan to better control mutations of state, not with monads, but implicit conversions. That is yet another ambitious challenge.

Small, replaceable, composable

If you ask ten developers what characterises good software design, chances are you will get ten completely different answers. There are many principles around, plenty enough to choose from. Just have a look at the wikipedia entry List of software development philosophies to get a glimpse.

Yet, if we synthesise the many principles, it all boils down to three fundamental rules:

  1. Make it small
  2. Make it replaceable
  3. Make it composable

Separation of concern, single responsibility, high cohesion/low coupling, information hiding, encapsulation, dependency inversion, composition over inheritance, etc.  are all subsumed by the three rules above.

Software design is about controlling dependencies between the parts of the software–the inner wiring of the software.

If you were to pick only one rule between the three, pick number two, “Make it replaceable”. Aiming at replaceability, you’re almost forced to make things small and composable.

Rule number two gives a very simple but very effective design tool: the question “is this object replaceable?”. If you can’t image another implementation of an object, not even a stub in a unit test, or if you can’t replace one object without replacing other objects, then you should maybe go back to the drawing board. That said, not all objects are replaceable. Domain entities for instance aren’t, and it’s fine.

The very same three principles apply to all levels of abstraction of the software: from classes to modules, from modules to systems, and from systems to systems-of systems. Microservices and SOA are only slightly disguised forms of “Make it small, replaceable, and composable” at the system level.

What is funny about software design, is that problems frequently arise because pieces of the system do too much–not too little. The natural tendency to fix the problem is the add more logic and make it even worse. It takes a lot of courage to instead follow the three principles and break something big into many small pieces. We must be brave if we want to make things small, replaceable, and composable in order to ultimately make them simpler.

Unit Testing Matters

Unit testing is a simple practice that can be explained in one sentence: each method should have an associated test that verifies its correctness. This idea is very simple. What is amazing with unit testing is how powerful this simple practice actually is. At first, unit testing seems like a simple approach to prevent coding mistakes. Its main benefit seems obvious:

Unit testing guarantees that the code does what it should.

This is actually very good, since it’s remarkably easy to make programming mistakes: typo in SQL statements, improper boundary conditions, unreachable code, etc. Unit tests will detect these flaws. Shortly after, you will realize that it’s way easier to test methods that are short and simple. This confers to unit testing a second benefit:

Unit testing favors clean code.

This is also very good. Unit testing forces developers to name things and break down code with more care. This will increase the readability of the code base. Now, armed with a growing suite of tests, you will feel more secure to change business logic, at least when the change has local effects. This is a third benefit of unit testing:

Unit testing provides the safety net that enables changes

This is excellent. Fear is one of the prime factor that leads to code rot. With unit tests, you can ensure that you don’t break existing behavior, and can cleanly refactor or extend the code base. You might object that many changes are not always localized, and that unit tests don’t help in such case. But remember: a non-local changes is nothing more than a sequence of local changes. Changes at the local level represent maybe 80% of the work; the remaining 20% is about making sure that the local changes fit together. Unit tests help for the 80% of the work. Integration tests and careful thinking will do for the other 20%. As you become enamoured with unit testing, you will try to cover every line you write with unit tests. You will make it a personal challenge to achieve full coverage every time. This isn’t always easy. You will embrace dependency inversion to decouple objects, and become proficient with mocks to abstract dependencies. You will systematically separate infrastructure code from business logic. With time, your production code will be organized so that your unit tests can always obtain an instance of the object to test easily. Along the way, you will have noticed that the classes you write are more focused and easier to understand. This is the fourth benefit of unit testing:

    Unit testing improves software design

This is amazing! Unit testing will literally highlight design smells. If writing unit tests for a class is painful, your code is waiting to be refactored. Maybe it depends on global state (Yes, I look at you Singleton), maybe it depends on the environment (Yes, I look at you java.lang.System), maybe it does too much (Yes, I look at you Blob), maybe it relies too much on other classes (Yes, I look at you Feature Envy). Unit testing is “a microscope for object interactions.” Unit testing will force you to think very carefully about your dependencies and minimize them as much as possible. It will naturally promote the SOLID principles, and lead to better a decomposition of the software.

Honestly, I find it amazing that such a simple practice can lead to so many benefits. There are many practices out there that improve software development in some way. What makes unit testing special is the ridiculous asymmetry between its simplicity and its outcome.


Package Visibility is Broken

In Java, classes and class members have by default package visibility. To restrict or increase the visibility of classes and class members, the access modifiers private, protected, and public must be used.

Modifier Class Package Subclass World
public Y Y Y Y
protected Y Y Y N
no modifier Y Y N N
private Y N N N

(from Controlling Access to Members)

These modifiers control encapsulation along two dimensions: one dimension is the packaging dimension, the other is the subclassing dimension. With these modifiers, it becomes possible to encapsulate code in flexible ways. Sadly, the two dimensions interfere in nasty ways.


A subclass might not see all methods of its superclass, and can thus redeclare a method with an existing name. This is called shadowing or name masking.  For instance, a class and its subclass can both declare a private method foo() without that overriding takes place. This situation is confusing and best to be avoided.

With package visibility, the situation gets worse. Let us consider the snippet below:

package a;
public class A {
int say() {return 1;};
package b;
public class B extends a.A {
int say() {return 2;};
package a;
class Test {
public static void main(String args[]) {
a.A a = new b.B();
System.out.println(a.say()); // prints 1, WTF!!
} }

 (from A thousand years of productivity: the JRebel Story)

The second method B.say() does not override A.say() but shadows it. Consequently, the static type at the call site defines which method will be invoked.

One could argue that everything works as intended, and that it is clear that B.say() does not override A.say() since there is no @Override annotation.

This argument makes sense when private methods are shadowed. In that case, the developer knows about the implementation of the class and can figure this out. For methods with package visibility, the argument is not acceptable since developers shouldn’t have to rely on implementation details of a class, only its visible interface.

The static types in a program should not influence the run-time semantics. The program should work the same whether the variable “a” has static type “A” or “B”.


With reflection, programmers have the ability to inspect and invoke methods in unanticipated ways. Reflections should honor the visibility rules and authorize only legitimate actions. Unfortunately, it’s hard to define what is legitimate or not. Let us consider the snippet below:

class Super {
public void methodOfSuper() {

public class Sub extends Super {

Method m = Sub.class.getMethod("methodOfSuper");
m.getAnnotations(); // WTF, empty list

Clearly, the method methodOfSuper is publicly exposed by instances of the class Sub. It’s legitimate to be able to reflect upon it from another package. The class Super is however not publicly visible, and its annotations are thus ignored by the reflection machinery.

Package visibility is broken

Package-visibility is a form of visibility between private and protected: some classes have access to the member, but not all (only those in the same package). This visibility sounds appealing to bundle code in small packages, exposing the package API using the public access modifier, and letting classes within the package freely access each others. Unfortunately, as the examples above have shown, this strategy breaks in certain cases.

Accessiblitiy in Java is in a way too flexible. The combination of the fours modifiers with the possibility to inherit and “widen” the visibility of classes and class members can lead to obscure behaviors.

Simpler forms of accessibility should then be preferred. Smalltalk supports for instance inheritance, but without access modifiers; methods are always public and fields are always protected. Go, on the other hand, embraces package visibility, but got rid of inheritance. Simple solutions are easier to get right.


  • In “Moderne Software-Architektur: Umsichtig planen, robust bauen mit Quasar” the author argues that method level visibility makes no sense. Instead, components consist of classes, which are either exposed to the outside (the component interface) of belond to the component’s internals and are hidden (the component implementation). This goes in the direction of OSGi and the future Java module system.

Lines Spent

Studies have shown that the productivity of a developer is about 10 LOC/day. Considering that modern software consists in millions of lines of code, this number is appalling.

Measuring productivity with LOC is of course a dangerous thing to do. With programming, quality does not correlate with quantity, and it is wise to remember the words of Dijkstra:

If we wish to count lines of code, we should not regard them as “lines produced” but as “lines spent”. — Dijkstra

Indeed, programming is not a production process but a design process. Before a piece of code reaches maturity, various directions might first need to be explored, refined, and analyzed. Qualities like readability, performance, or reliability are competing dimensions in the design space. There is rarely “one way” to program something. Effective programming is about finding the best possible tradeoff in the shortest timeframe.

Writing high-quality code requires also a high level of rigor. Code must obey the established naming conventions, idioms and patterns of the system; it must be systematically refactored to prevent technical debt to accumulate; and it must be exhaustively tested. This level of discipline pays off in the long-term, but requires more time in the short-term.

Also, as a system grows, it becomes progressively harder to maintain an accurate mental model of the system. Considerable time is thus spent assessing the existing system to temporarily reconstruct enough knowledge for the task at hand. During this time, no code is written.

One could argue that counting total lines of code is useless; what is relevant is the number of lines of code modified per commit. Lots of modifications but no additions would suggest a modular system where features can be adapted declaratively without “writing new code”. Unfortunately, metrics tools do not follow this view and treat modifications as second-class citizens.

So, is a low number of LOC/day per day good or bad? For an optimist, it might indicate a sign of code quality; the team designs carefully and takes care to write the minimum amount of code needed. For a pessimist, if might indicate that the project is stalled; you’re not delivering enough code to make the deadline. For a realist, one thing is sure: software engineering is a very expensive activity.


Software As Liability