Tag: design

Conceptual Integrity at Scale

The central argument of the Mythical Man Month from Fred Brooks is that conceptual integrity is the most important consideration in system design, and that conceptual integrity will only be achieved if the design comes from one, or a few resonant minds.

I will contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas.

[…]

Conceptual integrity in turn dictates that the design must proceed from one mind, or from a very small number of agreeing resonant minds.

If you’ve been the creative force in a group work, you will have experienced these challenges. Core ideas are misunderstood, insoncistencies start to pop up, and the result is a patchwork.

For my part, I can confirm that consistency erodes quickly if you don’t pay close attention. Maintaining conceptual integriy is hard work.

This doesn’t happen because people are dumb, neglecting or malevolent. It happens because as soon as you specialize, you lose sight of the whole. Someone does a change here, someone a change there, and both changes end up not being fully consistent with each other.

Unfortunaltely, unlike Brooks suggests, doing all the design work alone is usually not realisitc.

With a good review culture you can scale your design team from one head to a few: let people design parts of the system even if their understanding of the whole system is lacunary, and have one central person review how well the contributions fit it.

It’s like having mutliple authors for an article but having one person in charge of doing a complete pass on the article at the end to ensure consistency.

But if you want to tackle bigger challenges, you will have to scale your design team even more.

Ensuring conceptual integrity at scale is hard because it requires not only scaling knowledge but also standardizing the decision making process.

This is what guidelines try to achieve. Guidelines encode the principles, maxims, constraints, and goals of the system in a way that different people reach similar decisions. It’s evidently impossible to encode the complete decision making process in guidelines, given that so much subjective, but they help achieve a basic overall consistency.

As for the subjectivity: just take one of your colleague and ask yourself “what would he decide?” You might have a hunch at his decision, but chances are, you don’t know enough about all the thinking that went in his previous decisions to predict this one accurately. If you do, well, you’re two “reasonant” minds, as Brooks would say.

If you know lots of people will be involved in the design process, you will need more than guidelines and reviews. You will have to decompose the problem in parts that can be solved individually. Each part can be assigned one “mind”. The whole might not be fully consistent, but the solution at each level of abstraction will at least be consistent.

Following the newspaper analogy, a newspapers has an editor in chief that sets the tone of the writings and the overall orientation (these are guidelines). He or she will review the topics of the individual articles to make sure they fit in the issue of the newspaper, but he or she won’t edit every article himself (the parts).

No large system will be fully consistent (think of Microsoft Office, that our dear journalists might be using), but it doesn’t hurt too much, because no user will ever use all of the system.

Evolution will also bring some inconsistencies in the system. Moving from one system paradigm to the next is like moving from one local maxima to the next one. In between things will be worse, that is, less consistent. But if you think there’s a superior design paradigm for the whole system, it’s worth challenging the current one and see if there’s a path.

Fred Brooks is right that conceptual integrity is the most important aspect in system design. He’s also right that the more designers there are, the harder it is to ensure concistency. But for large systems that evolve, some inconsistencies are inevitable. Address them like other risks in your project.

Why a Calendar App is a Great Design Exercise

To check if a salesman is good, one classic is the “Sell me this pen” test. To check if a software designer is good, I propose the “Design me a calendar app” test.

That was one of the topic we chose for the software engineering lab, and I loved the results.

There are several reasons why it works well as a design exercise:

Everybody can relate − The domain is easily understood and everybody can relate. Who hasn’t used a calendar app?

It’s easy but not so easy − Managing events that occur once and are short is easy. But it gets more interesting as soon as events are recurring (series), span multiple days, are entered in different time zones, or have rooms associated with them. The design becomes more complex not because independent features pile up, but because the complexity of the core model increases.

Time is messy − A lot of complexity in business software comes from the fact that business rules are “arbitrary”. They make sense at the business level because of processes, domain knowledge, etc. but it’s hard to capture some clear “logic” behind them in software. Introducing such a business domain for an exercise is possible, but takes time. On the other hand, every body knows the idiosyncrasies of the Gregorian calendar already. There is little “logic” behind February having only 28 days and occasionally 29. But, yes, it means a month might sometimes overlap exactly 4 weeks and not 5. Deal with it in you UI.

It’s not just the server − This design exercise raises interesting questions not only in the backend, but also the frontend. What’s the right model? How can we display it fast? What’s the expectation of the user when the start time of a meeting is changed: to shorten it or to move it? These questions don’t have to do with the technology stack. They are inherent to the product. For questions like the last one, I recommend reading The Math of Easy-to-Use from Terry Crowley, former head of development for Microsoft Office, including Microsoft Outlook. He knows about calendar apps.

In the The Mythical Man Month, Fred Brooks explains that one of his favorite interview question is “Where is next November?”.

I have long enjoyed asking candidate programmers, “Where is next November?” If the question is too cryptic, then, “Tell me about your mental model of the calendar.” The really good programmers have strong spatial senses; they usually have geometric models of time; and they quite often understand the first question without elaboration.

Mental models of time are cultural. In western societies, time flows from left to right; the past is behind us and the future in front of us. In other societies, it’s the way around. So I wouldn’t quite expect a specific answer to this question. But I would agree with Fred Brooks that a good ability to model time is a predictor of good design skills in general.

If you don’t get much from this exercise, it will at least make you more aware of the problems that exist dealing with time in computer programs and to use libraries properly. This is a valuable programming skill on its own. The system I’m working on (a train dispatching system) doesn’t work correctly during the night of the daylight saving time (DST) change in autumn, since time jumps back if the DST offsets aren’t accounted for. If you’ve designed a calendar app once in your life, you are aware of such pitfalls.

So, please, don’t design todo apps as exercise. Design calendar apps. It develops real design skills and will make real-world software less buggy.

More

Things You Can’t Abstract

The art of programming is to a large extent the art of devising abstractions. Some might be very general and reusable in many contexts, some will be more specialized and applicable only in some domains.

The purpose of abstraction is to hide complexity so that we don’t need to care about details. Using abstractions, we can “raise the abstraction level”.

Data structures, relational databases, file systems or garbage collection are all examples of common programming abstractions. There are of course many more.

Abstracting is not unique to programming. For instance, the DNA, the cell, the organ and the organism are different abstraction levels in biology.

An abstraction defines a contract between a user and a provider. The less constraints there are in the contract the more freedom there is in the implementation possibilities. It’s tempting to abstract away all non functional aspects, but it’s actually a bad idea: you will need to understand them to use the abstraction correctly.

First, you can not abstract performance. Wether an operation takes O(1) or O(n) is not something you can ignore. Eventually, at some point you will have to care about the implementation of the abstraction to understand its performance characteristics. Abstracting performance and letting the runtime figure out the best optimization strategy look nice on paper but is the source of many headache. You will need to know how your data structure performs, how your database fetches data, how many files can reasonably exist in a folder, and when your garbage collection kicks in.

Second, you can not abstract failure modes. If something can fail, you can not ignore it. This is especially true of the network: if something is remote it can be inaccessible. Attempts to abstract the network as if everything were local simply do not work. An abstraction can have few failure modes, but there is no abstraction that never fails. You will need to understand how your data structure reacts when it can’t expand, how you database reacts when your commit is so big that its transaction log is full, when your file system is not reachable, and when the garbage collector can’t reclaim space.

And third, you can not abstract the consumption of shared resources. Since shared resources are finite and common to the whole system, every component is indirectly related to every other components. You will need to understand how much memory you data structure takes, how much of your data fit in the database cache, how much disk space your system consumes, how much clock cycles are eaten by garbage collection runs.

That makes a lot of aspects we can’t abstract. Joel Spolsky was right. All non-trivial abstractions eventually “leak“. Barbara Liskov was wrong. In practice, two abstractions with the same functionality cannot be “substituted“, unless they also have the same non-functional characteristics.

It is discouraging to realize we can’t abstract as much as we want, but it doesn’t mean abstraction doesn’t work. You will need to know a bit more about data structure, database, file systems, and garbage collection than you thought to use them correctly, but you can still ignore a lot of the internal details. The goal of hiding some complexity is achieved, but not of hiding all complexity.

More

Lateral Thinking

Lateral thinking is a term coined by Edward De Bono to characterize the generation of alternative ideas, as opposed to vertical thinking, which generates ideas based on logic and stepwise refinements. Another way to explain lateral thinking in a much common way is “thinking out of the box.”

Often finding the best solution to a problem requires a creative move to go away from the existing solution and start with a new angle. This is were lateral thinking can help.

As a reminder of the power of lateral thinking, let us take an egg and a spoon. You are doing a brunch. How do you provide assistance to help cut the egg?

With vertical thinking you might come up with this solution:


With lateral thinking, maybe with this one:


I was absolutely amazed the first time I saw this device in action. The cut is perfect. Also, I would probably never have come to this solution, no matter how long I stared at my egg.

Each time I discuss a design issue I remember my last brunch and try to take some distance with the situation to go back to the root of the problem to solve and ask: could we do this completely differently?

Sometimes the best way to cut an egg is to not cut it actually.

Gall’s Law

Gall’s law states that complex systems can only be the result of an evolutionary process, and not the result of a design from scratch:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.  – John Gall (1975, p.71)

A complex system evolves from simpler systems by adding successive deltas of complexity. The only way to build a complex a system is through iteration. That’s what evoluation is about.

Iterations enable us to get feedback, correct and improve the system. See what works and what doesn’t. Fix mistakes.

The system must be working after each iteration. You can add new features, as long as it refines the existing system and keeps it running.

A tadpole becomes a frog by developing its legs, then its arms, and finally shrinking its tail. The frog’s legs, arms and body aren’t developed individually and assembled at the end. That’s not how evolution works.

last_thumb1367178271

Also, you can not evolve everything at once, since in the meantime the system might not work. A tadpole develops its legs, then its arms, and finally shrink its tail. Each iteration needs focus.

Gall’s law is relieving. It’s OK to not be able to handle all the complexity at once. And it’s not only you–it’s everybody.

A complex system can not be built using only theory and first principles, because there will always be details of the environment that we were not aware of. The only way to make sure something will work is to test it for real. Practice trumps theory.

Obsessing with getting it right the first time is counter productive. Just start somewhere and iterate. Too much unknown blocks our creativity. But once we have something concrete, ideas to improve come easily.

The tadpole also teaches us a lesson here: it first develops a tail, which then disappears later on. The tail is a good idea in the water, but not so much on the ground. You will have to reinvent yourself occasionally.

Unit Testing Matters

Unit testing is a simple practice that can be explained in one sentence: each method should have an associated test that verifies its correctness. This idea is very simple. What is amazing with unit testing is how powerful this simple practice actually is. At first, unit testing seems like a simple approach to prevent coding mistakes. Its main benefit seems obvious:

Unit testing guarantees that the code does what it should.

This is actually very good, since it’s remarkably easy to make programming mistakes: typo in SQL statements, improper boundary conditions, unreachable code, etc. Unit tests will detect these flaws. Shortly after, you will realize that it’s way easier to test methods that are short and simple. This confers to unit testing a second benefit:

Unit testing favors clean code.

This is also very good. Unit testing forces developers to name things and break down code with more care. This will increase the readability of the code base. Now, armed with a growing suite of tests, you will feel more secure to change business logic, at least when the change has local effects. This is a third benefit of unit testing:

Unit testing provides the safety net that enables changes

This is excellent. Fear is one of the prime factor that leads to code rot. With unit tests, you can ensure that you don’t break existing behavior, and can cleanly refactor or extend the code base. You might object that many changes are not always localized, and that unit tests don’t help in such case. But remember: a non-local changes is nothing more than a sequence of local changes. Changes at the local level represent maybe 80% of the work; the remaining 20% is about making sure that the local changes fit together. Unit tests help for the 80% of the work. Integration tests and careful thinking will do for the other 20%. As you become enamoured with unit testing, you will try to cover every line you write with unit tests. You will make it a personal challenge to achieve full coverage every time. This isn’t always easy. You will embrace dependency inversion to decouple objects, and become proficient with mocks to abstract dependencies. You will systematically separate infrastructure code from business logic. With time, your production code will be organized so that your unit tests can always obtain an instance of the object to test easily. Along the way, you will have noticed that the classes you write are more focused and easier to understand. This is the fourth benefit of unit testing:

    Unit testing improves software design

This is amazing! Unit testing will literally highlight design smells. If writing unit tests for a class is painful, your code is waiting to be refactored. Maybe it depends on global state (Yes, I look at you Singleton), maybe it depends on the environment (Yes, I look at you java.lang.System), maybe it does too much (Yes, I look at you Blob), maybe it relies too much on other classes (Yes, I look at you Feature Envy). Unit testing is “a microscope for object interactions.” Unit testing will force you to think very carefully about your dependencies and minimize them as much as possible. It will naturally promote the SOLID principles, and lead to better a decomposition of the software.

Honestly, I find it amazing that such a simple practice can lead to so many benefits. There are many practices out there that improve software development in some way. What makes unit testing special is the ridiculous asymmetry between its simplicity and its outcome.

More

Your Language is a Start-up

Watching the TIOBE index of programming language popularity is depressing. PHP and Javascript rule the web, despite the consensus that they are horrible; Haskell and Smalltalk are relegated to academic prototyping, but unanimously praised for the conceptual purity. How technolgy adoption happens is a puzzling question.

Evidences seem to suggest that what matters is to attract a set of initial users, and then expand. The initial offer needn’t be particularly compelling. As long as it wins on one dimension  – maybe because it’s ambarassingly simple, or provides a very effective solution to a very specific problem – it might attract early adopters. PHP won because of its simplicity to get things done; Javascript won because it was the first to provide a solution to make HTML dynamic. After initial success in a niche, the technology can evolve to attract more users. PHP and Javascript  evolved later to fix their initial design flaws. They are both now mature object-oriented languages.

The price for fixing initial flaws is however extraordinary high. Once a language feature is designed and made available, it’s cast in stone. Evolving a language while maintaining backward compatibility is extremelly challenging, but breaking compatibility and dealing with multiple branches isn’t much of an easy solution neither. Notorious examples of evolutions in Java are the Java Memory Model and generics. It took years of research to plan them, and years of availability to reeducate the community. C++ is still trying to catch up, and still lacks feature that we take for granted on some plateforms, e.g. a standardized serialization.

Surprisingly, when adoption happens, it might be from a difference audience than the one expected.  “Languages designed for limited or local use can win a broad clientele, sometimes in environments and for applications that their designers never dreamed of.”  say the authors of Mastermind of programming. This is definitively true. Java was initially designed for embedded systems, but succeed instead in the enterprise. Javascript was thought as a thin veneer for web pages, but now powers the new generation of client-side web app and is even expanding to the server-side.

When a technology starts to decline after the adoption peak, don’t be too quick to claim it dead. It might enjoy an unexpected renaissance. Many have for instance claimed that Java was dead. They have failed however to recognize the the underlying JVM is a rocking beast, amazingly fast and versatile for those who know how to tame it. Nowadays, one of the best strategy to implement new languages is to leverage the JVM and provide interoperability with Java libraries. In turn, this massive adoption of the JVM by new language implementors is driving innovation in the JVM itself, which has been extended with new bytecode for languages other than Java. I doubt that James Gosling had anticipated this evolution of the platform.

Clearly, the key characteristics of a language are its syntax and semantics, since they define together its expressive power. Expressivity isn’t however the unique force at play for adoption. What a real-world check suggests is that expressivity is only one factor amongst many other technical and social factors. The ease of debugging or the existence of a friendly community could for instance turn out to more important for some users than the ease of writing code. Language designers typically understimate such factors, severly impeding their chances of success.

To foster adoption, one must also rekon that people are reluctant to change. What people are already familiar with must be taken in consideration: in 2008, mobile users wouldn’t have been ready for the minimalistic iOS 7 interface. They were however ready for the original skeuomorphic interface, and now that they have become familiar with it, they can get rid of the skeuomorphic ornaments. People don’t change for the sake of changing, they change to solve or problem, and they change only if the gain outweight the pain. For programming languages, the problem is productivity and the pain is learning a new platform. In 2013, developers might not be familiar enough with functional programming to adopt a pure functional language like Haskell, but they definitively are ready to adopt a hybrid language like Scala.

Together, these elements might help explain the failure of some great languages, for instance Lisp. Lisp is a beautiful programming language that offers amazing flexibility. For a skilled practitionner, lisp is a secret weapon. However, lisp does nothing particularly well out of the box. “Lisp isn’t a language, it’s a building material.”, dixit Alan Kay. Clojure, on the other hand, is a Lisp dialect with just enough direction to solve one very painful problem: writting concurrent code. Given that the problem is so painful, people won’t mind a few parenthesis to solve it. This choice paid off, and in 2012 Clojure moved in the “adopt” quadrant of Thoughwork’s technology radar.

The language business is a competitive business where idealism won’t prevail. For a language to be adopted, it must solve a problem for some early adopters, who will then create an attractive ecosystem that will convince the late majority. In other words: language designers should think of their language like a start-up.

Simplicity Prevails

We engineers are masters of self-deception when it comes to our aptitudes to handle complexity. We believe we are way better at handling complexity than we actually are. In practice the level of complexity that people can master (me including) is disappointing low.

Instead of self-deceiving ourselves, we should embrace our limitations and aim for simplicity. This is what all geniuses of our time have been preaching for. Simplicity prevails.

“Simplicity is the ultimate sophistication.” — Leonardo da Vinci

It is common for tools to be too complex. We all know that only a tiny fraction of the features of Word are used, still the usual trend for a product is towards adding more and more features. Often, the best thing for a product is taking something away from it. Only simple tools prevail.

One problem with simplicity is that it is often confused with easiness or triviality. Easiness and triviallity are subjective properties relative to a user. They refer to the sense of familiarity or ordinarity. Simplicity is an objective property that refers to purity, the absence of mixing distinct elements.

Let us look at some simple features that are clear wins.

Using Venn Diagrams for Access Control

Have you ever been confused by the security settings of an application, not knowing what would be accessible or not to other users? Venn diagrams are simple and can be used to make an interface intuitive. Win!

Using Pictures instead of Texts to Log

Have you ever found yourself overwhelmed by the difficutly to compare tens of print statements to understand state mutations over time? Comparing texts is hard; comparing images is easy. Using a visual log is simple and supports better debugging. Win!

Using Examples to Test Programs

To test your code you exercise it with chosen inputs that serve as examples? Well, you could have invented unit testing. As Fowler says: “Kent’s framework had a nice combination of absurd simplicity and just the right features for me”. Win!

Using Live Code instead of Static Code to Understand Programs

Have you ever felts exhausted trying to mentally run code in your head when inspecting static sources? The underlying question is: what prevented you from runinng it and see it live? Finding a suitable unit test and breaking at the start of the method you are inspecting can be automated to become a one-click operation. Win!

Optionless search

Have you ever been repelled by the sheer amount of options in a search form? For instance this one. That’s tackling the problem of search wrongly. Google, Airbnb, and Facebook got it right, offering essentially one unique text input and hiding the magic of relevance matching. Win!

These examples show what I consider to be the level of complexity we can handle, and should aim at. These features are so simple to use they will immediately look easy and trivial. This is a good thing.

Features Interaction

Hoare’s famous paper “Hints on Programming Language Design” distinguishes two main aspects of language design:

  • Part of language design consists of innovation. This activity leads to new language features in isolation.
  • The most difficult part of language design lies in integration: selecting a limited set of language features and polishing them until the result is a consistent simple framework that has no more rough edges.

I will try to keep here a list of inspiring articles that illustrate this two parts, especially edge cases in integration of individiual features.You can also follow Shoot Yourself In The Foot for examples of unexpected problems resulting from the language design rationale.

Abstraction-Level vs. Meta-Level

As per wikipedia, a model is a pattern, plan, representation (especially in miniature), or description designed to show the main object or workings of an object, system, or concept. In the case of software engineering, models are used to represent specific aspects of the software system, such as static structures, communication paths, algorithms, etc.

Models can be created along two axis:

Abstraction level

An abstraction level is a way of hiding the implementation of a particular set of functionality. Both representations are however two descriptions of the same reality. The representation can vary in their level of detail or in their expressiveness. Example

C code – assembler code.
Both representation describe a sequence of operation in an imperative way, but C is much more expressive than assembler.

Java code – sequence diagram.
Both representations describe the run-time behavior of the system. The sequence diagram does however frequently omit details of the call flow.

Meta-level

A meta-level (and meta-model) is way to highlight the properties – the nature – of a particular set of elements. A meta-model describes then the constraints or the semantics to any set of such elements.

Object – class.
In OO technologies, object and classes belong to two different meta-level. The class is a representation of the structure (constraints) that any of the instance will fulfill.

XML – DTD.
An XML file can be validated against a DTD, which is not a representation of the particular file, but of the structure (constraints) that the file must fulfill.

Tower of models

Models themselves can be refined towards either higher abstraction level or meta-levels.

Component diagram – class diagram – Java code.
All three artifacts are representations of the same reality at various level of refinement and reasoning.

XML – Schema – Meta schema.
In this case, each model is meta description of the underlying one. XML Schema themselves must conform to the schema http://www.w3.org/2001/XMLSchema.

As with the XML example, the tower of meta-model generally ends up in model specified in itself. That is, a recursion is introduced to stop the tower of meta-model from growing infinitely.
The term “modeling” refers commonly to “raising the abstraction level”; the term “meta-modelling” is used otherwise. Different tools/technologies don’t necessary represent different meta-levels. Inversly, one technology may be used to address different meta-level.
With respect to OO technologies, several UML diagrams and notations work at different meta-level. Object diagram and sequence diagrams describe the interaction between specific instances of objects. Class diagram specifies constraints (such as cardinality) or semantical relationship between objects (such as aggregation). Stereotypes can then be used to specifies constraints (such as “singleton”) or semantics (such as “entity”) to the class itself. For instance, an “entity” stereotype could be defined to flag all classes with an attribute “primary key” of type integer.
Level \ Technology

Language

 UML
L0 Object  Object diagram,  Sequence diagram
L1 Class Class diagram
L2 – Stereotype

Examples where meta-models are useful

Having three meta-level provides one extra level of indirection that can be very practical in some cases:

Level

XML transformation

OO refactoring

Annotation processing
L0 – information Data Object Class annotation (e.g. @Entity)
L1 – model Schema Class Annotation processor (e.g. JPA)
L2 – meta-model Meta schema Metaclass Classloader

Here are three common examples which actually leverage the strength of meta-model.

XML transformation

The success of XML is related to the existence of a meta-model which makes it a semi-structured format. Structured data are interpretable only by their host application which is the only one to know the model. Unstructured data (plain text, image, etc.) have not model and are not interpretable at all. Semi-structured data have a model which is itself understandable by any other application which knows the meta-model. Due to this extra-level, reusable parser, validator, encryptor or even arbitrary transformer (think XSLT) could be created. The schema could also be extended without necessary breaking backward compatibility (think XPath).

OO refactoring

Refactoring rules is a typical example of the usage of the OO meta-model: refactoring rules are proved to preserve the validity of the model against the meta-model, and are reusable for any model.

Annotation processing

Annotation processing is yet another example where a meta-model is used under the hood. Just like the class-loader knows how to load a class because of the meta-model of any class, it is also able to load the corresponding annotations. This reliefs the application from storing application-specific semantics in separate files (think deployment descriptor). The application (annotation processor) can then query the annotation at run-time to refine the behavior of the system. The annotations enrich classes with additional constraints and behaviour in a resuable way: the same annotation processor can be used for arbitrary class sets.