Understanding the Visibility of Side-Effects

This entry is all about the Java Memory Model and the happens-before partial order it formalizes.

The official specification of the JMM is authoritative but unreadable. Fortunately, there are useful resources around that aim at making it more accessible:

Happens-Before

The JMM defines a partial order called happens-before on all actions in the program. Actions are reads and writes to variables, locks and unlocks of monitors etc. The rules for happens-before are as follows:

  • Program order rule. Each action in a thread happens-before every action in that thread that comes later in the program order.
  • Monitor lock rule. An unlock on a monitor lock happens-before every subsequent lock on that same monitor lock.
  • Volatile variable rule. A write to a volatile field happens-before every subsequent read of that same field.
  • Thread start rule. A call to Thread.start on a thread happens-before any other thread detects that thread has terminated, either by successfully return from Thread.join or by Thread.isAlive returning false.

The happens-before partial order provides clear guidance about what values are allowed to be returned when the data are read. In other words: it formalizes the visibility of side-effect across threads.

A read is allowed to return the value of a write 1) if that write is the last write to that variable before the read along some path in the happens-before order, or 2) if the write is not ordered with respect to that read in the happens-before order.

Instructions can be reordered as long as the happens-before order is preserved. The code below

t1 = 1;
t2 = 2;
t3 = t1+t2;
t4 = 2*t1;

can for instance be be reordered as follows

t2 = 2;
t1 = 1;
t4 = 2*t1;
t3 = t1+t2;

If two threads shared data without using any synchronization, writes of a thread are considered unordered with respect to the other thread; according to condition 2), the execution is not deterministic and the writes of a thread might or might not be visible to the other thread.

Let us consider the two threads below, and one given execution:

  T1        T2

s = s+1 

          s = s+1

s = s+1 

The second increment might or might not see the previous increment in T1. Similarly, the thrid increment might or might not see the side-effect of the second increment.

In improperly synchronized programs, a read will return the value of one the possible matching writes, according to conditions 1) and 2). This corresponds to data races. In properly synchronized programs, a read has one unique ancestor write.

  T1        T2

lock m
  |
s = s+1 
  |
unlock m  
          \  
          lock m
             |
          s = s+1
             |
          unlock m
         /
lock m
  |
s = s+1 
  |
unlock m                  

Implementing Happens-Before

Conditions 1) and 2) specify that side-effects might or might not be visible consistently to theads in improperly synchronized programs. They enable the implementation to not read the shared memory all the time, either using CPU caches or compiler optimization. The specification does however never speak of specific implementation choice like caching and optimizations.

For instance, the code

t2 = s+1
t3 = s*2

could be rewritten as by a compiler optimization

t = s
t2 = t+1
t3 = t*2

, where t caches the value of the read to s.

Typically, the compiler will emit memory barriers to flush the CPU caches. The JSR-133 Cookbook provides guidance about implementation issues.

Data Races

Failure to properly synchronize accesses to shared data is called a data race. Inversely, a program is data race free if all accesses to shared state are synchronized. But what does data race freedom exactly mean?

Is the program data race free if,

  1. accesses to shared data happen within critical section?
  2. shared data is declared volatile?
  3. accesses to shared data are never concurrent?

Issues about the memory model are typically discussed using the double-checked locking pattern, and the “pausable” thread pattern. They provide only partial answer to these questions.

Point 1 is true. If the accesses to shared data are protected by the same cricital section, there is no possible data race.

Point 2 is true also. If you define variables as volatile, the side effect will be made visible to the other thread and there will be no data race. But remember: data race freedom does not mean that the behavior is correct. Consider the trival example below:

counter = counter + 1;

Making the counter volatile won’t suffice to ensure all increments are recorded.

Point 3 holds, but requires some clarification of the underlying assumptions (Answers to my stackoverflow question failed be clear cut and authoritative). Let us consider the situation when multiple threads manipulate an object X that is plain data, but alternate their temporal execution (so that X is never accessed concurrently) via another object Y that rely on concurrency control (wait, notify, synchronize). Should fields of object X be volatile or not?

If we assume that threads are synchronized using at least one the concurrency control primitive — an not just temporal alternance thanks, say, to a well-time sleep statements — this implies the alternate acquisition of at least a lock m, as for instance in:

while(true) {
    synchronized(m) { wait(); }
    counter = counter + 1;
    synchronized(m) { notify(); }
}

There is a happens-before dependency between the increment statement and the release of the wait lock. By construction, when the lock is released, exactly one thread will acquire it and wait again. So exactly one increment will happen-after the release of the wait lock. By consequence the increment is guaranteed to see only the value of the previous increment–no need of volatile.

More

Reflecting on Reflection

(A cheap title, I know)

Reflection is one of these concept that constantly move from “obvious” to “puzzling” in my head–you feel like you have a firm understanding of it, and then suddently it evades, you feel puzzled by certain aspects, and ultimately confused about the whole thing. Why?

Two Perspectives

One of the first source of confusion is that reflection can be apprehended with two perspective: code as data, and meta-objects.

The first one, “code as data”, corresponds to the ability to dynamically convert code of the running software into data, and to dynamically evaluate data representing code. The code and its representation are however not connected: modifications of the data have no impact on the running system until data are evaluated. This perspective is best represented with language of the lisp family.

The second one, “meta-objects”, corresponds to the ability to expose meta-objects that reifiy other aspects of the running system. Modifications of meta-objects impact the running system immediately since there is (ideally) a causal connection between the two. This perspective is best represented with language in the smalltalk family.

Understanding Reflection in Terms of Bindings

My take is that there are essentially two kinds reflective operations that can be understood in terms of bindings: (1) dynamic source code inspection and changes that reveal, update, or create bindings, and (2) dynamic code execution that exploit existing bindings.

Reflection can operate at two levels, the base and the interpreter level. The base level correspond to the program, and the interpreter-level to the execution of the program. (The interpreter-level is sometimes called meta-level) A program can reflect on its (a) own code and state, or (b) on the code of the interpreter and its state.

Category (1) corresponds to changes that could achieve by generating sources for the running software, persisting the state of the base and the meta level, modifing the source, restarting the software and restoring the state of the base and the meta level. Note that the state of the program might be transformed during the process. Category (2) corresponds to dynamic code execution that only exploit existing bindings. Unlike category 1, it can perform late binding using information known only at run-time, for which not source could be generated. Also, dynamic code execution can be used to jump across meta-level. Indeed, code at one level, is data for the level below. Note that when transfering data accross meta-level, data might be marshalled and unmarshalled.

Combining both categories (1)(2) and (a)(b) gives four kinds of reflective operations.

Both perspectives are equality powerfull, but each perspective represents a way of thinking about reflection that favors either (1) or (2).

Categorizing language features

Let us know walk through traditional reflective features and see how they can be categorized.

  • Enumerating fields — 1a
    The program inspects its own source code. For class-based languages, enumeration is usually performed with a first-class class that reifies the structure and behavior of a class. Special language construct could exist for enumaration, think for instance of  the for construct in Javascript.
  • Adding a field — 1a
    The program modifies its own source code. It is similar to enumeration.
  • Reflectively updating a field —  2a
    The program dynamically executes a statement that update an object, i.e. eval( "anObject.field = newValue"). The language might have special construst to do this, e.g. anObject at: "field" put: newValue in Smalltalk or might rely on first-class classes.
  • Reflective invocation of a method — 2a
    The program dynamically executes a state that invokes the method, i.e. eval("anObject.message(params)"). The language might have special constructs to do this, e.g. anObject perform: "method" with: arguments in Smalltalk or might rely on first-class classes.
  • #doesNotUnderstand trap — 1b
    The #doesNotUnderstand trap is a reification at the application level of code that run at the interpreter level and modify the semantics of message sends. This is conceptually equivalent to source code changes in the interpreter itself. Namely, instead of raising an error, message that can’t be delivered are for instance re-routed to another destination.
  • Method lookup — 1b
    Certain platform have other traps to customize method lookup and dynamic method dispatch, e.g. DLR, Java, Smalltalk/X. They are similar to #doesNotUnderstand.
  • thisContext meta-object — 2b
    the interpreter’s stack is reified at the applicative level via a meta-object. It corresponds conceptually to application code being evaluated at the interpreter level, so that it can inspect or modifies the internal state of the interpreter. The code of the application correspond to data that are evaluated one level below.
  • Continuations — 2b
    Continuations are similar to thisContext reification. The interpreter evaluates code which can access its internal state, and returns a data structure (the continuation) to the application; or it evaluates code that consume a data structure (the continuation) that is used to update its internal state.
  • First-class function — 2a
    First-class functions are gloryfied dynamic reflective invocations. A method, block, closure, function represents code. When invoked, it creates a new activation frame on the stack. Regular methods can be reified to be passed around for reflective invocations, e.g. aMethod.invoke( target, param ). Methods can also be simply reified as strings: target.perform( methodName, param ) or eval( target + "." + methodName + "(param)" ). A function that does not close over any variable is mostly an anonymous method. The main difference is that its reification as a first-class object already binds it to a given target, e.g. aBlock.value( params ). Functions with bound variables refer additionally to their outer context which is an activation frame.
  • Modification of behavior — 1a
    Modifying or addition behavior to existing objects correspond to source code change. New behavior can be loaded by evaluation code that return first-class function, e.g. eval( "[ 40+2 ]" ). Behavior can be added or modified by changing the corresponding reified entity such as method dictionary via first-class classes. The whole process might be exposed via other interfaces. For instance, Java’s Hotspot accept raw bytecode to replace a method.
  • Dynamic class loading — 1a
    Clearly, it correspond to dynamic source code changes. Dynamic class loading can be see as a one shot operation, or split into class creation, and incremental addition of structure and behavior. Class loading is a reflective operation that might be exposed via first-class classes (e.g. Smalltalk’s Class>>subclass:name:fields), or via specific interfaces (e.g. class loader).
  • Query all instances of a class — nothing
    If the class is provided as a literal, e.g. MyClass.class, there’s nothing really reflective going on. It could be considered as reflective if it is eposed via first-class classes, and reifies internal information of the interpreter.
  • Eval — 1a and 1b
    Since reflection is all about treating code as data, eval is the one superpower.  We have shown in the list above several eval of types 1a and 1b. Its type depends on the data being evaluated. In its unrestricted form, it can be a mix of both.

Structural and Behavioral Reflection

Reflection is sometimes split into structural and behavioral. Structural reflection deals with reifying structure (classes and methods as objects), while behavioral reflection deals with reifying execution (#doesNotUnderstand, thisContext). Behavioral reflection map to category (b) and structural reflection to category (a).

Correlated use of Reflection

Interestingly, if an application does not leverage operations of kind 1a, it does not need 2a neither.  Use of reflection (1a) and (2a) are correlated. Indeed, if reflection 1a is never leveraged, source never change, which means the entites are all known statically. While it might be usefull for consiceness to rely on operation of type 2a, such operation could be translated in an equivalent forms with non-reflective code. For instance, if methods are never added to a system, dynamic reflective invocations could be avoided with proper use interfaces, and exhaustive if-else cases (if( action="doIt" ) { doIt() } else if (action ="doThat" ) { doThat() } ...).

Conclusion

We tried to make sense of reflection and categorize existing features in a simple framework, and tried to relate two perspectives about reflection: code as data, and meta-objects. Going through a list of typcial features shows the advantage and disadvantage of both perspectives. For instance, reflective invocations are best understood in term of code as data, but stack reification is more easily though in term of meta-object. Ultimately, features can be explained in term of both perspectives, possibly with some mind twists.

references

my citeulike #reflection tag

Three Kinds of Typed Languages

There are a lot of questions on stackoverflow about difference between static and dynamic typing, e.g. why is Smalltalk considered dynamically typed. Here is my very own perception on the subject.

Background: Type systems are conservative static analysis

Here is the definition of a type system from Types and Programming Languages:

A type system is a tractable syntactic method for proving the absence of certain program behaviors by classifying phrases according to the kinds of values they compute.

The “kind of values they compute” is usually referred as a “type”. What the type system usually proves is that the program will not be “stuck”. Thinking of the semantics of the programming language as a set of evaluation rules, it means the evaluation of the program will terminate.

int x = 0;
if (false) {
   x = "hello";
} else {
   x = 1;
}

The above program will actually always terminate, since the true-branch will never be executed. However, it would fail to type-check, since the type system rejects x=”hello”. Reaching this statement would stuck the evaluation, since there is no consistent evaluation rules for it (unless the semantics define coercion rules). Type checking is a conservative analysis.

Run-time checks

For very simple language, evaluation rules do not perform run-time checks, and type checking usually means that the program terminates without run-time error. For any realistic program, there will be run-time checks for certain evaluation rules.

int x = 10;
int y = 0;
return x / y

One of the simple evaluation rules that requires a run-time check is integer division. Dividing by zero is not possible. If this situation occurs, a run-time error occurs. Run-time errors can either lead to the abrupt termination of the program, or the semantics can define error handling. Null pointers are other typical run-time checks.

Run-time type tags

Certain languages do not need run-time type tags to distinguish different kinds of structure in the heap. Evaluation rules depend on the static type solely, and the heap is flat array of unstructured data. (Lambda calculs with record and references, and C-like languages fall in this category, I guess.)

However, in many languages, run-time type tags are needed to distinguish different kinds of structures in the heap. Run-time type tags are requires to implement polymorphic behavior. In object-oriented language, this corresponds to interface and subclasses. The specific type of an object might be unknown, but it conforms to a type that is known statically.

Run-time type checks

Evaluation rules in such language perfrom run-time type checks. The check is necessary to “dynamically dispatch” the operation accordingly to the run-time type. Type systems can prove that the run-time type check will never fail, that is, the operation can always be dispatched, and the execution will not be stuck.

Three kinds of languages

The evaluation rules of the programming language, which define its semantics, can use static types and/or run-time types.

  1. Run-time type only — Let us consider Smalltalk. Smalltalk’s core semantics is entirely defined by the dynamic types: When you send a message to an object what must be executed depends on the dynamic types of the receiver. If a message can not be dispatched, the program is stuck. In practice, the error is reified into message not understood exception.
    The semantics of the language does not need a type system/type checking. You can however type check the code to guarantee that there will be no such run-time type error, see Strongtalk. With type inferrence, only minimal static types will be needed.  This is the approach that pluggable types promotes. The type system is optional.
  2. Static type only — A language with object, but without subtyping nor inteface. Such a language would not need type information at run-time. The heap is flat array of data whose type/structure can not be discovered at run-time. With proper information in the code about the expected type of objects, you can still define a valid semantics for the language: when you send a message to an object you know what must be executed since the type in statically known in the code. By definition such a language could not have dynamic dispatch and inheritance polymorphism. (The typed lambda calculus without subtyping is maybe of this kind).
  3. Static and run-time types — Let’s consider Java. It mixes both static and run-time types. Objects have dynamic types for dynamic dispatch and polymorphism. However,the exact semantics of the language rely also on static types. For instance, method overloading in Java depends on the static types of the paramters. A class can define methods print( A a ) and print( B b ). The semantics of anObject.print( anotherObject ) depends on the static type of anotherObject: when you send a message to an object, you define what must be executed depending on the dynamic type of the receiver, and the static types of the parameters. 

Note that by “static type”, I don’t mean the type is explicitely in the source. The static type is the result of a static analysis, and can be inferred. Inferred types can be the union of explicit types. The static types can be though of a “parametrization” of the evaluation rules.

function printHello( object ) {
     object.hello();
}
function printHelloA() {
     A a = new A();
     printHello( a );
}
function printHelloB()  {
     B b = new B();
     printHello( b );
}
Even if A and B are distinct types (not implement an interface or so), a type system can still figure out that printHello in passed either A or B and that both implement hello().

First-class functions

A language with first-class function can implement dynamic dispatch even without interface nor subclassing. Indeed, an object can store function that a client can access to emuate polymorphism. Functions have a type and are polymorphic by nature.
More pointers

The Object Cube

There’s the lambda cube, and the big data cube. Why not a an object cube? Here would be three axis:

  1. Object Existence — The object existence dimension can range for everything ia a mutable object, to more distinctions between mutable and immutable objects, or objects and values. Existence, identity and mutability are tightly related (“I have an identity therefore I exist”).
  2. Object Composition — The object composition dimension can range from no composition at all (an object define its unique behavior without any possiblity of reuse) to behavior reuse with class-based inheritance or prototype-base delegation, possibly with additional mechanisms like mixin and traits. Essential, the machinery of reuse implies late-binding and certain lookup rules. Composition is then also related to virtualization (of methods, classes). It is also about procedural abstraction.
  3. Object Interaction — The object interaction dimension can range from complete freedom (any object can interact with another object) to more displined approach with control visibility via module and namespaces, or behavioral restriction (read-only, priviledges, aliasing).

Let us consider inheritance. Inheritance itself is a composition feature: the behavior of a class can be flattened and duck typing is possible. The type system that enforce inheritance is an interaction feature that prevents run-time type errors, but also duck typing. Inheritance does not relate closely to existence (but hierarchies must be sound wrt to existence).

Object modularity could have been a way to subsume composition and interaction — making the cube a square — but composition and interaction capture two different kinds of features: composition is about defining the exact behavior of one object possibly involving other entites, while interaction is about the behavior of objects when two of them communicate. Both composition and interaction might rely on namespaces and scoping mechanisms, though. These mechanisms have no mean in itself, they serve a cause that is either composition or interaction.

More

2017.09.21: there’s also the Scale Cube!

Writing Immutable Objects with Elegance

Edit: This blog post has been turned into a draft paper.

Today, I got burned with design issues of Smalltalk’s dictionaries. They implement object comparison = using value equality, but are at the same time mutable. The internal state of a dictionary consists of an array of associations. In addition to problems of equality, the the internal state can be accessed and aliased with Dictionary>>associationsDo: and Dictionary>>add:

We should instead have an immutable dictionary using value comparison, and a mutable dictionary using identity comparision. Mixing styles is too dangerous. For the mutable dictionary, the internal state should never be accessible and data should be copied to ensure no dependencies on mutable state are established.

In an old post I rant on the conflict between the imperative object paradigm that promotes mutable state, and the functional paradigm that promotes immutable data structures. It is mostly a problem of readability, though.

Assignment Syntax

To make the use of immutable objects more readable, we could actually generalize operator assignment syntax like +=, -=, <<= to any message.  Postfixing a message with = would imply that the reference pointed to the receiver of the message is updated with the value of the message.

aReference message=: 5            
<-->      
aReference := aReference message: 5
aDictionray at: key put=: value   
<-->      
aDictionary := aDictionary at: key put=: value

When the selector has multiple arguments, the prefix comes at the very end. This is simple, efficient syntactic suggar.

What about self-sends? Accordind to the previous examples, self would be updated with the value of message. While it might sound counter-intuitive or plain absurd, this is what we need:

Point>>moveX: offsetX y: offsetY
self x=: self x + offsetX. 
self y=: self y + offsetY.
^ self

Assuming #x: and #y: return copies of the objects, wihth this semantics, the reference to self on the last line corresponds to the last copy created.

The only difference between self-sends and regular sends is the implicit contract that is assumed for the method. In the regular case, the message send can return any object. Any invocation of the returned object will happen with a regular send that will in the worst case raise a message not understood exception. For self-sends to work as in the example above, messages #x: and #y: must return instances of Point, so that the activation frame can be updated correctly. Updating the activation frame rebinds self to the new object, but preserves the temporary variables.

(I believe this would have an incidence on closures. More investigations are needed. The precise semantics could maybe be simulated with continuations)

Copy syntax

The previous proposal still leaves the burden of copying the objects to the developers. In the previous examples, #x: , #y: and #at:put: would need to first clone the receiver (self), then update it.

Point>>x: newX
^ ( self clone ) basicX: newX ; yourself.

Ugly, right? Following a similar syntactic approaches, message sends could be prefixed with % to indicate that the message must be delivered to a clone of the receiver:

aReference %message: 5 <--> aReference clone message: 5

We know that cloning is broken. However, it is not the subject of this post, so we will assume that we have a reasonable implementation of #clone. With % and = we have all ingredient to implement immutable structures easily.

Point>>x: newX
  self basicX: newX

Point>>moveX: offsetX y: offsetY
self %x=: self x + offsetX. 
self %y=: self y + offsetY.
^ self

(The accessor Point>>x is actually superfluous, since it is similar to basicX. It serves as example only.)

For an even more concise syntax, a third prefix/postfix could be introduced.

aReference ~message: 5 <--> aReference %message=: 5

Nested Objects

The proposed syntactic suggar has limited benefits for more complex mutation of objects with nested objects. Let’s consider an immutable circle with an immutable point as center.

Circle>>moveX: offsetX y: offsetY
   self ~center: (self center %moveX: offsetX y: offsetY )
  ^ self

But what we would really like to write is

Circle>>moveX: offsetX y: offsetY
   self center ~moveX: offsetX y: offsetY
  ^ self

Handling this situation so that the receiver “self center” is replaced with the new point, implies first the replacement of “self” with a new circle. The replacement of the receiver “self center” (that is not an L-value) could be achieved if by convention the corresponding setter is used. The above code would then execute implicitely “self ~center: xxx” to replace “self center”. This corresponds to the intended behavior. In other words,

self a ~m: args <--> self ~a: (self a %m: args)
self a b ~m: args <--> self ~a: (self a %b: (self a b %m: args))
etc.

The ~ can appear only before the last message send. The statement “self a ~b m: args” would be ill-defined.

More Links

Transformation for Class Immutability

Features Interaction

Hoare’s famous paper “Hints on Programming Language Design” distinguishes two main aspects of language design:

  • Part of language design consists of innovation. This activity leads to new language features in isolation.
  • The most difficult part of language design lies in integration: selecting a limited set of language features and polishing them until the result is a consistent simple framework that has no more rough edges.

I will try to keep here a list of inspiring articles that illustrate this two parts, especially edge cases in integration of individiual features.You can also follow Shoot Yourself In The Foot for examples of unexpected problems resulting from the language design rationale.

Development Hygiene

I recently attented three keynotes: Alan Kay and Andreas Kuehlmann at TOOLS and Jeff Kramer at ICSE. Hearing these talks, the software industry seems to be stuck: we don’t know how to move beyond manual development and testing of fine-grained software artefacts. However, as software engineering matures over time, a body of best practices emerges that increases quality of development. For instance, Jeff Kramer showed that certain ideas from the architecture community had indirectly made their way to industry under another form, e.g. dependency injection.

Andreas Kuehlmann used the word hygienic to refer to teams with sound engineering practices. The wikipedia definition of hygiene is a “set of  practices employed as preventative measures to reduce the incidence and spreading of disease”. High hygiene standards favor clean code. Assertions, contracts, refactoring are examples of hygienic practices. Hygienic practices lower the risk that your software become a drity pile of code. I like this analogy with health.

Progress in health promotion didn’t happen in one day. Medical and everyday practices such as sterilization, quarantine, treatment of wastes, dental care take took time to be established. Next to these practices, the healthcare industry enjoyed several breakthrough such as the creation of vaccines, or organ transplants that enabled this remarkable progress. Progress in development is similar to progress in healthcare.

  • Testing hygiene. Unit testing is similar to the daily routine of brushing ones teeth after each meal. It’s a defacto standard practice. Testing hygiene prevents bugs.
  • Code quality hygiene. IDE with code styling enforcement prevent unreadable code from spreading.
  • Modularity hygiene. A vast set of principles help keep software modular and prevent coupling and big balls of mud.
  • I/O hygiene. Sanitize your inputs and protect yourself from a malicious external environment. Don’t disclose sensitive information to the outside.
  • Monitoring hygiene. The first step to stay healthy is to identify problems and remedy before it’s too late. Yearly check-ups are becoming more and more common to detect symptoms of diseases. The same is true for applications, with monitoring, stress tests, and periodic reviews.

Hygiene refers to daily common pratices. Healthcare directly improves with improvement in hygiene, but also needs expert at time. When somebody is sick, he is sent to the hospital to follow a cure. For software, when an unreliable component is detected, it is put in quarantine, then analyzed and fixed with proper tools and techniques.  Expert troubleshooting is the equivalent of surgery.

Finally, what makes software remarkable is that similarly to organisms, software is grown, not built. Also, organisms and software are highly dynamic systems. This makes the analogy even stronger.

Hygienic programmers grow clean code and run healthy software.

MORE

Using Multiple Google Calendars with iOS 5

With Google calendars, you can create additional calendars linked to your account. This is convenient, say, to split your own events from events of others that don’t use any online calendar but that you want to track.

With iOS 4, adding a google account would display only the primary calendar, not the auxiliarary ones. The solution then was to add them individually as WebCal calendars. The WebCal URL is somehow cryptic but could be obtained from the ID of the calendar found in the settings.

After upgrading to iOS 5, all events are duplicated n times, where n is the number of auxialliary calendars. Sounds like an aweful bug, isn’t it? Actually not, things only got better. The auxiliary calendars are now correctly supported.

Go to m.google.com/sync and select the auxiliary calendars you want to sync. The google calendars you selected will all appear under you google account on iOS 5 (maybe you need to recreate it, though). You can remove the spurious individual WebCal calendars safely.

Sandboxed Evaluation

JSON is a data exchange format, and a the same time is valid javascript. You don’t need a parser to read them, and you can use the almighty eval to interpret the data. This is however a security breach. Malicious JSON data could be provide and would be evaluted without further notice. Parsing is therefore still necessary to validate the data. There’s nothing really new there: reflection and security are known to be orthogonal.

It doesn’t need necessary to be the case, though. Reflection could be limited so as to allow what is necessary and deny malicious actions. In the case of JSON evaluation, one should prevent the access to global data, and the creation of new functions. That is possible, e.g. with a custom first-class interpreter. The data might still not be valid JSON data from point of view of the data format, and might contain expressions. More advanced “security” policies could specify execution protocols that would be forbidden, so as to prevent the executions of certain expression, or sequences of expressions.

But then comes the question of error handling. Isn’t it “too late” to catch errors at run-time? An error–even if cought in time–might leave the system in a invalid transient state from which it is hard to recover. Validating the data statically prior any execution seems therefore an easier approach. This is for instance the approach taking by the Java platform: mobile code is verified when loaded.

Checking for potential errors statically is however more restrictive than checking for errors at run-time. Similarly to type systems, valid programs or data might be rejected because of the conservative nature of static checking.

Going back to a dynamic perspective on security, we need a way to (1) catch errors at run-time and (2) recover from errors at run-time. To provide the same level of security as its static counterpart, such an approach should make sure that nothing can happen at run-time that can not be undone. Recovery blocks, transactional memory, and first-class worlds are all techniques to enable undo easily.

What can be recovered must match what is allowed. Undoing changes to memory will not be enough if one can read from I/O streams ; either I/O operations should be denied, or the recovery mechanism should be strong enough to safely undo I/O read and write if needed. Spawning threads might be allowed, if we are able to detect and escape from infinite loop and reliably cancel execution of untrusted code.

Let’s take for instance that case of Java Enterprise Edition (JEE). Applications deployed on JEE application server must obey such specific restrictions. Most constraints arise from the transactional nature of EJBs, and the fact that clustring should be transparent. JEE applications run in a managed environement and should not spawn threads, access static data, performing blocking I/O, etc. they should be side-effect free. The sandboxing facilities of the JVM do not suffice to enforce these constraints, and many of them are conventions. The run-time itself does not prevent messing with the infrastructure.

We need a configurable runtime with adequate programming language abstractions to define and enforce security policies at various level of granularities. Such a runtime should be flexible enough to enforce the safe evaluation of JSON data, and the safe execution of JEE applications.

On Parallel Class Families

I discussed yesterday with Raffael Krebs, a master student at SCG, who was disappointed of his design in Java that lacked extensibility. The main problem was that he wanted parallel class hierachies, which you can’t do easily in Java.

This problem isn’t new. Let’s illustrate the problem and explore a bit the various design in pure Java, then with existing language extension.

Below is a sample situation. The core of the model is composed of Document, Item, Text and Image. The Reader reads data from a file and creates the model. The client uses the model. To be generic, it must be possible to specialize the model with as little effort as possible, that is, provide reuse in a natural way. One would like conceptually to specialize (that is, subclass) the model for instance for HTML documents.

This conceptual view can not be achieved with Java. There are several problems: (1) multiple inheritance is not supported, (2) the Reader class still references the core model and will not create instance of the more specialized one, and (3) in most statically-type languages, parameters in a subclass must be covariant to those in its superclass.

The two first problems can be circumvented with interfaces and factories. The third one might require a downcast in add(), if specialized classes have added methods. If only existing methods have been overriden, traditional polymorphism is enough.

Generics provide a partial solution to the problem. Document can accept a parameteric type <A extends Item>, so that HtmlDocument can be defined as an extension of Document<HtmlItem> (see this answer for an example). The downcast in method add() disappears. Item is however abstract, and the system assumes there are two implementations of it, Text and Image. With generics, the class Document can not create instances of these with regular instantiations new Text() or new Image(). First it would not comply necessary with the parametric type <A extends Item>, second, these instantiations would not be rewired correctly to their corresponding class in the other family. If classes were first-class, one could at least define abstract static factory methods createText() and createImage() in Item. This is however not  possible in Java, and static method can not be overriden, because classes are not first-class.

In a dynamicaly typed language, the usage of interface vanished, as well as the third issue. However, in both case problem (1) and (2) remain.

The lack of multiple inheritance in this case leads to code duplication between the classes for which not inheritance relationship exist. To avoid such code duplication, delegation could be used and an instance of HtmlText could delegate to an instance of BaseText. As pure delegation is not supported, this require writing boilerplate forwarding methods, which is still clusmy. Another way to avoid code duplication would be to factor the duplicated code into traits or mixin that can be reused among unrelated classes.

No matter what, this design is ugly–and it is really found in practice.

Revisiting the three problems

The  problems discussed before can be rephrased into three separated questions:

– How can the reader produce the right kind of document (either basic, or html) without invasive changes?

– How can variations of a class hierarchy be created with minimal efforts and no boilderplate code ?

– How can the type system be aware of logical relationship between types, e.g. HtmlDocument always goes with HtmlItem?

The fourth problem

One fourth problem not depicted but that happens in practice, is that two class hierarchy relate to each other, but in slightly incompatible ways. This typically happens due to software evolution, when method are renamed, signature are changed, etc. An example would be to change the “data” in Item from binary byte[]  to base64 string. While certain kinds of changes can be accomodated by providing convertion functions (they act as wrapper), this bloats the design and prevent clean evolution.  Sometimes it is impossible to create an inheritance relationship between the two variants of the classes, which  means that whole graphs must be adapted back and forth from one representation to the other. We state the fourth problem as follows:

– How can graph of objects be converted from one logical representation to another easily, whithout that the convertion logic bloats the code?

The fifth problem

One fifth problem not discussed so far is actually a requirement:

– How can the new families be introduced a posteriori, without entailing modification or invalidation of exisitng code.

This problem is usually referred to as modularity.

Language constructs

These problems are clearly related to each others, as is showed by the example.  A definitive solution to all three of them is to my knowledge still not available, but there has been progress in programming language to provide means to tackle part of them.

  • Design patterns (strategy, factory)
  • Generics
  • Static & dynamic reuse mechanisms (delegation, traits, talents)
  • Family polymorphism
  • Virtual classes (newspeak/gbeta, dependency injection, first-class class)
  • Object Algebras
  • Local rebinding (classbox)
  • Open classes
  • Multi-dimensional dispatch (subject-oriented programming, context-oriented programming, worlds, namespace selector)
  • Non-standard type coercion (translation polymorphism in Object Team & lifted Java, expander)
  • Type versioning (type hot swapping, upgradeJ, revision classes)

It seems to me there is room for something actually missing that would solve the problem depicted above–or myabe it exist but I am not aware of it?

Links