Scalability tips

• Asynchronous event-driven design: Avoid as much as possible any synchronous interaction with the data or business logic tier. Instead, use an event-driven approach and workflow
• Partitioning/Shards: You need to design your data model so that it will fit the partitioning model
• Parallel execution: Parallel execution should be used to get the most out of the available resources. A good place to use parallel execution is for processing users requests. In this case multiple instances of each service can take the requests from the messaging system and execute them in parallel. Another place for parallel processing is using MapReduce for performing aggregated requests on partitioned data
• Replication (read-mostly): In read-mostly scenarios (LinkedIN seems to fall into this category well), database replication can help load-balance the read load by splitting the read requests among the replicated database nodes
• Consistency without distributed transactions: That was one of the hot topics of the conference, which also sparked some discussion during one of the panels I participated in. An argument was made that to reach scalability you had to sacrifice consistency and handle consistency in your applications using things such as optimistic locking and asynchronous error-handling. It also assumes that you will need to handle idempotency in your code. My argument was that while this pattern addresses scalability, it creates complexity and is therefore error-prone. During another panel, Dan Pritchett argued that there are ways to avoid this level of complexity and still achieve the same goal, as I outlined in this blog post.
• Move the database to the background - There was violent agreement that the database bottleneck can only be solved if database interactions happen in the background.

(src- http://natishalom.typepad.com/nati_shaloms_blog/2007/11/lessons-from--1.html)

Roughless Pragmatism or RP

Rule 1 of RP: Never paint yourself into a corner

This rule basically states that when you design software in an agile manner (i.e evolutionary design) always make choices that allow for maximum flexibility. This could translate to different things to different people and is really quite subjective. That is what is great and wonderful with software development, its so creative and yet technical and abstract at the same time.

Software development is all about having abstract thought models in your head. Its very unnatural to live in one reality (our dimension) and act in it based on another reality (some conceptually compatible, yet a very much simplified one.) This is where OO, DLS and so on comes in. Tons of stuff have been written about this :)

More rules will be posted as they are realized...

Distributed web architecture with Spring

Overview
I have been working on a fairly interesting distributed architecture for some time now and this post describes this work. What makes this architecture interesting is the use of AOP to externalize coordination and communication between a various instances of the same server.

In this architecture a server can run in either:

• Single mode - in this mode the server encompass both the master and slave modes
  
• Master mode - in this mode the server serves as the main entry point for requests. A master can interact with several slaves depending on availability
  
• Slave mode - in this mode the server is serving as a slave awaiting requests from a master
  

A server running in single mode is what we typically understand when we talk about a server. It really only becomes interesting when we are breaking this server up into a distributed topology. All extra logic to handle the difference between these is handled by aspects. The master is the central coordinator in this configuration and dispatches requests to a potential slave when needed. The master contain all information on what requests to handle locally or which ones to dispatch to a slave.

AOP interceptors are used to handle the extra logic that has been extracted from the master and slave. Both method interceptors and request interceptors are used. The method interception is used for things like user creation, user update, invitation and registration. The request interception is used for functionality that should be handled by a slave.

This type or implementation gives a lot of flexibility in that once the single server is implemented the master and slave specifics are added on top as aspects. This makes the architecture easy to configure and modify. The architecture has been made modular such that the master and slaves together forms a single server that is driven by a single interface via a web browser.

The request dispatching mechanism cooperates with Registry in order to dispatch the slave specific requests. The configuration of these requests is stored in a application context configuration. The UI is constructed on the master/single server but individual requests are processed based on the request configuration. To make it simple, it is assumed that all requests are processed by the master/single server unless configured to be processed by a slave. What slave to dispatch the request to is dynamically determined by the Registry. All slave servers registers with a central master server upon startup.

Once a user logs onto an master, a session is also created with the slave associated to that user. The session is stored in the registry. The slave to authenticated against is defined during user registration where the slave is determined.

Both the master and slave server stores the same user information. This is needed in order to be able to handle all the various use cases.

The beauty of aspects I that they can be applied to code to add new functionality that was not originally thought of. Code or functionality can be decorated with new functionality. Another powerful benefit with aspects is that they allow for common code that crosscuts different conceptual areas to be externalized and thus be shared which reduces the amount of code that has to be written.

Writing a book

I am very excited. I am back home in my home town in Sweden and met up with a really smart friend of mine from when I was younger. We dont meet up very often but when we do we always spend quite some time discussing various topics from politics to computer science and philosophy (often not leaving a single drop of Glenmorangie left.)

We have both been active working in the software industry for quite a few years now so you can imagine that our discussions can become quite animated when it comes to sharing our experiences.

For some time now I have been experimenting with the idea of writing a book about my exeriences in this field and how I feel about developing software and my passion for it. And as I mentioned this to my friend he got so interested that he suggested what we should do this in a joint fashion, and I though that was an excellent idea.

So we have now started this little project which I think will be a really cool one. I dont expect anything really other than it being a learning expericence for me/us. And as my former coworker (mr potatoehead) - "Frick", once explained to me, once you are explaining something to someone, you are focusing and structuring your thoughts in such a briliant way that the world is clear as mud.

NORM

I have been working on a little framework that I think might be useful then developing applications on a Compact Framework device. Its an ORM for PDA's inspired by Hibernate. I developed the initial framework in a day or so just coding away in a nice flow fashion and I am quite happy with the result. Its called NORM and is available from here.

The basic idea was to try to abstract away all the SQL specific stuff from the programming model to allow for working with POJO's as easy as possible. Lets just say I got one step to bored with having to think/work/program in terms of Tables, Rows, DataSets and so on in .NET. For me that is just to low level. I want to work with proper objects (POJO's), and I want the CRUD operations related to these to be as transparent as possible. The less code I have to write the less I have to worry about.

The framework itself relies heavily on Annotations and Reflection as I am a strong believer in a self-contained representation model. Thus I should not have to configure something unnecessarily in a configuration file (XML or something similar) when the intent can be clearly specified in the class file itself.

Some basic assumptions regarding the framework

• The schema must match that of the domain model (currently working on an schema auto-generator)
• There is a one to one relationship between a table and a class name (unless otherwise specified using an Annotation)
  
• There is a one to one relationship between a field and a table column (unless otherwise specified using an Annotations)
• The many2one relationship must follow: (src table) id -- field_id (foreign table)
  

This is implemented so far:

• All fields are persisted unless [Transient] is specified
• All fields are persisted unless [Transient] is specified
• Only classes with [Entity] can be persisted
• Strings are assumed for all fields unless the Id is field
• The Id field is an Int32 which cannot be null so it is assumed that a class with Id == 0 is fresh and can be inserted
• Attributes to specify the persistence option:
• Entity (speficies that this class is persistable)
• Id (speficies this field to be the primary key)
• Component (flattens the the object graph)
• Transient (do not persist this field)
• Inserts an object graph (semicascading)
• (Only persists object with the [Entity]attribute specified)
• (Only fields != null are included)
• (If the id != 0 dont bother with generating an insert)
• Update individual objects
• (No dirty checking but only fields != null are included)
• Load an object and automatically populate it with the data from the DB
• Supports QBE whereby only fields != null are included
• Load an object graph (shallow cascading)

Features that I am working on at the moment

• Collection support (requires implementation of laziness)
  
• Dirty checking (requires a 1 level cache, Session or something)
• Support for more databases and dialects (currently only SQLite)
• Persistence by reachability (no dynamic proxies in CF, so have to rely on a CRUD service)
  

I am looking for developers that are interested in contributing to this framework. If you have been using NORM in a project yourself please drop me a line or two and tell me about your experience. Any feedback is more than welcome.

More information is available from the sourceforge.net site...

Groovy vs. Ruby

Both are very compelling and provide with very powerful abstractions...

but...

• I don't really like the syntax in Perl and unfortunately Ruby is living in that space..
• Groovy seems a bit lagging behind in areas like parallel assignment
  
• No good IDE support overall, maybe next year :)

despite that...

• Groovy really shines with its natural Java blending capability
• I love the Grails project, showing what is possible with Spring and Hibernate
  
• Dynamically typed languages are just sweet... write less and no interruptions in your flow
• Groovy just added Annotation support, that is way cool
  

I devoted some nerd-time to learn both last year and I really enjoyed the exposure. I think its important to experiment with different conceptual tools such as programming languages from different paradigms as it expands on the set of potential solutions you can dream up for a particular problem.


So for example...
Say for example how nice it would to be able to do something like this in Java:

Map stuff = {"locale" : "en", "theme" : "basic" }

...but of course we cant so we have to resort to something like this:

Map stuff = CollectionUtils.asMap(new String[][] {
{"locale", "en"}, {"theme", "basic"}} );

There is nothing in Java that is called CollectionUtils.asMap(String[][] map) so we have to create this ourselves, but without the exposure to the Map construct in a higher level language like Groovy it would be less likely that we would create a method asMap() as in the example above just because we had not experienced or perhaps not understood the power of such an abstraction.

Hmmm, so why is this so powerful, well it has to do with expressiveness and preciseness. Math is an excellent example of how you can cram vast amounts of precise knowledge together to describe a phenomena in nature... but just as Perl its syntax is quite obnoxious. I think I like math more :)

The way of Scrum

Was talking to my good friend about planning a software project and I wrote down this little overview on how I tend do it but it could change from one project to another...

Initial setup
You do an initial inception/elaboration phase where scoping takes place. Not all requirements are gathered here through. Maybe 30%-40% of the major ones.

Then for each sprint (iteration) you do analysis, design, development and testing over and over again. You find more requirements here maybe around up to 80%-90%. You should do integration here progressively.

Rule of thumb NEVER WAIT UNTIL LAST MOMENT WITH integration, testing, security and so on.

Then in the Closure/Transitional phase you complete/finalize the project.


Details around Scrum and Agile project development
So lets say you have a 4 week sprint (with is the name of one iteration when)...

So each software component would have to be broken down into smaller chunks that each should not take more than 8-16 hours (1-2 days) to complete. With completion here I mean:

Analyzed, designed, implemented and tested.

Backlogs
The team of stake holders would sit down and decide what the project should deliver. The following artifacts are produced:

Product Backlog - for the whole product (very high level, specifies the complete product)
Release Backlog - for a particular release (more details added, subset of the product backlog, for a release say v1.5)
Spring Backlog - features picked for a particular sprint, subset of release backlog (1 iteration worth of features)

The process and phases
So the Scrum project is divided into the following phases:

1. Review release plans (done by developers, picking from the backlog)
2. Distribution review and adjustment of product standards (done by developers)

This is where the majority of the work is done

3. Sprint (developers doing work, 4 weeks typically)
4. Sprint review (by stakeholders to determine next sprint, determine completion based on complexity and feedback)

5. Closure (last finishing activities, debugging, optimization, marketing and release)

So lets say for a 5 sub-projects and 5 components gives you 25 items in your backlog

So given that you have 5 months to complete the project you can break down the backlog into roughly 25/(4/20) = 125 chunks per sprint. So say you have 5 people in our team then each developer will work on around 25 chunks per iteration. That is your backlog "Burndown" rate... something called the project velocity in XP and other agile methodologies.

You need to be able to keep track of the Velocity in order to be try to track how the project is progressing. You need this as you would want to to be able to check your estimates... So lets say that one spring your velocity is 25... then if you calculating backwards you will find that you will be able to do 125 chunks based on having a team of 5 developers. But say that for one sprint your velocity is only 15 then you know that for the same number of developers you can only deliver 75 chunks so that will be your estimation baseline for the next sprint... So by using these instruments you can adjust the project plan and sprint backlog accordingly.