Friday, 25 January 2008

New adventures

Well this is it. I have now done 3.5 years at TEAMworks and its time for me to toss in the towel. It has been enriching to say the least. I don't think I have been working in a more tolerance, patient and mentally challenging environment in my life.

My motivation, focus and technological evangelism took a hefty smack on the head somewhere along the road but I am finally over that now.

I joined this company with the wish and dream that I could create something from nothing utilizing all my collected experience in all sorts of technological, social and educational settings but it seems that all the effort was for nothing. There are so many things that went wrong along the way so there is no need to detail it here. But worth mentioned maybe...
...the struggle with finding experienced and skilled staff and the problem with trying to keep in all together because nobody liked each other or to scared to say anything
...the boss/CEO who decided that nobody knew anything, who didn't wanted to listen to reason and advice from obvious expertise and who went his own way with his penny wise, pound foolish mentality
...the ignoring of my requests of key personnel and skills in interaction/graphical design with the motivation that "we can do it ourselves, it takes to long time, we have to go GLOBAL NOW!!"
After 3.5 years of working here my boss is someone you are afraid of. Every person that has left the company (or has been fired) refuses to come back to the office to visit us less the boss is not there.

Hmmm... its nice to be away from that hellhole. Initially I didn't really dislike everything that had to do with that company. So, it has been said. Its done I can move on.

Pax mobiscim

Wednesday, 16 January 2008

Simplistic Thinking

I recently came across a Google interview questions that kind of intrigued me. It went like this:
"You have a list of numbers which some of them are negative and some of them are positive, find out the largest sum of sequential subset of numbers from this list."

For example:

-5, 20, -4, 10, -18

The solution is the subset [20, -4, 10] which its sum is 26

The solution for this question is not summary the positive numbers or sort the list because the subset needs to be sequential.

A possible not efficient answer would be to to calculate the summary of all the possible subsets in the list but the complexity of this solution is O(N^2)

However the interviewer wanted a solution of O(N) !!!
This question is so full of subtleties that it cannot fail to entice even the most interview savvy reader. In order to be able to properly answer this question the interviewee not only has to understand big O, but also this particular type of problem. One brilliant aspect of the question is that it already contain parts of the answer. The answer can be found in the requested solution of O(N). This ought to give enough information on how the algorithm should be structured (or should it?)

O(N^2) solution (Groovy)

This solution illustrates a simple solution to this problem with O(N^2) complexity in Groovy.
def findLargestSubSequence(seq)
{
def tmpSeq = [], maxSeq = []
def maxVal = -9999999

seq.eachWithIndex { val, index ->
seq[index..seq.size()-1].eachWithIndex { innerVal, innerIndex ->
tmpSeq = sequence[index..innerIndex]
if(tmpSeq.sum() > maxVal) {
maxVal = tmpSeq.sum()
maxSeq = tmpSeq
}
}
}
maxSeq // no return needed, value is returned automatically
}

println findLargestSubSequence([-5, 20, -4, 10, -18])

And the O(N^2) comes from the nested iterations of course :)

O(N) solution

This solution illustrates one (not simple) solution to this problem with O(N) complexity in Java.

int[] findLargestSubSequence(int[] sequence)
{
int mini = -1, minv = 0, maxi1 = 0, maxi2 = 0, maxv = -9999999;
int[] seq = sequnce.clone();

for(int i = 0; i <> {
if (i > 0) {
seq[i] += seq[i-1];
}
if (seq[i]-minv >= maxv) {
maxv = seq[i] - minv;
maxi1 = mini + 1;
maxi2 = i;
}
if (seq[i] < minv) {
minv = seq[i];
mini = i;
}
}
return getSubArray(seq, maxi1, maxi2); // you have to write this :)
}
Here we see that there is only one iteration. The code is pretty hard to read though as it relies on a rewriting patterns where the path towards the solution is driven by successive alteration to the source input.

Saturday, 1 December 2007

5 abilities that improves a productive success rate

  • assuming personal responsibility for productive relationships
  • creating powerful partnerships
  • aligning individuals around a shared purpose
  • trusting when something is "just right"
  • and developing a collaborative mindset

This is a neat summary from the book: Teamwork Is an Individual Skill (ISBN 1576751554)

Agility cannot happen without collaboration, communication and feedback. But it will be even more efficient with team members realizing that its all about the people in the center... individuals are the ones that run the show :)

Another thing, as of now I have realized that the Agile term "self organizing teams" does not apply anymore because it carried a too chaotic connotation and implies weak or no leadership. There has to be a shared vision in order for a process to move forward.

Oh yeah I just remembered something I had to say here - its ok to say NO!

Over and out :)

Thursday, 29 November 2007

My kicks in life... yeah I am a nerd

I love programming!! End of story, talk to the hand.. I don't care what you think. It doesn't matter if I am working as an architect, technical director or janitor. I love programming and the creative mindset you are indulging in. To just sit and juggle with different concepts and translate them into running code is just so inspiring. I am trying to pick up a new language now and then and that has really helped me cross cutting several different paradigms.

If I am bored I just pick up a little problem that has bothered me and I just fix it. Like this other day. We have this product right where a mobile device is interacting with a Java server using Hessian. I wrote most of the Java server and the Mobile application so I should know how it works. There was something with the speed of the synchronization of the data that just didn't feel right so I sat down and started writing some tests for it. The mobile application is written in C# using Visual Studio. Unfortunately there are no freeware/opensource performance monitoring tools that I know of as of now that covers that platform so I had to resort to some of my own tricks. (I might write something myself as soon as the Compact Framework team is adding dynamic proxy support to the framework.)

So after some time I discovered that during the synchronization of the data some methods were called with their arguments evaluated dynamically. These dynamic arguments are quite expensive and should not be performed during sync. So we basically have this type of situation:


// running on the client in the syncService..
while(until no more dataChanges or exception)
{
serviceMethod(extractArguments(dataChange))
...
}
public void addToChangeLog(Date date, Operation op, ...)
{
if(IsSyncing)
return;

// add to changelog...
}

...
// in the proper service method, called during sync or normal operation..
public void serviceMethod(Type arg1, Type arg2, ...)
{
// 1) do work... save to DB and so on...
...
// 2) save changes to dataChangeLog to be picked up during next sync
// but should not be done if we are syncing already
syncService.addToChangeLog(currentDate, operation,
extractNewDataAsStringVeryExpensive(sourceObject));
}

In the call to syncService.addToChangeLog we have one argument extractNewDataAsStringVeryExpensive that is evaluated before the addToChangeLog is called. This method is very expensive in terms of execution time. But this is not directly obvious when running this piece of code just a few times. Lets say the argument takes 200ms to complete and this method is called 100 times then the total execution time would be 200ms x 100 = 20 seconds. That is 20 seconds wasted.

So by changing the code to something like...

public void serviceMethod(Type arg1, Type arg2, ...)
{
// 1) do work... save to DB and so on...
...
// 2) save changes to dataChangeLog to be picked up during next sync
// but should not be done if we are syncing already
if(!syncService.IsSyncing)
syncService.addToChangeLog(currentDate, operation,
extractNewDataAsStringVeryExpensive(sourceObject));
}

...we will fix the problem but have introduced a special case check in otherwise generic code. The check whether we are syncing already has been moved one level up to the calling code. Once would think that this problem could be handled with an around advice in an aspect where we have full control of the invocation of the target method, but no. The arguments would still be evaluated before the target method was called. If would be nice somehow to be able to have an advice feature whereby you would be able to decide on what arguments to resolve before the method is called, similar to the around advice.

Take away points:
  • the extractNewDataAsStringVeryExpensive method was not expensive initially. It evolved!!
  • it pays of having a CONTINUOUS PERFORMANCE task in place that would run as part of the continuous integration build so that performance increasing code changes can be detected properly. This is not a trivial exercise but fun never the less
Well anyways, it solved my problem for now and I don't think I will loose any sleep over it :)

If I come up with a better solution I will write about it here... Ciao

Tuesday, 27 November 2007

Book writing

After some initial excitement regarding writing a book about software development and my experiences with it I have discovered a new book written by Kent Beck (of XP fame) called Implementation Patterns which I believe perfectly covers some of the material I was planning to write about.

I am happy that Kent (whose ideas and work I respect a lot) has published this book and look forward to read it. I still have a few ideas though that I would like to work with in written form so more about this later. I might have to change the form and approach a bit :)

Monday, 26 November 2007

Jasypt experiences

We have been using the Jasypt framework for a bit more than a year in our product and its pretty good. Very easy to integrate, configure and extend to suit your own needs.

Few things though:

- Hibernate integration: you have to think a bit about your solution if you wanna encrypt PK-FK relationships, no out of the box solution... (yet?)

- In a development environment you probably don't want to use the PBE Filter + Servlet until deployment time. If you do, you probably want to automate the initialization of the PBE system at startup from a props file via a servlet listener or something..

Thursday, 15 November 2007

Scalability tips

  • Asynchronous event-driven design: Avoid as much as possible any synchronous interaction with the data or business logic tier. Instead, use an event-driven approach and workflow
  • Partitioning/Shards: You need to design your data model so that it will fit the partitioning model
  • Parallel execution: Parallel execution should be used to get the most out of the available resources. A good place to use parallel execution is for processing users requests. In this case multiple instances of each service can take the requests from the messaging system and execute them in parallel. Another place for parallel processing is using MapReduce for performing aggregated requests on partitioned data
  • Replication (read-mostly): In read-mostly scenarios (LinkedIN seems to fall into this category well), database replication can help load-balance the read load by splitting the read requests among the replicated database nodes
  • Consistency without distributed transactions: That was one of the hot topics of the conference, which also sparked some discussion during one of the panels I participated in. An argument was made that to reach scalability you had to sacrifice consistency and handle consistency in your applications using things such as optimistic locking and asynchronous error-handling. It also assumes that you will need to handle idempotency in your code. My argument was that while this pattern addresses scalability, it creates complexity and is therefore error-prone. During another panel, Dan Pritchett argued that there are ways to avoid this level of complexity and still achieve the same goal, as I outlined in this blog post.
  • Move the database to the background - There was violent agreement that the database bottleneck can only be solved if database interactions happen in the background.
(src- http://natishalom.typepad.com/nati_shaloms_blog/2007/11/lessons-from--1.html)