Complexity Creep: Data Scavenger

May 17, 2009

You’re given a simple task: get some XML data from a URL or web service, convert it to something else, and send it off down stream to some other system. Easy enough, right? Somewhere in the middle of your “80 percent done” report, you realize that the original XML is missing one silly little field – the customer’s middle name, the timezone on the date, the preferred nickname for their online avatar, whatever. Unfortunately, this little detail is critical for completing your task at hand.

If there’s no way to get this information, there’s very little you can do, besides ask for an enhancement and wait. But very often, the data’s there for the taking; you just have to go out and get it from somewhere else. Just as common, however, that “somewhere else” is not as accessible as you’d like. If you’re lucky, you can just pull another object or rowset out of the same database. If you’re not, good luck with that “80 percent” thing…

Here are some of the complications you may come across in a data scavenger hunt:

  • The data’s there, but you have to torture it out of data structures not meant to provide it (e.g. it’s buried in a string of text with no standard format, or in a numerical field with no clear designation for varying units)
  • The data’s on a remote server: performance may be impacted, you have to deal with the problem of making the call (if at all possible), handling errors, and so on
  • You don’t have the privilege to access the data
  • The data is not guaranteed to be transactionally consistent (you may be getting some stale data, or the new data reflects changes that aren’t seen by the rest of your data set)
  • The data is in a log file, system configuration file, admin-only database tables, or some other unholy “do not touch this!” artifact

Each one of these little beauties is a mini-rat’s nest of complexity creeps of their own. What do you do if your application doesn’t have privileges to get the data? Implement a “run-as” feature just to get the one field? Hard-code the root user password? Convince the guys on the other side of the fence to give you a user and wait?

In some cases, you may actually be able to modify the code itself to fit your needs. But that requires a new release of the software, which, if you’re accessing it from the outside, may not be an option (I’ll be posting another Complexity Creep article on this one some time soon). Also, this can lead to problems of its own: the “universal interface”, or the one-size-fits-all façade. I’ve worked with interfaces like this, where every new feature begets a new variation on the same old methods, just to avoid the need for scavenger hunting. It happens with database views, and in XML, too: to avoid making multiple remote requests, you keep adding new fields to your XML entities, until your Company XML document includes a list of all employees barfed up somewhere in the nested tags, complete with their “Address2” field and their in-flight meal preferences. This solution is the evil twin of the Data Scavenger: the Data Tar Baby.

Data scavenging is a common problem in integration projects, which is one of the reasons they can be so tough. But it can happen when building monitoring utilities, reporting features, or just about anywhere information is required. Unfortunately, when working in the “information technology” field, the odds are pretty high you’ll come across this more often than you’d like. And yet, when you do, it always seems to be that one last insignificant detail that turns your routine query into a scavenger hunt.

Complexity Creep: Aspect Infiltration

May 15, 2009

The other day, I was working on enhancements to a type of “job execution harness” that executes parallel tasks in our system. We had started out with the concept of “jobs”, essentially individual commands in a Command design pattern, and had recently evolved the idea of “groups” of jobs for managing dependencies between sets of parallel tasks. (note: just for fun I’m testing out yUML for the images here)

Harness schedules Groups and executes Jobs

Harness schedules Groups and executes Jobs

As with pretty much any complexity creep story, this one starts out with a pretty simple and elegant design. You basically had an execution harness, which took care of scheduling the jobs to be executed, and the jobs themselves, which were totally independent POJOs, with no knowledge of each other, nor of the harness itself. The harness itself provided monitoring capabilities which reported the start and end of the groups, and of the individual jobs.

Harness executes Jobs and reports to Harness Monitor

Harness executes Jobs and reports to Harness Monitor

We were living in separation-of-concerns bliss until the day we were given a new requirement: to support monitoring of job “steps”. “Um, what’s a job step?” we asked. It turns out that some of these jobs can take a looong time to run (several hours). Users wanted a way to see what was going on during these jobs in order to get a feel for when they would be done, and if everything was going ok.

Harness executes Jobs which contain Steps...?

Harness executes Jobs which contain Steps...

We wanted at all costs to preserve the harness-agnostic nature of our jobs. We thought about breaking the bigger jobs up into smaller jobs, but unfortunately, it wasn’t possible since they are essentially atomic units of work. We considered solutions for providing some sort of outside monitor which could somehow generically track the execution, but these steps were basically execution milestones that only the job itself could know. Finally, we knew we were defeated, and gave in: because of one little logging requirement, we were going to have to introduce some sort of callback mechanism to let the once harness-agnostic jobs signal to the harness whenever a step was completed.

Harness and Job both report to Harness Monitor

Harness and Job both report to Harness Monitor

From a pure layering perspective, you can see that we are in trouble if we are trying to keep all the harness code (in orange) in a top layer, and the business code below. So, what are some possible solutions to the problem? We could:

  1. Let the monitor know about the progress of the Job steps through some indirect method (e.g. through special text log statements, or indirect indications via data in the database). While it would avoid placing any explicit compile-time dependencies on the Job class to the harness, it would create a very fragile “know without knowing” relationship that the Jobs would have with the harness. Nasty.
  2. Create a special “StepLoggingJob” abstract class that these Jobs would extend in order to gain access to some special logging facilities. Basically, these Jobs would no longer be POJO classes, in the sense that I used the terms, since they would have to extend a harness-specific framework. Unfortunately, this introduces a circular dependency.
  3. Inject a special “StepLogger” utility class into the Jobs, either as a class member, or as a parameter on their “execute()” (or whatever) method
Option 1: Job writes special logging messages to a common store

Option 1: Job writes special logging messages to a common store

Job extends StepLoggingJob

Option 2: Job extends StepLoggingJob

Option 3: Job calls a StepLogger which reports to the Monitor

Option 3: Job calls a StepLogger which reports to the Monitor

Note that we still haven’t really solved the problem… the Job class still requires something in the “harness layer”. If we were using a dynamically typed language, we could do something of a mix between option 1 and option 3 by using duck typing (the Job would know it was getting SOMETHING that could log, but wouldn’t have to know it’s from the harness layer). In order to really separate the dependencies in Java, which we use, we have to create a new layer, the “harness API layer”, and place only the StepLogger interface there:

Job knows only about the StepLogger interface

Job knows only about the StepLogger interface

So, what happened? To summarize: because we wanted just a little more logging, we were forced to introduce a whole new layer into our application, and break the concept of 100% harness-agnostic commands. Is this an isolated case? Of course not. You see it all over the place, and logging is a great example of this. Have you ever heard someone talk about aspect-oriented programming (AOP), and give logging as an example? It’s PERFECT! With some simple configuration, you can automatically enable logging on all your methods without a single line of code. So, you can get rid of a ton of boiler-plate code related to logging, and focus on just the business logic, right? Wrong. If that were true, we all would have thrown our logging libraries in the garbage years ago. Instead, Log4J is still one of the most heavily-used tools in our library. Why? Because aspects work by wrapping your methods (and so on) with before and after logic, but they can’t get INSIDE the methods themselves.

The really useful logs are written by the method itself

The really useful logs are written by the method itself

I call this Aspect Infiltration: when your non-business infrastructure creeps into your business code. You can see this elsewhere, as well: in the J2EE container whose transaction control isn’t fine-grained enough for you (introduce a UserTransaction interface); in the security service that isn’t sufficiently detailed (give the code access to a security context), and so on. It’s a common issue for any container that wants to do all the work for you. There will come a time when the business code itself just knows better. And you’d better be ready to give it the API that it needs.

Complexity Creep

May 14, 2009

I’ve been silent in this blog for a while now not only because I’ve been busy with family, work, and organizing IASA encounters, but also because I’m reluctant to rehash anything that’s been written before. Fortunately, I think I’ve come across something worth the pixels its written on. While working on software design over the years, I’ve noticed a common pattern, no matter how unrelated design tasks are: every design starts out really simple and elegant, but at some point can grow into a warted perturbed perversion of the original idea. In some cases, in response to a new requirement or complication, a new solution latches on to the side of the core design, like a barnacle on the keel of a ship. Other times, the whole design can be turned belly-up, like the Poseidon after a storm. What’s fascinating to me is how the great upheavals in design are often caused by the smallest additional requirements.

Everyone has heard of the concept of “scope creep”, when the requirements seem to grow faster than you can code. I want to write about what I’d like to call “complexity creep”, those moments when a tiny little requirement can mean a whole lot of extra work for the development team, or even turn your basic design concepts on their head.

Since this will be the beginning of a series of posts, inspired by some of the most agonizing moments from my ongoing work as a software architect, I won’t post any examples here. Look instead for my next posts, coming soon, on two “patterns” (oh no! YADSDPC – Yet Another Damned Software Design Pattern Catalog*) of complexity creep: “Aspect Infiltration” and “Data Scavenger”. If you come across any moments of your own, please let me know!

* Note that I use the term “pattern” here loosely, as in a group of unrelated issues with recongnizable commonalities. I’m not really planning on documenting these like software design patterns.

Announcement: First meeting of the IASA-RJ study groups

March 13, 2009

The first official meeting of the IASA-RJ study groups will be happening on Thursday, March 26 from 7 to 9 PM. The idea is to eventually be running (at least) three self-organizing study groups separately. But for this first meeting, and until we decide that we have enough interest for each, we will try to touch on all three topics:

  • A study of the ATAM methodology for evaluating software architectures
  • Study and preparation for The Open Group’s ITAC certification for IT Architects
  • Study of The Open Group Architecture Framework (TOGAF) and preparation for those that want certification

That’s a lot to cover in a single meeting, and a very ambitious undertaking for a single study group. In this first get-together I will be giving an overview of the ATAM, and Marcelo Sávio will present the other two topics. Time permitting, we will then get down to business and discuss how we can organize the study group(s) and where to go from here.

If you’re interested in learning more about any of these topics, and happen to live in the Rio de Janeiro area, let me know. I’ll be happy to provide full information regarding the location of the meeting. Also, space is limited, so the sooner you let me know, the better. Lastly, if you are not yet a member of the IASA-RJ group, go to the Google Groups site and request permission to join, or send me your email and I’ll be happy to oblige.

Hope to see you soon!

Do we need Yet Another Method to communicate?

March 10, 2009

Lately I’ve been looking into Web 2.0-type tools to improve our commumication and workflow at work. There’s been some Enterprise 2.0 buzz about Twitter as a possible tool for communication on projects, and I’ve been scratching my head trying to figure out exactly how. One problem with Twitter is that it is wide out in the open, for anyone to read. If loose lips sink ships, Twitter is the Bermuda Triangle of company secrets. It hardly seems wise to be encouraging your employees to discuss the intacies of internal projects in such an open forum.

I recently came across Yammer, which is an enterprise-friendly version of Twitter. Basically, it’s the same thing, but your comments are only visible to members of your “company” (currently defined by the domain name of your email address). There are other features like private groups, but you get the idea.

Now, there’s been a lot of talk recently about the importance of quiet time for productivity (c.f. The Productive Programmer). To many “yammers” (or “yams”…?) might appear to be nothing more than an acronym for “Yet Another Method to Molest Everyone Relentlessly”. If there’s nothing more to this Yammer thing than posting your whims to the world, I wholeheartedly agree. We already use email for correspondence, IM for more immediate communication, Skype for voice and text conferences, web forums for context-related discussions, wikis, and on and on. So who needs yet another open window on our desktop, yet another way to have your train of thought interrupted, one more place you have to look to find a comment or note?

First, let’s take a look at the characteristics of the Yammer style of communication:

  1. It’s written (online)
  2. Very brief (about the size of an IM message)
  3. Can include attachments, but not common
  4. Visible to all in the enterprise (unless posted to private group)
  5. Messages can be tagged for organization/filtering by keywords
  6. Only your “followers” (or followers of a tag used) are “notified” – except for messages targeted at an individual, the author doesn’t choose the audience
  7. Mode of notification is configurable (from in-your-face to I’ll look it up when I remember)
  8. Messages are searchable by all

Twitter is billed as a microblog, where users post random thoughts and actions of the day (the form for posts asks you “What are you doing?”). The main benefit touted by users is that out of this chaos, you end up learning things about your friends you never knew. Yammer seems to be aiming for the same thing in a business setting (they ask “What are you working on?” in the desktop app. Online, they are more open-ended: “Share something with My Colleagues”). Unlike IMs and email, Twitter doesn’t seem to be for everyone; only those with a yen for connectivity really seem to grok it. In a business setting, particularly in a small company such as mine, it seems unlikely that we could achieve the critical mass necessary to make this work.

But there are some problems with communication out there in the corporate world that I have been looking to solve. The biggest issue I have is the need for a bottom-up “news from the trenches”-style way for developers to let me know what they do and I don’t. The types of things that may get mentioned at the water cooler and then forgotten; the fleeting idea that even the developer soon forgets; the magic shortcut command that only one guy in the corner is using; the little detail in the fine print of the new product we’re buying that means it’s not going to work with our software.

One would think that forums would work for this type of communication, but they don’t. Perhaps the problem is that there’s a certain sense of responsibility when you post to one: what if no one responds to my idea? What if people think it’s crap? A lot of people with really good ideas would rather remain anonymous than stick their necks out for what’s no more than a fleeting thought. But I think another part of the problem is just the sheer overhead involved in using forums: first you have to open the page, look to see if anyone’s posted something similar, create a new topic, write up some sort of backgroud story to fill up the big textarea… And who goes around browsing the forums for new posts, anyway?

Here’s where I think Twitter/Yammer-style communication may come in handy. It takes all of 10 seconds to post a single sentence, you can tag it so that it gets found by the people who care (#yammer #productivity hey! That chain icon on the plugin pastes a link to the current page), and there’s no expectation whatsoever that people will respond. The messages are searchable later, meaning that someone that wasn’t previously watching the tag could go and grab all the #productivity tips if they want to.

The other area I’m looking at is with project coordination and communication. An interesting thing that has evolved here at work is the use of Skype chats created specifically for members of a project team to coordinate their actions, ask questions, and so on. The one complaint I’ve heard about this approach is that people on the “outside” (not added to the chat) have no way of knowing what’s going on in the chat, and no way of looking at the history when they finally are. Yammer could be a solution to that problem, although it’s a terrible medium for active discussions.

So will it work? Will people use it freely, without putting a gun to their heads? Will it increase the sharing of knowledge at work and capture those important but elusive thoughts, or will it be just another icon on the desktop and another distraction from real work? I’m just starting to test it out here where I work, so I don’t have a conclusive answer yet, but I can make some initial observations.

First of all, It’s not a no-brainer. The best ideas catch on without any explanation, but with Yammer, people don’t seem to get it right away (me included). Twitter is the same way, so this doesn’t necessarily spell doom for this little experiment. Still, I sent out a general announcement to about 20 people, and got exactly 1 person who signed up (and since then hasn’t posted a single message). I next took the approach of asking people one at a time to join, explaining the basic concept to them in the process. While they seemed interested in the idea, I find I’m still pretty much the only one posting. Now I’m trying a 2-pronged approach of getting all the members of a small project to try it out for project-related communication, while encouraging developers to post some very specific ideas (around #productivity tips) when they have them.

I find that a few good examples help, and as I myself gain experience with it, I’m getting a feel for when to use it. Just today, I was in a meeting for one project, and had a quick thought about an unrelated subject. I grabbed my iPod and quickly “yammered” the idea out to the related tag. Too bad no one out there is listening…

So, perhaps this is an idea that will work, once people get used to it. Or perhaps Yammer’s value is no more that of Twitter: good for connecting the Web 2.0 addicts of your business in chaotic an unpredictable ways. Which is great for a fortune 500 enterprise, but sure feels lonely for us small guys.

97 Things Every Software Architect Should Know – Released!

March 6, 2009

I guess it’s my turn to write about this. I’ve mentioned before that I participated in an online community project called “97 Things Every Software Architect Should Know”. It’s an ongoing (although currently quiet) collaboration to share the wisdom and experiences of software architects, a group that has only recently been gaining recognition as anything more than a job title.

Well, I am proud to say that I contributed 4 of those 97 things, or “axioms” as they’re called, and now am happy to announce that the book was finally released! You can buy a copy yourself via O’Reilly,, or elsewhere. Or, if you prefer burning fossil fuels over killing trees, you can read the axioms for free (minus electric and internet fees) on the official web site. You’ll also find other nuggets of wisdom in the section entitled “Other Things Software Architects Should Know”, and can even contribute your own expertise via the “Community Axioms” page (I have two more out there, including one that I think I expressed better in a recent blog post).

As a side note, I’d just like to say that there was no pretense with this book to define that these principles are THE 97 things you need to know as a software architect, nor that there are only 97 (the additional contributions on the web site are a testament to that). In fact, the reason for the number 97 is a bit bizarre – I leave it as homework for you to find the explanation hidden in the forums of the 97 Things web site.  If this book is successful, O’Reilly may decide to print a second addition with more axioms, maybe even written by you! They are definitely going ahead with a whole new series of books in the “97 Things” style, on such subjects as programming, data architecture, and others. Keep an eye out for more opportunities to contribute to the community knowledge.

Lastly, if you really like the project or the book, and want to network with other contributors, I just created a group on for this purpose – feel free to join!

Causes of Decay: Mutating Design

February 23, 2009

AKA “Partial Refactor”

AKA “Good Ideas”

I have discussed in the past a phenomenon I call “Architecture by Accident”, in which the clarity of the design of a system may be ruined by rampant inconsistencies caused by a lack of attention for standards and reuse as the system evolves. But you don’t have to rely on chance to get there – we can achieve the same results absolutely intentionally.

Let’s say you have a system with a catalog of products, and that each of these products has a listing of parts. It’s probably a common pattern in the system to do something with the product itself, then go through the parts one by one and do a related activity. For example, the web page for the product probably lists the product’s name, code, and a description, and then shows each of the parts one by one in a similar fashion. The printed-out invoice may do the same. And let’s say the order fulfillment workflow does all sorts of funky calculations based on summing up the individual parts for things like calculating shipping weight, checking inventories, provisioning, whatever.

So the system designer goes ahead and says, “Hey everybody! Let’s create an iterator for products and their parts. From now on, whenever you need to do something to products, use a loop with the iterator.” Great. So, the team goes ahead and implements the web page and the invoice sheet using the really fancy iterator, with just a slight change to the contents of the “while” statement. So far, so good.

After a while, this “slight change to the contents” starts giving off a distinct copy-paste smell to the designer. So, one bright day, while browsing through their dog-eared copy of the GoF, they come across the Visitor pattern. “Aha! THIS is what we need!” exclaims our designer. The team has just been asked to implement that product-is-the-sum-of-its-parts weight algorithm I mentioned, and the designer decides it’s a good time to try out the pattern. What do you know?! It’s a fantastic improvement to the way they do things. “From now on, team, we use the Visitor pattern!” And it was so.

Time passes, and after a lot of summing up product parts in all sorts of incredibly meaningful ways, the designer starts to realize that their code base is lousy with one-hit-wonder Visitor classes that are created for some special purpose and are never used again. Fortunately, they are reading a book on the wonders of closures in Groovy. “Aha! THIS is what we need! We can just pass the code to be executed, without having to create a whole new class every time!” The team is all for it (all except one member, who’s forced to quit due to some unfortunate flashbacks to the 60’s inspired by the new language – especially tragic to happen to a young man of only 25), and goes about messing around with their products in Groovy.

Eventually, the team is able to hire a Groovy-compatible replacement for their fallen comrade. On the newbie’s first day on the job, she turns to one of her new coworkers and says, “Hey! I thought you said there was a full-time architect on this system.” Confused, he responds, “There is! Why?” “Well, then, why is this system such a mess? You said I’m supposed to be coding this product stuff in Groovy, but there’s a ton of these Visitor classes, brute-force loops, and all this other copy-pasted code. What up?”

From an outsider’s perspective, there’s little difference between “Architecture by Accident” (a lack of standards) and “Mutating Design” (too many standards). The result is pretty much the same: a patchwork quilt of approaches to solving the same problem in myriad ways. An architect or designer (or team) should strive for clarity in their designs. A system should speak for itself, but not if it’s going to say something different every time it opens its mouth.

So how does one avoid creating a system with a Mutating Design? There are only a few things you can do:

  1. Never change your design. Once you make a decision, write it in stone. This way, it will be easy for everyone to know how things are meant to be done. If anyone strays from the beaten path, it should be easy to identify and put things back on track. Unfortunately, this puts quite a burden on you to get things right from the beginning. This is basically synonymous with “waterfall methodology”, and has about the same chances of succeeding. However, it is worth noting that there may be times where the gain to be had by improving a design is outweighed by the damage the change would do to the clarity of the system.
  2. Refactor everything. The devil in a Mutating Design lies in inconsistency. You can exorcise it by going through a rigorous ritual of refactoring everything that had previously been implemented so that the whole system reflects the new design. This could mean a whole lot of work (and risk of introducing new bugs into previously working code) in the name of clarity.
  3. Isolate the changes. Again, the problem is with clarity, which can be occluded by inconsistency. So is there a way to provide clarity even when the design is incosistent? There is… if you’re clear about scope, and you provide a roadmap.

This last point is not obvious, but worth trying to understand and put into practice. The question you should ask yourself is: if the design keeps changing, how can developers know which pattern to use, and where? Ideally, the system should “speak for itself”, which means developers should be able to infer the design from existing implementations. Therefore, if you wish to change the design, do it in a way that can be consistent within the scope in which developers tend to work. If development teams are divided up by ownership of subsystems, for example, you can experiment with a new design in one of the subsystems – but then change the design for that whole subsystem. It may be inconsistent across the whole system, but in general, developers won’t feel the pain. Even if developers work on the whole system, it may be possible to choose a scope that makes sense to them. If the system is divided by modules, you can choose to change the design for one (entire) module. But then you must make it clear to developers that they should use whichever pattern is appropriate for the particular module they are working on.

This last approach can go really wrong if you don’t provide clear signals to developers as to where they are in the design. Because of this, I am working on a series of techniques (and blog posts) that I call “Visible Architecture”. The idea here is that the development team should be able to see the architecture relative to their code at any time. So, for example, if they are working on a module in which the Visitor pattern must be implemented to work with products, a document on this technique should “present itself” to the developers from within their IDE. If they then switch to a module using the new Groovy approach, the document will switch as well.

There aren’t very many tools that provide this type of functionality. I’m working with one called Structure101 which lets you do just that for layer diagrams. You can define dependency rules for a project, and they will actually show up as diagrams (with enforcement via compilation errors) in either an Eclipse or an IntelliJ IDE. You can publish a different set of diagrams for each Eclipse or IntelliJ project, which means if you wish to change these rules, it’s easy to do it for one project, and leave the old rules in effect everywhere else. I have also written a plug-in of my own for these two browsers called “Doclinks” which doesn’t enforce any rule, but allows you to link URLs to source code based on a wide variety of rules. This, together with a wiki-based architectural documentation, is another way to provide a context-specific roadmap to developers, reducing the confusion that can be caused by a Mutating Design.

I’ve previously shown you how a system can lose its clarity due to a lack of architecture. Now I’ve presented how the same thing can happen when it has too much architecture. As an architect or designer, you need to recognize the importance of standardization, but you also shouldn’t freeze your design in time. What’s important is to recognize that the evolution of the system is best done in stages, rather than through kaleidoscoping changes with no regard to what came before. Before you know it, your code may look like it’s from a B-Movie: The Attack of the Mutating Design!