Causes of Decay: Mutating Design

February 23, 2009

AKA “Partial Refactor”

AKA “Good Ideas”

I have discussed in the past a phenomenon I call “Architecture by Accident”, in which the clarity of the design of a system may be ruined by rampant inconsistencies caused by a lack of attention for standards and reuse as the system evolves. But you don’t have to rely on chance to get there – we can achieve the same results absolutely intentionally.

Let’s say you have a system with a catalog of products, and that each of these products has a listing of parts. It’s probably a common pattern in the system to do something with the product itself, then go through the parts one by one and do a related activity. For example, the web page for the product probably lists the product’s name, code, and a description, and then shows each of the parts one by one in a similar fashion. The printed-out invoice may do the same. And let’s say the order fulfillment workflow does all sorts of funky calculations based on summing up the individual parts for things like calculating shipping weight, checking inventories, provisioning, whatever.

So the system designer goes ahead and says, “Hey everybody! Let’s create an iterator for products and their parts. From now on, whenever you need to do something to products, use a loop with the iterator.” Great. So, the team goes ahead and implements the web page and the invoice sheet using the really fancy iterator, with just a slight change to the contents of the “while” statement. So far, so good.

After a while, this “slight change to the contents” starts giving off a distinct copy-paste smell to the designer. So, one bright day, while browsing through their dog-eared copy of the GoF, they come across the Visitor pattern. “Aha! THIS is what we need!” exclaims our designer. The team has just been asked to implement that product-is-the-sum-of-its-parts weight algorithm I mentioned, and the designer decides it’s a good time to try out the pattern. What do you know?! It’s a fantastic improvement to the way they do things. “From now on, team, we use the Visitor pattern!” And it was so.

Time passes, and after a lot of summing up product parts in all sorts of incredibly meaningful ways, the designer starts to realize that their code base is lousy with one-hit-wonder Visitor classes that are created for some special purpose and are never used again. Fortunately, they are reading a book on the wonders of closures in Groovy. “Aha! THIS is what we need! We can just pass the code to be executed, without having to create a whole new class every time!” The team is all for it (all except one member, who’s forced to quit due to some unfortunate flashbacks to the 60’s inspired by the new language – especially tragic to happen to a young man of only 25), and goes about messing around with their products in Groovy.

Eventually, the team is able to hire a Groovy-compatible replacement for their fallen comrade. On the newbie’s first day on the job, she turns to one of her new coworkers and says, “Hey! I thought you said there was a full-time architect on this system.” Confused, he responds, “There is! Why?” “Well, then, why is this system such a mess? You said I’m supposed to be coding this product stuff in Groovy, but there’s a ton of these Visitor classes, brute-force loops, and all this other copy-pasted code. What up?”

From an outsider’s perspective, there’s little difference between “Architecture by Accident” (a lack of standards) and “Mutating Design” (too many standards). The result is pretty much the same: a patchwork quilt of approaches to solving the same problem in myriad ways. An architect or designer (or team) should strive for clarity in their designs. A system should speak for itself, but not if it’s going to say something different every time it opens its mouth.

So how does one avoid creating a system with a Mutating Design? There are only a few things you can do:

  1. Never change your design. Once you make a decision, write it in stone. This way, it will be easy for everyone to know how things are meant to be done. If anyone strays from the beaten path, it should be easy to identify and put things back on track. Unfortunately, this puts quite a burden on you to get things right from the beginning. This is basically synonymous with “waterfall methodology”, and has about the same chances of succeeding. However, it is worth noting that there may be times where the gain to be had by improving a design is outweighed by the damage the change would do to the clarity of the system.
  2. Refactor everything. The devil in a Mutating Design lies in inconsistency. You can exorcise it by going through a rigorous ritual of refactoring everything that had previously been implemented so that the whole system reflects the new design. This could mean a whole lot of work (and risk of introducing new bugs into previously working code) in the name of clarity.
  3. Isolate the changes. Again, the problem is with clarity, which can be occluded by inconsistency. So is there a way to provide clarity even when the design is incosistent? There is… if you’re clear about scope, and you provide a roadmap.

This last point is not obvious, but worth trying to understand and put into practice. The question you should ask yourself is: if the design keeps changing, how can developers know which pattern to use, and where? Ideally, the system should “speak for itself”, which means developers should be able to infer the design from existing implementations. Therefore, if you wish to change the design, do it in a way that can be consistent within the scope in which developers tend to work. If development teams are divided up by ownership of subsystems, for example, you can experiment with a new design in one of the subsystems – but then change the design for that whole subsystem. It may be inconsistent across the whole system, but in general, developers won’t feel the pain. Even if developers work on the whole system, it may be possible to choose a scope that makes sense to them. If the system is divided by modules, you can choose to change the design for one (entire) module. But then you must make it clear to developers that they should use whichever pattern is appropriate for the particular module they are working on.

This last approach can go really wrong if you don’t provide clear signals to developers as to where they are in the design. Because of this, I am working on a series of techniques (and blog posts) that I call “Visible Architecture”. The idea here is that the development team should be able to see the architecture relative to their code at any time. So, for example, if they are working on a module in which the Visitor pattern must be implemented to work with products, a document on this technique should “present itself” to the developers from within their IDE. If they then switch to a module using the new Groovy approach, the document will switch as well.

There aren’t very many tools that provide this type of functionality. I’m working with one called Structure101 which lets you do just that for layer diagrams. You can define dependency rules for a project, and they will actually show up as diagrams (with enforcement via compilation errors) in either an Eclipse or an IntelliJ IDE. You can publish a different set of diagrams for each Eclipse or IntelliJ project, which means if you wish to change these rules, it’s easy to do it for one project, and leave the old rules in effect everywhere else. I have also written a plug-in of my own for these two browsers called “Doclinks” which doesn’t enforce any rule, but allows you to link URLs to source code based on a wide variety of rules. This, together with a wiki-based architectural documentation, is another way to provide a context-specific roadmap to developers, reducing the confusion that can be caused by a Mutating Design.

I’ve previously shown you how a system can lose its clarity due to a lack of architecture. Now I’ve presented how the same thing can happen when it has too much architecture. As an architect or designer, you need to recognize the importance of standardization, but you also shouldn’t freeze your design in time. What’s important is to recognize that the evolution of the system is best done in stages, rather than through kaleidoscoping changes with no regard to what came before. Before you know it, your code may look like it’s from a B-Movie: The Attack of the Mutating Design!

Causes of Decay: Architecture by Accident

September 4, 2008

AKA “Copy-paste Architecture”

I recently read a great answer to the question: “What is architecture?” Or rather, “What elements are architectural?” It’s an important question, and one seemingly impossible to answer: where do you draw the line between architecture and design? Architecture and just good practices? Architecture and “that’s just the way we did it”?

The answer is both obvious and disappointing: architecture is in the eye of the beholder. To an Enterprise Architect, the broad business concepts (domain entities) and system capabilities are architectural. To an Applications Architect, modules and layers, component interfaces and connection protocols are all architectural. To a Network Architect, network topologies and hardware are architectural. There is some overlap in what each cares about, but the point is that each type of architect, each software design and each developer has their own opinion about what is important to know.

But don’t despair! There is a way of defining exactly what is architectural to YOU! You see, what an architect does in the end is specify enough of the elements, patterns, technologies and structure of the system (or systems, or subsystem…) to guarantee that the implementation work that follows will satisfy the functional and non-functional requirements that it is supposed to. The rest is just design details that you can let a designer or developer decide for themselves – what’s important is that the performance, scalability, extensibility and all the other -ilities have been taken care of.

And that’s a healthy thing: to make all decisions up front for everyone else is not only boring, it would take forever, and take up volumes of hard-to-use and impossible-to-maintain documentation. What’s more, you run a big risk of overdesign.

Another definition, or characteristic, I have heard of architectural elements is that they are not generally one-off decisions regarding a specific implementation. What separates an architectural element from a design element is that the former represents a pattern that is meant to instruct and guide the design and implementation. Architecture in this sense is “reusable design”.

So what does all of this have to do with the decay of an application’s code base? Well, let’s say you’ve gone ahead and done your duty to specify those elements of your application that are architectural. Now it’s time for the team to write the software according to their whims, but within the constraints you have imposed. During this time, developers will naturally make decisions about the implementation. “Where should I put this class?” “Which package should I use?” “What should I call the class?” “Should I use a design pattern here? And which?”

Of course, none of these decisions are (you hope) important to the overall architecture of the system. But many of them are to solve common problems, and will likely need to be made again by other developers as they create and maintain the application. So, let’s say we have a simple “8-ball decision” (cf the commandment “Be decisive”) that a developer is making regarding the name of their Data Access Object for saving “Foo” to the database. It has an interface, but let’s say this particular implementation uses Hibernate to get the job done. Our hero could name the class:

  1. FooHibernateDAO
  2. FooDAOHibernate
  3. HibernateFooDAO
  4. HibernateFooDao
  5. FooHibernateDataAccessObject
  6. and so on…

So, our hero decides he likes option #4 because it sounds like an answer to the question “WHICH FooDao?”, and because all-caps acronyms run the risk of run-on acronymtences (Note: it’s always good to keep a justification for a Magic 8-ball Decision in your back pocket, because it ends arguments quicker than, “The 8-ball said its sources say yes.”). The next time our hero makes a DAO to save Bars, you can bet he’ll call it a “HibernateBarDao”.

In the following iteration (your process has iterations, right?), our heroine finds that she, too must write a DAO to save (you guessed it) Bazzes. Fortunately, she noticed that our hero has already done this before, so she copies over the basic code, kindly extracts any commonalities to a base class (you go, girl!), and calls her new implementation a “JdbcBazDao” (say it 10 times fast!). Although this was never specified in the original architecture, you now have the beginnings of a standard AND a framework:

  • All Data Access Objects should follow the naming convention: Implementation approach + Object type persisted + “Dao”
  • Common data management code is implemented in a BaseDao class, which all other DAOs should extend

Remember the previous definition that the architecture is “reusable design”? This is more or less what is happening here, even though it wasn’t intentionally specified by the architect. I call this “Architecture by Accident.” I also call this “Copy-paste Architecture” due to the way these implicit conventions often occur. But don’t get me wrong: there’s absolutely nothing wrong with the example I just gave you. It’s fantastic when standards and architecture and evolve properly on their own.

But let’s say a third developer, our sidekick, needs to write a DAO to save XPTOs. Because these “standards” were never made official, it’s quite possible the sidekick never got the memo. And after consulting his own Magic 8-ball (the official High School Musical edition! Wow!), decides to call his DAO “XPTOJDBCDAO” (DON’T try to say that one out loud). And one more for good measure: what better for saving XYZPDQs than an “XYZPDQJDBCDAO”?

Now enters into the story our villain. Actually, he’s not such a bad guy, but he does have his own incredibly important and convincing reasons for why his DAO must be called a YoMamaDataAccessObjectHibernate.

Our simple application now sports the classes:

  • HibernateFooDao
  • HibernateBarDao
  • JdbcBazDao
  • YoMamaDataAccessObjectHibernate

Confused yet? This is the problem with a “Copy-paste Architecture”. It isn’t so much who is making the decisions, or even why. The problem is that they aren’t being made consistently. As a result, your code base starts to rot. Clarity goes out the window. It’s hard to find classes you know are there because you can’t begin to guess what they’re called. You can’t do any generalized analysis, conversions or rule enforcement because there’s no one rule that can cover all the possibilities. And we’re just talking about the class names!

Of course, Architecture by Accident isn’t limited to just naming conventions. Design patterns, layering structures, library classes and infrastructure solutions are all at risk of having their wheels reinvented if there isn’t someone looking out to identify these patterns and maintain consistency. We’re all familiar with the results: increased complexity, reduced clarity and diminished productivity.

The only solution to this problem is collaboration. The whole team needs to recognize common patterns and solutions as they evolve, make them explicit, and let everyone know. But take my word for it: it won’t happen by accident.

Causes of Decay

August 29, 2008

In order to flesh out another aspect of the presentation I gave on “Remodularization”, I’ll be writing a series of posts discussing the reasons that the quality of software “decays” over its lifetime. This will by no means attempt to be a definitive guide on the subject, since I’m not an academic, but just some reflections on my personal experience and the experience of others in the field.

To me, one of the most interesting aspects of this discussion is the taken-as-fact assumption that software naturally decays over time. This is essentially driven by the second law of thermodynamics: entropy increases over time. It seems a stretch to apply this to the evolution of a code base, but I’m sure we have all felt this to be true in one way or another.

Inevitable Decay

Inevitable Decay

Scott Bain in his book “Emergent Design” makes the argument that this doesn’t have to be the case. He argues that by using design practices that permit entension in the future while minimizing the number of places you’ll have to change the code (the Open-Closed principle), a developer is able to add functionality without adding significantly to the complexity of the application. He says that, like doctors, developers should take something akin to the Hippocratic Oath (“First, do no harm”), committing themselves to always improving the code even as they add new functionality. He also proposes some simple design principles that can make this an almost natural occurrence.

"Emergent Design", Scott Bain, with slight modifications)

Rejecting Decay

He’s not alone in this assertion. Folks from the agile camps often talk about “technical debt” and the need to be courageous to refactor your code as you work. This philosophy recognizes that there is a cost associated with maintaining the quality of the software in the face of changes, but it reflects Bain’s ideal that developers should fight the good fight constantly, rather than leave it for later (when a complete rewrite may be necessary).

Personally, I have to agree that code decay is inevitable. As Neal Ford points out in his axiom “Simplify essential complexity; diminish accidental complexity”, there is such a thing as essential complexity. Even as you work to continuously improve the overall quality of the code, you will be adding essential complexity to the application. I don’t mean to imply that Bain would disagree with this; he is referring to a measure of quality, as opposed to the overall complexity of a growing system. Perhaps there is such a thing as a “maximum quality”, when all complexity is essential.

Whether or not you and your team have chosen to be keyboard medics, the fact is that software decay happens. In the following weeks, I’ll discuss some of the reasons that software complexity increases over time. Hopefully this discussion will provide you with the medicine you need to fight this cancer.