Project Triangle

Achieving efficiency in the software development process is one of the key motivators every team should strive for. Efficiency can be measured in a variety of ways. The most obvious measurements are cost, project timeline, and the feature set that can be implemented given the first two. In a sense, it boils down to the old project triangle (remember: pick any two of the criteria).

In essence, there is a trade-off between quality, timeline, and cost. For example, reducing the timeline at equal costs reduces quality just as implementing at a faster pace reduces quality. Yet I argue that the triangle approach is not necessarily valid anymore. Traditional development processes have clearly shown that just enhancing the timeline on a project to put special care into the design does not actually lead to higher quality software – quite the contrary.

Yet more dimensions are at play. The number of defects (“bugs”) found in a particular software directly translate into cost and time, especially when found late in development cycle, creating a dependency between testing quality, time, and cost. Inefficient software design increases the cost of introducing new functionality as requirements change and a lack of refactoring capabilities sooner or later lead to the need for a full re-development. The problems are amplified when the software spans multiple independent subsystems, which is often the case in modern web architectures which span across content management systems, web services, search engines, commerce engines, custom web applications, etc.

Agile development methodologies have tackled many of these problems in great detail through test-driven development (TDD) and time-boxed iterative release cycles. This article discusses a number of tactics you can deploy in addition to what you find in your agile toolkit: To speed up development and tackle complex problems with smaller teams in less time leveraging the key ideas of Model-Driven-Development (MDD).

Leaning on MDD

Model-Driven-Development (MDD) is a rather interesting software development paradigm which puts the modeling aspect of software engineering at the center of the development process.

MDD Overview

The most popular notion of MDD is the Model-Driven-Architecture standard by the Object Management Group (OMG). MDA is based on a variety of modeling artifacts. At top is a platform-independent model (PIM) which only captures the business requirements using appropriate domain-specific language. This model is then translated into any number of platform-specific models (PSM) using a platform definition model (PDM) for each platform. In essence, this is equivalent to modeling your software in a very high-level (business specific way) and then using a translator such as a code generator (the PDM) to convert the model into code (the platform specific model). Given the same business model, the software can automatically be built using C#, Java, and PHP given the correct translation routines.

MDA in theory has a number of advantages to traditional coding:

• It obviously appeals to the business owner who can finally re-use the conceptual business model across technology trends, i.e. re-implementing the solution using new technologies does not require a complete overhaul but is simply a matter of switching technologies. Numerous companies specialize in MDA and even rapid-prototyping tools exist which integrate agile development methodologies with MDA. Instead of developing software in iterations, the model is developed iteratively and can then be generated into executable code.
• When using code generation frameworks, such as the open source tool AndroMDA, one can quickly build applications using existing code generators. A simple UML domain diagram can immediately be translated into Spring MVC controllers, domain objects, Hibernate mappings, and much more.
• When the software spans multiple sub-systems, MDA nicely enforces the correct translation of the model across different technologies used in each one of these systems. While I prefer writing generic code to duplicating code via code generation, this isn’t always feasible (e.g. for XML configuration files or TeamSite CMS data capture templates). In MDD changes to the model can instantly be translated into multiple code artifacts using different technologies by the push of a button.

Yet I also see a number of serious issues with the OMG’s vision:

• As an “agilist” at heart I strongly oppose the idea of spending excessive time modeling software in great detail such as highly granular UML diagrams. Software is meant to be code, not a myriad of UML diagrams which are modeled without an in-depth understanding of the features and limitations of the underlying frameworks. I value the use of UML as a pictorial language, especially when illustrating concepts either on a white board or in documentation. But not when used in a strong forward-engineering paradigm.
• MDA reduces the application of a particular technology or framework to a simply technicality, i.e. the creation of a platform definition model. Yet building applications efficiently heavily relies on the capabilities and limitation of the underlying frameworks.
• Code generation is equivalent to duplicating a code template using the model as an input. However, I generally prefer writing generic, re-usable code to unnecessary duplication. The benefits of generic code are obvious, not only is the application smaller, but debugging and maintaining the code is by far easier. Generated code would force you to debug the same piece of logic in many places of the application and fixing it requires changing the code generator templates and ultimately re-generating the entire application.
• Building a platform dependent model, i.e. the code generator, for an entire application can be a huge undertaking. On the upside, many vendors and open source technologies, such as AndroMDA, ship with a variety of pre-built cartridges. However, by using existing code-generators one reduces implementation flexibility as well as maintainability of the application. Debugging and fixing issues in these pre-built code generators can be tedious and easily be overwritten by the next release of the generator. Further, generic code generators tend to be quite complex due to the fact that they have to be very generic.
• When building web applications, I usually like to encourage my teams to push the boundaries and leverage the latest technologies available. Using existing code generation frameworks obviously won’t leverage the bleeding edge of technology, forcing you to write your own.

Leveraging the Key Tenets of MDD

While I argue that in its pure form, that being the notion of building an entire application using this paradigm, MDD is not my first choice, I would also argue that it has an obvious allure to it. Writing generic code is not always an option as all modern frameworks require configuration, plumbing code, mapping directives, etc. This is exactly where the code generation aspect comes to fruition. Given a central domain model, many artifacts surrounding a domain object can be automatically generated.

A major objection I am confronted with often is whether this approach lacks flexibility as the code is generated according to the same pattern every time. My argument is that this is actually an advantage for the majority of every application. Of course the code generation framework needs to be able to handle special situations where the generic functionality needs to be extended.

Let’s consider this by an example. A software team is integrating an XML-based content management system with a web application. The CMS team is responsible for defining the content input forms in the CMS which are used by the end user to create the XML. The application team writes a parsing layer which parses the XML into domain objects and a web application on top of it. After the teams agree on a content model, i.e. the structure of the XML files, both teams can start implementing all necessary coding artifacts.

Sample Application

However, since all artifacts are developed manually, when integrating the pieces, the teams will encounter a number of bugs which are a result of the two separate systems relying on the same underlying domain model. Further, different bugs are most likely to be found each one of the content forms and the associated parsing layer because different developers make different mistakes.

Consistency may be another issue. Especially when multiple developers are working on the individual functionality, each usually adds their own spin to the code. Some of date fields in the CMS forms may have calendar buttons next to date fields, some may not. Some developers might use camel case, another one may not.

Of course both of these issues can be addressed by establishing sound coding conventions as well as doing impeccable up-front design of all the sub-systems. But reality shows this is rarely the case. Especially when reacting to changes during the development cycle, such as the web application noticing that they need additional fields in the CMS, the original design efforts are often neglected.

MDD with Code Generator

Consider the alternative which is more aligned with the MDD paradigm. The teams agree on a domain model and then build a vertical slice, i.e. a functional prototype of the system through all defined layers. Then, using this prototype, the teams build a code generator which takes the domain model as an input and automatically generates the CMS forms and application-level parsing layer for the resulting XML files. The domain model is then fed into the code generator and the application is automatically generated. The code generator automatically enforces consistency. If any bugs where to be encountered which are a result of the coupling of the two systems, the code generator would have to be changed and the application re-generated.

In addition to the maintenance and consistency advantages, the teams also saved time. In the traditional approach, each time had to manually build the application logic for each entity in the domain model in each of the participating subsystems. In the MDD scenario, the teams built a vertical slice prototype and then translated that into a code generator – which automatically generated the application.

Alternatives

Especially when building web applications a strong alternative to MDD is the use of Rails-like frameworks, such as Ruby on Rails, Grails, Monorail, and many others. The underlying core ideas are aligned with what I consider the key advantages of MDD:

• Don’t repeat yourself (DRY): Instead of repeating yourself, write code once and then have the framework take care of creating the plumbing code. This is typically done under the hood by the framework.
• Conventions over Configuration (CoC): Rather than having every aspect of the application be built differently, establish sound conventions and use these throughout the software to ensure consistency and eliminate unnecessary (and unmaintainable) bloated configuration files.

In essence, these frameworks try to solve the same underlying problem. Yet Rails frameworks focus on building web applications within the same technology stack. For simple (web) projects which are easily contained into one logical application, not spreading across multiple software systems, any Rails framework is an excellent way of building an application quickly and iteratively. Once an architecture spans multiple technologies, frameworks, or requires custom coding using proprietary products (such as a CMS), MDD proves to be the big brother of Rails-like frameworks.

Conclusion

I have used both the Rails and the MDD approach throughout my career. I have introduced the light-weight MDD approach in a number of recent projects at Razorfish, which led to us building a jumpstart kit that in its first iteration lets us quickly bootstrap projects with Interwoven TeamSite and .NET as an application platform. This has not only saved us a lot of time, but also a lot of headaches and long nights of debugging code. We are able to quickly react to changes during the development cycle. Changes to the domain model can be made within the matter of the short time it takes to open a UML editor and re-run the code generator.

I consequently see MDD as a vital part of agile enterprise development and a complementing technology which picks up where a Rails framework hits its limits.

Links

• Object Management Group’s MDA Page
• Wikipedia on Model-Driven-Architecture
• AndroMDA code generator tool
  
• Eclipse Modeling Framework (EMF)

I started thinking there’s got to be a way to automate this. Indeed, there are a number of downloadable utilities out there to create JPEG images of web sites, but I really wanted to build this myself. After some initial online research I decided to give it a spin in C#. I came across the System.Windows.Forms.WebBrowser class which, as it turns out, can do all these magic things for me directly on the server-side without requiring a desktop application.

The caveat is that the WebBrowser does need to execute in a standalone STA thread. As a side-note, the code is a total memory-hog the way it is written now, mostly due to IE.

Check out the full code right here:

using System;
using System.ComponentModel;
using System.Drawing;
using System.Drawing.Imaging;
using System.Threading;
using System.Windows.Forms;

namespace JToEE
{
    public class Shoot
    {
        /// <summary>
        /// Set URL for the screenshot
        /// </summary>
        public string Url { get; set; }

        /// <summary>
        /// Output filename
        /// </summary>
        public string Filename { get; set; }

        /// <summary>
        /// Width of the browser window
        /// </summary>
        private int? Width { set; get; }

        /// <summary>
        /// Initial height of the browser (get re-sized 
        /// automatically if the page is longer)
        /// </summary>
        private const int DefaultHeight = 768;

        /// <summary>
        /// Default width of the browser if not specified. This
        /// class doesn't re-size the width.
        /// </summary>
        private const int DefaultWidth = 1024;

        /// <summary>
        /// 
        /// </summary>
        /// <param name="url"></param>
        /// <param name="filename"></param>
        public Shoot(string url, string filename) : this()
        {
            Url = url;
            Filename = filename;
        }

        /// <summary>
        /// 
        /// </summary>        
        public Shoot()
        {
            Width = 1024;
        }

        /// <summary>
        /// Main entry method which created the screenshot
        /// </summary>
        public void CreateScreenshot()

        {
            ThreadStart ts = Run;
            Thread t = new Thread(ts);
            t.SetApartmentState(ApartmentState.STA);
            t.Start();
            t.Join();
        }

        /// <summary>
        /// Does the work in the thread
        /// </summary>
        private void Run()
        {
            using(Bitmap bitmap = CreateBitmap())
            {
                ImageCodecInfo jgpEncoder = GetEncoder(ImageFormat.Jpeg);
                var myEncoder = System.Drawing.Imaging.Encoder.Quality;
                var myEncoderParameters = new EncoderParameters(1);
                // encode jpeg at 75%
                var myEncoderParameter = new EncoderParameter(myEncoder, 75L);
                myEncoderParameters.Param[0] = myEncoderParameter;
                // save to file
                bitmap.Save(Filename, jgpEncoder, myEncoderParameters);
            }
        }

        /// <summary>
        /// Get a hold of a jpeg encoder
        /// </summary>
        /// <param name="format"></param>
        /// <returns></returns>
        private static ImageCodecInfo GetEncoder(ImageFormat format)
        {
            ImageCodecInfo[] codecs = ImageCodecInfo.GetImageDecoders();
            foreach (ImageCodecInfo codec in codecs)
            {
                if (codec.FormatID == format.Guid)
                {
                    return codec;
                }
            }
            return null;
        }

        /// <summary>
        /// Create a bitmap of the browser window. The bitmap's height is
        /// automatically adapted to the length of the page while the width
        /// is kept at the specified value.
        /// </summary>
        /// <returns></returns>
        private Bitmap CreateBitmap()
        {
            using (var browser = new WebBrowser())
            {
                browser.ScrollBarsEnabled = false;
                // set the initial height and width
                browser.Size = new Size(Width ?? 1024, DefaultHeight);
                browser.ScriptErrorsSuppressed = true;
                // supress creation of new windows
                browser.NewWindow += OnNewWindow;
                // navigate the browser to the URL
                browser.Navigate(Url);
                // wait for it to load
                WaitForBrowserReady(browser);
                // re-size the browser windows to the full height of the web page
                var finalHeight = browser.Document == null ? DefaultHeight :
                                      browser.Document.Body.ScrollRectangle.Height;
                // width of the bitmap
                var width = Width ?? DefaultWidth;
                // re-size the browser
                browser.Size = new Size(width, finalHeight);
                WaitForBrowserReady(browser);
                // create a bitmap                
                Console.WriteLine(string.Format("Creating bitmap of size {0}x{1}", width, finalHeight));
                var bitmap = new Bitmap(width, finalHeight);
                var rect = new Rectangle(0, 0, width, finalHeight);
                browser.DrawToBitmap(bitmap, rect);
                return bitmap;
            }
        }

        /// <summary>
        /// Waits for the browser to be ready loading the page
        /// </summary>
        /// <param name="browser"></param>
        private static void WaitForBrowserReady(WebBrowser browser)
        {
            Console.WriteLine("Waiting for browser..");
            while (browser.ReadyState != WebBrowserReadyState.Complete)
            {
                Application.DoEvents();
            }
            Console.WriteLine("Waiting ready");
        }

        /// <summary>
        /// New window delegate -- basically a popup blocker.
        /// </summary>
        /// <param name="sender"></param>
        /// <param name="e"></param>
        private static void OnNewWindow(object sender, CancelEventArgs e)
        {
            e.Cancel = true;
        }
    }
}

We’ve decided on a beautiful 1923 colonial-style house, and as any old house it needs a little bit of work. The most interesting part is the heating system. The house has nice old steam radiators which are powered by one of the scariest pieces of equipment I have seen in a basement to day. An old ca. 1930s furnace which seems to be and old coal furnace which was then converted to oil. The icing on the cake (yeah, really, the white stuff on the furnace) is lovely asbestos. At first, I was a wee bit freaked out about it, but I quickly learned that this is pretty normal in these old homes. Obviously the removal of the monster and the asbestos will be the first thing before we move in, but I wanted to share some pictures of this antique.

Also notice the quite beautiful old yellow-brownish gauge on the monster. I almost wish it had more switches, levers, and knobs on it. I actually got a good laugh out of the “SUPER FUEL CORP” sticker.

As a (non-fundamental) environmentalist I must really wonder what emissions this thing creates (aside from the atrocious efficiency). It’s got to be the equivalent of thousands of cars per year.

Youtube hasn’t been successfully turning the site into money yet, maybe this is a sad attempt at that. Frankly, I think it completely waters down their brand (if they really have one, that is). Who really associates premium content with the site that has an endless supply of kitten videos?

Then again, McGyver is hardly “premium” content. I wish they’d also add Airwolf and Knight Rider. You can already watch The Fall Guy on Hulu.

The road to… digital

I think in the long run we’ll see a lot of changes in the industry. Sites such as Hulu, TheWB.com, and others to follow will sooner or later revolutionize the way we watch TV. Cable operators are still trying to ignore this by doing things such as instantiating download limits. Yet it’s just a matter of time before we all have the Internet built directly into our TV and/or set top boxes and can watch our favorite House episode any time we’d like directly from the Internet and on the TV. This will most likely cannibalize traditional TV offerings, yet this trend offers huge potential even for the cable operators if they were to embrace it — and they’re slowly realizing it, otherwise Comcast would have never launched Fancast. Maybe that’s the real reason for the download limits — to skew competition and boot the cable provider’s own video sites?

There are still a lot of hurdles to be overcome. For example, Hulu is struggling with the cost of bandwidth resulting from the massive video streaming which seems in no proportion to the incoming advertising revenue. Also expect ad revenues to decline in the near term — we’re in a recession after all.

Today a co-worker forwarded an interesting article on AdvertisingAge about Viacom and HBO jumping on the bandwagon also. Then again, I already knew that, but I’d have to kill you if I told you why :-)

The Law&Order Effect

While I love the ability to watch a lot of things directly online, there’s one big thing about my TV that I miss online. TV is passive entertainment, you turn it on and that’s all you have to do. No effort involved, unless you feel like channel surfing. In the online world the myriad of options make discovering something to watch work — the editorial responsibilities are shifted from the channel to the end user. Sometimes, I really just want to watch Law&Order or be surprised by the evil-gargoyle-eats-soldiers movie of the day (SciFi channel, of course).

Maybe that’s an idea for another online video startup? Venture Capital, someone, please?

CDOs explained

CDOs, or collateralized-debt-obligations, are one of the big the financial weapons of mass destruction causing the mess:

Crisis explainer: Uncorking CDOs from Marketplace on Vimeo.

Where will it all go?

I wonder if in the long run the US will realize that the American model has simply failed and revert to a more moderate European capitalism. One that includes government oversight and social responsibility as a core pillar of the economy - rather than putting the focus on profit alone. Europe might not have quite the perfect mix either, but are years ahead in that regard. Unfortunately even the Europeans bought into Wall Street’s craziness and have the price to pay for it now. Globalization is one of the big reasons why many countries will have to work together to solve this underlying problem — the wrongdoings of one can just affect others too much. Think about it, the country of Iceland is basically bankrupt because of this. Unfortunately the Bush administration doesn’t exactly have a good track record of working with other nations (remember “either you’re with us or against us”?). Yet in many ways our system of countries and government seems too outdated to effectively combat the world of international finance.

On a side note, the national debt clock right around the corner from my office on 6th Ave in Manhattan ran out of digits today. There are many reasons for this, especially over the past 8 years. The focus of any adjustments that are made in the future to the financial system and economy as a whole have to focus on sustainability.

An interesting topic in the current election is the focus on universal health care. The US health care system is run by profit-oriented companies and their special interest organizations and really just hitting the fan. Many people aren’t cared for and many that are can’t afford to pay their bills. Having lived in Europe for most of my life I must say I prefer the European approach to things. The quality of live is just simply better and no-one has to worry about medical bills putting you out in the street. And I never got the impression I was cared for less over there — quite the contrary, American doctors annoy the crap out of me, all they ever want to do is make more money on tests and drugs without any real reason for it. I miss paying my 10 € a quarter even when going to a specialist. And not getting hit with about 10 bills after my $25 copay every time I go to the doctor (secretly my message here should be: eat more vegetables).

Let’s hope people learn out of this one. I’m really an optimist at heart :)

Mock Economist Cover (thanks Jason)

My predictions just became gloomier. I fully expect the Dow to go down as far as 6000-8000 points before it recovers. Interest rates will go down in the short term, I think the Fed might very well lower them at the end of this month. However, this will come back to haunt us by driving up inflation, which will in the longer term increase interest rates. I think we’ll see over 10% or more within the next few years, just like in the 80s.

I don’t think we’ll all be out of jobs or going into another major depression. Just a deep recession. Non-essential budgets will get reduced (think advertising, like the industry I’m in right now - oy).

Oh, and can they just keep Bernanke from speaking out publicly? Everytime that guy makes an announcement the Dow drops 300.

There’s nothing like getting to work early to be greeted with the total meltdown of the financial system. Lehman is bankrupt, Bank of America buys Merril, etc. What in the world is going on. Really?

Bank of America is still doing alright, that’s comforting. But it seems to me that the entire country has been in a drindling downward spiral for about a year now with no end in sight. Housing down, people defaulting on loans, not paying credit card debt, banks failing, rising inflation and unemployment. This is making the crisis of the 70s and 80s look like a joke in comparison.

I have no words beside: WTF?

Small update: While my initial bet yesterday morning was that the DOW was going to drop 1000+ points, it really “just” went down 504. But I read today that there is still potential for a total meltdown.

Honestly, what the f!@#$ where they thinking when they came up with the crap like CDOs?? “The average price of housing in the us will never drop” — sounds to me a lot like “We’ll never need more than 640k of memory”.

“Sun xVM Server is an outgrowth of the Xen project — which raises the question of why a company would go with Sun’s version rather than the Xen one. Apart from its support for SPARC and Solaris (as well as other chips and operating systems), Sun is also building a services and sales organization around a commercial version of xVM server… If you want to kick the tires or cut your costs, you can hop over to xVMServer.org, download the source (GPL 3) and join the community. But Sun is betting that, as deployments move from an initial testing phase to active usage, large organizations will be willing to pay for guaranteed support (starting at $500 per year per physical server).”

Check it out on xvmserver.org

After hitting the ok button, it tries the update again, fails. Ad infinitum.

WTF??

Do I have to use Chrome now afterall? ;)