Project Triangle

Achieving efficiency in the software development process is one of the key motivators every team should strive for. Efficiency can be measured in a variety of ways. The most obvious measurements are cost, project timeline, and the feature set that can be implemented given the first two. In a sense, it boils down to the old project triangle (remember: pick any two of the criteria).

In essence, there is a trade-off between quality, timeline, and cost. For example, reducing the timeline at equal costs reduces quality just as implementing at a faster pace reduces quality. Yet I argue that the triangle approach is not necessarily valid anymore. Traditional development processes have clearly shown that just enhancing the timeline on a project to put special care into the design does not actually lead to higher quality software – quite the contrary.

Yet more dimensions are at play. The number of defects (“bugs”) found in a particular software directly translate into cost and time, especially when found late in development cycle, creating a dependency between testing quality, time, and cost. Inefficient software design increases the cost of introducing new functionality as requirements change and a lack of refactoring capabilities sooner or later lead to the need for a full re-development. The problems are amplified when the software spans multiple independent subsystems, which is often the case in modern web architectures which span across content management systems, web services, search engines, commerce engines, custom web applications, etc.

Agile development methodologies have tackled many of these problems in great detail through test-driven development (TDD) and time-boxed iterative release cycles. This article discusses a number of tactics you can deploy in addition to what you find in your agile toolkit: To speed up development and tackle complex problems with smaller teams in less time leveraging the key ideas of Model-Driven-Development (MDD).

Leaning on MDD

Model-Driven-Development (MDD) is a rather interesting software development paradigm which puts the modeling aspect of software engineering at the center of the development process.

MDD Overview

The most popular notion of MDD is the Model-Driven-Architecture standard by the Object Management Group (OMG). MDA is based on a variety of modeling artifacts. At top is a platform-independent model (PIM) which only captures the business requirements using appropriate domain-specific language. This model is then translated into any number of platform-specific models (PSM) using a platform definition model (PDM) for each platform. In essence, this is equivalent to modeling your software in a very high-level (business specific way) and then using a translator such as a code generator (the PDM) to convert the model into code (the platform specific model). Given the same business model, the software can automatically be built using C#, Java, and PHP given the correct translation routines.

MDA in theory has a number of advantages to traditional coding:

• It obviously appeals to the business owner who can finally re-use the conceptual business model across technology trends, i.e. re-implementing the solution using new technologies does not require a complete overhaul but is simply a matter of switching technologies. Numerous companies specialize in MDA and even rapid-prototyping tools exist which integrate agile development methodologies with MDA. Instead of developing software in iterations, the model is developed iteratively and can then be generated into executable code.
• When using code generation frameworks, such as the open source tool AndroMDA, one can quickly build applications using existing code generators. A simple UML domain diagram can immediately be translated into Spring MVC controllers, domain objects, Hibernate mappings, and much more.
• When the software spans multiple sub-systems, MDA nicely enforces the correct translation of the model across different technologies used in each one of these systems. While I prefer writing generic code to duplicating code via code generation, this isn’t always feasible (e.g. for XML configuration files or TeamSite CMS data capture templates). In MDD changes to the model can instantly be translated into multiple code artifacts using different technologies by the push of a button.

Yet I also see a number of serious issues with the OMG’s vision:

• As an “agilist” at heart I strongly oppose the idea of spending excessive time modeling software in great detail such as highly granular UML diagrams. Software is meant to be code, not a myriad of UML diagrams which are modeled without an in-depth understanding of the features and limitations of the underlying frameworks. I value the use of UML as a pictorial language, especially when illustrating concepts either on a white board or in documentation. But not when used in a strong forward-engineering paradigm.
• MDA reduces the application of a particular technology or framework to a simply technicality, i.e. the creation of a platform definition model. Yet building applications efficiently heavily relies on the capabilities and limitation of the underlying frameworks.
• Code generation is equivalent to duplicating a code template using the model as an input. However, I generally prefer writing generic, re-usable code to unnecessary duplication. The benefits of generic code are obvious, not only is the application smaller, but debugging and maintaining the code is by far easier. Generated code would force you to debug the same piece of logic in many places of the application and fixing it requires changing the code generator templates and ultimately re-generating the entire application.
• Building a platform dependent model, i.e. the code generator, for an entire application can be a huge undertaking. On the upside, many vendors and open source technologies, such as AndroMDA, ship with a variety of pre-built cartridges. However, by using existing code-generators one reduces implementation flexibility as well as maintainability of the application. Debugging and fixing issues in these pre-built code generators can be tedious and easily be overwritten by the next release of the generator. Further, generic code generators tend to be quite complex due to the fact that they have to be very generic.
• When building web applications, I usually like to encourage my teams to push the boundaries and leverage the latest technologies available. Using existing code generation frameworks obviously won’t leverage the bleeding edge of technology, forcing you to write your own.

Leveraging the Key Tenets of MDD

While I argue that in its pure form, that being the notion of building an entire application using this paradigm, MDD is not my first choice, I would also argue that it has an obvious allure to it. Writing generic code is not always an option as all modern frameworks require configuration, plumbing code, mapping directives, etc. This is exactly where the code generation aspect comes to fruition. Given a central domain model, many artifacts surrounding a domain object can be automatically generated.

A major objection I am confronted with often is whether this approach lacks flexibility as the code is generated according to the same pattern every time. My argument is that this is actually an advantage for the majority of every application. Of course the code generation framework needs to be able to handle special situations where the generic functionality needs to be extended.

Let’s consider this by an example. A software team is integrating an XML-based content management system with a web application. The CMS team is responsible for defining the content input forms in the CMS which are used by the end user to create the XML. The application team writes a parsing layer which parses the XML into domain objects and a web application on top of it. After the teams agree on a content model, i.e. the structure of the XML files, both teams can start implementing all necessary coding artifacts.

Sample Application

However, since all artifacts are developed manually, when integrating the pieces, the teams will encounter a number of bugs which are a result of the two separate systems relying on the same underlying domain model. Further, different bugs are most likely to be found each one of the content forms and the associated parsing layer because different developers make different mistakes.

Consistency may be another issue. Especially when multiple developers are working on the individual functionality, each usually adds their own spin to the code. Some of date fields in the CMS forms may have calendar buttons next to date fields, some may not. Some developers might use camel case, another one may not.

Of course both of these issues can be addressed by establishing sound coding conventions as well as doing impeccable up-front design of all the sub-systems. But reality shows this is rarely the case. Especially when reacting to changes during the development cycle, such as the web application noticing that they need additional fields in the CMS, the original design efforts are often neglected.

MDD with Code Generator

Consider the alternative which is more aligned with the MDD paradigm. The teams agree on a domain model and then build a vertical slice, i.e. a functional prototype of the system through all defined layers. Then, using this prototype, the teams build a code generator which takes the domain model as an input and automatically generates the CMS forms and application-level parsing layer for the resulting XML files. The domain model is then fed into the code generator and the application is automatically generated. The code generator automatically enforces consistency. If any bugs where to be encountered which are a result of the coupling of the two systems, the code generator would have to be changed and the application re-generated.

In addition to the maintenance and consistency advantages, the teams also saved time. In the traditional approach, each time had to manually build the application logic for each entity in the domain model in each of the participating subsystems. In the MDD scenario, the teams built a vertical slice prototype and then translated that into a code generator – which automatically generated the application.

Alternatives

Especially when building web applications a strong alternative to MDD is the use of Rails-like frameworks, such as Ruby on Rails, Grails, Monorail, and many others. The underlying core ideas are aligned with what I consider the key advantages of MDD:

• Don’t repeat yourself (DRY): Instead of repeating yourself, write code once and then have the framework take care of creating the plumbing code. This is typically done under the hood by the framework.
• Conventions over Configuration (CoC): Rather than having every aspect of the application be built differently, establish sound conventions and use these throughout the software to ensure consistency and eliminate unnecessary (and unmaintainable) bloated configuration files.

In essence, these frameworks try to solve the same underlying problem. Yet Rails frameworks focus on building web applications within the same technology stack. For simple (web) projects which are easily contained into one logical application, not spreading across multiple software systems, any Rails framework is an excellent way of building an application quickly and iteratively. Once an architecture spans multiple technologies, frameworks, or requires custom coding using proprietary products (such as a CMS), MDD proves to be the big brother of Rails-like frameworks.

Conclusion

I have used both the Rails and the MDD approach throughout my career. I have introduced the light-weight MDD approach in a number of recent projects at Razorfish, which led to us building a jumpstart kit that in its first iteration lets us quickly bootstrap projects with Interwoven TeamSite and .NET as an application platform. This has not only saved us a lot of time, but also a lot of headaches and long nights of debugging code. We are able to quickly react to changes during the development cycle. Changes to the domain model can be made within the matter of the short time it takes to open a UML editor and re-run the code generator.

I consequently see MDD as a vital part of agile enterprise development and a complementing technology which picks up where a Rails framework hits its limits.

Links

• Object Management Group’s MDA Page
• Wikipedia on Model-Driven-Architecture
• AndroMDA code generator tool
  
• Eclipse Modeling Framework (EMF)

I started thinking there’s got to be a way to automate this. Indeed, there are a number of downloadable utilities out there to create JPEG images of web sites, but I really wanted to build this myself. After some initial online research I decided to give it a spin in C#. I came across the System.Windows.Forms.WebBrowser class which, as it turns out, can do all these magic things for me directly on the server-side without requiring a desktop application.

The caveat is that the WebBrowser does need to execute in a standalone STA thread. As a side-note, the code is a total memory-hog the way it is written now, mostly due to IE.

Check out a single-threaded version of the code right here (full download at bottom of page):

using System;
using System.ComponentModel;
using System.Drawing;
using System.Drawing.Imaging;
using System.Threading;
using System.Windows.Forms;

namespace JToEE
{
    public class Shoot
    {
        /// <summary>
        /// Set URL for the screenshot
        /// </summary>
        public string Url { get; set; }

        /// <summary>
        /// Output filename
        /// </summary>
        public string Filename { get; set; }

        /// <summary>
        /// Width of the browser window
        /// </summary>
        private int? Width { set; get; }

        /// <summary>
        /// Initial height of the browser (get re-sized 
        /// automatically if the page is longer)
        /// </summary>
        private const int DefaultHeight = 768;

        /// <summary>
        /// Default width of the browser if not specified. This
        /// class doesn't re-size the width.
        /// </summary>
        private const int DefaultWidth = 1024;

        /// <summary>
        /// 
        /// </summary>
        /// <param name="url"></param>
        /// <param name="filename"></param>
        public Shoot(string url, string filename) : this()
        {
            Url = url;
            Filename = filename;
        }

        /// <summary>
        /// 
        /// </summary>        
        public Shoot()
        {
            Width = 1024;
        }

        /// <summary>
        /// Main entry method which created the screenshot
        /// </summary>
        public void CreateScreenshot()

        {
            ThreadStart ts = Run;
            Thread t = new Thread(ts);
            t.SetApartmentState(ApartmentState.STA);
            t.Start();
            t.Join();
        }

        /// <summary>
        /// Does the work in the thread
        /// </summary>
        private void Run()
        {
            using(Bitmap bitmap = CreateBitmap())
            {
                ImageCodecInfo jgpEncoder = GetEncoder(ImageFormat.Jpeg);
                var myEncoder = System.Drawing.Imaging.Encoder.Quality;
                var myEncoderParameters = new EncoderParameters(1);
                // encode jpeg at 75%
                var myEncoderParameter = new EncoderParameter(myEncoder, 75L);
                myEncoderParameters.Param[0] = myEncoderParameter;
                // save to file
                bitmap.Save(Filename, jgpEncoder, myEncoderParameters);
            }
        }

        /// <summary>
        /// Get a hold of a jpeg encoder
        /// </summary>
        /// <param name="format"></param>
        /// <returns></returns>
        private static ImageCodecInfo GetEncoder(ImageFormat format)
        {
            ImageCodecInfo[] codecs = ImageCodecInfo.GetImageDecoders();
            foreach (ImageCodecInfo codec in codecs)
            {
                if (codec.FormatID == format.Guid)
                {
                    return codec;
                }
            }
            return null;
        }

        /// <summary>
        /// Create a bitmap of the browser window. The bitmap's height is
        /// automatically adapted to the length of the page while the width
        /// is kept at the specified value.
        /// </summary>
        /// <returns></returns>
        private Bitmap CreateBitmap()
        {
            using (var browser = new WebBrowser())
            {
                browser.ScrollBarsEnabled = false;
                // set the initial height and width
                browser.Size = new Size(Width ?? 1024, DefaultHeight);
                browser.ScriptErrorsSuppressed = true;
                // supress creation of new windows
                browser.NewWindow += OnNewWindow;
                // navigate the browser to the URL
                browser.Navigate(Url);
                // wait for it to load
                WaitForBrowserReady(browser);
                // re-size the browser windows to the full height of the web page
                var finalHeight = browser.Document == null ? DefaultHeight :
                                      browser.Document.Body.ScrollRectangle.Height;
                // width of the bitmap
                var width = Width ?? DefaultWidth;
                // re-size the browser
                browser.Size = new Size(width, finalHeight);
                WaitForBrowserReady(browser);
                // create a bitmap                
                Console.WriteLine(string.Format("Creating bitmap of size {0}x{1}", width, finalHeight));
                var bitmap = new Bitmap(width, finalHeight);
                var rect = new Rectangle(0, 0, width, finalHeight);
                browser.DrawToBitmap(bitmap, rect);
                return bitmap;
            }
        }

        /// <summary>
        /// Waits for the browser to be ready loading the page
        /// </summary>
        /// <param name="browser"></param>
        private static void WaitForBrowserReady(WebBrowser browser)
        {
            Console.WriteLine("Waiting for browser..");
            while (browser.ReadyState != WebBrowserReadyState.Complete)
            {
                Application.DoEvents();
            }
            Console.WriteLine("Waiting ready");
        }

        /// <summary>
        /// New window delegate -- basically a popup blocker.
        /// </summary>
        /// <param name="sender"></param>
        /// <param name="e"></param>
        private static void OnNewWindow(object sender, CancelEventArgs e)
        {
            e.Cancel = true;
        }
    }
}

Download the full source code here.

Our first idea was to start prototyping on the new Windows 2008 Server.

32 vs 64 Bit

Our initial gut instinct was to stick with what’s working, plus we where only installing on PCs with 4 Gigs of RAM. After the installation I tried to get Hyper-V (built-in virtualization) to work, however, Hyper-V is only supported on the 64 bit version of Windows. I then tried to install Virtual PC 2007 — it complained that the version of Windows I was using wasn’t compatible and that I at least needed to install IIS 6 compatibility modules.

We decided to re-install with the 64 bit version of Windows Server.

Hyper-V

As mentioned, Hyper-V requires 64 bit Windows. After installing all the optional patches and rebooting twice from Windows Update Hyper-V suddenly appeared. I proceeded to install a 64 bit Windows 2008 guest. Once it was installed and booted up I noticed it had no network access. Strange — Hyper-V was configured with a virtual adapter. As it turns out, the drivers for the virtual network adapter are part of the Windows Update patches for Server 2008 — so the guest couldn’t see the network without the updates and of course couldn’t download the updates without the network adapter — if Hyper-V wasn’t technically a CTP I’d be really pissed now. Luckily one can configure a “legacy network adapter” that a freshly installed guest can see at least long enough to fetch the updated drivers.

Hyper-V now runs along just fine with the small caveat that it randomly crashes with a nice message box “Worker process died”.

Verdict: This is not a stable development or even production environment yet.

MOSS 2007

At this point I decided to install MOSS 2007. After a download of the MOSS 2007 ISO the first installation attempt failed — Windows complained there were known compatibility issues with the software and the operating system. Now we downloaded the SP1 files for the WSS foundation and MOSS 2007. Unfortunately there is no consolidates ISO — you have copy the MOSS 2007 ISO to a temporary folder and manually extract the SP1 executables via the command line and copy the contents into the temporary folder with the ISO. The installation then fails with an error that the sample database cannot be set up.

Just trying to install WSS 3.0 SP1 by itself at least works fine.

Really?

At this point we decided to go back to Windows 2003 Server R2. Stay tuned for updates.

The nicest think IMHO about LINQ is the SQL bridge, which essentially implements a full OR-mapping framework like Hibernate in the .NET framework. Let’s dive into an example and map the following class to a database table:

[Table(Name = "c_schema")]
public class LinqSchema
{
private long id;
private string uuid;
private string name;
[Column]
private long id_creator = 1;
[Column]
private long id_owner = 1;
[Column]
private int version = 1;
[Column]
private bool is_public = true;
[Column(Name = "id", IsPrimaryKey = true)]
public long Id
{
get { return id; }
set { id = value; }
}
[Column(Name = "uuid")]
public string Uuid
{
get { return uuid; }
set { uuid = value; }
}
[Column(Name = "name")]
public string Name
{
get { return name; }
set { name = value; }
}
}

So far so good, that looks almost like an annotated Hibernate mapping in Java. Now let’s read some existing data out of the database:

string connectionString = @”…”;

DataContext db = new DataContext(connectionString);
Table<LinqSchema> schemata = db.GetTable<LinqSchema>();

// nice little LINQ expression
var query1 = from ss in schemata where ss.Name == “minimal” select ss;

// iterate of the query result
foreach (var qq in query1)

Console.WriteLine(string.Format(“Schema {0}”, qq.Uuid));

That’s pretty self-explanatory. Now the funny caveats. Initially, one of the columns was an integer type, and in .NET I put in a long. One would think integers can get casted into longs, yet the error I got was this:

I was quite stunned that the SqlDependency extensions for query caching which were introduced in SQL Server 2005 and .NET 2.0 didn’t work with LINQ at all. There are a few hacks out on the internet, but I couldn’t get them to work reliably.

I found association mappings to be somewhat inelegant, especially compared to Hibernate which doesn’t require you to redundantly map the IDs and the referenced objects (1-1 mapping shown in this example):

// column maps the id
[Column(Name="id_schema")]
public long id_schema;

// entity reference holds the reference object
private EntityRef<LinqSchema> _Schema;

// the actual association. Storage refers to the entity reference,
// ThisKey to the variable containing the id.
[Association(Storage = "_Schema", ThisKey = "id_schema", OtherKey = "Id")]
public LinqSchema Schema
{
get{ return _Schema.Entity;}
set { _Schema.Entity = value; }
}

All in all, however, a big step in the right direction. On the downside, .NET now has multiple competing options for remoting and persistence, and interoperability between all these options is not necessarily given.

The instructions out there on MSDN are nice, but there doesn’t seem to be a page that outlines all the things you actually have to do to use a custom key container. I’m also faced with a deployment scenario on ancient Windows 2000 servers, and the commands are slightly different here. It’s pretty straight forward, so for posterity I’ll outline the steps here.

The core tool is delivered with the .NET framework and called aspnet_regiis.exe.

Key Generation

First of all, let’s create a new custom container to hold the application keys. Another option would be to use the system default container NetFrameworkConfigurationKey, but let’s create a container specific our application:

C:\app> aspnet_regiis -pc MyCustomContainer
Creating RSA Key container...
Succeeded!


This now created the container and a new key inside it. The next step is to export the key into an XML file:

C:\app> aspnet_regiis -px MyCustomContainer mykeys.xml
Exporting RSA Keys to file...
Succeeded!

The file mykeys.xml now contains the keys. Keep this file safe, we will need to distribute it on the web farm later. The contents of the file looks like this:

<RSAKeyValue><Modulus>5C31Pc/ppGZjWuoUdQ9HjiAIxp8UaVYvgYXz3K4i3qzpW9al6s+2kG8ktHpSfZ/y6H4qDpaqI2TU5ltYdxr4Iv6sPhnTEKMv1N7AWq/FNwVTCM8XiWXqs6K+UW7mDfCI
ArF1Fqo+kFV1LJtuPUahy2TOmnji+ZUzzuwFqzqFop0=</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>

Encrypt your application configuration

First of all, a new encryption provider needs to be registered in the web.config which specified the custom key store to use. You can skip this when using the default key store. Pay special attention to the keyContainerName attribute, it must match the name of the container created.

<!-- sets up the encrypted configuration to use a special container -->
<configProtectedData>
<providers>
<add name="MyCustomProvider"
type="System.Configuration.RsaProtectedConfigurationProvider, System.Configuration, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
keyContainerName="MyCustomContainer"
useMachineContainer="true" />
</providers>
</configProtectedData>

To now encrypt the configuration file from the command line (you can easily embedd this into an msbuild/nant script), call the following command. Make sure the provider name passed in matches the provider name from the above configuration snippet. The following script will encrypt the connection string section:

aspnet_regiis -pef "connectionStrings" MyWebFilesDirectory -prov "MyCustomProvider"

The connection string section in the web.config file should now look like this:

<connectionStrings configProtectionProvider="RsaProtectedConfigurationProvider">
<EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
xmlns="http://www.w3.org/2001/04/xmlenc#">
<EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#tripledes-cbc" />
<KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
<EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
<EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
<KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
<KeyName>Rsa Key</KeyName>
</KeyInfo>
<CipherData>
<CipherValue>qDaaLdZtxtv1pbrXplfiHvUCKLq0ZpwJQorowmlm74QIH3ogcZyhNpYTgLCSNxaB4KaTLtcryRaivqPr+v/Lr6iv18d/VaIV1Ht0sCagA3sZCUQu/GuSTcTgfpQGdveU4Qsx5g4ERC/B8mQEwXtHs7a6VNVsYD4f6GFlO9g4iUI=</CipherValue>
</CipherData>
</EncryptedKey>
</KeyInfo>
<CipherData>
<CipherValue>lw9Y03jHQ6foqIAdN7UN0nOxc4y2zJ4ajb5dUjtxKyFa87vCmmMbBnTkatcoZ/6ZzibdEBg7Xmhf4SBqC4iCfQIgV5XIEb0j+P3gmDLSt+kC+enTLZGVlHt3ngND6dXJTF28C2jsVB2DIxI/u0p6cJ2UHRGfqvvetG24HnDPpZonlHuMGEyvKy/PhLf3OUv2Vc/v0aVi8srRwydjA1Ueqs9sGo+wGGqT1Leui3R/TCNiEwZrp1eO3Rt6W1NMEaRNkG3nz9tctHwOsNo9xiC+lM0IfGI/DFR04uwSzjuQX61Ph18nB+Q0zgcpauOsqRrNtorBSPqu+mfW8EL0bztU90T5F5i0TJ50BoXXo6F8kdTnJ1i90REFxsFBv/8jOXh++bUZMWRxnu3GUf+tJ5f7Kzr52GlApfkoMunTCLbd5zkPG8yga2iMJ+MGdqu9K2b+Qakpu8VNu6Ioo80l30Dbxw==</CipherValue>
</CipherData>
</EncryptedData>
</connectionStrings>


Register Keys on Web Farm

Register the key on all web servers running the application. First of all, create the container with the keys from the XML file created earlier:

C:\app> aspnet_regiis -pi MyCustomContainer mykeys.xml
Creating RSA Key container...
Succeeded!

Now the IIS application requires access to this container.

On Windows 2000, do this (assuming you’re not running IIS as a different user):

C:\> aspnet_regiis.exe -pa "MyCustomContainer" "ASPNET"
Adding ACL for access to the RSA Key container...
Succeeded!

On Windows 2003 and up, the following command shall work:

C:\> aspnet_regiis.exe -pa "MyCustomContainer" "NT AUTHORITY\NETWORK SERVICE"
Adding ACL for access to the RSA Key container...
Succeeded!

And that’s really it. Once the application with the encrypted web.config sections is deployed, IIS will automatically decrypt it.