ASP.NET MVC vs. WCSF

I've been thinking about the best way of building enterprise web applications after creating few sample applications with both ASP.NET MVC framework and WCSF and if I was to look at from a holistic perspective, both the frameworks provide excellent separation of concerns thus enhancing the modularity and testability of the application yet there are differences in their approaches.

ASP.NET MVC provides a new ground-up framework for building web applications that doesn't uses the core ASP.NET framework concepts like PostBacks, ViewState, Code Behind etc. It really enforces the discipline of creating simiplistic views, which are dependent on a controller to handle user events and pass the model to it. It forces one to think of Views as a pure rendering mechanism and doesn't allow the developer to pollute the .aspx files with all the gobbledygook of interactions with services to build a page. It provides a very clean way of handling all the HTTP verbs and provides a pluggable architecture to inject components to handle those requests. All the HTTP calls are routed to a controller which decides the action i.e. method to invoke, gets the right model and passes it to View to render the data. One can build very clean REST based solutions by using ASP.NET MVC framework. To learn more about the framework, I would recommend reading Scott Guthrie's blog and viewing these videos.

WCSF on the other hand builds uses the existing ASP.NET Forms based framework and provides a pattern based approach of building applications. It has guidance bundles which provides the right solution and project structure, code generation recipes, out of the box integration with various application blocks like Security, Exception Handling and Logging. It also provides good separation of concerns by following the MVP design pattern. Thus, a developer doesn't have to learn new patterns of building web applications and can utilize all the core concepts of PostBack,ViewState etc. and on top of it is given pre-configured access to the Logging, Exception Handling and Authorization features.

IMHO, WCSF is a better option if one was looking at capitalizing on ASP.NET strengths and, standardizing web development practices. ASP.NET MVC framework provides utmost flexbiity in creating web solutions but it's a 180 degree change in the way we have been building ASP.NET web applications and I'm not even sure how would it pan out while building enteprise applications. In the coming days, I would publish a detailed comparison of both the technologies.

Starting out with Web Client Software Factory a.k.a WCSF

Lately, I've been completely submerged in Software Factories - WCSF and WSSF(Web Service Software Factory) and have come to like one of them, WCSF. I'm bit surprised about the fact that it hasn't caught on to lot of people's imagination albeit having lots of features to create ASP.NET web applications in a standardized way. 

What it is?

It is designed to help one quickly and consistently create web applications that adhere to well known architecture and design principles and patterns. It is an integrated collection of VS.NET solution and project templates, various code recipes, design patterns and prescriptive guidance about creating large scale modular web sites. It synthesizes ASP.NET, ASP.NET AJAX, MVP Design Pattern, Application Blocks of Enterprise Library, Windows Workflow Foundation and VS.NET Solution templates and wizards to create these modular web sites.

Why should one care about it?

There are various reasons why one should care about the Client Software Factory but there are few that stand out -

• It provides a sound baseline for creating web applications which provides a mature web solution structure, out of the box integration with various application blocks like Security, Exception Handling and Logging and, good separation of concerns between layers(through Dependency Injection) thus improving the testability of the application.
• The Factory allows an organization to customize and extend the code recipes and templates to suit their architectural style.

What is the best way to get started?

The best way to get started on WCSF is to install it and go through the hands-on lab on codeplex. One needs the following before installing the client factory -

• Guidance Automation Extensions - This would enable the custom recipes of WCSF. Please install this before the GAT.
• Guidance Automation Toolkit (for VS 2005) - This would facilitate customizations. Please follow this link if you have VS 2008 installed
• Finally, install WCSF from here

Though there are lot of how-to articles and help on codeplex for WCSF but I've found David Hayden's Screencasts an invaluable resource in understanding WCSF. He has brief presentations covering various aspects of WCSF and I would highly recommend viewing all his screencasts before starting out.

Marriage of WCF and ASMX

I never imagined that writing a simple POX based application with correct interface semantics in the current release of WCF would be such a pain. Wait. It is impossible.

The intent was to expose a WCF service that can return POX when invoked through a regular well-formed URL with Query strings or submit HTML form data to it while doing a HTTP-POST -- Just like we used to do in the good ol' ASMX days. Guess what, this feature is not natively built into the current release of WCF.

Well, the WCF geek would be shrugging his head right now, itching to show me the way to build WCF services that can be consumed by HTTP-GET/POST verbs and can return POX. Not so fast, boy. I'm cognizant that one can build services which can return POX but they are loosely typed i.e. they have to declare methods which can only take Message type as an input. What does one do when one has to define operation contracts that need to define native data types as input parameters in the method signature?

Okay, I lied.

I know there is yet a better way -- To build a router, basically a message inspector, that can route the messages to the right operation contracts by inspecting the kind of action that is desired by the user. But, can you imagine how much code one would have to write to build this router in the context of a gigantic enterprise class application? I think you already know the answer. At least, I don't want to. As a matter of fact, I cringe imagining someone writing that kind of code as it would not be maintainable in the long run.

The good news is that WCF in .NET 3.5 would support the desired feature set out of the box but the bad news is that the upcoming framework is still in beta and to make matters worst, there isn't any confirmed release date. We have to roll out the system in production by January thus couldn't bet on the beta release. Thus, I ended up choosing ASMX to build our service delivery platform. But, I really wanted to harness the power of WCF to build the service delivery platform as it provides options galore to configure and host services.

So, what did I end up doing? Married both the technologies. Our service interfaces have both the WebService and ServiceContract Attributes, the methods have both the WebMethod and OperationContract Attributes. We ended up created ASP.NET Web Services project and decorated the WCF tags on the services which would allow us to switch to the new framework when it comes out without compiling the code. Of course, we would have to provide the right configuration to host it as a WCF service but still we wouldn't have to touch the code. Here is how a service interface looks like --

This way we de-risk ourselves by not adopting a beta product but still be ready to embrace the new framework. Have you grappled with this problem and have a better solution? I would appreciate if you can take the time out to share your thought process.

Agile Crazile

Blasphemy! You would say.It might be, especially when the whole software world treats agile methodologies as a panacea to all its problems. I have to admit that I've been in a perpetual state of skepticism about the benefits of agility and the fact of the matter is that I haven't found any valid reason to move out of my comfort zone of skepticism.

I think it tries hard to solve the problem of showing the ROI faster to the client but in the process ends up creating an even  bigger problem -- Chaos. Chaos has ensued in almost all the engagements that we have executed using agile methodology. Some of it definitely comes from our lack of understanding of the agile practices but most of it comes from the way the methodology evangelizes the projects to be executed. IMHO, agility has got benefits but it hasn't matured yet(again, could either be lack of our understanding or dearth of good case studies). Most of the material out there has been written by very gifted writers which ends up seducing everybody to espouse it but I think most of it is bit pedantic and lacks pragmatism. Furthermore, almost all of it has been written by comparing it with the wooly mammoth of Waterfall -- Waterfall is bad thus Agile is good, hence proved. Whoa!

What leads to this chaos? The culprit, more or less, in all the engagements has been the flux in requirements and the monstrosity involved in managing frequent iterations. I believe the whole notion of succinct user story is somewhat misleading. It doesn't give good enough idea to think about the design holistically and provide enough opportunity to put hooks in place so that the system can evolve over a period of time. Think of products like Operating Systems, Office based applications, your favorite IDE, Amazon.com, eBay, iTunes etc. -- do you really honestly believe that the engineers of these systems would have just gotten user stories, planned them in iterations, did few design spikes, few whiteboard sessions, created "just enough" documentation, planned enough for the first few months and starting churning out code without putting emphasis on rock solid design and what the future holds. Well, I don't think so.

I don't think one can adopt the agile philosophy to create systems which target millions of users. I believe all the lofty notions of brief user stories, design evolution and "design is code" can be put on a decorative mantle in these cases. While constructing such systems, wouldn't you want to detail out the vision of the system? Wouldn't you want to create detailed use cases from the perspectives of all the users? Wouldn't you want to take your time to design? Wouldn't you want to get submerged in creating comprehensive documentation to convey the thought process to the army of developers rather than doing the "just enough" documentation which can lead to ambiguity? I think the cost of refactoring the design and code and deploying them again is astronomical in this context thus it's better to follow a thorough approach rather than an agile one.

The other thing which contributes to the crazility is frequent releases. Somehow, the development team always gets bogged down by fixing the critical bugs from the previous iteration's feedback loop. One would think that the choice comes down to working on new set of requirements or bugs from the previous cycle but that generally isn't the case. Why? The thing is the development team has to roll out the agreed upon user stories and the customer also expects the team to fix the bugs thus the team dynamics goes out for a spin. It seems to me they get tied in a vicious cycle where they are always constantly refactoring the design and code to accommodate the new set of changes. It is like a dog chasing its tail.

The problem worsens when one doesn't have strong program management in place which can't push back and educate the customer about its fallout on the team dynamics. Ultimately the developer team has to work longer hours at work. This defeats one of the core objectives of agile which mandate the developers work only 40 hours week in regular iteration followed by a sprint towards the roll out of the product.

Don't get me wrong, I absolutely love the idea of frequent iterations as it provides real value to the customers as they get to be part of the growth of their software but detest it for its repercussions on team dynamics. I love it as it gives the customer an opportunity to provide early feedback to the team but abhor how the development team has to work long hours to incorporate that feedback. I think the core of both the above problems is that agile methodology believes there would be enough time to build the software the right way but the fact of the matter is that software is always conceived by a strong time bound market need and, the software would gather dust if it was to miss that timeline. In my experience, I've not once seen a piece of software being constructed without a time bound market need.

I believe most of the agile practices are best suited for a very ideal software construction setting where the developer works in tandem with the customer and both of them somehow magically agree on all timelines and what features have to be constructed. I think that kind of framework would let developers easily manage the flux in requirements and incorporate the changes from the feedback loop without any added pressure.

Nonetheless, as I said in the beginning, I'm a skeptic, but still working passionately towards to become a believer as there are lot of merits in following agility. Thus, I'm currently reading two of the famous works on Agile which have been recommended to me by people I respect a lot --

1. Agile Principles, Patterns, and Practices in C# by Robert C. Martin

2. Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results by David Anderson

I would love to hear any thoughts on how to execute projects on agile methodology, especially some good case studies of executing large scale enterprise projects.

Got Thoughts?

PS - Thanks to my friend Rajeev Singh for coming up with a very creative name for the problems we are facing on daily basis i.e. Crazy + Agile = Crazile

Designing Windows Folder Structure

For the first time, I was let down by my esteemed GSP(Google Search Professional) credentials when I couldn't search for a single white paper or article about data segmentation techniques for storing large number of files on a disk. Thus, I had to spend couple of hours to create a test harness and gather results to figure out the optimal number of files in a folder.

The Context

We have tens of millions of files to be served from our media server and we were looking for optimally arranging them in a folder structure so that it doesn't impact performance. The options were either stuffing files on a single folder or creating a hierarchial structure of folders to store these files. If we were to opt for the latter then we had to solve another problem -- the optimal number of folders in a given folder? (It turns out that this really is not a problem as the performance level for files and directories are very comparable)

The Approach

I decided to write a simple program which would create N number of files in a folder and would then try to locate a particular file by its name. It turned out to be very simple program to write but I had lot of "doh" moments while running the program. For creating the files, I just created a simple method to copy an image file and paste it with a unique name --

   private static void CreateFiles(int numberOfFiles)
   {
        Stopwatch stopWatch = Stopwatch.StartNew();
        for (int i = 0; i < numberOfFiles; i++)
           File.Copy(sourceFile, path + "thermometer" + i + ".jpg");
        stopWatch.Stop(); 

        Console.WriteLine("It took {0} seconds to create {1} files", stopWatch.ElapsedMilliseconds/1000, numberOfFiles);
        Console.Read();
   } 

And, to locate a file --

    private static void LocateFile(string filePath)
    {
         Stopwatch stopper = Stopwatch.StartNew();
         bool isFound = File.Exists(filePath);
         Console.WriteLine(isFound);
         stopper.Stop(); 

         Console.WriteLine("It took {0} milliseconds to find the file", stopper.ElapsedMilliseconds);
         Console.Read();
    } 

I ran the program for the following configurations --

• 1,000 files in a folder
  
• 10,000 files in a folder
  
• 15,000 files in a folder
  
• 18,000 files in a folder
  
• 1,000 files residing in their own unique folders. Thus, it had 1,000 folders.
  
• 10,000 files residing in their own unique folders. Thus, it had 10,000 folders.

The Results

I ran the tests in a single user mode on WindowsXP with 1.6 GHz single core, 1 Gb of RAM laptop having a 5400 RPM disk. I've taken out the results for configuration #5 and #6 as they were identical to #1 and #2 respectively.

Our overall SLA to serve an image is 500ms thus we want the image retrieval cost to be within 40-50ms as the application has to fire business rules before serving an image. According to the test results, I've come to the conclusion that the ideal size for our case is really 10K files in folder as the OS was able to serve an image in 37ms at the first invocation and 6ms in the subsequent invocations. The overall strategy that has been devised is --

• Create a Hash function which can take the name of the image file and output the folder name. The folder name will not be more than 4 digits so that it can support maximum of 10k entries.
  
• Create folders underneath the hashed folder with the unique ids(For our case, it would never be more than 10K at this level)

Next Steps

• Create an ASP.NET site which would serve the images.
  
• Run the ASP.NET site in a program like JMeter to check the results in a multi user mode.

Please feel free to comment on the above approach and share your experiences in case you have designed for such a scenario.

Attribute Driven Architecture

I think it's an Architect's primary responsibility to come up with the most bizzare and ambiguous terms to define very simple constructs - We somehow have to show that we exist for a reason :). I was recently asked to take a session on Arch. 101 to share the things that we consider before creating architecture. Thus sprung the idea of discussing all the "abilities" and how they act as a conduit to come up with the complete technology roadmap. Here is my perspective on all the design principles that can be followed to promote one ability over another --

Thou shalt not make architecture without having an ability in mind

What are these so-called abilities and how do they come in play while crafting the blueprint of an application?

• Maintainability
• Performability(is that even a word?) a.k.a Performance
• Scalability
• Reliability
• Flexibility

By the way, there are plethora of abilities but I will cover the important ones which come in play often enough.

Maintainability -- It is defined as an ability of a system to undergo repair and evolution. Following principles can help one achieve this ability – 

• Modularization

• Single Responsibility Principle – One should design classes in such a way that there is only a single focus for change.

• Open-Closed Principle – One should design frameworks which are open for extension but closed for modification

• Liskov Substitution Principle – One should design sub-types which are substitutable for their base types.

• Guidelines for Writing Good and Readable Code -- I'm cognizant that this is really not a design principle but helps tremendously in creating maintainable applications.

Performance – It is defined as the responsiveness of the system. Keeping the following principles in mind can help build a high performance application -- 

• Caching – One can follow the W3 principle of Caching – What to cache, When to Cache and Where to Cache.

• Asynchronous Behavior

• Coarse Grained Interfaces – Chunky interfaces always perform better than the Chatty ones.

• Data Partitioning

• Efficient Resource Management

Scalability – It is defined as the ability to maintain performance while system demand increases. Following design principles can help build a scalable application -- 

• Layering

• Loose Coupling

• Data Partitioning

Reliability – It is defined as the ability of the system to keep operating over a period of time. Designing the system while keeping the following things in mind promote reliability –

• Robustness

• Clustering

Flexibility – It is defined as the ease with which a system can be modified for use in applications or environments other than those for which it were specifically designed. Following principles promote flexibility of a system -- 

• Layering

• Loose Coupling

• Single Responsibility Principle

• Open-Closed Principle

• Code To Interface

• Interface Segregation Principle

Of course, there are always trade-offs one need to consider when thinking of one ability over another. For example, if one is designing for flexibility that one should be ready to let go of some performance constraints.

References – Implemeting System Quality Attributes by Gabriel Morgan