Old Code Part 4: I thought it was a Joke

About three years ago some senior executives at our company were in town for a big planning meeting.  As frequently happens during these times the entire office went out that night to show the visitors a good time.  During that get-together, the director of our office came to me with a question: Is it possible to have our web based application open a desktop app on the server and have a user over the web control that application.  I basically and politically said no, with the standard disclaimer that with enough time and money anything is possible of course.  So I thought that was the end of it.

The next afternoon I'm sitting at my desk working and in walks the director who promptly asks how I'm doing on making the web app launch and use the other app.  I was kind of stunned as I thought the topic was done but I start researching at the point.  Pretty much everything I can find either says it can't be done or you are stupid to even try.  The best answer on Google Groups was to quit working at a place with such unrealistic expectations.

So I took my findings to my boss who says "Yea, that's what I'm hearing from other people too, but you still need to make this work."  And with that I went to work on a prototype.

Over the next week I learned about using p/invoke to launch a windows app.  It was a little odd, being a web developer but within a week I was able to launch the windows app and send keystrokes to it.  Having proven that at least that much was possible my boss then assigned a couple more developers to work on this.  Luckily one of these developers had a windows background and so he took care of the lower layer.  We then had the prototype actually scraping the windows app and passing the values up through a windows service, to the web app.  At that point the web app would convert the values into screen coordinates and control types, and populate then with values.  We even had the app sending each keystroke from the client browser to the web app via AJAX and whenever the windows app updated itself we would refresh the screen to update the visible controls.  It would also spawn multiple instances of the web app, one for each user.

We had accomplished what we had thought was not possible!

When we showed this to the director, he was glad that we got it to work but to me didn't seem as impressed as he should have been :).  He did go off and tell the Japanese sponsors of the project about it though and before we knew it we were on a 8:00PM WebEx with Japan showing off our prototype.  Before this call got started we were under the impression that it was going to be just a couple of the executives from the earlier meeting in Canton.  When it actually got started however, it sounded like an entire conference room full of people.  We ended up finding out that the development team in Japan that had written the windows app had been tasked with this same goal earlier and had tried and failed.  I would NOT have wanted to be in that room when the inferior American developers showed it working :).

In the end, the project that we had proven possible was moved offshore where it was morphed into something else entirely.  At least we knew we had done it, and now so do you :).

dotTrace is a Wonderful Tool

Anyone that has done much programming, especially web programming, has from time to time had to look into a performance issue.  I have had to do this numerous times on numerous projects and the cause(s) can be incredibly varied.  Sometimes the culprit is a non-indexed table, other times large objects are loaded multiple times instead of shared throughout a page's lifecycle, or maybe someone decided a cursor was a good thing (a cursor is never a good thing btw).

This brings me to this week.  As some of you know, the application I'm currently working on is a large (> 2M lines of code) redistributed web based management application.  There has always been talk of it being slow in places but nothing quantitative until recently.  So this week the testing department ran some timing tests of the current production version of the application against the version that is currently in Beta and found that a few parts of the application are performing 3x slower than previously.  This isn't good.

Enter dotTrace.  I've heard about this application from multiple sources, most recently in twitter, and it has sounded like a good way to see what's going on.  After I did a quick look at the application it sounded good so I downloaded it and started the 10 day trial.  It didn't take long for me to completely fall in love with dotTrace.  I immediately referred it to the other developer also doing performance evaluation.  Soon my manager was looking to buy a few licenses, but I'm jumping ahead.

This brings us to what I found.  On initial testing nothing stood out as a bad offender, yes the pages in question weren't great, but nothing jumped out as a real trouble spot.  Then I was finally able to load a real customer database that the testing had been done against and re-ran the test.

This is what I first saw:

Wow.  95% of processing and > 17s all in Page_Load.  Now that's not unexpected as most method calls in are within that phase of the lifecycle but still, the numbers just jumped out like a red flag attached to a 2000lb bull being chased by a locomotive.  So I dug a little deeper and found this:

Hmmmm, 10,802 calls to GetGroupSummaryChild, which is doing auditing that results in 10 calls each to GetCustomAttributes.  That's 108,020 reflection calls which even at only .1 milliseconds each add up to almost 11 seconds.  Totally unacceptable.

So I looked a little deeper and found that there are 10,802 groups in the system.  Not all are needed for this page so the first though was to create a call that only returned the groups being needed.  This is still a good solution but when I looked even deeper at why pulling those groups was slow that's when I noticed the auditing calls that in turn are using the reflection calls which is where much of the time is actually being spent.  Which brought up the question "Why is this being audited, it's a read-only collection fetch"?

As it ends up a boolean was being hard-coded to true for all loads of GroupSummaryChild and there was never any update code in place.  So that resulted in the reflection calls to load attributes that would be needed in case of auditing but were never actually used.  So by flipping that hardcoded boolean from true to false, the total execution time of the page went from nearly 18 seconds to under 2.5 seconds when tested with all variables the same but that one boolean.  There is still room to improve the page performance but that now falls into the acceptable range.  Here is the trace file after just flipping that boolean to disable the loading of unused audit data:

Vast improvement there, we could have stopped at that point, but we kept going.  We decided to also address the issue of loading > 10,000 groups when we only needed about 25.  Seems like a good idea right?  In our system this entails not just adding the overload for the GetCollection method but also that new signature has to be passed up through Interfaces and Factories (we use reflection a lot) and of course the stored procedure needs modified as well.  Overall there would need to be changes in around half a dozen different places, not major changes, but files touched anyway. 

Well, unbeknownst to me the exact overload I needed was added about three weeks ago (the page calling the old version was built about two years ago), so all I had to do was modify the page to use the new version of GetCollection passing in the correct parameters and done.  Here's the trace after that fix was also tested:

That's right, under 1 second.  Exact same functionality as before any changes were made.  For doing everything this page is doing, that's not bad at all.

Best of all, all of this from running the initial trace against real data to testing the first fix against real data only took maybe 30 minutes.  Very worth a license of dotTrace as to find and fix that bottleneck without a good tracing tool would have been much more difficult and time consuming.

Old Code Part 3: Serious Hack Job

For this edition of Old Code I bring up a project I worked on back in 2002 that was a serious hack job.  The code was not pretty, the architecture was not pretty, but it solved the issue and I believe it is still running cleanly today.  This was for another big client too, the Ways and Means Committee of the U.S. House of Representatives.

Background: In those days I was working for a company that had a bit of an identity crisis.  They did both political and government work, under different business names.  This particular project started similar to others I had built for other House Committees, Representatives and political candidates.  We had an in-house built content management system with SQL Server 2000 as the data store and classic ASP as the front end.  It wasn't the best system but it worked well for it's purposes.  As with clients in pretty much every industry, specifications didn't really exist.  My direction in building this app had been quite sparse "Build it like you did the Financial Services Committee with a couple extra sections."  So that's what I did.  This one was a bit more complex as they had a ton of old information they wanted to import and quite a few extra sections but our system handled it with a little customization and we were good.  Or so I thought.

Problem:  First, I've got to say that it is a little intimidating to be in a big wood-paneled conference room on Capital Hill with a bunch of committee staffers sitting across from you.  So the handoff meeting was progressing nicely and then the committee staffers start to say "what about this piece of functionality?"  True to form our owner, who had worked out the details with the committee, had agreed to things that 1) our software couldn't do and 2) he didn't tell me.  When the smoke cleared I've got a laundry list of new functionality to add to the site including two wonderful items:

1. Integrate our CMS with the Library of Congress
2. Integrate our CMS with the Government Printing Office

Apparently the committee staff had been told that our system could not only manage the content in their website but also could communicate with the LoC and GPO systems seamlessly.  Having never even spoken with anyone from either of those organizations, I knew this was going to be a ton of fun.  Oh yea, and I only had about 4 weeks to get this all done.

Issues: After speaking with the people from both of those organizations I learned that the GPO integration wanted an SGML file FTPd to them nightly with all of the committee activity.  Included in the schema was if it were the final version among other things.  This being 2002 I hadn't built a site that FTPd info to another site before but as you'll see in the solution below, I found (hacked) a way to make it work).  The LoC was a different issue completely, I don't remember it fully but I do remember that they would crawl our site, if told where to look, and look for specially formatted links.  Also, they needed links to their site to do an HTTP POST, not a GET like every other link on the entire interwebs.

Solution:  The GPO issue was solved by creating a DTS package on the SQL server that would run nightly.  This package was configured to pull the results, format them into SGML, save it off to a text file, and finally FTP it to a server at the GPO.  Writing about it now it sounds simple but pre.NET and for someone that had little experience with DTS packages, it was a little daunting.  But it worked, and I believe it is still working today (I left that company four years ago so I can't be completely sure).

For the LoC the solution was a little messier but easier.  If I recall correctly, most of what we had to do was to create specially formulated links over to their site.  By links I mean embedded forms where the link actually posted the form to their site.  I know there was more to it than that but that part was hackish enough to remember.  That is possibly why they are still using the same classic asp site that went live more than 5 years ago.  Although .NET gives you much cleaner ways to accomplish that.

Conclusion:  Why is it that the code we feel has the worst hacks sticks around the longest?  Apparently I did something ok since the site is still up after 5 years and a power shift in congress.  That part means something since frequently when power shifts from one party to the other (why do we only have two parties again?)  they redo everything the first part already had.  Frequently for no other reason than they can.  So either I made the site good enough or so complicated that they don't want / can't figure out how to replace it.

Old Code Part 2: XML / XSLT Before it was Cool

For this issue of Old Code I'll be going way back. Before .NET, before Visual Studio, before broadband. I wrote this app before many of you were even working. This was probably my favorite project of all time, still stands up to that today. Since this was so long ago I can give more details than in the last issue so here we go.

Background: The year was 2000 and companies small and large were realizing they had to get online. Some because they had a business need to do so, others because "everyone else was'. This was a case of an actual business need.

I worked for a smallish web design / development company called DigitalDay, they had just changed their name from Mozes Cleveland and Company. We had some good sized clients like Sherwin Williams and Ernest & Young but our biggest (duh) was GE. We did a lot of work for GE Plastics and their Silicones division, incredibly huge multi-national companies. Anyway, one day I had just returned from TechEd 2000 in Orlando and was pulled into a conference call with a branch of GE Silicones in Europe. It was actually a joint venture between GE and Bayer (yes, the aspirin people). They had a problem.

Problem: When these two separate companies were merging their ERP systems it was concluded that they systems were incompatible so they switched both companies to SAP R/3. The E-Commerce team had been given a nearly impossible deadline of getting a system online using the new SAP system within just a few months. They did accomplish this goal (don't worry, we'll get to my role soon) as this was right after SAP had released the MySAP module which contained its own proprietary web server and exposed SAP modules online. Done, first goal accomplished. But there were issues.

Issues: While they met the milestone, no one was satisfied with the results. The web application wasn't branded GE Bayer Silicones but instead was branded MySAP including colors, logos and everything. Also, this was long before most manufacturing companies in Europe had broadband Internet and for them this application was incredibly slow. Customers complained. Then came the call.

So on this call they got us into the application to take a look around and wanted our opinion on if we could do anything to speed it up. It didn't take long before I had identified that the biggest issue affecting speed was the .js files being downloaded. They totaled around 300k. Even today in the AJAX and JSON world 300k .js files are a little excessive but in 2000 that was out of control. So the call ended and I went back to work only to be asked minutes later if I had a passport. Two weeks later I'm on a plane to Amsterdam to spend 10 days looking at the issue in Bergen Op Zoom Holland.

Once there I dug into the MySAP solution to see what I could do to fix it. So after three trips to Holland totaling over three months I had built them a solution that I am still proud of today.

Solution: The big requirement was to display a screen to the user with information on 300+ purchase orders in the quickest time possible. I crafted a solution where the page would have an embedded XML Data Island (linking XML was less reliable in those days) and linked XSL file, CSS file and JS file. The XSL/CSS/JS combination gave the ability to create HTML on the fly for two different layouts and allow sorting and filtering as well. All in client side code.

MySAP had already constrained them to using IE only, so I also had them use 5.0 or higher which gave me access to browser XML support and also having to only code the JavaScript once. I worked with ABAP programmers from Deloitte to get the information to me that I needed. I then had to learn SAP's server side language BHTML, which was simplistic but different, to craft the XML to send with the page. Due to the size constraints I went against "proper" XML design and used single character element names and no whitespace at all as every byte meant a lot when it would be repeated hundreds of times ((once in opening tag + once in closing tag) * number of results). Oh yea, and I built this entire app in Notepad. Not kidding. No IDE, not even Notepad Plus.

Conclusion: While using XML with client code in a browser may be common-place today, it was totally new at the time. XSL Transforms done in the client browser still isn't common, I did it in 2000. Basically between three months in Europe, totally cool technology and one of the best project managers I've ever worked under, this was my favorite project ever.

Lessons Learned:

• Two-Day shipping from Amazon to The Netherlands cost more than the book itself
• Never spend an anniversary in Amsterdam unless you can do it every year (I did, I haven't, I hear about it frequently)
• Never take wife to Europe unless you can move there (I did, I haven't, I hear about it frequently)

Old Code Part 1: Plug-In(able) Web Application

Today I'm starting a series that I'm calling "Old Code".  This will be where I go over some project I've worked on that was notable for some reason but may not directly translate into a "how-to" for developers today.  These posts will more be about some problem to which there was no simple solution but to which a solution was found that was usually either particularly creative, cutting edge (for its time) or a total hack but that got the job done.  Some of these projects I'm proud of, others I'll be telling you about even if it may not be my best work.  Overall though I think we can all gain by looking back at our past projects and letting the public review them for with us.  In most of these I won't be able to provide code samples or even give application / company names because of NDAs but I don't think that will impact the overall posts.

I this first installment I'm going to be talking about a project I was working on about four years ago. Disclaimer: My memory of 2004 may be a little fuzzy but I don't think I can take full credit or blame for this solution.

Background: The overall project was to create a web-based management application for complex devices which did have their own internal memory.  Basically this application was to replace numerous desktop applications with a multi-user, accessible anywhere, cool, user-friendly, do everything management application.  When the project got started it was 2004 and we were building it in C# and ASP.NET on the .NET 1.1 framework.

Problem:  Of these devices that the application was to manage there are numerous 'families' to which the capabilities of and communication to are drastically different.  This application was to allow the ability to be installed with the ability to support one or more family and have the ability to support future families possibly being added by third-party developers.  They didn't want to release the source code or to have to recompile every time a new family was to be added.  In theory a third party company that makes these devices would have the ability to create their own "plug-in" and install it at a customer location without our companies involvement.

Issues: With it being 2004 none of us had a ton of .NET experience but we did know a lot.  To our knowledge (and I still think this was true in 1.1) all code-behind files for a web app were compiled into a single DLL at build time.  This is where the real issue existed:  How do we let third-party developers seamlessly add pages to our .NET 1.1 web application?

Solution: Now the architecture that was devised to handle this is hard to explain without examples.  Who am I kidding, it's hard to explain with examples.  Many good developers took months to really grasp what we were doing, we'll see if you get it quicker.  Basically the application was split into pieces.  It was already n-tier with a full business object model based on Lhotka's CSLA framework.  Basically an architecture was created in which any family specific code would exist in satellite dlls that are loaded using reflection at run-time based on information coming from the SQL Server database.  This included the code-behind files for any pages or user controls that contained family specific controls/data.  Much work was done to construct generic widget classes that only contain fields that all widgets have regardless of family and then having the family specific classes derive from those classes and add their own fields / methods as well as all update logic.  While that is standard for OO business layer development, it's something else to apply that same paradigm to web pages. 

We ended up creating a structure where the base application would know that there was a widgetconfig.aspx page.  Then at run-time an intermediary page would intercept the request for the config page and append a family specific folder name in front so that the request would get to the correct page.  These was still the issue of the code-behind pages.  This is where it gets tricky.  For each family specfic page the actual aspx would only contain the generic header and footer user control declarations (this was pre-master pages) and a declaration for a user-control of the same name as the parent page.  In the directive for the family specific user control the assembly would be named to find the class in.  This assembly was a family specific dll that housed all code-behind files for that families pages.  In the class for that user control it would name the actual .ascx file to associate with the controls defined in the class. 

Through this method we were able to create a way to add a new family to an installed server by adding some dlls to the bin and some code-less aspx & ascx files to the application folders.  So it was possible to do this without touching the original source.

Conclusion:  After much coding effort and even more in education this strategy went untested for nearly four years.  Just today we got word that the first true use of this plug-in model had been successfully implemented using our SDK and without touching our source code.  While I'm sure that there were probably more elegant ways to accomplish this and that I'm glossed over much of the complexity this was a solution for the problem that worked.  Of course in .NET 2.0 it would have been much easier, but that didn't exist yet.  I was going to have a co-worker from that project review this before I posted it but what the hey, he can review it by reading it :).

Don't worry,  some posts in this series may contain actual code :).