dotTrace is a Wonderful Tool

Anyone that has done much programming, especially web programming, has from time to time had to look into a performance issue.  I have had to do this numerous times on numerous projects and the cause(s) can be incredibly varied.  Sometimes the culprit is a non-indexed table, other times large objects are loaded multiple times instead of shared throughout a page's lifecycle, or maybe someone decided a cursor was a good thing (a cursor is never a good thing btw).

This brings me to this week.  As some of you know, the application I'm currently working on is a large (> 2M lines of code) redistributed web based management application.  There has always been talk of it being slow in places but nothing quantitative until recently.  So this week the testing department ran some timing tests of the current production version of the application against the version that is currently in Beta and found that a few parts of the application are performing 3x slower than previously.  This isn't good.

Enter dotTrace.  I've heard about this application from multiple sources, most recently in twitter, and it has sounded like a good way to see what's going on.  After I did a quick look at the application it sounded good so I downloaded it and started the 10 day trial.  It didn't take long for me to completely fall in love with dotTrace.  I immediately referred it to the other developer also doing performance evaluation.  Soon my manager was looking to buy a few licenses, but I'm jumping ahead.

This brings us to what I found.  On initial testing nothing stood out as a bad offender, yes the pages in question weren't great, but nothing jumped out as a real trouble spot.  Then I was finally able to load a real customer database that the testing had been done against and re-ran the test.

This is what I first saw:

Wow.  95% of processing and > 17s all in Page_Load.  Now that's not unexpected as most method calls in are within that phase of the lifecycle but still, the numbers just jumped out like a red flag attached to a 2000lb bull being chased by a locomotive.  So I dug a little deeper and found this:

Hmmmm, 10,802 calls to GetGroupSummaryChild, which is doing auditing that results in 10 calls each to GetCustomAttributes.  That's 108,020 reflection calls which even at only .1 milliseconds each add up to almost 11 seconds.  Totally unacceptable.

So I looked a little deeper and found that there are 10,802 groups in the system.  Not all are needed for this page so the first though was to create a call that only returned the groups being needed.  This is still a good solution but when I looked even deeper at why pulling those groups was slow that's when I noticed the auditing calls that in turn are using the reflection calls which is where much of the time is actually being spent.  Which brought up the question "Why is this being audited, it's a read-only collection fetch"?

As it ends up a boolean was being hard-coded to true for all loads of GroupSummaryChild and there was never any update code in place.  So that resulted in the reflection calls to load attributes that would be needed in case of auditing but were never actually used.  So by flipping that hardcoded boolean from true to false, the total execution time of the page went from nearly 18 seconds to under 2.5 seconds when tested with all variables the same but that one boolean.  There is still room to improve the page performance but that now falls into the acceptable range.  Here is the trace file after just flipping that boolean to disable the loading of unused audit data:

Vast improvement there, we could have stopped at that point, but we kept going.  We decided to also address the issue of loading > 10,000 groups when we only needed about 25.  Seems like a good idea right?  In our system this entails not just adding the overload for the GetCollection method but also that new signature has to be passed up through Interfaces and Factories (we use reflection a lot) and of course the stored procedure needs modified as well.  Overall there would need to be changes in around half a dozen different places, not major changes, but files touched anyway. 

Well, unbeknownst to me the exact overload I needed was added about three weeks ago (the page calling the old version was built about two years ago), so all I had to do was modify the page to use the new version of GetCollection passing in the correct parameters and done.  Here's the trace after that fix was also tested:

That's right, under 1 second.  Exact same functionality as before any changes were made.  For doing everything this page is doing, that's not bad at all.

Best of all, all of this from running the initial trace against real data to testing the first fix against real data only took maybe 30 minutes.  Very worth a license of dotTrace as to find and fix that bottleneck without a good tracing tool would have been much more difficult and time consuming.

Pain in the Neck: Better

Quick follow-up on the last couple of posts on my neck.  I'm better.  All better.

Well, maybe not ALL better but definitely seriously improved.  Odds are that this is going to come back again, hopefully it will be another couple years before it happens though. 

So that's it. I'm good.

Poll: What is most important to you in a job?

The recent rash of people changing jobs has really made me think about reasons to switch jobs and about what makes a job worth the switch. 

So I would love to hear from people what they desire in a job.  So I'm going to throw out some items that are desirable in a job and why don't people rank them as to what they want/desire.

• Interesting project(s)
• Good co-workers (Talented)
• Good co-workers (Likeable)
• Good environment (City/Region)
• Good environment (Office)
• Cutting edge technology
• Difficulty of project
• Company size (Big)
• Company size (Small)
• Variety
• Salary

I've changed jobs for a lot of these reasons but here is a crack at my personal rankings.  My personal rankings will change from one moment to the next btw.

1. Interesting project(s)
2. Good co-workers (Likeable)
3. Difficulty of project
4. Cutting edge technology
5. Variety
6. Salary
7. Good co-workers (Talented)
8. Good environment (Office)

I also believe that any of these can be so good or so bad that it sways the rest of the list.  For instance, if the project is boring but the difficulty and co-workers are high, it still might make the cut (and in fact has).  Also, none of this is set in stone, numerous things can sway a job for good or for bad, I'm just trying to add some science to this chaotic habit of changing jobs. 

So let me hear your thoughts.  Feel free to add as much as you want to these items also.  Or use something completely different, I'm just curious.

How Dinosaur Web Developers Can Regain Their Shine: Step 2

Get Involved.

That's it. Step two is just to get involved.

Ok, there's a little more to it than that, it takes a commitment to stay involved. But luckily, just being involved helps you to stay involved, more on that in a second.

In-Person Events

User groups (UG), special interest groups (SIG) and conferences large and small are wonderful ways to learn new information than you would normally get through your normal 8-5 job. The Cleveland area has a bunch of disconnected user groups that cover many areas of web (and other) technologies. The Greater Cleveland PC User Group (GCPCUG) is an umbrella organization of which many smaller, more focused groups are members. It is a good starting place to find what you are looking for. Not all groups are part of it though which can make finding them difficult at times.

A few groups to note are:

• Cleveland .NET Special Interest Group
• Cleveland C#/VB.Net SIG
• Cleveland Web Standards Association

There are also events thrown by some of the large companies like Microsoft. Being a MS focused developer myself the events I'm most familiar with are events like Arcready, DevCares and MSDN Events but there are bound to be others for other technologies / companies. Some larger organizations also have their own internal groups that do the same things. I helped start an ASP.NET user group at the Cleveland Clinic and I know that Progressive has one as well, National City was starting an architect's book club when I was there last.

UPDATE: I forgot to mention a Google group set up to help find events: NEODevEvents

Online Tools

As much as social networking is a Web 2.0 buzzword (or should that be buzzphrase) it is truly a good way to meet and communicate with people of similar interests. Just like with Web Standards I can't say that I am great at this, or know everywhere to 'hang out' but I can say that just from the last couple of months I've already begun to know a wide range of developers, architects, managers and others in the field that have similar thoughts and ideas to my own. These people are at companies large and small are young and old, are experienced and beginners. What they all share though is a passion for what they do. And it's contagious. This is the part that helps keep you involved, everyone inspires/motivates each other. And you do it without trying.

My favorite at the moment is twitter. Everyone in the tech field it seems uses twitter, from all fields really but I only care about tech. When you first hear twitter described your (at least my) first thought is "what's the big deal?" and for the first week or two you still fail to see the allure. But then once you've found a group that's like yourself, then it all changes. Then when twitter is down (which unfortunately happens frequently) you will seem lost. One odd thing about twitter is that opposed to IM and similar things twitter does not seem to negatively affect productivity. It could be due to the asynchronous nature of it you don't need to reply right away, or even read things right away. Not only is the non-time sensitive nature of twitter capable it seems to be just the way things are done, it's very common to replay or receive a reply hours or even days after your message. One last thing about twitter that makes it unique is that you don't actually have to know someone to interact with them. You can follow nearly anyone you want to and see their messages. They don't see yours in normal conversation unless they are following you too but if you reply to them, they will see your post anyway. This lets unknown developers in Ohio respond and interact with 'big' names from around the world.

A few other relevant networking sites:

• LinkedIn
• Facebook
• Plaxo

Collaboration & Networking

What I've talked about are of course not the only ways to get involved, it's just a starting point. Another good thing to do is to participate in an open source project at sourceforge or codeplex. There are also professional organizations like ACM or IEEE among others. If you have another way you like to be involved pass it along, I'd love to hear about it.

The best things about being involved locally and online are that you are constantly exposed to new technologies and people that are passionate about using them. Typically the more active members of these groups spend time experimenting and learning new technologies long before they are widely used and as such are great contacts for when you too are trying to learn those same things. Getting involved with other people passionate about technology is by far the best way to keep yourself motivated. Just be careful not to spend so much time that the WAF (or HAF I suppose) dips too low.

Disclaimer: Due to severe neck pain I'm currently taking a few meds that have um, dulled my senses so if this is less than legible that's why :)

Pain in the Neck: Getting Better - Slowly

So I've been getting asked via numerous channels (twitter, email, in-person) how my neck is and how the Dr. apt went yesterday so here's a quick summary.

The nerve pain in my shoulder/neck area is definitely being lessened by the Prednisone. That was the biggest issue so there is much relief in that.  The Doc was very frustrated at the Insurance company for the exclusion rider on my back/neck/spine and as such didn't order an MRI since most cases of this clear themselves up with time anyway, my last occurrence of this cleared up on it's own too.  He also didn't charge me the full office visit charge, couldn't have, it was less than some co-pays.  Great doctor, if anyone in the Akron/Canton needs an orthopedic doctor hit me up for a recommendation.

It also appears that many, if not all, of the issues I've been having the last few days are side-effects of the Vicoden and Prednisone that are helping my pain.  I've got to say that some of the side-effects are pretty scary, like trembling so bad at times that even typing is difficult.  Knowing it's the drugs helps though. 

Anyone that has been seriously injured/ill knows that the psychological battle is half of it, if you worry yourself it makes it worse.  If nothing else the doc gave me hope that it's going to get better on it's own with just the Prednisone.

So I'm a bit better, pain is less and hopefully the trembling and insomnia will calm down soon.

Web Accessibility: Why should we care?

Several things have had me thinking a lot about accessibility lately.  A number of things all happened at around the same time which brought this to the forefront of my thoughts.  A manager at work referred another developer to me on an accessibility question, I found that Bobby is offline and this neck problem I'm having has scared me with the possibility that I could be a disabled web user one day.

A Little Background

This is a topic that I actually know quite a bit about.  In previous jobs in earlier years I had to know quite a bit about accessibility, both in the rules and regulations and also in how to construct a site that followed those rules.  I frequently had to instruct others in following these rules and for a while every single site I built had to conform to these rules.  I haven't had to worry about accessibility much for the last few years so I was worried that things would have changed so much that I would be playing catch-up again.  Luckily (for me at least) the same standards that were around then are still current.  I'm not going to get into what the standards are and how to code in an accessible way in this post, I'll be doing a whole series on those topics.

Why Should We Care

I would guess that most readers of this blog are developers.  Not designers (I'm not that cool), not simple HTML writers, not information architects.  It should be obvious be obvious to people in those roles why they need to think about accessibility when working on a site.  For the advanced .NET developer though, who is often handed a design and a requirements document, why do we need to think about accessibility. 

First, We should ensure that the sites/applications we build are useable to the widest possible user base.  Wether you do this out of a sense of helping the community (generosity), possibly increasing market share (money) or to avoid possible lawsuits (fear) it is something that needs done.  Currently Section 508 of the ADA only is enforced on government, school and similar types of organization but I believe that will change.  Just as office buildings and retail stores have ramps and accessible bathrooms now, regardless of who they serve, the ADA will eventually be applied to all commercial websites.  It's only a matter of time.

Secondly, The anti-nmp(tm) reason.  Depending on the size and structure of an organization, frequently lines of responsibility are drawn in the sand as to who is in charge of what.  Often this creates an attitude where even if someone sees a deficiency, if it is not in their area of responsibility they ignore it (Not My Problem).  Accessibility should never be treated this way.  People who use alternate methods of accessing a web site or application don't do it out of choice, they do it because it is their only practical way of using the information / services that are readily available to the rest of the surfing public.  As developers we should have expert knowledge of what the standards are and how our programming choices will affect them, for good or bad.  I'm not going to attempt to create a change in the politics of offices around the globe, what I am preaching evangelizing though is that accessibility is too important to let it slip when you know how to fix it.  So I'm saying we should practice the opposite of not my problem (anti-nmp) for this important topic.

Third, What if it were someone you loved.  What if your spouse, parent or child was suddenly not able to see or have full use of their hands or some other disability that affected their use and enjoyment of the Internet?  What if it were you?  If you've ever had your broadband connection at home go out for a day or two how did it impact you?  I bet you realized you use the Internet much more frequently than you had thought.  Now imagine that half, or really the majority of those sites were difficult or impossible to use because of a physical limitation.  It would probably impact the way you live your life, not just the disability, but the lack of accessible Internet sites.

What to do

I'm going to be writing a ton of posts on this topic as it is something I care about quite a bit.  Newer technologies such as AJAX, Silverlight and Flex bring new challenges to making accessible web applications but it is not insurmountable.  In this series I will try to explain the rules and address how to apply them with current web technologies.  I know that I have work to do to clean up this blog to make it more accessible, I'm not saying that I'm perfect in doing this, I'm just saying that web accessibility is something that we all should learn about, care about and practice daily in our jobs.

Old Code Part 3: Serious Hack Job

For this edition of Old Code I bring up a project I worked on back in 2002 that was a serious hack job.  The code was not pretty, the architecture was not pretty, but it solved the issue and I believe it is still running cleanly today.  This was for another big client too, the Ways and Means Committee of the U.S. House of Representatives.

Background: In those days I was working for a company that had a bit of an identity crisis.  They did both political and government work, under different business names.  This particular project started similar to others I had built for other House Committees, Representatives and political candidates.  We had an in-house built content management system with SQL Server 2000 as the data store and classic ASP as the front end.  It wasn't the best system but it worked well for it's purposes.  As with clients in pretty much every industry, specifications didn't really exist.  My direction in building this app had been quite sparse "Build it like you did the Financial Services Committee with a couple extra sections."  So that's what I did.  This one was a bit more complex as they had a ton of old information they wanted to import and quite a few extra sections but our system handled it with a little customization and we were good.  Or so I thought.

Problem:  First, I've got to say that it is a little intimidating to be in a big wood-paneled conference room on Capital Hill with a bunch of committee staffers sitting across from you.  So the handoff meeting was progressing nicely and then the committee staffers start to say "what about this piece of functionality?"  True to form our owner, who had worked out the details with the committee, had agreed to things that 1) our software couldn't do and 2) he didn't tell me.  When the smoke cleared I've got a laundry list of new functionality to add to the site including two wonderful items:

1. Integrate our CMS with the Library of Congress
2. Integrate our CMS with the Government Printing Office

Apparently the committee staff had been told that our system could not only manage the content in their website but also could communicate with the LoC and GPO systems seamlessly.  Having never even spoken with anyone from either of those organizations, I knew this was going to be a ton of fun.  Oh yea, and I only had about 4 weeks to get this all done.

Issues: After speaking with the people from both of those organizations I learned that the GPO integration wanted an SGML file FTPd to them nightly with all of the committee activity.  Included in the schema was if it were the final version among other things.  This being 2002 I hadn't built a site that FTPd info to another site before but as you'll see in the solution below, I found (hacked) a way to make it work).  The LoC was a different issue completely, I don't remember it fully but I do remember that they would crawl our site, if told where to look, and look for specially formatted links.  Also, they needed links to their site to do an HTTP POST, not a GET like every other link on the entire interwebs.

Solution:  The GPO issue was solved by creating a DTS package on the SQL server that would run nightly.  This package was configured to pull the results, format them into SGML, save it off to a text file, and finally FTP it to a server at the GPO.  Writing about it now it sounds simple but pre.NET and for someone that had little experience with DTS packages, it was a little daunting.  But it worked, and I believe it is still working today (I left that company four years ago so I can't be completely sure).

For the LoC the solution was a little messier but easier.  If I recall correctly, most of what we had to do was to create specially formulated links over to their site.  By links I mean embedded forms where the link actually posted the form to their site.  I know there was more to it than that but that part was hackish enough to remember.  That is possibly why they are still using the same classic asp site that went live more than 5 years ago.  Although .NET gives you much cleaner ways to accomplish that.

Conclusion:  Why is it that the code we feel has the worst hacks sticks around the longest?  Apparently I did something ok since the site is still up after 5 years and a power shift in congress.  That part means something since frequently when power shifts from one party to the other (why do we only have two parties again?)  they redo everything the first part already had.  Frequently for no other reason than they can.  So either I made the site good enough or so complicated that they don't want / can't figure out how to replace it.

ArcReady Cleveland - 2/5/08

This morning was the ArcReady Cleveland event with the topic of Service Lifecycle Management. This is a very valid topic these days with more and more organizations creating services and not really having any strategy on how they plan to update/upgrade those services.

Josh Holmes kicked off the event, introduced us to the speaker and then promptly disappeared until the giveaways at the end. Well, he was there for the break too. Josh not speaking at this event didn't detract from anything though as the speaker was quite capable and didn't need any hand-holding.

So the speaker, Chris Madrid (at least I think that was his name), had his slides together for a different event I'm guessing as he skipped over the majority of them but still managed to get his point across. Josh is supposed to provide a link to download the entire presentation for those interested. The first half of the talk was about concepts instead of practice and while that is very appropriate for an architect talk, he was moving fairly quickly and people weren't engaged. He then demonstrated a little codeplex app they have built that acts as a service gateway and adds capabilities such as versioning and logging. Nice app that they are hoping to keep up to date but as it's not a sanctioned product you get what you pay for.

Unlike the last couple of ArcReady events, the crowd was dead quiet until after the break. He about had to beg us to start talking. I think the biggest barrier to conversation was that very few people in the crowd had any relevant experiences to share. Once he got to demonstrating the app people really started to get it and conversation (mostly in the form of questions) started. The talk was oriented towards enterprise IT departments instead of at ISVs so not everything translated directly but he did make a good case for the usefulness of at least having a corporate-wide defined strategy as for how to deal with service lifecycles.

There were three to four times the number of attendees as were at the last ArcReady event including numerous people that I 'think' read this blog (which is a small number). I ran into an old co-worker, a current recruiter trying to lure me away and several other devs trying to get a code dojo/camp started. Hopefully with some of the contacts I made to day we will get enough momentum going to actually start an event.

As far as I know the next event I'll be reviewing will be the March 13th launch event. I may attend a SIG or UG before then but XNA and Ruby don't fit in my path so I doubt it.

Oh, and the streak continues, no prize for John.

Did IBM Kill Bobby?

Anyone that has had to build an accessible website, pretty much ever, has probably used Bobby. It has been a tool to allow a web developer, home hobbyist or school librarian to check to see if a given page or a whole site was useable by disabled visitors. This little application existed long before section 508 and has been around since the mid 90s.

Well, today I went to check to see if a site was valid with Bobby and there was a little message saying that it is no longer accessible. It appears that WatchFire, an IBM company, is now trying to make money from something that has been used by schools, non-profits and government agencies to ensure that their sites were accessible for more than a decade.

Nice work IBM, you've certainly made an impression on one web developer.

Pain in the Neck

For those reading this that are just looking for technical content, this post isn't for you.  Nothing to see here.  Move along now.  These are not the droids you are looking for.  Those of you who know me probably know a lot of this already but as this is my blog and not yours, I'm going to talk about it anyway :).

One Saturday morning day about two years ago I woke up in excruciating pain.  It felt as if my right shoulder and arm were on fire.  The pain was worse than anything I've felt before (and I've been injured a lot so that means something).  I spent the weekend laying in bed eating Motrin and Tylenol constantly.  Finally Monday morning the wife convinced me to go to stat care and get it looked at.  Well to shorten this story, after several weeks of incredible pain I ended up being diagnosed with a compressed nerve caused by a herniated disc in my cervical spine (neck).

Then the pain and muscle weakness (did I forget to mention that my right arm was so weak I couldn't open doors with it?) went away.  The orthopedic surgeon I was seeing said I would probably still need surgery one day but for now I was ok.  I got to look at the MRI images of my neck and while most of my spinal disks were perfectly round, one looked like a melted life-saver.  The surgery for this is nasty too since they have to go in through the front of your neck.  Yes, it leaves a big scar right in front. 

So for the last two years I've occasionally had pain in my neck or a twinge in my shoulder/arm but it didn't last long and wasn't too bad.  I ignored it.  About a year ago I was rear-ended on my drive home and had whip-lash fairly bad.  That scared me as the pain was right where I had the disk trouble already.

Also during this time I quit my job and became an independent contractor.  Which means no group insurance.  The first few insurance companies I approached wouldn't cover me at all because of the pre-existing condition.  Finally I found one (UHC) that would cover me but they put in an exclusion saying that they won't cover anything related to my neck, back or spine.  So if I have to go to the doctor, or worse have surgery, then I'm completely screwed.

Which brings us to now.  Friday morning I woke up and all was good until I stretched.  Then it got bad.  I heard/felt a pop and all of a sudden my neck hurt bad.  Real bad.  Not your typical neck pop, hurts for a day kind of pain either.  It's like the disk is gone completely and the vertebra are just crunching against each other.  Ouch.  Didn't get better either.  Went home early.

So now here I am, second day and it's not better, worrying over not having any insurance for it.  I've spent all morning looking up insurance laws and supplemental coverage and well, it's looking like my days of being a contractor are numbered as the only way for me to get covered is to be on a group plan.  So if this isn't better by Monday, I might have to start talking to people about getting on a group plan somehow.  Either that or sell my house and everything in it to pay for the medical bills.  Option one sounds better.

Great.