Originally posted on SharePoint Magazine
Before going into detail concerning each of the tests, I would like to point out two things about this list:
- These are not in any particular order.
- These are “Five Important Tests” not “The Five Most Important Tests”.
I have put them together for two main reasons:
- To compliment the existing test regime that I’m sure you already have in place.
- As a reminder (sure you never forget!) because these are the main reasons I see SharePoint solutions fail.
This is not a “How-To” article, but I have tried to include examples of “real world” experiences I’ve encountered along the way. Now, without further delay, lets take a stroll through Five Important Tests you should perform before “Go Live” with your SharePoint solution.
1. Fill ‘Er Up!
Add lots of data to your solution. SharePoint, fundamentally, is a data driven application. By that I mean, the application changes and adapts based on the data it is filled with. Most of the SharePoint solutions you build will be driven less by HTML and code, and more by data, including navigation, document libraries, lists, web parts and the list goes on. The result of this is that your solution will inevitably behave differently when it has lots of data in it, and at the risk of stating the obvious, it will rarely perform better!
SharePoint compounds this problem by having some serious limitations on the amount of data that it promises to perform well with. These are outlined in the "Planning for Software Boundaries” documentation (that every SharePoint developer should know by heart).
Of course, our development environments never have much data in them, which means that “everything works fine on my machine”. Unfortunately, problems related to data volumes often fail to show up until months after the solution has been deployed, and by that time, it can be very difficult to go back and “Fix it”.
One final note, back in my RRE (Rapid Response Engineer) days, the #1 reason for a CritSit (critical situation/server down) were custom Site Navigation controls. This was usually a web part that built out a tree control using a recursive loop of all sites. When you have hundreds of sites, combined with the need to dispose (which almost no one did), you have an application pool recycle problem on your hands!
Working with large lists in Office SharePoint Server 2007 (must read)
SharePoint 2007 Test Data Population Tool
Plan for software boundaries (Office SharePoint Server)
Plan for software boundaries (Windows SharePoint Services)
SharePoint Dispose Checker Tool
Load up with data until you know when you’re creaking.
2. You entered what?!?
Sometimes it feels like the people using the solutions I build have completely different keyboards to the one I have. They are always entering the weirdest characters, in the weirdest places, leaving me with the most infuriating bugs. Infuriating because they are so easy to fix, and I know, deep down, that I really, really should have picked it up. This of course is vastly more complicated in an international environment.
21 Apps Blog – Andrew Woodward has some great resources on Unit Testing and SharePoint
TypeMock for SharePoint
Have some testing resources (automated or otherwise) dedicated to entering the strangest characters you can imagine, into the strangest places you can imagine.
3. Are you allowed to do that?
I couldn’t imagine a solution built on SharePoint that has only a single permission level. SharePoint is a collaboration platform, so by its nature, it brings people together, safely, by ensuring everyone can do only what they should be able to do. The custom solutions we build are no different.
With that said, developers are typically running with every permissions available (rightly or wrongly!). The problem of course is that running with all these permissions makes it very difficult to use the system exactly as an end user would. This problem is amplified by the SharePoint Object Model which, thankfully, will always respect a users permissions and in whose context much of our code will later run. The classic symptom of code that works when logged on as an administrator, but not as a “normal” user, is an authentication box when trying to access a site they otherwise have permission to access.
Another aspect of this is giving deep thought to exactly what rights a user should really have as an “Administrator”. For example, initially, our blogging solution made the person who owned the blog a Site Collection administrator. We very quickly realised that this was a bad idea. Why would the owner of a blog need to activate new features? Modify the navigation? This just ended up confusing people, so now, a blog owner gets a reduced set of permissions tuned specifically to the solution we have built for them, simplifying things, which after all, is our job as solution designers.
Best Practices for Elevated Privilege in SharePoint and Elevated Privilege with SPSite by Daniel Larson
Elegant SPSite Elevation by Keith Dahlby
Make sure you are testing the system using the roles you have defined, and make sure you are giving people only the permissions they really need.
4. Two Servers are better than one
SharePoint, through its Solution Framework, does a great job of making it easy to install solutions across a farm. That is of course if you have one, many customers don’t need need either the scale or the availability to warrant one. However, in my mind, this doesn’t excuse developers from worrying about whether our solutions will work in a farm environment. You never know what the future holds.
So, what can go wrong? Well, let me tell you a little story. Many years ago, in the SPS2003 days, I was part of a team that put together a solution that stored navigation information for SharePoint sites in an XML file (that’s another story). The solution worked great “on the developers machine”, but on our medium server farm we had a problem, when someone updated the navigation on one front-end, that change was not reflected on the other front end. Obvious when you think about it, but you will be amazed how often you don’t.
The solution of course was also obvious, move the XML file onto shared storage, somewhere both machines could access it. We tested away, and again, on our development environment it worked. We went ahead and made the change, only to find that in production, nothing worked. The problem of course, was that we were hitting that classic of all issues, the NTLM Double Hop problem (to date I have lost around 6.3 days of my life to that this issue).
Now, that had a relatively simple solution too, impersonation, but the important bit is not how simple the solution is, but what it ended up doing to our timelines, and how embarrassed I felt! <grin>
Anyway, double hop, resource storage, state management (potentially) and Web.Config modifications are just a few of the issues you can encounter once you move to a farm environment.
How To: Modify the web.config file in SharePoint using SPWebConfigModification by Mark Wagner
Test on a multi-server environment sooner rather than later.
5. Baseline it!
Your application works, job well done. You have completed all the testing you possibly can, everyone has signed everything off. You are ready to go into production.
Wait, before you do, there is one more thing!
Create a baseline. That is, an accurate representation of exactly how your application performs under a controlled set of parameters. Its critical to ensuring that the future upgrades you make are safe.
This is actually simpler than it sounds (which is not to say its simple), for example it could be:
- A virtual machine that includes your application, fully configured, and with a complete set of sample data. Ensure that it is representative of what you will find when it does go into production.
- Visual Studio Test Edition suite, or basically any load testing tool (see resources).
Then, pick your metrics, rinse, record and repeat. You need to get to a point where you have a pretty good grasp of how the system you are testing behaves while operational.
With this in hand, and before your next release, before going live, upgrade the environment and repeat exactly the same steps. The important thing then is to look for variation. If you suddenly find an unexpected increase of 50% in memory consumption, then you know something isn’t right. In this case, its the performance relative to the baseline that’s important, not the actual numbers.
MOSS2007 – Visual Studio 2008 Testing (Load Test) by Liam Cleary
MS Web Application Stress Tool
Baseline the performance of your solution so that you can compare it to future releases.
As I said in my opening, there are many, many other important tests that should be performed. So lets start a discussion, what test do you perform before “Go Live”, and what do you think are the most important?