<Update 17-7-2009>
Did an update on this post at New MOSS SP2 bug .. continued and problem solved!
In short, I was copying old dll’s in the GAC using WSPBuilder. In my project I had old references to SharePoint .dll’s copied locally in my project folder. So while executing the “Copy to GAC” method of WSPBuilder, the ‘old’ .dl’s were copied into the GAC. By doing that it caused my WFE thinking that it was running an old version while the database was thinking it was running a new version.
</Update>
I twittered about this problem two weeks ago I merely thought that it’s possibly my own fault because of the development work I was doing.. So recreated my environment by reinstalling SharePoint and everything was fine (though it’s quite irritating that when you attach your ‘old’ contentdatabases to your new farm it starts indexing everything from scratch and the mssearch/msddmn and sqlserver starts consuming all resources.. and that’t not funny on a VPC ;) but after a clean SharePoint (with SP2) install everything was fine again..’Till now! I got the same error back again.
First it starts with this when navigating to a SharePoint site in my browser..
[SPUpgradeException: An error has occurred on the server.http://go.microsoft.com/fwlink?LinkID=96177]
Microsoft.SharePoint.Administration.SPContentDatabase.ValidateDatabaseSchemaCompatibility() +1805
Microsoft.SharePoint.SPSite.PreinitializeServer(SPRequest request) +61
Microsoft.SharePoint.SPWeb.InitializeSPRequest() +290
Microsoft.SharePoint.SPWeb.EnsureSPRequest() +58
Microsoft.SharePoint.SPWeb.get_Request() +54
Microsoft.SharePoint.SPWeb.GetWebPartPageContent(Uri pageUrl, PageView requestedView, HttpContext context, Boolean forRender, Boolean includeHidden, Boolean mainFileRequest, Boolean fetchDependencyInformation, Boolean& ghostedPage, Byte& verGhostedPage, String& siteRoot, Guid& siteId, Int64& bytes, Guid& docId, UInt32& docVersion, String& timeLastModified, Byte& level, Object& buildDependencySetData, UInt32& dependencyCount, Object& buildDependencies, SPWebPartCollectionInitialState& initialState, Object& oMultipleMeetingDoclibRootFolders, String& redirectUrl, Boolean& ObjectIsList, Guid& listId) +1492
Microsoft.SharePoint.ApplicationRuntime.SPRequestModuleData.FetchWebPartPageInformationForInit(HttpContext context, SPWeb spweb, Boolean mainFileRequest, String path, Boolean impersonate, Boolean& fGhostedPage, Byte& verGhostedPage, Guid& docId, UInt32& docVersion, String& timeLastModified, SPFileLevel& spLevel, String& masterPageUrl, String& customMasterPageUrl, String& webUrl, String& siteUrl, Guid& siteId, Object& buildDependencySetData, SPWebPartCollectionInitialState& initialState, String& siteRoot, String& redirectUrl, Object& oMultipleMeetingDoclibRootFolders, Boolean& objectIsList, Guid& listId, Int64& bytes) +718
Microsoft.SharePoint.ApplicationRuntime.SPRequestModuleData.GetFileForRequest(HttpContext context, SPWeb web, Boolean exclusion, String virtualPath) +232
Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.InitContextWeb(HttpContext context, SPWeb web) +104
Microsoft.SharePoint.WebControls.SPControl.SPWebEnsureSPControl(HttpContext context) +395
Microsoft.SharePoint.WebControls.SPControl.GetContextWeb(HttpContext context) +31
Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.PostResolveRequestCacheHandler(Object oSender, EventArgs ea) +383
System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +68
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +75
And the EventViewer showed this error:
The schema version (3.1.8.0) of the database SharePoint_AdminContent_3b1b1115-48b5-40d1-afe3-142692142cd6
on dev is not consistent with the expected database schema version (3.1.3.0) on dev.
Connections to this database from this server have been blocked to avoid data loss.
Upgrade the web front end or the content database to ensure that these versions match.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Looks quite scary since the SharePoint_AdminContent database is the contentdatabase of the Central Administration WebApplication. But it makes sense for me, since I’m using this Web Application extensively in my solutions. On the other hand I find it quite strange because it’s not the first time that I’m doing this and did not do anything that obscure or scary.
So of course, I tried to do an upgrade using the STSADM –o upgrade –url <centraladmin> –inplace.. When I checked in the Upgrade.log I found this error
[SPDelegateManager] [DEBUG] [5/29/2009 11:54:04 AM]: No assembly manifest found.
[SPDelegateManager] [ERROR] [5/29/2009 11:54:04 AM]: Failed to call GetTypes()
[SPDelegateManager] [ERROR] [5/29/2009 11:54:04 AM]: Could not load type 'Microsoft.Office.Server.Administration.Health.SPHealthAnalysisRule' from assembly 'Microsoft.Office.Server, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c'.
[SPDelegateManager] [ERROR] [5/29/2009 11:54:04 AM]: Failed to call GetTypes()
[SPDelegateManager] [ERROR] [5/29/2009 11:54:04 AM]: Could not load type 'Microsoft.Office.Server.Administration.Health.SPHealthAnalysisRule' from assembly 'Microsoft.Office.Server, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c'.
[SPManager] [ERROR] [5/29/2009 11:54:04 AM]: CanUpgrade [SPFarm Name=SharePoint_Config] failed.
Now if I’m not mistaken, the Health class is added to the Microsoft.Office.Administration namespace since SP2 right?
Next thing was to try do an upgrade using the PSConfig tool.. and I got an Access Denied error
An exception of type System.Security.SecurityException was thrown. Additional exception information: Access denied.
System.Security.SecurityException: Access denied.
at Microsoft.SharePoint.Administration.SPPersistedObject.Update()
at Microsoft.SharePoint.Administration.SPService.Update()
at Microsoft.SharePoint.Administration.SPPersistedObject.Update(Boolean ensure)
at Microsoft.SharePoint.PostSetupConfiguration.ServicesTask.InstallServiceInConfigDB(Boolean provisionTheServiceToo, String serviceRegistryKeyName)
at Microsoft.SharePoint.PostSetupConfiguration.ServicesTask.InstallServices(Boolean provisionTheServicesToo)
at Microsoft.SharePoint.PostSetupConfiguration.ServicesTask.Run()
at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()
The Zone of the assembly that failed was:
MyComputer
So I think the error about the “Could not load type..” and the SecurityException are linked to each other.
I tried the ‘execadmsvcjobs’ command to see if that give me any errors.. it didn’t.. but the ‘sync’ command did throw an error and in the ULS it showed this :
05/29/2009 13:47:02.21 STSADM.EXE (0x17E0)
0x0C44 1597 6y3d Exception
System.Security.SecurityException: Access denied.
at Microsoft.SharePoint.Administration.SPPersistedObject.Update()
at Microsoft.SharePoint.Administration.SPJobDefinition.Update()
at Microsoft.Office.Server.CommandLine.SyncCommands.ProcessScheduleParams(
String multipleWebAppUrls, SPSchedule sweepSchedule, SPSchedule syncSchedule)
at Microsoft.Office.Server.CommandLine.SyncCommands.Execute()
The Zone of the assembly that failed was: MyComputer
Some background information on what I was doing when this happened, I was working on a custom timer job and restarted the Timer service a lot of times.. And ‘suddenly' the environment was down. When it happened a couple of weeks ago I did a solution install and during an installation of a solution package the Timer Service is also restarted. And when a Timer service restarts, a lot of timer jobs are also being executed right? Maybe the upgrade timer job was still busy trying to update the content database of my Central Administration but never got the chance or something..
<Update>
Found some blogposts that describe the problem after applying a hotfix and servicepacks
Only ‘weird’ thing is that this didn’t happen to me after a the service pack update but after two weeks after applying the service pack. So I’m still convinced it’s a timer job issue..
</Update>
Up to this point I don’t have a solution for this problem, but if I do I will update this post.. or if you have the solution.. let me know! ;)