icelava.net

why be normal?
Welcome to icelava.net Sign in | Help

Reply to an Existing Message

The message you are replying to: SharePoint 2010: Application Server job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance
icelava wrote the following post at 03-30-2011 7:05:

In these present weeks I am pretty caught up trying to ramp up my knowledge on SharePoint 2010 as much as possible. To get started, I obtained a VirtualBox virtual server from my colleague that he had already prepared with the basic installation bits. As time went by, I met a problem with my copy that he had not encounter on his:

The Search Service Application in Central Administration would throw out a generic error page when I attempted to visit the Crawling > Content Sources page. Peeking at the Windows Application event log startled me.

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Server
Date:          3/30/2011 7:00:21 PM
Event ID:      6481
Task Category: Shared Services
Level:         Error
Keywords:     
User:          NETWORK SERVICE
Computer:      WIN-NG0HQ5HJR6U
Description:
Application Server job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (785c55eb-c13e-424d-9393-ec92dc65f1d0).

Reason: The device is not ready. 

Technical Support Details:
System.IO.FileNotFoundException: The device is not ready. 
   at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
   at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)
Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-SharePoint Products-SharePoint Server" Guid="{C33B4F2A-64E9-4B39-BD72-F0C2F27A619A}" />
    <EventID>6481</EventID>
    <Version>14</Version>
    <Level>2</Level>
    <Task>3</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2011-03-30T11:00:21.643551100Z" />
    <EventRecordID>13946</EventRecordID>
    <Correlation ActivityID="{E7673D0A-CBDD-4DB9-A025-105212ED356A}" />
    <Execution ProcessID="1200" ThreadID="3060" />
    <Channel>Application</Channel>
    <Computer>WIN-NG0HQ5HJR6U</Computer>
    <Security UserID="S-1-5-20" />
  </System>
  <EventData>
    <Data Name="string0">Microsoft.Office.Server.Search.Administration.SearchServiceInstance</Data>
    <Data Name="string1">785c55eb-c13e-424d-9393-ec92dc65f1d0</Data>
    <Data Name="string2">The device is not ready.  </Data>
    <Data Name="string3">System.IO.FileNotFoundException: The device is not ready. 
   at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
   at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)</Data>
  </EventData>
</Event>

 

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          3/30/2011 7:00:28 PM
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:     
User:          NETWORK SERVICE
Computer:      WIN-NG0HQ5HJR6U
Description:
The Execute method of job definition Microsoft.Office.Server.Search.Administration.IndexingScheduleJobDefinition (ID e201e3a9-b58c-4c6f-b6cb-a5937e9606c8) threw an exception. More information is included below.

The search application '373c2f03-0d80-42ac-930e-892b76a6c7d8' on server WIN-NG0HQ5HJR6U did not finish loading. View the event logs on the affected server for more information.
Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-SharePoint Products-SharePoint Foundation" Guid="{6FB7E0CD-52E7-47DD-997A-241563931FC2}" />
    <EventID>6398</EventID>
    <Version>14</Version>
    <Level>1</Level>
    <Task>12</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2011-03-30T11:00:28.549801100Z" />
    <EventRecordID>13947</EventRecordID>
    <Correlation ActivityID="{F7E5AD67-B631-4720-AB04-E5899B7E5423}" />
    <Execution ProcessID="1200" ThreadID="2308" />
    <Channel>Application</Channel>
    <Computer>WIN-NG0HQ5HJR6U</Computer>
    <Security UserID="S-1-5-20" />
  </System>
  <EventData>
    <Data Name="string0">Microsoft.Office.Server.Search.Administration.IndexingScheduleJobDefinition</Data>
    <Data Name="string1">e201e3a9-b58c-4c6f-b6cb-a5937e9606c8</Data>
    <Data Name="string2">The search application '373c2f03-0d80-42ac-930e-892b76a6c7d8' on server WIN-NG0HQ5HJR6U did not finish loading. View the event logs on the affected server for more information.</Data>
  </EventData>
</Event>

 

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          3/30/2011 7:00:48 PM
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:     
User:          NETWORK SERVICE
Computer:      WIN-NG0HQ5HJR6U
Description:
The Execute method of job definition Microsoft.Office.Server.Search.Administration.CrawlReportJobDefinition (ID 04998e51-5d88-4a48-a979-134ca4a77468) threw an exception. More information is included below.

The search application '373c2f03-0d80-42ac-930e-892b76a6c7d8' on server WIN-NG0HQ5HJR6U did not finish loading. View the event logs on the affected server for more information.
Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-SharePoint Products-SharePoint Foundation" Guid="{6FB7E0CD-52E7-47DD-997A-241563931FC2}" />
    <EventID>6398</EventID>
    <Version>14</Version>
    <Level>1</Level>
    <Task>12</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2011-03-30T11:00:48.534176100Z" />
    <EventRecordID>13948</EventRecordID>
    <Correlation ActivityID="{B4749C97-4A81-48CD-A889-BFC969976672}" />
    <Execution ProcessID="1200" ThreadID="2308" />
    <Channel>Application</Channel>
    <Computer>WIN-NG0HQ5HJR6U</Computer>
    <Security UserID="S-1-5-20" />
  </System>
  <EventData>
    <Data Name="string0">Microsoft.Office.Server.Search.Administration.CrawlReportJobDefinition</Data>
    <Data Name="string1">04998e51-5d88-4a48-a979-134ca4a77468</Data>
    <Data Name="string2">The search application '373c2f03-0d80-42ac-930e-892b76a6c7d8' on server WIN-NG0HQ5HJR6U did not finish loading. View the event logs on the affected server for more information.</Data>
  </EventData>
</Event>

Device not ready? What device? Are data being stored in a removable thumb drive that I don't know of? Peculiar condition indeed; my colleague hadn't experienced such problems before either. A search around did not net any favourable explanation to what is happening, other than the fact that I am not alone with this.

Further digging suggested the complete removal of the existing Search Application and removal of the existing index. I did remove the Search Application, but when I went for the Crawling > Index Reset option, another generic error page is blown out. Swell.

The event log was also recorded with the following entries, especially when I restart the SharePoint Server Search 14 Windows service.

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Server Search
Date:          3/30/2011 5:53:04 PM
Event ID:      74
Task Category: Content Index Server
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      WIN-NG0HQ5HJR6U
Description:
Component: 373c2f03-0d80-42ac-930e-892b76a6c7d8
 An index corruption of type WidSetFormat was detected in catalog Portal_Content. Stack trace is
     tquery offset=0x0000000000034FEC (0x000007FED8A44FEC)
     tquery offset=0x000000000001E311 (0x000007FED8A2E311)
     tquery offset=0x00000000000ED3E4 (0x000007FED8AFD3E4)
     tquery offset=0x000000000012BC00 (0x000007FED8B3BC00)
     tquery offset=0x000000000012C3C3 (0x000007FED8B3C3C3)
     tquery offset=0x0000000000124326 (0x000007FED8B34326)
     tquery offset=0x0000000000124BF7 (0x000007FED8B34BF7)
     tquery offset=0x0000000000126821 (0x000007FED8B36821)

Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-SharePoint Products-SharePoint Server Search" Guid="{C8263AFE-83A5-448C-878C-1E5F5D1C4252}" />
    <EventID>74</EventID>
    <Version>14</Version>
    <Level>2</Level>
    <Task>138</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2011-03-30T09:53:04.042634000Z" />
    <EventRecordID>13803</EventRecordID>
    <Correlation />
    <Execution ProcessID="2088" ThreadID="4840" />
    <Channel>Application</Channel>
    <Computer>WIN-NG0HQ5HJR6U</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="string0">Component: 373c2f03-0d80-42ac-930e-892b76a6c7d8
</Data>
    <Data Name="string1">WidSetFormat</Data>
    <Data Name="string2">Portal_Content</Data>
    <Data Name="string3">
     tquery offset=0x0000000000034FEC (0x000007FED8A44FEC)
     tquery offset=0x000000000001E311 (0x000007FED8A2E311)
     tquery offset=0x00000000000ED3E4 (0x000007FED8AFD3E4)
     tquery offset=0x000000000012BC00 (0x000007FED8B3BC00)
     tquery offset=0x000000000012C3C3 (0x000007FED8B3C3C3)
     tquery offset=0x0000000000124326 (0x000007FED8B34326)
     tquery offset=0x0000000000124BF7 (0x000007FED8B34BF7)
     tquery offset=0x0000000000126821 (0x000007FED8B36821)
</Data>
  </EventData>
</Event>

 

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Server Search
Date:          3/30/2011 5:53:04 PM
Event ID:      71
Task Category: Content Index Server
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      WIN-NG0HQ5HJR6U
Description:
Content index on Component: 373c2f03-0d80-42ac-930e-892b76a6c7d8
 could not be initialized. Error Portal_Content.The content index is corrupt.   0xc0041800
Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-SharePoint Products-SharePoint Server Search" Guid="{C8263AFE-83A5-448C-878C-1E5F5D1C4252}" />
    <EventID>71</EventID>
    <Version>14</Version>
    <Level>2</Level>
    <Task>138</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2011-03-30T09:53:04.042634000Z" />
    <EventRecordID>13804</EventRecordID>
    <Correlation />
    <Execution ProcessID="2088" ThreadID="4840" />
    <Channel>Application</Channel>
    <Computer>WIN-NG0HQ5HJR6U</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="string0">Component: 373c2f03-0d80-42ac-930e-892b76a6c7d8
</Data>
    <Data Name="string1">Portal_Content</Data>
    <Data Name="string2">The content index is corrupt.   0xc0041800</Data>
  </EventData>
</Event>

 

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Server Search
Date:          3/30/2011 5:53:04 PM
Event ID:      12
Task Category: Gatherer
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      WIN-NG0HQ5HJR6U
Description:
The plug-in in OSearch14.Indexer.1 cannot be initialized.

Context: Application '373c2f03-0d80-42ac-930e-892b76a6c7d8', Catalog 'Portal_Content'

Details:
  (0xc0041800)
Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-SharePoint Products-SharePoint Server Search" Guid="{C8263AFE-83A5-448C-878C-1E5F5D1C4252}" />
    <EventID>12</EventID>
    <Version>14</Version>
    <Level>2</Level>
    <Task>137</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2011-03-30T09:53:04.042634000Z" />
    <EventRecordID>13805</EventRecordID>
    <Correlation />
    <Execution ProcessID="2088" ThreadID="4840" />
    <Channel>Application</Channel>
    <Computer>WIN-NG0HQ5HJR6U</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="string0">OSearch14.Indexer.1</Data>
    <Data Name="string1">

Context: Application '373c2f03-0d80-42ac-930e-892b76a6c7d8', Catalog 'Portal_Content'

Details:
  (0xc0041800)</Data>
  </EventData>
</Event>

 

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Server Search
Date:          3/30/2011 5:53:04 PM
Event ID:      11
Task Category: Gatherer
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      WIN-NG0HQ5HJR6U
Description:
The gatherer object cannot be initialized.

Context: Application 'Search_Service_Application', Catalog 'Portal_Content'

Details:
  (0xc0041800)
Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-SharePoint Products-SharePoint Server Search" Guid="{C8263AFE-83A5-448C-878C-1E5F5D1C4252}" />
    <EventID>11</EventID>
    <Version>14</Version>
    <Level>2</Level>
    <Task>137</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2011-03-30T09:53:04.058259000Z" />
    <EventRecordID>13806</EventRecordID>
    <Correlation />
    <Execution ProcessID="2088" ThreadID="4840" />
    <Channel>Application</Channel>
    <Computer>WIN-NG0HQ5HJR6U</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="string0">

Context: Application 'Search_Service_Application', Catalog 'Portal_Content'

Details:
  (0xc0041800)</Data>
  </EventData>
</Event>

 

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Server Search
Date:          3/30/2011 5:53:04 PM
Event ID:      25
Task Category: Gatherer
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      WIN-NG0HQ5HJR6U
Description:
The application cannot be initialized.

Context: Application 'Search_Service_Application'

Details:
  (0xc0041800)
Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-SharePoint Products-SharePoint Server Search" Guid="{C8263AFE-83A5-448C-878C-1E5F5D1C4252}" />
    <EventID>25</EventID>
    <Version>14</Version>
    <Level>2</Level>
    <Task>137</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2011-03-30T09:53:04.058259000Z" />
    <EventRecordID>13807</EventRecordID>
    <Correlation />
    <Execution ProcessID="2088" ThreadID="4840" />
    <Channel>Application</Channel>
    <Computer>WIN-NG0HQ5HJR6U</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="string0">

Context: Application 'Search_Service_Application'

Details:
  (0xc0041800)</Data>
  </EventData>
</Event>

Ok, so the index, seemingly "Portal_Content", is corrupt. And I cannot delete it. Where is it stored anyway? What's an unfortunate soul like me gonna do now? Reinstall SharePoint 2010 all over again? But my gut feeling was to stick with it, because there is something to be learnt from this occasion; simply reverting to a fresh installation would destroy the opportunity.

After much searching for experienced comments (which there were none), I gave up and loaded up Sysinternals procmon on the virtual server. And there began my long, long track of monitoring.

See what I see?

C:\Program Files\Microsoft Office Servers\14.0\Data\Office Server\Applications\373c2f03-0d80-42ac-930e-892b76a6c7d8\Projects\Portal_Content

That is where the index is stored. Within is an "Indexer" folder. I took that out, and let SharePoint Server Search 14 restart. No errors this time round; the index could be regenerated afresh. The Search Administration was subsequently able to display its Crawl History data grid and viewing the Content Sources was possible as well. Now we know one area of resilience SharePoint 2010 is still lacking.

Thanks, yet again, to Sysinternals.


   
Tags




Powered by Community Server, by Telligent Systems