Hi all
I have had an odd outage with my production servers and trying to work out what happened. We have two SSRS instances in a mirrored and load balanced configuration running SQL 2008 (R1) on Server 2003. They each have Dual Xeon processors with
4GB ram. Last night every report request and subscription started failing resulting in me getting called out. When I logged on to the primary server ReportingServicesService.exe was churning away at 100% CPU even though no jobs were running
at that instant. I restarted the service and CPU returned to normal and reports started being issued again.
When I looked in the SSRS log file it was full of instances the error message below which is normally thrown on a time out. However, the reports that were timing out take about 20 seconds to run in normal usage and our execution time out is set to
30 minutes for some of our longer reports. I've checked the SQL server log and the windows application and system logs and there are no other errors indicated. Does anyone have any ideas to help me work out what happened please? I've
put some log extracts below.
library!WindowsService_22!970!12/14/2014-22:05:06:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerStorageException: An error occurred within the report server database. This may be due to a connection failure, timeout
or low disk condition within the database., ;
chunks!WindowsService_22!970!12/14/2014-22:05:06:: w WARN: Rolling back shared chunk transaction for snapshot '2d19a9e3-8cae-4d23-a43f-8fdc1b23c86f', Permanent=True.
notification!WindowsService_22!970!12/14/2014-22:05:06:: e ERROR: Error occured processing notification. An error occurred within the report server database. This may be due to a connection failure, timeout or low disk condition within the database.
library!WindowsService_22!ec8!12/15/2014-00:05:08:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerStorageException: An error occurred within the report server database. This may be due to a connection failure, timeout
or low disk condition within the database., ;
chunks!WindowsService_22!ec8!12/15/2014-00:05:08:: w WARN: Rolling back shared chunk transaction for snapshot 'de3cc1ba-d468-4078-9906-321401804aae', Permanent=True.
library!WindowsService_22!ec8!12/15/2014-00:05:08:: i INFO: Initializing EnableExecutionLogging to 'True' as specified in Server system properties.
library!WindowsService_22!ec8!12/15/2014-00:05:08:: e ERROR: Schedule 40bd9808-bd8a-4000-96d9-42a0a1d4077e failed. Error Microsoft.ReportingServices.Diagnostics.Utilities.RSException: An error occurred within the report server database. This may be due
to a connection failure, timeout or low disk condition within the database. ---> Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerStorageException: An error occurred within the report server database. This may be due to a connection failure,
timeout or low disk condition within the database. ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Microsoft.ReportingServices.Library.InstrumentedSqlCommand.<ExecuteNonQuery>b__0()
at Microsoft.ReportingServices.Library.SqlBoundaryWithReturn`1.Invoke(Method m)
--- End of inner exception stack trace ---
at Microsoft.ReportingServices.Library.Storage.WrapAndThrowKnownExceptionTypes(Exception e)
at Microsoft.ReportingServices.Library.SqlBoundaryWithReturn`1.Invoke(Method m)
at Microsoft.ReportingServices.Library.InstrumentedSqlCommand.ExecuteNonQuery()
at Microsoft.ReportingServices.Library.CancelableSqlCommand.ExecuteNonQuery()
at Microsoft.ReportingServices.Library.SegmentChunkDbInterface.IsSegmentedChunk(Guid snapshotId, Boolean isPermanent, String chunkName, Int32 chunkType)
at Microsoft.ReportingServices.Library.SnapshotChunkStreamFactory.CreateReadStream(Guid snapshotDataID, Boolean isPermanentSnapshot, String chunkName, Int32 chunkType, Boolean rawStreamRequested, ConnectionManager connectionManager, Boolean supportReadWrite,
WriteOptions updateMode, Boolean isForUpgrade, String& mimeType)
at Microsoft.ReportingServices.Library.ServerSnapshot.GetChunk(String name, ReportChunkTypes type, Boolean supportReadWrite, WriteOptions updateMode, String& mimeType)
at Microsoft.ReportingServices.Library.ServerSnapshot.GetChunk(String name, ReportChunkTypes type, String& mimeType)
at Microsoft.ReportingServices.Library.ReportSnapshot.Microsoft.ReportingServices.ReportProcessing.IChunkFactory.GetChunk(String chunkName, ReportChunkTypes type, ChunkMode mode, String& mimeType)
at Microsoft.ReportingServices.Library.ReadOnlyChunkFactory.GetChunk(String chunkName, ReportChunkTypes type, ChunkMode mode, String& mimeType)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.DeserializeKatmaiReport(IChunkFactory chunkFactory, String chunkName, Boolean keepReferences, GlobalIDOwnerCollection globalIDOwnerCollection, IDOwner parentIDOwner, ReportItem parentReportItem,
Boolean processUsingCachedData)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.DeserializeKatmaiReport(IChunkFactory chunkFactory)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.GetAllDataSources(CatalogItemContext reportContext, IChunkFactory getCompiledDefinitionFactory, OnDemandSubReportDataSourcesCallback subReportCallback, DataSourceInfoCollection dataSources,
Boolean checkIfUsable, ServerDataSourceSettings serverDatasourceSettings, Boolean& needsUpgrade)
at Microsoft.ReportingServices.Library.RSService.ProcessingGetAllDataSources(ReportProcessing repProc, CatalogItemContext reportContext, ReportSnapshot intermediateSnapshot, DataSourceInfoCollection thisReportDataSources, Boolean checkIfUsable)
at Microsoft.ReportingServices.Library.RSService.GetAllDataSources(ReportProcessing repProc, BaseReportCatalogItem report, ReportSnapshot intermediateSnapshot, Guid reportID, Boolean checkIfUsable)
at Microsoft.ReportingServices.Library.RSService.GetAllDataSources(BaseReportCatalogItem report, Boolean checkIfUsable, Boolean useServiceConnectionForRepublishing, ReportSnapshot& compiledDefinition)
at Microsoft.ReportingServices.Library.RSReportContext.RetrieveAllDataSources(Boolean checkIfUsable, Boolean userServiceConnectionForRepublishing)
at Microsoft.ReportingServices.Library.CreateSnapshotExecutor.CreateSnapshot(Boolean unattended, Boolean permanent, Boolean isForHistory)
at Microsoft.ReportingServices.Library.UpdateExecutionSnapshotAction.InternalUpdateSnapshot()
at Microsoft.ReportingServices.Library.UpdateExecutionSnapshotCancelableStep.Execute()
at Microsoft.ReportingServices.Diagnostics.CancelablePhaseBase.ExecuteWrapper()
--- End of inner exception stack trace ---
at Microsoft.ReportingServices.Diagnostics.CancelablePhaseBase.ExecuteWrapper()
at Microsoft.ReportingServices.Library.UpdateExecutionSnapshotAction.PerformActionNow()
at Microsoft.ReportingServices.Library.RSSoapAction`1.Execute()
at Microsoft.ReportingServices.Library.ReportExecutionSnapshotScheduleEventHandler.PerformActionHandler(ICatalogQuery catalogQuery, ArrayList reportActions)
at Microsoft.ReportingServices.Library.ScheduleFireEventHandlerBase.HandleScheduleEvent(ICatalogQuery catalogQuery, String eventData, PerformEventActions handleFunction)