Missing TFS Alerts

One day our TFS server stopped sending out e-mail alerts.

We first checked and rechecked our Email Alert Settings, but they were all in order.

Our next guess was our server’s access to the SMTP server had been blocked (we are in an environment where only certain machines are allowed to send e-mail messages via SMTP). But we ruled that out when we were able to send out e-mail messages with a simple test application from that server.

After much looking around on the internet and on the TFS server, we came across this error in the Tfs_Configuration.tbl_JobHistory table:

TF400797: Job extension had an unhandled error: Microsoft.TeamFoundation.Framework.Server.IdentityNotFoundException:
TF14045: The identity with TeamFoundationId 6a358919-xxxx-xxxx-xxxx-2c2f342dc379 could not be found.
at Microsoft.TeamFoundation.Framework.Server.Subscription.AfterReadSubscription(TeamFoundationRequestContext requestContext)
at Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension.ExpandEvents(TeamFoundationRequestContext requestContext)
at Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime jobQueueTime, String& resultMessage)
at Microsoft.TeamFoundation.Framework.Server.JobRunner.ExecuteJob()

We used the following SQL query to find this:

SELECT TOP 1000 [HistoryId]

      ,[JobSource]

      ,[JobId]

      ,[QueueTime]

      ,[StartTime]

      ,[EndTime]

      ,[AgentId]

      ,[Result]

      ,[ResultMessage]

      ,[QueuedReasons]

      ,[QueueFlags]

      ,[Priority]

  FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]

  WHERE ResultMessage IS NOT NULL

  and StartTime > (GETDATE() - 1)

  and Result = 2

  ORDER BY EndTime desc

We looked up the user associated with the GUID listed in the error message and discovered it was related to a user who had left the project a few months earlier. Their user account had been deleted from the Windows domain.

The following TFS bug appears to document the issue we were encountering: https://connect.microsoft.com/VisualStudio/feedback/details/779506/all-tfs-e-mail-notifications-stop-because-of-any-exception-regarding-a-user-alert-or-user-identity (link is no longer valid)

Using the GUID from the error message and the SQL query below, we were able to find the alert subscription that was causing our problems:

select *
from Tfs_PharmacyTechnology.dbo.tbl_EventSubscription
where SubscriberId='6A358919-xxxx-xxxx-xxxx-2C2F342DC379'

Using the id field from the results of the query above, we were able to delete the offending subscription using the BisSubscribe.exe command shown below:

> BisSubscribe.exe /unsubscribe /id {id from query above} /collection http://{TFS Server Name}:8080/tfs/{TFS Collection Name}

Once the bad alert subscription was deleted, the other e-mail alert notifications started to be sent.