Lync 2010 response group members phones keep ringing even if the remote party hangs up the phone
In this blog post, I’ll be talking about a response group problem where the response group members phones keep ringing even if the user calling from pstn side hangs up the phone. I would like to talk about how I troubleshooted this issue:
We first collected ETL traces and network traces from the mediation server to better understand the problem (actually mediation server and frontend server were running on the same machine). Some more details about the response group:
Response group number: 8900
Response group members: user1@contoso.com / user2@contoso.com / user3@contoso.com / user4@contoso.com / user5@contoso.com
In the S4 ETL trace, we can see that once the call is made to response group number, response group application starts ringing each response group member with 15 seconds intervals while it keeps playing music on hold to the caller:
Since none of the response group members answer, response group application goes through each member in a loop fashion. (actually the exact behavior here is identified by response group charateristics like workflow, routing method etc)
Interestingly, even the user who initiated the call to response group hangs up the phone, the rtp traffic from voip gateway keeps coming towards the mediation server:
98312 2012-11-23 08:59:05.383 0.015 172.16.1.7 172.17.1.100RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x3FF62C47, Seq=19584, Time=964130050
98313 2012-11-23 08:59:05.388 0.004 172.17.1.100172.16.1.7 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x6ED4D5BE, Seq=33285, Time=1945495369
98320 2012-11-23 08:59:05.403 0.015 172.16.1.7 172.17.1.100RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x3FF62C47, Seq=19585, Time=964130210
98322 2012-11-23 08:59:05.408 0.004 172.17.1.100172.16.1.7 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x6ED4D5BE, Seq=33286, Time=1945495529
98323 2012-11-23 08:59:05.423 0.015 172.16.1.7 172.17.1.100RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x3FF62C47, Seq=19586, Time=964130370
98324 2012-11-23 08:59:05.428 0.004 172.17.1.100172.16.1.7 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x6ED4D5BE, Seq=33287, Time=1945495689
98326 2012-11-23 08:59:05.443 0.015 172.16.1.7 172.17.1.100RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x3FF62C47, Seq=19587, Time=964130530
So we should have got a SIP BYE from VoIP gateway side when the remote caller hung up the phone...
This looks like a VoIP gateway side problem. We then made some internal testing to verify the expected behavior. Here are the results from the internal testing:
Expected behavior:
===============
Note: The test response group has one member only so the response group application keeps ringing the same user
- The remote party calls the Response group number
- Response group answers the call and then it starts ringing the response group member (joe@contoso.com)
- Since the agent doesn’t pick up the phone for 15 seconds, it drops that session with a SIP CANCEL to the group member and it starts another session with the same agent (because there’s only one agent)
- The person calling from behind the VoIP gateway hangs up the phone after 23 seconds or so after the initial call
- So the second dialog with the agent is terminated
- The second call to the agent is made at 13:36:17
- The remote caller hangs up the phone at 13:36:22 (23 seconds after the response group answers the call)
- And the mediation server gets a SIP BYE from VoIP gateway since the remote caller hangs up the phone:
TL_INFO(TF_PROTOCOL) [0]0C4C.1410::11/23/2012-13:36:22.847.0002b91f (S4,SipMessage.DataLoggingHelper:sipmessage.cs(686))[3311788563]
<<<<<<<<<<<<Incoming SipMessage c=[<SipTlsConnection_386DD23>], 192.168.1.20:5071<-192.168.1.20:50807
BYE sip:napool.contoso.com:5071;grid;ms-fe=nafe02.contoso.com SIP/2.0
FROM: <sip:18006934501;phone-context=DefaultProfile@contoso.com;user=phone>;epid=8D443A6D50;tag=928547717d
TO: <sip:17042563016;phone-context=DefaultProfile@contoso.com;user=phone>;epid=14B9F3DDE9;tag=872d3a4ad0
CSEQ: 3828 BYE
CALL-ID: d9789202-c026-4661-88d8-adfc12c37995
MAX-FORWARDS: 69
VIA: SIP/2.0/TLS 192.168.1.20:50807;branch=z9hG4bK6FF51B0B.C998AF162BD50886;branched=FALSE
VIA: SIP/2.0/TLS 192.168.1.15:54616;branch=z9hG4bK97e1a6aa;ms-received-port=54616;ms-received-cid=726C00
CONTACT: <sip:nafe01.contoso.com@contoso.com;gruu;opaque=srvr:MediationServer:gJSXgywGXl2uoBnZnm4R7QAA;grid=1792a8d837324735acd180c81887b8b0>;isGateway
CONTENT-LENGTH: 0
SUPPORTED: ms-dialog-route-set-update
SUPPORTED: gruu-10
USER-AGENT: RTCC/4.0.0.0 MediationServer
P-ASSERTED-IDENTITY: "PBX 01"<sip:18006934501;phone-context=DefaultProfile@contoso.com;user=phone>
ms-diagnostics: 10027;source="nafe01.contoso.com";reason="Normal termination response from gateway after the call was established";component="MediationServer"
ms-diagnostics-public: 10027;reason="Normal termination response from gateway after the call was established";component="MediationServer"
ms-endpoint-location-data: NetworkScope;ms-media-location-type=intranet
------------EndOfIncoming SipMessage
Right after the SIP BYE is received from VoIP gateway, response group stops ringing the agent with a SIP CANCEL request:
We can see the same behavior in network trace as well:
=> The call starts and RTP traffic starts flowing:
=> At 15:36:23, VoIP gateway (Asteriks PBX) sends a SIP BYE to mediation server since the remote caller hangs up the phone, and the RTP traffic also stops at this point:
Summary:
========
We advised our customer to further check their VoIP gateway to better understand why it doesn’t send a SIP BYE to mediation server when the remote caller hangs up the phone (and because the SIP BYE is not received, response group member phones keep ringing)
Hope this helps
Thanks,
Murat