Quantcast

Mediaproxy hanging sessions on high load

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Mediaproxy hanging sessions on high load

Daniel Zanutti
I'm using mediaproxy on several installations and have noticed that when the machine is on high load (> 700 sessions), the media-relay process starts to hang some sessions.

These sessions doesn't have any RTP being sent/received anymore and they never hangup. After some hours of frozen sessions, the media-relay process doesn't connect to the dispatcher anymore, but keep using high CPU on the machine. Maybe it's on loop internally, not sure.

Is there any solution for this? Maybe a timer to cleanup old sessions (2 or 4+ hours old).

Thanks



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mediaproxy hanging sessions on high load

Daniel Zanutti
Any idea guys?

On Tue, Mar 7, 2017 at 11:10 AM, Daniel Zanutti <[hidden email]> wrote:
I'm using mediaproxy on several installations and have noticed that when the machine is on high load (> 700 sessions), the media-relay process starts to hang some sessions.

These sessions doesn't have any RTP being sent/received anymore and they never hangup. After some hours of frozen sessions, the media-relay process doesn't connect to the dispatcher anymore, but keep using high CPU on the machine. Maybe it's on loop internally, not sure.

Is there any solution for this? Maybe a timer to cleanup old sessions (2 or 4+ hours old).

Thanks




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mediaproxy hanging sessions on high load

Daniel Zanutti
In reply to this post by Daniel Zanutti
Hi guys

I sent this email a few days ago, anyone from Mediaproxy team could take a look? I could debug it, just need some directions on where to look.

Thanks

On Tue, Mar 7, 2017 at 11:10 AM, Daniel Zanutti <[hidden email]> wrote:
I'm using mediaproxy on several installations and have noticed that when the machine is on high load (> 700 sessions), the media-relay process starts to hang some sessions.

These sessions doesn't have any RTP being sent/received anymore and they never hangup. After some hours of frozen sessions, the media-relay process doesn't connect to the dispatcher anymore, but keep using high CPU on the machine. Maybe it's on loop internally, not sure.

Is there any solution for this? Maybe a timer to cleanup old sessions (2 or 4+ hours old).

Thanks




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mediaproxy hanging sessions on high load

Dan Pascu

On 13 Mar 2017, at 18:58, Daniel Zanutti wrote:

> Hi guys
>
> I sent this email a few days ago, anyone from Mediaproxy team could take a look? I could debug it, just need some directions on where to look.

We have never encountered this problem, so I', not sure what to suggest, even more considering that the description is not very clear. What do you mean when you say the relay starts to hang some sessions? That it timeouts on them not having traffic and initiates a BYE for those sessions? Because in the next paragraph you imply that they never timeout.

>
> Thanks
>
> On Tue, Mar 7, 2017 at 11:10 AM, Daniel Zanutti <[hidden email]> wrote:
> I'm using mediaproxy on several installations and have noticed that when the machine is on high load (> 700 sessions), the media-relay process starts to hang some sessions.
>
> These sessions doesn't have any RTP being sent/received anymore and they never hangup. After some hours of frozen sessions, the media-relay process doesn't connect to the dispatcher anymore, but keep using high CPU on the machine. Maybe it's on loop internally, not sure.
>
> Is there any solution for this? Maybe a timer to cleanup old sessions (2 or 4+ hours old).
>
> Thanks
>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


--
Dan





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mediaproxy hanging sessions on high load

Dan Pascu

One thing came to mind. A case when the relay could get overloaded is if a lot of clients start sessions and only one endpoint sends media. That is the only case where the relay would have to deal with the media traffic itself and having hundreds of such sessions at the same time could overload the relay.

The way the relay works is for each call it starts listening on 4 ports (2 for RTP and 2 for RTCP). Each endpoint will send 2 streams (1 RTP one RTCP) and initially the relay will just listen on these ports and when it receives data it learns the endpoint's address. After it learns both endpoint's addresses, it adds a conntrack rule in the kernel to allow the kernel to directly relay the media streams between the endpoints and it will never see a media packet from the endpoints again until the call ends. This allows for very efficient data forwarding because it's done entirely in the kernel with no data being transferred from kernel to user-space and back like traditional solutions. We have seen media relays handling hundreds of calls at a time with 0% CPU load on the relay.

So the only thing I can think of causing something like what you describe (even though I'm still not sure what you meant by hanging up sessions), is that somehow this process didn't finish setting up completely and the relay directly receives media streams from hundreds of devices because only one endpoint sends data (or the other endpoint's data gets filtered at some firewall), and because it cannot learn both endpoint's addresses it cannot setup the kernel conntrack rule to move data forwarding to the kernel.

On 14 Mar 2017, at 13:38, Dan Pascu wrote:

>
> On 13 Mar 2017, at 18:58, Daniel Zanutti wrote:
>
>> Hi guys
>>
>> I sent this email a few days ago, anyone from Mediaproxy team could take a look? I could debug it, just need some directions on where to look.
>
> We have never encountered this problem, so I', not sure what to suggest, even more considering that the description is not very clear. What do you mean when you say the relay starts to hang some sessions? That it timeouts on them not having traffic and initiates a BYE for those sessions? Because in the next paragraph you imply that they never timeout.
>
>>
>> Thanks
>>
>> On Tue, Mar 7, 2017 at 11:10 AM, Daniel Zanutti <[hidden email]> wrote:
>> I'm using mediaproxy on several installations and have noticed that when the machine is on high load (> 700 sessions), the media-relay process starts to hang some sessions.
>>
>> These sessions doesn't have any RTP being sent/received anymore and they never hangup. After some hours of frozen sessions, the media-relay process doesn't connect to the dispatcher anymore, but keep using high CPU on the machine. Maybe it's on loop internally, not sure.
>>
>> Is there any solution for this? Maybe a timer to cleanup old sessions (2 or 4+ hours old).
>>
>> Thanks
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> --
> Dan
>
>
>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


--
Dan





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mediaproxy hanging sessions on high load

Daniel Zanutti
Hi Dan

Looks like this problem is only happening on virtual machines, not on physical machines. And only while they are on high load. 

But i'm not sure about the kernel rule, is there any way to check it? 

Please take a look at this case, this Relay will never halt because there are more than 3k sessions that will never finish internally (the call has already hangup hours ago):

82.2.2.22.6.144h01'05"
112.03kbps3045
audio 3045
Halting

Some of these calls:












728From: 222222@4.4.4.4
To: [hidden email]
unknown agentHG4000/1.06.6.6.6:556322.2.2.2:466402.2.2.2:468667.7.7.7:4170activeG729audio21h35'34"00
729From: 2222222@4.4.4.4:5064
To: 33333333@sip.aaa.com.br
TS-v4.6.0-11eWAgitel GSM Bridge v2.06.6.6.6:349082.2.2.2:581582.2.2.2:543727.7.7.7:16846activeG729audio16h11'51"00
730From: 22222222@4.4.4.4
To: 33333333@sip.aaa.com.br
Mediant 2000/v.6.60A.328.003unknown agent6.6.6.6:463242.2.2.2:501562.2.2.2:481827.7.7.7:18516activeG729audio19h45'38"00
731From: 222222@4.4.4.4:5061
To: 33333333333@sip.aaa.com.br
TS-v4.6.0-14bgsm-gw-3.4.16.6.6.6:548002.2.2.2:439982.2.2.2:461447.7.7.7:12360activeG729audio19h09'41"00
732From: [hidden email]
To: 333333333333@sip.aaa.com.br
Trinit IVRHG4000/1.06.6.6.6:188542.2.2.2:519242.2.2.2:405127.7.7.7:4200activeG729audio19h37'59"00

Is there any way to drop these sessions? Maybe using the internal timeout system of mediaproxy?

If you could take a look personally, we could negotiate an hourly rate.

Thanks again



On Thu, Mar 16, 2017 at 10:54 AM, Dan Pascu <[hidden email]> wrote:

One thing came to mind. A case when the relay could get overloaded is if a lot of clients start sessions and only one endpoint sends media. That is the only case where the relay would have to deal with the media traffic itself and having hundreds of such sessions at the same time could overload the relay.

The way the relay works is for each call it starts listening on 4 ports (2 for RTP and 2 for RTCP). Each endpoint will send 2 streams (1 RTP one RTCP) and initially the relay will just listen on these ports and when it receives data it learns the endpoint's address. After it learns both endpoint's addresses, it adds a conntrack rule in the kernel to allow the kernel to directly relay the media streams between the endpoints and it will never see a media packet from the endpoints again until the call ends. This allows for very efficient data forwarding because it's done entirely in the kernel with no data being transferred from kernel to user-space and back like traditional solutions. We have seen media relays handling hundreds of calls at a time with 0% CPU load on the relay.

So the only thing I can think of causing something like what you describe (even though I'm still not sure what you meant by hanging up sessions), is that somehow this process didn't finish setting up completely and the relay directly receives media streams from hundreds of devices because only one endpoint sends data (or the other endpoint's data gets filtered at some firewall), and because it cannot learn both endpoint's addresses it cannot setup the kernel conntrack rule to move data forwarding to the kernel.

On 14 Mar 2017, at 13:38, Dan Pascu wrote:

>
> On 13 Mar 2017, at 18:58, Daniel Zanutti wrote:
>
>> Hi guys
>>
>> I sent this email a few days ago, anyone from Mediaproxy team could take a look? I could debug it, just need some directions on where to look.
>
> We have never encountered this problem, so I', not sure what to suggest, even more considering that the description is not very clear. What do you mean when you say the relay starts to hang some sessions? That it timeouts on them not having traffic and initiates a BYE for those sessions? Because in the next paragraph you imply that they never timeout.
>
>>
>> Thanks
>>
>> On Tue, Mar 7, 2017 at 11:10 AM, Daniel Zanutti <[hidden email]> wrote:
>> I'm using mediaproxy on several installations and have noticed that when the machine is on high load (> 700 sessions), the media-relay process starts to hang some sessions.
>>
>> These sessions doesn't have any RTP being sent/received anymore and they never hangup. After some hours of frozen sessions, the media-relay process doesn't connect to the dispatcher anymore, but keep using high CPU on the machine. Maybe it's on loop internally, not sure.
>>
>> Is there any solution for this? Maybe a timer to cleanup old sessions (2 or 4+ hours old).
>>
>> Thanks
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> --
> Dan
>
>
>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


--
Dan





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mediaproxy hanging sessions on high load

Adrian Georgescu
Perhaps your virtual machine simply cannot handle the load. The commands to close sessions may also be dropped or lost under such environment.

Adrian


On 16 Mar 2017, at 11:22, Daniel Zanutti <[hidden email]> wrote:

Hi Dan

Looks like this problem is only happening on virtual machines, not on physical machines. And only while they are on high load. 

But i'm not sure about the kernel rule, is there any way to check it? 

Please take a look at this case, this Relay will never halt because there are more than 3k sessions that will never finish internally (the call has already hangup hours ago):

82.2.2.22.6.144h01'05"
112.03kbps3045
audio 3045
Halting

Some of these calls:












728From: 222222@4.4.4.4
To: [hidden email]
unknown agentHG4000/1.06.6.6.6:556322.2.2.2:466402.2.2.2:468667.7.7.7:4170activeG729audio21h35'34"00
729From: 2222222@4.4.4.4:5064
To: 33333333@sip.aaa.com.br
TS-v4.6.0-11eWAgitel GSM Bridge v2.06.6.6.6:349082.2.2.2:581582.2.2.2:543727.7.7.7:16846activeG729audio16h11'51"00
730From: 22222222@4.4.4.4
To: 33333333@sip.aaa.com.br
Mediant 2000/v.6.60A.328.003unknown agent6.6.6.6:463242.2.2.2:501562.2.2.2:481827.7.7.7:18516activeG729audio19h45'38"00
731From: 222222@4.4.4.4:5061
To: 33333333333@sip.aaa.com.br
TS-v4.6.0-14bgsm-gw-3.4.16.6.6.6:548002.2.2.2:439982.2.2.2:461447.7.7.7:12360activeG729audio19h09'41"00
732From: [hidden email]
To: 333333333333@sip.aaa.com.br
Trinit IVRHG4000/1.06.6.6.6:188542.2.2.2:519242.2.2.2:405127.7.7.7:4200activeG729audio19h37'59"00

Is there any way to drop these sessions? Maybe using the internal timeout system of mediaproxy?

If you could take a look personally, we could negotiate an hourly rate.

Thanks again



On Thu, Mar 16, 2017 at 10:54 AM, Dan Pascu <[hidden email]> wrote:

One thing came to mind. A case when the relay could get overloaded is if a lot of clients start sessions and only one endpoint sends media. That is the only case where the relay would have to deal with the media traffic itself and having hundreds of such sessions at the same time could overload the relay.

The way the relay works is for each call it starts listening on 4 ports (2 for RTP and 2 for RTCP). Each endpoint will send 2 streams (1 RTP one RTCP) and initially the relay will just listen on these ports and when it receives data it learns the endpoint's address. After it learns both endpoint's addresses, it adds a conntrack rule in the kernel to allow the kernel to directly relay the media streams between the endpoints and it will never see a media packet from the endpoints again until the call ends. This allows for very efficient data forwarding because it's done entirely in the kernel with no data being transferred from kernel to user-space and back like traditional solutions. We have seen media relays handling hundreds of calls at a time with 0% CPU load on the relay.

So the only thing I can think of causing something like what you describe (even though I'm still not sure what you meant by hanging up sessions), is that somehow this process didn't finish setting up completely and the relay directly receives media streams from hundreds of devices because only one endpoint sends data (or the other endpoint's data gets filtered at some firewall), and because it cannot learn both endpoint's addresses it cannot setup the kernel conntrack rule to move data forwarding to the kernel.

On 14 Mar 2017, at 13:38, Dan Pascu wrote:

>
> On 13 Mar 2017, at 18:58, Daniel Zanutti wrote:
>
>> Hi guys
>>
>> I sent this email a few days ago, anyone from Mediaproxy team could take a look? I could debug it, just need some directions on where to look.
>
> We have never encountered this problem, so I', not sure what to suggest, even more considering that the description is not very clear. What do you mean when you say the relay starts to hang some sessions? That it timeouts on them not having traffic and initiates a BYE for those sessions? Because in the next paragraph you imply that they never timeout.
>
>>
>> Thanks
>>
>> On Tue, Mar 7, 2017 at 11:10 AM, Daniel Zanutti <[hidden email]> wrote:
>> I'm using mediaproxy on several installations and have noticed that when the machine is on high load (> 700 sessions), the media-relay process starts to hang some sessions.
>>
>> These sessions doesn't have any RTP being sent/received anymore and they never hangup. After some hours of frozen sessions, the media-relay process doesn't connect to the dispatcher anymore, but keep using high CPU on the machine. Maybe it's on loop internally, not sure.
>>
>> Is there any solution for this? Maybe a timer to cleanup old sessions (2 or 4+ hours old).
>>
>> Thanks
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> --
> Dan
>
>
>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


--
Dan





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

signature.asc (210 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mediaproxy hanging sessions on high load

Daniel Zanutti
Adrian

You may be correct, overload can be the problem. But since the call is already finished, how can I remove from the relay? The final problem is the relay processing freezing, i need to avoid this.

Thanks

On Thu, Mar 16, 2017 at 10:40 PM, Adrian Georgescu <[hidden email]> wrote:
Perhaps your virtual machine simply cannot handle the load. The commands to close sessions may also be dropped or lost under such environment.

Adrian



On 16 Mar 2017, at 11:22, Daniel Zanutti <[hidden email]> wrote:

Hi Dan

Looks like this problem is only happening on virtual machines, not on physical machines. And only while they are on high load. 

But i'm not sure about the kernel rule, is there any way to check it? 

Please take a look at this case, this Relay will never halt because there are more than 3k sessions that will never finish internally (the call has already hangup hours ago):

82.2.2.22.6.144h01'05"
112.03kbps3045
audio 3045
Halting

Some of these calls:












728From: 222222@4.4.4.4
To: [hidden email]
unknown agentHG4000/1.06.6.6.6:556322.2.2.2:466402.2.2.2:468667.7.7.7:4170activeG729audio21h35'34"00
729From: 2222222@4.4.4.4:5064
To: 33333333@sip.aaa.com.br
TS-v4.6.0-11eWAgitel GSM Bridge v2.06.6.6.6:349082.2.2.2:581582.2.2.2:543727.7.7.7:16846activeG729audio16h11'51"00
730From: 22222222@4.4.4.4
To: 33333333@sip.aaa.com.br
Mediant 2000/v.6.60A.328.003unknown agent6.6.6.6:463242.2.2.2:501562.2.2.2:481827.7.7.7:18516activeG729audio19h45'38"00
731From: 222222@4.4.4.4:5061
To: 33333333333@sip.aaa.com.br
TS-v4.6.0-14bgsm-gw-3.4.16.6.6.6:548002.2.2.2:439982.2.2.2:461447.7.7.7:12360activeG729audio19h09'41"00
732From: [hidden email]
To: 333333333333@sip.aaa.com.br
Trinit IVRHG4000/1.06.6.6.6:188542.2.2.2:519242.2.2.2:405127.7.7.7:4200activeG729audio19h37'59"00

Is there any way to drop these sessions? Maybe using the internal timeout system of mediaproxy?

If you could take a look personally, we could negotiate an hourly rate.

Thanks again



On Thu, Mar 16, 2017 at 10:54 AM, Dan Pascu <[hidden email]> wrote:

One thing came to mind. A case when the relay could get overloaded is if a lot of clients start sessions and only one endpoint sends media. That is the only case where the relay would have to deal with the media traffic itself and having hundreds of such sessions at the same time could overload the relay.

The way the relay works is for each call it starts listening on 4 ports (2 for RTP and 2 for RTCP). Each endpoint will send 2 streams (1 RTP one RTCP) and initially the relay will just listen on these ports and when it receives data it learns the endpoint's address. After it learns both endpoint's addresses, it adds a conntrack rule in the kernel to allow the kernel to directly relay the media streams between the endpoints and it will never see a media packet from the endpoints again until the call ends. This allows for very efficient data forwarding because it's done entirely in the kernel with no data being transferred from kernel to user-space and back like traditional solutions. We have seen media relays handling hundreds of calls at a time with 0% CPU load on the relay.

So the only thing I can think of causing something like what you describe (even though I'm still not sure what you meant by hanging up sessions), is that somehow this process didn't finish setting up completely and the relay directly receives media streams from hundreds of devices because only one endpoint sends data (or the other endpoint's data gets filtered at some firewall), and because it cannot learn both endpoint's addresses it cannot setup the kernel conntrack rule to move data forwarding to the kernel.

On 14 Mar 2017, at 13:38, Dan Pascu wrote:

>
> On 13 Mar 2017, at 18:58, Daniel Zanutti wrote:
>
>> Hi guys
>>
>> I sent this email a few days ago, anyone from Mediaproxy team could take a look? I could debug it, just need some directions on where to look.
>
> We have never encountered this problem, so I', not sure what to suggest, even more considering that the description is not very clear. What do you mean when you say the relay starts to hang some sessions? That it timeouts on them not having traffic and initiates a BYE for those sessions? Because in the next paragraph you imply that they never timeout.
>
>>
>> Thanks
>>
>> On Tue, Mar 7, 2017 at 11:10 AM, Daniel Zanutti <[hidden email]> wrote:
>> I'm using mediaproxy on several installations and have noticed that when the machine is on high load (> 700 sessions), the media-relay process starts to hang some sessions.
>>
>> These sessions doesn't have any RTP being sent/received anymore and they never hangup. After some hours of frozen sessions, the media-relay process doesn't connect to the dispatcher anymore, but keep using high CPU on the machine. Maybe it's on loop internally, not sure.
>>
>> Is there any solution for this? Maybe a timer to cleanup old sessions (2 or 4+ hours old).
>>
>> Thanks
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> --
> Dan
>
>
>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


--
Dan





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mediaproxy hanging sessions on high load

Dan Pascu
In reply to this post by Daniel Zanutti

On 16 Mar 2017, at 16:22, Daniel Zanutti wrote:

> Hi Dan
>
> Looks like this problem is only happening on virtual machines, not on physical machines. And only while they are on high load.

We have recommended since forever that people run a media relays on real hardware, yet time and again people ignore that and use virtual machines. I rest my case.

>
> But i'm not sure about the kernel rule, is there any way to check it?

You can check the conntrack rules in /proc/net/ip_conntrack

>
> Please take a look at this case, this Relay will never halt because there are more than 3k sessions that will never finish internally (the call has already hangup hours ago):

Ok, so that is what you meant by "the relay starts to hangup sessions".

This is something we encountered as well, but not to the extent you mention. In our case we have some 30-50 sessions in that state after 1-2 months of usage. It seems to always be associated with some network failures where sessions are not ended properly. We do not know exactly what causes it and given it's small impact for us it was never a priority to debug it. All I know is that it seems to be correlated with dialog updates that remove a RTP stream by setting its port to 0, without ending the session.

> Is there any way to drop these sessions? Maybe using the internal timeout system of mediaproxy?

Mediaproxy already employs several timeout mechanisms, both of its own as well as leveraging the timeout mechanism the linux conntrack implementation provides for rules that receive no traffic for a while. The problem is not the lack of those, the problem seems to be that under certain conditions which we have not yet identified, those timeouts seem to be ignored.

--
Dan





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mediaproxy hanging sessions on high load

Dan Pascu
In reply to this post by Daniel Zanutti

On 17 Mar 2017, at 3:54, Daniel Zanutti wrote:

> Adrian
>
> You may be correct, overload can be the problem. But since the call is already finished, how can I remove from the relay?

One way is to issue commands to the dispatcher to end certain sessions, in the same way that opensips issues them when it receives a BYE. But this is easier said than done, because you will need to find out the call-id , from-tag and to-tag of the call in order to do that.

At some point we had the idea to add this kind of functionality to the monitoring web page allowing you to click a button next to a call to forcefully end it, but it never come to fruition.

The only thing you can do for now is make sure you have at least another relay connected to the dispatcher, so it can absorb new calls, the run /etc/init.d/media-relay stop-gracefully and wait until this relay has no more active traffic, then you can run /etc/init.d/media-relay restart to restart it.

--
Dan





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mediaproxy hanging sessions on high load

Daniel Zanutti
Hi Dan

Thanks for replying.

I understand that best case scenario is to run on a physical dedicated server, but unfortunately this is impossible on all cases and virtual machines is the only viable ($$) solution.

About the "frozen" sessions, as as workaround, I'll test dropping these sessions directly using dispatcher interface. If it works, a console will solve the problem.

The problem is that I cannot reproduce this scenario, on some clients it happens, on some I have same quantity of calls and never happens. I'll try to find a workaround on the timer.

Thanks for all help, you guys are great.


On Fri, Mar 17, 2017 at 3:08 PM, Dan Pascu <[hidden email]> wrote:

On 17 Mar 2017, at 3:54, Daniel Zanutti wrote:

> Adrian
>
> You may be correct, overload can be the problem. But since the call is already finished, how can I remove from the relay?

One way is to issue commands to the dispatcher to end certain sessions, in the same way that opensips issues them when it receives a BYE. But this is easier said than done, because you will need to find out the call-id , from-tag and to-tag of the call in order to do that.

At some point we had the idea to add this kind of functionality to the monitoring web page allowing you to click a button next to a call to forcefully end it, but it never come to fruition.

The only thing you can do for now is make sure you have at least another relay connected to the dispatcher, so it can absorb new calls, the run /etc/init.d/media-relay stop-gracefully and wait until this relay has no more active traffic, then you can run /etc/init.d/media-relay restart to restart it.

--
Dan





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Loading...