Issues with mediaproxy-dispatcher

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Issues with mediaproxy-dispatcher

Koldo Aingeru Marcos Fernández
Hi everyone,

we are starting to have some strange issues with mediaproxy. Our setup is kamailio 4.2 + mediaproxy 2.6.1 on CentOS 6.6.

Recently we have two incidents were the communication between the dispatcher and the relays (we got two of them, one in the same server) get lost, digging in the logs we see this:

Jun 25 13:07:22 kamailio media-dispatcher[14459]: session with call_id [hidden email]:5060 from relay AAA.AAA.AAA.AAA did timeout
Jun 25 13:07:22 kamailio media-dispatcher[14459]: Traceback (most recent call last):
Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
Jun 25 13:07:22 kamailio media-dispatcher[14459]:     why = selectable.doRead()
Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/gnutls/interfaces/twisted/__init__.py", line 79, in doRead
Jun 25 13:07:22 kamailio media-dispatcher[14459]:     return tcp.Connection.doRead(self)
Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/internet/tcp.py", line 214, in doRead
Jun 25 13:07:22 kamailio media-dispatcher[14459]:     return self._dataReceived(data)
Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/internet/tcp.py", line 220, in _dataReceived
Jun 25 13:07:22 kamailio media-dispatcher[14459]:     rval = self.protocol.dataReceived(data)
Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/protocols/basic.py", line 454, in dataReceived
Jun 25 13:07:22 kamailio media-dispatcher[14459]:     self.lineReceived(line)
Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/mediaproxy/dispatcher.py", line 239, in lineReceived
Jun 25 13:07:22 kamailio media-dispatcher[14459]:     session.expire_time = time()
Jun 25 13:07:22 kamailio media-dispatcher[14459]: TypeError: 'module' object is not callable
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Connection with relay at AAA.AAA.AAA.AAA was lost: 'module' object is not callable
Jun 25 13:07:22 kamailio media-relay[19712]: error: Connection with dispatcher at AAA.AAA.AAA.AAA:25060 was lost: A TLS packet with unexpected length was received.
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: failed to end dialog: 404 Requested Dialog not found
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
Jun 25 13:07:22 kamailio media-dispatcher[14459]: warning: user requested media_relay AAA.AAA.AAA.AAA is not available
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: No suitable relay found
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
Jun 25 13:07:22 kamailio media-dispatcher[14459]: warning: user requested media_relay AAA.AAA.AAA.AAA is not available
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: No suitable relay found
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected

And after that we had to restart the dispatcher. The connections to the management interface stopped working too.

We don't have more calls or traffic than other days and the resources on the machines seems fine (this is a "top" when the incident happened, I hope that doesn't get too much distorted in the mail):

top - 10:19:53 up 30 days,  9:34,  7 users,  load average: 0.07, 0.05, 0.02
Tasks: 288 total,   2 running, 286 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.8%us,  0.3%sy,  0.0%ni, 97.7%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  16315980k total,   970248k used, 15345732k free,    59004k buffers
Swap:  4194300k total,    57876k used,  4136424k free,   455560k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                        
18896 root      20   0  261m  29m 1956 S 14.6  0.2   6:58.93 media-relay                                                                                                                     
18903 root      20   0  411m  24m 1992 S  7.3  0.2   2:48.80 media-dispatcher                                                                                                                 
25921 root      20   0  244m 5228  404 S  1.3  0.0 210:41.31 rsyslogd                                                                                                                        
18856 root      20   0 1198m  38m  33m S  0.7  0.2   0:11.34 kamailio                                                                                                                        
18860 root      20   0 1198m  31m  26m S  0.7  0.2   0:11.36 kamailio                                                                                                                        
31712 root      20   0  192m 8296 8184 S  0.7  0.1 108:03.22 captagent                                                                                                                       
18846 root      20   0 1198m  31m  26m S  0.3  0.2   0:11.68 kamailio                                                                                                                        
18847 root      20   0 1198m  38m  33m S  0.3  0.2   0:11.38 kamailio    

 
We are starting to suspect that maybe the old python version that Centos 6 uses or from other packages (twisted?) may be the cause but we don't discard other options (accouting, some sysctl fine tunning option...)

Any ideas would be appreciated.
 
Thanks!

========================================
Koldo Aingeru Marcos Fdez.  
Ingeniería de Sistemas
Sarenet S.A.
944209470
Parque Tecnológico, Edificio 103
48170 Zamudio, Bizkaia
========================================





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Issues with mediaproxy-dispatcher

Dan Pascu

This exception cannot happen unless you modified the mediaproxy code yourself and made a mistake or if the python interpreter you use is seriously broken (unlikely).

On 25 Jun 2015, at 17:26, Koldo Aingeru Marcos Fernández wrote:

> Hi everyone,
>
> we are starting to have some strange issues with mediaproxy. Our setup is kamailio 4.2 + mediaproxy 2.6.1 on CentOS 6.6.
>
> Recently we have two incidents were the communication between the dispatcher and the relays (we got two of them, one in the same server) get lost, digging in the logs we see this:
>
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: session with call_id [hidden email]:5060 from relay AAA.AAA.AAA.AAA did timeout
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: Traceback (most recent call last):
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     why = selectable.doRead()
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/gnutls/interfaces/twisted/__init__.py", line 79, in doRead
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     return tcp.Connection.doRead(self)
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/internet/tcp.py", line 214, in doRead
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     return self._dataReceived(data)
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/internet/tcp.py", line 220, in _dataReceived
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     rval = self.protocol.dataReceived(data)
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/protocols/basic.py", line 454, in dataReceived
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     self.lineReceived(line)
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/mediaproxy/dispatcher.py", line 239, in lineReceived
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     session.expire_time = time()
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: TypeError: 'module' object is not callable
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Connection with relay at AAA.AAA.AAA.AAA was lost: 'module' object is not callable

--
Dan





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Issues with mediaproxy-dispatcher

Saúl Ibarra Corretgé
In reply to this post by Koldo Aingeru Marcos Fernández
As Dan pointed out, that’s the result of your local changes. Since you submitted them to me, I had a quick look, and in media proxy/dispatcher.py you changed “from time import time” to “import time”. Since no other change was made to that file, I assume it was a mistake, reverting that should make it Just Work.


Cheers,

> On 25 Jun 2015, at 16:26, Koldo Aingeru Marcos Fernández <[hidden email]> wrote:
>
> Hi everyone,
>
> we are starting to have some strange issues with mediaproxy. Our setup is kamailio 4.2 + mediaproxy 2.6.1 on CentOS 6.6.
>
> Recently we have two incidents were the communication between the dispatcher and the relays (we got two of them, one in the same server) get lost, digging in the logs we see this:
>
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: session with call_id [hidden email]:5060 from relay AAA.AAA.AAA.AAA did timeout
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: Traceback (most recent call last):
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     why = selectable.doRead()
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/gnutls/interfaces/twisted/__init__.py", line 79, in doRead
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     return tcp.Connection.doRead(self)
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/internet/tcp.py", line 214, in doRead
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     return self._dataReceived(data)
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/internet/tcp.py", line 220, in _dataReceived
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     rval = self.protocol.dataReceived(data)
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/twisted/protocols/basic.py", line 454, in dataReceived
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     self.lineReceived(line)
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:   File "/usr/lib64/python2.6/site-packages/mediaproxy/dispatcher.py", line 239, in lineReceived
> Jun 25 13:07:22 kamailio media-dispatcher[14459]:     session.expire_time = time()
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: TypeError: 'module' object is not callable
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Connection with relay at AAA.AAA.AAA.AAA was lost: 'module' object is not callable
> Jun 25 13:07:22 kamailio media-relay[19712]: error: Connection with dispatcher at AAA.AAA.AAA.AAA:25060 was lost: A TLS packet with unexpected length was received.
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: failed to end dialog: 404 Requested Dialog not found
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: warning: user requested media_relay AAA.AAA.AAA.AAA is not available
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: No suitable relay found
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: warning: user requested media_relay AAA.AAA.AAA.AAA is not available
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: No suitable relay found
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
> Jun 25 13:07:22 kamailio media-dispatcher[14459]: error: Relay for this session (AAA.AAA.AAA.AAA) is no longer connected
>
> And after that we had to restart the dispatcher. The connections to the management interface stopped working too.
>
> We don't have more calls or traffic than other days and the resources on the machines seems fine (this is a "top" when the incident happened, I hope that doesn't get too much distorted in the mail):
>
> top - 10:19:53 up 30 days,  9:34,  7 users,  load average: 0.07, 0.05, 0.02
> Tasks: 288 total,   2 running, 286 sleeping,   0 stopped,   0 zombie
> Cpu(s):  1.8%us,  0.3%sy,  0.0%ni, 97.7%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
> Mem:  16315980k total,   970248k used, 15345732k free,    59004k buffers
> Swap:  4194300k total,    57876k used,  4136424k free,   455560k cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 18896 root      20   0  261m  29m 1956 S 14.6  0.2   6:58.93 media-relay
> 18903 root      20   0  411m  24m 1992 S  7.3  0.2   2:48.80 media-dispatcher
> 25921 root      20   0  244m 5228  404 S  1.3  0.0 210:41.31 rsyslogd
> 18856 root      20   0 1198m  38m  33m S  0.7  0.2   0:11.34 kamailio
> 18860 root      20   0 1198m  31m  26m S  0.7  0.2   0:11.36 kamailio
> 31712 root      20   0  192m 8296 8184 S  0.7  0.1 108:03.22 captagent
> 18846 root      20   0 1198m  31m  26m S  0.3  0.2   0:11.68 kamailio
> 18847 root      20   0 1198m  38m  33m S  0.3  0.2   0:11.38 kamailio
>
>
> We are starting to suspect that maybe the old python version that Centos 6 uses or from other packages (twisted?) may be the cause but we don't discard other options (accouting, some sysctl fine tunning option...)
>
> Any ideas would be appreciated.
>
> Thanks!
>
> ========================================
> Koldo Aingeru Marcos Fdez.
> Ingeniería de Sistemas
> Sarenet S.A.
> 944209470
> Parque Tecnológico, Edificio 103
> 48170 Zamudio, Bizkaia
> www.sarenet.es
> ========================================
>
>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
--
Saúl Ibarra Corretgé
AG Projects




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

signature.asc (859 bytes) Download Attachment