Multiple media relays on one server

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple media relays on one server

JimDoesVoip
Hi All,
  I have the impression (perhaps wrong) that we can run multiple media-relay processes on a single server and have them connect to the same media dispatcher(s).  I also have the perception that media-relay not multi-threaded.  Searched the mail list and didn't find indications that anyone was doing this.  Here are more details of what we are attempting:

The scenario is that we have 4 bare metal servers running centOS 6.  Two of these servers are running openSIPS and media-dispatcher processes.  We are using mediaproxy 2.5.2.  To take advantage of the multiple cores [each server has 4 cores] we'd like to run about the same number of media relay processes on the servers that aren't running openSIPS.  Each of the media-relay processes would connect to both dispatchers to have things be as flexible / resilient as possible.  

We are able to start multiple media-relays, run out of different directories, with different media port ranges on a single server using the --pid option.  When doing this, and connecting 2 of these media-relays to the same media-dispatcher we see errors in the logs that indicate that the media-relays are being disconnected and reconnected.  We are attempting to run the media-relays on the same IP address (well.. the media relays that are on the same server).  Do the media relays connecting to a particular dispatcher need their own IP address?  Do we need to run the media-relays on different IP addresses on the servers?  

The logs are really hinting to us that the dispatcher identifies the relays only by their IP address.  Here is a sample of the messages we are seeing (2 relays and 1 dispatcher running on a single server as a test):


Oct 19 09:52:34 bstnma-osips1-b1 media-dispatcher[31150]: warning: Relay at 10.128.66.75 reconnected, closing old connection
Oct 19 09:52:34 bstnma-osips1-b1 media-dispatcher[31150]: warning: Old connection with relay at 10.128.66.75 was lost
Oct 19 09:52:34 bstnma-osips1-b1 media-relay[31129]: error: Connection with dispatcher at 10.128.66.75:25060 was lost: A TLS packet with unexpected length was received.
Oct 19 09:52:36 bstnma-osips1-b1 media-dispatcher[31150]: warning: Relay at 10.128.66.75 reconnected, closing old connection
Oct 19 09:52:36 bstnma-osips1-b1 media-dispatcher[31150]: warning: Old connection with relay at 10.128.66.75 was lost
Oct 19 09:52:36 bstnma-osips1-b1 media-relay[31140]: error: Connection with dispatcher at 10.128.66.75:25060 was lost: A TLS packet with unexpected length was received.


Thanks,

Jim O
Reply | Threaded
Open this post in threaded view
|

Re: Multiple media relays on one server

Saúl Ibarra Corretgé
Hi Jim,

On Oct 19, 2011, at 4:08 PM, JimDoesVoip wrote:

> Hi All,
>  I have the impression (perhaps wrong) that we can run multiple media-relay
> processes on a single server and have them connect to the same media
> dispatcher(s).  I also have the perception that media-relay not
> multi-threaded.  Searched the mail list and didn't find indications that
> anyone was doing this.  Here are more details of what we are attempting:
>
> The scenario is that we have 4 bare metal servers running centOS 6.  Two of
> these servers are running openSIPS and media-dispatcher processes.  We are
> using mediaproxy 2.5.2.  To take advantage of the multiple cores [each
> server has 4 cores] we'd like to run about the same number of media relay
> processes on the servers that aren't running openSIPS.  Each of the
> media-relay processes would connect to both dispatchers to have things be as
> flexible / resilient as possible.  
>

You don't need to do that with MediaProxy. MediaProxy is single threaded (uses Twisted) and it will open 2 pairs of ports for each connection. When *a single* RTP packet was received from both sides, a conntrack rule is created in kernel space, so no packets are relayed in userspace anymore.

Thus, you'll not get better relaying performance for running multiple instances, the kernel will do the relaying for you.

You can run the relay in the same host as the dispatcher, btw.

> We are able to start multiple media-relays, run out of different
> directories, with different media port ranges on a single server using the
> --pid option.  When doing this, and connecting 2 of these media-relays to
> the same media-dispatcher we see errors in the logs that indicate that the
> media-relays are being disconnected and reconnected.  We are attempting to
> run the media-relays on the same IP address (well.. the media relays that
> are on the same server).  Do the media relays connecting to a particular
> dispatcher need their own IP address?  Do we need to run the media-relays on
> different IP addresses on the servers?  
>

The --pid option is not enough. You need different configuration, different port range and a different IP address. The dispatcher maps the relays by IP address, so you can't have two on the same. What you see in the logs is the dispatcher being confused by two connections from the same IP, which he thinks is a single relay.


Regards,

--
Saúl Ibarra Corretgé
AG Projects




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Multiple media relays on one server

osiris123d
I think I remember reading somewhere that they only thing you really care about on a Mediaproxy server is the network interface... Processor and memory doesn't matter.

On , Saúl Ibarra Corretgé <[hidden email]> wrote:

> Hi Jim,
>
>
>
> On Oct 19, 2011, at 4:08 PM, JimDoesVoip wrote:
>
>
>
> > Hi All,
>
> >  I have the impression (perhaps wrong) that we can run multiple media-relay
>
> > processes on a single server and have them connect to the same media
>
> > dispatcher(s).  I also have the perception that media-relay not
>
> > multi-threaded.  Searched the mail list and didn't find indications that
>
> > anyone was doing this.  Here are more details of what we are attempting:
>
> >
>
> > The scenario is that we have 4 bare metal servers running centOS 6.  Two of
>
> > these servers are running openSIPS and media-dispatcher processes.  We are
>
> > using mediaproxy 2.5.2.  To take advantage of the multiple cores [each
>
> > server has 4 cores] we'd like to run about the same number of media relay
>
> > processes on the servers that aren't running openSIPS.  Each of the
>
> > media-relay processes would connect to both dispatchers to have things be as
>
> > flexible / resilient as possible.
>
> >
>
>
>
> You don't need to do that with MediaProxy. MediaProxy is single threaded (uses Twisted) and it will open 2 pairs of ports for each connection. When *a single* RTP packet was received from both sides, a conntrack rule is created in kernel space, so no packets are relayed in userspace anymore.
>
>
>
> Thus, you'll not get better relaying performance for running multiple instances, the kernel will do the relaying for you.
>
>
>
> You can run the relay in the same host as the dispatcher, btw.
>
>
>
> > We are able to start multiple media-relays, run out of different
>
> > directories, with different media port ranges on a single server using the
>
> > --pid option.  When doing this, and connecting 2 of these media-relays to
>
> > the same media-dispatcher we see errors in the logs that indicate that the
>
> > media-relays are being disconnected and reconnected.  We are attempting to
>
> > run the media-relays on the same IP address (well.. the media relays that
>
> > are on the same server).  Do the media relays connecting to a particular
>
> > dispatcher need their own IP address?  Do we need to run the media-relays on
>
> > different IP addresses on the servers?
>
> >
>
>
>
> The --pid option is not enough. You need different configuration, different port range and a different IP address. The dispatcher maps the relays by IP address, so you can't have two on the same. What you see in the logs is the dispatcher being confused by two connections from the same IP, which he thinks is a single relay.
>
>
>
>
>
> Regards,
>
>
>
> --
>
> Saúl Ibarra Corretgé
>
> AG Projects
>
>
>
>
>
>
>
>
>
> _______________________________________________
>
> Users mailing list
>
> [hidden email]
>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Multiple media relays on one server

JimDoesVoip
In reply to this post by Saúl Ibarra Corretgé
Hi Saúl,

Thanks for the detailed reply.  Perhaps only to get the idea of running more than one media relay /server out of my system:  

Is there any reason (performance gain / resiliency / other?) to run 2 different instances of media-relay on an individual server, each connecting to different/independent dispatchers?  I assume the two relay processes here could share a single IP because they aren't connecting to the same dispatcher?

Sad to say that I'm disappointed that I won't be using those extra cores :)  Perhaps it speaks to the elegance of the mediaproxy implementation.

Thanks again,

Jim O







_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Reply | Threaded
Open this post in threaded view
|

Re: Multiple media relays on one server

Saúl Ibarra Corretgé
Hi,

On Oct 19, 2011, at 7:16 PM, JimDoesVoip wrote:

> Hi Saúl,
>
> Thanks for the detailed reply.  Perhaps only to get the idea of running more
> than one media relay /server out of my system:  
>
> Is there any reason (performance gain / resiliency / other?) to run 2
> different instances of media-relay on an individual server, each connecting
> to different/independent dispatchers?  I assume the two relay processes here
> could share a single IP because they aren't connecting to the same
> dispatcher?
>

No, there is no reason to do that unless you'd want different IP addresses for relaying the traffic. A single relay may connect to any amount of dispatchers, so, for resilience, you can make those 2 relays connect simultaneously to the 2 dispatchers.

> Sad to say that I'm disappointed that I won't be using those extra cores :)
> Perhaps it speaks to the elegance of the mediaproxy implementation.
>

Actually I think you will. We pass the control to the Linux kernel, so it's up to her to sue those cores. It's an iptables thing, so I guess it uses multiple cores, but I can't say with 100% certainty.

Regards,

--
Saúl Ibarra Corretgé
AG Projects




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Multiple media relays on one server

Jeff Pyle
In reply to this post by JimDoesVoip
Perhaps slightly off topic, but would running irqbalance spread out the interrupts associated with forwarding all those packets across the multiple processing cores available?


- Jeff


On Oct 19, 2011, at 1:16 PM, JimDoesVoip wrote:

> Hi Saúl,
>
> Thanks for the detailed reply.  Perhaps only to get the idea of running more
> than one media relay /server out of my system:  
>
> Is there any reason (performance gain / resiliency / other?) to run 2
> different instances of media-relay on an individual server, each connecting
> to different/independent dispatchers?  I assume the two relay processes here
> could share a single IP because they aren't connecting to the same
> dispatcher?
>
> Sad to say that I'm disappointed that I won't be using those extra cores :)
> Perhaps it speaks to the elegance of the mediaproxy implementation.
>
> Thanks again,
>
> Jim O
>
>
>
>
>
>
>
> _______________________________________________
> Users mailing list
> Users@.opensips
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> --
> View this message in context: http://opensips-open-sip-server.1449251.n2.nabble.com/Multiple-media-relays-on-one-server-tp6908831p6909602.html
> Sent from the OpenSIPS - Users mailing list archive at Nabble.com.
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Multiple media relays on one server

JimDoesVoip
Hi Jeff, All, This seems(ed) quite interesting reading through the documentation (http://irqbalance.org/documentation.html). I then ran a command I saw in this writeup on irqbalance: http://kb.fusionio.com/KB/a65/irqbalance-avoid-overloading-cpu-0-with-interrupt-requests.aspx cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       
>>snip other things
  77:    3441350    3442277    3440052    3437999   IO-APIC-fasteoi   eth0
  78:    2900004    2899359    2901479    2903668   IO-APIC-fasteoi   eth1
>>snip other things
It appears the interrupts for our bonded gige interfaces are already being spread across all the cores quite evenly. Perhaps some of this has been better integrated into the kernel. This is CentOS 6. Thanks, Jim O
Jeff Pyle wrote
Perhaps slightly off topic, but would running irqbalance spread out the interrupts associated with forwarding all those packets across the multiple processing cores available? - Jeff _______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Multiple media relays on one server

JimDoesVoip
One more data point... I tried the same on one of our RedHat EL 5.5 servers, same hardware configuration, and things look quite different.

cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
>>SNIP
193:        167 1775740922        145     301880   IO-APIC-level  eth0
201:          1  347154062      61887      23650   IO-APIC-level  eth1
>>SNIP