Max TCP connections

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Max TCP connections

John Quick
Does anyone know what the practical limit is for the maximum number of TCP
connections to OpenSIPS?
It is a question that often comes up with clients who are developing
SIP-based apps for smart phones. Persistent TCP seems to be strongly
favoured over UDP for connections with mobile devices such as iPhone, iPad,
etc. However, I cannot even get OpenSIPS to start if I set
tcp_max_connections to a number greater than about 55000. Have tried it with
v1.7 and 1.8. Is there a way to overcome this ceiling? Has anyone tested it
with 10's of thousands of TCP connections?

John Quick
Smartvox Limited




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Max TCP connections

Ali Pey
I am also interested in this. Any responds/updates?

Regards,
Ali Pey

On Wed, Sep 12, 2012 at 9:34 AM, John Quick <[hidden email]> wrote:
Does anyone know what the practical limit is for the maximum number of TCP
connections to OpenSIPS?
It is a question that often comes up with clients who are developing
SIP-based apps for smart phones. Persistent TCP seems to be strongly
favoured over UDP for connections with mobile devices such as iPhone, iPad,
etc. However, I cannot even get OpenSIPS to start if I set
tcp_max_connections to a number greater than about 55000. Have tried it with
v1.7 and 1.8. Is there a way to overcome this ceiling? Has anyone tested it
with 10's of thousands of TCP connections?

John Quick
Smartvox Limited




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Max TCP connections

Muhammad Shahzad
Every thing is a file in UNIX / Linux, so every TCP connection is a file too. Also persistent connects are sometime memory intensive too. So make sure adjust following things carefully.

1. Number of Open Files Limit, use either ulimit -n <number> in opensips startup script or set open_files_limit in global section on opensips dial plan. I usually use second options and set open_files_limit to 1024000.
2. Increase TCP children, tcp_children, default is i think 8, increase it to 16 or even 32 if you have very large deployment, this will speed up connection processing significantly, however consume much more memory. Also make sure to set all other TCP related opensips variables to reasonable values, such as tcp_connect_timeout, tcp_connection_lifetime etc. etc. Look here for details,


3. If you have a dedicated SIP server, running only OpenSIPs then reduce stack size as well. I found setting it to 1024KB or even less gives good results.

Now some practical results. On Dell server R510 with 8GB RAM and 2.4GHz quad core Intel processor I was able to test 104,981 concurrent SIP registrations with load average between 1.0 to 1.5.

Two important things to remember.

1. What is estimated / expected number of requests per second?
2. What is SIP re-register time?

For me, i tested up to 350 requests per second with re-register time as 300 seconds this gives theoretically possible concurrent SIP connections to 350 x 300 = 105,000. Which means you can't have more then this number of concurrent connections. So i can safely say per my testing that opensips can practically handle up to ~100K SIP concurrent connections.

Thank you.


On Fri, Sep 28, 2012 at 7:21 PM, Ali Pey <[hidden email]> wrote:
I am also interested in this. Any responds/updates?

Regards,
Ali Pey


On Wed, Sep 12, 2012 at 9:34 AM, John Quick <[hidden email]> wrote:
Does anyone know what the practical limit is for the maximum number of TCP
connections to OpenSIPS?
It is a question that often comes up with clients who are developing
SIP-based apps for smart phones. Persistent TCP seems to be strongly
favoured over UDP for connections with mobile devices such as iPhone, iPad,
etc. However, I cannot even get OpenSIPS to start if I set
tcp_max_connections to a number greater than about 55000. Have tried it with
v1.7 and 1.8. Is there a way to overcome this ceiling? Has anyone tested it
with 10's of thousands of TCP connections?

John Quick
Smartvox Limited




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




--
Muhammad Shahzad
-----------------------------------
CISCO Rich Media Communication Specialist (CRMCS)
CISCO Certified Network Associate (CCNA)
Cell: +92 334 422 40 88
MSN: [hidden email]
Email: [hidden email]


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Max TCP connections

Ali Pey
Thanks Muhammad. These are great information. Thank you sharing it with us.

Two follow up questions:
1- What tool did you use to send registers over TCP and keep connections open?
2- What did you set tcp_max_connections to in opensips?

Regards,
Ali Pey


On Sat, Sep 29, 2012 at 10:59 AM, Muhammad Shahzad <[hidden email]> wrote:
Every thing is a file in UNIX / Linux, so every TCP connection is a file too. Also persistent connects are sometime memory intensive too. So make sure adjust following things carefully.

1. Number of Open Files Limit, use either ulimit -n <number> in opensips startup script or set open_files_limit in global section on opensips dial plan. I usually use second options and set open_files_limit to 1024000.
2. Increase TCP children, tcp_children, default is i think 8, increase it to 16 or even 32 if you have very large deployment, this will speed up connection processing significantly, however consume much more memory. Also make sure to set all other TCP related opensips variables to reasonable values, such as tcp_connect_timeout, tcp_connection_lifetime etc. etc. Look here for details,


3. If you have a dedicated SIP server, running only OpenSIPs then reduce stack size as well. I found setting it to 1024KB or even less gives good results.

Now some practical results. On Dell server R510 with 8GB RAM and 2.4GHz quad core Intel processor I was able to test 104,981 concurrent SIP registrations with load average between 1.0 to 1.5.

Two important things to remember.

1. What is estimated / expected number of requests per second?
2. What is SIP re-register time?

For me, i tested up to 350 requests per second with re-register time as 300 seconds this gives theoretically possible concurrent SIP connections to 350 x 300 = 105,000. Which means you can't have more then this number of concurrent connections. So i can safely say per my testing that opensips can practically handle up to ~100K SIP concurrent connections.

Thank you.


On Fri, Sep 28, 2012 at 7:21 PM, Ali Pey <[hidden email]> wrote:
I am also interested in this. Any responds/updates?

Regards,
Ali Pey


On Wed, Sep 12, 2012 at 9:34 AM, John Quick <[hidden email]> wrote:
Does anyone know what the practical limit is for the maximum number of TCP
connections to OpenSIPS?
It is a question that often comes up with clients who are developing
SIP-based apps for smart phones. Persistent TCP seems to be strongly
favoured over UDP for connections with mobile devices such as iPhone, iPad,
etc. However, I cannot even get OpenSIPS to start if I set
tcp_max_connections to a number greater than about 55000. Have tried it with
v1.7 and 1.8. Is there a way to overcome this ceiling? Has anyone tested it
with 10's of thousands of TCP connections?

John Quick
Smartvox Limited




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




--
Muhammad Shahzad
-----------------------------------
CISCO Rich Media Communication Specialist (CRMCS)
CISCO Certified Network Associate (CCNA)
Cell: <a href="tel:%2B92%20334%20422%2040%2088" value="+923344224088" target="_blank">+92 334 422 40 88
MSN: [hidden email]
Email: [hidden email]


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Max TCP connections

Muhammad Shahzad

For udp i used sipp and for tcp i used libnet-sip-perl. I think tcp max connections was 819200, max open files was 1024000. Also i used 6 machines to produce this load. The test lasted 30 minutes.

Thank you.

On Oct 1, 2012 6:13 PM, "Ali Pey" <[hidden email]> wrote:
>
> Thanks Muhammad. These are great information. Thank you sharing it with us.
>
> Two follow up questions:
> 1- What tool did you use to send registers over TCP and keep connections open?
> 2- What did you set tcp_max_connections to in opensips?
>
> Regards,
> Ali Pey
>
>
> On Sat, Sep 29, 2012 at 10:59 AM, Muhammad Shahzad <[hidden email]> wrote:
>>
>> Every thing is a file in UNIX / Linux, so every TCP connection is a file too. Also persistent connects are sometime memory intensive too. So make sure adjust following things carefully.
>>
>> 1. Number of Open Files Limit, use either ulimit -n <number> in opensips startup script or set open_files_limit in global section on opensips dial plan. I usually use second options and set open_files_limit to 1024000.
>> 2. Increase TCP children, tcp_children, default is i think 8, increase it to 16 or even 32 if you have very large deployment, this will speed up connection processing significantly, however consume much more memory. Also make sure to set all other TCP related opensips variables to reasonable values, such as tcp_connect_timeout, tcp_connection_lifetime etc. etc. Look here for details,
>>
>> http://www.opensips.org/Resources/DocsCoreFcn18#toc75
>>
>> 3. If you have a dedicated SIP server, running only OpenSIPs then reduce stack size as well. I found setting it to 1024KB or even less gives good results.
>>
>> Now some practical results. On Dell server R510 with 8GB RAM and 2.4GHz quad core Intel processor I was able to test 104,981 concurrent SIP registrations with load average between 1.0 to 1.5.
>>
>> Two important things to remember.
>>
>> 1. What is estimated / expected number of requests per second?
>> 2. What is SIP re-register time?
>>
>> For me, i tested up to 350 requests per second with re-register time as 300 seconds this gives theoretically possible concurrent SIP connections to 350 x 300 = 105,000. Which means you can't have more then this number of concurrent connections. So i can safely say per my testing that opensips can practically handle up to ~100K SIP concurrent connections.
>>
>> Thank you.
>>
>>
>> On Fri, Sep 28, 2012 at 7:21 PM, Ali Pey <[hidden email]> wrote:
>>>
>>> I am also interested in this. Any responds/updates?
>>>
>>> Regards,
>>> Ali Pey
>>>
>>>
>>> On Wed, Sep 12, 2012 at 9:34 AM, John Quick <[hidden email]> wrote:
>>>>
>>>> Does anyone know what the practical limit is for the maximum number of TCP
>>>> connections to OpenSIPS?
>>>> It is a question that often comes up with clients who are developing
>>>> SIP-based apps for smart phones. Persistent TCP seems to be strongly
>>>> favoured over UDP for connections with mobile devices such as iPhone, iPad,
>>>> etc. However, I cannot even get OpenSIPS to start if I set
>>>> tcp_max_connections to a number greater than about 55000. Have tried it with
>>>> v1.7 and 1.8. Is there a way to overcome this ceiling? Has anyone tested it
>>>> with 10's of thousands of TCP connections?
>>>>
>>>> John Quick
>>>> Smartvox Limited
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> [hidden email]
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> [hidden email]
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>
>>
>>
>> --
>> Muhammad Shahzad
>> -----------------------------------
>> CISCO Rich Media Communication Specialist (CRMCS)
>> CISCO Certified Network Associate (CCNA)
>> Cell: +92 334 422 40 88
>> MSN: [hidden email]
>> Email: [hidden email]
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Max TCP connections

John Quick
In reply to this post by Ali Pey
Hi Ali,

Inspired by that feedback from Muhammad (thanks for that), I just did some
more testing.
It is possible to set the value of tcp_max_connections to a large number
(e.g. 200000) but only if you increase the setting for “private memory per
process” above the default value of 1 Mb. In version 1.8.x you can do this
with a command line parameter –M. In older versions you must edit the
config.h file and rebuild from source as described here:
http://www.opensips.org/Resources/DocsTsInm

My tests suggest that a value of 8 Mb is enough to let you set
tcp_max_connections=200000.

So if you are using v1.8 the command line to start opensips could be like
this:
opensips -M 8 -m 128

-M sets the Private memory and  -m sets the Shared memory. I also used the
open_files_limit to a value slightly greater than the value of
tcp_max_connections.
I have been using SIPp to test it but have not yet run a full test to
anywhere near "max connections" limit.

John Quick
Smartvox Limited


From: Ali Pey [mailto:[hidden email]]
Sent: 01 October 2012 17:13
To: OpenSIPS users mailling list
Cc: [hidden email]
Subject: Re: [OpenSIPS-Users] Max TCP connections

Thanks Muhammad. These are great information. Thank you sharing it with us.

Two follow up questions:
1- What tool did you use to send registers over TCP and keep connections
open?
2- What did you set tcp_max_connections to in opensips?

Regards,
Ali Pey

On Sat, Sep 29, 2012 at 10:59 AM, Muhammad Shahzad
<[hidden email]> wrote:
Every thing is a file in UNIX / Linux, so every TCP connection is a file
too. Also persistent connects are sometime memory intensive too. So make
sure adjust following things carefully.

1. Number of Open Files Limit, use either ulimit -n <number> in opensips
startup script or set open_files_limit in global section on opensips dial
plan. I usually use second options and set open_files_limit to 1024000.
2. Increase TCP children, tcp_children, default is i think 8, increase it to
16 or even 32 if you have very large deployment, this will speed up
connection processing significantly, however consume much more memory. Also
make sure to set all other TCP related opensips variables to reasonable
values, such as tcp_connect_timeout, tcp_connection_lifetime etc. etc. Look
here for details,

http://www.opensips.org/Resources/DocsCoreFcn18#toc75

3. If you have a dedicated SIP server, running only OpenSIPs then reduce
stack size as well. I found setting it to 1024KB or even less gives good
results.

Now some practical results. On Dell server R510 with 8GB RAM and 2.4GHz quad
core Intel processor I was able to test 104,981 concurrent SIP registrations
with load average between 1.0 to 1.5.

Two important things to remember.

1. What is estimated / expected number of requests per second?
2. What is SIP re-register time?

For me, i tested up to 350 requests per second with re-register time as 300
seconds this gives theoretically possible concurrent SIP connections to 350
x 300 = 105,000. Which means you can't have more then this number of
concurrent connections. So i can safely say per my testing that opensips can
practically handle up to ~100K SIP concurrent connections.

Thank you.

On Fri, Sep 28, 2012 at 7:21 PM, Ali Pey <[hidden email]> wrote:
I am also interested in this. Any responds/updates?

Regards,
Ali Pey

On Wed, Sep 12, 2012 at 9:34 AM, John Quick <[hidden email]>
wrote:
Does anyone know what the practical limit is for the maximum number of TCP
connections to OpenSIPS?
It is a question that often comes up with clients who are developing
SIP-based apps for smart phones. Persistent TCP seems to be strongly
favoured over UDP for connections with mobile devices such as iPhone, iPad,
etc. However, I cannot even get OpenSIPS to start if I set
tcp_max_connections to a number greater than about 55000. Have tried it with
v1.7 and 1.8. Is there a way to overcome this ceiling? Has anyone tested it
with 10's of thousands of TCP connections?

John Quick
Smartvox Limited




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




--
Muhammad Shahzad
-----------------------------------
CISCO Rich Media Communication Specialist (CRMCS)
CISCO Certified Network Associate (CCNA)
Cell: +92 334 422 40 88
MSN: [hidden email]
Email: [hidden email]


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users