auto scaling

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

auto scaling

Slava Bendersky-2
Hello Everyone,
Trying test auto scaling feature with following config

auto_scaling_profile = PRF_SIP
     scale up to 24 on 70% for 4 cycles within 5
     scale down to 4 on 20% for 8 cycles

On box 4096 Mb memory and 4 virtual CPU

Getting in log 

CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues


Not clear how to calculate workers for tcp or udp

tcp_workers=10 use_auto_scaling_profile PRF_SIP    
listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP


[root@vprx00 ~]# opensips -V
version: opensips 3.0.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 10:06:47 Mar  7 2019 with gcc 8


volga629

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: auto scaling

Bogdan-Andrei Iancu-2
Hi Slava,

Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ?

Also I just pushed a small commit to improve the date provided by the bug log.

In regards to the "not clear how to calculate"...could you be specific ?

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/10/2019 05:12 PM, Slava Bendersky wrote:
Hello Everyone,
Trying test auto scaling feature with following config

auto_scaling_profile = PRF_SIP
     scale up to 24 on 70% for 4 cycles within 5
     scale down to 4 on 20% for 8 cycles

On box 4096 Mb memory and 4 virtual CPU

Getting in log 

CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues


Not clear how to calculate workers for tcp or udp

tcp_workers=10 use_auto_scaling_profile PRF_SIP    
listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP


[root@vprx00 ~]# opensips -V
version: opensips 3.0.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 10:06:47 Mar  7 2019 with gcc 8


volga629


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: auto scaling

Slava Bendersky-2
Hello Bogdan,
My question how to calculate correctly 

tcp_workers
udp_workers

to properly assign with in  auto scaling profile. Which parameters need use as base ?

As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers.
It message is appear in log on first attempt to fork in scale up.

volga629 
   

From: "Bogdan-Andrei Iancu" <[hidden email]>
To: "OpenSIPS users mailling list" <[hidden email]>, "Slava Bendersky" <[hidden email]>
Sent: Monday, March 11, 2019 4:44:57 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Hi Slava,

Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ?

Also I just pushed a small commit to improve the date provided by the bug log.

In regards to the "not clear how to calculate"...could you be specific ?

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/10/2019 05:12 PM, Slava Bendersky wrote:
Hello Everyone,
Trying test auto scaling feature with following config

auto_scaling_profile = PRF_SIP
     scale up to 24 on 70% for 4 cycles within 5
     scale down to 4 on 20% for 8 cycles

On box 4096 Mb memory and 4 virtual CPU

Getting in log 

CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues


Not clear how to calculate workers for tcp or udp

tcp_workers=10 use_auto_scaling_profile PRF_SIP    
listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP


[root@vprx00 ~]# opensips -V
version: opensips 3.0.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 10:06:47 Mar  7 2019 with gcc 8


volga629


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: auto scaling

Bogdan-Andrei Iancu-2
Slava,

Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error.

Thanks,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/12/2019 05:51 AM, Slava Bendersky wrote:
Hello Bogdan,
My question how to calculate correctly 

tcp_workers
udp_workers

to properly assign with in  auto scaling profile. Which parameters need use as base ?

As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers.
It message is appear in log on first attempt to fork in scale up.

volga629 
   

From: "Bogdan-Andrei Iancu" [hidden email]
To: "OpenSIPS users mailling list" [hidden email], "Slava Bendersky" [hidden email]
Sent: Monday, March 11, 2019 4:44:57 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Hi Slava,

Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ?

Also I just pushed a small commit to improve the date provided by the bug log.

In regards to the "not clear how to calculate"...could you be specific ?

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/10/2019 05:12 PM, Slava Bendersky wrote:
Hello Everyone,
Trying test auto scaling feature with following config

auto_scaling_profile = PRF_SIP
     scale up to 24 on 70% for 4 cycles within 5
     scale down to 4 on 20% for 8 cycles

On box 4096 Mb memory and 4 virtual CPU

Getting in log 

CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues


Not clear how to calculate workers for tcp or udp

tcp_workers=10 use_auto_scaling_profile PRF_SIP    
listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP


[root@vprx00 ~]# opensips -V
version: opensips 3.0.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 10:06:47 Mar  7 2019 with gcc 8


volga629


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: auto scaling

Slava Bendersky-2
I will do pull today.

volga629

From: "Bogdan-Andrei Iancu" <[hidden email]>
To: "Slava Bendersky" <[hidden email]>
Cc: "OpenSIPS users mailling list" <[hidden email]>
Sent: Tuesday, March 12, 2019 4:52:06 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Slava,

Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error.

Thanks,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/12/2019 05:51 AM, Slava Bendersky wrote:
Hello Bogdan,
My question how to calculate correctly 

tcp_workers
udp_workers

to properly assign with in  auto scaling profile. Which parameters need use as base ?

As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers.
It message is appear in log on first attempt to fork in scale up.

volga629 
   

From: "Bogdan-Andrei Iancu" [hidden email]
To: "OpenSIPS users mailling list" [hidden email], "Slava Bendersky" [hidden email]
Sent: Monday, March 11, 2019 4:44:57 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Hi Slava,

Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ?

Also I just pushed a small commit to improve the date provided by the bug log.

In regards to the "not clear how to calculate"...could you be specific ?

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/10/2019 05:12 PM, Slava Bendersky wrote:
Hello Everyone,
Trying test auto scaling feature with following config

auto_scaling_profile = PRF_SIP
     scale up to 24 on 70% for 4 cycles within 5
     scale down to 4 on 20% for 8 cycles

On box 4096 Mb memory and 4 virtual CPU

Getting in log 

CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues


Not clear how to calculate workers for tcp or udp

tcp_workers=10 use_auto_scaling_profile PRF_SIP    
listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP


[root@vprx00 ~]# opensips -V
version: opensips 3.0.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 10:06:47 Mar  7 2019 with gcc 8


volga629


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: auto scaling

Slava Bendersky-2
In reply to this post by Slava Bendersky-2
Hello Bogdan,
After we doing more testing with auto scaling we observed issue with TCP, where opensips try scale down the processes it tear down them and break TCP connections with end points and that cause blocked connection state where opensips stops process TCP connections.

volga629 

From: "Slava Bendersky" <[hidden email]>
To: "Bogdan-Andrei Iancu" <[hidden email]>
Cc: "OpenSIPS users mailling list" <[hidden email]>
Sent: Wednesday, March 13, 2019 12:21:14 AM
Subject: Re: [OpenSIPS-Users] auto scaling

I will do pull today.

volga629

From: "Bogdan-Andrei Iancu" <[hidden email]>
To: "Slava Bendersky" <[hidden email]>
Cc: "OpenSIPS users mailling list" <[hidden email]>
Sent: Tuesday, March 12, 2019 4:52:06 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Slava,

Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error.

Thanks,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/12/2019 05:51 AM, Slava Bendersky wrote:
Hello Bogdan,
My question how to calculate correctly 

tcp_workers
udp_workers

to properly assign with in  auto scaling profile. Which parameters need use as base ?

As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers.
It message is appear in log on first attempt to fork in scale up.

volga629 
   

From: "Bogdan-Andrei Iancu" [hidden email]
To: "OpenSIPS users mailling list" [hidden email], "Slava Bendersky" [hidden email]
Sent: Monday, March 11, 2019 4:44:57 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Hi Slava,

Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ?

Also I just pushed a small commit to improve the date provided by the bug log.

In regards to the "not clear how to calculate"...could you be specific ?

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/10/2019 05:12 PM, Slava Bendersky wrote:
Hello Everyone,
Trying test auto scaling feature with following config

auto_scaling_profile = PRF_SIP
     scale up to 24 on 70% for 4 cycles within 5
     scale down to 4 on 20% for 8 cycles

On box 4096 Mb memory and 4 virtual CPU

Getting in log 

CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues


Not clear how to calculate workers for tcp or udp

tcp_workers=10 use_auto_scaling_profile PRF_SIP    
listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP


[root@vprx00 ~]# opensips -V
version: opensips 3.0.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 10:06:47 Mar  7 2019 with gcc 8


volga629


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: auto scaling

Bogdan-Andrei Iancu-2
Slava,

What about the upscaling error you originally reported ? is there any way to reproduce it ? do you do auto-scaling only for TCP (no UDP,timer) ?

Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ? Is is a proper close of the connection ? or ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/14/2019 02:14 PM, Slava Bendersky wrote:
Hello Bogdan,
After we doing more testing with auto scaling we observed issue with TCP, where opensips try scale down the processes it tear down them and break TCP connections with end points and that cause blocked connection state where opensips stops process TCP connections.

volga629 

From: "Slava Bendersky" [hidden email]
To: "Bogdan-Andrei Iancu" [hidden email]
Cc: "OpenSIPS users mailling list" [hidden email]
Sent: Wednesday, March 13, 2019 12:21:14 AM
Subject: Re: [OpenSIPS-Users] auto scaling

I will do pull today.

volga629

From: "Bogdan-Andrei Iancu" [hidden email]
To: "Slava Bendersky" [hidden email]
Cc: "OpenSIPS users mailling list" [hidden email]
Sent: Tuesday, March 12, 2019 4:52:06 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Slava,

Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error.

Thanks,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/12/2019 05:51 AM, Slava Bendersky wrote:
Hello Bogdan,
My question how to calculate correctly 

tcp_workers
udp_workers

to properly assign with in  auto scaling profile. Which parameters need use as base ?

As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers.
It message is appear in log on first attempt to fork in scale up.

volga629 
   

From: "Bogdan-Andrei Iancu" [hidden email]
To: "OpenSIPS users mailling list" [hidden email], "Slava Bendersky" [hidden email]
Sent: Monday, March 11, 2019 4:44:57 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Hi Slava,

Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ?

Also I just pushed a small commit to improve the date provided by the bug log.

In regards to the "not clear how to calculate"...could you be specific ?

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/10/2019 05:12 PM, Slava Bendersky wrote:
Hello Everyone,
Trying test auto scaling feature with following config

auto_scaling_profile = PRF_SIP
     scale up to 24 on 70% for 4 cycles within 5
     scale down to 4 on 20% for 8 cycles

On box 4096 Mb memory and 4 virtual CPU

Getting in log 

CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues


Not clear how to calculate workers for tcp or udp

tcp_workers=10 use_auto_scaling_profile PRF_SIP    
listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP


[root@vprx00 ~]# opensips -V
version: opensips 3.0.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 10:06:47 Mar  7 2019 with gcc 8


volga629


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: auto scaling

Slava Bendersky-2
In reply to this post by Slava Bendersky-2
Hello Bogdan,
Upscaling error is didn't show up any more in logs at least I didn't so it. Might need trigger bigger load.

do you do auto-scaling only for TCP ?

Our  setup in most  multi homed, as example public side TCP private side UDP


Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ?

Yes, that how is look like end result that opensips start blocking TCP connections. If specific process  handle sip registration via tcp and opensips trying close to down scale end point need send re register in order do not loose registration. Other wise it cause stale connections. Just close properly connection not enough I think.


volga629





From: "Bogdan-Andrei Iancu" <[hidden email]>
To: "OpenSIPS users mailling list" <[hidden email]>, "Slava Bendersky" <[hidden email]>
Sent: Thursday, March 14, 2019 1:45:20 PM
Subject: Re: [OpenSIPS-Users] auto scaling

Slava,

What about the upscaling error you originally reported ? is there any way to reproduce it ? do you do auto-scaling only for TCP (no UDP,timer) ?

Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ? Is is a proper close of the connection ? or ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/14/2019 02:14 PM, Slava Bendersky wrote:
Hello Bogdan,
After we doing more testing with auto scaling we observed issue with TCP, where opensips try scale down the processes it tear down them and break TCP connections with end points and that cause blocked connection state where opensips stops process TCP connections.

volga629 


From: "Slava Bendersky" [hidden email]
To: "Bogdan-Andrei Iancu" [hidden email]
Cc: "OpenSIPS users mailling list" [hidden email]
Sent: Wednesday, March 13, 2019 12:21:14 AM
Subject: Re: [OpenSIPS-Users] auto scaling

I will do pull today.

volga629

From: "Bogdan-Andrei Iancu" [hidden email]
To: "Slava Bendersky" [hidden email]
Cc: "OpenSIPS users mailling list" [hidden email]
Sent: Tuesday, March 12, 2019 4:52:06 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Slava,

Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error.

Thanks,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/12/2019 05:51 AM, Slava Bendersky wrote:
Hello Bogdan,
My question how to calculate correctly 

tcp_workers
udp_workers

to properly assign with in  auto scaling profile. Which parameters need use as base ?

As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers.
It message is appear in log on first attempt to fork in scale up.

volga629 
   

From: "Bogdan-Andrei Iancu" [hidden email]
To: "OpenSIPS users mailling list" [hidden email], "Slava Bendersky" [hidden email]
Sent: Monday, March 11, 2019 4:44:57 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Hi Slava,

Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ?

Also I just pushed a small commit to improve the date provided by the bug log.

In regards to the "not clear how to calculate"...could you be specific ?

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/10/2019 05:12 PM, Slava Bendersky wrote:
Hello Everyone,
Trying test auto scaling feature with following config

auto_scaling_profile = PRF_SIP
     scale up to 24 on 70% for 4 cycles within 5
     scale down to 4 on 20% for 8 cycles

On box 4096 Mb memory and 4 virtual CPU

Getting in log 

CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues


Not clear how to calculate workers for tcp or udp

tcp_workers=10 use_auto_scaling_profile PRF_SIP    
listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP


[root@vprx00 ~]# opensips -V
version: opensips 3.0.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 10:06:47 Mar  7 2019 with gcc 8


volga629


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: auto scaling

Bogdan-Andrei Iancu-2
Hi Slava,

OK, I will need to run more tests on the TCP conns stuff - the intended behavior is *NOT* to close the connections handled by the process to be terminated.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/15/2019 12:47 PM, Slava Bendersky wrote:
Hello Bogdan,
Upscaling error is didn't show up any more in logs at least I didn't so it. Might need trigger bigger load.

do you do auto-scaling only for TCP ?

Our  setup in most  multi homed, as example public side TCP private side UDP


Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ?

Yes, that how is look like end result that opensips start blocking TCP connections. If specific process  handle sip registration via tcp and opensips trying close to down scale end point need send re register in order do not loose registration. Other wise it cause stale connections. Just close properly connection not enough I think.


volga629





From: "Bogdan-Andrei Iancu" [hidden email]
To: "OpenSIPS users mailling list" [hidden email], "Slava Bendersky" [hidden email]
Sent: Thursday, March 14, 2019 1:45:20 PM
Subject: Re: [OpenSIPS-Users] auto scaling

Slava,

What about the upscaling error you originally reported ? is there any way to reproduce it ? do you do auto-scaling only for TCP (no UDP,timer) ?

Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ? Is is a proper close of the connection ? or ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/14/2019 02:14 PM, Slava Bendersky wrote:
Hello Bogdan,
After we doing more testing with auto scaling we observed issue with TCP, where opensips try scale down the processes it tear down them and break TCP connections with end points and that cause blocked connection state where opensips stops process TCP connections.

volga629 


From: "Slava Bendersky" [hidden email]
To: "Bogdan-Andrei Iancu" [hidden email]
Cc: "OpenSIPS users mailling list" [hidden email]
Sent: Wednesday, March 13, 2019 12:21:14 AM
Subject: Re: [OpenSIPS-Users] auto scaling

I will do pull today.

volga629

From: "Bogdan-Andrei Iancu" [hidden email]
To: "Slava Bendersky" [hidden email]
Cc: "OpenSIPS users mailling list" [hidden email]
Sent: Tuesday, March 12, 2019 4:52:06 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Slava,

Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error.

Thanks,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/12/2019 05:51 AM, Slava Bendersky wrote:
Hello Bogdan,
My question how to calculate correctly 

tcp_workers
udp_workers

to properly assign with in  auto scaling profile. Which parameters need use as base ?

As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers.
It message is appear in log on first attempt to fork in scale up.

volga629 
   

From: "Bogdan-Andrei Iancu" [hidden email]
To: "OpenSIPS users mailling list" [hidden email], "Slava Bendersky" [hidden email]
Sent: Monday, March 11, 2019 4:44:57 AM
Subject: Re: [OpenSIPS-Users] auto scaling

Hi Slava,

Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ?

Also I just pushed a small commit to improve the date provided by the bug log.

In regards to the "not clear how to calculate"...could you be specific ?

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/10/2019 05:12 PM, Slava Bendersky wrote:
Hello Everyone,
Trying test auto scaling feature with following config

auto_scaling_profile = PRF_SIP
     scale up to 24 on 70% for 4 cycles within 5
     scale down to 4 on 20% for 8 cycles

On box 4096 Mb memory and 4 virtual CPU

Getting in log 

CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues


Not clear how to calculate workers for tcp or udp

tcp_workers=10 use_auto_scaling_profile PRF_SIP    
listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP


[root@vprx00 ~]# opensips -V
version: opensips 3.0.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 10:06:47 Mar  7 2019 with gcc 8


volga629


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users