opensips 2.2.5 stop responding/processing requests. - Centos 6.5

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

opensips 2.2.5 stop responding/processing requests. - Centos 6.5

Jonathan Hunter

Hi Guys,

Has anyone had the following behaviour with this release of opensips?

It has been running fine for a number of months, and then suddenly today the server which was running as the primary suddenly stopped responding to SIP requests, or stopped processing them, or there was a long pause in a sequence completing.

So in some cases, INVITES or REGISTER messages were ignored and the initial requests werent always logged in the opensips log file (even though we do this for all messages), however the SIP requests were hitting the server as I was tracing them.

In the logs I can see the following warnings we havent seen before;

Oct 30 13:31:25 sgw6 VU-SIP-Proxy[11159]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 9639891650 ms (now 9639891850 ms), it may overlap.
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <nh-timer> already scheduled for 9640994180 ms (now 9641175280 ms), it may overlap..
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <ul-timer> already scheduled for 9640980270 ms (now 9641175280 ms), it may overlap..

A restart of the opensips application didnt help, and service was only restored fully when I failed over to the secondary server.

Has anyone had these issues before? There are no obvious server level issues, and as I mentioned the logs dont contain anything too unusual.

Any help appreciated.

Many thanks

Jon


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips 2.2.5 stop responding/processing requests. - Centos 6.5

Johan De Clercq
If you use acc with radius, check radius config.

Br

On 30 Oct 2017 18:33, "Jonathan Hunter" <[hidden email]> wrote:

Hi Guys,

Has anyone had the following behaviour with this release of opensips?

It has been running fine for a number of months, and then suddenly today the server which was running as the primary suddenly stopped responding to SIP requests, or stopped processing them, or there was a long pause in a sequence completing.

So in some cases, INVITES or REGISTER messages were ignored and the initial requests werent always logged in the opensips log file (even though we do this for all messages), however the SIP requests were hitting the server as I was tracing them.

In the logs I can see the following warnings we havent seen before;

Oct 30 13:31:25 sgw6 VU-SIP-Proxy[11159]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 9639891650 ms (now 9639891850 ms), it may overlap.
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <nh-timer> already scheduled for 9640994180 ms (now 9641175280 ms), it may overlap..
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <ul-timer> already scheduled for 9640980270 ms (now 9641175280 ms), it may overlap..

A restart of the opensips application didnt help, and service was only restored fully when I failed over to the secondary server.

Has anyone had these issues before? There are no obvious server level issues, and as I mentioned the logs dont contain anything too unusual.

Any help appreciated.

Many thanks

Jon


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips 2.2.5 stop responding/processing requests. - Centos 6.5

Jonathan Hunter

Hi Johan,


Thanks for the response I appreciate it.


We do use radius for accounting and fire all the information to a central server, but it appears all that was fine and connectivity/configuration was good at the time of the issue.


Anywhere in particular it could break things? I have seen it when opensips cant reach radius it starts to be none responsive, is that a scenario you have seen?


Many thanks


Jon




From: Users <[hidden email]> on behalf of Johan De Clercq <[hidden email]>
Sent: 30 October 2017 17:53
To: OpenSIPS users mailling list
Subject: Re: [OpenSIPS-Users] opensips 2.2.5 stop responding/processing requests. - Centos 6.5
 
If you use acc with radius, check radius config.

Br

On 30 Oct 2017 18:33, "Jonathan Hunter" <[hidden email]> wrote:

Hi Guys,

Has anyone had the following behaviour with this release of opensips?

It has been running fine for a number of months, and then suddenly today the server which was running as the primary suddenly stopped responding to SIP requests, or stopped processing them, or there was a long pause in a sequence completing.

So in some cases, INVITES or REGISTER messages were ignored and the initial requests werent always logged in the opensips log file (even though we do this for all messages), however the SIP requests were hitting the server as I was tracing them.

In the logs I can see the following warnings we havent seen before;

Oct 30 13:31:25 sgw6 VU-SIP-Proxy[11159]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 9639891650 ms (now 9639891850 ms), it may overlap.
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <nh-timer> already scheduled for 9640994180 ms (now 9641175280 ms), it may overlap..
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <ul-timer> already scheduled for 9640980270 ms (now 9641175280 ms), it may overlap..

A restart of the opensips application didnt help, and service was only restored fully when I failed over to the secondary server.

Has anyone had these issues before? There are no obvious server level issues, and as I mentioned the logs dont contain anything too unusual.

Any help appreciated.

Many thanks

Jon


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips 2.2.5 stop responding/processing requests. - Centos 6.5

robert


This might help with debugging it.





From: Users <[hidden email]> on behalf of Jonathan Hunter <[hidden email]>
Sent: Monday, October 30, 2017 2:25 PM
To: OpenSIPS users mailling list
Subject: Re: [OpenSIPS-Users] opensips 2.2.5 stop responding/processing requests. - Centos 6.5
 

Hi Johan,


Thanks for the response I appreciate it.


We do use radius for accounting and fire all the information to a central server, but it appears all that was fine and connectivity/configuration was good at the time of the issue.


Anywhere in particular it could break things? I have seen it when opensips cant reach radius it starts to be none responsive, is that a scenario you have seen?


Many thanks


Jon




From: Users <[hidden email]> on behalf of Johan De Clercq <[hidden email]>
Sent: 30 October 2017 17:53
To: OpenSIPS users mailling list
Subject: Re: [OpenSIPS-Users] opensips 2.2.5 stop responding/processing requests. - Centos 6.5
 
If you use acc with radius, check radius config.

Br

On 30 Oct 2017 18:33, "Jonathan Hunter" <[hidden email]> wrote:

Hi Guys,

Has anyone had the following behaviour with this release of opensips?

It has been running fine for a number of months, and then suddenly today the server which was running as the primary suddenly stopped responding to SIP requests, or stopped processing them, or there was a long pause in a sequence completing.

So in some cases, INVITES or REGISTER messages were ignored and the initial requests werent always logged in the opensips log file (even though we do this for all messages), however the SIP requests were hitting the server as I was tracing them.

In the logs I can see the following warnings we havent seen before;

Oct 30 13:31:25 sgw6 VU-SIP-Proxy[11159]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 9639891650 ms (now 9639891850 ms), it may overlap.
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <nh-timer> already scheduled for 9640994180 ms (now 9641175280 ms), it may overlap..
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <ul-timer> already scheduled for 9640980270 ms (now 9641175280 ms), it may overlap..

A restart of the opensips application didnt help, and service was only restored fully when I failed over to the secondary server.

Has anyone had these issues before? There are no obvious server level issues, and as I mentioned the logs dont contain anything too unusual.

Any help appreciated.

Many thanks

Jon


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.


Thank you for your compliance.



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips 2.2.5 stop responding/processing requests. - Centos 6.5

Bogdan-Andrei Iancu-2
In reply to this post by Jonathan Hunter
Hi Jonathan,

It looks like you have kind on task inside OpenSIPS that is using a lot of time/process resources - shortly, this translate into "there are no available worker processes to handle new jobs".

* do you have any threshold set to catch any DB/DNS delays ?
* have you checked the CPU usage ?
* have you monitor the "load:" statistics group ?

Regards,
Bogdan-Andrei Iancu
  OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
On 10/30/2017 07:33 PM, Jonathan Hunter wrote:
Hi Guys,

Has anyone had the following behaviour with this release of opensips?

It has been running fine for a number of months, and then suddenly today the server which was running as the primary suddenly stopped responding to SIP requests, or stopped processing them, or there was a long pause in a sequence completing.

So in some cases, INVITES or REGISTER messages were ignored and the initial requests werent always logged in the opensips log file (even though we do this for all messages), however the SIP requests were hitting the server as I was tracing them.

In the logs I can see the following warnings we havent seen before;

Oct 30 13:31:25 sgw6 VU-SIP-Proxy[11159]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 9639891650 ms (now 9639891850 ms), it may overlap.
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <nh-timer> already scheduled for 9640994180 ms (now 9641175280 ms), it may overlap..
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <ul-timer> already scheduled for 9640980270 ms (now 9641175280 ms), it may overlap..

A restart of the opensips application didnt help, and service was only restored fully when I failed over to the secondary server.

Has anyone had these issues before? There are no obvious server level issues, and as I mentioned the logs dont contain anything too unusual.

Any help appreciated.

Many thanks

Jon



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips 2.2.5 stop responding/processing requests. - Centos 6.5

Jonathan Hunter

Hi Guys,


Thanks for the responses,appreciate it!


In the end it was an overloaded DNS server, which caused a delay in another SBC passing requests to the opensips server, so overloading it.


Thanks for the advice everyone.  It wasnt OpenSIPS! 😊


Jon




From: Bogdan-Andrei Iancu <[hidden email]>
Sent: 31 October 2017 12:04
To: OpenSIPS users mailling list; Jonathan Hunter
Subject: Re: [OpenSIPS-Users] opensips 2.2.5 stop responding/processing requests. - Centos 6.5
 
Hi Jonathan,

It looks like you have kind on task inside OpenSIPS that is using a lot of time/process resources - shortly, this translate into "there are no available worker processes to handle new jobs".

* do you have any threshold set to catch any DB/DNS delays ?
* have you checked the CPU usage ?
* have you monitor the "load:" statistics group ?

Regards,
Bogdan-Andrei Iancu
  OpenSIPS Founder and Developer
  http://www.opensips-solutions.com

On 10/30/2017 07:33 PM, Jonathan Hunter wrote:
Hi Guys,

Has anyone had the following behaviour with this release of opensips?

It has been running fine for a number of months, and then suddenly today the server which was running as the primary suddenly stopped responding to SIP requests, or stopped processing them, or there was a long pause in a sequence completing.

So in some cases, INVITES or REGISTER messages were ignored and the initial requests werent always logged in the opensips log file (even though we do this for all messages), however the SIP requests were hitting the server as I was tracing them.

In the logs I can see the following warnings we havent seen before;

Oct 30 13:31:25 sgw6 VU-SIP-Proxy[11159]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 9639891650 ms (now 9639891850 ms), it may overlap.
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <nh-timer> already scheduled for 9640994180 ms (now 9641175280 ms), it may overlap..
Oct 30 13:52:48 sgw6 VU-SIP-Proxy[11159]: WARNING:core:timer_ticker: timer task <ul-timer> already scheduled for 9640980270 ms (now 9641175280 ms), it may overlap..

A restart of the opensips application didnt help, and service was only restored fully when I failed over to the secondary server.

Has anyone had these issues before? There are no obvious server level issues, and as I mentioned the logs dont contain anything too unusual.

Any help appreciated.

Many thanks

Jon



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users