Debug Assistance (All the gateways are disabled)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Debug Assistance (All the gateways are disabled)

Callum Guy
Hi All,

This morning one of our OpenSIPs instances (2.1.4 (x86_64/linux)) had an issue that lasted around a minute, I'm trying to establish cause and wondered if anyone would be able to offer suggestions based on our log output. At the time of the issue the server was receiving around genuine 80 calls per second however some did not get timely responses (took ~2.5 seconds) which caused the requests to fail. 

The following log messages represent the start and end of the failure period:

2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:00:21  ERROR:tm:uri2proxy: bad host name in URI <sip:[hidden email]>
2017-11-27 08:00:21  ERROR:tm:t_forward_nonack: failure to add branches
2017-11-27 08:00:21  ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp proxy
2017-11-27 08:00:21  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:00:21  ERROR:tm:uri2proxy: bad host name in URI <sip:[hidden email]  >
...
2017-11-27 08:01:17  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:01:17  ERROR:tm:uri2proxy: bad host name in URI <sip:[hidden email]  >
2017-11-27 08:01:17  ERROR:tm:t_forward_nonack: failure to add branches

Where the logs state internal.address these are normal and correct domains which have been redacted.

Is anyone able to offer any advice on how to interpret those messages? In particular the initial messages "All the gateways are disabled" seem like a good lead however I am unclear how to interpret these.

Any advice would be greatly appreciated.

Many thanks,

Callum
--
Callum Guy
Head of Information Security
X-on


0333 332 0000  |  www.x-on.co.uk  |       
X-on is a trading name of Storacall Technology Ltd a limited company registered in England and Wales.
Registered Office : Avaland House, 110 London Road, Apsley, Hemel Hempstead, Herts, HP3 9SD. Company Registration No. 2578478.
The information in this e-mail is confidential and for use by the addressee(s) only. If you are not the intended recipient, please notify X-on immediately on +44(0)333 332 0000 and delete the
message from your computer. If you are not a named addressee you must not use, disclose, disseminate, distribute, copy, print or reply to this email.
Views or opinions expressed by an individual
within this email may not necessarily reflect the views of X-on or its associated companies. Although X-on routinely screens for viruses, addressees should scan this email and any attachments
for viruses. X-on makes no representation or warranty as to the absence of viruses in this email or any attachments.


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Debug Assistance (All the gateways are disabled)

Bogdan-Andrei Iancu-2
Hi Callum,

That message means that all the gateways provisioned under the selected carrier are marked as disabled. So, how do you disable the gws ? do you do dr_disable() from script ?

Regards,
Bogdan-Andrei Iancu
  OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
On 11/27/2017 03:34 PM, Callum Guy wrote:
Hi All,

This morning one of our OpenSIPs instances (2.1.4 (x86_64/linux)) had an issue that lasted around a minute, I'm trying to establish cause and wondered if anyone would be able to offer suggestions based on our log output. At the time of the issue the server was receiving around genuine 80 calls per second however some did not get timely responses (took ~2.5 seconds) which caused the requests to fail. 

The following log messages represent the start and end of the failure period:

2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:00:21  ERROR:tm:uri2proxy: bad host name in URI <a class="moz-txt-link-rfc2396E" href="sip:441767300000@internal.address"><sip:441767300000@...>
2017-11-27 08:00:21  ERROR:tm:t_forward_nonack: failure to add branches
2017-11-27 08:00:21  ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp proxy
2017-11-27 08:00:21  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:00:21  ERROR:tm:uri2proxy: bad host name in URI <a class="moz-txt-link-rfc2396E" href="sip:442083400000@internal.address "><sip:442083400000@...  >
...
2017-11-27 08:01:17  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:01:17  ERROR:tm:uri2proxy: bad host name in URI <a class="moz-txt-link-rfc2396E" href="sip:441216100000@internal.address "><sip:441216100000@...  >
2017-11-27 08:01:17  ERROR:tm:t_forward_nonack: failure to add branches

Where the logs state internal.address these are normal and correct domains which have been redacted.

Is anyone able to offer any advice on how to interpret those messages? In particular the initial messages "All the gateways are disabled" seem like a good lead however I am unclear how to interpret these.

Any advice would be greatly appreciated.

Many thanks,

Callum
--
Callum Guy
Head of Information Security
X-on


0333 332 0000  |  www.x-on.co.uk  |       
X-on is a trading name of Storacall Technology Ltd a limited company registered in England and Wales.
Registered Office : Avaland House, 110 London Road, Apsley, Hemel Hempstead, Herts, HP3 9SD. Company Registration No. 2578478.
The information in this e-mail is confidential and for use by the addressee(s) only. If you are not the intended recipient, please notify X-on immediately on +44(0)333 332 0000 and delete the
message from your computer. If you are not a named addressee you must not use, disclose, disseminate, distribute, copy, print or reply to this email.
Views or opinions expressed by an individual
within this email may not necessarily reflect the views of X-on or its associated companies. Although X-on routinely screens for viruses, addressees should scan this email and any attachments
for viruses. X-on makes no representation or warranty as to the absence of viruses in this email or any attachments.



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Debug Assistance (All the gateways are disabled)

Brian :
In reply to this post by Callum Guy
Dns resolution issue? Do you use IPs or hostnames for your drouting gateways? 

On Monday, November 27, 2017, Callum Guy <[hidden email]> wrote:
Hi All,

This morning one of our OpenSIPs instances (2.1.4 (x86_64/linux)) had an issue that lasted around a minute, I'm trying to establish cause and wondered if anyone would be able to offer suggestions based on our log output. At the time of the issue the server was receiving around genuine 80 calls per second however some did not get timely responses (took ~2.5 seconds) which caused the requests to fail. 

The following log messages represent the start and end of the failure period:

2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:00:21  ERROR:tm:uri2proxy: bad host name in URI <sip:441767300000@internal.address>
2017-11-27 08:00:21  ERROR:tm:t_forward_nonack: failure to add branches
2017-11-27 08:00:21  ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp proxy
2017-11-27 08:00:21  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:00:21  ERROR:tm:uri2proxy: bad host name in URI <sip:442083400000@internal.address  >
...
2017-11-27 08:01:17  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:01:17  ERROR:tm:uri2proxy: bad host name in URI <sip:441216100000@internal.address  >
2017-11-27 08:01:17  ERROR:tm:t_forward_nonack: failure to add branches

Where the logs state internal.address these are normal and correct domains which have been redacted.

Is anyone able to offer any advice on how to interpret those messages? In particular the initial messages "All the gateways are disabled" seem like a good lead however I am unclear how to interpret these.

Any advice would be greatly appreciated.

Many thanks,

Callum
--
Callum Guy
Head of Information Security
X-on


0333 332 0000  |  www.x-on.co.uk  |       
X-on is a trading name of Storacall Technology Ltd a limited company registered in England and Wales.
Registered Office : Avaland House, 110 London Road, Apsley, Hemel Hempstead, Herts, HP3 9SD. Company Registration No. 2578478.
The information in this e-mail is confidential and for use by the addressee(s) only. If you are not the intended recipient, please notify X-on immediately on +44(0)333 332 0000 and delete the
message from your computer. If you are not a named addressee you must not use, disclose, disseminate, distribute, copy, print or reply to this email.
Views or opinions expressed by an individual
within this email may not necessarily reflect the views of X-on or its associated companies. Although X-on routinely screens for viruses, addressees should scan this email and any attachments
for viruses. X-on makes no representation or warranty as to the absence of viruses in this email or any attachments.


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Debug Assistance (All the gateways are disabled)

Callum Guy
Hi Bogdan/Brian,

It was a false alarm I am afraid and the fault was external to OpenSIPs however the error message did throw me off the scent of the fault for a while!

The configuration file used on the affected instance of OpenSIPs relies on an external web service to handle some routing scenarios. This component was running slow on the day in question which caused occupation of all OpenSIPs threads and a backlog of messages was created. We plan on designing the reliance on this service out of the system in our next release. 

Apologies for the delayed response - I missed your initial reply.

Best Regards,

Callum

On Wed, Nov 29, 2017 at 11:42 PM Brian : <[hidden email]> wrote:
Dns resolution issue? Do you use IPs or hostnames for your drouting gateways? 

On Monday, November 27, 2017, Callum Guy <[hidden email]> wrote:
Hi All,

This morning one of our OpenSIPs instances (2.1.4 (x86_64/linux)) had an issue that lasted around a minute, I'm trying to establish cause and wondered if anyone would be able to offer suggestions based on our log output. At the time of the issue the server was receiving around genuine 80 calls per second however some did not get timely responses (took ~2.5 seconds) which caused the requests to fail. 

The following log messages represent the start and end of the failure period:

2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  ERROR:drouting:route2_carrier: All the gateways are disabled
2017-11-27 08:00:21  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:00:21  ERROR:tm:uri2proxy: bad host name in URI <sip:[hidden email]>
2017-11-27 08:00:21  ERROR:tm:t_forward_nonack: failure to add branches
2017-11-27 08:00:21  ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp proxy
2017-11-27 08:00:21  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:00:21  ERROR:tm:uri2proxy: bad host name in URI <sip:[hidden email]  >
...
2017-11-27 08:01:17  CRITICAL:core:mk_proxy: could not resolve hostname: "internal.address"
2017-11-27 08:01:17  ERROR:tm:uri2proxy: bad host name in URI <sip:[hidden email]  >
2017-11-27 08:01:17  ERROR:tm:t_forward_nonack: failure to add branches

Where the logs state internal.address these are normal and correct domains which have been redacted.

Is anyone able to offer any advice on how to interpret those messages? In particular the initial messages "All the gateways are disabled" seem like a good lead however I am unclear how to interpret these.

Any advice would be greatly appreciated.

Many thanks,

Callum
--
Callum Guy
Head of Information Security
X-on


0333 332 0000  |  www.x-on.co.uk  |       
X-on is a trading name of Storacall Technology Ltd a limited company registered in England and Wales.
Registered Office : Avaland House, 110 London Road, Apsley, Hemel Hempstead, Herts, HP3 9SD. Company Registration No. 2578478.
The information in this e-mail is confidential and for use by the addressee(s) only. If you are not the intended recipient, please notify X-on immediately on <a href="tel:+44%20333%20332%200000" value="+443333320000" target="_blank">+44(0)333 332 0000 and delete the
message from your computer. If you are not a named addressee you must not use, disclose, disseminate, distribute, copy, print or reply to this email.
Views or opinions expressed by an individual
within this email may not necessarily reflect the views of X-on or its associated companies. Although X-on routinely screens for viruses, addressees should scan this email and any attachments
for viruses. X-on makes no representation or warranty as to the absence of viruses in this email or any attachments.

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
--
Callum Guy
Head of Information Security
X-on


0333 332 0000  |  www.x-on.co.uk  |       
X-on is a trading name of Storacall Technology Ltd a limited company registered in England and Wales.
Registered Office : Avaland House, 110 London Road, Apsley, Hemel Hempstead, Herts, HP3 9SD. Company Registration No. 2578478.
The information in this e-mail is confidential and for use by the addressee(s) only. If you are not the intended recipient, please notify X-on immediately on +44(0)333 332 0000 and delete the
message from your computer. If you are not a named addressee you must not use, disclose, disseminate, distribute, copy, print or reply to this email.
Views or opinions expressed by an individual
within this email may not necessarily reflect the views of X-on or its associated companies. Although X-on routinely screens for viruses, addressees should scan this email and any attachments
for viruses. X-on makes no representation or warranty as to the absence of viruses in this email or any attachments.


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users