Opensips stops responding

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Opensips stops responding

S. Rosenberg
Hi, I've had recently a few times that OpenSIPSs didnt crash but it
stopped responding to certain requests, perhaps INVITES that requeired
DB access, but I havent verified it becuase I always wanted to bring
it back up ASAP because it seems like it did respond to my requests
that my keepalived VRRP script which sends a notify to OpenSIPs and
OpenSIPS does a DB lookup and only on success it returns a true thats
how I make sure that MYSQL is working in addition to  OpenSIPs, but
opensips seems to have responded because it did not failover, and
thats why I didnt have time to investigate further to which requests
it responds and to which it didnt.

Here are the syslogs from the time of the crash, maybe it can shed
some light to the issue, the query in there that failed is the query
from keepalived, I assume it failed only once and then did respond,
otherwise the IP would of failed over to the backup, but its only an
assumption.

Jul 10 20:13:13 sipsvr1 /sbin/opensips[29446]:
INFO:db_mysql:switch_state_to_disconnected: disconnect event for
0x7f2ba43e7600
Jul 10 20:13:13 sipsvr1 /sbin/opensips[29446]:
INFO:db_mysql:reset_all_statements: resetting all statements on
connection: (0x7f2ba44062a0) 0x7f2ba43e7600
Jul 10 20:13:13 sipsvr1 /sbin/opensips[29446]:
INFO:db_mysql:connect_with_retry: re-connected successful for
0x7f2ba43e7600
Jul 10 20:13:13 sipsvr1 /sbin/opensips[29446]:
INFO:db_mysql:db_mysql_do_prepared_query: reconnected to mysql server
-> re-init the statement
Jul 10 20:13:13 sipsvr1 /sbin/opensips[29486]:
WARNING:core:handle_timer_job: utimer job <tm-utimer> has a 30000 us
delay in execution
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29400]:
INFO:clusterer:do_action_trans_2: Ping reply not received, node [2] is
down
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29400]:
INFO:clusterer:do_action_trans_2: Ping reply not received, node [2] is
down
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29478]:
INFO:clusterer:handle_internal_msg: Node [2] is UP
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29478]:
INFO:clusterer:handle_internal_msg: Node [2] is UP
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29422]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29425.374568.2]
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29419]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29431.1000819.1]
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29428]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29411.161103.9]
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29446]:
CRITICAL:db_mysql:wrapper_single_mysql_stmt_execute: driver error
(1180): Got error 35 "Resource deadlock avoided" during COMMIT
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29446]:
ERROR:usrloc:db_update_ucontact: updating database failed
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29446]: ERROR:usrloc:wb_timer:
updating contact in db failed
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29388]:
CRITICAL:db_mysql:wrapper_single_mysql_real_query: driver error
(1047): WSREP has not yet prepared node for application use
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29388]:
ERROR:core:db_do_raw_query: error while submitting query
Jul 10 20:13:15 sipsvr1 /sbin/opensips[29388]:
ERROR:avpops:db_query_avp: raw_query failed: db0(usr_preferences)
select domain from domain where domain='...
Jul 10 20:13:15 sipsvr1 Keepalived_vrrp[514]: pid 2092 exited with status 1
Jul 10 20:13:16 sipsvr1 /sbin/opensips[29428]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29422.274079.61]
Jul 10 20:13:16 sipsvr1 /sbin/opensips[29429]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29421.236751.61]
Jul 10 20:13:16 sipsvr1 /sbin/opensips[29425]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29419.224887.28]
Jul 10 20:13:16 sipsvr1 /sbin/opensips[29422]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29413.165448.9]
Jul 10 20:13:16 sipsvr1 /sbin/opensips[29419]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29429.711965.1]
Jul 10 20:13:16 sipsvr1 /sbin/opensips[29411]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29431.1000786.2]
Jul 10 20:13:16 sipsvr1 /sbin/opensips[29428]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29411.161104.10]
Jul 10 20:13:16 sipsvr1 /sbin/opensips[29421]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29419.224888.10]
Jul 10 20:13:16 sipsvr1 /sbin/opensips[29433]:
INFO:presence:update_presentity: *** found in db but not in htable
[a.1561468443.29421.236752.28]

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips stops responding

S. Rosenberg
I want to add that a restart of OpenSIPs fixed the problem every time, I
didnt restart the MySQL server.



--
Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips stops responding

S. Rosenberg
Another update, seems like my assumption that some requests were handeld was wrong,  the keepalive script had a bug and therefore didn't failover, I now believe OpenSIPs ignored all requests after the initial DB error, this seems like a bug in OpenSIPs it shouldnt just stop responding to requests because of one DB attempt that went wrong.

On Thu, Jul 11, 2019, 12:39 PM S. Rosenberg <[hidden email]> wrote:
I want to add that a restart of OpenSIPs fixed the problem every time, I
didnt restart the MySQL server.



--
Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips stops responding

Răzvan Crainea-2
Hi, Schneur!

If this happens again, can you please run an `opensipsctl trap` command
- this could help us understand why it was not handling traffic.
BTW: could it be possible that the keepalived script also removed the IP
from the interface, and OpenSIPS was no longer getting any traffic
whatsoever?

Best regards,
Răzvan

On 7/12/19 1:38 PM, Schneur Rosenberg wrote:

> Another update, seems like my assumption that some requests were handeld
> was wrong,  the keepalive script had a bug and therefore didn't
> failover, I now believe OpenSIPs ignored all requests after the initial
> DB error, this seems like a bug in OpenSIPs it shouldnt just stop
> responding to requests because of one DB attempt that went wrong.
>
> On Thu, Jul 11, 2019, 12:39 PM S. Rosenberg <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I want to add that a restart of OpenSIPs fixed the problem every time, I
>     didnt restart the MySQL server.
>
>
>
>     --
>     Sent from:
>     http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html
>
>     _______________________________________________
>     Users mailing list
>     [hidden email] <mailto:[hidden email]>
>     http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>

--
Răzvan Crainea
OpenSIPS Core Developer
   http://www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users