Opensips restart and hanging process (dialogs?)

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Opensips restart and hanging process (dialogs?)

Max Mühlbronner
Hi,

when restarting Opensips it will shut down all the processes
immediately, but there is still a single process left "hanging" (cpu
load) which eventually exits after some time. It is not possible to
restart Opensips until this process is killed/quits.

root@opensips1:~# /etc/init.d/opensips restart
Restarting opensips: opensipsroot@opensips1:~#


Opensips version is 1.11.4-tls. But we also had the same issue with
1.7.x and 1.10.X.

There are ~3000 active dialogs, but it should not take that long to dump
everything into the database. (db_mode is set to "2") right?

Any hints/ideas are appreciated.


Best Regards


Max M.

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips restart and hanging process (dialogs?)

Max Mühlbronner

Comparing "opensipsctl ps" and the pid of the remaining process shows
it's the attendant process.

root@opensips1:/etc/opensips# opensipsctl ps

Process:: ID=0 PID=25935 Type=attendant



root@opensips1:/etc/opensips# /etc/init.d/opensips restart

Restarting opensips: opensips^[[Aroot@opensips1:/etc/opensips# ps
uax|grep opensips

opensips 25935 0.0 2.1 962160 21768 ? S 09:44 0:00 /sbin/opensips -P
/var/run/opensips/opensips.pid -m 896 -M 32 -u opensips -g opensips


After a few seconds the attendant process is gone, and opensips is not
running anymore. Did anyone experience something like this before?


BR

Max M.

On 05.05.2015 11:00, Max Mühlbronner wrote:

> Hi,
>
> when restarting Opensips it will shut down all the processes
> immediately, but there is still a single process left "hanging" (cpu
> load) which eventually exits after some time. It is not possible to
> restart Opensips until this process is killed/quits.
>
> root@opensips1:~# /etc/init.d/opensips restart
> Restarting opensips: opensipsroot@opensips1:~#
>
>
> Opensips version is 1.11.4-tls. But we also had the same issue with
> 1.7.x and 1.10.X.
>
> There are ~3000 active dialogs, but it should not take that long to
> dump everything into the database. (db_mode is set to "2") right?
>
> Any hints/ideas are appreciated.
>
>
> Best Regards
>
>
> Max M.
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips restart and hanging process (dialogs?)

Podrigal, Aron

What distro are u using? This sounds more like a problem in the init script,  where it tries to restart opensips before the last process has existed, which then fails to start.  Try adding  - - retry option to start-stop-daemon.

On May 7, 2015 4:01 AM, "Max Mühlbronner" <[hidden email]> wrote:

Comparing "opensipsctl ps" and the pid of the remaining process shows it's the attendant process.

root@opensips1:/etc/opensips# opensipsctl ps

Process:: ID=0 PID=25935 Type=attendant



root@opensips1:/etc/opensips# /etc/init.d/opensips restart

Restarting opensips: opensips^[[Aroot@opensips1:/etc/opensips# ps uax|grep opensips

opensips 25935 0.0 2.1 962160 21768 ? S 09:44 0:00 /sbin/opensips -P /var/run/opensips/opensips.pid -m 896 -M 32 -u opensips -g opensips


After a few seconds the attendant process is gone, and opensips is not running anymore. Did anyone experience something like this before?


BR

Max M.

On 05.05.2015 11:00, Max Mühlbronner wrote:
Hi,

when restarting Opensips it will shut down all the processes immediately, but there is still a single process left "hanging" (cpu load) which eventually exits after some time. It is not possible to restart Opensips until this process is killed/quits.

root@opensips1:~# /etc/init.d/opensips restart
Restarting opensips: opensipsroot@opensips1:~#


Opensips version is 1.11.4-tls. But we also had the same issue with 1.7.x and 1.10.X.

There are ~3000 active dialogs, but it should not take that long to dump everything into the database. (db_mode is set to "2") right?

Any hints/ideas are appreciated.


Best Regards


Max M.

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips restart and hanging process (dialogs?)

Max Mühlbronner
Hi,

Debian. testing now. Could this have any other undesired side-effects?

BR

Max M.

On 07.05.2015 10:12, Podrigal, Aron wrote:

What distro are u using? This sounds more like a problem in the init script,  where it tries to restart opensips before the last process has existed, which then fails to start.  Try adding  - - retry option to start-stop-daemon.

On May 7, 2015 4:01 AM, "Max Mühlbronner" <[hidden email]> wrote:

Comparing "opensipsctl ps" and the pid of the remaining process shows it's the attendant process.

root@opensips1:/etc/opensips# opensipsctl ps

Process:: ID=0 PID=25935 Type=attendant



root@opensips1:/etc/opensips# /etc/init.d/opensips restart

Restarting opensips: opensips^[[Aroot@opensips1:/etc/opensips# ps uax|grep opensips

opensips 25935 0.0 2.1 962160 21768 ? S 09:44 0:00 /sbin/opensips -P /var/run/opensips/opensips.pid -m 896 -M 32 -u opensips -g opensips


After a few seconds the attendant process is gone, and opensips is not running anymore. Did anyone experience something like this before?


BR

Max M.

On 05.05.2015 11:00, Max Mühlbronner wrote:
Hi,

when restarting Opensips it will shut down all the processes immediately, but there is still a single process left "hanging" (cpu load) which eventually exits after some time. It is not possible to restart Opensips until this process is killed/quits.

root@opensips1:~# /etc/init.d/opensips restart
Restarting opensips: opensipsroot@opensips1:~#


Opensips version is 1.11.4-tls. But we also had the same issue with 1.7.x and 1.10.X.

There are ~3000 active dialogs, but it should not take that long to dump everything into the database. (db_mode is set to "2") right?

Any hints/ideas are appreciated.


Best Regards


Max M.

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


-- 
Max Mühlbronner
42com Telecommunication GmbH
Straße der Pariser Kommune 12-16
10243 Berlin

E-Mail: [hidden email]
Web: www.42com.com

Firmenangaben/Company information:
Handelsregister/Commercial register: Amtsgericht Berlin HRB 99071 B
Umsatzsteuer-ID/VAT-ID: DE223812306
Geschäftsführer/CEO: Thomas Reinig, Alexander Reinig

Diese E-Mail enthält Informationen von 42com Telecommunication GmbH. Diese sind möglicherweise vertraulich und ausschließlich für den Adressaten bestimmt. 
Sollten Sie diese elektronische Nachricht irrtümlicherweise erhalten haben, so informieren Sie uns bitte unverzüglich telefonisch oder per E-Mail.
This message is intended only for the use of the individual or entity to which it is addressed. 
If you have received this message by mistake, please notify us immediately.

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips restart and hanging process (dialogs?)

Bogdan-Andrei Iancu-2
In reply to this post by Max Mühlbronner
Hi Max,

The attendant is responsible for doing the cleanup for all modules and
core - this is why it may take longer to complete. Do you have memory
logging or some expensive DB flushing on shutdown ?

BTW: see you in Amsterdam for the OpenSIPS Summit !

Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 07.05.2015 11:01, Max Mühlbronner wrote:

>
> Comparing "opensipsctl ps" and the pid of the remaining process shows
> it's the attendant process.
>
> root@opensips1:/etc/opensips# opensipsctl ps
>
> Process:: ID=0 PID=25935 Type=attendant
>
>
>
> root@opensips1:/etc/opensips# /etc/init.d/opensips restart
>
> Restarting opensips: opensips^[[Aroot@opensips1:/etc/opensips# ps
> uax|grep opensips
>
> opensips 25935 0.0 2.1 962160 21768 ? S 09:44 0:00 /sbin/opensips -P
> /var/run/opensips/opensips.pid -m 896 -M 32 -u opensips -g opensips
>
>
> After a few seconds the attendant process is gone, and opensips is not
> running anymore. Did anyone experience something like this before?
>
>
> BR
>
> Max M.
>
> On 05.05.2015 11:00, Max Mühlbronner wrote:
>> Hi,
>>
>> when restarting Opensips it will shut down all the processes
>> immediately, but there is still a single process left "hanging" (cpu
>> load) which eventually exits after some time. It is not possible to
>> restart Opensips until this process is killed/quits.
>>
>> root@opensips1:~# /etc/init.d/opensips restart
>> Restarting opensips: opensipsroot@opensips1:~#
>>
>>
>> Opensips version is 1.11.4-tls. But we also had the same issue with
>> 1.7.x and 1.10.X.
>>
>> There are ~3000 active dialogs, but it should not take that long to
>> dump everything into the database. (db_mode is set to "2") right?
>>
>> Any hints/ideas are appreciated.
>>
>>
>> Best Regards
>>
>>
>> Max M.
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips restart and hanging process (dialogs?)

Max Mühlbronner
Hi,

Thanks for your reply. No, nothing special. Even changed db_mode for
dialog module to realtime. No usrloc.

Delaying the start-stop-daemon (--retry) like Aron suggested is a
workaround for restarting, although i am still wondering why the process
is taking so long to quit.

Indeed, see you at the Summit.


BR

Max M.

On 07.05.2015 18:35, Bogdan-Andrei Iancu wrote:

> Hi Max,
>
> The attendant is responsible for doing the cleanup for all modules and
> core - this is why it may take longer to complete. Do you have memory
> logging or some expensive DB flushing on shutdown ?
>
> BTW: see you in Amsterdam for the OpenSIPS Summit !
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developer
> http://www.opensips-solutions.com
>
> On 07.05.2015 11:01, Max Mühlbronner wrote:
>>
>> Comparing "opensipsctl ps" and the pid of the remaining process shows
>> it's the attendant process.
>>
>> root@opensips1:/etc/opensips# opensipsctl ps
>>
>> Process:: ID=0 PID=25935 Type=attendant
>>
>>
>>
>> root@opensips1:/etc/opensips# /etc/init.d/opensips restart
>>
>> Restarting opensips: opensips^[[Aroot@opensips1:/etc/opensips# ps
>> uax|grep opensips
>>
>> opensips 25935 0.0 2.1 962160 21768 ? S 09:44 0:00 /sbin/opensips -P
>> /var/run/opensips/opensips.pid -m 896 -M 32 -u opensips -g opensips
>>
>>
>> After a few seconds the attendant process is gone, and opensips is
>> not running anymore. Did anyone experience something like this before?
>>
>>
>> BR
>>
>> Max M.
>>
>> On 05.05.2015 11:00, Max Mühlbronner wrote:
>>> Hi,
>>>
>>> when restarting Opensips it will shut down all the processes
>>> immediately, but there is still a single process left "hanging" (cpu
>>> load) which eventually exits after some time. It is not possible to
>>> restart Opensips until this process is killed/quits.
>>>
>>> root@opensips1:~# /etc/init.d/opensips restart
>>> Restarting opensips: opensipsroot@opensips1:~#
>>>
>>>
>>> Opensips version is 1.11.4-tls. But we also had the same issue with
>>> 1.7.x and 1.10.X.
>>>
>>> There are ~3000 active dialogs, but it should not take that long to
>>> dump everything into the database. (db_mode is set to "2") right?
>>>
>>> Any hints/ideas are appreciated.
>>>
>>>
>>> Best Regards
>>>
>>>
>>> Max M.
>>>
>>> _______________________________________________
>>> Users mailing list
>>> [hidden email]
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>


--
Max Mühlbronner
42com Telecommunication GmbH
Straße der Pariser Kommune 12-16
10243 Berlin

E-Mail: [hidden email]
Web: www.42com.com

Firmenangaben/Company information:
Handelsregister/Commercial register: Amtsgericht Berlin HRB 99071 B
Umsatzsteuer-ID/VAT-ID: DE223812306
Geschäftsführer/CEO: Thomas Reinig, Alexander Reinig

Diese E-Mail enthält Informationen von 42com Telecommunication GmbH. Diese sind möglicherweise vertraulich und ausschließlich für den Adressaten bestimmt.
Sollten Sie diese elektronische Nachricht irrtümlicherweise erhalten haben, so informieren Sie uns bitte unverzüglich telefonisch oder per E-Mail.
This message is intended only for the use of the individual or entity to which it is addressed.
If you have received this message by mistake, please notify us immediately.


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips restart and hanging process (dialogs?)

Bogdan-Andrei Iancu-2
Max,

What you can do is to try to attach with GDB to the attendant process
while doing the cleanup. It will be interesting to see what is going on.

Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 08.05.2015 09:58, Max Mühlbronner wrote:

> Hi,
>
> Thanks for your reply. No, nothing special. Even changed db_mode for
> dialog module to realtime. No usrloc.
>
> Delaying the start-stop-daemon (--retry) like Aron suggested is a
> workaround for restarting, although i am still wondering why the
> process is taking so long to quit.
>
> Indeed, see you at the Summit.
>
>
> BR
>
> Max M.
>
> On 07.05.2015 18:35, Bogdan-Andrei Iancu wrote:
>> Hi Max,
>>
>> The attendant is responsible for doing the cleanup for all modules
>> and core - this is why it may take longer to complete. Do you have
>> memory logging or some expensive DB flushing on shutdown ?
>>
>> BTW: see you in Amsterdam for the OpenSIPS Summit !
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com
>>
>> On 07.05.2015 11:01, Max Mühlbronner wrote:
>>>
>>> Comparing "opensipsctl ps" and the pid of the remaining process
>>> shows it's the attendant process.
>>>
>>> root@opensips1:/etc/opensips# opensipsctl ps
>>>
>>> Process:: ID=0 PID=25935 Type=attendant
>>>
>>>
>>>
>>> root@opensips1:/etc/opensips# /etc/init.d/opensips restart
>>>
>>> Restarting opensips: opensips^[[Aroot@opensips1:/etc/opensips# ps
>>> uax|grep opensips
>>>
>>> opensips 25935 0.0 2.1 962160 21768 ? S 09:44 0:00 /sbin/opensips -P
>>> /var/run/opensips/opensips.pid -m 896 -M 32 -u opensips -g opensips
>>>
>>>
>>> After a few seconds the attendant process is gone, and opensips is
>>> not running anymore. Did anyone experience something like this before?
>>>
>>>
>>> BR
>>>
>>> Max M.
>>>
>>> On 05.05.2015 11:00, Max Mühlbronner wrote:
>>>> Hi,
>>>>
>>>> when restarting Opensips it will shut down all the processes
>>>> immediately, but there is still a single process left "hanging"
>>>> (cpu load) which eventually exits after some time. It is not
>>>> possible to restart Opensips until this process is killed/quits.
>>>>
>>>> root@opensips1:~# /etc/init.d/opensips restart
>>>> Restarting opensips: opensipsroot@opensips1:~#
>>>>
>>>>
>>>> Opensips version is 1.11.4-tls. But we also had the same issue with
>>>> 1.7.x and 1.10.X.
>>>>
>>>> There are ~3000 active dialogs, but it should not take that long to
>>>> dump everything into the database. (db_mode is set to "2") right?
>>>>
>>>> Any hints/ideas are appreciated.
>>>>
>>>>
>>>> Best Regards
>>>>
>>>>
>>>> Max M.
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> [hidden email]
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> [hidden email]
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>
>
>


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips restart and hanging process (dialogs?)

Max Mühlbronner

Here is a pastebin of the bt (attendant process, while restarting).
Seems to be the database operations/dialogs?


http://pastebin.com/9zHQjMMq


BR

Max M.

On 08.05.2015 11:01, Bogdan-Andrei Iancu wrote:

> Max,
>
> What you can do is to try to attach with GDB to the attendant process
> while doing the cleanup. It will be interesting to see what is going on.
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developer
> http://www.opensips-solutions.com
>
> On 08.05.2015 09:58, Max Mühlbronner wrote:
>> Hi,
>>
>> Thanks for your reply. No, nothing special. Even changed db_mode for
>> dialog module to realtime. No usrloc.
>>
>> Delaying the start-stop-daemon (--retry) like Aron suggested is a
>> workaround for restarting, although i am still wondering why the
>> process is taking so long to quit.
>>
>> Indeed, see you at the Summit.
>>
>>
>> BR
>>
>> Max M.
>>
>> On 07.05.2015 18:35, Bogdan-Andrei Iancu wrote:
>>> Hi Max,
>>>
>>> The attendant is responsible for doing the cleanup for all modules
>>> and core - this is why it may take longer to complete. Do you have
>>> memory logging or some expensive DB flushing on shutdown ?
>>>
>>> BTW: see you in Amsterdam for the OpenSIPS Summit !
>>>
>>> Regards,
>>>
>>> Bogdan-Andrei Iancu
>>> OpenSIPS Founder and Developer
>>> http://www.opensips-solutions.com
>>>
>>> On 07.05.2015 11:01, Max Mühlbronner wrote:
>>>>
>>>> Comparing "opensipsctl ps" and the pid of the remaining process
>>>> shows it's the attendant process.
>>>>
>>>> root@opensips1:/etc/opensips# opensipsctl ps
>>>>
>>>> Process:: ID=0 PID=25935 Type=attendant
>>>>
>>>>
>>>>
>>>> root@opensips1:/etc/opensips# /etc/init.d/opensips restart
>>>>
>>>> Restarting opensips: opensips^[[Aroot@opensips1:/etc/opensips# ps
>>>> uax|grep opensips
>>>>
>>>> opensips 25935 0.0 2.1 962160 21768 ? S 09:44 0:00 /sbin/opensips
>>>> -P /var/run/opensips/opensips.pid -m 896 -M 32 -u opensips -g opensips
>>>>
>>>>
>>>> After a few seconds the attendant process is gone, and opensips is
>>>> not running anymore. Did anyone experience something like this before?
>>>>
>>>>
>>>> BR
>>>>
>>>> Max M.
>>>>
>>>> On 05.05.2015 11:00, Max Mühlbronner wrote:
>>>>> Hi,
>>>>>
>>>>> when restarting Opensips it will shut down all the processes
>>>>> immediately, but there is still a single process left "hanging"
>>>>> (cpu load) which eventually exits after some time. It is not
>>>>> possible to restart Opensips until this process is killed/quits.
>>>>>
>>>>> root@opensips1:~# /etc/init.d/opensips restart
>>>>> Restarting opensips: opensipsroot@opensips1:~#
>>>>>
>>>>>
>>>>> Opensips version is 1.11.4-tls. But we also had the same issue
>>>>> with 1.7.x and 1.10.X.
>>>>>
>>>>> There are ~3000 active dialogs, but it should not take that long
>>>>> to dump everything into the database. (db_mode is set to "2") right?
>>>>>
>>>>> Any hints/ideas are appreciated.
>>>>>
>>>>>
>>>>> Best Regards
>>>>>
>>>>>
>>>>> Max M.
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> [hidden email]
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> [hidden email]
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>>
>>>
>>
>>
>


--
Max Mühlbronner
42com Telecommunication GmbH
Straße der Pariser Kommune 12-16
10243 Berlin

E-Mail: [hidden email]
Web: www.42com.com

Firmenangaben/Company information:
Handelsregister/Commercial register: Amtsgericht Berlin HRB 99071 B
Umsatzsteuer-ID/VAT-ID: DE223812306
Geschäftsführer/CEO: Thomas Reinig, Alexander Reinig

Diese E-Mail enthält Informationen von 42com Telecommunication GmbH. Diese sind möglicherweise vertraulich und ausschließlich für den Adressaten bestimmt.
Sollten Sie diese elektronische Nachricht irrtümlicherweise erhalten haben, so informieren Sie uns bitte unverzüglich telefonisch oder per E-Mail.
This message is intended only for the use of the individual or entity to which it is addressed.
If you have received this message by mistake, please notify us immediately.


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips restart and hanging process (dialogs?)

Bogdan-Andrei Iancu-2
Yes indeed, it looks like flushing dialog info into DB. How many dialog
do you have ongoing and how fast your db is ??

Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 08.05.2015 13:05, Max Mühlbronner wrote:

>
> Here is a pastebin of the bt (attendant process, while restarting).
> Seems to be the database operations/dialogs?
>
>
> http://pastebin.com/9zHQjMMq
>
>
> BR
>
> Max M.
>
> On 08.05.2015 11:01, Bogdan-Andrei Iancu wrote:
>> Max,
>>
>> What you can do is to try to attach with GDB to the attendant process
>> while doing the cleanup. It will be interesting to see what is going on.
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com
>>
>> On 08.05.2015 09:58, Max Mühlbronner wrote:
>>> Hi,
>>>
>>> Thanks for your reply. No, nothing special. Even changed db_mode for
>>> dialog module to realtime. No usrloc.
>>>
>>> Delaying the start-stop-daemon (--retry) like Aron suggested is a
>>> workaround for restarting, although i am still wondering why the
>>> process is taking so long to quit.
>>>
>>> Indeed, see you at the Summit.
>>>
>>>
>>> BR
>>>
>>> Max M.
>>>
>>> On 07.05.2015 18:35, Bogdan-Andrei Iancu wrote:
>>>> Hi Max,
>>>>
>>>> The attendant is responsible for doing the cleanup for all modules
>>>> and core - this is why it may take longer to complete. Do you have
>>>> memory logging or some expensive DB flushing on shutdown ?
>>>>
>>>> BTW: see you in Amsterdam for the OpenSIPS Summit !
>>>>
>>>> Regards,
>>>>
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developer
>>>> http://www.opensips-solutions.com
>>>>
>>>> On 07.05.2015 11:01, Max Mühlbronner wrote:
>>>>>
>>>>> Comparing "opensipsctl ps" and the pid of the remaining process
>>>>> shows it's the attendant process.
>>>>>
>>>>> root@opensips1:/etc/opensips# opensipsctl ps
>>>>>
>>>>> Process:: ID=0 PID=25935 Type=attendant
>>>>>
>>>>>
>>>>>
>>>>> root@opensips1:/etc/opensips# /etc/init.d/opensips restart
>>>>>
>>>>> Restarting opensips: opensips^[[Aroot@opensips1:/etc/opensips# ps
>>>>> uax|grep opensips
>>>>>
>>>>> opensips 25935 0.0 2.1 962160 21768 ? S 09:44 0:00 /sbin/opensips
>>>>> -P /var/run/opensips/opensips.pid -m 896 -M 32 -u opensips -g
>>>>> opensips
>>>>>
>>>>>
>>>>> After a few seconds the attendant process is gone, and opensips is
>>>>> not running anymore. Did anyone experience something like this
>>>>> before?
>>>>>
>>>>>
>>>>> BR
>>>>>
>>>>> Max M.
>>>>>
>>>>> On 05.05.2015 11:00, Max Mühlbronner wrote:
>>>>>> Hi,
>>>>>>
>>>>>> when restarting Opensips it will shut down all the processes
>>>>>> immediately, but there is still a single process left "hanging"
>>>>>> (cpu load) which eventually exits after some time. It is not
>>>>>> possible to restart Opensips until this process is killed/quits.
>>>>>>
>>>>>> root@opensips1:~# /etc/init.d/opensips restart
>>>>>> Restarting opensips: opensipsroot@opensips1:~#
>>>>>>
>>>>>>
>>>>>> Opensips version is 1.11.4-tls. But we also had the same issue
>>>>>> with 1.7.x and 1.10.X.
>>>>>>
>>>>>> There are ~3000 active dialogs, but it should not take that long
>>>>>> to dump everything into the database. (db_mode is set to "2") right?
>>>>>>
>>>>>> Any hints/ideas are appreciated.
>>>>>>
>>>>>>
>>>>>> Best Regards
>>>>>>
>>>>>>
>>>>>> Max M.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> [hidden email]
>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> [hidden email]
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips restart and hanging process (dialogs?)

Max Mühlbronner

~3000 Dialogs. The DB is supposed to be fast, at least slow query
logging did not reveal anything when testing earlier.

Currently the plan is to try a different/locally installed DB Server to
also rule out the possibility of network problems/lags.


BR

Max M.

On 08.05.2015 12:16, Bogdan-Andrei Iancu wrote:

> Yes indeed, it looks like flushing dialog info into DB. How many
> dialog do you have ongoing and how fast your db is ??
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developer
> http://www.opensips-solutions.com
>
> On 08.05.2015 13:05, Max Mühlbronner wrote:
>>
>> Here is a pastebin of the bt (attendant process, while restarting).
>> Seems to be the database operations/dialogs?
>>
>>
>> http://pastebin.com/9zHQjMMq
>>
>>
>> BR
>>
>> Max M.
>>
>> On 08.05.2015 11:01, Bogdan-Andrei Iancu wrote:
>>> Max,
>>>
>>> What you can do is to try to attach with GDB to the attendant
>>> process while doing the cleanup. It will be interesting to see what
>>> is going on.
>>>
>>> Regards,
>>>
>>> Bogdan-Andrei Iancu
>>> OpenSIPS Founder and Developer
>>> http://www.opensips-solutions.com
>>>
>>> On 08.05.2015 09:58, Max Mühlbronner wrote:
>>>> Hi,
>>>>
>>>> Thanks for your reply. No, nothing special. Even changed db_mode
>>>> for dialog module to realtime. No usrloc.
>>>>
>>>> Delaying the start-stop-daemon (--retry) like Aron suggested is a
>>>> workaround for restarting, although i am still wondering why the
>>>> process is taking so long to quit.
>>>>
>>>> Indeed, see you at the Summit.
>>>>
>>>>
>>>> BR
>>>>
>>>> Max M.
>>>>
>>>> On 07.05.2015 18:35, Bogdan-Andrei Iancu wrote:
>>>>> Hi Max,
>>>>>
>>>>> The attendant is responsible for doing the cleanup for all modules
>>>>> and core - this is why it may take longer to complete. Do you have
>>>>> memory logging or some expensive DB flushing on shutdown ?
>>>>>
>>>>> BTW: see you in Amsterdam for the OpenSIPS Summit !
>>>>>
>>>>> Regards,
>>>>>
>>>>> Bogdan-Andrei Iancu
>>>>> OpenSIPS Founder and Developer
>>>>> http://www.opensips-solutions.com
>>>>>
>>>>> On 07.05.2015 11:01, Max Mühlbronner wrote:
>>>>>>
>>>>>> Comparing "opensipsctl ps" and the pid of the remaining process
>>>>>> shows it's the attendant process.
>>>>>>
>>>>>> root@opensips1:/etc/opensips# opensipsctl ps
>>>>>>
>>>>>> Process:: ID=0 PID=25935 Type=attendant
>>>>>>
>>>>>>
>>>>>>
>>>>>> root@opensips1:/etc/opensips# /etc/init.d/opensips restart
>>>>>>
>>>>>> Restarting opensips: opensips^[[Aroot@opensips1:/etc/opensips# ps
>>>>>> uax|grep opensips
>>>>>>
>>>>>> opensips 25935 0.0 2.1 962160 21768 ? S 09:44 0:00 /sbin/opensips
>>>>>> -P /var/run/opensips/opensips.pid -m 896 -M 32 -u opensips -g
>>>>>> opensips
>>>>>>
>>>>>>
>>>>>> After a few seconds the attendant process is gone, and opensips
>>>>>> is not running anymore. Did anyone experience something like this
>>>>>> before?
>>>>>>
>>>>>>
>>>>>> BR
>>>>>>
>>>>>> Max M.
>>>>>>
>>>>>> On 05.05.2015 11:00, Max Mühlbronner wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> when restarting Opensips it will shut down all the processes
>>>>>>> immediately, but there is still a single process left "hanging"
>>>>>>> (cpu load) which eventually exits after some time. It is not
>>>>>>> possible to restart Opensips until this process is killed/quits.
>>>>>>>
>>>>>>> root@opensips1:~# /etc/init.d/opensips restart
>>>>>>> Restarting opensips: opensipsroot@opensips1:~#
>>>>>>>
>>>>>>>
>>>>>>> Opensips version is 1.11.4-tls. But we also had the same issue
>>>>>>> with 1.7.x and 1.10.X.
>>>>>>>
>>>>>>> There are ~3000 active dialogs, but it should not take that long
>>>>>>> to dump everything into the database. (db_mode is set to "2")
>>>>>>> right?
>>>>>>>
>>>>>>> Any hints/ideas are appreciated.
>>>>>>>
>>>>>>>
>>>>>>> Best Regards
>>>>>>>
>>>>>>>
>>>>>>> Max M.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> [hidden email]
>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> [hidden email]
>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>


--
Max Mühlbronner
42com Telecommunication GmbH
Straße der Pariser Kommune 12-16
10243 Berlin

E-Mail: [hidden email]
Web: www.42com.com

Firmenangaben/Company information:
Handelsregister/Commercial register: Amtsgericht Berlin HRB 99071 B
Umsatzsteuer-ID/VAT-ID: DE223812306
Geschäftsführer/CEO: Thomas Reinig, Alexander Reinig

Diese E-Mail enthält Informationen von 42com Telecommunication GmbH. Diese sind möglicherweise vertraulich und ausschließlich für den Adressaten bestimmt.
Sollten Sie diese elektronische Nachricht irrtümlicherweise erhalten haben, so informieren Sie uns bitte unverzüglich telefonisch oder per E-Mail.
This message is intended only for the use of the individual or entity to which it is addressed.
If you have received this message by mistake, please notify us immediately.


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Opensips restart and hanging process (dialogs?)

Bogdan-Andrei Iancu-2
DB mode 2 should be fine, 3K dialogs should not be such a burn. It
depends on the DB performance IMHO.

Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 08.05.2015 14:04, Max Mühlbronner wrote:

>
> ~3000 Dialogs. The DB is supposed to be fast, at least slow query
> logging did not reveal anything when testing earlier.
>
> Currently the plan is to try a different/locally installed DB Server
> to also rule out the possibility of network problems/lags.
>
>
> BR
>
> Max M.
>
> On 08.05.2015 12:16, Bogdan-Andrei Iancu wrote:
>> Yes indeed, it looks like flushing dialog info into DB. How many
>> dialog do you have ongoing and how fast your db is ??
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com
>>
>> On 08.05.2015 13:05, Max Mühlbronner wrote:
>>>
>>> Here is a pastebin of the bt (attendant process, while restarting).
>>> Seems to be the database operations/dialogs?
>>>
>>>
>>> http://pastebin.com/9zHQjMMq
>>>
>>>
>>> BR
>>>
>>> Max M.
>>>
>>> On 08.05.2015 11:01, Bogdan-Andrei Iancu wrote:
>>>> Max,
>>>>
>>>> What you can do is to try to attach with GDB to the attendant
>>>> process while doing the cleanup. It will be interesting to see what
>>>> is going on.
>>>>
>>>> Regards,
>>>>
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developer
>>>> http://www.opensips-solutions.com
>>>>
>>>> On 08.05.2015 09:58, Max Mühlbronner wrote:
>>>>> Hi,
>>>>>
>>>>> Thanks for your reply. No, nothing special. Even changed db_mode
>>>>> for dialog module to realtime. No usrloc.
>>>>>
>>>>> Delaying the start-stop-daemon (--retry) like Aron suggested is a
>>>>> workaround for restarting, although i am still wondering why the
>>>>> process is taking so long to quit.
>>>>>
>>>>> Indeed, see you at the Summit.
>>>>>
>>>>>
>>>>> BR
>>>>>
>>>>> Max M.
>>>>>
>>>>> On 07.05.2015 18:35, Bogdan-Andrei Iancu wrote:
>>>>>> Hi Max,
>>>>>>
>>>>>> The attendant is responsible for doing the cleanup for all
>>>>>> modules and core - this is why it may take longer to complete. Do
>>>>>> you have memory logging or some expensive DB flushing on shutdown ?
>>>>>>
>>>>>> BTW: see you in Amsterdam for the OpenSIPS Summit !
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Bogdan-Andrei Iancu
>>>>>> OpenSIPS Founder and Developer
>>>>>> http://www.opensips-solutions.com
>>>>>>
>>>>>> On 07.05.2015 11:01, Max Mühlbronner wrote:
>>>>>>>
>>>>>>> Comparing "opensipsctl ps" and the pid of the remaining process
>>>>>>> shows it's the attendant process.
>>>>>>>
>>>>>>> root@opensips1:/etc/opensips# opensipsctl ps
>>>>>>>
>>>>>>> Process:: ID=0 PID=25935 Type=attendant
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> root@opensips1:/etc/opensips# /etc/init.d/opensips restart
>>>>>>>
>>>>>>> Restarting opensips: opensips^[[Aroot@opensips1:/etc/opensips#
>>>>>>> ps uax|grep opensips
>>>>>>>
>>>>>>> opensips 25935 0.0 2.1 962160 21768 ? S 09:44 0:00
>>>>>>> /sbin/opensips -P /var/run/opensips/opensips.pid -m 896 -M 32 -u
>>>>>>> opensips -g opensips
>>>>>>>
>>>>>>>
>>>>>>> After a few seconds the attendant process is gone, and opensips
>>>>>>> is not running anymore. Did anyone experience something like
>>>>>>> this before?
>>>>>>>
>>>>>>>
>>>>>>> BR
>>>>>>>
>>>>>>> Max M.
>>>>>>>
>>>>>>> On 05.05.2015 11:00, Max Mühlbronner wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> when restarting Opensips it will shut down all the processes
>>>>>>>> immediately, but there is still a single process left "hanging"
>>>>>>>> (cpu load) which eventually exits after some time. It is not
>>>>>>>> possible to restart Opensips until this process is killed/quits.
>>>>>>>>
>>>>>>>> root@opensips1:~# /etc/init.d/opensips restart
>>>>>>>> Restarting opensips: opensipsroot@opensips1:~#
>>>>>>>>
>>>>>>>>
>>>>>>>> Opensips version is 1.11.4-tls. But we also had the same issue
>>>>>>>> with 1.7.x and 1.10.X.
>>>>>>>>
>>>>>>>> There are ~3000 active dialogs, but it should not take that
>>>>>>>> long to dump everything into the database. (db_mode is set to
>>>>>>>> "2") right?
>>>>>>>>
>>>>>>>> Any hints/ideas are appreciated.
>>>>>>>>
>>>>>>>>
>>>>>>>> Best Regards
>>>>>>>>
>>>>>>>>
>>>>>>>> Max M.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> [hidden email]
>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> [hidden email]
>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users