core dump in dialog

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

core dump in dialog

Jennifer Hashimoto
Hi guys,
I’m getting ready to switch to using opensips 2.4 but as soon as i put traffic on it (like 3 calls, not a lot) it crashed.
it happened twice - once when I didn’t have the dump cores on, so I turned it on, then started and it crashed almost immediately after that

I thought maybe it was because it did a insert on my postgres DB which doesn’t happen every call, 

It’s really strange because when i try test calls it seems fine, but real traffic it doesn’t’ like :(

 
I think it’s part of where i do lookups for limit like this tutorial: https://www.opensips.org/Documentation/Tutorials-ConcurrentCallsLimitation


version: opensips 2.4.2 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on  with gcc 4.9.2

Here are some relevant bits from my opensips.cfg

loadmodule "db_virtual.so"
modparam("db_virtual", "db_urls", "define debdb FAILOVER")
modparam("db_virtual", "db_urls", "<a href="postgres://opensips:" class="">postgres://opensips:passwd@192.168.5.65/opensips_cazcore")
modparam("db_virtual", "db_urls", "<a href="postgres://opensips:" class="">postgres://opensips:passwd@192.168.6.65/opensips_cazcore")

#### DIALOG module
loadmodule "dialog.so"
modparam("dialog", "dlg_match_mode", 1)
modparam("dialog", "default_timeout", 21600)  # 6 hours timeout
modparam("dialog", "db_mode", 1)  
modparam("dialog", "db_update_period", 5)  

#modparam("dialog", "db_url", "<a href="postgres://opensips:passwd@192.168.5.65/opensips_cazcore" class="">postgres://opensips:passwd@192.168.5.65/opensips_cazcore") 
modparam("dialog", "db_url", "<a href="virtual://debdb" class="">virtual://debdb") 
modparam("dialog", "dialog_replication_cluster", 1)
modparam("dialog", "profile_replication_cluster", 1)
#tags for VIP handling in cluster
modparam("dialog", "dlg_sharing_tag", "vipA=MY_VIPA")
modparam("dialog", "dlg_sharing_tag", "vipB=MY_VIPB")
# ani dnis to limit calls from ani or to dnis. /b means replicated 
modparam("dialog", "profiles_with_value", "dnis/b;ani/b")

#I called the lookup like this

route("limit_dnis",$var(limit_dnis),2);
route("limit_ani",$var(limit_ani),2);


#The lookup route looks like this

route[limit_dnis]
{
# first add to the profile, just to avoid "test and set" false results
set_dlg_profile("dnis/b","$param(1)");


# do the actual test - see how many calls the user has so far
get_profile_size("dnis/b","$param(1)","$var(calls)");
xlog("$ft $Ri There are $var(calls) ongoing calls to  $param(1), limit is $param(2)\n");

# check within limit
if( $var(calls)>$param(2) )
{
xlog("$ft $Ri LIMIT_DNIS [$var(calls)/$param(2)]: From:$fU To:$param(1) Callid:$ci\n");
avp_db_query("insert into caztel.opensips_log(event,fulltext,linenr,digits,hostname) values ('LIMIT DNIS','$ft $Ri LIMIT_DNIS [$var(calls)/$param(2)]: From:$fU To:$param(1) Callid:$ci','$fU','$tU','MY_HOSTNAME')");
send_reply("486","DNIS LIMIT");
exit;
# terminating this call will automatically remove the call from the profile
}

# call was added to the profile without exceeding the limit, simply continue
}


route[limit_ani]
{
# first add to the profile, just to avoid "test and set" false results
set_dlg_profile("ani/b","$param(1)");


# do the actual test - see how many calls the user has so far
get_profile_size("ani/b","$param(1)","$var(calls_ani)");
xlog("$ft $Ri User $param(1) has $var(calls_ani) ongoing calls so far, limit is $param(2)\n");


# check within limit
if( $var(calls_ani)>$param(2) )
{
xlog("$ft $Ri LIMIT_ANI [$var(calls_ani)/$param(2)]: From:$param(1) To:$tU Callid:$ci\n");
avp_db_query("insert into caztel.opensips_log(event,fulltext,linenr,digits,hostname) values ('LIMIT ANI','$ft $Ri LIMIT_ANI [$var(calls_ani)/$param(2)]: From:$param(1) To:$tU Callid:$ci','$fU','$tU','MY_HOSTNAME')");
send_reply("486", "ANI LIMIT\n");
exit;
# terminating this call will automatically remove the call from the profile
}

# call was added to the profile without exceeding the limit, simply continue
}


from the log: 

Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: 20138040-1546 10.10.10.109 There are 1 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: 20138040-1546 10.10.10.109 User 4186880000 has 1 ongoing calls so far, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: CRITICAL:core:sig_usr: segfault in process pid: 5690, id: 11
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 Call from: caz5400. Outcarrierid = 24. <a href="sip:4186880000@10.10.10.118" class="">sip:4186880000@10.10.10.118 -> <a href="sip:24491232412323@10.10.10.109:5060" class="">sip:24491232412323@10.10.10.109:5060
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 There are 2 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 User 4186880000 has 2 ongoing calls so far, limit is 2
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 Call from: caz5400. Outcarrierid = 24. <a href="sip:4186880000@10.10.10.118" class="">sip:4186880000@10.10.10.118 -> <a href="sip:24491232412323@10.10.10.109:5060" class="">sip:24491232412323@10.10.10.109:5060
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 There are 3 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 LIMIT_DNIS [3/2]: From:4186880000 To:491232412323 Callid:[hidden email]
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5749]: CRITICAL:core:handle_worker: dead child 11 (EOF received), pid 5690
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: INFO:core:handle_sigs: child process 5690 exited by a signal 11
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: INFO:core:handle_sigs: core was generated
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: INFO:core:handle_sigs: terminating due to SIGCHLD

backtrace

GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/opensips...(no debugging symbols found)...done.
[New LWP 5690]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /run/opensips/opensips.pid -f /etc/opensips/opensips.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f5afe73b35c in fetch_dlg_value () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
(gdb) bt full
#0  0x00007f5afe73b35c in fetch_dlg_value () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#1  0x00007f5afe78022b in get_shtag_state () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#2  0x00007f5afe742a05 in get_profile_size () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#3  0x00007f5afda28022 in lb_route () from /usr/lib/x86_64-linux-gnu/opensips/modules/load_balancer.so
No symbol table info available.
#4  0x00007f5afda1ef35 in ?? () from /usr/lib/x86_64-linux-gnu/opensips/modules/load_balancer.so
No symbol table info available.
#5  0x0000000000433585 in do_action ()
No symbol table info available.
#6  0x0000000000438390 in run_action_list ()
No symbol table info available.
#7  0x0000000000462f3c in eval_expr ()
No symbol table info available.
#8  0x0000000000462b89 in eval_expr ()
No symbol table info available.
#9  0x0000000000462b33 in eval_expr ()
No symbol table info available.
#10 0x0000000000433526 in do_action ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#11 0x0000000000438390 in run_action_list ()
No symbol table info available.
#12 0x00000000004384ba in ?? ()
No symbol table info available.
#13 0x000000000043594b in do_action ()
No symbol table info available.
#14 0x0000000000438390 in run_action_list ()
No symbol table info available.
#15 0x0000000000436b3c in do_action ()
No symbol table info available.
#16 0x0000000000438390 in run_action_list ()
No symbol table info available.
#17 0x0000000000438648 in run_top_route ()
No symbol table info available.
#18 0x000000000043dd28 in receive_msg ()
No symbol table info available.
#19 0x000000000052e7d8 in ?? ()
No symbol table info available.
#20 0x0000000000519ad9 in udp_start_processes ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#21 0x000000000041c5a6 in main ()
No symbol table info available.
(gdb)
---------------------------------------------------
Jennifer Akemi Hashimoto
Caztel Communications
[hidden email]
905-836-5445 


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: core dump in dialog

Bogdan-Andrei Iancu-2
Hi Jennifer,

The crash seems to happen when checking the sharing tag used for dialog replication. Maybe you could install the debug symbols (it is a separate package) to get more information in the backtrace.

As a note, if you want to only do sharing profiles, you do not have to do dialog sharing also - the two (dialog sharing and profile sharing) are completely independent.

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/24/2018 10:33 PM, Jennifer Hashimoto wrote:
Hi guys,
I’m getting ready to switch to using opensips 2.4 but as soon as i put traffic on it (like 3 calls, not a lot) it crashed.
it happened twice - once when I didn’t have the dump cores on, so I turned it on, then started and it crashed almost immediately after that

I thought maybe it was because it did a insert on my postgres DB which doesn’t happen every call, 

It’s really strange because when i try test calls it seems fine, but real traffic it doesn’t’ like :(

 
I think it’s part of where i do lookups for limit like this tutorial: https://www.opensips.org/Documentation/Tutorials-ConcurrentCallsLimitation


version: opensips 2.4.2 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on  with gcc 4.9.2

Here are some relevant bits from my opensips.cfg

loadmodule "db_virtual.so"
modparam("db_virtual", "db_urls", "define debdb FAILOVER")
modparam("db_virtual", "db_urls", "<a href="postgres://opensips:" class="" moz-do-not-send="true">postgres://opensips:passwd@192.168.5.65/opensips_cazcore")
modparam("db_virtual", "db_urls", "<a href="postgres://opensips:" class="" moz-do-not-send="true">postgres://opensips:passwd@192.168.6.65/opensips_cazcore")

#### DIALOG module
loadmodule "dialog.so"
modparam("dialog", "dlg_match_mode", 1)
modparam("dialog", "default_timeout", 21600)  # 6 hours timeout
modparam("dialog", "db_mode", 1)  
modparam("dialog", "db_update_period", 5)  

#modparam("dialog", "db_url", "<a href="postgres://opensips:passwd@192.168.5.65/opensips_cazcore" class="" moz-do-not-send="true">postgres://opensips:passwd@192.168.5.65/opensips_cazcore") 
modparam("dialog", "db_url", "<a href="virtual://debdb" class="" moz-do-not-send="true">virtual://debdb") 
modparam("dialog", "dialog_replication_cluster", 1)
modparam("dialog", "profile_replication_cluster", 1)
#tags for VIP handling in cluster
modparam("dialog", "dlg_sharing_tag", "vipA=MY_VIPA")
modparam("dialog", "dlg_sharing_tag", "vipB=MY_VIPB")
# ani dnis to limit calls from ani or to dnis. /b means replicated 
modparam("dialog", "profiles_with_value", "dnis/b;ani/b")

#I called the lookup like this

route("limit_dnis",$var(limit_dnis),2);
route("limit_ani",$var(limit_ani),2);


#The lookup route looks like this

route[limit_dnis]
{
# first add to the profile, just to avoid "test and set" false results
set_dlg_profile("dnis/b","$param(1)");


# do the actual test - see how many calls the user has so far
get_profile_size("dnis/b","$param(1)","$var(calls)");
xlog("$ft $Ri There are $var(calls) ongoing calls to  $param(1), limit is $param(2)\n");

# check within limit
if( $var(calls)>$param(2) )
{
xlog("$ft $Ri LIMIT_DNIS [$var(calls)/$param(2)]: From:$fU To:$param(1) Callid:$ci\n");
avp_db_query("insert into caztel.opensips_log(event,fulltext,linenr,digits,hostname) values ('LIMIT DNIS','$ft $Ri LIMIT_DNIS [$var(calls)/$param(2)]: From:$fU To:$param(1) Callid:$ci','$fU','$tU','MY_HOSTNAME')");
send_reply("486","DNIS LIMIT");
exit;
# terminating this call will automatically remove the call from the profile
}

# call was added to the profile without exceeding the limit, simply continue
}


route[limit_ani]
{
# first add to the profile, just to avoid "test and set" false results
set_dlg_profile("ani/b","$param(1)");


# do the actual test - see how many calls the user has so far
get_profile_size("ani/b","$param(1)","$var(calls_ani)");
xlog("$ft $Ri User $param(1) has $var(calls_ani) ongoing calls so far, limit is $param(2)\n");


# check within limit
if( $var(calls_ani)>$param(2) )
{
xlog("$ft $Ri LIMIT_ANI [$var(calls_ani)/$param(2)]: From:$param(1) To:$tU Callid:$ci\n");
avp_db_query("insert into caztel.opensips_log(event,fulltext,linenr,digits,hostname) values ('LIMIT ANI','$ft $Ri LIMIT_ANI [$var(calls_ani)/$param(2)]: From:$param(1) To:$tU Callid:$ci','$fU','$tU','MY_HOSTNAME')");
send_reply("486", "ANI LIMIT\n");
exit;
# terminating this call will automatically remove the call from the profile
}

# call was added to the profile without exceeding the limit, simply continue
}


from the log: 

Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: 20138040-1546 10.10.10.109 There are 1 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: 20138040-1546 10.10.10.109 User 4186880000 has 1 ongoing calls so far, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: CRITICAL:core:sig_usr: segfault in process pid: 5690, id: 11
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 Call from: caz5400. Outcarrierid = 24. <a href="sip:4186880000@10.10.10.118" class="" moz-do-not-send="true">sip:4186880000@10.10.10.118 -> <a href="sip:24491232412323@10.10.10.109:5060" class="" moz-do-not-send="true">sip:24491232412323@10.10.10.109:5060
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 There are 2 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 User 4186880000 has 2 ongoing calls so far, limit is 2
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 Call from: caz5400. Outcarrierid = 24. <a href="sip:4186880000@10.10.10.118" class="" moz-do-not-send="true">sip:4186880000@10.10.10.118 -> <a href="sip:24491232412323@10.10.10.109:5060" class="" moz-do-not-send="true">sip:24491232412323@10.10.10.109:5060
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 There are 3 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 LIMIT_DNIS [3/2]: From:4186880000 To:491232412323 Callid:[hidden email]
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5749]: CRITICAL:core:handle_worker: dead child 11 (EOF received), pid 5690
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: <a class="moz-txt-link-freetext" href="INFO:core:handle_sigs">INFO:core:handle_sigs: child process 5690 exited by a signal 11
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: <a class="moz-txt-link-freetext" href="INFO:core:handle_sigs">INFO:core:handle_sigs: core was generated
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: <a class="moz-txt-link-freetext" href="INFO:core:handle_sigs">INFO:core:handle_sigs: terminating due to SIGCHLD

backtrace

GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/opensips...(no debugging symbols found)...done.
[New LWP 5690]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /run/opensips/opensips.pid -f /etc/opensips/opensips.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f5afe73b35c in fetch_dlg_value () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
(gdb) bt full
#0  0x00007f5afe73b35c in fetch_dlg_value () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#1  0x00007f5afe78022b in get_shtag_state () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#2  0x00007f5afe742a05 in get_profile_size () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#3  0x00007f5afda28022 in lb_route () from /usr/lib/x86_64-linux-gnu/opensips/modules/load_balancer.so
No symbol table info available.
#4  0x00007f5afda1ef35 in ?? () from /usr/lib/x86_64-linux-gnu/opensips/modules/load_balancer.so
No symbol table info available.
#5  0x0000000000433585 in do_action ()
No symbol table info available.
#6  0x0000000000438390 in run_action_list ()
No symbol table info available.
#7  0x0000000000462f3c in eval_expr ()
No symbol table info available.
#8  0x0000000000462b89 in eval_expr ()
No symbol table info available.
#9  0x0000000000462b33 in eval_expr ()
No symbol table info available.
#10 0x0000000000433526 in do_action ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#11 0x0000000000438390 in run_action_list ()
No symbol table info available.
#12 0x00000000004384ba in ?? ()
No symbol table info available.
#13 0x000000000043594b in do_action ()
No symbol table info available.
#14 0x0000000000438390 in run_action_list ()
No symbol table info available.
#15 0x0000000000436b3c in do_action ()
No symbol table info available.
#16 0x0000000000438390 in run_action_list ()
No symbol table info available.
#17 0x0000000000438648 in run_top_route ()
No symbol table info available.
#18 0x000000000043dd28 in receive_msg ()
No symbol table info available.
#19 0x000000000052e7d8 in ?? ()
No symbol table info available.
#20 0x0000000000519ad9 in udp_start_processes ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#21 0x000000000041c5a6 in main ()
No symbol table info available.
(gdb)
---------------------------------------------------
Jennifer Akemi Hashimoto
Caztel Communications
[hidden email]
905-836-5445 



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: core dump in dialog

Jennifer Hashimoto
Thanks Bogdan,

I will check out the debug symbols. Per your advice that I don’t need to share both profiles and dialogs, I turned off profile sharing since I do want dialog replication for the master/master aspect of stuff. Turning off profile replication fixed the problem and now it is running smoothly.

Jennifer

---------------------------------------------------
Jennifer Akemi Hashimoto
Caztel Communications
[hidden email]
905-836-5445 

On Oct 26, 2018, at 2:29 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:

Hi Jennifer,

The crash seems to happen when checking the sharing tag used for dialog replication. Maybe you could install the debug symbols (it is a separate package) to get more information in the backtrace.

As a note, if you want to only do sharing profiles, you do not have to do dialog sharing also - the two (dialog sharing and profile sharing) are completely independent.

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/24/2018 10:33 PM, Jennifer Hashimoto wrote:
Hi guys,
I’m getting ready to switch to using opensips 2.4 but as soon as i put traffic on it (like 3 calls, not a lot) it crashed.
it happened twice - once when I didn’t have the dump cores on, so I turned it on, then started and it crashed almost immediately after that

I thought maybe it was because it did a insert on my postgres DB which doesn’t happen every call, 

It’s really strange because when i try test calls it seems fine, but real traffic it doesn’t’ like :(

 
I think it’s part of where i do lookups for limit like this tutorial: https://www.opensips.org/Documentation/Tutorials-ConcurrentCallsLimitation


version: opensips 2.4.2 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on  with gcc 4.9.2

Here are some relevant bits from my opensips.cfg

loadmodule "db_virtual.so"
modparam("db_virtual", "db_urls", "define debdb FAILOVER")
modparam("db_virtual", "db_urls", "<a href="postgres://opensips" class="" moz-do-not-send="true">postgres://opensips:passwd@192.168.5.65/opensips_cazcore")
modparam("db_virtual", "db_urls", "<a href="postgres://opensips" class="" moz-do-not-send="true">postgres://opensips:passwd@192.168.6.65/opensips_cazcore")

#### DIALOG module
loadmodule "dialog.so"
modparam("dialog", "dlg_match_mode", 1)
modparam("dialog", "default_timeout", 21600)  # 6 hours timeout
modparam("dialog", "db_mode", 1)  
modparam("dialog", "db_update_period", 5)  

#modparam("dialog", "db_url", "<a href="postgres://opensips:passwd@192.168.5.65/opensips_cazcore" class="" moz-do-not-send="true">postgres://opensips:passwd@192.168.5.65/opensips_cazcore") 
modparam("dialog", "db_url", "<a href="virtual://debdb" class="" moz-do-not-send="true">virtual://debdb") 
modparam("dialog", "dialog_replication_cluster", 1)
modparam("dialog", "profile_replication_cluster", 1)
#tags for VIP handling in cluster
modparam("dialog", "dlg_sharing_tag", "vipA=MY_VIPA")
modparam("dialog", "dlg_sharing_tag", "vipB=MY_VIPB")
# ani dnis to limit calls from ani or to dnis. /b means replicated 
modparam("dialog", "profiles_with_value", "dnis/b;ani/b")

#I called the lookup like this

route("limit_dnis",$var(limit_dnis),2);
route("limit_ani",$var(limit_ani),2);


#The lookup route looks like this

route[limit_dnis]
{
# first add to the profile, just to avoid "test and set" false results
set_dlg_profile("dnis/b","$param(1)");


# do the actual test - see how many calls the user has so far
get_profile_size("dnis/b","$param(1)","$var(calls)");
xlog("$ft $Ri There are $var(calls) ongoing calls to  $param(1), limit is $param(2)\n");

# check within limit
if( $var(calls)>$param(2) )
{
xlog("$ft $Ri LIMIT_DNIS [$var(calls)/$param(2)]: From:$fU To:$param(1) Callid:$ci\n");
avp_db_query("insert into caztel.opensips_log(event,fulltext,linenr,digits,hostname) values ('LIMIT DNIS','$ft $Ri LIMIT_DNIS [$var(calls)/$param(2)]: From:$fU To:$param(1) Callid:$ci','$fU','$tU','MY_HOSTNAME')");
send_reply("486","DNIS LIMIT");
exit;
# terminating this call will automatically remove the call from the profile
}

# call was added to the profile without exceeding the limit, simply continue
}


route[limit_ani]
{
# first add to the profile, just to avoid "test and set" false results
set_dlg_profile("ani/b","$param(1)");


# do the actual test - see how many calls the user has so far
get_profile_size("ani/b","$param(1)","$var(calls_ani)");
xlog("$ft $Ri User $param(1) has $var(calls_ani) ongoing calls so far, limit is $param(2)\n");


# check within limit
if( $var(calls_ani)>$param(2) )
{
xlog("$ft $Ri LIMIT_ANI [$var(calls_ani)/$param(2)]: From:$param(1) To:$tU Callid:$ci\n");
avp_db_query("insert into caztel.opensips_log(event,fulltext,linenr,digits,hostname) values ('LIMIT ANI','$ft $Ri LIMIT_ANI [$var(calls_ani)/$param(2)]: From:$param(1) To:$tU Callid:$ci','$fU','$tU','MY_HOSTNAME')");
send_reply("486", "ANI LIMIT\n");
exit;
# terminating this call will automatically remove the call from the profile
}

# call was added to the profile without exceeding the limit, simply continue
}


from the log: 

Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: 20138040-1546 10.10.10.109 There are 1 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: 20138040-1546 10.10.10.109 User 4186880000 has 1 ongoing calls so far, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: CRITICAL:core:sig_usr: segfault in process pid: 5690, id: 11
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 Call from: caz5400. Outcarrierid = 24. <a href="sip:4186880000@10.10.10.118" class="" moz-do-not-send="true">sip:4186880000@10.10.10.118 -> <a href="sip:24491232412323@10.10.10.109:5060" class="" moz-do-not-send="true">sip:24491232412323@10.10.10.109:5060
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 There are 2 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 User 4186880000 has 2 ongoing calls so far, limit is 2
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 Call from: caz5400. Outcarrierid = 24. <a href="sip:4186880000@10.10.10.118" class="" moz-do-not-send="true">sip:4186880000@10.10.10.118 -> <a href="sip:24491232412323@10.10.10.109:5060" class="" moz-do-not-send="true">sip:24491232412323@10.10.10.109:5060
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 There are 3 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 LIMIT_DNIS [3/2]: From:4186880000 To:491232412323 Callid:[hidden email]
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5749]: CRITICAL:core:handle_worker: dead child 11 (EOF received), pid 5690
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: <a class="moz-txt-link-freetext" href="info:core:handle_sigs">INFO:core:handle_sigs: child process 5690 exited by a signal 11
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: <a class="moz-txt-link-freetext" href="info:core:handle_sigs">INFO:core:handle_sigs: core was generated
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: <a class="moz-txt-link-freetext" href="info:core:handle_sigs">INFO:core:handle_sigs: terminating due to SIGCHLD

backtrace

GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/opensips...(no debugging symbols found)...done.
[New LWP 5690]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /run/opensips/opensips.pid -f /etc/opensips/opensips.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f5afe73b35c in fetch_dlg_value () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
(gdb) bt full
#0  0x00007f5afe73b35c in fetch_dlg_value () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#1  0x00007f5afe78022b in get_shtag_state () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#2  0x00007f5afe742a05 in get_profile_size () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#3  0x00007f5afda28022 in lb_route () from /usr/lib/x86_64-linux-gnu/opensips/modules/load_balancer.so
No symbol table info available.
#4  0x00007f5afda1ef35 in ?? () from /usr/lib/x86_64-linux-gnu/opensips/modules/load_balancer.so
No symbol table info available.
#5  0x0000000000433585 in do_action ()
No symbol table info available.
#6  0x0000000000438390 in run_action_list ()
No symbol table info available.
#7  0x0000000000462f3c in eval_expr ()
No symbol table info available.
#8  0x0000000000462b89 in eval_expr ()
No symbol table info available.
#9  0x0000000000462b33 in eval_expr ()
No symbol table info available.
#10 0x0000000000433526 in do_action ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#11 0x0000000000438390 in run_action_list ()
No symbol table info available.
#12 0x00000000004384ba in ?? ()
No symbol table info available.
#13 0x000000000043594b in do_action ()
No symbol table info available.
#14 0x0000000000438390 in run_action_list ()
No symbol table info available.
#15 0x0000000000436b3c in do_action ()
No symbol table info available.
#16 0x0000000000438390 in run_action_list ()
No symbol table info available.
#17 0x0000000000438648 in run_top_route ()
No symbol table info available.
#18 0x000000000043dd28 in receive_msg ()
No symbol table info available.
#19 0x000000000052e7d8 in ?? ()
No symbol table info available.
#20 0x0000000000519ad9 in udp_start_processes ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#21 0x000000000041c5a6 in main ()
No symbol table info available.
(gdb)
---------------------------------------------------
Jennifer Akemi Hashimoto
Caztel Communications
[hidden email]
905-836-5445 



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: core dump in dialog

Bogdan-Andrei Iancu-2
Hi Jennifer,

OK, I'm glad you found a way to keep things under control - it seems the crash may be more related to the profile replication. Nevertheless, please post the full backtrace after installing the dbg symbols.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/30/2018 09:38 PM, Jennifer Hashimoto wrote:
Thanks Bogdan,

I will check out the debug symbols. Per your advice that I don’t need to share both profiles and dialogs, I turned off profile sharing since I do want dialog replication for the master/master aspect of stuff. Turning off profile replication fixed the problem and now it is running smoothly.

Jennifer

---------------------------------------------------
Jennifer Akemi Hashimoto
Caztel Communications
[hidden email]
905-836-5445 

On Oct 26, 2018, at 2:29 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:

Hi Jennifer,

The crash seems to happen when checking the sharing tag used for dialog replication. Maybe you could install the debug symbols (it is a separate package) to get more information in the backtrace.

As a note, if you want to only do sharing profiles, you do not have to do dialog sharing also - the two (dialog sharing and profile sharing) are completely independent.

Best regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/24/2018 10:33 PM, Jennifer Hashimoto wrote:
Hi guys,
I’m getting ready to switch to using opensips 2.4 but as soon as i put traffic on it (like 3 calls, not a lot) it crashed.
it happened twice - once when I didn’t have the dump cores on, so I turned it on, then started and it crashed almost immediately after that

I thought maybe it was because it did a insert on my postgres DB which doesn’t happen every call, 

It’s really strange because when i try test calls it seems fine, but real traffic it doesn’t’ like :(

 
I think it’s part of where i do lookups for limit like this tutorial: https://www.opensips.org/Documentation/Tutorials-ConcurrentCallsLimitation


version: opensips 2.4.2 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on  with gcc 4.9.2

Here are some relevant bits from my opensips.cfg

loadmodule "db_virtual.so"
modparam("db_virtual", "db_urls", "define debdb FAILOVER")
modparam("db_virtual", "db_urls", "<a href="postgres://opensips" class="" moz-do-not-send="true">postgres://opensips:passwd@192.168.5.65/opensips_cazcore")
modparam("db_virtual", "db_urls", "<a href="postgres://opensips" class="" moz-do-not-send="true">postgres://opensips:passwd@192.168.6.65/opensips_cazcore")

#### DIALOG module
loadmodule "dialog.so"
modparam("dialog", "dlg_match_mode", 1)
modparam("dialog", "default_timeout", 21600)  # 6 hours timeout
modparam("dialog", "db_mode", 1)  
modparam("dialog", "db_update_period", 5)  

#modparam("dialog", "db_url", "<a href="postgres://opensips:passwd@192.168.5.65/opensips_cazcore" class="" moz-do-not-send="true">postgres://opensips:passwd@192.168.5.65/opensips_cazcore") 
modparam("dialog", "db_url", "<a href="virtual://debdb" class="" moz-do-not-send="true">virtual://debdb") 
modparam("dialog", "dialog_replication_cluster", 1)
modparam("dialog", "profile_replication_cluster", 1)
#tags for VIP handling in cluster
modparam("dialog", "dlg_sharing_tag", "vipA=MY_VIPA")
modparam("dialog", "dlg_sharing_tag", "vipB=MY_VIPB")
# ani dnis to limit calls from ani or to dnis. /b means replicated 
modparam("dialog", "profiles_with_value", "dnis/b;ani/b")

#I called the lookup like this

route("limit_dnis",$var(limit_dnis),2);
route("limit_ani",$var(limit_ani),2);


#The lookup route looks like this

route[limit_dnis]
{
# first add to the profile, just to avoid "test and set" false results
set_dlg_profile("dnis/b","$param(1)");


# do the actual test - see how many calls the user has so far
get_profile_size("dnis/b","$param(1)","$var(calls)");
xlog("$ft $Ri There are $var(calls) ongoing calls to  $param(1), limit is $param(2)\n");

# check within limit
if( $var(calls)>$param(2) )
{
xlog("$ft $Ri LIMIT_DNIS [$var(calls)/$param(2)]: From:$fU To:$param(1) Callid:$ci\n");
avp_db_query("insert into caztel.opensips_log(event,fulltext,linenr,digits,hostname) values ('LIMIT DNIS','$ft $Ri LIMIT_DNIS [$var(calls)/$param(2)]: From:$fU To:$param(1) Callid:$ci','$fU','$tU','MY_HOSTNAME')");
send_reply("486","DNIS LIMIT");
exit;
# terminating this call will automatically remove the call from the profile
}

# call was added to the profile without exceeding the limit, simply continue
}


route[limit_ani]
{
# first add to the profile, just to avoid "test and set" false results
set_dlg_profile("ani/b","$param(1)");


# do the actual test - see how many calls the user has so far
get_profile_size("ani/b","$param(1)","$var(calls_ani)");
xlog("$ft $Ri User $param(1) has $var(calls_ani) ongoing calls so far, limit is $param(2)\n");


# check within limit
if( $var(calls_ani)>$param(2) )
{
xlog("$ft $Ri LIMIT_ANI [$var(calls_ani)/$param(2)]: From:$param(1) To:$tU Callid:$ci\n");
avp_db_query("insert into caztel.opensips_log(event,fulltext,linenr,digits,hostname) values ('LIMIT ANI','$ft $Ri LIMIT_ANI [$var(calls_ani)/$param(2)]: From:$param(1) To:$tU Callid:$ci','$fU','$tU','MY_HOSTNAME')");
send_reply("486", "ANI LIMIT\n");
exit;
# terminating this call will automatically remove the call from the profile
}

# call was added to the profile without exceeding the limit, simply continue
}


from the log: 

Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: 20138040-1546 10.10.10.109 There are 1 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: 20138040-1546 10.10.10.109 User 4186880000 has 1 ongoing calls so far, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5690]: CRITICAL:core:sig_usr: segfault in process pid: 5690, id: 11
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 Call from: caz5400. Outcarrierid = 24. <a href="sip:4186880000@10.10.10.118" class="" moz-do-not-send="true">sip:4186880000@10.10.10.118 -> <a href="sip:24491232412323@10.10.10.109:5060" class="" moz-do-not-send="true">sip:24491232412323@10.10.10.109:5060
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 There are 2 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:29 CazcoreA /usr/sbin/opensips[5688]: 20138040-1546 10.10.10.109 User 4186880000 has 2 ongoing calls so far, limit is 2
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 Call from: caz5400. Outcarrierid = 24. <a href="sip:4186880000@10.10.10.118" class="" moz-do-not-send="true">sip:4186880000@10.10.10.118 -> <a href="sip:24491232412323@10.10.10.109:5060" class="" moz-do-not-send="true">sip:24491232412323@10.10.10.109:5060
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 There are 3 ongoing calls to  491232412323, limit is 2
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5686]: 20138040-1546 10.10.10.109 LIMIT_DNIS [3/2]: From:4186880000 To:491232412323 Callid:[hidden email]
Oct 24 12:21:30 CazcoreA /usr/sbin/opensips[5749]: CRITICAL:core:handle_worker: dead child 11 (EOF received), pid 5690
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: <a class="moz-txt-link-freetext" href="info:core:handle_sigs" moz-do-not-send="true">INFO:core:handle_sigs: child process 5690 exited by a signal 11
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: <a class="moz-txt-link-freetext" href="info:core:handle_sigs" moz-do-not-send="true">INFO:core:handle_sigs: core was generated
Oct 24 12:21:32 CazcoreA /usr/sbin/opensips[5678]: <a class="moz-txt-link-freetext" href="info:core:handle_sigs" moz-do-not-send="true">INFO:core:handle_sigs: terminating due to SIGCHLD

backtrace

GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/opensips...(no debugging symbols found)...done.
[New LWP 5690]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /run/opensips/opensips.pid -f /etc/opensips/opensips.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f5afe73b35c in fetch_dlg_value () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
(gdb) bt full
#0  0x00007f5afe73b35c in fetch_dlg_value () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#1  0x00007f5afe78022b in get_shtag_state () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#2  0x00007f5afe742a05 in get_profile_size () from /usr/lib/x86_64-linux-gnu/opensips/modules/dialog.so
No symbol table info available.
#3  0x00007f5afda28022 in lb_route () from /usr/lib/x86_64-linux-gnu/opensips/modules/load_balancer.so
No symbol table info available.
#4  0x00007f5afda1ef35 in ?? () from /usr/lib/x86_64-linux-gnu/opensips/modules/load_balancer.so
No symbol table info available.
#5  0x0000000000433585 in do_action ()
No symbol table info available.
#6  0x0000000000438390 in run_action_list ()
No symbol table info available.
#7  0x0000000000462f3c in eval_expr ()
No symbol table info available.
#8  0x0000000000462b89 in eval_expr ()
No symbol table info available.
#9  0x0000000000462b33 in eval_expr ()
No symbol table info available.
#10 0x0000000000433526 in do_action ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#11 0x0000000000438390 in run_action_list ()
No symbol table info available.
#12 0x00000000004384ba in ?? ()
No symbol table info available.
#13 0x000000000043594b in do_action ()
No symbol table info available.
#14 0x0000000000438390 in run_action_list ()
No symbol table info available.
#15 0x0000000000436b3c in do_action ()
No symbol table info available.
#16 0x0000000000438390 in run_action_list ()
No symbol table info available.
#17 0x0000000000438648 in run_top_route ()
No symbol table info available.
#18 0x000000000043dd28 in receive_msg ()
No symbol table info available.
#19 0x000000000052e7d8 in ?? ()
No symbol table info available.
#20 0x0000000000519ad9 in udp_start_processes ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#21 0x000000000041c5a6 in main ()
No symbol table info available.
(gdb)
---------------------------------------------------
Jennifer Akemi Hashimoto
Caztel Communications
[hidden email]
905-836-5445 



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users