opensips will not start while loading huge blacklist

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

opensips will not start while loading huge blacklist

basitengr
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

basitengr
Hi team,

I am thinking to do horizontal scaling through dispatcher module and sharding the blacklist to 2 or 3 opensips nodes local cache.
or using redis for blacklist repository but local cache is even more quick. 

Any comments or experience to share about the problem I am facing currently?

--
regards,

abdul basit


On Fri, 28 Sep 2018 at 06:10, Abdul Basit <[hidden email]> wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

Jon Abrams
I have used memached to store 20-30 million entry blacklists while maintaining 6000-9000 responses per second on a dipping server. The downside is its not persistent.

I've used redis as well to store the whole North American LNP database in memory, but I've only tried pushing it to a couple thousand CPS so far. I would imagine it would perform at least as well as the memcached if needed.

I wouldn't worry about the speed difference between local cache speed and using Redis/Memcached over a socket in a normal deployment scenario.

- Jon

On Sat, Sep 29, 2018 at 6:17 PM Abdul Basit <[hidden email]> wrote:
Hi team,

I am thinking to do horizontal scaling through dispatcher module and sharding the blacklist to 2 or 3 opensips nodes local cache.
or using redis for blacklist repository but local cache is even more quick. 

Any comments or experience to share about the problem I am facing currently?

--
regards,

abdul basit


On Fri, 28 Sep 2018 at 06:10, Abdul Basit <[hidden email]> wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

basitengr
Hi Jon,

Good to know.
If possible, can you share the settings you have for opensips?
Whats the server hardware specification?
Whats opensips service startup time?

--
regards,

abdul basit

On Sun, 30 Sep 2018 at 05:17, Jon Abrams <[hidden email]> wrote:
I have used memached to store 20-30 million entry blacklists while maintaining 6000-9000 responses per second on a dipping server. The downside is its not persistent.

I've used redis as well to store the whole North American LNP database in memory, but I've only tried pushing it to a couple thousand CPS so far. I would imagine it would perform at least as well as the memcached if needed.

I wouldn't worry about the speed difference between local cache speed and using Redis/Memcached over a socket in a normal deployment scenario.

- Jon

On Sat, Sep 29, 2018 at 6:17 PM Abdul Basit <[hidden email]> wrote:
Hi team,

I am thinking to do horizontal scaling through dispatcher module and sharding the blacklist to 2 or 3 opensips nodes local cache.
or using redis for blacklist repository but local cache is even more quick. 

Any comments or experience to share about the problem I am facing currently?

--
regards,

abdul basit


On Fri, 28 Sep 2018 at 06:10, Abdul Basit <[hidden email]> wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

Jon Abrams
On the high CPS box its an old OpenSIPs 1.9 instance running 64 children IIRC on a pair of old 6 core Westmere Core Xeons. On a 2.2 box the code looks sort of like this:

# memcached config
modparam("cachedb_memcached", "cachedb_url", "memcached:blacklist://127.0.0.1:11211/")


# in the main route block
if (cache_fetch("memcached: blacklist ", "$avp(key)", $avp(value)))       
{
   #reject the call or something
}

The only other possible downside to memcached is the lack of longest prefix match, which you may not need. You also need to build the Memcached.

I don't have any OpenSIPs Redis code samples handy, as I think I've only used it with that other open source SIP server.

Since the data is not being loaded into OpenSIPs memory, startup times are considerably faster. With memcached you need to remember or script it to reload the blacklist into memory on reboot or restart of memcached.

With Redis you can persist it, although make sure check the performance hit when you turn on persistence. AOF isn't bad, especially with a SSD disk. The background save can add a bunch of latency when it kicks off.
If its a lot of data, I usually do a BGSAVE once a day during off-peak times with a cron job. But I don't think 20-30 million entries will cause much disturbance.

- Jon

On Sun, Sep 30, 2018 at 9:13 AM Abdul Basit <[hidden email]> wrote:
Hi Jon,

Good to know.
If possible, can you share the settings you have for opensips?
Whats the server hardware specification?
Whats opensips service startup time?

--
regards,

abdul basit

On Sun, 30 Sep 2018 at 05:17, Jon Abrams <[hidden email]> wrote:
I have used memached to store 20-30 million entry blacklists while maintaining 6000-9000 responses per second on a dipping server. The downside is its not persistent.

I've used redis as well to store the whole North American LNP database in memory, but I've only tried pushing it to a couple thousand CPS so far. I would imagine it would perform at least as well as the memcached if needed.

I wouldn't worry about the speed difference between local cache speed and using Redis/Memcached over a socket in a normal deployment scenario.

- Jon

On Sat, Sep 29, 2018 at 6:17 PM Abdul Basit <[hidden email]> wrote:
Hi team,

I am thinking to do horizontal scaling through dispatcher module and sharding the blacklist to 2 or 3 opensips nodes local cache.
or using redis for blacklist repository but local cache is even more quick. 

Any comments or experience to share about the problem I am facing currently?

--
regards,

abdul basit


On Fri, 28 Sep 2018 at 06:10, Abdul Basit <[hidden email]> wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

basitengr
Hi Jon,

Thanks, I'll check it out with memcached and Redis both. 
Current disturbance is opensips restart due to localcache loading from mysql. Once loaded, it works like a charm.

Any idea about -m and -M vales? How much RAM was allocated to the box?

Opensips performed better when I increased shared and package memory. 
I want to avoid overkill of resources.

--
regards,

abdul basit

On Sun, 30 Sep 2018 at 20:00, Jon Abrams <[hidden email]> wrote:
On the high CPS box its an old OpenSIPs 1.9 instance running 64 children IIRC on a pair of old 6 core Westmere Core Xeons. On a 2.2 box the code looks sort of like this:

# memcached config
modparam("cachedb_memcached", "cachedb_url", "memcached:blacklist://127.0.0.1:11211/")


# in the main route block
if (cache_fetch("memcached: blacklist ", "$avp(key)", $avp(value)))       
{
   #reject the call or something
}

The only other possible downside to memcached is the lack of longest prefix match, which you may not need. You also need to build the Memcached.

I don't have any OpenSIPs Redis code samples handy, as I think I've only used it with that other open source SIP server.

Since the data is not being loaded into OpenSIPs memory, startup times are considerably faster. With memcached you need to remember or script it to reload the blacklist into memory on reboot or restart of memcached.

With Redis you can persist it, although make sure check the performance hit when you turn on persistence. AOF isn't bad, especially with a SSD disk. The background save can add a bunch of latency when it kicks off.
If its a lot of data, I usually do a BGSAVE once a day during off-peak times with a cron job. But I don't think 20-30 million entries will cause much disturbance.

- Jon

On Sun, Sep 30, 2018 at 9:13 AM Abdul Basit <[hidden email]> wrote:
Hi Jon,

Good to know.
If possible, can you share the settings you have for opensips?
Whats the server hardware specification?
Whats opensips service startup time?

--
regards,

abdul basit

On Sun, 30 Sep 2018 at 05:17, Jon Abrams <[hidden email]> wrote:
I have used memached to store 20-30 million entry blacklists while maintaining 6000-9000 responses per second on a dipping server. The downside is its not persistent.

I've used redis as well to store the whole North American LNP database in memory, but I've only tried pushing it to a couple thousand CPS so far. I would imagine it would perform at least as well as the memcached if needed.

I wouldn't worry about the speed difference between local cache speed and using Redis/Memcached over a socket in a normal deployment scenario.

- Jon

On Sat, Sep 29, 2018 at 6:17 PM Abdul Basit <[hidden email]> wrote:
Hi team,

I am thinking to do horizontal scaling through dispatcher module and sharding the blacklist to 2 or 3 opensips nodes local cache.
or using redis for blacklist repository but local cache is even more quick. 

Any comments or experience to share about the problem I am facing currently?

--
regards,

abdul basit


On Fri, 28 Sep 2018 at 06:10, Abdul Basit <[hidden email]> wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

Bogdan-Andrei Iancu-2
In reply to this post by basitengr
Hi Abdul,

IF I understand correctly, you use startup route to load (from DB?) some blacklists into localcache ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 09/28/2018 04:10 AM, Abdul Basit wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

basitengr
Yes. That's right.
The code I am using is as below. globalblacklist table has complete number as prefix with and whitelist status.   

route {
.....
if ( !check_blacklist("globalblacklist") ) {
xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not Allowed. Dropping call.\n");
sl_send_reply("403", "Forbidden");
exit;
}
else {
xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
route(10);
exit;
}
.....
}


--
regards,

abdul basit


On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Abdul,

IF I understand correctly, you use startup route to load (from DB?) some blacklists into localcache ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 09/28/2018 04:10 AM, Abdul Basit wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

Bogdan-Andrei Iancu-2
Hi Abdul,

OK, but what is the blacklist related operation you do in startup route ?? As far as I know, the blacklist module is not doing any kind of caching, so any check you do against the blacklists will translate into a DB query, right ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/01/2018 06:58 PM, Abdul Basit wrote:
Yes. That's right.
The code I am using is as below. globalblacklist table has complete number as prefix with and whitelist status.   

route {
.....
if ( !check_blacklist("globalblacklist") ) {
xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not Allowed. Dropping call.\n");
sl_send_reply("403", "Forbidden");
exit;
}
else {
xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
route(10);
exit;
}
.....
}


--
regards,

abdul basit


On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Abdul,

IF I understand correctly, you use startup route to load (from DB?) some blacklists into localcache ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 09/28/2018 04:10 AM, Abdul Basit wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

basitengr
Hi Bogdan,

I don't think so its translating to DB query on run time, instead its caching the blacklist.
I am saying so because opensips response is quick, where as when I query manually from DB, it take some time even prefix col. is indexed.
Moreover, while opensips restart i get following warnings in syslog: 

Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <nh-timer> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <dlg-reinvite-pinger> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <dlg-options-pinger> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <dlg-timer> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <tm-timer> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <blcore-expire> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 100 ms (now 18020 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 100 ms (now 18120 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 100 ms (now 18220 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: INFO:userblacklist:reload_sources: got 1386901 entries from 'globalblacklist' 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job: utimer job <tm-utimer> has a 18170000 us delay in execution 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job: timer job <nh-timer> has a 17280000 us delay in execution 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job: timer job <dlg-reinvite-pinger> has a 17280000 us delay in execution 

Once globalblacklist loaded 1386901 entries , opensips behave normally. 
When I will add more records to globalblacklist, opensips wouldn't start due to service timeout.
One way might be to increase service timeout to allow opensips keep loading the entries but it may take ages. 

--
regards,

abdul basit

On Tue, 2 Oct 2018 at 13:19, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Abdul,

OK, but what is the blacklist related operation you do in startup route ?? As far as I know, the blacklist module is not doing any kind of caching, so any check you do against the blacklists will translate into a DB query, right ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/01/2018 06:58 PM, Abdul Basit wrote:
Yes. That's right.
The code I am using is as below. globalblacklist table has complete number as prefix with and whitelist status.   

route {
.....
if ( !check_blacklist("globalblacklist") ) {
xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not Allowed. Dropping call.\n");
sl_send_reply("403", "Forbidden");
exit;
}
else {
xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
route(10);
exit;
}
.....
}


--
regards,

abdul basit


On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Abdul,

IF I understand correctly, you use startup route to load (from DB?) some blacklists into localcache ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 09/28/2018 04:10 AM, Abdul Basit wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

Jon Abrams
OpenSIPs is loading the global blacklist into a digit trie structure into shared memory on startup (or reload_blacklist command). That causes the startup delay.

- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <[hidden email]> wrote:
Hi Bogdan,

I don't think so its translating to DB query on run time, instead its caching the blacklist.
I am saying so because opensips response is quick, where as when I query manually from DB, it take some time even prefix col. is indexed.
Moreover, while opensips restart i get following warnings in syslog: 

Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <nh-timer> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <dlg-reinvite-pinger> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <dlg-options-pinger> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <dlg-timer> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <tm-timer> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <blcore-expire> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 100 ms (now 18020 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 100 ms (now 18120 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 100 ms (now 18220 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: INFO:userblacklist:reload_sources: got 1386901 entries from 'globalblacklist' 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job: utimer job <tm-utimer> has a 18170000 us delay in execution 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job: timer job <nh-timer> has a 17280000 us delay in execution 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job: timer job <dlg-reinvite-pinger> has a 17280000 us delay in execution 

Once globalblacklist loaded 1386901 entries , opensips behave normally. 
When I will add more records to globalblacklist, opensips wouldn't start due to service timeout.
One way might be to increase service timeout to allow opensips keep loading the entries but it may take ages. 

--
regards,

abdul basit

On Tue, 2 Oct 2018 at 13:19, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Abdul,

OK, but what is the blacklist related operation you do in startup route ?? As far as I know, the blacklist module is not doing any kind of caching, so any check you do against the blacklists will translate into a DB query, right ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/01/2018 06:58 PM, Abdul Basit wrote:
Yes. That's right.
The code I am using is as below. globalblacklist table has complete number as prefix with and whitelist status.   

route {
.....
if ( !check_blacklist("globalblacklist") ) {
xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not Allowed. Dropping call.\n");
sl_send_reply("403", "Forbidden");
exit;
}
else {
xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
route(10);
exit;
}
.....
}


--
regards,

abdul basit


On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Abdul,

IF I understand correctly, you use startup route to load (from DB?) some blacklists into localcache ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 09/28/2018 04:10 AM, Abdul Basit wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

Bogdan-Andrei Iancu-2
Hi,

Taking a closer look at the module (to be honest I do not user it, as drouting does a similar job) I see the module is loading the data into memory. It is doing this during the init stage, holding back the entire OpenSIPS startup.
Module doing similar heavy db load were changed to do the load post-startup. Let patch the module to decouple the loading from the startup, so that OpenSIPS will start without delays.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:04 PM, Jon Abrams wrote:
OpenSIPs is loading the global blacklist into a digit trie structure into shared memory on startup (or reload_blacklist command). That causes the startup delay.

- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <[hidden email]> wrote:
Hi Bogdan,

I don't think so its translating to DB query on run time, instead its caching the blacklist.
I am saying so because opensips response is quick, where as when I query manually from DB, it take some time even prefix col. is indexed.
Moreover, while opensips restart i get following warnings in syslog: 

Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <nh-timer> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <dlg-reinvite-pinger> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <dlg-options-pinger> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <dlg-timer> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <tm-timer> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer task <blcore-expire> already scheduled for 990 ms (now 17920 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 100 ms (now 18020 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 100 ms (now 18120 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 100 ms (now 18220 ms), it may overlap.. 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: <a class="moz-txt-link-freetext" href="INFO:userblacklist:reload_sources">INFO:userblacklist:reload_sources: got 1386901 entries from 'globalblacklist' 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job: utimer job <tm-utimer> has a 18170000 us delay in execution 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job: timer job <nh-timer> has a 17280000 us delay in execution 
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job: timer job <dlg-reinvite-pinger> has a 17280000 us delay in execution 

Once globalblacklist loaded 1386901 entries , opensips behave normally. 
When I will add more records to globalblacklist, opensips wouldn't start due to service timeout.
One way might be to increase service timeout to allow opensips keep loading the entries but it may take ages. 

--
regards,

abdul basit

On Tue, 2 Oct 2018 at 13:19, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Abdul,

OK, but what is the blacklist related operation you do in startup route ?? As far as I know, the blacklist module is not doing any kind of caching, so any check you do against the blacklists will translate into a DB query, right ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/01/2018 06:58 PM, Abdul Basit wrote:
Yes. That's right.
The code I am using is as below. globalblacklist table has complete number as prefix with and whitelist status.   

route {
.....
if ( !check_blacklist("globalblacklist") ) {
xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not Allowed. Dropping call.\n");
sl_send_reply("403", "Forbidden");
exit;
}
else {
xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
route(10);
exit;
}
.....
}


--
regards,

abdul basit


On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Abdul,

IF I understand correctly, you use startup route to load (from DB?) some blacklists into localcache ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 09/28/2018 04:10 AM, Abdul Basit wrote:
Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records loaded into local cache. 

While blacklist was growing upto 1.3m records, opensips startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to 6144 and package memory to 256.
Filter works fine here yet. Call load is not that much only blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m records in mysql. Opensips is not starting successfully and service startup requests timeout after huge CPU spikes while trying to populate local cache from mysql. 
I increased shared memory to 12288 and package memory to 1024 but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful startup. 

--
regards,

abdul basit


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

Bogdan-Andrei Iancu-2
OK, I pushed on master a patch to move the initial data load after the startup sequence. The patch should apply to 2.4 also. 

    https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728

While checking the module, I found some issues with it:

    * before my patch, the load (for same data) was independently done in each opensips process (even if the data is shared) :)

    * the "reload" is no error pron - if the new data load fails, the old existing data is lost

    * during the data reload the module has no data, so it cannot operate (it will actually block) - normally it should continue with the old data until the new data is fully loaded and validated.

Let me know if the patch makes a different for you and I will backport it to 2.4 too.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:
Hi,

Taking a closer look at the module (to be honest I do not user it, as drouting does a similar job) I see the module is loading the data into memory. It is doing this during the init stage, holding back the entire OpenSIPS startup.
Module doing similar heavy db load were changed to do the load post-startup. Let patch the module to decouple the loading from the startup, so that OpenSIPS will start without delays.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:04 PM, Jon Abrams wrote:
OpenSIPs is loading the global blacklist into a digit trie structure into shared memory on startup (or reload_blacklist command). That causes the startup delay.

- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <[hidden email]> wrote:
Hi Bogdan,

I don't think so its translating to DB query on run time, instead its caching the blacklist.
I am saying so because opensips response is quick, where as when I query manually from DB, it take some time even prefix col. is indexed.
Moreover, while opensips restart i get following warnings in syslog: 




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

basitengr
Thanks a lot. 
Earlier status
Opensips service was started in (real 1m6.474s)

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'

real    0m3.948s
user    0m0.039s
sys     0m0.013s

time systemctl stop opensips 

real    0m1.424s
user    0m0.004s
sys     0m0.000s

I pulled source from master.

# time systemctl start opensips 
Oct  2 15:57:41 srouter-215 systemd[1]: Starting OpenSIPS is a very fast and flexible SIP (RFC3261) server... 
Oct  2 15:57:41 srouter-215 opensips[15906]: Listening on 
Oct  2 15:57:41 srouter-215 opensips[15906]: udp: x.x.x.x [x.x.x.x]:9020 
Oct  2 15:57:41 srouter-215 opensips[15906]: udp: x6:x6:x6:x6:x6:x6 [[x6:x6:0:0:x6:x6:x6:x6]]:9020 
Oct  2 15:57:41 srouter-215 opensips[15906]: Aliases: 
Oct  2 15:57:41 srouter-215 opensips[15906]: *: x.x.x.x:* 

real    0m0.066s
user    0m0.003s
sys     0m0.002s
root@srouter-215:~# Oct  2 15:57:41 srouter-215 systemd[1]: Started OpenSIPS is a very fast and flexible SIP (RFC3261) server. 

/var/log/opensips.log is here ... https://pastebin.com/8md6sBiP

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'
Oct  2 16:17:24 srouter-215 SIPSrv[16185]: INFO:userblacklist:reload_sources: got 1386901 entries from 'globalblacklist' 

real    0m41.551s
user    0m0.037s
sys     0m0.015s

# time systemctl stop opensips 
Oct  2 16:06:38 srouter-215 systemd[1]: Stopping OpenSIPS is a very fast and flexible SIP (RFC3261) server... 
Oct  2 16:06:50 srouter-215 systemd[1]: Stopped OpenSIPS is a very fast and flexible SIP (RFC3261) server

real    0m12.134s
user    0m0.003s
sys     0m0.002s


Looks like issue has been fixed.. :)

--
regards,

abdul basit


On Tue, 2 Oct 2018 at 18:55, Bogdan-Andrei Iancu <[hidden email]> wrote:
OK, I pushed on master a patch to move the initial data load after the startup sequence. The patch should apply to 2.4 also. 

    https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728

While checking the module, I found some issues with it:

    * before my patch, the load (for same data) was independently done in each opensips process (even if the data is shared) :)

    * the "reload" is no error pron - if the new data load fails, the old existing data is lost

    * during the data reload the module has no data, so it cannot operate (it will actually block) - normally it should continue with the old data until the new data is fully loaded and validated.

Let me know if the patch makes a different for you and I will backport it to 2.4 too.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:
Hi,

Taking a closer look at the module (to be honest I do not user it, as drouting does a similar job) I see the module is loading the data into memory. It is doing this during the init stage, holding back the entire OpenSIPS startup.
Module doing similar heavy db load were changed to do the load post-startup. Let patch the module to decouple the loading from the startup, so that OpenSIPS will start without delays.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:04 PM, Jon Abrams wrote:
OpenSIPs is loading the global blacklist into a digit trie structure into shared memory on startup (or reload_blacklist command). That causes the startup delay.

- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <[hidden email]> wrote:
Hi Bogdan,

I don't think so its translating to DB query on run time, instead its caching the blacklist.
I am saying so because opensips response is quick, where as when I query manually from DB, it take some time even prefix col. is indexed.
Moreover, while opensips restart i get following warnings in syslog: 




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

Bogdan-Andrei Iancu-2
Hi Abdul,

Thanks for the feedback. I just want to be sure you understand that the change I did does not speed up the loading process, but it simply decouples the startup sequence from the data loading process. Shortly OpenSIPS does not have to wait for the DB data to be loaded in order to start up.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 07:22 PM, Abdul Basit wrote:
Thanks a lot. 
Earlier status
Opensips service was started in (real 1m6.474s)

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'

real    0m3.948s
user    0m0.039s
sys     0m0.013s

time systemctl stop opensips 

real    0m1.424s
user    0m0.004s
sys     0m0.000s

I pulled source from master.

# time systemctl start opensips 
Oct  2 15:57:41 srouter-215 systemd[1]: Starting OpenSIPS is a very fast and flexible SIP (RFC3261) server... 
Oct  2 15:57:41 srouter-215 opensips[15906]: Listening on 
Oct  2 15:57:41 srouter-215 opensips[15906]: udp: x.x.x.x [x.x.x.x]:9020 
Oct  2 15:57:41 srouter-215 opensips[15906]: udp: x6:x6:x6:x6:x6:x6 [[x6:x6:0:0:x6:x6:x6:x6]]:9020 
Oct  2 15:57:41 srouter-215 opensips[15906]: Aliases: 
Oct  2 15:57:41 srouter-215 opensips[15906]: *: x.x.x.x:* 

real    0m0.066s
user    0m0.003s
sys     0m0.002s
root@srouter-215:~# Oct  2 15:57:41 srouter-215 systemd[1]: Started OpenSIPS is a very fast and flexible SIP (RFC3261) server. 

/var/log/opensips.log is here ... https://pastebin.com/8md6sBiP

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'
Oct  2 16:17:24 srouter-215 SIPSrv[16185]: <a class="moz-txt-link-freetext" href="INFO:userblacklist:reload_sources">INFO:userblacklist:reload_sources: got 1386901 entries from 'globalblacklist' 

real    0m41.551s
user    0m0.037s
sys     0m0.015s

# time systemctl stop opensips 
Oct  2 16:06:38 srouter-215 systemd[1]: Stopping OpenSIPS is a very fast and flexible SIP (RFC3261) server... 
Oct  2 16:06:50 srouter-215 systemd[1]: Stopped OpenSIPS is a very fast and flexible SIP (RFC3261) server

real    0m12.134s
user    0m0.003s
sys     0m0.002s


Looks like issue has been fixed.. :)

--
regards,

abdul basit


On Tue, 2 Oct 2018 at 18:55, Bogdan-Andrei Iancu <[hidden email]> wrote:
OK, I pushed on master a patch to move the initial data load after the startup sequence. The patch should apply to 2.4 also. 

    https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728

While checking the module, I found some issues with it:

    * before my patch, the load (for same data) was independently done in each opensips process (even if the data is shared) :)

    * the "reload" is no error pron - if the new data load fails, the old existing data is lost

    * during the data reload the module has no data, so it cannot operate (it will actually block) - normally it should continue with the old data until the new data is fully loaded and validated.

Let me know if the patch makes a different for you and I will backport it to 2.4 too.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:
Hi,

Taking a closer look at the module (to be honest I do not user it, as drouting does a similar job) I see the module is loading the data into memory. It is doing this during the init stage, holding back the entire OpenSIPS startup.
Module doing similar heavy db load were changed to do the load post-startup. Let patch the module to decouple the loading from the startup, so that OpenSIPS will start without delays.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:04 PM, Jon Abrams wrote:
OpenSIPs is loading the global blacklist into a digit trie structure into shared memory on startup (or reload_blacklist command). That causes the startup delay.

- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <[hidden email]> wrote:
Hi Bogdan,

I don't think so its translating to DB query on run time, instead its caching the blacklist.
I am saying so because opensips response is quick, where as when I query manually from DB, it take some time even prefix col. is indexed.
Moreover, while opensips restart i get following warnings in syslog: 





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: opensips will not start while loading huge blacklist

basitengr
Yes. I understand and I check that point specifically. 
Thank you for your quick solution.

--
regards,

abdul basit


On Wed, 3 Oct 2018 at 00:37, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Abdul,

Thanks for the feedback. I just want to be sure you understand that the change I did does not speed up the loading process, but it simply decouples the startup sequence from the data loading process. Shortly OpenSIPS does not have to wait for the DB data to be loaded in order to start up.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 07:22 PM, Abdul Basit wrote:
Thanks a lot. 
Earlier status
Opensips service was started in (real 1m6.474s)

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'

real    0m3.948s
user    0m0.039s
sys     0m0.013s

time systemctl stop opensips 

real    0m1.424s
user    0m0.004s
sys     0m0.000s

I pulled source from master.

# time systemctl start opensips 
Oct  2 15:57:41 srouter-215 systemd[1]: Starting OpenSIPS is a very fast and flexible SIP (RFC3261) server... 
Oct  2 15:57:41 srouter-215 opensips[15906]: Listening on 
Oct  2 15:57:41 srouter-215 opensips[15906]: udp: x.x.x.x [x.x.x.x]:9020 
Oct  2 15:57:41 srouter-215 opensips[15906]: udp: x6:x6:x6:x6:x6:x6 [[x6:x6:0:0:x6:x6:x6:x6]]:9020 
Oct  2 15:57:41 srouter-215 opensips[15906]: Aliases: 
Oct  2 15:57:41 srouter-215 opensips[15906]: *: x.x.x.x:* 

real    0m0.066s
user    0m0.003s
sys     0m0.002s
root@srouter-215:~# Oct  2 15:57:41 srouter-215 systemd[1]: Started OpenSIPS is a very fast and flexible SIP (RFC3261) server. 

/var/log/opensips.log is here ... https://pastebin.com/8md6sBiP

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'
Oct  2 16:17:24 srouter-215 SIPSrv[16185]: INFO:userblacklist:reload_sources: got 1386901 entries from 'globalblacklist' 

real    0m41.551s
user    0m0.037s
sys     0m0.015s

# time systemctl stop opensips 
Oct  2 16:06:38 srouter-215 systemd[1]: Stopping OpenSIPS is a very fast and flexible SIP (RFC3261) server... 
Oct  2 16:06:50 srouter-215 systemd[1]: Stopped OpenSIPS is a very fast and flexible SIP (RFC3261) server

real    0m12.134s
user    0m0.003s
sys     0m0.002s


Looks like issue has been fixed.. :)

--
regards,

abdul basit


On Tue, 2 Oct 2018 at 18:55, Bogdan-Andrei Iancu <[hidden email]> wrote:
OK, I pushed on master a patch to move the initial data load after the startup sequence. The patch should apply to 2.4 also. 

    https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728

While checking the module, I found some issues with it:

    * before my patch, the load (for same data) was independently done in each opensips process (even if the data is shared) :)

    * the "reload" is no error pron - if the new data load fails, the old existing data is lost

    * during the data reload the module has no data, so it cannot operate (it will actually block) - normally it should continue with the old data until the new data is fully loaded and validated.

Let me know if the patch makes a different for you and I will backport it to 2.4 too.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:
Hi,

Taking a closer look at the module (to be honest I do not user it, as drouting does a similar job) I see the module is loading the data into memory. It is doing this during the init stage, holding back the entire OpenSIPS startup.
Module doing similar heavy db load were changed to do the load post-startup. Let patch the module to decouple the loading from the startup, so that OpenSIPS will start without delays.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:04 PM, Jon Abrams wrote:
OpenSIPs is loading the global blacklist into a digit trie structure into shared memory on startup (or reload_blacklist command). That causes the startup delay.

- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <[hidden email]> wrote:
Hi Bogdan,

I don't think so its translating to DB query on run time, instead its caching the blacklist.
I am saying so because opensips response is quick, where as when I query manually from DB, it take some time even prefix col. is indexed.
Moreover, while opensips restart i get following warnings in syslog: 





_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users