[Request for Brain[storming]] New types of routes in config

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[Request for Brain[storming]] New types of routes in config

Bogdan-Andrei Iancu
Hi,

Following some discussion on mailing lists and IRC chat, I think there
is a need to enhance the routing possibilities in the OpenSIPS, in order
to simplify things and to allow more complex scenarios.

But I would like to boil a bit the ideas here, with all of you (more
brains are definitely better than one ;))


1. Init Route (new)

A route to be executed at startup only - this will allow the script
writer to load stuff from DB , to populate memcache-ul, so, to prepare
the routing process. More or less you can do this first time load even
now, but you need to check all the time if you already did it or not (if
not yet loaded, load it now)...


2. Reply Route (change)

Not a new type of route, but to be able to set onreply_route per branch
- it will simplify a lot of things (complex routing because there is a
single reply_route block per transaction)


3. Reply Route (change) versus branch_completed Route

The idea is that receiving some reply may be an event that trigger the
creation of new branches.
Ex (from Thomas Gelf):
       a) do parallel forking to a MOH server and to a GW
       b) MOH sends 183 and start playing media
       c) GW fails and at this point I want to create a new branch (GW
failover) without affecting the MOH branch (and I cannot use
failure_route as the transaction still have the MOH branch ongoing)

Changing Reply_route to allow the usage of t_reply() (+family) in this
route - will allow to create new branches from there - it is the most
flexible, but a bit tricky as the reply route is not protected to
retransmissions.

Adding branch_completed Route will be more sane and structured, but more
limited - a branch can be created only when a another branch is
completed. For example you cannot create a new branch when you receive
180 on another branch.


4. Timer based route

There was a request for doing parallel forking, but with asynchronous
branches (the branches are fires one at each 10 seconds)..The idea will
be to arm a trigger to execute a route after some period of time (this
will work only in stateful mode)



Any idea, comments, improvements, etc about this are welcome!

Thanks and regards,
Bogdan


_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: [Request for Brain[storming]] New types of routes in config

Thomas Gelf
Hi Bogdan,

here my 2 Euro Cents, following your brainstorm request:


> 1. Init Route (new)

Personally I do not have immmediate need for this feature, but as soon
as I start digging into memcache support it will be really useful. As
it also shouldn't be that hard to implement: please do it!

> 2. Reply Route (change)

I have encountered some issues, for example with uac_replace_from calls
for specific branches - I'm pretty sure such an enhancement would clean
up such scenarios / make them easier to implement.

The following should IMO be possible:

route[] {
    t_on_reply(1);
}

branch_route[] {
    if (whatever) {
        # this will not affect other branches
        t_on_reply(2);
    }
}

If this is how it would work: great, I would really welcome such an
enhancement.


> 3. Reply Route (change) versus branch_completed Route

To summarize my questions / your explainations on IRC:

- branch_completed[] will be triggered when a branch is completed = when
  a final response is received on that branch (reply >=200)
- Calling drop() in branch_completed[] will prevent failure_route from
  being triggered for replies >=400
- When triggering a new branch in branch_completed[], minor branches
  will still remain active and are not going to be CANCELed

This seems very useful and would allow clean routing scripts for complex
scenarios. I would welcome also this addition!


> 4. Timer based route

This ould be a nice feature. It's not something I have immediate need
for - and I guess it could be tricky to implement it, probably with some
side effects... If you want to face the challenge: feel free to do so ;)

My personal favourites are 2. and 3. - 1. should not be that hard, and
if there is enough time and interest for 4.: I'm absolutely not against
it. Can someone provide a concrete example where this could make sense?
And: we would also need individual timeouts for each delayed branch,
correct?

Kind regards,
Thomas Gelf




_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: [OpenSIPS-Users] [Request for Brain[storming]] New types of routes in config

Bogdan-Andrei Iancu
Hi Thomas,

Thomas Gelf wrote:

> Hi Bogdan,
>
> here my 2 Euro Cents, following your brainstorm request:
>
>
>  
>> 1. Init Route (new)
>>    
>
> Personally I do not have immmediate need for this feature, but as soon
> as I start digging into memcache support it will be really useful. As
> it also shouldn't be that hard to implement: please do it!
>  
yes, it is not hard to do it, and as time as it brings some added value,
why not :)

>  
>> 2. Reply Route (change)
>>    
>
> I have encountered some issues, for example with uac_replace_from calls
> for specific branches - I'm pretty sure such an enhancement would clean
> up such scenarios / make them easier to implement.
>
> The following should IMO be possible:
>
> route[] {
>     t_on_reply(1);
> }
>
> branch_route[] {
>     if (whatever) {
>         # this will not affect other branches
>         t_on_reply(2);
>     }
> }
>
> If this is how it would work: great, I would really welcome such an
> enhancement.
>  

yes, that's my idea - if you set onreply_route from request_route -> it
will be used for all branches; but if you set it from Branch_route ->
only for that branch.

>
>  
>> 3. Reply Route (change) versus branch_completed Route
>>    
>
> To summarize my questions / your explainations on IRC:
>
> - branch_completed[] will be triggered when a branch is completed = when
>   a final response is received on that branch (reply >=200)
> - Calling drop() in branch_completed[] will prevent failure_route from
>   being triggered for replies >=400
> - When triggering a new branch in branch_completed[], minor branches
>   will still remain active and are not going to be CANCELed
>
> This seems very useful and would allow clean routing scripts for complex
> scenarios. I would welcome also this addition!
>  

so you are in favour of branch_complete route, instead of chaginf the
reply_route to allow new branch creation? Actually, thinking more on
this, these 2 approaches do not exclude one each other :D

>
>  
>> 4. Timer based route
>>    
>
> This ould be a nice feature. It's not something I have immediate need
> for - and I guess it could be tricky to implement it, probably with some
> side effects... If you want to face the challenge: feel free to do so ;)
>
> My personal favourites are 2. and 3. - 1. should not be that hard, and
> if there is enough time and interest for 4.: I'm absolutely not against
> it. Can someone provide a concrete example where this could make sense?
>  
An example was on the mailing list - asyncron parallel forking (instead
of firing all branches in the same time, they will fired with some delays).
> And: we would also need individual timeouts for each delayed branch,
> correct?
>  
timeout per branch during parallel forking? interesting.....I guess it
can be done....

Regards,
Bogdan

_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: [OpenSIPS-Users] [Request for Brain[storming]] New types of routes in config

Brett Nemeroff
In reply to this post by Bogdan-Andrei Iancu
1. Init Route (new)

A route to be executed at startup only - this will allow the script
writer to load stuff from DB , to populate memcache-ul, so, to prepare
the routing process. More or less you can do this first time load even
now, but you need to check all the time if you already did it or not (if
not yet loaded, load it now)...

I just wanted to say that I'd have a lot of use for this. I've got some scenarios together with memcache and love it. However if/once you allow using an external cache store, this need may disappear.

I had some other requests. I hope you don't mind me sticking them in here:
1. Something else I've mentioned (and I think I stuck on the tracker) is that I think some of the module needs to be able to load arbitrary data while returning rows. This is by far my #1 request. I feel like I'm always playing tricks to do this different ways. 

For example:
A. Trusted module should allow loading of AVPs based on trusted entry. An example of use for this would be call limit for source IP. Another example would be an "Account Code" Which would be inserted as a header when this source IP is used. So maybe even the option to load multiple values.

B. DB Routing module should set an avp indicating WHICH rule # was used and which Gateway ID # was used. Furthermore both the rule and the gateway should allow setting custom AVPs. An example of this is: I only have 5 channels to 411. Or Gateway Id #100 only has 5 available channels. That's actually really important since today, if Gateway Id #100 only has 5 channels, but gets picked first, I'll have to roll off it every time to find out it's full rather than "smartly" skipping it. I'm sure there are more examples of this.

The basic idea is that every module that "returns" a value should also return some sort of primary key that was used for that value. Then it can be logged into ACC. Displayed on logging "Sending call to Gateway #100"


2. Native combined CDR format. One row, start, answer, end, duration. Yes, I know, not a B2BUA.. It'll have limitations, and a bit fat disclaimer and I'm sure we'll have "that talk" again. :)

3. New DRouting fifo commands: route lookup, route count, route last reload, route table dump (ek?). Sometimes it's good to know that what's in memory is really what you expected. However, that's not very useful in production. 

4. Memcache: cache_fetch should return time to expiry for retrieved objects. 

5. Memcache: full cache purge fifo option (not just 1 object, but everything)

6. Memcache: partial cache purge (not sure how to do this. Idea is "Purge all accountcodes" or "Purge all account limits" (to force loading new ones)

7. Dialog: dlg_list should show profiles dialog belongs to. The dialog table (when db is used) should also show this.

That's it for now. As always, I appreciate everyone's hard work to make this product what it is. Thank you!
-Brett


_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: [OpenSIPS-Users] [Request for Brain[storming]] New types of routes in config

Bogdan-Andrei Iancu
Hi Brett,

Brett Nemeroff wrote:

>
>     1. Init Route (new)
>
>     A route to be executed at startup only - this will allow the script
>     writer to load stuff from DB , to populate memcache-ul, so, to prepare
>     the routing process. More or less you can do this first time load even
>     now, but you need to check all the time if you already did it or
>     not (if
>     not yet loaded, load it now)...
>
>
> I just wanted to say that I'd have a lot of use for this. I've got
> some scenarios together with memcache and love it. However if/once you
> allow using an external cache store, this need may disappear.
there multiple tools that may need script init (localcache, shared
variables, etc). Of course, once we have the implementation for memchace
support (memchace server), so of the needs may disappear, but there are
cases where you want to use the localcache only and you need to init it.

>
> I had some other requests. I hope you don't mind me sticking them in here:
> 1. Something else I've mentioned (and I think I stuck on the tracker)
> is that I think some of the module needs to be able to load arbitrary
> data while returning rows. This is by far my #1 request. I feel like
> I'm always playing tricks to do this different ways.
>
> For example:
> A. Trusted module should allow loading of AVPs based on trusted entry.
> An example of use for this would be call limit for source IP. Another
> example would be an "Account Code" Which would be inserted as a header
> when this source IP is used. So maybe even the option to load multiple
> values.
>
> B. DB Routing module should set an avp indicating WHICH rule # was
> used and which Gateway ID # was used. Furthermore both the rule and
> the gateway should allow setting custom AVPs. An example of this is: I
> only have 5 channels to 411. Or Gateway Id #100 only has 5 available
> channels. That's actually really important since today, if Gateway Id
> #100 only has 5 channels, but gets picked first, I'll have to roll off
> it every time to find out it's full rather than "smartly" skipping it.
> I'm sure there are more examples of this.
>
> The basic idea is that every module that "returns" a value should also
> return some sort of primary key that was used for that value. Then it
> can be logged into ACC. Displayed on logging "Sending call to Gateway
> #100"
I see your point and I agree with that - personally I had this need also
and this is why I added in Drouting module the "attrs" field for the GWs
definition - see
http://www.opensips.org/html/docs/modules/devel/drouting.html#id272135.

Of course, the same mechanism can be added to several other modules
(like permissions as you suggested).

In this area I intend to add support for JSON (json.org) to be able to
structure more info in a single string value - you have a generic attr
field that may encode multiple values, keys, etc using json scheme. So,
the idea will be to add the json transformation to extract from the
encode string a certain attribute.
>
>
> 2. Native combined CDR format. One row, start, answer, end, duration.
> Yes, I know, not a B2BUA.. It'll have limitations, and a bit fat
> disclaimer and I'm sure we'll have "that talk" again. :)
you mean to get directly CDRs, instead of acc events?
>
> 3. New DRouting fifo commands: route lookup, route count, route last
> reload, route table dump (ek?). Sometimes it's good to know that
> what's in memory is really what you expected. However, that's not very
> useful in production.
maybe this can be split between stat vars (for counting rules and GW)
and MI commands (like triggering a routing request from outside).
>
> 4. Memcache: cache_fetch should return time to expiry for retrieved
> objects.
Ok
>
> 5. Memcache: full cache purge fifo option (not just 1 object, but
> everything)
like a command to purge the entire content?
>
> 6. Memcache: partial cache purge (not sure how to do this. Idea is
> "Purge all accountcodes" or "Purge all account limits" (to force
> loading new ones)
but how do you suggest to identify the attributes to be purges as they
have different names? regexp matching ?
>
> 7. Dialog: dlg_list should show profiles dialog belongs to. The dialog
> table (when db is used) should also show this.
this is something I'm working on - to have persistence across reboot for
profiles, dialog values and flags.

Best regards,
Bogdan
>
> That's it for now. As always, I appreciate everyone's hard work to
> make this product what it is. Thank you!
> -Brett
>


_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: [OpenSIPS-Users] [Request for Brain[storming]] New types of routes in config

Brett Nemeroff
The basic idea is that every module that "returns" a value should also return some sort of primary key that was used for that value. Then it can be logged into ACC. Displayed on logging "Sending call to Gateway #100"
I see your point and I agree with that - personally I had this need also and this is why I added in Drouting module the "attrs" field for the GWs definition - see http://www.opensips.org/html/docs/modules/devel/drouting.html#id272135.

Of course, the same mechanism can be added to several other modules (like permissions as you suggested).

In this area I intend to add support for JSON (json.org) to be able to structure more info in a single string value - you have a generic attr field that may encode multiple values, keys, etc using json scheme. So, the idea will be to add the json transformation to extract from the encode string a certain attribute.
That's a very interesting idea. That would certainly satisfy many needs. I like it..I'd like to see something like this many modules. :)
 
2. Native combined CDR format. One row, start, answer, end, duration. Yes, I know, not a B2BUA.. It'll have limitations, and a bit fat disclaimer and I'm sure we'll have "that talk" again. :)
you mean to get directly CDRs, instead of acc events?
Yes, exactly. I've asked this directly on the list, but no one has answered. For me, combining ACC events into real CDR is a pain. it's really the single biggest wrench for the overall architecture. I have to use external scripts that make assumptions about SIP to combine the records. For example, as my rating scripts process the ACC events, it makes assumptions on weather the dialog has been destroyed yet or not to determine if it's time to rate it, or if it's still in progress. I don't know about you, but I'd much rather the SIP Stack track that, rather than duplicate that kind of logic on the rating engine. I feel that with all of the fancy dialog stuff going on, it should be able to easily identify events like start, pdd, ring, stop and gracefully handle events like reinvites. I don't work much on the core, but I bet there is a lot of this logic already in there for other purposes.
 


3. New DRouting fifo commands: route lookup, route count, route last reload, route table dump (ek?). Sometimes it's good to know that what's in memory is really what you expected. However, that's not very useful in production.
maybe this can be split between stat vars (for counting rules and GW) and MI commands (like triggering a routing request from outside).
Sounds good to me. :)
 
5. Memcache: full cache purge fifo option (not just 1 object, but everything)
like a command to purge the entire content?
Yeah: 100% destroy all cached objects (ie: I want you to suck up new data from the database). Of course, usage of this command is highly dependent on good scriptwriting. Else you may end up breaking a lot of stuff.
 
6. Memcache: partial cache purge (not sure how to do this. Idea is "Purge all accountcodes" or "Purge all account limits" (to force loading new ones)
but how do you suggest to identify the attributes to be purges as they have different names? regexp matching ?
You know, I'm not sure how this should be done.. regex sounds neat, but could just lead to a bunch of sloppy scripting. I *hate* to borrow ideas from asterisk, but caching families may be a neat idea.. examples of caching families are:
1. Account Limits
2. Destination Rates
3. Account Codes

It would require some tweaking of the caching functions, but I imagine something like:
cache_store_family("local","account_limits","10000001","24","3600")
(translation: store to local cache,in the account_limits family, account 10000001 has 24 channels. remember this for 3600 seconds)

Then you can issue a fifo:
cache_purge_family account_limits

And then all account_limits are all *gone*. but "Account Codes" and "Destination Rates" still remain.

Thanks again for everything, your hard work is much appreciated. :)
-Brett


_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: [OpenSIPS-Users] [Request for Brain[storming]] New types of routes in config

Bogdan-Andrei Iancu
Brett,

Brett Nemeroff wrote:

>
>         2. Native combined CDR format. One row, start, answer, end,
>         duration. Yes, I know, not a B2BUA.. It'll have limitations,
>         and a bit fat disclaimer and I'm sure we'll have "that talk"
>         again. :)
>
>     you mean to get directly CDRs, instead of acc events?
>
> Yes, exactly. I've asked this directly on the list, but no one has
> answered. For me, combining ACC events into real CDR is a pain. it's
> really the single biggest wrench for the overall architecture. I have
> to use external scripts that make assumptions about SIP to combine the
> records. For example, as my rating scripts process the ACC events, it
> makes assumptions on weather the dialog has been destroyed yet or not
> to determine if it's time to rate it, or if it's still in progress. I
> don't know about you, but I'd much rather the SIP Stack track that,
> rather than duplicate that kind of logic on the rating engine. I feel
> that with all of the fancy dialog stuff going on, it should be able to
> easily identify events like start, pdd, ring, stop and gracefully
> handle events like reinvites. I don't work much on the core, but I bet
> there is a lot of this logic already in there for other purposes.
have you tried the mysql procedure used by opensips-cp for creating CDR?
the CP is consuming the ACC data to create CDRs in a different table via
a mysql procedure that is triggered by cron. So far even with large CDR
volume, I got good results with this.

But making opensips to directly generate CDRs is complicated as in many
case you do not have a dialog info, so you never know if a START is
already there or not (in order to do an UPDATE or INSERT for BYE).

>  
>
>
>
>         3. New DRouting fifo commands: route lookup, route count,
>         route last reload, route table dump (ek?). Sometimes it's good
>         to know that what's in memory is really what you expected.
>         However, that's not very useful in production.
>
>     maybe this can be split between stat vars (for counting rules and
>     GW) and MI commands (like triggering a routing request from outside).
>
> Sounds good to me. :)
>  
>
>         5. Memcache: full cache purge fifo option (not just 1 object,
>         but everything)
>
>     like a command to purge the entire content?
>
> Yeah: 100% destroy all cached objects (ie: I want you to suck up new
> data from the database). Of course, usage of this command is highly
> dependent on good scriptwriting. Else you may end up breaking a lot of
> stuff.
>  
>
>         6. Memcache: partial cache purge (not sure how to do this.
>         Idea is "Purge all accountcodes" or "Purge all account limits"
>         (to force loading new ones)
>
>     but how do you suggest to identify the attributes to be purges as
>     they have different names? regexp matching ?
>
> You know, I'm not sure how this should be done.. regex sounds neat,
> but could just lead to a bunch of sloppy scripting. I *hate* to borrow
> ideas from asterisk, but caching families may be a neat idea..
> examples of caching families are:
> 1. Account Limits
> 2. Destination Rates
> 3. Account Codes
>
> It would require some tweaking of the caching functions, but I imagine
> something like:
> cache_store_family("local","account_limits","10000001","24","3600")
> (translation: store to local cache,in the account_limits family,
> account 10000001 has 24 channels. remember this for 3600 seconds)
>
> Then you can issue a fifo:
> cache_purge_family account_limits
>
> And then all account_limits are all *gone*. but "Account Codes" and
> "Destination Rates" still remain.
I have nothing against borrowing ideas, as time as they are useful and
applicable. I found really interesting the idea of "families" or
"classes" for memcache support  and it shouldn't be too difficult to
use. We can keep the current format for the functions for a default
"Generic" class and create a new set of functions to take an extra one
parameter, the name of the class.

Regards,
Bogdan

_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: [OpenSIPS-Users] [Request for Brain[storming]] New types of routes in config

Brett Nemeroff

have you tried the mysql procedure used by opensips-cp for creating CDR? the CP is consuming the ACC data to create CDRs in a different table via a mysql procedure that is triggered by cron. So far even with large CDR volume, I got good results with this.

But making opensips to directly generate CDRs is complicated as in many case you do not have a dialog info, so you never know if a START is already there or not (in order to do an UPDATE or INSERT for BYE).

I haven't looked at the opensips-cp method for doing that.. I'll have to do that right away. :) Sure, I understand there are issues with doing it natively, but thought I'd bring it up. I understand architecturally it isn't the best mechanism.  
 

I have nothing against borrowing ideas, as time as they are useful and applicable. I found really interesting the idea of "families" or "classes" for memcache support  and it shouldn't be too difficult to use. We can keep the current format for the functions for a default "Generic" class and create a new set of functions to take an extra one parameter, the name of the class.
I think the memcache API has a "namespace" param that may be a simple 1:1 drop in for this.Of course, the family should be a string, avp, or pv, right? :) Hey, should I document this on the feature request tracker?


Thanks!
-Brett


_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: [OpenSIPS-Users] [Request for Brain[storming]] New types of routes in config

Bogdan-Andrei Iancu
Brett Nemeroff wrote:

>
>
>     have you tried the mysql procedure used by opensips-cp for
>     creating CDR? the CP is consuming the ACC data to create CDRs in a
>     different table via a mysql procedure that is triggered by cron.
>     So far even with large CDR volume, I got good results with this.
>
>     But making opensips to directly generate CDRs is complicated as in
>     many case you do not have a dialog info, so you never know if a
>     START is already there or not (in order to do an UPDATE or INSERT
>     for BYE).
>
>
> I haven't looked at the opensips-cp method for doing that.. I'll have
> to do that right away. :) Sure, I understand there are issues with
> doing it natively, but thought I'd bring it up. I understand
> architecturally it isn't the best mechanism.  
>  
>
>
>     I have nothing against borrowing ideas, as time as they are useful
>     and applicable. I found really interesting the idea of "families"
>     or "classes" for memcache support  and it shouldn't be too
>     difficult to use. We can keep the current format for the functions
>     for a default "Generic" class and create a new set of functions to
>     take an extra one parameter, the name of the class.
>
> I think the memcache API has a "namespace" param that may be a simple
> 1:1 drop in for this.Of course, the family should be a string, avp, or
> pv, right? :) Hey, should I document this on the feature request tracker?

Please, do so :)

Regards,
Bogdan


_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel