Clusterer and Full Sharing USRLOC sync state/persistence

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Clusterer and Full Sharing USRLOC sync state/persistence

Jock McKechnie
Good afternoon;

I've been poking with getting a four-node cluster doing fully shared
USRLOC data up and running and I'm having troubles getting the restart
persistency to work reliably. The cluster comes up without issues and
all nodes appear to find each other, and I can stop various nodes and
the USRLOC database will back-fill on node start later ('opensipsctl
ul show' populates as expected).

I have discovered, however, that after a certain period of run time -
I haven't worked out what the magic switch is yet - interspersed with
node restarts I get a cluster State of "not synced" at which point
when nodes restart they will no longer sync the USRLOC data back and
sort of become orphans. So far the only way I've found to get things
working again is to restart _all_ nodes simultaneously, which brings
them back into an "Ok" cluster State (opensipsctl fifo
clusterer_list_cap).

When the cluster is in a 'not synced' state I haven't managed to work
out how to restart nodes to bring the cluster back into a working
state if they're not all cycled together.

This is my first experience with the clusterer system and I'm not
completely famaliar with its ins, outs and quirks as yet.

I'm wondering if there's a correct procedure to bring a cluster back
into sync after it's fallen out of sync - if there's a way to
determine who is the node at fault and bring things back in such a
manner that won't require a complete restart and losing the data set
in memory.

We are a MySQL and Cassandra house and, at discussed in a previous
thread, the only working cluster source for this data is a MongoDB.
Unfortunately from a corporate stand point I'm unlikely to get anyone
willing to support a MongoDB set just to support offline restart
persistence of the OpenSIPS cluster so I'm hoping that I can get the
in-memory set up reliable enough that I can use it.

My thanks for your time, and any hints you might be able to provide;

 - Jock

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Clusterer and Full Sharing USRLOC sync state/persistence

vasilevalex
Hi Jock.

I also use full-sharing cluster with just two nodes. Everything in memory,
no DB.

Do you have "seed" node? Because without it, as far as I tested, it is
impossible to get out from "not synced state". All nodes unsynced so no one
can be taken as trusted point.

I made seed flag for one server. So the second server starts and trusts seed
node. If I restart seeding node, it starts clean (and this state is
considered trusted and synced for itself) then I have to run "opensipsctl
fifo ul_cluster_sync" on it to add all the data from cluster to it's clean
state. So it became synced and containing all data.

May be this is not the best way, but I don't know, how also can be solved
this state.



--
Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Clusterer and Full Sharing USRLOC sync state/persistence

Liviu Chircu
Hi all,

Jock: just like Alexei said, I assume you don't have a "seed" node. 
Make sure one node of your opensips.clusterer table has "seed" in the
"flags" column.  Regarding offline restart persistence: you may skip the
"working_mode_preset" parameter and individually set "cluster_mode" [1]
and "restart_persistency" [2] in order to achieve a restart-persistent
cluster using an SQL DB, rather than point-to-point sync.

Alexei: that is exactly how we meant it to be used.  With 0 nodes
online, you have to bootstrap the cluster somehow -- this is the purpose
of the "seed" node.

Cheers,

[1]:
https://opensips.org/html/docs/modules/3.0.x/usrloc.html#param_cluster_mode
[2]:
https://opensips.org/html/docs/modules/3.0.x/usrloc.html#restart_persistency

Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com

On 18.10.2018 09:28, vasilevalex wrote:

> Hi Jock.
>
> I also use full-sharing cluster with just two nodes. Everything in memory,
> no DB.
>
> Do you have "seed" node? Because without it, as far as I tested, it is
> impossible to get out from "not synced state". All nodes unsynced so no one
> can be taken as trusted point.
>
> I made seed flag for one server. So the second server starts and trusts seed
> node. If I restart seeding node, it starts clean (and this state is
> considered trusted and synced for itself) then I have to run "opensipsctl
> fifo ul_cluster_sync" on it to add all the data from cluster to it's clean
> state. So it became synced and containing all data.
>
> May be this is not the best way, but I don't know, how also can be solved
> this state.
>
>
>
> --
> Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Clusterer and Full Sharing USRLOC sync state/persistence

Jock McKechnie
Thank you Alexei and Liviu;

To close the loop, this must've been it. I was having issue with the
seed node sync'ing off the other nodes after it was cycled earlier,
but that must have been something else, because nominating a single
seed seems to have brought everything into a comfortably reliable
service.

Thank you again,

 - Jock
On Thu, Oct 18, 2018 at 4:04 AM Liviu Chircu <[hidden email]> wrote:

>
> Hi all,
>
> Jock: just like Alexei said, I assume you don't have a "seed" node.
> Make sure one node of your opensips.clusterer table has "seed" in the
> "flags" column.  Regarding offline restart persistence: you may skip the
> "working_mode_preset" parameter and individually set "cluster_mode" [1]
> and "restart_persistency" [2] in order to achieve a restart-persistent
> cluster using an SQL DB, rather than point-to-point sync.
>
> Alexei: that is exactly how we meant it to be used.  With 0 nodes
> online, you have to bootstrap the cluster somehow -- this is the purpose
> of the "seed" node.
>
> Cheers,
>
> [1]:
> https://opensips.org/html/docs/modules/3.0.x/usrloc.html#param_cluster_mode
> [2]:
> https://opensips.org/html/docs/modules/3.0.x/usrloc.html#restart_persistency
>
> Liviu Chircu
> OpenSIPS Developer
> http://www.opensips-solutions.com
>
> On 18.10.2018 09:28, vasilevalex wrote:
> > Hi Jock.
> >
> > I also use full-sharing cluster with just two nodes. Everything in memory,
> > no DB.
> >
> > Do you have "seed" node? Because without it, as far as I tested, it is
> > impossible to get out from "not synced state". All nodes unsynced so no one
> > can be taken as trusted point.
> >
> > I made seed flag for one server. So the second server starts and trusts seed
> > node. If I restart seeding node, it starts clean (and this state is
> > considered trusted and synced for itself) then I have to run "opensipsctl
> > fifo ul_cluster_sync" on it to add all the data from cluster to it's clean
> > state. So it became synced and containing all data.
> >
> > May be this is not the best way, but I don't know, how also can be solved
> > this state.
> >
> >
> >
> > --
> > Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html
> >
> > _______________________________________________
> > Users mailing list
> > [hidden email]
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users