Troubleshoot Registry Name Service and Database Synchronization Issues
The Registry Name Service and Database Synchronization are complex and there are many factors that can affect the smooth running of the service. Every host, including the Primary Registry Name Server, has a Registry Name Service Cache, which they use to look up all Service Group information that are applicable to them.
All of the servers that provide these services are implemented using the REST services (clients may or may not have REST services installed, but this should not be relevant to the overall service), and as such, one of the first checks is to make sure the relevant REST services are running. Use an appropriate ps (man1) command to show the process information:
# ps -ef | egrep "pblight|pbconfig" root 124824 1 0 12:12 pts/1 00:00:00 /usr/lib/beyondtrust/pb/rest/sbin/pblighttpd-svc -d -i #mon (wait) root 124825 124824 0 12:12 pts/1 00:00:01 /usr/lib/beyondtrust/pb/rest/sbin/pblighttpd-svc -d -i #sched (sleep) pblight 124826 124824 0 12:12 pts/1 00:00:00 pblighttpd -D -m /usr/lib/beyondtrust/pb/rest/lib -f /usr/lib/beyondtrust/pb/rest/etc/pblighttpd.conf root 124827 124826 0 12:12 pts/1 00:00:00 /usr/lib/beyondtrust/pb/rest/sbin/pbconfigd root 124828 124826 0 12:12 pts/1 00:00:00 /usr/lib/beyondtrust/pb/rest/sbin/pbconfigd
The number of pbconfigd processes running differs according to the host role within the enterprise.
If these services are not running, check the corresponding system logs and pblighttpd/REST logs for errors, and use the platform specific method of restarting the services.
Next, check that the host has the correct Service Group information.
Find out which Service Groups the host is a member of on the Primary Registry Name Server:
# pbdbutil -P --svc -L -L pbulprimrns
{
"hostid": 1,
"cn": "pbulprimrns",
"uuid": "3d13a9eb-7340-4199-aa47-1570941bd50f",
"fqdn": "pbulprimrns.org.com",
"addrs": [
{
"family": 4,
"addr": "192.168.1.1",
"port": 24351
}
],
"tnlzone": 0,
"updated_usec": "2016-11-11 12:03:47",
"deleted": false,
"svcs": [
{
"svcgid": 1,
"hostid": 1,
"role": "primary",
"sorder": 1,
"created_usec": "2016-11-11 12:03:47",
"updated_usec": "2016-11-11 12:03:47",
"svcgname": "registry_name_service",
"svc": "registry",
"deleted": 0
},
{
"svcgid": 4,
"hostid": 1,
"role": "secondary",
"sorder": 2,
"created_usec": "2016-11-11 12:04:35",
"updated_usec": "2016-11-11 12:03:47",
"svcgname": "dflt_sudopolicy_service",
"svc": "sudopolicy",
"deleted": 0
}
]
}
Check the IP Addresses and List of Service Groups
Then check on the host to make sure the Registry Name Service Cache shows corresponding information:
# pbdbutil -P --scache -l
{
"svcgname": "registry_name_service",
"svc": "registry",
"sorder": 1,
"cn": "pbulprimrns",
"uuid": "3d13a9eb-7340-4199-aa47-1570941bd50f",
"fqdn": "pbulprimrns.org.com",
"addrs": [
{
"family": 4,
"addr": "192.168.1.1",
"port": 24351
}
],
"role": "primary",
"lastupdated_usec": "2016-11-11 12:03:47"
}
{
"svcgname": "dflt_sudopolicy_service",
"svc": "sudopolicy",
"sorder": 1,
"cn": "pbtest",
"uuid": "12345676789",
"fqdn": "pbtest",
"addrs": [
{
"family": 4,
"addr": "192.168.1.5",
"port": 24351
}
],
"role": "primary",
"lastupdated_usec": "2016-11-11 12:04:23"
}
{
"svcgname": "dflt_sudopolicy_service",
"svc": "sudopolicy",
"sorder": 2,
"cn": "pbulprimrns",
"uuid": "3d13a9eb-7340-4199-aa47-1570941bd50f",
"fqdn": "pbulprimrns.org.com",
"addrs": [
{
"family": 4,
"addr": "192.168.1.1",
"port": 24351
}
],
"role": "secondary",
"lastupdated_usec": "2016-11-11 12:04:35"
}
If this information differs, and the lastupdated times are significantly different, refresh the cache:
# pbdbutil --scache -R
If this produces an error, then re-initialize the Registry Name Service Cache on the host using:
# pbdbutil --scache -N '{"hostname" : "<primaryRNS>, "appid": "<appid>": "appkey" : "<appkeys>"}' --force
This clears the existing Registry Name Service Cache database and reloads it from the Primary Registry Name Server.
If databases are not correctly synchronizing across servers within the Service Group, firstly identify the primary server within that Service Group:
# pbdbutil -P --svc -g '{ "primary" : "registry_name_service" }'
{
"svcgid": 1,
"svcgname": "registry_name_service",
"svc": "registry",
"updated_usec": "2016-11-11 12:03:47",
"deleted": false,
"hostid": 1,
"role": "primary",
"sorder": 1,
"created_usec": "2016-11-11 12:03:47",
"cn": "pbulprimrns",
"uuid": "3d13a9eb-7340-4199-aa47-1570941bd50f",
"fqdn": "pbulprimrns.org.com",
"addrs": [
{
"family": 4,
"addr": "192.168.1.1",
"port": 24351
}
],
"tnlzone": 0
}
Next, on that host list the contents of the Database Synchronization Summary Database:
pbdbutil --dbsync -l
{"cn":"pbtest","svc":"registry","dbid":1,"dbname":"/etc/pb.db","dbuuid":"9b332c26-b8bb-4546-9ed2-bf93146dd08c","lastupdated":"2016-11-11 15:10:44","lasttid":0}
{"cn":"pbtest","svc":"registry","dbid":2,"dbname":"/opt/pbul/dbs/pbrstkeys.db","dbuuid":"a22e8a75-fc6c-4f0e-aee9-b0d764c7e820","lastupdated":"2016-11-11 15:10:44","lasttid":0}
{"cn":"pbtest","svc":"registry","dbid":257,"dbname":"/opt/pbul/dbs/pbsvc.db","dbuuid":"97423b4f-2c5e-42c2-a87a-aeeaaa826c5b","lastupdated":"2016-11-11 15:12:14","lasttid":40}
{"cn":"pbtest","svc":"registry","dbid":258,"dbname":"/opt/pbul/dbs/pbregclnt.db","dbuuid":"f08b8679-8bcb-4f7d-b431-1bbab688e0c1","lastupdated":"2016-11-11 15:12:14","lasttid":0}
And then for each individual database check for outstanding transactions:
# pbdbutil --dbsync -l /opt/pbul/dbs/pbsvc.db
{"path":"/opt/pbul/dbs/pbsvc.db","sz":0}
If necessary, reset the Summary Database information and force a resynchronization:
# pbdbutil --dbsync -R registry
Finally, check that the Scheduling Service is running, and has recently run. This service provides the regular Registry Name Service Cache and Database Synchronization updates:
# pbdbutil --info --sched
{"id":"svccache_update","grp":"system","epoch":"2016-11-11 14:36:57","reoccurs":110,"retry":0,"backoff":0,"retried":0}
{"id":"database_sync","grp":"system","epoch":"2016-11-11 14:35:37","reoccurs":30,"retry":0,"backoff":0,"retried":0}
