Conflict detection

Replication works extremely well in most cases without any additional complexity. When it comes to password randomization, however, merely maintaining synchronization is not good enough. Not only must all nodes record the same password for an account but it must be the correct password for the account on its target system as well. The delayed nature of replication means that if two nodes perform a randomization simultaneously, the value recorded for the password on each node is unpredictable because incoming replication messages from a replica will overwrite the current stored value. Depending on the account, if the wrong password is set in a node’s database, the impact can range from completely inconsequential to a catastrophic outage. Bravura Privilege is now capable of detecting cases where the passwords stored in its database may be incorrect or out of synchronization with its replicas and has a conflict resolution module to correct them.

Consider a replicated environment with only two nodes: A and B. When A randomizes an account’s password, it immediately writes to its database the password it intends to set and replicates that same value to B. B follows exactly the same process when it performs a randomization. If B begins a randomization near the same time as A’s randomization, the password that is stored in the database depends on how its constituent operations interact with A’s randomization. In the following three scenarios, involving two nodes randomizing the same account, different replication timing leads to different, sometimes incorrect, results.

Case 1: Serialized randomization

First A records password #1, then B records password #1, then B records password #2, then A records password #2. Ignoring any replication delay, both nodes correctly agree (that is, they have stored the same password in their databases) at all times as to the current password. Ideally, all randomizations follow this flow: each node is allowed to complete and fully replicate each randomization before any other node begins a new one.

Case 2: Perfectly parallel randomization

Suppose now that node B initiates its randomization earlier:

In this case, password #2 is the last password that was stored at A while password #1 is the last password that was stored at B. The nodes disagree and the instance is desynchronized. Because these are two completely independent nodes, each corresponding pair of steps may occur at exactly the same time on each node. By the time A learns that B is in the process of randomizing, its randomization is already complete, and vice versa.

Case 3: Interleaved parallel randomization

In this variant, A and B are at first in disagreement about the provisional password. Then, because A is a little slower than B, both nodes first store B’s password #2 and are in agreement, and then A’s password #1. But A’s connector completed before B’s connector. Although both nodes agree on password #1, the current password on the target system is password #2.

Case 4: Incomplete randomization

This is another case that is handled by Bravura Security Fabric 's password conflict detection technology, though it is unrelated to replication. If a node launches a connector to perform a randomization and the connector stops responding or times out, there is no way to know whether the connector was actually able to set the password on the target. It may be that the connector stopped responding immediately after startup and wasn't even able to connect to the target, or it may be that the target accepted the password randomization and network latency caused its acknowledgment to the connector to be dropped, leading to a timeout.

An ancestry tree is used to detect and correct simultaneous randomizations. In an ancestry tree, each randomization is linked to the last successful randomization that came before it (this randomization is called its parent), according to the node that issued it. This linkage creates a tree of randomizations. The root of the tree is the earliest randomization, and the tip is the latest randomization.

In the ideal case of a single node with no conflicts, each randomization has at most one child and at most one parent, and the tree is entirely vertical, as in Figure 1. With multiple nodes, each node maintains its own copy of the randomization tree, as in Figure 2, “Copy”. When a node receives a replication message informing it of a randomization, it incorporates that randomization into its tree, as in Figure 3, “Incorporation”. A replication message that creates a new randomization includes information about its parent and the node that initiated the randomization.

Figure 1. No conflicts

A simple randomization tree with no conflicts. Randomization 1 is the root and Randomization 3 is the tip

Figure 2. Copy

Node B performs a randomization and Node A has not yet processed its replication message. The replication message for Randomization 2 records that it was performed by Node B (in green), and that its parent is Randomization 1

Figure 3. Incorporation

Node A incorporates Node B’s randomization into its tree

Simultaneous randomizations are defined as two randomizations with the same parent. Such randomizations trigger a conflict called a tree conflict. In Figure 4, “Simultaneous randomization”, both nodes have performed different randomizations with the same parent. When their replication messages to each other are processed, they create a complex tree, shown in Figure 5, “Conflict detection”. Each node checks its randomization tree with each addition and searches for such conflicts.

Figure 4. Simultaneous randomization

Nodes A (in red) and B (in green) randomize the same account simultaneously and replicate to each other

Figure 5. Conflict detection

Nodes A (in red) and B (in green) randomize the same account simultaneously and replicate to each other

There are four primary tables used for randomization:

All passwords that have ever been generated are always retained in one of these tables. They are only discarded using the rmidarchivehis program or a similar administrator-controlled process.

All passwords have a status, regardless of table location (though some tables imply some subsets of these statuses). The important statuses for conflict resolution are:

Each node’s randomization tree is rooted at the sigkey specified in wstnpwdcur, which marks the current password for the account. This is the password that will be disclosed when checked out.

Randomization proceeds through a four-stage process:

Implementation of the fourth stage differs depending on the node. On the node where randomization was initiated, the idarch service is signalled to begin resolution immediately. On other nodes it is not possible to signal any services because the tree validation takes place within a stored procedure in the context of the database service processing the incoming replication queue. Instead, the idarch service periodically polls the database to find accounts that have been marked conflicted by incoming replication messages. Note that conflicts are detected immediately when replicated, and the account marked appropriately; database polling is only required to initiate conflict resolution.