Background
SearchStax Managed Search service’s Solr clients occasionally report that the same query produces inconsistent results when run repeatedly! This complaint is not a common one, but it alarms people when they see it.
The load balancer of a Solr cluster sends incoming queries to the Solr nodes on a rotating or random basis. A query may be sent to node 1 on the first try, but may go to node 2 the next time.
If the replicas for a collection are not fully synchronized across the nodes, search results can be inconsistent.
Anytime a document gets indexed, the replicas will be out of sync temporarily. Heavy system loads (indexing, querying) can delay replication. This balance in priorities is controlled by Solr itself, and is not a mechanism that SearchStax can configure. However, when system load eases again, the replicas should rebalance quickly.
Remediation
Solr replication is a complex mechanism. On the rare occasions when the replication service fails, there is little point in trying to pinpoint the exact cause. The cure is to do a rolling restart of the cluster, which relaunches the replication service. SearchStax Support engineers will be happy to help you with this step, which does not involve down time.
Will It Happen Again?
A replication service failure is a rare event that does not usually become a chronic problem.
Self-Diagnosis
If you suspect a replication failure, how can you verify it yourself? The replicas of a collection should all contain the same number of documents. When they don’t, we have identified the problem.
Solr 8+
In Solr 8 and above, it is trivial to check the number of documents in a replica. The Solr Dashboard puts that information right up front.
Open the Solr Dashboard and navigate to the Cloud > Nodes page. Look for the replicas and their doc counts in the rightmost column of the table.
Solr 7-
In Solr release before Solr 8, you have to work a little to compare document counts between replicas.
Go into the Solr Dashboard and select a collection. Open up shard1, and look for the names of the replicas. (You might want to refer to our page on Solr terminology.)
Use the cluster’s Solr Endpoint and the replica names to construct queries in this format:
<Solr Endpoint><Replica Name>/select?q=*:*&rows=0For example,
https://ss123456-jkzior0-ca-central-1-aws.searchstax.com/solr/filmcollection_shard1_replica_n1/select?q=*:*&rows=0
Paste this URL into the address field of a web browser and observe the number of documents in that replica. (Note that IP Filtering and Solr Basic Auth security will apply to this request.)
Perform the same test for the other replica. If the numFound figures don’t match, it’s time to restart the system.
Questions?
Do not hesitate to contact the SearchStax Support Desk.