SearchStax Managed Search service clients sometimes want to add a node to a Solr cluster to meet seasonal query loads, such as during Christmas shopping or peak summer travel. They want to ensure High Availability during peak seasons without overpaying during the rest of the year.
Upgrade Deployment or Add Nodes?
User queries are automatically distributed over all nodes of your cluster. If the deployment is stressed by queries, adding nodes will address the issue.
Indexing of any specific collection occurs on one node only. If the indexing node is overloaded, upgrading to a deployment with more CPUs, memory, and disk space is the better path.
High Availability is easily achieved by adding nodes to a Solr cluster. This spreads out the query traffic across more servers, which lowers the response time for individual queries.
It is possible to add one or more nodes to a SearchStax cluster and then remove them again later. This is a self-service activity although SearchStax premium clients (gold, platinum, platinum plus) may ask for our help.
(Note that one cannot add a node to a single-node deployment. Single-node architecture does not lend itself to this kind of scaling. To expand a single node system, follow the A-to-B Upgrade path. You can downgrade later if your index size permits.)
Upsize: Adding a Node and Replica
Let’s assume that we have a standard NDC cluster with two Solr nodes (and three independent Zookeeper nodes). Our collection is present as two replicas, one on each Solr node.
To add a node to the cluster, pull down the Options menu in the upper-right corner of the screen. Select the Add Nodes option.
This pops up a modal dialog asking how many nodes, and giving some insight into what this upgrade will cost. Dial in the number of new nodes and click the Add button.
It will take a few minutes for SearchStax to provision the new node.
Now we have a third Solr node, but only the original two replicas of the collection. We need to create another replica.
To add the missing third replica, slick the cluster’s HTTP Endpoint, go to the Solr Dashboard, navigate to the collection and click on shard1. There’s a add replica button in the lower right corner.
This opens a list of nodes. Choose the new node, or just let Solr find it for you. It seems pretty smart about that.
Now we have three nodes and three matching replicas. Our cluster upsizing is complete.
Zookeeper automatically responds to the new node by copying Solr configurations. It responds to the new replica by populating it with index files. This process is transparent to the end user.
Downsize: Removing a Node and Replica
When the busy season is over, you might want to scale back your cluster. It is important to remove both the extra node and the extra replica. This is not intuitive. Deleting the node also delete the physical replica, but Solr and Zookeeper “remember” it. This can cause problems with backups later on.
To remove the extra node, navigate to the Deployment Details page. Click on the Delete Node link.
It takes a few minutes to decommission the node.
This leaves us with the original two-node cluster, but we also have a “ghost” replica haunting Solr.
To remove the ghost, navigate into the Solr Dashboard to the collection and the list of replicas. Click the red X beside the decommissioned replica.
That will remove the “memory” of the replica from Solr so it won’t cause any problems downstream.
That concludes the procedure about how to add a node to a Solr cluster to meet seasonal demands.
Questions?
Do not hesitate to contact the SearchStax Support Desk.