Well, if you have ever spent two hours doing something completely wrong then you know where I have been today.
I have a Solr Cloud configuration with three shards and no replicas at this time. I am inserting documents then pausing and doing some load tests, then inserting more, until I get to 5 billion.
Well, the linux boxes running Solr and Zookeeper were restarted and Solr was restarted before Zookeeper. Then when Zookeeper was restarted somehow the clusterstate.json was updated so that I now had 1 shard with 2 replicas.
So, all I need to do is fix the clusterstate.json. I stop the Solr instances and go to work. Fortunately I have a copy of the clusterstate.json and I highly recommend saving a copy of your clusterstate.json somewhere safe.
Here is where the trouble starts.
I had found online that you just have to set the clusterstate.json. I assume that you can do this from the zkCli command prompt.
So, I run zkCli.sh and at the prompt I try this:
set /clusterstate.json "`cat /local/path/to/clusterstate.json`"
I get a java NumberFormat Exception.
I tried escaping quotes in the json, removing line feeds, everything I could think of.
Well, long story short, you don't do the command from the zkCli command prompt.
You run the command in your shell/console.
I now do this from the zookeeper bin directory:
#sh zkCli.sh -server 127.0.0.1:2181 set /clusterstate.json "`cat /local/path/to/clusterstate.json`"
Everything worked great.
So, if you are trying to set the clusterstate.json while in ZkCli.sh you will get this error:
zkcli Command failed: java.lang.NumberFormatException: For input string:
Don't run it inside of ZkCli.sh, run the command from the shell.