A Removed Node was not Removed Properly from the Seed Node List¶
Failed to create materialized view after node was removed from the cluster.
InvalidRequest: Error from server: code=2200 [Invalid query] message="Can't create materialized views until the whole cluster has been upgraded"
A removed node was not removed properly from the seed node list.
How to Verify¶
Scylla logs show the error message above.
To verify that the node wasn’t remove properly use the nodetool gossipinfo command
A three nodes cluster, with one node (18.104.22.168) removed.
nodetool gossipinfo /22.214.171.124 generation:1172279348 heartbeat:7212 LOAD:2.0293227179E10 INTERNAL_IP:10.240.0.83 DC:E1 STATUS:NORMAL,-872190912874367364312 HOST_ID:12fdcf43-4642-53b1-a987-c0e825e4e10a RPC_ADDRESS:10.240.0.83 RACK:R1 /126.96.36.199 generation:1657463198 heartbeat:8135 LOAD:2.0114638716E12 INTERNAL_IP:10.240.0.93 DC:E1 STATUS:NORMAL,-258152127640110957173 HOST_ID:99acbh55-1013-24a1-a987-s1w718c1e01b RPC_ADDRESS:10.240.0.93 RACK:R1 /188.8.131.52 generation:1657463198 heartbeat:7022 LOAD:2.5173672157E48 INTERNAL_IP:10.240.0.103 DC:E1 STATUS:NORMAL,-365481201980413697284 HOST_ID:99acbh55-1301-55a1-a628-s4w254c1e01b RPC_ADDRESS:10.240.0.103 RACK:R1
We can see that node
184.108.40.206 is still part of the cluster and need to be removed.
Remove the relevant node from the other nodes seed list (under scylla.yaml) and restart the nodes one by one.
Seed list before remove the node
- seeds: "10.240.0.83,10.240.0.93,10.240.0.103"
Seed list after removing the node
- seeds: "10.240.0.83,10.240.0.93"
Restart Scylla nodes