Remove server from Cluster Print

to remove a node out of the cluster you need to do a few things

please note

YOU CAN NOT ADD THE SAME NODE AGAIN AFTER YOU REMOVED IT WITH OUT A FRESH INSTALL

so now that is said , lets start the tutorial

first things first

Remove monitor in Ceph 
Remove Manager is Ceph
Remove OSD disks in Ceph
make sure nothing is linked to the server from ceph anymore

then we can go to step 2

move all vps servers to a other node, the one you like to keep atleast
stop all vps servers on this node , Really all vps servers that are left
make sure HA is disabled, for ALL vps servers on this node
Remove all avaible Mounting points like iso libary or any other mounts FROM this node/vps on this node

from this point on , you CANT undo  anymore

we going start the real business now

begin proxmox info :D

Remove a Cluster Node

Read carefully the procedure before proceeding, as it could not be what you want or need.

Move all virtual machines from the node. Make sure you have no local data or backups you want to keep, or save them accordingly. In the following example we will remove the node hp4 from the cluster.

Log in to a different cluster node (not hp4), and issue a pvecm nodes command to identify the node ID to remove:

hp1# pvecm nodes

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
         1          1 hp1 (local)
         2          1 hp2
         3          1 hp3
         4          1 hp4

At this point you must power off hp4 and make sure that it will not power on again (in the network) as it is.

As said above, it is critical to power off the node before removal, and make sure that it will never power on again (in the existing cluster network) as it is. If you power on the node as it is, your cluster will be screwed up and it could be difficult to restore a clean cluster state.

After powering off the node hp4, we can safely remove it from the cluster.

 hp1# pvecm delnode hp4

If the operation succeeds no output is returned, just check the node list again with pvecm nodes or pvecm status. You should see something like:

hp1# pvecm status

Quorum information
~~~~~~~~~~~~~~~~~~
Date:             Mon Apr 20 12:44:28 2015
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1/8
Quorate:          Yes

Votequorum information
~~~~~~~~~~~~~~~~~~~~~~
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
0x00000001          1 192.168.15.90 (local)
0x00000002          1 192.168.15.91
0x00000003          1 192.168.15.92

If, for whatever reason, you want this server to join the same cluster again, you have to

• reinstall Proxmox VE on it from scratch

• then join it, as explained in the previous section.

After removal of the node, its SSH fingerprint will still reside in the known_hosts of the other nodes. If you receive an SSH error after rejoining a node with the same IP or hostname, run pvecm updatecerts once on the re-added node to update its fingerprint cluster wide.

 

end of proxmox info , now lets the cleaning start

You probaly already noticed that you dit not saw those nodes already in the list when you powered down the node

so lets start with the second part

cd /etc/pve/nodes/
ls
rm {node}

{node} is the node to remove ofcourse

now lets move on

do below command on ALL NODES exept ofcourse the one you removed ( as it should be down ... and need to stay down ) 

systemctl stop pve-ha-crm

now execute followign comamnd on a SINGLE node

rm /etc/pve/ha/manager_status

and then we start up everything again

 

systemctl start pve-ha-crm

and then finaly you also remove the node from the crush map ( ceph )

ceph osd crush rm {node}

after this everything should be cleaned out

and now it just look like nothing is changed and that that server was not there in the first place :D