Gabriele.vivinetto: Created page with "Un cluster normale non necessita di quorum, ma non può gestire il failover in automatico, facendo ripartire le VM da un nodo morto ad uno vivo. * Proxmox 3.x supporta un clu..."

2016-01-19T15:40:24Z

Created page with "Un cluster normale non necessita di quorum, ma non può gestire il failover in automatico, facendo ripartire le VM da un nodo morto ad uno vivo. * Proxmox 3.x supporta un clu..."

New page

Un cluster normale non necessita di quorum, ma non può gestire il failover in automatico, facendo ripartire le VM da un nodo morto ad uno vivo.

* Proxmox 3.x supporta un cluster HA a 2 nodi +1 1 qorum disk e necessita della configurazione del fencing
* Proxmox 4x invece necessita di almeno 3 nodi, NON supporta il quorum disk, ma NON necessita della configurazione del fencing, perchè è autogestito.

=Proxmox 3.x=

==Configurazione del Fencing tramite IPMI==

sudo apt-get install ipmitool
sudo modprobe ipmi_si
sudo modprobe ipmi_devintf
sudo modprobe ipmi_msghandler

sudoedit /etc/modules

ipmi_devintf
ipmi_si
ipmi_msghandler

sudo /etc/init.d/kmod start

sudoedit /etc/default/ipmievd

ENABLED=true

sudo /etc/init.d/ipmievd restart

sudo ipmitool -I open chassis status
sudo ipmitool lan set 1 ipsrc static
sudo ipmitool lan print 1
sudo ipmitool lan set 1 ipaddr 192.168.41.32
sudo ipmitool lan set 1 netmask 255.255.255.0
sudo ipmitool lan set 1 defgw ipaddr 192.168.41.254
sudo ipmitool lan set 1 defgw macaddr 00:0f:fe:24:36:0b
sudo ipmitool lan set 1 arp respond on
ping 192.168.41.32
sudo ipmitool lan set 1 auth ADMIN MD5
sudo ipmitool lan set 1 access on
sudo ipmitool lan print 1
sudo ipmitool user set name 2 admin
sudo ipmitool user set password 2
sudo ipmitool channel setaccess 1 2 link=on ipmi=on callin=on privilege=4
sudo ipmitool user enable 2

Da un altro pc

ipmitool -I lan -H 192.168.41.31 -E chassis status

ipmitool -I lan -H 192.168.41.31 -E chassis power off
fa shutdown

ipmitool -I lan -H 192.168.41.31 -E chassis power on

sudo apt-get remove --purge acpi acpid

ipmitool -I lan -H 192.168.41.31 -E chassis power off

si spegne hard

ipmitool -I lan -H 192.168.41.31 -E chassis power on

===Abilitazione del Fencing===

Eseguire su tutti i nodi:

* Enable fencing in /etc/default/redhat-cluster-pve (Just uncomment the last line, see below):

sudoedit /etc/default/redhat-cluster-pve

FENCE_JOIN="yes"

sudo /etc/init.d/cman restart

* Join teh fence domain

sudo fence_tool join

* Verificare
fence_tool ls

* Probabilmente il cluster andrà out fo sync, su tutti i nodi restartare:
service pve-cluster restart

===Configurazione fencing===

* Fare una copia del file di configurazione
sudo cp /etc/pve/cluster.conf /etc/pve/cluster.conf.new

* Incrementare la versione
sudoedit /etc/pve/cluster.conf.new

<cluster alias="hpiloclust" config_version="12" name="hpiloclust">

* Imposatre la modalità a due nodi:
<cman keyfile="/var/lib/pve-cluster/corosync.authkey"
two_node="1" expected_votes="1"/>

* Aggiungere i fence devices fuori dai tag >clusternodes>:
<fencedevices>
<fencedevice agent="fence_ipmilan" name="galprox01-fence" login="" ipaddr="192.168.41.31" passwd="gal80xl700" power_wait="5"/>
<fencedevice agent="fence_ipmilan" name="galprox02-fence" login="" ipaddr="192.168.41.32" passwd="gal80xl700" power_wait="5"/>
</fencedevices>

* Ora in ogni <clusternode></clusternode> inserire quale fencing device va usato:

<clusternodes>
<clusternode name="galprox01" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="galprox01-fence" action="off"/>
</method>
</fence>
</clusternode>

<clusternodes>
<clusternode name="galprox02" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="galprox02-fence" action="off"/>
</method>
</fence>
</clusternode>

* Testare la sintassi:
sudo ccs_config_validate -v -f /etc/pve/cluster.conf.new

* Ecco un esempio:

<pre>
<?xml version="1.0"?>
<cluster config_version="12" name="M4">
<cman keyfile="/var/lib/pve-cluster/corosync.authkey"
two_node="1" expected_votes="1"/>

<fencedevices>
<fencedevice agent="fence_ipmilan" name="galprox01-fence" ipaddr="192.168.41.31" login="" passwd="gal80xl700" power_wait="5"/>
<fencedevice agent="fence_ipmilan" name="galprox02-fence" ipaddr="192.168.41.32" login="" passwd="gal80xl700" power_wait="5"/>
</fencedevices>

<clusternodes>
<clusternode name="galprox01" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="galprox01-fence" action="off"/>
</method>
</fence>
</clusternode>
<clusternode name="galprox02" nodeid="2" votes="1">
<fence>
<method name="1">
<device name="galprox02-fence" action="off"/>
</method>
</fence>
</clusternode>
</clusternodes>
<rm/>
</cluster>

</pre>
* Tornare nella gui, in Datacenter/HA e premere
Activate

* testare ora dal nodo2, deve spegnersi il nodo1
fence_node galprox01 -vv

fence galprox01 dev 0.0 agent fence_ipmilan result: success
agent args: action=off nodename=galprox01 agent=fence_ipmilan ipaddr=192.168.41.31 login= passwd=gal80xl700 power_wait=5
fence galprox01 success

* Riavviarlo via ipmi

==Impostazione disco di quorum==

* Utilizzando un qorum disk, si evitano i problemi dell'architettura a 2 nodi, simulandone un terzo

* Un disco di quorum può essere ricavato sottranedo 100mb ad un volume lvm di swap da uno storage condiviso

* Verificare che lo swap possa essere disabilitato
$ free
total used free shared buffers cached
Mem: 6112636 5402828 709808 0 11204 5259216
-/+ buffers/cache: 132408 5980228
Swap: 3903484 0 3903484

*Disablitarlo
sudo swapoff -a

*Verificare

$ free
total used free shared buffers cached
Mem: 6112636 5402780 709856 0 11308 5260048
-/+ buffers/cache: 131424 5981212
Swap: 0 0 0

* Identificare il volume di swap:
cat /etc/fstab| grep swap
/dev/mapper/vgroot-swap none swap sw 0 0

* Ridurlo di 100Mb

sudo lvreduce -L -100M /dev/mapper/vgroot-swap

WARNING: Reducing active logical volume to 3.62 GiB
THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce swap? [y/n]: y
Reducing logical volume swap to 3.62 GiB
Logical volume swap successfully resized

* Ricreare lo swap
sudo mkswap /dev/mapper/vgroot-swap

mkswap: /dev/mapper/vgroot-swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.
Setting up swapspace version 1, size = 3801084 KiB
no label, UUID=bcfbdfef-72f8-49e1-8a44-961aec0b22ff

* Riabilitare lo swap:
$ sudo swapon -a
mnt.vvngrl@galstorage02[2015-07-21 19:50:23]
~
$ free
total used free shared buffers cached
Mem: 6112636 5409352 703284 0 11648 5264896
-/+ buffers/cache: 132808 5979828
Swap: 3801084 0 3801084

* Create the LV to be shared via iSCSI:

sudo lvcreate -n proxmox2quorumdisk -L 10M vgroot
* Verificare
sudo lvdisplay | grep proxmox2quorumdisk

* Installare iscsi
sudo apt-get install iscsitarget iscsitarget-dkms

* Abilitare:
sudoedit /etc/default/iscsitarget

ISCSITARGET_ENABLE=true

* Dichiare le risorse sharata
sudoedit /etc/iet/ietd.conf

Target iqn.2015-07.priv.example:myserver.lun1
IncomingUser iscsi_username iscsi_password
OutgoingUser
#make sure the partition isn't mounted :
Lun 0 Path=/dev/mapper/vgroot-proxmox2quorumdisk,Type=fileio
Alias LUN1
#MaxConnections 6

* Consentire la connessione
sudoedit /etc/iet/initiators.allow

ALL ALL
sudo /etc/init.d/iscsitarget restart

===Collegamento quorum disk===

* Sui nodi proxmox, collegare il quorum disk

* Impostare l'autenticazione e la partenza automatica del demone
sudoedit /etc/iscsi/iscsid.conf

<pre>
node.startup = automatic
node.session.auth.username = iscsi_username
node.session.auth.password = iscsi_password
discovery.sendtargets.auth.username = iscsi_username
discovery.sendtargets.auth.password = iscsi_password
</pre>

sudo /etc/init.d/open-iscsi restart

* Elencare le LUN esposte dallo storage:
sudo iscsiadm -m discovery -t sendtargets -p 192.168.41.103

192.168.41.103:3260,1 iqn.2015-07.priv.galimberti.m4:galstorage02.lun1

* Collgeare la LUN con l'id precedente con il comando:

sudo iscsiadm --mode node --targetname iqn.2015-07.priv.galimberti.m4:galstorage02.lun1 --portal 192.168.41.103 --login

<pre>
Logging in to [iface: default, target: iqn.2015-07.priv.galimberti.m4:galstorage02.lun1, portal: 192.168.41.103,3260] (multiple)
Login to [iface: default, target: iqn.2015-07.priv.galimberti.m4:galstorage02.lun1, portal: 192.168.41.103,3260] successful.
</pre>

* Verificare che sia stato aggiunto il disco:

sudo less /var/log/syslog

Jul 22 09:31:26 galprox02 kernel: scsi 7:0:0:0: Direct-Access IET VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Jul 22 09:31:26 galprox02 kernel: sd 7:0:0:0: Attached scsi generic sg4 type 0
Jul 22 09:31:26 galprox02 kernel: sd 7:0:0:0: [sdb] 24576 512-byte logical blocks: (12.5 MB/12.0 MiB)
Jul 22 09:31:26 galprox02 kernel: sd 7:0:0:0: [sdb] Write Protect is off
Jul 22 09:31:26 galprox02 kernel: sd 7:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jul 22 09:31:26 galprox02 kernel: sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jul 22 09:31:26 galprox02 kernel: sdb: unknown partition table
Jul 22 09:31:26 galprox02 kernel: sd 7:0:0:0: [sdb] Attached SCSI disk
Jul 22 09:31:26 galprox02 iscsid: Connection1:0 to [target: iqn.2015-07.priv.galimberti.m4:galstorage02.lun1, portal: 192.168.41.103,3260] through [iface: default] is operational now

===Preparazione del quorum disk===
* Non occorrerà montare il disco, verrà gestito direttamente dal cluster

* Creare la partizione sul quorum disk (utilizzare l'unità corretta):
sudo fdisk /dev/sdb
...

* Formattare il quorum disk:
sudo mkqdisk -c /dev/sdb1 -l proxmox1_qdisk

mkqdisk v1364188437

Writing new quorum disk label 'proxmox1_qdisk' to /dev/sdb1.
WARNING: About to destroy all data on /dev/sdb1; proceed [N/y] ? y
Initializing status block for node 1...
Initializing status block for node 2...
Initializing status block for node 3...
Initializing status block for node 4...
Initializing status block for node 5...
Initializing status block for node 6...
Initializing status block for node 7...
Initializing status block for node 8...
Initializing status block for node 9...
Initializing status block for node 10...
Initializing status block for node 11...
Initializing status block for node 12...
Initializing status block for node 13...
Initializing status block for node 14...
Initializing status block for node 15...
Initializing status block for node 16...

* Procedere al collegamento del quorum disk sugli altri nodi, senza ricreare il quorum disk, naturalmente

===Impostazione del cluster per utilizzo quorum disk===
* Creare la nuova configurazione:
sudo cp /etc/pve/cluster.conf /etc/pve/cluster.conf.new

* Editarla
sudoedit /etc/pve/cluster.conf.new

* Incrementare il numero di configurazione
* Rimuovere la direttiva two_node="1"
* Impostare expected_votes="3"
* Aggiungere le direttive per l'utilizzo del quorum disk:

<pre>
<?xml version="1.0"?>
<cluster config_version="14" name="M4">
<cman keyfile="/var/lib/pve-cluster/corosync.authkey"
expected_votes="3"/>
<quorumd votes="1" allow_kill="0" interval="1" label="proxmox1_qdisk" tko="10"/>
<totem token="54000"/>

</pre>

* Validare la configurazione
sudo ccs_config_validate -v -f /etc/pve/cluster.conf.new

* Attivarla da web gui

* Verificare lo stato del cluster:

sudo pvecm s

Version: 6.2.0
Config Version: 14
Cluster Name: M4
Cluster Id: 206
Cluster Member: Yes
Cluster Generation: 356
Membership state: Cluster-Member
Nodes: 2
Expected votes: 3
Total votes: 2
Node votes: 1
Quorum: 2
Active subsystems: 6
Flags:
Ports Bound: 0 177
Node name: galprox02
Node ID: 2
Multicast addresses: 239.192.0.206
Node addresses: 192.168.41.102

=== Abilitazione quorum disk===

* Stoppare rgmanager:
sudo /etc/init.d/rgmanager stop
Stopping Cluster Service Manager: [ OK ]
root@galprox01[2015-07-22 10:11:25]
~

* Fare il reload delal configurazione del custer, per attivare il qorum disk:

sudo /etc/init.d/cman reload

Stopping cluster:
Leaving fence domain... [ OK ]
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping qdiskd... [ OK ]
Stopping cman... [ OK ]
Waiting for corosync to shutdown:[ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Starting qdiskd... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Tuning DLM kernel config... [ OK ]
Unfencing self... [ OK ]
Joining fence domain... [ OK ]

* Restartare rgmanager se fermo:

sudo /etc/init.d/rgmanager status

rgmanager is stopped

sudo /etc/init.d/rgmanager start

Starting Cluster Service Manager: [ OK ]

* Verificare che il quorum disk sia attivo:
sudo clustat

Cluster Status for M4 @ Wed Jul 22 10:16:34 2015
Member Status: Quorate

Member Name ID Status
------ ---- ---- ------
galprox01 1 Online, Local
galprox02 2 Online
/dev/block/8:17 0 Online, Quorum Disk

* Verificare lo stato,e notare che ora c'è:

sudo pvecm s

Quorum device votes: 1
Total votes: 3

* Farlo su tutti i nodi

==Test==
* Aggiungere una VM in HA via web gui
* Spostare la VM su un nodo in modo che ci sia solo quella
* spegnere il nodo via ipmi
* Nel syslog del nodo attivo si vedrà che verrà fatto il fencing, e quando avrà avuto successo, verrà riavviata la VM
* Se al nodo viene fisicamente scollegata l'alimentazione o viene scollegato dalla lan, il fnecing non avrà mai successo, e verrà ritentato infinitamente
Jul 22 14:35:53 galprox02 fence_ipmilan: Failed: Unable to obtain correct plug status or plug is not available

* In questo caso, occorre confermare l'avvenuto spegnmento della macchina scollegata con
sudo fence_ack_manual galprox01
[sudo] password for mnt.vvngrl:
About to override fencing for galprox01.
Improper use of this command can cause severe file system damage.

Continue [NO/absolutely]? absolutely
Done

* Appena fatto questo, la VM in HA verrà riavviata.

==Riferimenti==

*[https://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster#Configuring_Fencing Two-Node High Availability Cluster - Proxmox VE]
*[http://pve.proxmox.com/wiki/Fencing#Test_fencing Fencing - Proxmox VE]
*[http://forum.proxmox.com/threads/14162-Error-fencing-proxmox-cluster-node Error fencing proxmox cluster node]
*[http://www.redhat.com/archives/linux-cluster/2011-July/msg00005.html Re: [Linux-cluster] fence_ipmilan fails to reboot - SOLVED]
*[http://www.redhat.com/archives/linux-cluster/2011-July/msg00005.html Re: [Linux-cluster] fence_ipmilan fails to reboot - SOLVED]
*[http://ibiblio.org/gferg/ldp/IPMI_on_Debian.html IPMI HOWTO for Debian GNU/Linux on the Intel SR2300 (Server Board SE7501WV2)]
*[https://pve.proxmox.com/wiki/Fencing#IPMI_.28generic.29 Fencing - Proxmox VE]
*[http://insanelabs.com/linux/debian-open-iscsi-use-iscsi-initiator-to-connect-to-a-san/ Debian: open-iscsi, use iSCSI initiator to connect to a SAN | Ali Aboosaidi]

*[http://www.vionblog.com/debian-iscsi-initiator-and-target/ Debian iSCSI Initiator and Target - VION Technology Blog]

=Proxmox 4.x=
TBD

Configurare un Cluster in HA in Proxmox - Revision history

Gabriele.vivinetto: Created page with "Un cluster normale non necessita di quorum, ma non può gestire il failover in automatico, facendo ripartire le VM da un nodo morto ad uno vivo. * Proxmox 3.x supporta un clu..."