Saturday, May 22, 2010

Zimbra backup - open source way

Every information technology system must be backed up frequently. Backup if performed correctly - should allow you for fast recovery after any disaster which might damage the entire system or for recovery of accidentally deleted or corrupted files. Unlike the Professional (commercial) version - open-source Zimbra is not equipped with any build in backup system. You need to take care of backing up your MTA by yourselve. Shell scripting and crontab prove to be useful once again.

#########################################
# Zimbra backup cut and paste howto
#########################################

mkdir /backup

mkdir /backup/zimbra

crontab -e

# Backup Zimbra Nightly
0 1 * * * /usr/local/bin/backup.sh


vim /usr/local/bin/backup.sh

#!/bin/bash

echo Time backup started = $(date +%T)
before="$(date +%s)"

rsync -avHK --delete /opt/zimbra/ /backup/zimbra

before2="$(date +%s)"

su - zimbra -c"/opt/zimbra/bin/zmcontrol stop"
sleep 15

kill -9 `ps -u zimbra -o "pid="`

rsync -avHK --delete /opt/zimbra/ /backup/zimbra

su - zimbra -c "/opt/zimbra/bin/zmcontrol start"

after="$(date +%s)"
elapsed="$(expr $after - $before2)"
hours=$(($elapsed / 3600))
elapsed=$(($elapsed - $hours * 3600))
minutes=$(($elapsed / 60))
seconds=$(($elapsed - $minutes * 60))
echo Server was down for: "$hours hours $minutes minutes $seconds seconds"

su - zimbra -c "zmcontrol -v > /backup/zimbra/conf/zimbra_version.txt"

echo Displaying Zimbra services status...
su - zimbra -c "/opt/zimbra/bin/zmcontrol status"

tar -zcvf /tmp/mail.backup.tgz -C /backup/zimbra .

echo Time backup finished = $(date +%T)

after="$(date +%s)"
elapsed="$(expr $after - $before)"
hours=$(($elapsed / 3600))
elapsed=$(($elapsed - $hours * 3600))
minutes=$(($elapsed / 60))
seconds=$(($elapsed - $minutes * 60))
echo Time taken: "$hours hours $minutes minutes $seconds seconds"

[root@mta002 ~]# less /usr/local/bin/backup.sh
[root@mta002 ~]# date
Mon Mar 29 09:57:26 EDT 2010
[root@mta002 ~]# date
Mon Mar 29 09:57:31 EDT 2010
[root@mta002 ~]# crontab -e
crontab: installing new crontab
[root@mta002 ~]# clear
[root@mta002 ~]# less /usr/local/bin/backup.sh
[root@mta002 ~]# cat /usr/local/bin/backup.sh

echo Time backup started = $(date +%T)
before="$(date +%s)"

rsync -avHK --delete /opt/zimbra/ /backup/zimbra

before2="$(date +%s)"

su - zimbra -c"/opt/zimbra/bin/zmcontrol stop"
sleep 15

kill -9 `ps -u zimbra -o "pid="`

rsync -avHK --delete /opt/zimbra/ /backup/zimbra

su - zimbra -c "/opt/zimbra/bin/zmcontrol start"

after="$(date +%s)"
elapsed="$(expr $after - $before2)"
hours=$(($elapsed / 3600))
elapsed=$(($elapsed - $hours * 3600))
minutes=$(($elapsed / 60))
seconds=$(($elapsed - $minutes * 60))
echo Server was down for: "$hours hours $minutes minutes $seconds seconds"

su - zimbra -c "zmcontrol -v > /backup/zimbra/conf/zimbra_version.txt"

echo Displaying Zimbra services status...
su - zimbra -c "/opt/zimbra/bin/zmcontrol status"

tar -zcvf /tmp/mail.backup.tgz -C /backup/zimbra .

#Copy to NAS or Windows share

#change 192.168.0.6 to the IP of your network device
#change yourname and yourpassword to the user that has
#permissions to use your network device

mount -t cifs //192.168.0.6/backup /mnt/bac001/ -o "username=yourname,password=yourpassword"
mount -t cifs //192.168.0.7/backup /mnt/bac002/ -o "username=yourusername,password=yourpassword"
cp /tmp/mail.backup.tgz /mnt/bac001/
cp /tmp/mail.backup.tgz /mnt/bac002/
umount /mnt/bac00*
######################################

Wednesday, May 19, 2010

When stats fail

This summary is not available. Please click here to view the post.

Tuesday, May 18, 2010

MIGRATING ZIMBRA FROM OPEN SOURCE TO NETWORK EDITION

First of all start with blocking your VM's IP address at your firewall. In my case the IP address of Zimbra VM is 10.20.0.26. This is a standard procedure while migrating any MTA - any message will now bounce off the server, it will be delayed. Any server trying to send an email will now wait up to 5 days until your MTA is available again. This way you will not loose any e-mails.

Next - VERY IMPORTANT - TAKE A BACKUP of your current installation, tar it and copy it somewhere. Only then you are allowed to start the installation.

############################################
INSTALL ZIMBRA NE
############################################

Get the newest version of Zimbra from here: http://www.zimbra.com/downloads/ne-downloads.html

wget http://files2.zimbra.com/downloads/6.0.6_GA/zcs-NETWORK-6.0.6_GA_2324.RHEL5_64.20100406133038.tgz

According to official Zimbra documentation you cannot install ZCS without a license. This however is not entirely true. I was having issues with Zimbra installer not being able to recognize the valid license and I recovered from it which I'll explain later on in this post. So if you get an error in regards to the license - don't panic - just continue the installation.

Before the upgrade, place the license in /opt/zimbra/conf/ZCSLicense.xml.

tar -xvf zcs-NETWORK-6.0.6_GA_2324.RHEL5_64.20100406133038.tgz

cd zcs-NETWORK-6.0.6_GA_2324.RHEL5_64.20100406133038

./install.sh --platform-override

Select the packages to install

Install zimbra-ldap [Y] y

Install zimbra-logger [Y] y

Install zimbra-mta [Y] y

Install zimbra-snmp [Y] y

Install zimbra-store [Y] y

Install zimbra-apache [Y] y

Install zimbra-spell [Y] y

Install zimbra-memcached [N] y

Install zimbra-proxy [N] y


Looking for valid license to install...failed to install license.
*******ERROR
Failed to install a license - this will prevent your server from functioning properly
Please contact Zimbra to obtain a license
Press RETURN to continue

It probably is a permission problem - try this:

chown zimbra:zimbra /opt/zimbra/conf/ZCSLicense.xml
chmod 444 /opt/zimbra/conf/ZCSLicense.xml

If it didn't work then don't panic - just continue with the installation. Zimbra will install anyway and you will be able to upload the license from the administration console.


Zimbra - Installation

This howto is a step-by-step guide on installing the Zimbra Collaboration Suite (ZCS) on Centos 5.4 64bit. Zimbra is an open source collaboration suite. It consists of the following: email, group calendaring, contacts, web document management and authoring. It is fully compatible with clients such as Microsoft Outlook, Apple Mail, and Novell Evolution so that mail, contacts, and calendar items can be synchronised from these to the ZCS server as well as to many mobile devices. But all of those features fully work only with Zimbra Network Edition.

It's usually good idea to virtualize your servers - so we will install Zimbra MTA on a Virtual Machine where the quest operating system is going to be CentOS 5.4 64 bit sitting on top of VmWare ESXi 4.0. This will make your installation fully hardware independent thus allowing for migration to more capable hardware in the feature.

I also decided to explain how to configure your DNS in order for your MTA to be visible from the outside world. I noticed that most authors of the tutorials don't do that. I'm using http://atrianetworks.com/ as my ISP and http://ccjclearline.com/ to manage my domains and DNS records plus BIND as my internal DNS.

Lets assume the following:

INTERNAL IP (The IP of your VM): 10.20.0.26
EXTERNAL IP :226.111.111.111
IP OF YOUR INTERNAL DNS/DHCP SERVER: 10.20.0.10


########################################
CREATE A VM (ESXi 4.0)
########################################

* Create a VM with the following settings:
HD: 100GB thin provisioning.
RAM: 4096
Version: Red Hat Enterprise Linux 5 (64bit)

* Install Centos 5.4 x86_64

########################################
INSTALL VMWARE-TOOLS INSIDE YOUR GUEST OS
########################################

yum install gcc gcc-c++ kernel-devel

---------------------------------------------------------------
NOW RIGHT CLICK YOUR VM >> GUEST >> INSTALL VMWARE TOOLS
---------------------------------------------------------------

mkdir /usr/src/linux

ln -s /usr/src/kernels/2.6.18-164.15.1.el5-x86_64 /usr/src/linux

mkdir /mnt/cdrom

mount /dev/cdrom /mnt/cdrom

cp /mnt/cdrom/VMwareTools-4.0.0-171294.tar.gz /tmp/

umount /mnt/cdrom

cd /tmp

tar -xvf VMwareTools-4.0.0-171294.tar.gz

cd vmware-tools-distrib

./vmware-install.pl

reboot

At his moment people usually jump straight to Zimbra installation which is a mistake. First you need to configure the DNS entries in several places. Otherwise your installation will fail and/or your Zimbra will not be visible from outside the world. As I've already said above - I'm using ccjclearline to manage my domains and DNS antries, the general process though should be fairly similar with any provider.

########################################
DNS CONFIG (EXTERNAL)
########################################
https://ccjclearline.managednsservice.com/
login: yourlogin
pass:yourpass

Find yourdomain.com and hit "zone management" button.

Add A record: HOSTNAME: mail.yourdomain.com IP Address: 226.111.111.111
Add A record: HOSTNAME: yourdomain.com IP Address: 226.111.111.111

Add MX record: HOSTNAME: mail.yourdomain.com Mail Exchange: mail.yourdomain.com. Preference: 10

Hit "Update zone preferences" button.

########################################
FIREWALL Config
########################################

Create NAT rules

Source: 10.20.0.26
Interface: outside
Use IP address:226.111.111.111
Enable Port Address Translation TCP Original Port: smtp Translated Port: smtp

Add similiar rules for: imap4 , http nad and 993

Permit tcp/smtp from 10.20.0.26 to ANY !!!


It might be that you will not need to perform the next step - It highly depends on your particular setup - I had to do it though.
########################################
DNS CONFIG (ISP)
########################################
Send a message to : support@atrianetworks.com with the following body:

Can you please add a reverse DNS entry for 226.111.111.111 to mail.yourdomain.com ?

########################################
DNS CONFIG (INTERNAL - BIND)
########################################

Ssh to 10.20.0.10

vim /etc/bind/yourdomain.com.internal.zone
########################################
$ORIGIN .
$TTL 86400 ; 1 day
yourdomain.com IN SOA ns1.yourdomain.com. admin\@yourdomain.com. (
2010012205 ; serial
10800 ; refresh (3 hours)
15 ; retry (15 seconds)
604800 ; expire (1 week)
10800 ; minimum (3 hours)
)
NS ns1.yourdomain.com.
NS ns2.yourdomain.com.
$ORIGIN yourdomain.com.
IN MX 10 mail.yourdomain.com.
IN A 10.20.0.26
mail IN A 10.20.0.26
########################################

vim /etc/bind/named.conf.local
########################################
zone "yourdomain.com" {
type master;
file "/etc/bind/yourdomain.com.internal.zone";
allow-update { key DHCP_UPDATER; };
allow-query { internal_networks; };
allow-transfer { internal_networks; };
notify yes;
};
########################################

rndc reload & tail -f /var/log/daemon.log

nslookup mail.yourdomain.com

you should get:
...
Name: mail.yourdomain.com
Address: 10.20.0.26

On 10.20.0.26 (your Zimbra VM) do: vim /etc/hosts
########################################
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
10.20.0.26 mail.yourdomain.com mta002
########################################

rndc thaw #on the DNS - 10.20.0.10 rndc thaw. This command allows the changes to be reloaded and the zone to participate in dynamic updates.

tail -f /var/log/daemon.log /var/log/named-auth.info /var/log/update-debug.log |grep 10.20.0.26

########################################
DHCP CONFIG (on 10.20.0.10)
########################################

vim /etc/dhcp3/dhcpd.conf

host mta002 {
hardware ethernet 00:0C:29:C8:C8:C8;
option host-name "mta002";
option domain-name "yourdomain.com";
ddns-hostname "mta002";
ddns-domainname "yourdomain.com";
ddns-rev-domainname "0.20.10.in-addr.arpa";
fixed-address 10.20.0.26;
}

Only after all the above steps you can now move to install Zimbra.

########################################
ZIMBRA INSTALLATION
########################################

chkconfig sendmail off
chkconfig ip6tables off
chkconfig iptables off

yum install libtool-ltdl

yum install gmp

yum install compat-libstdc++-33

yum update

yum install openssh-server

wget http://h.yimg.com/lo/downloads/6.0.4_GA/zcs-6.0.4_GA_2038.RHEL5_64.20091214192925.tgz

tar xvfz zcs-6.0.4_GA_2038.RHEL5_64.20091214192925.tgz

cd /tmp/zcs-6.0.4_GA_2038.RHEL5_64.20091214192925

./install.sh --platform-override

Select the packages to install

Install zimbra-ldap [Y] y

Install zimbra-logger [Y] y

Install zimbra-mta [Y] y

Install zimbra-snmp [Y] y

Install zimbra-store [Y] y

Install zimbra-apache [Y] y

Install zimbra-spell [Y] y

Install zimbra-memcached [N] y

Install zimbra-proxy [N] y

########################################

Main menu

1) Common Configuration:
2) zimbra-ldap: Enabled
3) zimbra-store: Enabled
+Create Admin User: yes
+Admin user to create: admin@mail.yourdomain.com
******* +Admin Password UNSET
+Enable automated spam training: yes
+Spam training user: spam.fd9i2srv_@mail.yourdomain.com
+Non-spam(Ham) training user: ham.m6eg4zfo@mail.yourdomain.com
+Global Documents Account: wiki@mail.yourdomain.com
+SMTP host: mail.yourdomain.com
+Web server HTTP port: 80
+Web server HTTPS port: 443
+Web server mode: http
+IMAP server port: 7143
+IMAP server SSL port: 7993
+POP server port: 7110
+POP server SSL port: 7995
+Use spell check server: yes
+Spell server URL: http://mail.yourdomain.com:7780/aspell.php
+Enable version update checks: TRUE
+Enable version update notifications: TRUE
+Version update notification email: admin@mail.yourdomain.com
+Version update source email: admin@mail.yourdomain.com

4) zimbra-mta: Enabled
5) zimbra-snmp: Enabled
6) zimbra-logger: Enabled
7) zimbra-spell: Enabled
8) zimbra-proxy: Enabled
9) Default Class of Service Configuration:
r) Start servers after configuration yes
s) Save config to file
x) Expand menu
q) Quit

Address unconfigured (**) items (? - help)

########################################
Hit 3 and Enter

Hit 4 and Enter

Hit r

Hit s

https://10.20.0.26:7071/zimbraAdmin/

https://10.20.0.26:7071

Log in to https://10.20.0.26:7071/zimbraAdmin/ and add domain:
Domain name: yourdomain.com
Public service host name: yourdomain.com
Inbound SMTP host name:: mail.yourdomain.com


You are done. Your server should be available from both - internal and external network.

Add some users, run some test and enjoy :)

Followers