Saturday, May 22, 2010

Zimbra backup - open source way

Every information technology system must be backed up frequently. Backup if performed correctly - should allow you for fast recovery after any disaster which might damage the entire system or for recovery of accidentally deleted or corrupted files. Unlike the Professional (commercial) version - open-source Zimbra is not equipped with any build in backup system. You need to take care of backing up your MTA by yourselve. Shell scripting and crontab prove to be useful once again.

#########################################
# Zimbra backup cut and paste howto
#########################################

mkdir /backup

mkdir /backup/zimbra

crontab -e

# Backup Zimbra Nightly
0 1 * * * /usr/local/bin/backup.sh


vim /usr/local/bin/backup.sh

#!/bin/bash

echo Time backup started = $(date +%T)
before="$(date +%s)"

rsync -avHK --delete /opt/zimbra/ /backup/zimbra

before2="$(date +%s)"

su - zimbra -c"/opt/zimbra/bin/zmcontrol stop"
sleep 15

kill -9 `ps -u zimbra -o "pid="`

rsync -avHK --delete /opt/zimbra/ /backup/zimbra

su - zimbra -c "/opt/zimbra/bin/zmcontrol start"

after="$(date +%s)"
elapsed="$(expr $after - $before2)"
hours=$(($elapsed / 3600))
elapsed=$(($elapsed - $hours * 3600))
minutes=$(($elapsed / 60))
seconds=$(($elapsed - $minutes * 60))
echo Server was down for: "$hours hours $minutes minutes $seconds seconds"

su - zimbra -c "zmcontrol -v > /backup/zimbra/conf/zimbra_version.txt"

echo Displaying Zimbra services status...
su - zimbra -c "/opt/zimbra/bin/zmcontrol status"

tar -zcvf /tmp/mail.backup.tgz -C /backup/zimbra .

echo Time backup finished = $(date +%T)

after="$(date +%s)"
elapsed="$(expr $after - $before)"
hours=$(($elapsed / 3600))
elapsed=$(($elapsed - $hours * 3600))
minutes=$(($elapsed / 60))
seconds=$(($elapsed - $minutes * 60))
echo Time taken: "$hours hours $minutes minutes $seconds seconds"

[root@mta002 ~]# less /usr/local/bin/backup.sh
[root@mta002 ~]# date
Mon Mar 29 09:57:26 EDT 2010
[root@mta002 ~]# date
Mon Mar 29 09:57:31 EDT 2010
[root@mta002 ~]# crontab -e
crontab: installing new crontab
[root@mta002 ~]# clear
[root@mta002 ~]# less /usr/local/bin/backup.sh
[root@mta002 ~]# cat /usr/local/bin/backup.sh

echo Time backup started = $(date +%T)
before="$(date +%s)"

rsync -avHK --delete /opt/zimbra/ /backup/zimbra

before2="$(date +%s)"

su - zimbra -c"/opt/zimbra/bin/zmcontrol stop"
sleep 15

kill -9 `ps -u zimbra -o "pid="`

rsync -avHK --delete /opt/zimbra/ /backup/zimbra

su - zimbra -c "/opt/zimbra/bin/zmcontrol start"

after="$(date +%s)"
elapsed="$(expr $after - $before2)"
hours=$(($elapsed / 3600))
elapsed=$(($elapsed - $hours * 3600))
minutes=$(($elapsed / 60))
seconds=$(($elapsed - $minutes * 60))
echo Server was down for: "$hours hours $minutes minutes $seconds seconds"

su - zimbra -c "zmcontrol -v > /backup/zimbra/conf/zimbra_version.txt"

echo Displaying Zimbra services status...
su - zimbra -c "/opt/zimbra/bin/zmcontrol status"

tar -zcvf /tmp/mail.backup.tgz -C /backup/zimbra .

#Copy to NAS or Windows share

#change 192.168.0.6 to the IP of your network device
#change yourname and yourpassword to the user that has
#permissions to use your network device

mount -t cifs //192.168.0.6/backup /mnt/bac001/ -o "username=yourname,password=yourpassword"
mount -t cifs //192.168.0.7/backup /mnt/bac002/ -o "username=yourusername,password=yourpassword"
cp /tmp/mail.backup.tgz /mnt/bac001/
cp /tmp/mail.backup.tgz /mnt/bac002/
umount /mnt/bac00*
######################################

Wednesday, May 19, 2010

When stats fail

This summary is not available. Please click here to view the post.

Tuesday, May 18, 2010

MIGRATING ZIMBRA FROM OPEN SOURCE TO NETWORK EDITION

First of all start with blocking your VM's IP address at your firewall. In my case the IP address of Zimbra VM is 10.20.0.26. This is a standard procedure while migrating any MTA - any message will now bounce off the server, it will be delayed. Any server trying to send an email will now wait up to 5 days until your MTA is available again. This way you will not loose any e-mails.

Next - VERY IMPORTANT - TAKE A BACKUP of your current installation, tar it and copy it somewhere. Only then you are allowed to start the installation.

############################################
INSTALL ZIMBRA NE
############################################

Get the newest version of Zimbra from here: http://www.zimbra.com/downloads/ne-downloads.html

wget http://files2.zimbra.com/downloads/6.0.6_GA/zcs-NETWORK-6.0.6_GA_2324.RHEL5_64.20100406133038.tgz

According to official Zimbra documentation you cannot install ZCS without a license. This however is not entirely true. I was having issues with Zimbra installer not being able to recognize the valid license and I recovered from it which I'll explain later on in this post. So if you get an error in regards to the license - don't panic - just continue the installation.

Before the upgrade, place the license in /opt/zimbra/conf/ZCSLicense.xml.

tar -xvf zcs-NETWORK-6.0.6_GA_2324.RHEL5_64.20100406133038.tgz

cd zcs-NETWORK-6.0.6_GA_2324.RHEL5_64.20100406133038

./install.sh --platform-override

Select the packages to install

Install zimbra-ldap [Y] y

Install zimbra-logger [Y] y

Install zimbra-mta [Y] y

Install zimbra-snmp [Y] y

Install zimbra-store [Y] y

Install zimbra-apache [Y] y

Install zimbra-spell [Y] y

Install zimbra-memcached [N] y

Install zimbra-proxy [N] y


Looking for valid license to install...failed to install license.
*******ERROR
Failed to install a license - this will prevent your server from functioning properly
Please contact Zimbra to obtain a license
Press RETURN to continue

It probably is a permission problem - try this:

chown zimbra:zimbra /opt/zimbra/conf/ZCSLicense.xml
chmod 444 /opt/zimbra/conf/ZCSLicense.xml

If it didn't work then don't panic - just continue with the installation. Zimbra will install anyway and you will be able to upload the license from the administration console.


Zimbra - Installation

This howto is a step-by-step guide on installing the Zimbra Collaboration Suite (ZCS) on Centos 5.4 64bit. Zimbra is an open source collaboration suite. It consists of the following: email, group calendaring, contacts, web document management and authoring. It is fully compatible with clients such as Microsoft Outlook, Apple Mail, and Novell Evolution so that mail, contacts, and calendar items can be synchronised from these to the ZCS server as well as to many mobile devices. But all of those features fully work only with Zimbra Network Edition.

It's usually good idea to virtualize your servers - so we will install Zimbra MTA on a Virtual Machine where the quest operating system is going to be CentOS 5.4 64 bit sitting on top of VmWare ESXi 4.0. This will make your installation fully hardware independent thus allowing for migration to more capable hardware in the feature.

I also decided to explain how to configure your DNS in order for your MTA to be visible from the outside world. I noticed that most authors of the tutorials don't do that. I'm using http://atrianetworks.com/ as my ISP and http://ccjclearline.com/ to manage my domains and DNS records plus BIND as my internal DNS.

Lets assume the following:

INTERNAL IP (The IP of your VM): 10.20.0.26
EXTERNAL IP :226.111.111.111
IP OF YOUR INTERNAL DNS/DHCP SERVER: 10.20.0.10


########################################
CREATE A VM (ESXi 4.0)
########################################

* Create a VM with the following settings:
HD: 100GB thin provisioning.
RAM: 4096
Version: Red Hat Enterprise Linux 5 (64bit)

* Install Centos 5.4 x86_64

########################################
INSTALL VMWARE-TOOLS INSIDE YOUR GUEST OS
########################################

yum install gcc gcc-c++ kernel-devel

---------------------------------------------------------------
NOW RIGHT CLICK YOUR VM >> GUEST >> INSTALL VMWARE TOOLS
---------------------------------------------------------------

mkdir /usr/src/linux

ln -s /usr/src/kernels/2.6.18-164.15.1.el5-x86_64 /usr/src/linux

mkdir /mnt/cdrom

mount /dev/cdrom /mnt/cdrom

cp /mnt/cdrom/VMwareTools-4.0.0-171294.tar.gz /tmp/

umount /mnt/cdrom

cd /tmp

tar -xvf VMwareTools-4.0.0-171294.tar.gz

cd vmware-tools-distrib

./vmware-install.pl

reboot

At his moment people usually jump straight to Zimbra installation which is a mistake. First you need to configure the DNS entries in several places. Otherwise your installation will fail and/or your Zimbra will not be visible from outside the world. As I've already said above - I'm using ccjclearline to manage my domains and DNS antries, the general process though should be fairly similar with any provider.

########################################
DNS CONFIG (EXTERNAL)
########################################
https://ccjclearline.managednsservice.com/
login: yourlogin
pass:yourpass

Find yourdomain.com and hit "zone management" button.

Add A record: HOSTNAME: mail.yourdomain.com IP Address: 226.111.111.111
Add A record: HOSTNAME: yourdomain.com IP Address: 226.111.111.111

Add MX record: HOSTNAME: mail.yourdomain.com Mail Exchange: mail.yourdomain.com. Preference: 10

Hit "Update zone preferences" button.

########################################
FIREWALL Config
########################################

Create NAT rules

Source: 10.20.0.26
Interface: outside
Use IP address:226.111.111.111
Enable Port Address Translation TCP Original Port: smtp Translated Port: smtp

Add similiar rules for: imap4 , http nad and 993

Permit tcp/smtp from 10.20.0.26 to ANY !!!


It might be that you will not need to perform the next step - It highly depends on your particular setup - I had to do it though.
########################################
DNS CONFIG (ISP)
########################################
Send a message to : support@atrianetworks.com with the following body:

Can you please add a reverse DNS entry for 226.111.111.111 to mail.yourdomain.com ?

########################################
DNS CONFIG (INTERNAL - BIND)
########################################

Ssh to 10.20.0.10

vim /etc/bind/yourdomain.com.internal.zone
########################################
$ORIGIN .
$TTL 86400 ; 1 day
yourdomain.com IN SOA ns1.yourdomain.com. admin\@yourdomain.com. (
2010012205 ; serial
10800 ; refresh (3 hours)
15 ; retry (15 seconds)
604800 ; expire (1 week)
10800 ; minimum (3 hours)
)
NS ns1.yourdomain.com.
NS ns2.yourdomain.com.
$ORIGIN yourdomain.com.
IN MX 10 mail.yourdomain.com.
IN A 10.20.0.26
mail IN A 10.20.0.26
########################################

vim /etc/bind/named.conf.local
########################################
zone "yourdomain.com" {
type master;
file "/etc/bind/yourdomain.com.internal.zone";
allow-update { key DHCP_UPDATER; };
allow-query { internal_networks; };
allow-transfer { internal_networks; };
notify yes;
};
########################################

rndc reload & tail -f /var/log/daemon.log

nslookup mail.yourdomain.com

you should get:
...
Name: mail.yourdomain.com
Address: 10.20.0.26

On 10.20.0.26 (your Zimbra VM) do: vim /etc/hosts
########################################
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
10.20.0.26 mail.yourdomain.com mta002
########################################

rndc thaw #on the DNS - 10.20.0.10 rndc thaw. This command allows the changes to be reloaded and the zone to participate in dynamic updates.

tail -f /var/log/daemon.log /var/log/named-auth.info /var/log/update-debug.log |grep 10.20.0.26

########################################
DHCP CONFIG (on 10.20.0.10)
########################################

vim /etc/dhcp3/dhcpd.conf

host mta002 {
hardware ethernet 00:0C:29:C8:C8:C8;
option host-name "mta002";
option domain-name "yourdomain.com";
ddns-hostname "mta002";
ddns-domainname "yourdomain.com";
ddns-rev-domainname "0.20.10.in-addr.arpa";
fixed-address 10.20.0.26;
}

Only after all the above steps you can now move to install Zimbra.

########################################
ZIMBRA INSTALLATION
########################################

chkconfig sendmail off
chkconfig ip6tables off
chkconfig iptables off

yum install libtool-ltdl

yum install gmp

yum install compat-libstdc++-33

yum update

yum install openssh-server

wget http://h.yimg.com/lo/downloads/6.0.4_GA/zcs-6.0.4_GA_2038.RHEL5_64.20091214192925.tgz

tar xvfz zcs-6.0.4_GA_2038.RHEL5_64.20091214192925.tgz

cd /tmp/zcs-6.0.4_GA_2038.RHEL5_64.20091214192925

./install.sh --platform-override

Select the packages to install

Install zimbra-ldap [Y] y

Install zimbra-logger [Y] y

Install zimbra-mta [Y] y

Install zimbra-snmp [Y] y

Install zimbra-store [Y] y

Install zimbra-apache [Y] y

Install zimbra-spell [Y] y

Install zimbra-memcached [N] y

Install zimbra-proxy [N] y

########################################

Main menu

1) Common Configuration:
2) zimbra-ldap: Enabled
3) zimbra-store: Enabled
+Create Admin User: yes
+Admin user to create: admin@mail.yourdomain.com
******* +Admin Password UNSET
+Enable automated spam training: yes
+Spam training user: spam.fd9i2srv_@mail.yourdomain.com
+Non-spam(Ham) training user: ham.m6eg4zfo@mail.yourdomain.com
+Global Documents Account: wiki@mail.yourdomain.com
+SMTP host: mail.yourdomain.com
+Web server HTTP port: 80
+Web server HTTPS port: 443
+Web server mode: http
+IMAP server port: 7143
+IMAP server SSL port: 7993
+POP server port: 7110
+POP server SSL port: 7995
+Use spell check server: yes
+Spell server URL: http://mail.yourdomain.com:7780/aspell.php
+Enable version update checks: TRUE
+Enable version update notifications: TRUE
+Version update notification email: admin@mail.yourdomain.com
+Version update source email: admin@mail.yourdomain.com

4) zimbra-mta: Enabled
5) zimbra-snmp: Enabled
6) zimbra-logger: Enabled
7) zimbra-spell: Enabled
8) zimbra-proxy: Enabled
9) Default Class of Service Configuration:
r) Start servers after configuration yes
s) Save config to file
x) Expand menu
q) Quit

Address unconfigured (**) items (? - help)

########################################
Hit 3 and Enter

Hit 4 and Enter

Hit r

Hit s

https://10.20.0.26:7071/zimbraAdmin/

https://10.20.0.26:7071

Log in to https://10.20.0.26:7071/zimbraAdmin/ and add domain:
Domain name: yourdomain.com
Public service host name: yourdomain.com
Inbound SMTP host name:: mail.yourdomain.com


You are done. Your server should be available from both - internal and external network.

Add some users, run some test and enjoy :)

Friday, April 16, 2010

Zimbra - perfect replacement for MS Exchange

I recently faced the need of adding new MTA server to our corporate cloud.
The company that I'm working for is a specific one - we invent new technologies, then we create a new enterprise around the project, next - the enterprise gets sold.
From the Systems Architect perspective I need to create an entire system for each enterprise that is being developed so it can be easily transfered to a different location. Also I need to think about the costs - my goal always was to build a system which is inexpensive but rock solid and agile.
Now - Microsoft Exchange is anything but inexpensive, a very basic setup would cost at least $30000 and it looks like there is nothing equally good on the market - Or is there ?
I have already tried MDS (Mandriva Directory Server) and I walked out a little disappointed.First there is very little community support around this product. Second - Although the main page of the project advertises that there is 24/7 Support Center, one would find that it's not entirely truth. Yes they have a Call Center - but only in French :) otherwise you need to send an email and wait. Not a very good solution for a production system. So I plunged myself in the madness of research and few hours and three coffee cups later I've found it ! The product is called Zimbra Collaboration Suite, the community support seems to be all right and the cost of the server is around $2000 if you need to go for the support. Otherwise you will be absolutely fine using the Open Source version of the product. The installation of Zimbra Collaboration Suite is simple, fast and intuitive.The Graphic User Interface is so similar to that of MS Outlook that our users love so much that the transition between both systems should be seamless.
Zimbra stands for it's name, it's not just an email server but truly a Collaboration Suite. It allows for creating and managing meetings in a very similar way that Outlook does. You can cerate e-mail aliases, resources and distribution groups.
The next feature worth mentioning and one the cannot be found in Exchange is that Zimbra has opened the architecture which allows for creation of plugins which they call Zimlets. Zimlets allow for integration of various components of the Zimbra system within the mailbox as well as integration of various features from external third party systems.
This means you don't have to live the e-mail window in order to make phone call - just highlight the telephone number inside your e-mail message and use your favorite PC Softphone to make a call. Yahoo Maps zimlet allows for quickly finding an address - just hover your mouse over the address inside your email for the map window to appear.
Zimbra Collaboration Suite looks like a really great replacement for MS Exchange.


Followers