Complete guide to backups

**thisisit3** · 04-30-2007, 01:27 AM

This guide explains how to setup backups on VPS and Dedicated servers.

STEP 0 - Requirements

- One or more Dedicated or VPS servers
- A backup system (not required to be online all the time)
- Dedicated (static) IP addresses for all the above, the backup system
may use a dynamic IP address if there is a dynamic DNS setup.
- *nix operating system on the server
- *nix or windoze on the backup system
- Bacula Network Backup Tool
- MySQL on the backup system

STEP 1 - Planning stage

I have made the decision to use Bacula, its one of the best open source backup tools, its also enterprise-level software (more than matches the best commercial backup solutions) which also means that its VERY complex.

You have to plan a few things before you begin, otherwise you'll end up in a dead end later on. Bacula is very complex and tries to accomodate for the most advanced systems/networks, so it doesn't come as one piece of software. Bacula comes in 5 major components: Director, Console, File, Storage and Monitor components.

Depending on the number of Dedicated and VPS servers you plan on backing up, those 5 components will multiply. In this guide, i'll assume the following setup:

- 1 Dedicated or VPS server hosted by Jag (we'll call this system: JAGSERVER)
- It runs on a dedicated (static) IP address
- It has a fully qualified domain name like: server.domain.com. For example, if you have a VPS by Jag then you are given "vps.domain.com" to resolve to your main IP address that you also use to for virtual hosted domains.

- 1 backup system running at your office/home (we'll call this system: BACKUPSERVER)
- With an internet connection (doesnt matter if its always on)
- With a static IP address or a static domain name. If you are on a dynamic IP address, you can get a free dynamic domain from dyndns.org. Otherwise don't even bother to continue with this guide, it just won't work without it.
- Running MySQL (not required to be accessible from the internet, just localy)
- Enough disk space to store your backups (you may burn them to DVD later on)

The backup system could be your own desktop or a dedicated backup computer, you can even use an old laptop if you are too desperate.

The idea behind the above setup is this:

You run the Console component (a command line client, but you can replace it with graphical or even web clients) on the BACKUPSERVER and via the Console you give the command to backup your JAGSERVER, that command goes to the Director.

The Director, which also runs on the BACKUPSERVER, will process the command and will assign the required storage space from the Storage component also running on the BACKUPSERVER.

Then the Director will send a command to the File component (which runs on the JAGSERVER) and request the start of the backup procedure.

The File component on the JAGSERVER will initiate a connection to the Storage component and will start reading and transmitting file data directly to the Storage component. The Director component will just wait for the process to finish.

The following diagram shows the general idea of the backup procedure:

STEP 2 - Download files

Download the required software.

Go to the Bacula download files page. Based on our plan from STEP 1, you need:

- "bacula-client" package for the JAGSERVER, since that system will only act as a dumb client for reading/writting files.
- "bacula-mysql" if you use a *nix system on the BACKUPSERVER
- "winbacula" if you use windoze on the BACKUPSERVER

For *nix systems, use the available RPM packages, much easier than compiling. For example, if you have for JAGSERVER:

CentOS 4.4 on the JAGSERVER then you get: bacula-client-2.0.3-1.el4.i386.rpm
Fedora Core 6 (32bit) on the JAGSERVER then you get: bacula-client-2.0.3-1.fc6.i386.rpm

and respectively for BACKUPSERVER:

Fedora Core 6 (32bit) on the BACKUPSERVER then you get: bacula-mysql-2.0.3-1.fc6.i386.rpm, bacula-mtx-2.0.3-1.fc6.i386.rpm
Windoze XP on the BACKUPSERVER then you get: winbacula-2.0.3.exe

Note, that only versions must match between bacula components running on different operating systems. Also remember that "bacula-mysql" has an RPM dependency for "bacula-mtx".

Once you've downloaded the above, install them on their respective system, a simple "rpm -ivh <file>" should be enough. In my case, the RPM return several errors that the RPM couldn't create the MySQL database. I had to erase the RPM (rpm -e <files>), create the /root/.my.cnf file with the following options:

Code:

[client]
user="root"
pass="password_to_mysql"

and re-run the RPM installation (rpm -ivh <files>). Its possible that your system doesn't have a password set for root (it should!), thus you won't have any errors if you try to install the Bacula rpms the first time.

I am not aware how Bacula installs under Windoze, you are on your own there. Please read The Windows Version of Bacula documentation.

STEP 3 - Setup of the server

First, we'll setup the JAGSERVER, which is the easiest configuration. Once you've installed bacula-client, you'll end up with the directory /etc/bacula/ containing all the configuration files.

Edit the file /etc/bacula/bacula-fd.conf

Code:

Director {
  Name = BACKUPSERVER-dir
  Password = "mypassword"
}

Director {
  Name = BACKUPSERVER-mon
  Password = "mypassword"
  Monitor = yes
}

FileDaemon {
  Name = JAGSERVER-fd
  FDport = 9102
  WorkingDirectory = /var/bacula
  Pid Directory = /var/run
  Maximum Concurrent Jobs = 20
}

Messages {
  Name = Standard
  director = BACKUPSERVER-dir = all, !skipped, !restored
}

Delete the default settings (keep a backup just in case) and copy/paste the above. You may also string search&replace the "JAGSERVER" and "BACKUPSERVER" with your own strings. Just remember to do this for all the configuration files that we'll mention later on. Thats it, save the file.

STEP 4 - Setup of the backup system

Here is the hard part, the Director and Storage configurations. Remember that your BACKUPSERVER also has a File component, but its not used, so we won't edit the "bacula-fd.conf" configuration file.

Edit the file /etc/bacula/bacula-dir.conf

Code:

Director {
  Name = BACKUPSERVER-dir
  DIRport = 9101
  QueryFile = "/etc/bacula/query.sql"
  WorkingDirectory = "/var/bacula"
  PidDirectory = "/var/run"
  Maximum Concurrent Jobs = 1
  Password = "mypassword"
  Messages = Daemon
}

Job {
  Name = "JAGSERVERBackup"
  Type = Backup
  Level = Differential
  Client = JAGSERVER-fd
  FileSet = "Full Set"
  #Schedule = "DailyCycle"
  Storage = File
  Messages = Standard
  Pool = Default
  Priority = 10
  Write Bootstrap = "/var/bacula/BACKUPSERVERBackup.bsr"
}

Job {
  Name = "JAGSERVERRestore"
  Type = Restore
  Client = JAGSERVER-fd
  FileSet = "Full Set"
  Storage = File
  Pool = Default
  Messages = Standard
  Where = /
}

FileSet {
  Name = "Full Set"
  Include {
    Options {
      signature = SHA1
      compression = GZIP
    }
    File = /home
    #FILE = /
  }
  #Exclude {
  #  File = /dev
  #  File = /proc
  #  File = /tmp
  #  File = /var/tmp
  #  File = /var/log
  #  File = /etc/mtab
  #  File = /.autofsck
  #}
}

Schedule {
  Name = "DailyCycle"
  Run = Full 1st sun at 23:05
  Run = Differential mon-sat at 23:05
}

Client {
  Name = JAGSERVER-fd
  Address = server.domain.com
  FDPort = 9102
  Catalog = MySQLCatalog
  Password = "mypassword"
  File Retention = 30 days
  Job Retention = 6 months
  AutoPrune = yes
}

Storage {
  Name = File
  Address = backup.domain.com
  SDPort = 9103
  Password = "mypassword"
  Device = FileStorage
  Media Type = File
}

Catalog {
  Name = MySQLCatalog
  dbname = bacula; user = bacula; password = ""
}

Messages {
  Name = Standard
  console = all, !skipped, !saved
}

Messages {
  Name = Daemon
  console = all, !skipped, !saved
}

Pool {
  Name = Default
  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 365 days
  Maximum Volumes = 1
  Maximum Volume Bytes = 30g
}

Console {
  Name = BACKUPSERVER-mon
  Password = "mypassword"
  CommandACL = status, .status
}

Edit the file /etc/bacula/bacula-sd.conf

Code:

Storage {
  Name = BACKUPSERVER-sd
  SDPort = 9103
  WorkingDirectory = "/var/bacula"
  Pid Directory = "/var/run"
  Maximum Concurrent Jobs = 20
}

Director {
  Name = BACKUPSERVER-dir
  Password = "mypassword"
}

Director {
  Name = BACKUPSERVER-mon
  Password = "mypassword"
  Monitor = yes
}

Device {
  Name = FileStorage
  Device Type = File
  Media Type = File
  Archive Device = /tmp
  LabelMedia = yes;
  Random Access = Yes;
  AutomaticMount = yes;
  RemovableMedia = no;
  AlwaysOpen = no;
}

Messages {
  Name = Standard
  director = BACKUPSERVER-dir = all
}

First, string search&replace JAGSERVER and BACKUPSERVER as done in the File component (bacula-fd.conf), then do the same for the "server.domain.com" (its the address of your hosted server here at Jag) and for "backup.domain.com" which should be replaced either with the dedicated (static) IP address of your backup system or with a dynamic DNS address from a service such as DynDNS.org.

The above configurations imply that you are going to backup /home. You may do some tests by replacing the line "File = /home" with something like "File = /etc/bacula" to backup the bacula configuration files for a testing period and once you've made sure that everything is working fine then you may set it to "File = /" which will backup the ENTIRE SERVER.

Its also important to remember that the "Where = /" tells Bacula to restore files to their original location. If you don't want that then you may set it to something else, like "Where = /tmp" and restored files will go under that prefix (for example: /tmp/usr/local/etc/file.txt instead of /usr/local/etc/file.txt).

You will notice that the option Schedule (Schedule = "DailyCycle") is commented out, thats because its a good idea to test your backup setup first before running it manualy. Once you are confident with the entire system then you may unset the Schedule option. You may modify the schedule from the "run" options: Run = Full 1st sun at 23:05 and Run = Differential mon-sat at 23:05 by default.

Finally, the option "Maximum Volume Bytes = 30g" tells Bacula to store 30 gigs of backup data. You should change that number based on the amount of data you plan to backup, i believe it should be set to twice the amount of that data. For example, if you want to backup 30GB then set it to 60GB. Ofcourse the backed up data will be compressed and will use much less storage, but this way we allow for multiple "versions" of our data to exist.

STEP 5 - Setup of the Console component

Now that we have our main Bacula configuration set, we need to modify our Console configuration. This also depends on which console you are using, by default Bacula comes with "bconsole" which is a simple shell console. Others include gnome-console (graphical version of bconsole), wxconsole (a WX-Windows widget, much more advanced than the previous consoles).

Edit the file /etc/bacula/bconsole.conf

Code:

Director {
  Name = BACKUPSERVER-dir
  DIRport = 9101
  address = localhost
  Password = "mypassword"
}

Again remember to replace the nessesary strings.

STEP 6 - Start your engines

First, start the File component on the JAGSERVER with:

Code:

service bacula-fd start

Second, start the Storage and Director components on the BACKUPSERVER with:

Code:

service bacula-sd start
service bacula-dir start

I've put the Director last because thats the most prone to errors, the other components shouldn't give you any trouble. In case of errors, go back to the line thats reported to give the error and see whats wrong, it could be a spelling mistake or a configuration error, a common mistake is to write a string like BACKUPSERVER instead of a domain name backup.domain.com. Another common mistake is an unrecognized password, since each "connection" can have its own password and by default the RPM packages initialize the default configuration files with different random passwords.

Now you may also start the bconsole (or whichever console you have chosen to use), for bconsole just type "/etc/bacula/bconsole" as root and you should see something like this (the * is the command prompt):

Code:

Connecting to Director localhost:9101
1000 OK: BACKUPSERVER-dir Version: 2.0.3 (06 March 2007)
Enter a period to cancel a command.
*

Now type the command "status" and select 1, then retype "status" and redo the procedure in order to test all components. If at any point you get an error, then you probably have a connection/authentication problem with your configuration files.

STEP 7 - Backup & Restore

I'm going to describe the bare metal version of doing a backup and a restore. Please read the complete documentation on Running a Job (doing a backup) and Restoring Your Files.

The following steps execute the default backup procedure (commands are after the * prompt, its what you are supposed to type):

Code:

*autodisplay on

(we run the above autodisplay command in order to get automatic responces from the components)

*run

Using default Catalog name=MySQLCatalog DB=bacula
A job name must be specified.
The defined Job resources are:
     1: JAGSERVERBackup
     2: JAGSERVERRestore
Select Job resource (1-2):

*1

Run Backup job
JobName:  JAGSERVERBackup
FileSet:  Full Set
Level:    Differential
Client:   JAGSERVERBackup-fd
Storage:  File
Pool:     Default
When:     2007-04-28 14:18:57
OK to run? (yes/mod/no):

*yes

Please use the "label"  command to create a new Volume for:
    Storage:      FileStorage
    Media type:   File
    Pool:         Default

*label

The defined Storage resources are:
     1: File
Item 1 selected automatically.
Enter new Volume name:

*ThisIsMyBackup

(lots of messages here...)

The above should end in something that contains the string "Termination: Backup OK". In order to restore your files, just type "restore" at the asterisk prompt and follow the questions.

ChangeLog

- first version of the document, requires some refinement and explanations.

TODO

- TLS encryption and authentication

**thisisit3** · 04-30-2007, 01:31 AM

damn i'm good

**AlexKall** · 04-30-2007, 04:17 AM

Did you get compression to work? All i get is Software Compression None when i've done a backup and its so damn annoying! I do have the Compression = gzip line, still refuse to work (yes bacula was restarted)

**thisisit3** · 04-30-2007, 06:30 AM

Mine is not compressed either, but i'm not using gzip compression at all.

It sounds like your Bacula binary wasn't compiled with zlib support:

http://www.bacula.org/rel-manual/Bac...00000000000000

**AlexKall** · 04-30-2007, 06:48 AM

The problem is that it was lol

**thecoalman** · 11-11-2007, 02:44 AM

Thank you very much and having read some of your other posts I'd like to thank you for the future ones I'll read. I'm a newbie to running a server this will undoubtedly save me some grief at some point in the future. One thing i will note as I have some pretty good experience with DVD video:

you may burn them to DVD later on)

DVD media quality can vary greatly, some of the cheaper brands can have very large failure rates, 50% or even higher is not uncommon for the real cheap brands. Even the best brands have some amount of failure, maybe 1 in 1000. Personally I would only use DVD as a backup medium for the second backup or for archiving where reliability isn't that important.

There's an excellent guide to media quality here if anyone is interested:

http://www.digitalfaq.com/media/

**Daniel_DBS** · 11-11-2007, 06:31 AM

thanks thisisit... this guide is great... i just finished setting up my backup system with bacula... excellent

**thisisit3** · 11-11-2007, 12:42 PM

I'm grand you found my guide helpful. I hope i'll find some free time to update it and add a few extra sections...

**Redemption** · 11-12-2007, 08:51 AM

I will be trying this myself, now this is backing the whole vps?

**thisisit3** · 11-12-2007, 09:35 AM

The example configuration will backup the entire /home tree. Look at the bacula-dir.conf, under the "FileSet {" section you'll see:

File = /home
#FILE = /

If you want to backup the entire system, just comment the first line and uncomment the second:

#File = /home
FILE = /

I strongly recommend to add a lot of exclude lines, so that you don't backup stuff that shouldn't really be backed up, like /tmp, /var/tmp, /var/log, /var/run and others.

**Vin DSL** · 11-12-2007, 10:52 AM

Originally Posted by thisisit3

I strongly recommend to add a lot of exclude lines, so that you don't backup stuff that shouldn't really be backed up, like /tmp, /var/tmp, /var/log, /var/run and others.

Do you guys have a 'www' symlink? If so, I strongly recommend excluding that...

Otherwise, you'll backup your web space twice...

**Daniel_DBS** · 11-12-2007, 11:02 AM

Originally Posted by Vin DSL

Do you guys have a 'www' symlink? If so, I strongly recommend excluding that...

Otherwise, you'll backup your web space twice...

haha good point... i best catch that before the backup thing runs tonight... thanks vin

**thisisit3** · 11-12-2007, 11:06 AM

What Vin doesn't know is that Bacula is a lot more clever than that. Bacula understands all the different types of file system entities, including symbolic links, hard links, pipes, sockets, and others.

In other words, the "www" symbolic links will NOT backup the same files twice, Bacula will instead backup the link as a link and restore it as such.

**Daniel_DBS** · 11-12-2007, 11:07 AM

Originally Posted by thisisit3

What Vin doesn't know is that Bacula is a lot more clever than that. Bacula understands all the different types of file system entities, including symbolic links, hard links, pipes, sockets, and others.

In other words, the "www" symbolic links will NOT backup the same files twice, Bacula will instead backup the link as a link and restore it as such.

hmm i did not know that... thanks

**Redemption** · 11-17-2007, 08:52 AM

Originally Posted by thisisit3

The example configuration will backup the entire /home tree. Look at the bacula-dir.conf, under the "FileSet {" section you'll see:

File = /home
#FILE = /

If you want to backup the entire system, just comment the first line and uncomment the second:

#File = /home
FILE = /

I strongly recommend to add a lot of exclude lines, so that you don't backup stuff that shouldn't really be backed up, like /tmp, /var/tmp, /var/log, /var/run and others.

Are the SQL databases located in the home directory, if that is the case then all I need to back up is the home.

LinkBack

Thread Tools

Display

Complete guide to backups

Bookmarks

Bookmarks

Posting Permissions