Logo

  • Register
  • | Login
  • Facebook
  • Twitter
  • Gplus
  • Linkedin

Our forum Area

Page 1 of 2 12 LastLast
Results 1 to 15 of 16

This is a discussion on Complete guide to backups in the Dedicated Servers forum
This guide explains how to setup backups on VPS and Dedicated servers. STEP 0 - Requirements - One or more Dedicated or VPS servers - ...

  1. #1
    Loyal Client thisisit3's Avatar
    Join Date
    Mar 2007
    Posts
    725

    Lightbulb Complete guide to backups

    This guide explains how to setup backups on VPS and Dedicated servers.

    STEP 0 - Requirements

    - One or more Dedicated or VPS servers
    - A backup system (not required to be online all the time)
    - Dedicated (static) IP addresses for all the above, the backup system
    may use a dynamic IP address if there is a dynamic DNS setup.
    - *nix operating system on the server
    - *nix or windoze on the backup system
    - Bacula Network Backup Tool
    - MySQL on the backup system


    STEP 1 - Planning stage

    I have made the decision to use Bacula, its one of the best open source backup tools, its also enterprise-level software (more than matches the best commercial backup solutions) which also means that its VERY complex.

    You have to plan a few things before you begin, otherwise you'll end up in a dead end later on. Bacula is very complex and tries to accomodate for the most advanced systems/networks, so it doesn't come as one piece of software. Bacula comes in 5 major components: Director, Console, File, Storage and Monitor components.

    Depending on the number of Dedicated and VPS servers you plan on backing up, those 5 components will multiply. In this guide, i'll assume the following setup:

    - 1 Dedicated or VPS server hosted by Jag (we'll call this system: JAGSERVER)
    - It runs on a dedicated (static) IP address
    - It has a fully qualified domain name like: server.domain.com. For example, if you have a VPS by Jag then you are given "vps.domain.com" to resolve to your main IP address that you also use to for virtual hosted domains.

    - 1 backup system running at your office/home (we'll call this system: BACKUPSERVER)
    - With an internet connection (doesnt matter if its always on)
    - With a static IP address or a static domain name. If you are on a dynamic IP address, you can get a free dynamic domain from dyndns.org. Otherwise don't even bother to continue with this guide, it just won't work without it.
    - Running MySQL (not required to be accessible from the internet, just localy)
    - Enough disk space to store your backups (you may burn them to DVD later on)

    The backup system could be your own desktop or a dedicated backup computer, you can even use an old laptop if you are too desperate.

    The idea behind the above setup is this:

    You run the Console component (a command line client, but you can replace it with graphical or even web clients) on the BACKUPSERVER and via the Console you give the command to backup your JAGSERVER, that command goes to the Director.

    The Director, which also runs on the BACKUPSERVER, will process the command and will assign the required storage space from the Storage component also running on the BACKUPSERVER.

    Then the Director will send a command to the File component (which runs on the JAGSERVER) and request the start of the backup procedure.

    The File component on the JAGSERVER will initiate a connection to the Storage component and will start reading and transmitting file data directly to the Storage component. The Director component will just wait for the process to finish.

    The following diagram shows the general idea of the backup procedure:




    STEP 2 - Download files

    Download the required software.

    Go to the Bacula download files page. Based on our plan from STEP 1, you need:

    - "bacula-client" package for the JAGSERVER, since that system will only act as a dumb client for reading/writting files.
    - "bacula-mysql" if you use a *nix system on the BACKUPSERVER
    - "winbacula" if you use windoze on the BACKUPSERVER

    For *nix systems, use the available RPM packages, much easier than compiling. For example, if you have for JAGSERVER:

    CentOS 4.4 on the JAGSERVER then you get: bacula-client-2.0.3-1.el4.i386.rpm
    Fedora Core 6 (32bit) on the JAGSERVER then you get: bacula-client-2.0.3-1.fc6.i386.rpm

    and respectively for BACKUPSERVER:

    Fedora Core 6 (32bit) on the BACKUPSERVER then you get: bacula-mysql-2.0.3-1.fc6.i386.rpm, bacula-mtx-2.0.3-1.fc6.i386.rpm
    Windoze XP on the BACKUPSERVER then you get: winbacula-2.0.3.exe

    Note, that only versions must match between bacula components running on different operating systems. Also remember that "bacula-mysql" has an RPM dependency for "bacula-mtx".

    Once you've downloaded the above, install them on their respective system, a simple "rpm -ivh <file>" should be enough. In my case, the RPM return several errors that the RPM couldn't create the MySQL database. I had to erase the RPM (rpm -e <files>), create the /root/.my.cnf file with the following options:

    Code:
    [client]
    user="root"
    pass="password_to_mysql"
    and re-run the RPM installation (rpm -ivh <files>). Its possible that your system doesn't have a password set for root (it should!), thus you won't have any errors if you try to install the Bacula rpms the first time.

    I am not aware how Bacula installs under Windoze, you are on your own there. Please read The Windows Version of Bacula documentation.


    STEP 3 - Setup of the server

    First, we'll setup the JAGSERVER, which is the easiest configuration. Once you've installed bacula-client, you'll end up with the directory /etc/bacula/ containing all the configuration files.

    Edit the file /etc/bacula/bacula-fd.conf

    Code:
    Director {
      Name = BACKUPSERVER-dir
      Password = "mypassword"
    }
    
    Director {
      Name = BACKUPSERVER-mon
      Password = "mypassword"
      Monitor = yes
    }
    
    FileDaemon {
      Name = JAGSERVER-fd
      FDport = 9102
      WorkingDirectory = /var/bacula
      Pid Directory = /var/run
      Maximum Concurrent Jobs = 20
    }
    
    Messages {
      Name = Standard
      director = BACKUPSERVER-dir = all, !skipped, !restored
    }
    Delete the default settings (keep a backup just in case) and copy/paste the above. You may also string search&replace the "JAGSERVER" and "BACKUPSERVER" with your own strings. Just remember to do this for all the configuration files that we'll mention later on. Thats it, save the file.


    STEP 4 - Setup of the backup system

    Here is the hard part, the Director and Storage configurations. Remember that your BACKUPSERVER also has a File component, but its not used, so we won't edit the "bacula-fd.conf" configuration file.

    Edit the file /etc/bacula/bacula-dir.conf

    Code:
    Director {
      Name = BACKUPSERVER-dir
      DIRport = 9101
      QueryFile = "/etc/bacula/query.sql"
      WorkingDirectory = "/var/bacula"
      PidDirectory = "/var/run"
      Maximum Concurrent Jobs = 1
      Password = "mypassword"
      Messages = Daemon
    }
    
    Job {
      Name = "JAGSERVERBackup"
      Type = Backup
      Level = Differential
      Client = JAGSERVER-fd
      FileSet = "Full Set"
      #Schedule = "DailyCycle"
      Storage = File
      Messages = Standard
      Pool = Default
      Priority = 10
      Write Bootstrap = "/var/bacula/BACKUPSERVERBackup.bsr"
    }
    
    Job {
      Name = "JAGSERVERRestore"
      Type = Restore
      Client = JAGSERVER-fd
      FileSet = "Full Set"
      Storage = File
      Pool = Default
      Messages = Standard
      Where = /
    }
    
    FileSet {
      Name = "Full Set"
      Include {
        Options {
          signature = SHA1
          compression = GZIP
        }
        File = /home
        #FILE = /
      }
      #Exclude {
      #  File = /dev
      #  File = /proc
      #  File = /tmp
      #  File = /var/tmp
      #  File = /var/log
      #  File = /etc/mtab
      #  File = /.autofsck
      #}
    }
    
    Schedule {
      Name = "DailyCycle"
      Run = Full 1st sun at 23:05
      Run = Differential mon-sat at 23:05
    }
    
    Client {
      Name = JAGSERVER-fd
      Address = server.domain.com
      FDPort = 9102
      Catalog = MySQLCatalog
      Password = "mypassword"
      File Retention = 30 days
      Job Retention = 6 months
      AutoPrune = yes
    }
    
    Storage {
      Name = File
      Address = backup.domain.com
      SDPort = 9103
      Password = "mypassword"
      Device = FileStorage
      Media Type = File
    }
    
    Catalog {
      Name = MySQLCatalog
      dbname = bacula; user = bacula; password = ""
    }
    
    Messages {
      Name = Standard
      console = all, !skipped, !saved
    }
    
    Messages {
      Name = Daemon
      console = all, !skipped, !saved
    }
    
    Pool {
      Name = Default
      Pool Type = Backup
      Recycle = yes
      AutoPrune = yes
      Volume Retention = 365 days
      Maximum Volumes = 1
      Maximum Volume Bytes = 30g
    }
    
    Console {
      Name = BACKUPSERVER-mon
      Password = "mypassword"
      CommandACL = status, .status
    }
    Edit the file /etc/bacula/bacula-sd.conf

    Code:
    Storage {
      Name = BACKUPSERVER-sd
      SDPort = 9103
      WorkingDirectory = "/var/bacula"
      Pid Directory = "/var/run"
      Maximum Concurrent Jobs = 20
    }
    
    Director {
      Name = BACKUPSERVER-dir
      Password = "mypassword"
    }
    
    Director {
      Name = BACKUPSERVER-mon
      Password = "mypassword"
      Monitor = yes
    }
    
    Device {
      Name = FileStorage
      Device Type = File
      Media Type = File
      Archive Device = /tmp
      LabelMedia = yes;
      Random Access = Yes;
      AutomaticMount = yes;
      RemovableMedia = no;
      AlwaysOpen = no;
    }
    
    Messages {
      Name = Standard
      director = BACKUPSERVER-dir = all
    }
    First, string search&replace JAGSERVER and BACKUPSERVER as done in the File component (bacula-fd.conf), then do the same for the "server.domain.com" (its the address of your hosted server here at Jag) and for "backup.domain.com" which should be replaced either with the dedicated (static) IP address of your backup system or with a dynamic DNS address from a service such as DynDNS.org.

    The above configurations imply that you are going to backup /home. You may do some tests by replacing the line "File = /home" with something like "File = /etc/bacula" to backup the bacula configuration files for a testing period and once you've made sure that everything is working fine then you may set it to "File = /" which will backup the ENTIRE SERVER.

    Its also important to remember that the "Where = /" tells Bacula to restore files to their original location. If you don't want that then you may set it to something else, like "Where = /tmp" and restored files will go under that prefix (for example: /tmp/usr/local/etc/file.txt instead of /usr/local/etc/file.txt).

    You will notice that the option Schedule (Schedule = "DailyCycle") is commented out, thats because its a good idea to test your backup setup first before running it manualy. Once you are confident with the entire system then you may unset the Schedule option. You may modify the schedule from the "run" options: Run = Full 1st sun at 23:05 and Run = Differential mon-sat at 23:05 by default.

    Finally, the option "Maximum Volume Bytes = 30g" tells Bacula to store 30 gigs of backup data. You should change that number based on the amount of data you plan to backup, i believe it should be set to twice the amount of that data. For example, if you want to backup 30GB then set it to 60GB. Ofcourse the backed up data will be compressed and will use much less storage, but this way we allow for multiple "versions" of our data to exist.


    STEP 5 - Setup of the Console component

    Now that we have our main Bacula configuration set, we need to modify our Console configuration. This also depends on which console you are using, by default Bacula comes with "bconsole" which is a simple shell console. Others include gnome-console (graphical version of bconsole), wxconsole (a WX-Windows widget, much more advanced than the previous consoles).

    Edit the file /etc/bacula/bconsole.conf

    Code:
    Director {
      Name = BACKUPSERVER-dir
      DIRport = 9101
      address = localhost
      Password = "mypassword"
    }
    Again remember to replace the nessesary strings.


    STEP 6 - Start your engines

    First, start the File component on the JAGSERVER with:

    Code:
    service bacula-fd start
    Second, start the Storage and Director components on the BACKUPSERVER with:

    Code:
    service bacula-sd start
    service bacula-dir start
    I've put the Director last because thats the most prone to errors, the other components shouldn't give you any trouble. In case of errors, go back to the line thats reported to give the error and see whats wrong, it could be a spelling mistake or a configuration error, a common mistake is to write a string like BACKUPSERVER instead of a domain name backup.domain.com. Another common mistake is an unrecognized password, since each "connection" can have its own password and by default the RPM packages initialize the default configuration files with different random passwords.

    Now you may also start the bconsole (or whichever console you have chosen to use), for bconsole just type "/etc/bacula/bconsole" as root and you should see something like this (the * is the command prompt):

    Code:
    Connecting to Director localhost:9101
    1000 OK: BACKUPSERVER-dir Version: 2.0.3 (06 March 2007)
    Enter a period to cancel a command.
    *
    Now type the command "status" and select 1, then retype "status" and redo the procedure in order to test all components. If at any point you get an error, then you probably have a connection/authentication problem with your configuration files.


    STEP 7 - Backup & Restore

    I'm going to describe the bare metal version of doing a backup and a restore. Please read the complete documentation on Running a Job (doing a backup) and Restoring Your Files.

    The following steps execute the default backup procedure (commands are after the * prompt, its what you are supposed to type):

    Code:
    *autodisplay on
    
    (we run the above autodisplay command in order to get automatic responces from the components)
    
    *run
    
    Using default Catalog name=MySQLCatalog DB=bacula
    A job name must be specified.
    The defined Job resources are:
         1: JAGSERVERBackup
         2: JAGSERVERRestore
    Select Job resource (1-2):
    
    *1
    
    Run Backup job
    JobName:  JAGSERVERBackup
    FileSet:  Full Set
    Level:    Differential
    Client:   JAGSERVERBackup-fd
    Storage:  File
    Pool:     Default
    When:     2007-04-28 14:18:57
    OK to run? (yes/mod/no):
    
    *yes
    
    Please use the "label"  command to create a new Volume for:
        Storage:      FileStorage
        Media type:   File
        Pool:         Default
    
    *label
    
    The defined Storage resources are:
         1: File
    Item 1 selected automatically.
    Enter new Volume name:
    
    *ThisIsMyBackup
    
    (lots of messages here...)
    The above should end in something that contains the string "Termination: Backup OK". In order to restore your files, just type "restore" at the asterisk prompt and follow the questions.


    ChangeLog

    - first version of the document, requires some refinement and explanations.


    TODO

    - TLS encryption and authentication

  2. #2

  3. #3
    Loyal Client
    Join Date
    Apr 2006
    Location
    Sweden
    Posts
    262
    Did you get compression to work? All i get is Software Compression None when i've done a backup and its so damn annoying! I do have the Compression = gzip line, still refuse to work (yes bacula was restarted)
    My VPS server:
    www.myfedoraserver.com


  4. #4
    Loyal Client thisisit3's Avatar
    Join Date
    Mar 2007
    Posts
    725
    Mine is not compressed either, but i'm not using gzip compression at all.

    It sounds like your Bacula binary wasn't compiled with zlib support:

    http://www.bacula.org/rel-manual/Bac...00000000000000

  5. #5
    Loyal Client
    Join Date
    Apr 2006
    Location
    Sweden
    Posts
    262
    The problem is that it was lol
    My VPS server:
    www.myfedoraserver.com


  6. #6
    Nearly 100% Pure Carbon thecoalman's Avatar
    Join Date
    Nov 2007
    Location
    Northeast Pennsylvania
    Posts
    673
    Thank you very much and having read some of your other posts I'd like to thank you for the future ones I'll read. I'm a newbie to running a server this will undoubtedly save me some grief at some point in the future. One thing i will note as I have some pretty good experience with DVD video:

    you may burn them to DVD later on)
    DVD media quality can vary greatly, some of the cheaper brands can have very large failure rates, 50% or even higher is not uncommon for the real cheap brands. Even the best brands have some amount of failure, maybe 1 in 1000. Personally I would only use DVD as a backup medium for the second backup or for archiving where reliability isn't that important.

    There's an excellent guide to media quality here if anyone is interested:

    http://www.digitalfaq.com/media/

  7. #7
    I didn't do it! Daniel_DBS's Avatar
    Join Date
    Aug 2007
    Location
    Mars
    Posts
    1,201
    thanks thisisit... this guide is great... i just finished setting up my backup system with bacula... excellent
    -Daniel

    If the automobile had followed the same development cycle as the computer, a Rolls-Royce today would cost $100, get a million miles to the gallon, and explode once every few weeks.

    My scripts never have bugs. They just develop random features.

  8. #8
    Loyal Client thisisit3's Avatar
    Join Date
    Mar 2007
    Posts
    725
    I'm grand you found my guide helpful. I hope i'll find some free time to update it and add a few extra sections...

  9. #9
    Loyal Client
    Join Date
    Aug 2006
    Posts
    61
    I will be trying this myself, now this is backing the whole vps?

  10. #10
    Loyal Client thisisit3's Avatar
    Join Date
    Mar 2007
    Posts
    725
    The example configuration will backup the entire /home tree. Look at the bacula-dir.conf, under the "FileSet {" section you'll see:

    File = /home
    #FILE = /

    If you want to backup the entire system, just comment the first line and uncomment the second:

    #File = /home
    FILE = /

    I strongly recommend to add a lot of exclude lines, so that you don't backup stuff that shouldn't really be backed up, like /tmp, /var/tmp, /var/log, /var/run and others.

  11. #11
    Yeah, I know a LOT! Vin DSL's Avatar
    Join Date
    Mar 2003
    Location
    Arizona Uplands
    Posts
    10,665
    Quote Originally Posted by thisisit3 View Post
    I strongly recommend to add a lot of exclude lines, so that you don't backup stuff that shouldn't really be backed up, like /tmp, /var/tmp, /var/log, /var/run and others.
    Do you guys have a 'www' symlink? If so, I strongly recommend excluding that...

    Otherwise, you'll backup your web space twice...
    DISCLAIMER Any resemblance between the views expressed above and those of the owners and operators of this system is purely coincidental. Any resemblance between these views and my own are non-deterministic. The existence of Vin DSL is questionable. The existence of views in the absence of anyone to hold them is problematic. The existence of the reader is left as an exercise in the second-order coefficient.

    No Guts, No Story! VinDSL 2010

  12. #12
    I didn't do it! Daniel_DBS's Avatar
    Join Date
    Aug 2007
    Location
    Mars
    Posts
    1,201
    Quote Originally Posted by Vin DSL View Post
    Do you guys have a 'www' symlink? If so, I strongly recommend excluding that...

    Otherwise, you'll backup your web space twice...
    haha good point... i best catch that before the backup thing runs tonight... thanks vin
    -Daniel

    If the automobile had followed the same development cycle as the computer, a Rolls-Royce today would cost $100, get a million miles to the gallon, and explode once every few weeks.

    My scripts never have bugs. They just develop random features.

  13. #13
    Loyal Client thisisit3's Avatar
    Join Date
    Mar 2007
    Posts
    725
    What Vin doesn't know is that Bacula is a lot more clever than that. Bacula understands all the different types of file system entities, including symbolic links, hard links, pipes, sockets, and others.

    In other words, the "www" symbolic links will NOT backup the same files twice, Bacula will instead backup the link as a link and restore it as such.

  14. #14
    I didn't do it! Daniel_DBS's Avatar
    Join Date
    Aug 2007
    Location
    Mars
    Posts
    1,201
    Quote Originally Posted by thisisit3 View Post
    What Vin doesn't know is that Bacula is a lot more clever than that. Bacula understands all the different types of file system entities, including symbolic links, hard links, pipes, sockets, and others.

    In other words, the "www" symbolic links will NOT backup the same files twice, Bacula will instead backup the link as a link and restore it as such.
    hmm i did not know that... thanks
    -Daniel

    If the automobile had followed the same development cycle as the computer, a Rolls-Royce today would cost $100, get a million miles to the gallon, and explode once every few weeks.

    My scripts never have bugs. They just develop random features.

  15. #15
    Loyal Client
    Join Date
    Aug 2006
    Posts
    61
    Quote Originally Posted by thisisit3 View Post
    The example configuration will backup the entire /home tree. Look at the bacula-dir.conf, under the "FileSet {" section you'll see:

    File = /home
    #FILE = /

    If you want to backup the entire system, just comment the first line and uncomment the second:

    #File = /home
    FILE = /

    I strongly recommend to add a lot of exclude lines, so that you don't backup stuff that shouldn't really be backed up, like /tmp, /var/tmp, /var/log, /var/run and others.
    Are the SQL databases located in the home directory, if that is the case then all I need to back up is the home.

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •