Configuration file has a separate section for each repository. Each section contains
two blocks: information needed to find the repository (host, user, repository path) and
its parent (reference to parent configuration block) and the templates for the three
commands that will be used to process information in the repository:
- Command to list repository contents
- Command to calculate the digest of the file in the repository
- Command to transfer a file from the parent repository
#===================================================================
# FILE: /etc/sshsync/sshsync.conf
# DESCRIPTION: SSH synchronization tooling config
#===================================================================
[repomaster]
user = repoowner
password = ******
host = 192.168.0.1
path = /home/repoowner/apprepo
listcmd = find ${REPOPATH} -type f
digestcmd = if [ -f ${FILEPATH} ]; then sha1sum ${FILEPATH} | cut -d' ' -f1; else echo \'NONE\'; fi
[repoproxy]
user = proxyrunner
password = ******
host = 192.168.1.1
path = /dat/repoproxy
parent = repomaster
listcmd = find ${REPOPATH} -type f
digestcmd = if [ -f ${FILEPATH} ]; then sha1sum ${FILEPATH} | cut -d' ' -f1; else echo \'NONE\'; fi
synccmd = scp -p -i /home/proxyrunner/.ssh/id_dsa ${REMOTEUSER}@${REMOTEHOST}:${REMOTEDIR}/${FILE} ${LOCALDIR}/${FILE}
[appnode1]
user = john
password = ******
host = 192.168.2.1
path = /apprepo/latest
parent = repoproxy
listcmd = find ${REPOPATH} -type f
digestcmd = if [ -f ${FILEPATH} ]; then digest -a sha1 ${FILEPATH}; else echo \'NONE\'; fi
synccmd = scp -p -i /home/john/.ssh/id_dsa ${REMOTEUSER}@${REMOTEHOST}:${REMOTEDIR}/${FILE} ${LOCALDIR}/${FILE}
[appnode2]
user = sheila
password = ******
host = 192.168.2.2
path = /home/sheila/apprepo
parent = repoproxy
listcmd = find ${REPOPATH} -type f
digestcmd = if [ -f ${FILEPATH} ]; then digest -a sha1 ${FILEPATH}; else echo \'NONE\'; fi
synccmd = scp -p -i /home/sheila/.ssh/id_dsa ${REMOTEUSER}@${REMOTEHOST}:${REMOTEDIR}/${FILE} ${LOCALDIR}/${FILE}As you can see I did not use the tools that Python’s ConfigParser module gives me to resolve variables defined
in other parts of the file. I use the variables in a shell-like notation and replace them in the script. In my
opinion this is easier to maintain, as I can update the parent-child relationship between repositories much esier.
Patterns for listcmd, digestcmd and synccmd are bash oneliners, but nothing prevents them to be more
sophisticated, separate scripts. They can be different for each repository - as you can see, I sometimes
use digest command to calculate SHA1 of a file, and sometimes I use sha1sum for the same purpose. I do it because
some hosts do not have sha1sum installed. The important thing is, that these commands return exactly the same strings.
Let me present the control script in chunks.
#!/bin/env python
"""
File repository synchronization with SSH.
Compares digests of files, and transfers them if necessary.
Requires passwordless ssh logins, no output on ssh login, availability of
list, digest and transfer commands on all respective repo servers.
"""
import os
import sys
import getopt
import ConfigParser
import logging, logging.config
import commands
This is a standard shebang to allow calling the script as a command plus docstring and necessary imports.
Modules os and sys are needed for operating system related stuff, getopt for option parsing,
ConfigParser for handling our configuration file, logging for logging support and commands for
calling command line programs and collecting their return code and output.
class UsageException(Exception):
"""
Exception thrown when parameters are wrong.
"""
def __init__(self, msg):
self.msg = msg
class InternalErrorException(Exception):
"""
Exception thrown when internal application error occurs.
"""
def __init__(self, msg):
self.msg = msg
class ProcessingErrorException(Exception):
"""
Exception thrown when application encountered an external error.
"""
def __init__(self, msg):
self.msg = msg
Well, here I deviate a bit from my personal standards. Usually I define one parent exception class and
make all the exceptions that a given application can throw extend it. That way it is easy for any code
importing our module to catch all our exceptions.
In this case I am being lazy, I just define three separate exceptions: one for bad usage, one for
internal bugs, and one for outside conditions that may make the application to fail.
config = None
log = None
cmd = commands.getstatusoutput
scriptconfig = '/etc/sshsync/sshsync.conf'
logconfig = '/etc/sshsync/logging.conf'
pidfile = '/var/run/sshsync.pid'
Globals. Config and log are to be instantiated later in the code, cmd is a mere shortcut, the rest
are the locations of files used by the script. I do not explain logconfig here, as this is plain, standard
configuration file for Python logging.
def print_usage():
"""
Print usage information.
"""
print >>sys.stdout, """\
Usage:
sshsync --option=<value> ...
Options:
-h / --help
Print this help message and exit
-s / --sync <repository>
Synchronize <repository> with it's parent
"""
return 0
This is just a funtion called to print a standard help message. Notice that it returns, not exits.
def lock(lockfile):
""" """
rc = True
if os.path.exists(lockfile):
rc = False
log.debug('Can not create lock file. Lock file already exists: %s' % lockfile)
else:
log.debug('Creating lock file: %s' % lockfile)
try:
f = open(lockfile, 'w')
f.write(str(os.getpid()))
f.close()
log.debug('Lock file created: %s' % lockfile)
except IOError, error:
rc = False
log.error('Can not create lock file. I/O error: %s' % error)
return rc
def unlock(lockfile):
""" """
rc = True
if not os.path.exists(lockfile):
rc = False
log.error('Unlock called, but lock file does not exist: %s' % lockfile)
else:
log.debug('Removing lock file: %s' % lockfile)
try:
os.unlink(lockfile)
log.debug('Lock file deleted: %s' % lockfile)
except IOError, error:
rc = False
log.error('Can not delete lock file. I/O error: %s' % error)
return rc
Our script should have only one, single instance at any given time. I use a PID file to
find out whether there is any other instance running, or not.
def runrmtcmd(host, user, password, rmtcmd):
""" """
status, out = cmd('ssh -l %s %s "%s"' % (user, host, rmtcmd))
if status != 0:
raise ProcessingErrorException('Can not run command "%s" on host "%s": %s' % (rmtcmd, host, out))
return out
A tool to run a command on the remote system. Password is ignored. I keep it just in case if later I
want to be more trendy and use Twisted libraries to properly login with SSH.
This function relies on passwordless logins enabled for SSH. Notice that non-zero return code triggers
throwing of exception.
As you can see, I am pretty lazy with writing docstrings.
def getrmtfiles(repository):
""" """
files = []
log.debug('Getting list of files in the repository: %s' % repository)
repo_host = config.get(repository, 'host')
repo_user = config.get(repository, 'user')
repo_password = config.get(repository, 'password')
repo_path = os.path.normpath(config.get(repository, 'path'))
listcmd = config.get(repository, 'listcmd')
listcmd = listcmd.replace('${REPOPATH}', repo_path)
out = runrmtcmd(repo_host, repo_user, repo_password, listcmd)
prefix = len(repo_path) + 1
lines = out.splitlines()
for line in lines:
f = line[prefix:]
files.append(f)
log.debug('Repository %s has file: %s' % (repository, f))
return files
This is the function that we call to get a list of files in the repository.
Config is a global that must be initialized before we call this function. Again, this is being
lazy on my part, I would not write such a thing in a library class. Yes, I agree, it would be
cleaner to have a Repository object instead of a bunch of functions and global variables...
Config is wise enough to fetch strings from our configuration file. Read ConfigParser documentation
to learn more about it.
I am cutting off the repository path from the file name, because the repositories can be placed in different
places on different machines. I use substring instead of os.path functions, because I do not want to deal
with leading dots in file paths.
def getrmtdigest(repository, fpath):
""" """
log.debug('Getting digest from repository %s for file: %s' % (repository, fpath))
repo_host = config.get(repository, 'host')
repo_user = config.get(repository, 'user')
repo_password = config.get(repository, 'password')
repo_path = os.path.normpath(config.get(repository, 'path'))
digestcmd = config.get(repository, 'digestcmd')
digestcmd = digestcmd.replace('${FILEPATH}', '%s/%s' % (repo_path, fpath))
digest = runrmtcmd(repo_host, repo_user, repo_password, digestcmd)
log.debug('Digest from repository %s for file %s : %s' % (repository, fpath, digest))
return digest
This is the function that gets the digest of a file in a repository. As you can see, I am
repeating myself to fetch the user, password and host from the configuration. If this code were to
be used longer, I would make a repository object and fetch all the configuration only once in __init__
function.
I leave this improvement as an exercise to the visitors of my blog.
def syncrmtfile(repository, fpath):
""" """
repo_host = config.get(repository, 'host')
repo_user = config.get(repository, 'user')
repo_password = config.get(repository, 'password')
repo_path = os.path.normpath(config.get(repository, 'path'))
parent = config.get(repository, 'parent')
parent_host = config.get(parent, 'host')
parent_user = config.get(parent, 'user')
parent_password = config.get(parent, 'password')
parent_path = os.path.normpath(config.get(parent, 'path'))
rmtcmd = config.get(repository, 'synccmd')
rmtcmd = rmtcmd.replace('${FILE}', fpath)
rmtcmd = rmtcmd.replace('${REMOTEHOST}', parent_host)
rmtcmd = rmtcmd.replace('${REMOTEUSER}', parent_user)
rmtcmd = rmtcmd.replace('${REMOTEPASSWORD}', parent_password)
rmtcmd = rmtcmd.replace('${REMOTEDIR}', parent_path)
rmtcmd = rmtcmd.replace('${LOCALHOST}', repo_host)
rmtcmd = rmtcmd.replace('${LOCALUSER}', repo_user)
rmtcmd = rmtcmd.replace('${LOCALPASSWORD}', repo_password)
rmtcmd = rmtcmd.replace('${LOCALDIR}', repo_path)
log.info('Syncing file in repository %s: %s' % (repository, fpath))
cmdout = runrmtcmd(repo_host, repo_user, repo_password, rmtcmd )
for line in cmdout.splitlines():
log.debug("Host %s: %s" % (repo_host, line))
Code used to copy a single file from the parent repository to the current one.
Again, DRY principle violated. Sorry about that.
def syncrepo(repository):
""" """
log.info('Syncing repository: %s' % repository)
log.debug('This script assumes passwordless logins with ssh!')
parent = config.get(repository, 'parent')
parent_files = getrmtfiles(parent)
for f in parent_files:
pdigest = getrmtdigest(parent, f)
rdigest = getrmtdigest(repository, f)
if pdigest == rdigest:
log.debug('File %s in %s and %s have the same digest (%s). Synchronization is not needed.' % (f, parent, repository, rdigest))
else:
syncrmtfile(repository, f)
rdigest = getrmtdigest(repository, f)
if pdigest == rdigest:
log.info('File %s has been successfully copied from %s to %s. File digest is: %s' % (f, parent, repository, rdigest))
else:
log.error('Transfer of file %s from %s to %s failed. Digests mismatch (%s != %s)' % (f, parent, repository, pdigest, rdigest))
A function capable of syncing all the files in a single repository. It gets a list of files and for each file
compares its digest in the current and parent repositories. If the digests do not match, the file is being copied from the parrent.
That is wasting the bandwidth, but at least easy to write.
def checkconfig(repository):
""" """
rc = True
parent = None
if config.has_section(repository):
if not config.has_option(repository, 'host') or not config.get(repository, 'host'):
rc = False
log.error('Missing host for repository %s' % repository)
if not config.has_option(repository, 'user') or not config.get(repository, 'user'):
rc = False
log.error('Missing user for repository %s' % repository)
if not config.has_option(repository, 'password') or not config.get(repository, 'password'):
rc = False
log.error('Missing password for repository %s' % repository)
if not config.has_option(repository, 'path') or not config.get(repository, 'path'):
rc = False
log.error('Missing path for repository %s' % repository)
if not config.has_option(repository, 'digestcmd') or not config.get(repository, 'digestcmd'):
rc = False
log.error('Missing digestcmd for repository %s' % repository)
if not config.has_option(repository, 'synccmd') or not config.get(repository, 'synccmd'):
rc = False
log.error('Missing synccmd for repository %s' % repository)
if config.has_option(repository, 'parent'):
parent = config.get(repository, 'parent')
if not parent:
rc = False
log.error('Parent repository name missing for %s' % repository)
else:
if not config.has_option(parent, 'host') or not config.get(parent, 'host'):
rc = False
log.error('Missing host for repository %s' % parent)
if not config.has_option(parent, 'user') or not config.get(parent, 'user'):
rc = False
log.error('Missing user for repository %s' % parent)
if not config.has_option(parent, 'password') or not config.get(parent, 'password'):
rc = False
log.error('Missing password for repository %s' % parent)
if not config.has_option(parent, 'path') or not config.get(parent, 'path'):
rc = False
log.error('Missing path for repository %s' % parent)
if not config.has_option(parent, 'listcmd') or not config.get(parent, 'listcmd'):
rc = False
log.error('Missing listcmd for repository %s' % parent)
if not config.has_option(parent, 'digestcmd') or not config.get(parent, 'digestcmd'):
rc = False
log.error('Missing digestcmd for repository %s' % parent)
else:
rc = False
log.error('Repository %s does not have parent repository.' % repository)
else:
rc = False
log.error('Missing configuration for repository: %s' % repository)
return rc
Before we start doing the job, we want to know if our configuration file can be parsed and contains
all the information that we will need. This function is supposed to take care of that.
I am checking if all the keys are present and if they are not empty.
def main(argv=None):
""" """
if argv is None:
argv = sys.argv[1:]
global config
global log
config = ConfigParser.ConfigParser()
config.read(scriptconfig)
logging.config.fileConfig(logconfig)
log = logging.getLogger('mbrepo')
opts, args = getopt.getopt(argv, 'hs:', ["help","sync="])
syncqueue = []
for opt, arg in opts:
if opt in ('-h', '--help'):
return print_usage()
elif opt in ('-s', '--sync'):
if checkconfig(arg):
syncqueue.append(arg)
log.debug('Scheduled repo for syncing: %s' % arg)
else:
raise ProcessingErrorException("Missing configuration to sync repository: %s" % arg)
else:
raise UsageException('Unknown option: ' + opt)
if lock(pidfile):
try:
log.info('Begin repository synchronization. Syncqueue: %s' % ', '.join(syncqueue))
for repo in syncqueue:
syncrepo(repo)
finally:
unlock(pidfile)
else:
log.warning('Terminating - could not create run lock.')
Well, pretty standard main function, doing all the job a script is supposed to do.
Get the arguments cutting off the script name, initialize configuration and logging machinery, parse
the options, create lock, synchronize everything, remove the lock. Allow all exceptions to
percolate (let the ones who call us decide what to do with exceptions), only make sure that the lock
file is not left behind - otherwise cron would not be able to start the script again until the
operator removes the lock manually.
if __name__ == "__main__":
rc = 0
try:
rc = main(sys.argv[1:])
except getopt.error, error:
logging.error('Bad call: %s' % error.msg)
print >>sys.stderr, error.msg
print >>sys.stderr, "For help use --help"
rc = 1
except UsageException, err:
logging.error('Bad call: %s' % err.msg)
print >>sys.stderr, err.msg
print >>sys.stderr, "For help use --help"
rc = 2
except ConfigParser.ParsingError, err:
print >>sys.stderr, err
print >>sys.stderr, "Please correct your configuration file!"
rc = 3
except ConfigParser.NoSectionError, error:
print >>sys.stderr, error
print >>sys.stderr, "Please extend your configuration file!"
rc = 4
except ConfigParser.NoOptionError, error:
print >>sys.stderr, error
print >>sys.stderr, "Please correct your configuration file!"
rc = 3
except InternalErrorException, err:
logging.critical('Internal error: %s' % err.msg)
print >>sys.stderr, 'Internal application error occured: %s' % err.msg
rc = 5
except ProcessingErrorException, err:
logging.critical('Processing error: %s' % err.msg)
print >>sys.stderr, 'Terminating on processing error: %s' % err.msg
rc = 6
except:
raise
sys.exit(rc)
This is the fragment that calls the main function. We call it only if run
directly from command line, catch the exceptions and convert them to error
messages and error return codes.
