1. Introduction
Here is what we are aiming to achieve. We would like to keep a record of the changes made to a policy store under revision control, and to be able to see when policy changes have occurred and view the differences between revisions.
The above picture shows the revision history diaog and in the background (in blue/red/black) a diff comparison of the current revision with the policy store from 6 days ago.
Note: But first a warning: this software is a CA Community development initiative, it is not an official CA product, and this is a first release of the new ssobackup functionality. Having said that the underlying mechanisms (git & ssh in particular) are fairly solid, so we expect it will be fairly reliable, but there will still be some teething issues and possibly some structural changes. We are releasing it in the hope that people will try it, and give us some feedback and thoughts on how the process should evolve - Cheers Mark.
This document is about implementing the ssobackup process, If you are after an overview and demo showing viewing change history then this shorter article is more appropriate : Tech Note : Storing SSO policy changes in Revision Control - viewing changes
The tools for storing text files and source code under revision control have a long history are very robust. Also the SSO XPSExport policy store export format is a close, but not quite perfect, fit for storage under revision control. What we do in this article (and program) is :
a) Tinker with the SSO xpsexport format a little to make it compatible with a revision control system.
b) Write a batch job that commits differences in the modified policy changes.
c) Modify the SMPolicyReader so it can list the git revision history and compare revision versions.
Process Overview : Here is what we have implemented :
The concept is to be able to store a group of related policy stores, eg, DEV, QA, PROD etc in a group in the same PolicyRootEnv directory or repository.
The programs used are :
ssobackup: we have a small program ssobackup (part of the SMPolicyReader package) that runs regularly (once per hour) via a scheduled task/cron job on a policy server in each of the environments. When it runs it does an XPSExport and manipulates the output to make it more "revision control" friendly, and checks in any changes that are found.
SMPolicyReader: The policy reader has the ability to fetch the current policy stores stored in Git and also to view and read the revision history. It can display a list of changes for the policy store as a whole or list of changes for a selected app/domain. The user has some control over what amount of history is shown, and can then view or compare revisions.
The model we are using is revision control stored in a GIT server with access to the Git server via a SSH client. We are conscious that others will have slightly different requirements, so in our design we've tried to isolate the interface to the revision control software to to allow other implementations to be easily developed (For example we expect to have a GIT server via HTTPS access (with UN/PW) to github without too much trouble in the near future).
The ssobackup process and SMPolicyReader are designed and run on Windows and Linux platforms (I tend to develop on Linux, and then run the testing on Windows) - so it should run on both those platforms.
In the below steps we setup and install the ssobackup and policy reader process on one policy server. The process can be repeated on each different environment and on any developer or admin desktop where access is needed. The install steps, each a section below are :
- Prerequisites
- Installing Git & SSH
- Setup of SSH to use public/private key pairs
- Install SMPolicyReader on dev policy server.
- Setup initial repository and dev policy env
- Setup ssobackup program to run on policy server.
- View policy change history on dev policy server.
2. Prerequisites
Requirements for this setup are :
Git : The revision control software : https://git-scm.com/
ssh: Secure Shell : https://www.openssh.com/
SSO SMPolicy Reader : Siteminder Policy Reader via the CA Communitry page
For Linux/unix Git/ssh generally come as a package to install with the operating system. For windows, there are a number of different packages to choose and to keep it simple I walk though the Windows installs of git and ssh that I have used.
The SMPolicyReader, which can already do a compare/diff of XPS exported policy stores, now has a subdirectory/module called : ssobackup.
3. Installing Git & SSH
We need to install Git and ssh on a policy server in each environment, and on any desktop that intends to view the policy store.
3.1 On Windows
For the windows program installs I have left it to use the default install locations.
Git for windows
Here is where I downloaded my GIT client for windows :
"Git for Windows" https://git-for-windows.github.io/
https://git-scm.com/download/win
This guide was also useful:
http://guides.beanstalkapp.com/version-control/git-on-windows.html
SSH for windows
There are a number of ports/implementations of openSSH for windows, putty being the most common one. But to keep it simple, the one I used was - although on install you only need the client part (ie not the sshd server component).
https://www.mls-software.com/opensshd.html
3.2 On Linux
You probably already have them installed, otherwise fire up the package manager and select them - then get a cup of coffee wait for your windows colleagues to catch up, and then move to the next section.
4. Setup of SSH to use public/private key pairs
To communicate with the Git server, via SSH we need to generate an RSA key pair. Note: When prompted for a password to store the key pair please just hit enter so they are stored without a password. This means the user has access to the public/private key pair whenever they are logged on or from a process that is running as a scheduled job by the user. The process is identical on Linux and Windows.
"c:\Program Files\OpenSSH\bin\ssh-keygen.exe" -t rsa
You will need the contents of the id_rsa.pub key, which is your public key, in the next section. Basically you need to place the public key (id_rsa.pub) in the Git server so that when you connect it knows you are a trusted connection. The rsa public key looks like this:
Note: the other file id_rsa is your private key and you should not share this file with anyone - its a bit like giving your password away. It is not that unusual for an unknowing user sends both the public and the private key files to a third party.
5. Create a Repository on your Git Server
We need a Git repository to store the date, that can either be one on your own Git Server, or the Git server can be one provided by an external supplier, we show both below, and for the external supplier we've used github.com.
5.1 Your own Git Server & Repository
Hopefully you have someone that has already setup your company Git server, and you can just ask them to add you to the project. I did set one up and this guide was what I used. For the remote access I've used SSH access (rather than https - latter on we will revisit the https option).
Generating Git Server & creating a Git Repository
The guide for setting up a server and repository on Unix is:
https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server
For me, after setting up the git server, this was how I created the repository:
root@odoma04-vm2:~# su - git
git@odoma04-vm2:~$ cd /opt/git
git@odoma04-vm2:/opt/git$ ls
project.git
git@odoma04-vm2:/opt/git$ mkdir PolicyEnvRoot.git
git@odoma04-vm2:/opt/git$ cd PolicyEnvRoot.git/
git@odoma04-vm2:/opt/git/PolicyEnvRoot.git$ git init --bare
Initialized empty Git repository in /opt/git/PolicyEnvRoot.git/
git@odoma04-vm2:/opt/git/PolicyEnvRoot.git$
Note: after creating the repository, I reset the git user so it is not interactive as per the instructions link above.:
Adding your public key to the git server.
For Linux you just need to append your public key to the end of the ~git/.ssh/authorized_keys file :
cat >> ~git/.ssh/authorized_keys
But presumably you can send your public key to someone who will do that for you:
5.2 Using Github Repository
An alternative to rolling you own git server, is Github. Github.com is a cloud based provider for storing git repositories, they makes setting up and using Git repositories fairly easy. By default the (free) service give public access to your repositories, presumably if that is our choice you will opt for the paid for service.
Creating a Git Repository
The githib.com screen for generating a new repository is :
Adding your public key to the github server.
The github screen for adding your RSA public key to your profile. you click on your picture, and select "settings' and then select "SSH and GPG keys".
5.3 Test Access to you Github Repository
In my case, with my own private git server. I've added an entry into the /etc/hosts file to point the name gitserver to the IP address of the host I installed a git server on.
To access the git repository, we need to run : ssh git@gitserver
On first access to a new host, it will ask if you trust that remote host, that protects against future man in the middle attacks
In the above example it then prompts for a password, that was because I had not setup the ssh keypair as yet.
I then set them up, and added the public key as trusted and the 2nd access below, then connects and authenticates, but then logs me out, that is because the git user is set on the server to not allow interactive sessions - but that is ok, we only wanted to verify the authentication.
We have now completed testing access to our git server.
6. Install SMPolicyReader on dev policy server
The SMPolicyReader can be downloaded from the CA communities site : Siteminder Policy Reader
It is a simple .zip file eg : SMPolicyReader-4_0-ALPHA-426_bin.zip It contains both the SMPolicyReader and the ssobackup programs.
I tend to install the program into a c:\apps directory - I also usually to keep the version # name, but you may wish to rename it to a non versioned name - as we will be latter adding a scheduled task pointing into this directory.
To run the SMPolicyReader usually via run.bat/run.sh gives the best results.
Note1: If you directly click on the .jar file, often you end up running via the 32bit java, even if you have 64bit java installed. 64bit gives more space, and so it is better to click on the run.bat / run.sh file to start it.
Note2: The runOracle.bat is only needed if you are trying to connect to an Oracle ODBC policy store to read the contents as there were some extra .jar files that we could not dynamically add to the path. However, here we are only accessing via the XPSExport, it does not hurt, but you do not need to run that way.
7. Setup initial repository and dev policy env
We want to create a PolicyEnvRoot, which is a directory where we will have subdirectory for each different env realted to this group of policy stores. In the below example we will only create one env a DEV environment, but the process is similar when creating multiple ones. We then will push the directory structure as our initial commit into the git reprository.
Note: We expect to create cmd line version of these steps in future to assist with install on remote non gui servers.
7.1 Create PolicyEnvRoot directory
First we create the PolicyEnvRoot directory. It is where we will store the collection of policy environments. It is also the root directory that is stored in the Git repository. We create this from a menu in the SMPolicyReader and after prompting for some parameters it creates the directory and also pushes the initial setup to the repository. The screens are : ( there is also a cmd line option as well - will discuss in latter post).
First select the menu item Options/New Env Root.
On the policy server we want the "Create New" option, since we don't have a PolicyEnvRoot as yet. But if we already have a PolicyEnvRoot created, then we can use clone to connect to it, rather than creating a new instance.
Once selected init or clone, we get to the revision control template currently we have a choice of one : "git_ssh" - but hopefully more in future "git_https" and "batch_script" option are ones we would like to add - but for now select "git_ssh".
We now select a directory where we want to create the PolicyEnvRoot - We want the parent directory, for example the selection below is for "C:\work" (normally you would only have one PolicyEnvRoot directory, the extra ones you see here are the result from me running these tests)
We then enter the name of the repository/path subdirectory name eg :
We are then prompted for a Admin password for this PolicyEnvRoot - this password needed to create a new env, and is also used to store a copy of the xpsexport passwords for each environment so they can be recovered latter if needed.
Finally we enter the path to the Git repository:
And then we have the confirmation screen :
And if we continue, we get the final screen confirming what directory is created and committed to the repository:
After creation the current "Policy Env Root Settings" can be viewed from "Options/Policy Root Env Settings" menu item:
You can also verify the Admin password and reset it.
(Note; the astute observers amongst you will have noticed some variation of the repository names and directories - yours should all be the same, these screenshots are an accumulation from a few sessions I had)
7.2 Create DEV Policy Store environment
The next step is to create a directory to store store our DEV policy store. This is also done via a menu item off SMPolicyReader.
Select the Options/New Env menu item.
You are prompted for the PolicyEnvRoot Admin password, which you need to enter correctly before it will let you add a new environment entry.
Then a short env code "DEV" is what we have entered here.
You are then prompted for the password that will be used with XPSExport for this environment (each environment can have a different xpsexport password)..
Keeping an eye on the cmd line window for Git add commit and push commands, and then finally conformation the DEV environment is created and committed to revision control.
After creation of the DEV environment the setting can be viewed via menu item "Options/Policy Env Settings" :
You can also check the XPSExport password, or recover it using the PolicyEnvRoot Admin password, or set it to a new value.
7.3 PolicyRootEnv and DEV env Directory Structure
At the end of that process we will have create an PolicyEnvRoot/ssoenv_DEV directory :
With some subdirectories roughly as seen here :
The settings for the current PolicyEnvRoot and which environment are stored in the userSettings file (this is where the batch program will get its settings) :
(There is a menu item on SMPolicyReader Options/Edit User Options that will open the file)
8. Test run of ssobackup on SSO policy server.
The environment is now setup, so now we want to do a test run of the ssobackup process.
From a DOS cmd window we cd to the ssobackup directory and run the program for the first time.
cd \apps\SMPolicyReader-4_0-ALPHA-426_bin\ssobackup
runBackup.bat
The ssobackup program runs XPSExport, manipulates and splits the export file and then commits the data to Git. It writes the output to timestamped log file (it keeps 10 of them).
The ssobackup_stdout_<timestamp>.log file can be checked to ensure the export and commit ran without problems.
I've tidied up most loose ends for adding files to the Git repository, but you can check the status of the created files via :
git status -s
will give a good idea of any additional files that need to be committed :
Also you can run other git commands, such as "git add" "git rm" etc, to tidy up any extra files or directories you or the ssobackup program has added : "git log policystore.xml" show show the main revision history.
Once an environment is setup and running the files under revision control are usually fairly stable, and should not need many manual changes.
9. Setup ssobackup program to run on policy server.
Now that our runBackup.bat is working the next step it to automate it, so that it runs every hour.
9.1 On Windows
On Windows we do that by adding a scheduled job. From the "Start" menu search for the "Tesk Scheduler" program and start that.
Here are the screenshots of the scheduled task that I have created :
The overview:
We set it up to run daily, and to repeat every 1 hours.
The Action is to run the runBackup.bat script and we specify the startup directory as the same directory that the .bat file is in (in our case \apps\SMPolicyReader-4_0-ALPHA-426_bin\ssobackup)
Once setup you can check the main "Task Scheduler" screen to see if the program has run, and also check the ssobackup_stdout_<timestamp>.log log files to verify that it is running correctly.
9.2 On Linux
On Linux you would setup the runBackup.sh to be run regularly via cron - with somethig like :
You may need to tinker a bit so that it runs in the same directory (added a cd in the ssobackup directory for example).
10. View policy change history on dev policy server.
The usage of the SMPolicyReader viewing history I will cover in another post.
But for a quick sanity check, we can run the SMPolicyReader either select:
the menu : Env/"Open Policy Version History"
right click on a Application and select "Show Version History"
You can then select two versions and hit compare.
Hopefully you now see something similar to the above - if not then contact us via the page here give some details of the error you see, and we will see what we can do.
11. Conclusion
And that is it - we now have the DEV policy store being stored under revision control.
You can repeat the above for each different environment QA, PROD etc. You will need to add the ssoenv_QA etc directories to the existing PolicyEnvRoot Git Repository rather than creating a new one each time.
For the stand alone developer, the setup is similar, except they do not have a separate environment and do not need to run the batch job.
Cheers - Mark
----
Mark O'Donohue
Snr Principal Support Engineer - Global Customer Success