I recently acquired three 1TB drives and decided to do something about my lack of storage at home. Always trying to make best use of existing kit (and save money) I decided to stick the drives in to my HP ML110 and try something in a VM instead of doing the sensible thing of lobbing them in to a dedicated NAS box.

After wasting a few hours I realised that the onboard SATA RAID controller of the ML110 just can’t do RAID5 and to make matters worse, when I gave up and created a RAID1 array with a hot spare, vSphere 4.1 didn’t recognise the array and instead saw the drives as 3 individual drives. I saw this as a chance to try out the WAFL-alike ZFS file system. FreeNAS had been my NAS of choice recently so I chose to try ZFS in that.

I point blank refused to create 3 1TB VMDK’s (one of each of the three drives) so I set about figuring out how to create Raw Device Mappings (RDMs) of the local SATA drives. There were a couple of posts on the net that got me a little closer, but no guide/article had the whole thing down, so that’s my aim with this blog post.

Step 1

Once you had your drives installed, SSH to your ESXi box (now even easier in vSphere 4.1) and go to the /dev/disks directory. There, if you perform a ls -l, you’ll see your drives listed:

Ignore the instances of your drives which show them as VM stores (vm1.*****). We want to look at the raw devices.

Step 2

Now move to the /vmfs/volumes folder. Here you can see your existing local datastore(s). If, like me, you had a solitary hard-drive, you’ll just see localdisk01 or whatever you chose to name the local datastore:

Step 3

Now we are going to use the vmkfstools utility to create our RDM’s. Remember that a RDM is just another VMDK, but instead of the VMDK pointing to a xxx-flat.vmdk file (which is the actual virtual hard disk), the VMDK points to our physical device. Being as we still need to create this VMDK file we need to save it somewhere. Since we just have the one local datastore, we are going to create the RDM VMDK files in it’s root.

The following command creates the RDM VMDK for us:

vmkfstools -z /vmfs/devices/disks/<name of RAW device from Step 1> <location to store VMDK>/<RDM name>.vmdk

In my personal example below, I am creating an RDM called rdm_WD2DWCAVU0477582.vmdk and it is being stored in the location /vmfs/volumes/localdisk01/ I chose the name of the VMDK to match the name of the serial number of the physical drive (and what is shown in Step 1) to help with troubleshooting in the future when I get an inevitable drive failure). You can call your RDM’s whatever you wish.

The name of the RAW device (t10.ATA____WDC_WD10EARS2D00Z5B1__________________________WD2DCAVU0477582 in my example) you will have noted from Step 1 when you listed all local devices attached to your ESXi host. This is why the tech Gods created Copy n Paste! You will want to copy the full device name as shown in Step 1 in to the vmkfstools command.

Step 4

Once you have repeated the steps for all of your local SATA drives, you can navigate to where you created the RDM’s (in my case /vmfs/volumes/localdisk01) and perform an ls -l *.vmdk to see the new VMDK’s you have created:

Don’t panic – the xxx-rdmp.vmdk files will reflect the size of the RAW devices they are mapping to, but rest assured it will be taking no more space than a few bytes on your local disk!

Step 5

You can now add your RDM’s to an existing VM. vSphere doesn’t recognise this as a true RDM (to a SAN) so you just browse the local disk datastore for the VMDK files that we created.

Edit the properties of an existing VM and click Add…

Step 6

Select Use an existing virtual disk and click Next >

Step 7

Click Browse. You now need to navigate your local datastore and select the VMDK’s that we created in Step 3).

Once complete you will be shown a confirmation window. Repeat Steps 5 through 7 to add additional RDM’s to your VM.

Step 8

You should now see your new Hard Disk’s in your VM and vSphere will correctly identify them as Mapped Raw LUN.

NOTE: One thing I forgot to show in the screen shots, is that you should create your RDM’s on a new SCSI controller! You do this by simply selecting a new SCSI ID starting with 1:x instead of 0:x. Existing VMDK’s should be on SCSI Controller 0. Your RDM’s should be on SCSI controller 1. Although my screenshot shows 0:3 this should read 1:3.

You can now save your VM configuration. Your VM will now access the RAW SATA drives  and be able to use things like SMART to monitor its health.

See below; I am adding my three 1TB drive to FreeNAS to create a new ZFS pool.

Stay tuned for an upcoming blog post on FreeNAS and NexentaStor which may or may not put you off ZFS [in a VM] altogether!

The MSA 2000 series has a pretty dire web interface – when you compare it with other SAN’s – and to make matters work, it mixes and swaps terminology with HP’s own EVA’s (a vdisk on an MSA refers to a RAID array made up of physical disks. A vdisk on an EVA is a virtual RAID array which sits on top of the underlying RAID 0 structure). The web interface is challenging enough as it is when it works, but recently the web page of the management interface has been timing out.

Thankfully SSH access still works and responds well. I’ve been using…

restart mc a

…to restart the management interface which helped things for a short while but ultimately I found myself falling back to SSH more and more.

The following procedure follows the steps I took to rename and prevent a volume to a host.

To give a host a friendly name (from its WWN):

set host-wwn-name host <wwn> <friendlyHostName>

To rename a volume:

set volume <volName> name <new_volName>

To view what hosts a volume is currently presented to and what volumes map to what hosts:

show host-maps [<friendlyHostName>]
show volume-maps [<volName>]

To actually present a volume to a host:

map volume <volName> [lun <lunID>] host <friendlyHostName> access rw

In theory the LUN ID can be left blank, but in practise I found that it would not map unless I specified an ID. Don’t forget to map both HBA’s for a host.

After presenting the volume to the Windows 2003 host I was dismayed when the Storage MMC could not see the new lun. Hitting F5, of course, was not enough. On the Disk Management section folder in the left hand side of the MMC window, right-click and select Rescan Disks.

I’ll update thist post if I use and feel the need to remember some other command line options for the MSA 2000 but hopefully we’ll be migrating off it to EVA soon!

In the new environment we let Netbackup dictate which tapes are in the scratch pool and free for re-use based on the expiration policy of the jobs written to those tapes. In theory this should mean that you only ever use as many tapes as you need for a given backup job. The problem with this set-up is that although it is, in theory, more efficient, you don’t always know which tapes you will need each night and how many you have free.

Each day guys were running NBU command lines to show which tapes were in the scratch pool, but having to manually sift through them to see which tapes were currently sat in drives/libraries and of the ones held off-site, which tapes were of which format (LTO2, 3 or 4).

The following script can be scheduled to email out a list of tapes in the scratch pool, which are in what library and also groups the offsite tapes by format. It was knocked up in a hurry so isn’t as parameterised or dynamic as it should be, but the comments should help you fix it for your needs…

Here’s the script and HTML header & footer…

scratch_list.ps1

$debugpreference = "continue"

#This "output" variable runs and stores the output of the NBU command line
#Be sure that this .exe is in your path, otherwise change the below to include the full path to the executable.
$output = vmquery.exe -b -pn Scratch
$htmlfile = ".\output.txt"
$today = Get-Date

#Arrays to hold and sort info about our tapes.
#NOTE: this script is hard-coded to work for LTO2 (HCART2), LTO3 (HCART3) and LTO4 (HCART) tape drives/libraries
[array]$aTapes = @()
[array]$aHOffsite = @()
[array]$aH2Offsite = @()
[array]$aH3Offsite = @()

#NOTE: this script is hard coded to include the specific names of tape drives attached (via various media servers) to the master server
$aRobots = @{"TLD0" = 0; "TLD1" = 0; "TLD2" = 0; "TLD3" = 0; "TLD4" = 0; "TLD5" = 0; "TLD6" = 0; "TLD7" = 0; "TLD8" = 0; "TLD9" = 0; "TLD10" = 0; "TLD11" = 0; "TLD12" = 0; "TLD13" = 0; "TLD14" = 0}
$emailBody = Get-Content "header.html"
$aOffsite = 0

#Function to send mail (currently configured to send via Exchange)
function sendEmail() {
 $eServer = New-Object system.net.mail.smtpClient("exchange_server01")

 $From = "netbackup@yourcompany.com"
 $To = "nbu_reports@yourcompany.com"

 $todayDate = [string] $today.day + "/" + [string] $today.month + "/" + [string] $today.year

 $title = "Scratch tape list for $todayDate"

 $Body = $emailBody
 $eMsg = New-Object System.Net.Mail.MailMessage($From,$To,$title,$body)

 $eMsg.IsBodyHTML = $True

 $eServer.send($eMsg)    
}

#Take output of NBU command line util, and for each line pick out bits of the string
#Chuck everything in to a new array called $aTapes
foreach ($line in $output){
 if (($line.contains("HCART")) -and (-not $line.contains("A00"))) {

 $tName = ($line.substring(0, 6)).trim()
 $tType = ($line.substring(8, 6)).trim()
 $tRobot = $line.substring(23, 5).trim() #this captures a wide area with some white space that needs trimming

 $tRobot = $tRobot.replace(" ", "")

 $aTapes += ,($tName, $tRobot, $tType)
 }
}

#Go through the array we just created and then put the contents in to seperate arrays
#dependant on whether they are LTO, LTO2 or LTO3 tapes...
for ($j = 0; $j -lt $aTapes.length; $j++) {
 $tapeID = $aTapes[$j][0]
 $robot = $aTapes[$j][1]
 $type = $aTapes[$j][2]

 if ($robot.contains("-")) {
 switch ($type ) {
 "HCART" {$aHOffsite += ,($tapeID)}
 "HCART2" {$aH2Offsite += ,($tapeID)}
 "HCART3" {$aH3Offsite += ,($tapeID)}
 }

 } else {#If a library type does not contain HCART, HCART2 or HCART3 we must assume it's offsite (or in a safe))
 $temp = "TLD$robot"
 $aRobots[$temp] = $aRobots[$temp] + 1
 }
}

#Take the contents of $aRobots (offsite tapes) and put it in to a variables ready to be dumped in to our html output
foreach ($element in $aRobots) {
 $element > $htmlfile
}

#Start pipping everything out to the $emailBody variable which will make the body of our email
$emailBody += '<table><tr><td>Tape drive</td><td># scratch tapes</td></tr>'
foreach ($line in Get-Content $htmlfile) {

 if ($line.contains("TLD")) {
 $emailBody += "<tr>"

 $robot = $line.substring(0, 5)
 $count = $line.substring(31, 3)

 $emailBody += "<td>$robot</td><td>$count</td>"

 $emailBody += "</tr>"
 }
}
$emailBody += '</table><br/><br/>'

$emailBody += "<b>Offsite tapes...</b><br/>"

$emailBody += "<b>HCART</b><br>"
foreach($element in $aHOffsite) {
 $emailBody += "$element<br/>"
}
$emailBody += "<br/><b>HCART2</b><br>"
foreach($element in $aH2Offsite) {
 $emailBody += "$element<br/>"
}
$emailBody += "<br/><b>HCART3</b><br>"
foreach($element in $aH3Offsite) {
 $emailBody += "$element<br/>"
}

$emailBody += Get-Content "footer.html"

sendEmail

As per my other NBU scripts I use HTML files to store the CSS and opening HTML in order to easily format nice looking emails. See below for examples…

header.html

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
<head>
 <title>Scratch tape list</title>
 <style type="text/css">
 body {
 font-family: Calibri, Arial, Helvetica;
 color: black;
 font-size: small;
 background-color: #FFFFFF }
 p {
 font-size: small }
 .big {
 font-size: small }
 table {
 font-size: small;
 padding: 0px ;
 border-spacing: 2px ;
 empty-cells: hide ;
 border: 1px solid #AAAAAA }
 td {
 background-color: #EEEEEE;
 vertical-align: top }
 th {
 background-color: #EEEEEE;
 vertical-align: top ;
 text-align: left}
 </style>
</head>
<body>

<p>*** This email is now automatically generated ***<br></p>

footer.html

<br/>
<p>Regards,<br/><br/>
</p>
</body>
</html>

Let me know if this helps you out at all or if you can suggest any improvements…

Recently I’ve installed FreeNAS on to an old PC to (a) start getting some stability to my storage solution, and (b) play around with ZFS and iSCSI at home. I tried OpenFiler but it pales in comparison to FreeNAS. I’ve gone against my better judgement and I’m using some of the cooler plugins including the Torrent client, Dynamic DNS and I’ve even installed SABNzbd on it. FreeNAS comes with rsync built in and even Unison which is an interesting solution to cross platform backup/synchronisation, but I’ve payed for my CrashPlan Central plan so want to make use of it!Only problem now is backing up my data!

CrashPlan doesn’t support backing up from a network share – either mapped drive or UNC path. There is a work around on the CrashPlan site which works but is a little convoluted so wanted to post a nice, quick version here.

Read on after the break… The problem is simply that CrashPlan runs as a service using the SYSTEM account. To allow us to backup a network share we have a couple of options….

1. Change the service to run as a standard user account (e.g. the one you’re logged in to your machine as)
2. Map a drive under the SYSTEM account so that it is visible to CrashPlan

I’d recommend the latter but if you just plan on using this on a desktop machine and/or want a quick and dirty solution, here’s how you change the logon account for the service…

• Open up the Services MMC and find the CrashPlan Backup Service. Double-click it to open its properties
  
• Click the Log On tab and then select the This account radio button. 
• Enter the username and password of a local/domain user account with full permissions to the network share you want to back up 
  
• Click OK and then restart the service 
  
• Relaunch the CrashPlan application and you’ll now be able to see drives that you currently have mapped as a backup selection as well as local drives

The preferred method is to map a drive as the SYSTEM account. That in itself is no big deal, the thing is getting the drive mapped as soon as the machine boots and doing it each and every time your machine restarts. To do this we simply make use of Vista/7′s Task Scheduler.

The batch script, which we’ll save as c:\mount.bat simply uses a NET USE command. The one drawback to this method is that the username and password need to be saved in the script. You must use a username which has permissions to the network share. This need not be a local username, in the case of FreeNAS, if you have it joined to a domain as I do.

Here’s the code:

net use M: \\freenas\media /USER:dave@mydomain.lan myP4ssw0rd

Once you have that saved open up the Task Scheduler…

Create a new task (I keep my tasks in a seperate folder to the default Windows ones simply for easier administration). Give your ask a name, for example Mount drive for CrashPlan. Select Run whether user is logged on or not, tick the Run with highest privileges box and then click Change User or Group…

Type system and then Check Names (it should capitalise and underline). Click OK.

Back in the Create Task window slect the Triggers tab and click New… Select At log on… from the drop down list and click OK to close the window.

Now select the Actions tab and, again, click New… leave the action as Start a program and then browse to your batch that we created above.

Click OK to close the New Action window and again for the Create Task window.

Job done! You can right click on the task and select Run to try it out.

A much more elegant solution would be to have a native FreeNAS (or just BSD) client for CrashPlan or to make use of rsync/Unison. This will do me for now though.

I wrote this script to report on the status of snapshots. The filer name is passed as a variable and a list of volume names is stored as a variable at the top of the script. The output is nicely formatted email.

1. The script uses SSH to connect to the filer (see my SSH guide for Netapp filers)
2. For each vol listed in the array, the list of snapshots is returned
3. Any line with “__daily” in is processed – all others are ignored (we only care about our nightly verified snapshots which are all named __daily)
4. The snapshot name contains the date/time it was taken – this is read and if the date is within 24 hours of now a success is return (since reports are run the day after the verified job). If the date is over 24 hours a failure is returned.
5. A nicely formated HTML email is sent out which looks a little something like this…

Please see below for today’s backup report.

See the script and HTML files after the jump…

Don’t forget to change the name of your mail server, your sending and recipient email addresses and, most importantly, the location of plink.exe and your DSA keys used to establish the SSH connection (although you could easily modify this script to use RSH instead).

snapReport.ps1 script

# ==============================================================================================
#
# NAME:     Netapp filer snapshot reports
#
# AUTHOR:     David Warburton
# DATE:     12/03/2009
#
# PARAMETERS:    1. $filer
#        Specifies the name of the remote filer to connect to.
#        example: snapReport.ps1 myfiler01
#
# ==============================================================================================
$debugPreference = "continue"
$errorActionPreference = "stop"
$filer = $Args[0]
$htmlFile = "D:\Logs\Exchange Snapshot Report\exchange_Snapshot.html"
$volumes = ("Vol_Exc02_DB", "Vol_Exc02_LOG", "Vol_Exc02_DB1", "Vol_Exc02_LOG1", "Vol_Exc03_DB", "Vol_Exc03_LOG", "Vol_Exc03_DB1", "Vol_Exc03_LOG1")
$header = get-content "header.html"
$header > $htmlFile
$today = get-date
function sendEmail() {
 $eServer = new-object system.net.mail.smtpClient("mailserver.yourdomain.com")
 $From = "youraddress@yourcompany.com"
 $To = "backup_reports@yourcompany.com"
 if ($checkType -eq "weekly") {
 $title = "Weekly "
 } else {
 $title = "Daily "
 }
 $todayDate = [string] $today.day + "/" + [string] $today.month + "/" + [string] $today.year
 $title += "Exchange snapshot Report for $todayDate"
 $Body = get-content $htmlFile
 $eMsg = new-object System.Net.Mail.MailMessage($from,$to,$Title,$body)
 $eMsg.IsBodyHTML = $True
 #$eMsg.cc.add($cc)
 $eServer.send($eMsg)
}
function checkSnap($volName) {
 $output = D:\Utils\PuTTY\plink.exe svc-ssh@$filer -i D:\Utils\PuTTY\id_dsa.ppk snap list $volName
 $validsnap = $False
 $snapname = ""
 foreach ($line in $output){
 if ($line.contains("__daily")) {
 $sMonth = $line.substring(62, 2)
 $sDay = $line.substring(65, 2)
 $sYear = $line.substring(68, 4)
 $sDate = [datetime] "$sMonth $sDay $sYear"
 if (($sDate.date).addDays(1) -eq $today.date) {
 $validsnap = $True
 $snapname = $line
 }
 }
 if ($validsnap) {
 return $snapname
 }
 }
 return $False
}
foreach ($vol in $volumes) {
 $line = "<tr><td>$vol</td>"
 $status = checkSnap $vol
 if ($status -eq $false) {
 $line += "<td><font color=red>FAILED</font></td>"
 } else {
 $line += "<td><font color=blue>Success</font></td>"
 }
 $line += "<td>$status</td></tr>"
 $line >> $htmlFile
}
$footer = Get-Content "footer.html"
$footer >> $htmlFile
sendEmail

Here are the HTML bits for the header and footer of the email…

header.html

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
<head>
 <title>Backup Report</title>
 <style type="text/css">
 body {
 font-family: Calibri, Arial, Helvetica;
 color: black;
 font-size: xx-small;
 background-color: #FFFFFF }
 p {
 font-size: x-small }
 .big {
 font-size: x-small;
 font-weight: bold }
 table {
 padding: 0px ;
 border-spacing: 2px ;
 empty-cells: hide ;
 border: 1px solid #AAAAAA }
 td {
 background-color: #EEEEEE;
 vertical-align: top }
 th {
 background-color: #EEEEEE;
 vertical-align: top ;
 text-align: left}
 </style>
</head>
<body>
<p>*** This email is now automatically generated ***<br><br><b>Please see below for today's backup report.</b></p>
<p>
 <b>NOTE: </b><br/>
 - Exchange and SQL snapshots are managed by SnapManager and are alerted independantly to this report.<br/>
 - Not <u>all</u> snapshots are shown in these reports. Hourly snapshots are currently not reported upon.
</p>
<table>
<tr>
 <th>VOL Name</th><th>Status</th><th>Number of Snapshots</th><th>Last snapshot taken</th>
</tr>

footer.html

</table>
<br/>
<p>Regards.</p>
</body>
</html>

When copying files from a CIFS snapshot using the hidden .snapshot directory NTFS permissions are lost. Instead they are inherited from the folder you are copying to.

To maintain permissions, use NDMPCOPY specifying the snapshot name in the source (first) parameter…

ndmpcopy "filer1:/vol/vol_A/qtree_A/.snapshot/nightly.1/MyDocs" "filer1:/vol/vol_A/qtree_A/MyDocs"

You can enclose the path in quotes if the path has spaces in.

The great thing is that all the storage processing is offloaded from vCenter and is performed entirely on the filers. I’ve not had a chance to play with the RCU yet but this just looks utterly awesome! Check out this preview blog post and video from Netapp:

http://blogs.netapp.com/virtualstorageguy/2009/12/preview-rapid-cloning-utility-30-vcenter-plug-in.html

If you don’t have time to sit and read, then at least check out the video. I dare you not to be impressed!

Do any other storage vendors have similar tools?

In either case scripts are not processed on the filers themselves. They are scheduled and run on Windows/Linux systems using their native languages (e.g. Vbscript or Powershell). These scripts simply call native OnTAP commands against a filer which then returns data in the form of text. This is important to note as although Powershell is a useful language to use OnTAP will not return native PS objects which means you are limited to text/string based manipulation.

RSH

This method is the least secure but is the quickest and easiest to set up. Users need not necessarily be filer administrators. Although there is some degree of security in the sense that access is restricted to a set user account and accessible only via a set IP address, data is not encrypted.

RSH is not natively supported in Windows 2008 [from the command line] as it is in Windows 2003.

To enable RSH on the filer use the following command:

options rsh.enable on

You must also specify user accounts and IP addresses via the FilerView web interface.

SSH

Windows 2008 does not natively support SSH either so this is accomplished using plink.

To enable SSH on the filer use the following commands:

secureadmin setup
secureadmin enable ssh

Read more after the break…

Configuring SSH

1. First create a local user account on the filer using the following command:

    useradmin user add svc-ssh -g Administrators

   1. the passwords are kept in the Password spreadsheet
   2. The user must be a member of Administrators to be granted the Login_ssh permission on the filer
2. Using Powershell (whilst running with your Admin Account) change directory to the hidden share on the filer:

    cd \\filer\c$\etc\sshd\

3. Create a folder with the same name as your account created in step 1 and then another folder to hold the SSH keys:

    md svc-ssh
    md svc-ssh\.ssh

4. On the script server lunch PUTTYGEN.EXE and under the “Key” menu change the type to “SSH-2 DSA key”. Leave everything else on its default setting.
5. Click on the Generate button and follow the prompt
6. Enter the same password from step 1 in the “Key passphrase” and “Confirm passphrase” areas (they do not HAVE to be the same as step 1 but are being kept so for simplicity)
7. When complete, click on Save public key. Save it to the PuTTY folder with the name:
   id_dsa.pub
8. Save the private key to the same place with a similar name:
   id_dsa.ppk
9. Copy the contents of the “Public key for pasting into OpenSSH authorised_key file” area and paste it in to a new text document. You should use Wordpad to do this as Notepad does not handle the line breaks properly. Save this document as authorized_keys.
10. You are now finished. You can test connection using PuTTY by selecting your private key. The first time you connect you will be prompted to accept the hosts key. You will still be prompted for a username but you should not need to enter a password as this is what the SSH keys are for.

For scripting purposes we use plink.exe to issue a command. For example:

.\plink.exe svc-ssh@filer -i "id_dsa.ppk" df -s

Scripting using plink

The basic use of plink.exe is as follows:

.\plink.exe svc-ssh@filer -i "id_dsa.ppk" df -s

Two methods can be used to script the filers depending on what you’re trying to achieve:

NOTE: when calling plink.exe in Powershell the full or abbreviated path must be specified. So use either a full path, for example, D:\Utils\PuTTY\plink.exe or just .\plink.exe

Reporting

From a Powershell command prompt simply pipe the output of an SSH command in to a variable for later manipulation:

$output = .\plink.exe svc-ssh@filer -i “id_dsa.ppk” df -s

Maintenance

If multiple command need to be issued – for example, you need to delete a snapshot, rename the previous 5 snapshots, and then take a new snapshot – you should issue them all via the one SSH connection.

Commands can be concatenated using a semi-colon. Just be sure to enclose everything in quotes. For example:

 .\plink.exe svc-ssh@filer -i "id_dsa.ppk" "snap delete Vol_A hourly.6; snap rename Vol_A hourly.5 hourly.4; snap rename Vol_A hourly.4 hourly.3; snap rename Vol_A hourly.3 hourly.2; snap rename Vol_A hourly.2 hourly.1; snap create Vol_A hourly.0"

Well that’s it. It’s a pretty brief post – if you have any questions please feel free to post them in the comments…

Until now our users have demanded daily reports to be emailed out to key individuals. The format of these emails was quite basic in theory…it simply showed a list of all servers (clients) that were backed up, their status from last night (e.g. successful, failed, etc…) but then, crucially, a list of files that had been skipped.

The reason for listing each file that had failed was due to Netbackup’s poor poor (don’t get me started on how poor) feedback in the Activity Monitor and it’s built in reports. You see sometimes jobs could be marked as “Successful with warnings” if the job completed but the odd file had been skipped (if it were in use, for example). But other times, although it still listed which files had been missed if you double-clicked on each job, it showed the overall status as Successful. This meant that to report on the servers in the format that was required by our users we had to manually trawl through the 360 jobs across two Netbackup servers just to copy and paste the lines of the files that had been missed.

Needless to say it didn’t take me long to get bored of this so I started looking at a way of reproducing our reports automatically. After some time, I’ve come up with a solution – it uses VBScript to parse the logs files which I output using a few key Netbackup command-line tools.

The basic process and components to the script are as follows…

1. A VBscript calls a number of external batch scripts
2. These batch script run NBU commands and spit their output to text files
3. The VBscript then reads these text files
4. The VBscript reads a servers.txt file to determine what jobs to report on*
5. A LOT of array and string manipulation is used to catalogue which jobs have run and which have errors and if so, it collates all skipped files
6. These arrays are then parsed and dumped to either an HTML file (which can be used directly in the body of an email) or, as I’ve recently changed to script, outputs to an Excel file.

There are some nice bits to the script(s) and some really nasty bits – I welcome you to improve on the original and share it with everyone (for example, I know there is a much neater way of calling and parsing the output of batch jobs rather than the nasty wscript.sleep I have used).

* A text file (called servers.txt) is kept along side the scripts. This file is manually edited and must be kept up to date. The file is a number of lines, each one representing a client/job that we wish to report on. The format is as follows:

60, myserver, mypolicy, GroupName

The number at the start represents what days of the week the report should be generated. myserver is obviously the client that is to be reported on. mypolicy is the name of a policy that you are reporting on (since a client may me a member of multiple policies). Finally, GroupName is free text and can be used to group similar clients (that may not necessarily be grouped by the same policy.

The scripts necessary to generate a report are…

1. generateReport.vbs
   The primary script which calls the others and also performs of the processing and HTML/Excel generation
2. jobs_summary.bat
   Issues the bpdjobs – simmary command and pipes output to a text file. This provides a simple overview of number of total jobs and the number that have succeeded, failed or errored.
3. jobs_report.bat
   This is the main batch script upon which the report is generated. It is a hideous mess of space-formatted garbage and why so much processing is needed to generate a useful and nice looking report.

Clicking the scripts above will take you to a separate page with the code and more explanations of how to use it in your environment.

UPDATE: Okay, so enough of you have expressed an interest that I decided to put it up for public download. Please note that it is not currently protected via any kind of license so please use your discretion. I have included a very basic READ ME that I encourage you to go through. Post your comments here if you have any problems.

Download: Dave’s Netbackup Reporting script