Cisco Prime 2.0 Backup Remote Repository Speed Issues

Published on: Jan 8, 2014 @ 22:34
CiscoPrime

You just got your Prime Infrastructure 2.0 all setup and running and collecting data.  So the next best thing is to back up your new Cisco Prime Infrastructure 2.0 to a remote repository data store so that its available off site in the event of failure.

Isn’t that best practice?  Of course it is.    Well I have run into a bug (awaiting # ) and its a silly one at that.

Scenario
You have a Cisco Prime Infrastructure 2.0 Pro OVA which we will call PI going forward.  You decided to create a remote SFTP or FTP repository so that you can back up your PI Application backup nightly.

It’s now time to transfer that file from PI to a remote repository i.e. SFTP or FTP server on the backend.  There are a couple of different ways of doing this.

First:  This is what happens behind the scenes for local backups to the defaultRepo on a PI server.

Local backup to defaultRepo,  CLI view

PRIME-VM1/admin# backup PRIME-VM1 repository defaultRepo application NCS
% Creating backup with timestamped filename: PRIME-VM1-140107-1449.tar.gpg
Backup Started at : 01/07/14 14:49:20
Stage 1 of 7: Database backup …
Database size: 85G
— completed at 01/07/14 15:08:42
Stage 2 of 7: Database copy …
— completed at 01/07/14 15:08:42
Stage 3 of 7: Backing up support files …
— completed at 01/07/14 15:11:15
Stage 4 of 7: Compressing Backup …
— completed at 01/07/14 15:11:48
— completed at 01/07/14 15:35:16
Stage 6 of 7: Encrypting backup file …
— completed at 01/07/14 15:45:17
Stage 7 of 7: Transferring backup file …
– completed at 01/07/14 15:55:40
Total Backup duration is: 1h:6m:20s

Two: The following is an example of a Remote Repository backup via SFTP

Lets start at the beginning and create a new repository for the remote sftp as follows.

PRIME-VM1/admin#
PRIME-VM1/admin(config)# repository RemoteSFTP
PRIME-VM1/admin(config-Repository)# url sftp://10.x.x.2
PRIME-VM1/admin(config-Repository)# user test password plain cisco123 (password gets Hashed later)
PRIME-VM1/admin(config-Repository)#end

Next step is to start the backup. Keep in mind you can either start the process via the GUI Background task or CLI.  I prefer the CLI when doing a traces because you can see things that you normally don’t see in the GUI

PRIME-VM1/admin# backup PRIME-VM1 repository RemoteSFTP application NCS
% Creating backup with timestamped filename: PRIME-VM1-140107-1600.tar.gpg
omitted….
Total Backup duration is: 11h:10m:22s

Lets look at the results.  We should also note that the two servers are on the same network. PI is using the default (ESX 5.0) VMware 1gig interface and the SFTP Windows 2008 R2 Ent 64bit is using 10g interface.

  • Backup to the local repository 1h:6m:20s
  • Backup to the remote repository 11h:10m:22s

Wow 11 hours to back up to a remote server (and yes I tried this as a FTP with the same results +/- 10min). Now if you don’t need this backup finished fast then you’re ok.  But if you’re like me and needs it done now and moved fast then look deeper into the whats going on.

Next Steps
 The next logical step is to engage your peers and Cisco Tac.  Well good news I was already on with TAC filing another bug when this occurred. ( we will leave that for another post because you will love the next one)

So why is this happening?
Cisco TAC’s response “To get the data off fast you need to”…. I paused, and said “What’chu talkin’ ’bout, Willis?” but then I listened and could see where he was going with this one.  

Workaround by TAC
First we need to prepare the PI from the CLI only.

PRIME-VM1/admin# root_enable
Password :
Root enabled

PRIME-VM1/admin#
PRIME-VM1/admin(config)# repository localdisk
PRIME-VM1/admin(config-Repository)# url disk:/
PRIME-VM1/admin(config-Repository)# end

PRIME-VM1/admin#
PRIME-VM1/admin# backup PRIME-VM1 repository localdisk application NCS
omitted you know the rest from above…..
Total Backup duration is: 1h:6m:20s

NEXT

PRIME-VM1/admin# root
Enter root password :
Starting root bash shell …
ade # cd /localdisk
ade # dir
PRIME-VM1-140107-1800.tar.gpg crash defaultRepo ftp lost+found ssh telnet tftp

ade # sftp test@10.x.x.2
The authenticity of host ’10.x.x.2 (10.x.x.2)’ can’t be established.
RSA key fingerprint is BLAH:BLAH:BLAH:BLAH:BLAH:BLAH:BLAH:BLAH:BLAH
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ’10.x.x.2′ (RSA) to the list of known hosts.
test@10.x.x.2’s password:
Connected to 10.x.x.2

sftp> put PRIME-VM1-140107-1800.tar.gpg
Uploading PRIME-VM1-140107-1800.tar.gpg to /PRIME-VM1-140107-1800.tar.gpg
PRIME-VM1-140107-1800.tar.gpg 100% 11GB 3.5MB/s 58:11 #
sftp> exit
ade # exit
PRIME-VM1/admin#

Note Backup Time 58:11

As Mr. Spock would say “Fascinating

But wait there’s more.  I decide to do one more test for myself.   Move the PRIME-VM-140107-1800.tar.gpg to the local FTP directory on PI and leach the file directly from the ftp server built into PI. (are you still with me)?

From Windows Server 2008R2 64 Ent.
D:\SFTP_Root>ftp
ftp> open 10.x.x.1
Connected to 10.x.x.1.
220 Service ready for new user
User (10.x.x.1:(none)): Prime
331 User name okay, need password for Prime
Password:
230 User logged in, proceed
ftp> get  PRIME-VM1-140107-1800.tar.gpg
200 Command PORT okay
150 File status okay; about to open data connection
226 Closing data connection
ftp: 11691215091 bytes received in 389.57Seconds 30010.56Kbytes/sec.
ftp>

FTP “get” alternative transfer in (6min 49sec) even more fascinating.

Summary

So after talking to TAC apparently the root ADE OS is much faster for transferring data when pushing from PI to remote “its a linux thing”.  But if you just simply leech the file after changing the backup location to the local ftp directory then use your ftp client and it transfers like a Thunderbolt.

My recommendation is to change the default backup repository to the ftp server and protect you’re FTP on PI with ACL’s and create a script on your FTP client to “get” the file nightly but,  you are going to have to be creative as the file name is always changing.

Many of you may or may not encounter this but its good to note that this little gotcha is out there.  I sure hope that this will be addressed in the next release. Prime Infrastructure 2.1 perhaps..

A good friend of mine would say “Wow what a kludge”.  Yes that it is.

@WirelessStew
excuse the typos my brain moves faster than my fingers.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s