It's becoming clearer and clearer that things are changing drastically in the IT infrastructure (IT Pro) world.  Last year I attended a talk by Don Jones who scared the ***t out of an audience.  He said that in a few years time, there would be much fewer IT Pro jobs.  What we'd have is a smaller set of junior or operator engineers, almost no one in the middle and a small group of senior engineers.  Those who would rise to the top and stay in IT would be those who could learn something inside-out and learn to leverage automation.  And to get to that level, these engineers will have to be interested in their jobs, not be one of those 10-till-4 types I've discussed before.  Key to their success will be the ability to learn on their own and to provide business solutions, not IT ones.

How this is going to happen is becoming evident now.  You've probably heard of Cloud Computing and SaaS but I'll quickly talk about them. 

Would you build a nuclear power plant in your back yard if you need electricity in your house?  Probably not, but we take this approach whenever we need a new business application.  Take a CRM application.  It might need a database and an application server.  If fault tolerance is required then you need more servers, clustering, etc.  All this IT complexity is added to non-IT companies every day and they find themselves becoming accidental IT companies.  Sure, there are consultancy and field engineering companies but they don't take the pain away.  For example, when a CRM must be upgraded there's more servers, operating systems, a costly project and a data migration.  The non-IT company finds itself immersed in an IT project that consumes time and money and puts their business data at risk.  It's not just CRM either ... it's everything from the SBS server, tape backups, databases, ERP systems, risk management, etc.

The principle of Software as a Service (SaaS) is that you should avoid this on-site installation and consume applications on an as-needed basis.  Your service should be like a household utility, e.g. sign a supply contract for electricity and turn the power switch on and off as required.  You know how much a unit costs and you can budget accordingly.  We've seen how companies like SalesForce and Google have done this with their services.  Microsoft isn't far behind (BPOS) either.  SaaS isn't just for special online solutions.  You can cut the costs and complexities of owning many solutions, e.g. a DR site, your internal IT systems in an outsourced deployment, etc.

This all requires a service delivery mechanism.  Some companies like Google and MS are big enough and skilled to host their own solutions on the Internet in high quality data centres.  However, smaller or niche companies looking to build a SaaS service can't build something of that quality.  They need a quality data centre (not a computer room) because their business is dependent on this facility.  Not all data centres are the same either.  You'll want to check them out and get advice from people who know the industry.  Don't base any decisions on web sites, press releases, marketing or your own IT experiences.  The data centre world is very complex and full of many deep pitfalls that can end your career.

So if you cannot build your own data centre for your SaaS product then you can use Cloud Computing.  The idea is simple.  A service provider owns, manages and leases an infrastructure.  You simply subscribe for the functionality you require as you need it.  Most software developers aren't IT infrastructure experts so building and managing a best practice and secure architecture is hard for them.  With a good hosting partner, they can use this black box solution called Cloud Computing to rapidly get the server/network resources they need and grow/shrink them as customer demand changes.

Before I go any further, not every application will be SaaS and not every server will migrate to the cloud.  Some organisations just won't be able to for security, unionisation or complexity reasons.

Here's the rub for IT pros.  If you don't work for one of these cloud computing firms you might not have a job in 10 years.  Think about this ... if your employer can reduce costs and complexity by using SaaS applications or servers that reside in the cloud then do they really need you?  They already perceive IT infrastructure departments as a cost centre that eats up budget and delivers 80% of what they promise ... late.

So where I am thinking most IT pro jobs will be in the future?  In the data centre that hosts cloud computing infrastructures.  Operators are junior staff that look after the physical infrastructure.  They rack servers, run cables and look after the NOC.  They're the first point of call for support issues.  There's always a good number of these folks to maintain a 24 operation (... or there should be.  Try knocking on the door of any data centre you're considering at 2am to check out promises of 24 hour on site presence.  You'll be surprised who makes claims and who fails to live up to them!).  The folks who design and deploy systems will be the senior engineers.  In a large facility these will often be specialists, e.g. firewall CCIE's, messaging experts, server OS guru's, DBA's, etc.  Not only must they understand complex technologies but they must know how to handle huge workloads efficiently.  They'll be managing a huge number of machines and applications so automation will be critical.  Using correctly designed solutions they can take control of the network.  Have a look at the concepts of Optimised Infrastructure and you'll see what I'm getting at.

We've heard the talk about outsourcing before.  Some companies did bring in IT companies to replace their IT staff but there's no cost savings there.  Quite the opposite to be honest when you replace like for like at a higher daily rate.  But being able to access subscription based services from a quality data centre with centralised expertise and management systems will give the financial and business reasons for employers to reconsider their IT situation.  This isn't just me talking, it's every big brain out there.  I attended a session in Barcelona that said IT will have completed a swing that way in 10 years time.  If you're an application developer then you actually need to be engineering your SaaS solution now or it's already too late!

If you're an IT Pro and that's your career choice rather than an accident then my advice is to get really good at something and learn how to use automation to manage a network.

This page has a collection of useful links for installing and managing Hyper-V.

The first public beta release of OpsMgr 2007 R2 is available for testing now.  Features include:

• Manage UNIX and Linux seamlessly.  I've seen this in action.  It is not a bolt on.  Operationally, it looks very nice.  They decided to do it only in R2 because it needed some changes in how to use the role user accounts.
• More VMM integration
• Improved web application monitoring
• The SLA stuff appears to be integrated rather than being a bolt on as it currently is
• A faster console
• Better management pack ... management (stuck for a better word there!)
• Simplified notification  - it was needed because it's a maze to figure out for the first time
• Improved and simplified authoring - I really hope so because discovery is a nightmare

Microsoft currently sells a subscription service product for consumer computer security called Live OneCare.  It takes care of AV, firewall, spyware, etc.  I've used the trial and I reckoned it was pretty good for the domestic user.  I didn't subscribe - I've been using AVG free for a while now (Avast beforehand) and I find it pretty good.

According to Bink, MS are going to stop selling OneCare via retail in June 2009 and replace it with a free product.  The aim is to get as many people protected as possible, thus giving Windows consumers the protection from malware that they need.  The decision to phase out OneCare allows MS to focus their efforts on a single consumer product.  Making it free spreads the cover of their protection to the maximum possible install base.

Credit: Bink.

I decided to have a play with Live Mesh tonight.  I've wanted a way to synch my Favourites folder between laptop, desktop and my work laptop.  That rules out using folder redirection on my network at home - anyway I'm thinking of flattening the SBS box and reusing the machine for something else.

I installed it from the web site and synched the favourites from my personal laptop.  I could then sign into mesh on my work laptop and view the folder contents.  I was then able to install the client on that machine and view the folders.  I opened Favourites from my personal laptop (on my work machine) and copied in the work machine's Favourites.  They instantly appeared over on my laptop.  They now stay in synch whenever I change a file.

It looks like you can share your files via Mesh as well.  You could think of it as a very limited (in function and size) version of Groove.

I also tried out the remote desktop feature.  It appears that is uses Remote Assistance tunnelled over HTTPS.  That bypasses those ISP's who've been blocking "work" network protocols such as PPTP, RDP and IPsec.  The performance was quite good.  The only downside was that it required an approval for the connection on the target machine.

I've used a "security server" running DL360's with WSUS and AV in several jobs now.  They're great candidates for virtualisation so the security server at work was my first target to convert to a virtual machine, thus freeing up some h/w for profit making.

The P2V process of VMM 2008 is pretty easy.  I found no fault with it.  However, I did have some problems that were non-VMM 2008 related.

The VM would hang on boot up.  I got it into safe mode and disabled the HP services.  They were trying to access hardware that didn't exist.  Ideally you would uninstall this stuff before P2V but I needed to keep the physical machine online until the virtual was ready.

Once the VM was ready I installed the integration components in VMM 2008.  I fired up the VM and tried to log in ... uh oh!  It needed to be reactivated.  Luckily I'd put the machine on a test network with Internet access so that was done.  Then I had a service failure pop-up.  The event log showed that was OK, the server was looking for the domain and not finding it ... it's still on the test network while the physical machine is still providing services.

Now the killer.  I got a pop up about WMIPRVSE failing.  That repeated 9 times when I closed it.  I also had dozens of WINMGMT errors in the application log.  To troubleshoot I made a checkpoint and started googling and trying things out.  In the end here's what it came down to:

• Uninstall anything related to HP.
• Edit the registry and searched for anything to do with HPWBEM.  I deleted the relevant keys/values.  Some needed to be edited instead of deleted.  This took ages!
• Searched for HP services in CurrentControlSet\Services.  They weren't removed by uninstall's.
• Rebooted
• Removed HP folders from Program Files.
• Uninstalled the OpsMgr agent (I wasn't taking chances now - because I was still getting the error after reboots).
• Removed the ATI driver which I'd forgotten to remove.
• I reset the WMI repository.
• After a reboot the WMI errors disappeared.

As I said, the P2V worked perfectly.  Any problems were related to the HP software, e.g. not uninstalling correctly.  There seemed to be loads that needed to be done.  I'd tried lots of combinations in various attempts by restoring the checkpoint.  Looking back on it, I doubt the OpsMgr agent was a factor but I removed it anyway in case it was doing some heavy WMI stuff that was no longer applicable.

CAUTION: Edit the registry at your own risk.  I'm not recommending it.  It's just what I did to solve my problem.  If you screw up your server then it's your problem, not mine.

John Howard from MS wrote a 5 page article on how to grant remote admin rights using the Hyper-V MMC to Hyper-V servers that were not in your domain, e.g. in an un-trusted domain or in a workgroup.  It was 5 long pages of detailed instructions where anything could go wrong.  It was quite off-putting.

He's just shared a new tool that will do the job for you.  HVRemote works quite simply, you just tell it to add or remove a user's admin rights.  Well done John!

Another nice post by Kurt.  This time he briefly discusses BranchCache (I covered it a bit in my TechEd EMEA 2008 posts on day #1).  There's some good diagrams in Kurt's post to illustrate the solution.

I was just reviewing this stuff this morning on the laptop while on the train.  I checked my RSS feeds and I saw that Kurt Roggen was doing some blogging recently (and doing a nice job too).

Understanding things like VMBus, VSC's and VSP's is recommended when working with Hyper-V.  This post will teach you some of this.

What I'll add to this is that your VM's (child partitions) have a 1-1 connection to the parent partition.  This secure channel, the VMBus, is at Ring 0 and is protected by Data Execution Protection (DEP).  This is why turning this on in the BIOS is a requirement for installing Hyper-V.

Credit: Kurt Roggen.

Small Business Server 2008 and Essential Business Server 2008 may have RTM'd a while back but they've just had their launch events.  MS Ireland launched on the 13th (I'm only catching up on news after a nuts couple of weeks).  I recognise two of the people in the photo and I know one of the quoted people ... Man!  People are right about me seeming almost like an MS person.

I've used SkyDrive but really only in a limited fashion.  When speaking, I keep my presentation on the laptop, USB and on SkyDrive.  That's rescued me twice.  It has had a limited capacity but that's set to change.  It's growing to 25Gb in synch with a launch of new services from Windows Live.  You'll be able to use this expanded storage and services such as Mesh to keep your work machine, your home machine, your laptop and your Windows Mobile phone in synch.  The things of importance to me are contacts and favourites and Live will manage those for you.  Of little interest to me is the social side of stuff (Bebo me ****!) but MS appears to be going after that market as well.

Bink did a nice job of gathering a series of guides from MS on installing and migrating to EBS 2008:

• Windows Essential Business Server Installation Guide
• Migrating WSUS to Windows EBS
• Migrating Exchange Server to Windows EBS
• Migrating from Windows SBS 2003 to Windows EBS
• Windows Essential Business Server Getting Started Guide

Credit: Bink.

My calendar is filling up with events.  I'm speaking at IDC in Dublin next week about managed services.  On the non-IT front I'll be talking about my experiences in wildlife photography.  Microsoft Ireland plans to run a "Best of TechEd" event in 3 venues around Ireland in February and I'm lined up to be one of the speakers.  It also looks like I'll be doing something at Minasi Forum 2009 in Virginia Beach.

I'll briefly mention some of the things I'm really liking that I was unaware of before putting VMM into action.

One of the things I was dreading with a VHD library was disk wastage.  PSS don't like anything other than pass through disks and fixed size VHD's in production.  Without VMM I was building sysprepped dynamically expanding VHD's.  I'd store those in a shared folder.  I'd copy the image VHD to a host and then "convert" (it actually creates a new file) the VHD into a fixed size VHD which my new VM would use.

VMM is a little more clever.  It allows you to convert a disk in place.  I like that.  I'm storing my dynamic VHD's in the library.  To save more space I've compressed the library - hey, disk is money to us and disk IS NOT CHEAP!  I can build a VM and convert the disks to fixed size before powering it up.  This means I can conserve disk space in the VMM library and still build fixed size VHD's from templates without incurring nasty amounts of work.

Without Hyper-V using ISO images for the CD/DVD was a similar painful process -> copy the ISO to the Hyper-V box and load it up.  With VMM I can load the ISO into the library and it can be loaded on the VM via the VMM console over the network.

The idea of a template is different in VMM than it is in ESX ... or even any OS deployment solution ... more MS renaming!  It'll take me a while to get used to but it mightn't be a bad thing - I have to try it in anger first.  A template is a machine configuration, e.g. 1 processor, 2GB RAM, etc.  The VHD image is a totally different thing altogether.  So that 1 CPU & 2GB RAM machine description can be paired up with different OS images by the looks of it.

I'm seeing more and more how VMM makes managing multiple Hyper-V boxes easier.  It is different to ESX which I found quite natural (other than nested resource pools to be honest) but that difference isn't naturally a bad thing.

BTW, after the early issues that I sorted out (and blogged about) it's running very sweetly.  The diagram view went down very nice with the boss.  It's nice to show the people in charge where all the money went :)

I was just discussing with a colleague earlier about how big the Microsoft site is: you can't see the forest from the trees.  I've been a TechNet subscriber for over 2 years.  I haven't even looked at one of the services bundled with it.  We get access to a bunch of free Computer Based Training (CBT) courses in Microsoft E-Learning, many of which provide essential materials for MS exams.  I also was parsing through my MVP emails.  One of the perks we get as MVP's is vouchers for free E-Learning for those courses that must be paid for.  Fantastic!

I signed into the site and I could browse my free courses from TechNet.  There's some good stuff in there.  But, I prefer to read this stuff when I don't have Internet access, e.g. when I'm on the train or down the countryside on a rainy day where there's no coverage.  Not to worry, there's a E-Learning Offline Player.  You select your course materials and you download them to your laptop. 

That means I can read this stuff when I'm commuting to the office - something I'm doing more often for the next few weeks by the looks of it.  At least those 2 hours a day on the train will be put to some good use.  I need to upgrade some of my MCSE electives and start bashing out the upgrade exams from MCSE W2003 to MCITP: Server Administrator W2008.

The downside?  I hate exams.  It took me 4 years to get around to sitting down for the 2nd W2003 upgrade exam.  It wasn't because of lack of preparation.  It was purely because of dread.

Don't worry if you're not in these programs, there's loads of free stuff on Microsoft Learning.  If you're lucky enough to work for a company/organisation that has an Enterprise Agreement then you may have access to a large library of materials too.

I alluded to a second problem with VMM 2008 and Hyper-V earlier.  The issue was that after a while, a previously healthy host would change to "requires attention".  The virtualisation status would change to unknown and the agent would stop communicating.  The host refresh would fail as follows:

Error (2927)
A Hardware Management error has occurred trying to contact server server.domain.local.
(Unknown error (0x80338104))

Recommended Action
Check that WinRM is installed and running on server server.domain.local. For more information use the command "winrm helpmsg hresult".

That WinRm error translates to "access denied".  I confirmed all the networking stuff and WinRM were actually OK.

Not long after that, every VM on that Hyper-V cluster would become unmanageable in VMM.  There's a big clue that it's a VMM issue.  Hyper-V and Failover Clustering stay healthy.  The VM's are manageable in Hyper-V and run perfectly well.

Warning (13921)
Highly available virtual machine VM001 is not supported by VMM because one or more of its network adapters is not configured correctly. 

Recommended Action
Ensure that all of the virtual network adapters are either disconnected or connected to highly available virtual networks.

I'd set up PRO earlier that afternoon.  It was pretty simple.  An OpsMgr console is installed on the VMM server.  I installed PRO Tips on the OpsMgr 2007 SP1 server.  That also sets up the VMM console and the management packs on the RMS.  On the VMM server, I set up the FWDN of the OpsMgr server and the URL of the OpsMgr reporting server.  That's it!

Everything was good when I left work.  When I got home I saw those above errors had taken place an VMM thought my cluster was messed up.  OpsMgr alerted me about the status of the VM's.  Excellent!  I checked the supplied knowledge and it was more than I'd found in a day of googling.  As it turns out, my search terms sucked.  The provided expertise in the alert gave me the search term I needed and I found an excellent blog post on the issue.

I use Active Directory Group Policy restricted groups to control membership of the local administrators groups.  The VMM server was added by the agent install to the local administrators group to allow WMI and WinRM access.  My GPO would refresh after several hours and wipe out that group membership.  To fix this I reconfigured my GPO to add the VMM server to the Hyper-V host local administrators group and forced a GPO refresh on the server in question (GPUPDATE /FORCE).  I restarted WinRM (and VMM agent) on the affected host.  Finally I refreshed the host on the VMM server and the VM listings.  Everything was back to normal in just a few seconds.

Thank you PRO!

I've just configured the VMM 2008 web based portal.  It was pretty easy.  The idea is that you give people a web interface that allows them to manage VM's, their properties and "KVM" access to them via the web site.  If you provide templates and VHD's in your library (as well has prepared disk for your cluster) you can allow users to build their own VM's.  To be honest, this would be impossible to control without a cluster file system - what's to stop a user taking a 1TB LUN for a 100GB VHD?  You also are going to have trouble with restricting control over VLAN tags.  You can control VM resource consumption by using a points system, e.g. assign a score to a VM template and deduct it from a user's point allocation as they deploy machines.

However, if you restrict full access to administrators and allow KVM/power control access to VM owners then you've got a nice solution.  You'll want to do some clever group management and permissioning.

You'll need an AD group for "VMM Administrators".  Put your VMM administrator accounts/groups into that group.  For every customer there will be a group, e.g. "Cust-Group".  They will also have a user called "Cust-User".  Cust-User and VMM-Administrators are members of Cust-Group. 

Create a folder/group in the VMM console to put that customer's VM's into called "Cust".  When a VM is set up for them assign the owner of the machine as "Cust-Group". 

Set up a self service role called "Cust-Self" service and add Cust-Group to that role.  Give it the scope required, i.e. the VMM folder/group called Cust.  Give the role the required permissions over the VM's in that group, e.g. start, stop, pause & resume, remote connection and shutdown.

Now configure the portal with SSL access (simple IIS7 operation) and share the URL.  The user will log in using domain\Cust-User.  Their console will load an only show their VM's.  They will only be able to do the actions you assigned to them.

My very early experiences were positive but the wrinkles are now only being worked out.  My advice for using VMM 2008 is this: deploy it before you deploy Hyper-V.  Adding an existing Hyper-V cluster to VMM 2008 is not recommended by me.

The first thing I'm going to talk about it virtual networks.  I have an existing Hyper-V cluster.  I created 2 virtual networks (mapped to 2 NIC's) on each host.  Networking on very host was configured and named identically at every level by me.  The failover cluster validation report was a pass and VM's failed over and back while maintaining network connectivity.  Perfect!

Then I added VMM 2008 to the mix.  It sucked in the cluster and deployed it's agents.  The first problem (which I'm still trying to resolve) is a Win-RM access denied issue that appears after a host has been managed for several hours.  I've no idea why.  I've got a call open on this with MS so I hope to post a resolution at sometime soon.

The second problem is the one I'm going to talk about now.  After a while my highly available VM's started going red with a status of "unsupported cluster configuration".  I knew the cluster was OK because of my report and because of how the VM's moved OK.  I was getting an error on my VM's telling me my networking was at fault.  Anthony Crotty sent me up a link that described a scenario when this error occurs.  It wasn't identical but it did point out non-identical networking across the cluster hosts.  As far as I could see my networking was identical across all the hosts. 

There's a property box for a Hyper-V cluster in VMM 2008.  In there you'll find a networking tab to show networks that reside across all the hosts.  My 2 virtual networks weren't there.  Intriguing!  I added a test private network to host 1 and it was automatically built across the other hosts.  That's handy!

I deleted virtual network #2 and rebuilt it on host 1.  It was recreated on the other hosts and now it appeared in the cluster properties networking tab.

That leads me to this theory.  If you create virtual networks by hand (as you have to if you don't yet have VMM 2008) then there's a hidden configuration that's required by VMM managed clusters that is not created.  I'm thinking there's supposed to be a common ID in the hidden properties of the virtual network across the hosts.

So I recreated my 2 virtual networks, mapped the NIC's and configured trunking on host 1 and they were created across the hosts.  I did a quick test, luckily.  VMM 2007 may set up the virtual networks on the other hosts but it did not map the NIC or configure trunking.  I replayed those steps on the other hosts.  You have to be patient when doing this.  You may see the dialog disappear when you click OK but there's a job still running in the background to carry out the configuration.  Don't jump straight back into the dialog box expecting to see your new configuration.  Watch the properties of the host update and wait for the job to complete.

Hopefully I can post something a little later about WinRM.  I expect it'll be something stupid like the above.

One of the things that is critical to a virtualisation platform deployment is converting existing physical machines into virtual machines (P2V).  I attempted my first P2V last night and I really saw how much VMM 2008 adds to managing Hyper-V.  A lot of the work was done for me.

VMM 2008 P2V deploys a temporary agent to the physical machine (PM) to perform the conversion.  From what I've heard, this performs a VSS backup of the PM and send the data stream to be rebuilt on the VM.  It works a little like this:

• Deploy disk to your host/cluster.  In a cluster, note down the GUID of the disk being used.
• Find a nice quiet period where you can work on the PM.  I learned this the hard way when doing ESX P2V last year :-)  People can get so mad when their server's CPU and/or disk spike at 100% activity for hours on end ;-)
• Start up the wizard.  You'll need admin credentials for the PM and the name of the PM.
• Configure the VM: disk (you can set the type of VHD and make it bigger if required), CPU, RAM, power up/down and network.
• I leave the network disconnected and the machine set not to automatically power up.  This allows me to leave the PM running while testing the VM.  That's not ideal in all scenarios, e.g. converting a mail or DB server.
• Pick a host.  If using a cluster, ensure that this is the host with the disk you've provided.
• In the disk configuration, ensure that the correct drive is selected.  I found it best to refresh the disk selection dialog if using GUID drives.   Otherwise the last second check would fail.
• Once the check passes, continue with the wizard.
• Now an agent on the PM copies the PM to the new VM.  The VM exists on the cluster at this point ... the job will fail if the VMM service fails or the PM restarts so repeating the wizard will require deleting the VM.  The contents of the VM's disk will get populated over the following hours.

I left that wizard running overnight.  I'll be checking it when I get into work in a few hours.  My opinion so far is that it made managing GUID's easier than bare Hyper-V.  It's still behind Virtual Center because we don't have that single storage made possible by a cluster file system such as VMFS.  However, that will come in Windows Server 2008 R2.