Aidan Finn

If you've looked at putting Hyper-V in a cluster you might have read Jose Barreto's blog post on clustering options, viewed Dave Northey's videos demonstrating it in action or considered trying to recreate what ESX with Virtual Center does.  You'll soon see that to have failover or mobility on a per-VM basis with Hyper-V on Windows Server 2008, each VM must reside in it's on disk/LUN on your shared storage.  Windows Server 2008 doesn't have the ability (yet) to do shared file systems like that in ESX's VMFS.

You'll now think ... I can have 16 nodes in a cluster and potentially dozens of VM's in my N+1 or N+2 architecture.  Wait ... how many drive letters am I going to need?  I've already consumed A, B, C and D ... does this mean a cluster can have only 22 VM's?  This is probably something where some certain-product-fanatic gets to write some blog FUD without digging just a little deeper.  It's amazing to see how prejudice is tainting the commentary and reviews that are out there right now :-)

You have the option to use "letterless" drives in Windows Server 2008.  Instead of using a drive letter to identify the physical drive that each VM can reside on, you can use a GUID to identify the drives. 

The only question now is, how do you use these drives?  VirtuallyAware has done a post on the subject.  The hardest part of the process is getting the GUID of the LUN that you're working with.  Who really wants to type out something nasty like "fc247e42-0a5e-11dd-94db-001b785788b0"?  PowerShell helps at there as the blog post indicates. 

You'll now have a virtually unlimited set of drive identifiers that will allow your cluster to scale out to the limitations of your CPU, storage and RAM.

On a tangent, this is just another example of where PowerShell is a necessary skill, not only in PowerShell but in all new MS technologies.  I've started learning it.  It's different, that's for sure, but it's not optional any longer.

This article gives you a quick and easy to understand look under the hood so you can see how snapshots work in Hyper-V.

Credit: HyperVoria.

I have a strong dislike for auditing.  It's a time consuming process.  But you know, if you use the right systems management tools it doesn't need to be.  Microsoft's Optimised Infrastructure model and Dynamic Systems Initiative preach automation and expertise built into the network.  The latest generation of System Centre allows for this.  Microsoft released a short white paper that looks at data centre auditing.  It's not something I'd really considered until the last few months.

Network and some *NIX administrators have long used SYSLOG tools.  The idea is that all events are forwarded to a central store.  It gives a synchronised view of what is happening across a multitude of devices.  It allows for diagnostics.  But from an auditors point of view, it gives an audit trail of who did what and when.  You can get this sort of functionality going with Windows as well.  I'm not a network or *NIX admin but I'm guessing their security logs are not that different to one on a Windows box, i.e. lots of noise and they require significant time to filter through to figure out what was really going on.

System Centre Operations Manager (SCOM or OpsMgr) 2007 includes Audit Collection Services.  I first heard of ACS at TechEd Europe in Amsterdam in 2004.  It was going to be a standalone tool but after a lengthy delay it finally saw the light as a part of OpsMgr.  You can turn on ACS on your OpsMgr agents to enable centralised security logging for Windows platforms.  What makes it different to SYSLOG is that Microsoft's developers have identified the important events that illustrate what is going on and they only forward those events to the ACS database.  The ACS database is separate to the rest of the OpsMgr databases so you can permission it differently, i.e. only your auditors or security staff would have access to it if required.

I don't know if the new Cross Platform Extensions for OpsMgr will allow for ACS on Linux platforms.  I suspect that they won't.  Anyway, you're going to still need SYSLOG for your network devices.  From what I'm seeing recently, network monitoring tools (which are often freeware) seem to run and be supported best when running on Linux.  Yes, you read that on my blog ... something running best on Linux.  I am open to non-MS products!

That's great for monitoring your security activities, but that's only half of the story.  You need to build a secure and regulatory complaint infrastructure and maintain that integrity.  I knew a security consultant in Germany who spent a huge amount of time building an automated auditing tool set that dumped data into a central store and allowed for reporting.  It covered all sorts of platforms.  It was a really great idea.  But this guy was an alpha geek.  Owning and running that toolset required his level of abilities, I'm guessing.

System Center Configuration Management (SCCM or ConfigMgr) 2007 features Desired Configuration Management (DCM).  DCM allows you to use either a set of pre-built or custom made templates to audit your Microsoft network on a recurring and automated basis.  That means there's no more logging into each box to check out the configuration of the box.  Everything is automated.  You're also building that expertise into the network by using templates.  Heck, Microsoft even gives away a set of DCM packs for the products to cover regulators like SOX, FISMA, EUDPD, HIPAA and more!  Now you can just tell your auditors to run a report to see the configuration health of your network.  No more wasted admin or auditor time or complexity, e.g. delegated admin rights on servers and applications.  The DCM tool is easy enough to get your head around in order to build your own templates for auditing 3rd party or internal applications. 

If you're in a regulated market, e.g. finance, health, pharmaceuticals, etc, then you're probably required to have these sorts of controls.  If you're using System Centre then it makes sense to look into and enable these functions to make your job easier.  Sure, you may require another server and some storage but when you compare time savings VS capital costs, there's really only one logical way forward: build that expertise into the network and leverage the available automation.

We have a support contract at work for our IBM servers and storage.  The contract defines it as 24*7 with 4 hours response time.  I logged a call 24 hours ago for a failed disk.  24 hours later I get a phone call from "Droopy" who can't get me an engineer.  What?  Breach of contract (by 20 hours) is what IBM offers as an enterprise service.  I asked to speak to his manager.  "He's busy".  OK, I'll speak to his manager's manager.  "He's busy too".  Friggin muppets.  Imagine how much worse it'll be when IBM hands over their server and storage brands to Lenovo?

Anyone looking at IBM hardware - forget it.  Do yourself a favour and talk to Dell or HP. 

I released the July updates onto our network this past weekend.  I'd also deployed our new AV the previous week.  Let's just say that AV mixed with Hyper-V and followed by a reboot made for a nice mess.

I logged into the Hyper-V lab this morning to find half of my VM's were missing.  They're sitting find (but idle) on the storage.  It's just Hyper-V has "forgotten" that they ever existed.

I trawled through the Windows Event logs (Application and Service logs - Microsoft - Windows - Hyper-V-Config - Admin) and found a series of these:

Source: Hyper-V-Config

Event ID: 4096

Level: Error

The Virtual Machines configuration <big long GUID> at <path to VM> is no longer accessible: The requested operation cannot be performed on a file with a user-mapped section open. (0x800704C8)

Ok.  A bit of googling found an entry on the TechNet forums that says you need to disable scanning for the VHD's and the XML files of your VM's.  Ouch!

OK, so I did that and rebooted by lab server.  Still no dice.  Actually, Hyper-V doesn't even bother attempting to load these VM's now.  OK, I'll do what I would in any other virtualisation product; I'll open them.  Ick ... no open command.  Import?  Nope; because MS in their wisdom (!) decided that the import/export format should be different to that of a normal VM. 

So I've got a plethora of VM's that are sitting on my disk in a saved state that I cannot load up.  My only way forward is to re-add the virtual hard disks as new VM's.  This is a pain:

• I lose my saved states.
• I have to reconfigure every single VM that is missing.
• Each VM has to do the PNP dance with a "new" NIC and I have to reconfigure IPv4 addressing.
• It's just lots of work I shouldn't have to do.

I've logged a bug report with MS.  I'm open to any constructive suggestions.

I inherited a number of IBM servers with this job.  They perform a critical business service for our customers.  Luckily, the architecture we use is very fault tolerant.

Over the weekend we deployed updates in a staged manner to our production network - after testing of course.  On Sunday morning, I woke up to an email from System Center Operations Manager 2007 (gotta love it!) saying that one of the servers we patched on Saturday night was not responding to agent heartbeat requests.  Uh oh!  This was one of those IBM boxes.  We have triplicate redundancy so I knew I could let it wait until Monday morning.  To be safe, I suspended updates for the remaining production boxes.  I didn't suspect an update but I wasn't taking any chances.

I came into the data centre this morning and found the server sitting on a BIOS prompt.  Hmm.  That's not good.  It had detected a problem with the external disk storage and was waiting for administrator approval to boot up.  What?  Hello?  Note: the failure was nothing to do with the server-internal boot disks.

I checked the Direct Attached Storage (DAS) and it was all green.  I booted up the server and saw the DAS was not being connected.  I shut down the server and powered down the DAS.  I powered up the DAS and was greeted with beeping ... non-stop beeping.  The front panel now showed a chassis alert on the DAS and one of the disks in the RAID5 array was alerting as well.  Huh!?!  Why didn't it tell me this when the server already knew there was a problem?

I powered up the server.  Now it didn't prompt me.  But it did tell me the external disk was degraded.  Fine, the hardware knows there's a problem.

I logged in and found there were no hardware logs or any sort of interface into the IBM director agent.  Nothing.  Sweet F.A.  The consultants (before my time) who installed the hardware had set up an IBM director console on another box for centralised monitoring.  I logged into it and sure enough, there were no alerts.  Hold an a *beep*ing minute; the hardware knows there's a problem but the monitoring agent from the hardware vendor doesn't have a clue?

OK, maybe it was the central console at fault?  I've never trusted it.  I went on to the SCOM console but found no alerts or health degradation on the IBM Director monitors.  That made it certain in my mind, the IBM Director agent was clueless.

So here's my summary why I would recommend people to steer clear of IBM hardware in an enterprise deployment based on this little story:

1. The DAS failed to show an alert on the front panel or disk despite the server not being able to boot up because it detected a failure.
2. The IBM Director agent failed to report an incident of any kind.
3. There's no user interface to the IBM director agent on the server.
4. A failure of a single disk in a RAID5 array in a DAS caused a server not to boot up.  That's just stupid.
5. We've all heard that Lenovo are taking over the server and storage business.  My experience of them with their support was awful - A call open for around 4 months and 2 months of that with the regional director taking a personal interest.

I'm now left wondering how long I've had a failed disk on this server considering it didn't give any monitoring alert or visible notification until I reset the DAS chassis.

How would HP handle this?

1. The SIM agent would have alerted on this and shown it in the HP SIM log and in the SIM web page on the server.
2. The HP SCOM management pack for SIM would have alerted and sent all of the required/responsible administrators/operators/"business owners" a notification of the failure.
3. The disk would have shown an alert light immediately.
4. It's unlikely that the server would have been prevented from booting up unless there was a complete failure of the boot disk.
5. I would have had the storage back to a healthy state within 4 hours of opening a call with HP.

That's a very different experience and one you expect to have from enterprise class servers and storage.

EDIT

As you can guess, I was concerned with the lack of h/w monitoring that the IBM Director agent gave me.  The horrid response from the MD was that we'd have to check that the logical disks in question were present on a daily/manual presence.  Yuk!  I'd a better idea: let SCOM do the work for me.  I've created a distributed application that entails on the dependancies I can think of for this service, including the presence and health of the logical disk in question.

It was funny to see that the HP management pack allowed me to include discovered HP hardware objects but there were no classes for IBM hardware.  Come on IBM; you gotta play better with others!  Not everyone wants to buy consultancy-ware like Tivoli.

The author of CoreConfigurator has had to pull the plug on CoreConfigurator.  Like many of us, he had that awful clause in his employment contract that gives the employer ownership rights over all intellectual property he created while employed by that firm, even if he did it on his own time and at home. 

That one brought up some interesting discussions when I took my current job because some of the things I've been writing are already the property of a publisher.  My employer was able to confirm that legally the contract could not extend to my independent work at home.

The author, Guy Teverovsky, has had to hand over ownership to his now former employers.  What they'll do with the code is uncertain.  This sucks because it was a great little tool for those new to the ways of command prompt.

But, not that I condone piracy in any way, you will find this tool out and about in the wilds of the Internet if you Google hard enough.

The Virtual PC guy has done a very nice job in explaining how a VM is aware of the difference between a physical and a logical processor.

I've previously discussed how RAM is used by Hyper-V in terms of:

• The parent partition
• Hyper-V services
• Drivers
• Guest RAM allocation overhead.

I've put together an Excel spreadsheet that calculates how much RAM is consumed by a VM as you load it onto a host.  Using it is easy:

1. Specify how much RAM is in the physical host machine.
2. Add each guest VM and enter how much RAM (in GB) you want to allocate to the guest.
3. The RAM utilised by the guest is calculated and the amount remaining on the host is presented.

The numbers you need to enter are highlighted in yellow.

The formula used assumes maximum RAM overhead, i.e. the worst case scenario of 32MB for the first GB and 8MB for each GB after that on a per VM basis.  I'm also allowing 300MB in addition to the 2GB recommended as the reserve for the parent partition.  Often, this can be considered a part of the 2GB.  You can recalculate things by adding in another line item to specify driver requirements for the parent OS if you want.

There's been plenty of blog posts out there saying that there is no support for SCSI in Hyper-V.  That's not true.  What is true is this.  You can use SCSI controllers for disks but not for your boot disk.  Your boot disk must be on an IDE controller.

Some facts:

IDE

Hyper-V uses en emulated IDE controller.  This means there is a little bit of overhead in processing disk operations.  That's not so bad for lightweight VM's.  You can have two IDE controllers, each with 2 devices.  One of these is your virtual DVD drive which you should probably disconnect when you don't need it.

SCSI

Hyper-V uses a SCSI controller that is not emulated.  Instead it uses the virtual machine bus which is much faster and requires less CPU overhead. 

Setting Up VM's

How do you set up your VM's?  You have no choice about your boot disk.  You must use a disk connected to the IDE controller.  You can't move that to the SCSI controller because you cannot boot from a Hyper-V SCSI controller.  Lightweight VM's can probably put everything on one virtual disk and run on the IDE controller.

However, best practice is to separate your data/workload from your operating system.  Consider a virtual application server where the operating system is on C: and the workload is on D:.  C: will be a virtual disk on the IDE controller.  D: should be a virtual disk on a SCSI controller.  This makes the most of the underlying Hyper-V architecture and optimises CPU utilisation on the host server.

I'd recommend that you read some of the blog entries on All Topics Performance.  I found two (1 and 2) to be rather interesting.

Judging purely by all my Hyper-V posts as of late, you might think I was anti-VMware.  Far from it.  I reckon they have a great product.  It's not perfect.  In my opinion, they need to start playing nice with others and provide better end-end management of their enterprise solution from centralised management solutions.  I'm not just talking about placement of VM's (Virtual Center); I'm talking about health of hardware, health of VM's, hypervisor performance, etc.  On that they could take a page from Citrix, e.g. Presentation Server.  Pricing is something else they need to reconsider too, e.g.pay less for the product and maybe make the profits from the support or management services.

Now I've just read something rather interesting on Bink.  Some regional director is spouting that the days of the operating system will be over in 5-10 years.  Oka-ay then.  I for one, am not throwing my OS books into a fire in the back garden tonight.  Is this really how VMware are thinking?  Is this how they think they will survive the serious entry of Microsoft/Citrix into the enterprise hypervisor world?  This is not good; not good at all.

Is industry moving to the hypervisor?  Yes.  But not all machines are candidates for virtualisation.  I certainly don't see the virtual appliance replacing the operating system.  Is the operating system changing?  Definitely.  Windows Server 2008 already features component based installation.  You can get an even smaller footprint using a Core installation.  And Core isn't so scary as I found out for myself.

Credit these comments to the kook bank and don't expect to see this looneytune making a big splash in the future.

Credit: Bink.

Why is System Center Operations Manager 2007 different to everything else?  You've already heard about management packs: how they use state models instead of just traditional triggers and how they use the monitored products vendor expertise.  The other big difference is that SCOM recognises that IT is their to serve a customer.  Think of this from the ITIL or MOF point of view.  IT provides services to a customer, either someone in the same organisation or a client who subscribes for the service.  That customer doesn't care about IIS sites, disk utilisation or CPU interrupt time.  They care about the uptime and performance of their service, e.g. the user who complains about there "being no Internet" doesn't care if a network switch is dead.  Their service enables their business.  SCOM gives you the ability to model that service using a distributed application model.  Up to now, to give the customer visibility to their service was messy.

Microsoft has just released the Service Level Dashboard Management Pack for Operations Manager 2007.  This allows you to use an accelerator to present the availability and performance of the service to a customer in a more accessible manner.

You can watch a video on the subject on MSN.  There's also an executive summary on TechNet (note the MS link are mostly dead for this one so use my link).

Here's what Microsoft has to say:

"The Service Level Dashboard Management Pack for Operations Manager 2007 assists you in tracking, managing, and reporting on your line-of-business (LOB) application service level compliance. It displays a list of applications and their performance and availability against a target SLA.

The application or service is defined using the Operations Manager distributed application model. This model allows the user to define all components of the application or service that affect the health state and SLA calculation. When an application does not meet the defined performance or availability thresholds, it is placed into a warning or error state within Operations Manager. This state shows the current status of an application relative to its defined thresholds.

The Service Level Dashboard report uses the history of the state of an application to calculate the time the application was in each state over the duration of the report. Based on this information, the report derives a performance and availability percentage for the time period that the report covers".

If you have a Microsoft Enterprise Agreement or a Select Volume License agreement then you will want to read this PowerPoint.  It details changes to Microsoft's media shipping policy:

• "Beginning September 1, 2007 Microsoft Enterprise Agreement and Select Agreement Volume License customers worldwide with active media subscriptions will receive monthly product subscription kits of the most widely-used Microsoft products.
• Updates of all other products will be available either via download from the Microsoft Volume Licensing Services website or by purchasing media (CDs/DVDs) from a Microsoft Reseller.
• This change will reduce the quantity of media in shipments to create a better customer experience".

Basically, everything they produce media for, you will get in the kit.  You only have to license it (according to your license program) to use it.  No more waiting around for the media to arrive in the post after your already lengthy internal purchasing process.

Bink has reported that Microsoft will put SQL 2008 on the price list in August.  That means that SQL Server 2008 will be released pretty soon.  Microsoft delayed the release of SQL 2008 to make sure it was right.  This is pretty admirable.  They did the same for other releases, e.g. FTP publishing for Windows Server 2008 and Hyper-V.

Credit: Bink.

.I've just read about the Microsoft Offline Virtual Machine Servicing Tool.  The idea is that many organisations keep a certain number of VM's in an offline state.  Maybe they are used once in a while.  Maybe they are archived for regulatory reasons.  However, there is a chance they need to be powered up once in a while.  What is the risk that they power up and are not sufficiently secured by updates?  Are you really going to manually power them up every month to deploy updates and human resources?

Microsoft has the Virtual Machine Service Tool accelerator to take care of this for you.  It runs a servicing job (using PowerShell scripts) to power up the VM, deploy updates using either WSUS 3.0 or SCCM 2007 and then powers down the VM.

The product has recently been released by Microsoft.  It's a free download and well worth checking out if you have a limited virtual lab or large production environment that utilises MS virtualisation. 

I don't see any support for Hyper-V yet.  Maybe that's coming - I've sent in a question to find out.  There's a dependancy on VMM.  VMM 2007 only supports Virtual Server 2005 R2 SP1 so I guess we'll have to wait and see on VMM 2008 (RTM Q4 2008).

Its requirements are:

• Supported Operating Systems: Windows Server 2003 R2 (32-Bit x86); Windows Server 2003 R2 x64 editions; Windows Server 2003 Service Pack 2.
• Other Requirements: .NET Framework 2.0, .NET Framework 3.0, IIS with ASP .NET installed, Windows Remote Managment, Windows PowerShelll 1.0, Configuration Manager 2007, WSUS3.0, Virtual Machine Manager 2007 (VMM), Virtual Server 2005 R2 SP1 or higher, Windows Server 2003 R2 SP1 or higher, Active Directory, SQL Server 2005 SP1 or higher, SQL Server 2005 Express Edition (VMM only).

I've just read about some licensing changes for the SMSE CAL.  This CAL entitles you to license a physical device (and hence all the VM's on it) for the following System Center products:

• Operations Manager
• Configuration Manager
• Data Protection Manager
• Virtual Machine Manager

You're probably aware that VMM 2008 is in beta at the moment.  The System Center team blog says that it will be released in Q4 2008.  Along with this it will be included to replace VMM 2007 in the SMSE.  VMM 2008 will also be available as a standalone product.  It can make sense to use the SMSE because it works out around the price of the CAL's for two of these products.  VMM and SCOM can work very closely together offering an almost consultancy/advisor like experience for MS virtualisation (Virtual Server and Hyper-V).

It is expected that the cost of the SMSE will increase by 10-15% at this time.

Credit: Bink.

I finally got around to installing Hyper-V on Server 2008 Core at home.  I'm building a lab to do some work on SCCM 2007 SP1 and R2 and I figured I rebuild my Vista desktop with Core so I could provision more RAM for my VM's.

My desktop is a 4GB RAM "home build" that I got on Komplett back in September 2006.  Imagine trying to install ESX on that?  LOL.  One of the great perks of Hyper-V is that it will support anything that Windows Server 2008 can install on.

Here's what I did to get going:

• Enabled CPU virtualisation assistance and DEP in the BIOS.
• Installed Windows Server 2008 Enterprise x64 (Core Installation).  This takes no time at all.  It was using about 6GB of disk and around 500MB of RAM.
• Changed the regional settings:  I am in Ireland but my wireless keyboard is USA: control intl.cpl.
• Determined the NIC ID (I have a few in it): netsh interface ipv4 show interfaces.
• Set the IP address for NIC