| Aidan's profileAidan Finn - Please Go I...BlogListsNetwork | Help |
|
7/31/2008 OpsMgr 2007 Management Pack: SharePoint Monitoring ToolkitThis Solution Accelerator provides a set of System Center Ops Manager 2007 Management Packs for Windows SharePoint Services 3.0 and Microsoft Office SharePoint Server 2007 (including SP1). The Toolkit also includes user documentation for each of the Management Packs. Dublin is a Small Town1 month ago, we sent out a job offer to an Irish guy who was raised in mid-west America. He accepted the job and was excited about it. It was a permanent role (of value in a slowing economy) and would offer him the chance to get into server administration in an advanced infrastructure (it's pretty leading edge). We would expose him to lots of stuff and he was bringing some valued skills, i.e. Linux. In the meantime we made plans for him. He was going to have lots of interesting work, not just "pressing buttons". I spent time preparing his laptop, getting his accounts ready in our data centre, and planning his induction. "Heavy-D" was due in the office yesterday morning. Nothing. No sign of him. I tried to ring but his phone rang out. I mailed him to let him know that we were assuming he didn't want to join the company. Then he decided to call our MD. He claimed that he got his dates mixed up. Huh!?!? That seems like a lack of attention to detail because it was clearly printed in his employment contract. He was told to be in today at 9 sharp. 10:00am comes and goes and "Heavy-D" still didn't turn up. OK, I was done with this chump. I wasn't having some one start their job reporting to me like this. I reported it to the MD. My opinion of Heavy-D now is that he has a lack of attention to detail, is unprofessional and unreliable, i.e. I deem him to be unemployable. The only excuse is if he fell under the #72 bus. If you're not from Dublin or not experienced it then here's the crux of the story. Dublin may have 1+ million residents but it is a small town. No one in business is more than 1 or 2 degrees away from anyone else (like the Kevin Bacon game), e.g. if you want to know about someone then you ask around a little and you find someone who has worked with them, sold to them or bought from them. Ruining your reputation with one person is not a good career move because people in Dublin like to talk. Example, I had looked into "Heavy-D" through a friend. I'm now telling that friend about "Heavy-D" who'll probably tell "Heavy-D's" former workmates about the story. So, "Heavy-D", you've lost out on the chance to work on a super infrastructure. You've also gone and shot your career in Dublin. Maybe you should go click your heels like Dorothy and vamoose back from whence you came. 7/29/2008 A Particularly Odd OpsMgr 2007 Problem (And Solution)The Operations Manager 2007 agent and management server communicate with each other and perform mutual authentication using Kerberos. They're in the same forest and hence in the same Kerberos domain. But what happens if you have agents outside the forest? If you read anything from Microsoft (or the OpsMgr book I just bought) you'd be left under the impression that you must install the OpsMgr gateway. You'd then install a custom X.509 cert (requiring a cert server running on Windows Enterprise Edition) on that machine and on the OpsMgr server. There's two problems with this:
If you are persistent with your searches you will find that:
Duncan appears to be the only person to have attempted to document this process so he deserves credit for it. The MS documentation folks have done a poor job with OpsMgr, e.g. failing to cover this subject and failing to document complete management pack authoring. The instructions for setting up the CA are in the OpsMgr 2007 Security Guide and Duncan walks you through installing the agent. The only missing step is you need to install and import CA and agent certs on the OpsMgr management server(s) so that they have a means for mutual authentication with the agents. I'd been doing this successfully on servers and then I hit one server where the agent could not use the cert. I saw the following in the Operations Manager Event Log: Source: OpsMgr Connector Type: Error Event ID: 21036 The certificate specified in the registry at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Machine Settings cannot be used for authentication. The error is The credentials supplied to the package were not recognized I reissued that cert, re-imported it, re-installed the agent half a dozen times. I'd opened a call with MS (thanks to IT Pro Momentum) but the first PSS agent was not the Mae West to deal with. He kept claiming the my CA was at fault but I knew it wasn't - other agents were fine. Finally the ticket got reassigned to Brian who was a pleasure to work with. He started coming up with some new ideas straight away. The first was maybe the cert store was corrupt. I tried a fix for that (CERTUTIL -F -REPAIRSTORE MY “<thumbprint of agent cert>”) but that didn't fix the problem. Brian asked if we could look at the server together using "EasyAssist" ... it's MS's answer to WebEx or LogMeIn so they can get Remote Assistance over web friendly protocols. We poked around and saw something interesting.
We manually imported the cert into there and the certification path was still screwed. We re-imported the CA cert but it was still screwed. We re-imported the CA cert and the operations manager copy of the cert. The certification path was fine but the agent didn't appear to be using it. We re-ran MOMCERTIMPORT and the certification path was invalid again. OK ... I thought we'd try this:
Lo and behold ... it worked! In fact, it worked so well that we detected a hardware fault on the server that we hadn't known about. Sweet; OpsMgr rules! A big "Thank You" to Brian for helping out on that one. For the most part, I've always had good dealings with MS PSS agents going back to 2003. It was good to see this one being rescued so professionally. Windows 2008 Group Policy MP for OpsMgr 2007Microsoft has released a new Operations Manager 2007 management pack for managing group policy on Windows Server 2008 and Windows Server 2003. 7/28/2008 Official: Support for Operations Manager 2007 on Windows Server 2008Microsoft has just given us the green light to install OpsMgr 2007 on W2008. We've been waiting since February but we finally have support and as I mentioned earlier today, we saw the first few management packs hit the streets. It's a complicated process to be compliant before installing SCOM 2007 on Windows 2008. You have to first install 3 updates:
Then you need to install a hotfix rollup. The "Mojave Experiment"Microsoft is going to launch their fresh attempt at marketing Windows Vista tomorrow, entitled the "Mojave Experiment" (pronounced mo-have-ee - after the desert). Microsoft has faced a lot of negative press about Vista, right from the early days, e.g. 2003. The hardware requirements were pretty steep when it was launched compared to what people had bought in the previous few years. Heck, I remember reading the requirements in 2003 when we'd ordered hundreds of PC's and thinking that we might never run Vista - it required hardware that wasn't publicly available back then. When it hit the market in late 2006, there was plenty of hardware on the market that wasn't really suitable but people bought it with/for Vista and had a bad experience. Then there's the OS itself. A lot has changed. I'm not a big fan of the network management in it (I am a fan of the new network stack!). I'm also not a fan of renaming and moving things about for the sake of it. Some things just seem hardware for the sake of it. The security is locked down some. A lot of legacy applications just won't work on Vista so that's messed up organisations with large application catalogues. Comments like "give out to your suppliers" or "Use compatibility toolkits" don't go down well with those organisations because they see that as unnecessary work - XP runs just fine as is so why upgrade for what they see as an upgrade for the sake of upgrading? I think MS might have gotten things all messed up. I remember hearing the story of how MS were trying to market how "pretty" Vista is. What? Why does a corporate want to hear about pretty? When Vista was launched all we saw was the new <ALT-TAB> and stories about some granny in the USA who wanted to burn photos on her DVD drive. Why would a university or bank care about that? The home user was alienated too. The OS changed so much that old hardware was insufficient and trusted home applications or peripherals no longer worked. How's a home user expected to resolve those issues? They barely know how to use Office and print. What ended up happening is that most business consumers shrugged their shoulders and kept deploying XP. Home users complained about poor performance and old purchases not working anymore. CIO's and CEO's happen to be home users. These decision makers saw trouble at home and didn't want that experience on their networks. The jungle grapevine is powerful too. I see it all the time at social occasions when I'm asked about a prospective new PC purchase and someone pipes in about Vista being awful. Vista isn't awful, but I think it's gotten mixed up. There are some vast improvements and some things that aren't great at all. So MS is going to tackle the perception that Vista is awful. They rounded up loads of people in San Francisco who disliked Vista. They sat them down in front of a PC, asked them to try an operating system and video recorded their experience. Surprise! It was Vista all along. The videos will be played online starting from tomorrow (probably night Irish time). Hyper-V Deployment GuideMicrosoft has released a deployment guide for Hyper-V. I'll be giving it a read later today to see what it's like, assuming that meetings don't eat up my day. Operations Manager Management Packs for Windows Server 2008Finally! Microsoft has released a set of management packs that include monitoring support for Windows Server 2008. These include:
I haven't seen anything on agent support for 2008 yet. I was under the impression that a patch would be required. Hold off on deploying agents to 2008 until you read something official from MS. 7/25/2008 Windows Essentials DocumentationMicrosoft has released a bunch of document for Windows Essential Business Server:
A Forgotten Skill: ListeningIn some ways in my education and career, I've been lucky. In college we did "Communications" for two years where we were forced to do public presentations and learn about how to interact with customers, etc. I'm not saying I perfected this (because I didn't!) but I picked up a few handy tips. One of them is listening. I am a geek that does get excited about my work and I love to get involved in discussing a problem. I've found that there's times where I need to force myself to sit back and say nothing. The benefits of doing this cannot be measured. In my first job after college, I was lucky to work with some great consultants and I got to see masters in action. The best of these was one of the quietest people you'd ever meet; not exactly something you expect for a consultant that cost customers £1,000/day back in the mid 90's. I was involved in a politically sensitive project a few years ago. I was working as a consultant on a site where an implementation project had been slow to get off the ground. The project manager and the staff felt uncomfortable with the projects architecture and direction. With no knowledge of the customer I was sent in to see what I could do to help. I spent two days in a meeting with 20 or so staff members. For the first 4 or 5 hours, I did nothing but ask short quick questions, sit back and take notes. My notebook (which I take everywhere) was filling up fast. This customer was complex both in terms of infrastructure and organisation. I wrote up a summary and a general plan for how to move forward. The feedback was positive. In fact, they were genuinely interested. We ended up have a series of these meetings where we would focus on different goals. I'd kick things off and let the staff explore the issues. My input was to either steer things back on course or to steer the exploration towards new sub-issues. I was purely exploring the problems and the possibilities of potential solutions. In fact, in the meetings I talked very little at all. Most of my talking was before/after the meetings or at lunch. I'd submit a document with my findings and proposal. This would then be followed up by the staff (who were capable but relatively inexperienced with the technology in question) or some of our other consultants. The key to success was forcing myself to listen. It's amazing what the difference is between hearing and listening. I'm on both sides of that fence now. I'm a service provider and a consumer of services/goods. As a service provider I still have to listen to the market and to the individual client. I tend to work with clients who might not be experienced in what we do so I have to get quite involved in teasing out their requirements and proposing alternate/better directions for them. The key is in hearing their business and technology requirements and translating that into a platform that they can build on. My experience as a consumer (for the first time since 2005) has been interesting to say the least. For most things, I tend to be self sufficient. Firms I work for (that let me do things my way) don't need consulting skills the way that some others do - they save money and develop internal expertise. But there are times where I need specialist skills. In 2003-2005 I was lucky to work with a hardware supplier who I treated as a partner. Our sales contact was educated about their products and I got great service from them. Today, I work with a great network service provider who I can trust the same way. But not everything is smelling of roses. We're about to make a significant hardware purchase. Unlike most companies, this isn't something finite with X CPU's and Y GB's of disk; this is just a foundation which will be followed by continual purchasing. I've been leading the interaction with several hardware vendors of different types. I couldn't have been clearer about telling them each to listen and to work well with me on this. I am evaluating them to see if they are firms I can work with over the coming 3 years. It's funny because the number of competitors whittled themselves down very, very quickly. The losing competitors are ruling themselves out because they haven't read emails or listened to me in meetings/on the phone. Most salesmen seem to think that people only think in numbers. Me? That's still very important but enjoying my day at work is important too. I don't need some person wrecking my head all day long and ruining our relationship with our clients. A simple skill that requires no €2,000 training courses such as listening can be a major tool in your arsenal. I struggle myself at times with it but when I force myself, things work out much better. I'd highly recommend it to anyone that's a service provider. As a consumer, I'd recommend that you evaluate your service providers ability to listen too. 7/24/2008 Hyper-V Clusters - There Are Only 26 Letters In the AlphabetIf you've looked at putting Hyper-V in a cluster you might have read Jose Barreto's blog post on clustering options, viewed Dave Northey's videos demonstrating it in action or considered trying to recreate what ESX with Virtual Center does. You'll soon see that to have failover or mobility on a per-VM basis with Hyper-V on Windows Server 2008, each VM must reside in it's on disk/LUN on your shared storage. Windows Server 2008 doesn't have the ability (yet) to do shared file systems like that in ESX's VMFS. You'll now think ... I can have 16 nodes in a cluster and potentially dozens of VM's in my N+1 or N+2 architecture. Wait ... how many drive letters am I going to need? I've already consumed A, B, C and D ... does this mean a cluster can have only 22 VM's? This is probably something where some certain-product-fanatic gets to write some blog FUD without digging just a little deeper. It's amazing to see how prejudice is tainting the commentary and reviews that are out there right now :-) You have the option to use "letterless" drives in Windows Server 2008. Instead of using a drive letter to identify the physical drive that each VM can reside on, you can use a GUID to identify the drives. The only question now is, how do you use these drives? VirtuallyAware has done a post on the subject. The hardest part of the process is getting the GUID of the LUN that you're working with. Who really wants to type out something nasty like "fc247e42-0a5e-11dd-94db-001b785788b0"? PowerShell helps at there as the blog post indicates. You'll now have a virtually unlimited set of drive identifiers that will allow your cluster to scale out to the limitations of your CPU, storage and RAM. On a tangent, this is just another example of where PowerShell is a necessary skill, not only in PowerShell but in all new MS technologies. I've started learning it. It's different, that's for sure, but it's not optional any longer. Hyper-V Snapshots Under The HoodThis article gives you a quick and easy to understand look under the hood so you can see how snapshots work in Hyper-V. Credit: HyperVoria. 7/23/2008 Auditing Your Data CentreI have a strong dislike for auditing. It's a time consuming process. But you know, if you use the right systems management tools it doesn't need to be. Microsoft's Optimised Infrastructure model and Dynamic Systems Initiative preach automation and expertise built into the network. The latest generation of System Centre allows for this. Microsoft released a short white paper that looks at data centre auditing. It's not something I'd really considered until the last few months. Network and some *NIX administrators have long used SYSLOG tools. The idea is that all events are forwarded to a central store. It gives a synchronised view of what is happening across a multitude of devices. It allows for diagnostics. But from an auditors point of view, it gives an audit trail of who did what and when. You can get this sort of functionality going with Windows as well. I'm not a network or *NIX admin but I'm guessing their security logs are not that different to one on a Windows box, i.e. lots of noise and they require significant time to filter through to figure out what was really going on. System Centre Operations Manager (SCOM or OpsMgr) 2007 includes Audit Collection Services. I first heard of ACS at TechEd Europe in Amsterdam in 2004. It was going to be a standalone tool but after a lengthy delay it finally saw the light as a part of OpsMgr. You can turn on ACS on your OpsMgr agents to enable centralised security logging for Windows platforms. What makes it different to SYSLOG is that Microsoft's developers have identified the important events that illustrate what is going on and they only forward those events to the ACS database. The ACS database is separate to the rest of the OpsMgr databases so you can permission it differently, i.e. only your auditors or security staff would have access to it if required. I don't know if the new Cross Platform Extensions for OpsMgr will allow for ACS on Linux platforms. I suspect that they won't. Anyway, you're going to still need SYSLOG for your network devices. From what I'm seeing recently, network monitoring tools (which are often freeware) seem to run and be supported best when running on Linux. Yes, you read that on my blog ... something running best on Linux. I am open to non-MS products! That's great for monitoring your security activities, but that's only half of the story. You need to build a secure and regulatory complaint infrastructure and maintain that integrity. I knew a security consultant in Germany who spent a huge amount of time building an automated auditing tool set that dumped data into a central store and allowed for reporting. It covered all sorts of platforms. It was a really great idea. But this guy was an alpha geek. Owning and running that toolset required his level of abilities, I'm guessing. System Center Configuration Management (SCCM or ConfigMgr) 2007 features Desired Configuration Management (DCM). DCM allows you to use either a set of pre-built or custom made templates to audit your Microsoft network on a recurring and automated basis. That means there's no more logging into each box to check out the configuration of the box. Everything is automated. You're also building that expertise into the network by using templates. Heck, Microsoft even gives away a set of DCM packs for the products to cover regulators like SOX, FISMA, EUDPD, HIPAA and more! Now you can just tell your auditors to run a report to see the configuration health of your network. No more wasted admin or auditor time or complexity, e.g. delegated admin rights on servers and applications. The DCM tool is easy enough to get your head around in order to build your own templates for auditing 3rd party or internal applications. If you're in a regulated market, e.g. finance, health, pharmaceuticals, etc, then you're probably required to have these sorts of controls. If you're using System Centre then it makes sense to look into and enable these functions to make your job easier. Sure, you may require another server and some storage but when you compare time savings VS capital costs, there's really only one logical way forward: build that expertise into the network and leverage the available automation. 7/22/2008 IBM Support Sucks TooWe have a support contract at work for our IBM servers and storage. The contract defines it as 24*7 with 4 hours response time. I logged a call 24 hours ago for a failed disk. 24 hours later I get a phone call from "Droopy" who can't get me an engineer. What? Breach of contract (by 20 hours) is what IBM offers as an enterprise service. I asked to speak to his manager. "He's busy". OK, I'll speak to his manager's manager. "He's busy too". Friggin muppets. Imagine how much worse it'll be when IBM hands over their server and storage brands to Lenovo? Anyone looking at IBM hardware - forget it. Do yourself a favour and talk to Dell or HP. 7/21/2008 Beware Anti-Virus and Hyper-VI released the July updates onto our network this past weekend. I'd also deployed our new AV the previous week. Let's just say that AV mixed with Hyper-V and followed by a reboot made for a nice mess. I logged into the Hyper-V lab this morning to find half of my VM's were missing. They're sitting find (but idle) on the storage. It's just Hyper-V has "forgotten" that they ever existed. I trawled through the Windows Event logs (Application and Service logs - Microsoft - Windows - Hyper-V-Config - Admin) and found a series of these: Source: Hyper-V-Config Event ID: 4096 Level: Error The Virtual Machines configuration <big long GUID> at <path to VM> is no longer accessible: The requested operation cannot be performed on a file with a user-mapped section open. (0x800704C8) Ok. A bit of googling found an entry on the TechNet forums that says you need to disable scanning for the VHD's and the XML files of your VM's. Ouch! OK, so I did that and rebooted by lab server. Still no dice. Actually, Hyper-V doesn't even bother attempting to load these VM's now. OK, I'll do what I would in any other virtualisation product; I'll open them. Ick ... no open command. Import? Nope; because MS in their wisdom (!) decided that the import/export format should be different to that of a normal VM. So I've got a plethora of VM's that are sitting on my disk in a saved state that I cannot load up. My only way forward is to re-add the virtual hard disks as new VM's. This is a pain:
I've logged a bug report with MS. I'm open to any constructive suggestions. Why I Dislike IBM DirectorI inherited a number of IBM servers with this job. They perform a critical business service for our customers. Luckily, the architecture we use is very fault tolerant. Over the weekend we deployed updates in a staged manner to our production network - after testing of course. On Sunday morning, I woke up to an email from System Center Operations Manager 2007 (gotta love it!) saying that one of the servers we patched on Saturday night was not responding to agent heartbeat requests. Uh oh! This was one of those IBM boxes. We have triplicate redundancy so I knew I could let it wait until Monday morning. To be safe, I suspended updates for the remaining production boxes. I didn't suspect an update but I wasn't taking any chances. I came into the data centre this morning and found the server sitting on a BIOS prompt. Hmm. That's not good. It had detected a problem with the external disk storage and was waiting for administrator approval to boot up. What? Hello? Note: the failure was nothing to do with the server-internal boot disks. I checked the Direct Attached Storage (DAS) and it was all green. I booted up the server and saw the DAS was not being connected. I shut down the server and powered down the DAS. I powered up the DAS and was greeted with beeping ... non-stop beeping. The front panel now showed a chassis alert on the DAS and one of the disks in the RAID5 array was alerting as well. Huh!?! Why didn't it tell me this when the server already knew there was a problem? I powered up the server. Now it didn't prompt me. But it did tell me the external disk was degraded. Fine, the hardware knows there's a problem. I logged in and found there were no hardware logs or any sort of interface into the IBM director agent. Nothing. Sweet F.A. The consultants (before my time) who installed the hardware had set up an IBM director console on another box for centralised monitoring. I logged into it and sure enough, there were no alerts. Hold an a *beep*ing minute; the hardware knows there's a problem but the monitoring agent from the hardware vendor doesn't have a clue? OK, maybe it was the central console at fault? I've never trusted it. I went on to the SCOM console but found no alerts or health degradation on the IBM Director monitors. That made it certain in my mind, the IBM Director agent was clueless. So here's my summary why I would recommend people to steer clear of IBM hardware in an enterprise deployment based on this little story:
I'm now left wondering how long I've had a failed disk on this server considering it didn't give any monitoring alert or visible notification until I reset the DAS chassis. How would HP handle this?
That's a very different experience and one you expect to have from enterprise class servers and storage. EDIT As you can guess, I was concerned with the lack of h/w monitoring that the IBM Director agent gave me. The horrid response from the MD was that we'd have to check that the logical disks in question were present on a daily/manual presence. Yuk! I'd a better idea: let SCOM do the work for me. I've created a distributed application that entails on the dependancies I can think of for this service, including the presence and health of the logical disk in question. It was funny to see that the HP management pack allowed me to include discovered HP hardware objects but there were no classes for IBM hardware. Come on IBM; you gotta play better with others! Not everyone wants to buy consultancy-ware like Tivoli. 7/20/2008 This Sucks: CoreConfigurator Is DiscontinuedThe author of CoreConfigurator has had to pull the plug on CoreConfigurator. Like many of us, he had that awful clause in his employment contract that gives the employer ownership rights over all intellectual property he created while employed by that firm, even if he did it on his own time and at home. That one brought up some interesting discussions when I took my current job because some of the things I've been writing are already the property of a publisher. My employer was able to confirm that legally the contract could not extend to my independent work at home. The author, Guy Teverovsky, has had to hand over ownership to his now former employers. What they'll do with the code is uncertain. This sucks because it was a great little tool for those new to the ways of command prompt. But, not that I condone piracy in any way, you will find this tool out and about in the wilds of the Internet if you Google hard enough. 7/18/2008 How A Hyper-V VM Perceives Logical Processors (Cores)The Virtual PC guy has done a very nice job in explaining how a VM is aware of the difference between a physical and a logical processor. Hyper-V RAM CalculatorI've previously discussed how RAM is used by Hyper-V in terms of:
I've put together an Excel spreadsheet that calculates how much RAM is consumed by a VM as you load it onto a host. Using it is easy:
The numbers you need to enter are highlighted in yellow. The formula used assumes maximum RAM overhead, i.e. the worst case scenario of 32MB for the first GB and 8MB for each GB after that on a per VM basis. I'm also allowing 300MB in addition to the 2GB recommended as the reserve for the parent partition. Often, this can be considered a part of the 2GB. You can recalculate things by adding in another line item to specify driver requirements for the parent OS if you want. EDIT: I've done some testing on hosts with 32GB RAM and the theory seems to match the practice. 7/17/2008 Hyper-V Controllers: IDE or SCSI?There's been plenty of blog posts out there saying that there is no support for SCSI in Hyper-V. That's not true. What is true is this. You can use SCSI controllers for disks but not for your boot disk. Your boot disk must be on an IDE controller. When using emulated storage controllers, i.e. no integration components then IDE is slower than SCSI. However, there is no discernable difference between SCSI and IDE when using sythentic drivers, i.e. integration components or VM additions. Setting Up VM's How do you set up your VM's? You have no choice about your boot disk. You must use a disk connected to the IDE controller. You can't move that to the SCSI controller because you cannot boot from a Hyper-V SCSI controller. Lightweight VM's can probably put everything on one virtual disk and run on the IDE controller. However, best practice is to separate your data/workload from your operating system. Consider a virtual application server where the operating system is on C: and the workload is on D:. C: will be a virtual disk on the IDE controller. D: should be a virtual disk on a SCSI controller if you don't have integration components. This makes the most of the underlying Hyper-V architecture and optimises CPU utilisation on the host server. However, if you have integration components then it makes no difference whether you use SCSI or IDE for the workload disk. What really makes a difference is the underlying physical storage and the types of VHD that you use. Passthrough disks are physical speed. Fixed Sized VHD currently get to within 6% of the speed of the underlying physical LUN, assuming you have 1 VHD per LUN. Dynamic and Differencing VHD's have great impacts on performance. |
|
|