Aidan Finn: Commentary

A Forgotten Skill: Listening

Fri, 25 Jul 2008 08:58:47 GMT

In some ways in my education and career, I've been lucky. In college we did "Communications" for two years where we were forced to do public presentations and learn about how to interact with customers, etc. I'm not saying I perfected this (because I didn't!) but I picked up a few handy tips. One of them is listening. I am a geek that does get excited about my work and I love to get involved in discussing a problem. I've found that there's times where I need to force myself to sit back and say nothing. The benefits of doing this cannot be measured. In my first job after college, I was lucky to work with some great consultants and I got to see masters in action. The best of these was one of the quietest people you'd ever meet; not exactly something you expect for a consultant that cost customers £1,000/day back in the mid 90's.

I was involved in a politically sensitive project a few years ago. I was working as a consultant on a site where an implementation project had been slow to get off the ground. The project manager and the staff felt uncomfortable with the projects architecture and direction. With no knowledge of the customer I was sent in to see what I could do to help. I spent two days in a meeting with 20 or so staff members. For the first 4 or 5 hours, I did nothing but ask short quick questions, sit back and take notes. My notebook (which I take everywhere) was filling up fast. This customer was complex both in terms of infrastructure and organisation. I wrote up a summary and a general plan for how to move forward. The feedback was positive. In fact, they were genuinely interested. We ended up have a series of these meetings where we would focus on different goals. I'd kick things off and let the staff explore the issues. My input was to either steer things back on course or to steer the exploration towards new sub-issues. I was purely exploring the problems and the possibilities of potential solutions. In fact, in the meetings I talked very little at all. Most of my talking was before/after the meetings or at lunch. I'd submit a document with my findings and proposal. This would then be followed up by the staff (who were capable but relatively inexperienced with the technology in question) or some of our other consultants.

The key to success was forcing myself to listen. It's amazing what the difference is between hearing and listening.

I'm on both sides of that fence now. I'm a service provider and a consumer of services/goods. As a service provider I still have to listen to the market and to the individual client. I tend to work with clients who might not be experienced in what we do so I have to get quite involved in teasing out their requirements and proposing alternate/better directions for them. The key is in hearing their business and technology requirements and translating that into a platform that they can build on.

My experience as a consumer (for the first time since 2005) has been interesting to say the least. For most things, I tend to be self sufficient. Firms I work for (that let me do things my way) don't need consulting skills the way that some others do - they save money and develop internal expertise. But there are times where I need specialist skills. In 2003-2005 I was lucky to work with a hardware supplier who I treated as a partner. Our sales contact was educated about their products and I got great service from them. Today, I work with a great network service provider who I can trust the same way.

But not everything is smelling of roses. We're about to make a significant hardware purchase. Unlike most companies, this isn't something finite with X CPU's and Y GB's of disk; this is just a foundation which will be followed by continual purchasing. I've been leading the interaction with several hardware vendors of different types. I couldn't have been clearer about telling them each to listen and to work well with me on this. I am evaluating them to see if they are firms I can work with over the coming 3 years. It's funny because the number of competitors whittled themselves down very, very quickly. The losing competitors are ruling themselves out because they haven't read emails or listened to me in meetings/on the phone. Most salesmen seem to think that people only think in numbers. Me? That's still very important but enjoying my day at work is important too. I don't need some person wrecking my head all day long and ruining our relationship with our clients.

A simple skill that requires no €2,000 training courses such as listening can be a major tool in your arsenal. I struggle myself at times with it but when I force myself, things work out much better. I'd highly recommend it to anyone that's a service provider. As a consumer, I'd recommend that you evaluate your service providers ability to listen too.

IBM Support Sucks Too

Tue, 22 Jul 2008 11:29:34 GMT

We have a support contract at work for our IBM servers and storage. The contract defines it as 24*7 with 4 hours response time. I logged a call 24 hours ago for a failed disk. 24 hours later I get a phone call from "Droopy" who can't get me an engineer. What? Breach of contract (by 20 hours) is what IBM offers as an enterprise service. I asked to speak to his manager. "He's busy". OK, I'll speak to his manager's manager. "He's busy too". Friggin muppets. Imagine how much worse it'll be when IBM hands over their server and storage brands to Lenovo?

Anyone looking at IBM hardware - forget it. Do yourself a favour and talk to Dell or HP.

Why I Dislike IBM Director

Mon, 21 Jul 2008 09:42:52 GMT

I inherited a number of IBM servers with this job. They perform a critical business service for our customers. Luckily, the architecture we use is very fault tolerant.

Over the weekend we deployed updates in a staged manner to our production network - after testing of course. On Sunday morning, I woke up to an email from System Center Operations Manager 2007 (gotta love it!) saying that one of the servers we patched on Saturday night was not responding to agent heartbeat requests. Uh oh! This was one of those IBM boxes. We have triplicate redundancy so I knew I could let it wait until Monday morning. To be safe, I suspended updates for the remaining production boxes. I didn't suspect an update but I wasn't taking any chances.

I came into the data centre this morning and found the server sitting on a BIOS prompt. Hmm. That's not good. It had detected a problem with the external disk storage and was waiting for administrator approval to boot up. What? Hello? Note: the failure was nothing to do with the server-internal boot disks.

I checked the Direct Attached Storage (DAS) and it was all green. I booted up the server and saw the DAS was not being connected. I shut down the server and powered down the DAS. I powered up the DAS and was greeted with beeping ... non-stop beeping. The front panel now showed a chassis alert on the DAS and one of the disks in the RAID5 array was alerting as well. Huh!?! Why didn't it tell me this when the server already knew there was a problem?

I powered up the server. Now it didn't prompt me. But it did tell me the external disk was degraded. Fine, the hardware knows there's a problem.

I logged in and found there were no hardware logs or any sort of interface into the IBM director agent. Nothing. Sweet F.A. The consultants (before my time) who installed the hardware had set up an IBM director console on another box for centralised monitoring. I logged into it and sure enough, there were no alerts. Hold an a *beep*ing minute; the hardware knows there's a problem but the monitoring agent from the hardware vendor doesn't have a clue?

OK, maybe it was the central console at fault? I've never trusted it. I went on to the SCOM console but found no alerts or health degradation on the IBM Director monitors. That made it certain in my mind, the IBM Director agent was clueless.

So here's my summary why I would recommend people to steer clear of IBM hardware in an enterprise deployment based on this little story:

The DAS failed to show an alert on the front panel or disk despite the server not being able to boot up because it detected a failure.
The IBM Director agent failed to report an incident of any kind.
There's no user interface to the IBM director agent on the server.
A failure of a single disk in a RAID5 array in a DAS caused a server not to boot up. That's just stupid.
We've all heard that Lenovo are taking over the server and storage business. My experience of them with their support was awful - A call open for around 4 months and 2 months of that with the regional director taking a personal interest.

I'm now left wondering how long I've had a failed disk on this server considering it didn't give any monitoring alert or visible notification until I reset the DAS chassis.

How would HP handle this?

The SIM agent would have alerted on this and shown it in the HP SIM log and in the SIM web page on the server.
The HP SCOM management pack for SIM would have alerted and sent all of the required/responsible administrators/operators/"business owners" a notification of the failure.
The disk would have shown an alert light immediately.
It's unlikely that the server would have been prevented from booting up unless there was a complete failure of the boot disk.
I would have had the storage back to a healthy state within 4 hours of opening a call with HP.

That's a very different experience and one you expect to have from enterprise class servers and storage.

EDIT

As you can guess, I was concerned with the lack of h/w monitoring that the IBM Director agent gave me. The horrid response from the MD was that we'd have to check that the logical disks in question were present on a daily/manual presence. Yuk! I'd a better idea: let SCOM do the work for me. I've created a distributed application that entails on the dependancies I can think of for this service, including the presence and health of the logical disk in question.

It was funny to see that the HP management pack allowed me to include discovered HP hardware objects but there were no classes for IBM hardware. Come on IBM; you gotta play better with others! Not everyone wants to buy consultancy-ware like Tivoli.

Are VMware Bonkers?

Wed, 16 Jul 2008 13:52:20 GMT

Judging purely by all my Hyper-V posts as of late, you might think I was anti-VMware. Far from it. I reckon they have a great product. It's not perfect. In my opinion, they need to start playing nice with others and provide better end-end management of their enterprise solution from centralised management solutions. I'm not just talking about placement of VM's (Virtual Center); I'm talking about health of hardware, health of VM's, hypervisor performance, etc. On that they could take a page from Citrix, e.g. Presentation Server. Pricing is something else they need to reconsider too, e.g.pay less for the product and maybe make the profits from the support or management services.

Now I've just read something rather interesting on Bink. Some regional director is spouting that the days of the operating system will be over in 5-10 years. Oka-ay then. I for one, am not throwing my OS books into a fire in the back garden tonight. Is this really how VMware are thinking? Is this how they think they will survive the serious entry of Microsoft/Citrix into the enterprise hypervisor world? This is not good; not good at all.

Is industry moving to the hypervisor? Yes. But not all machines are candidates for virtualisation. I certainly don't see the virtual appliance replacing the operating system. Is the operating system changing? Definitely. Windows Server 2008 already features component based installation. You can get an even smaller footprint using a Core installation. And Core isn't so scary as I found out for myself.

Credit these comments to the kook bank and don't expect to see this looneytune making a big splash in the future.

Credit: Bink.

Ramsay's IT Nightmares Revisited

Mon, 07 Jul 2008 10:23:10 GMT

Way, way back I talked about the parallels between running a restaurant and running an IT infrastructure/business. I'd been watching Ramsay's Kitchen Nightmares and everything Gordon Ramsay was saying rang true to me.

In my original post I pointed out the following as being keys for a successful IT implementation or business:

Ingredients: Use the best products that are suitable for your business. Don't just look for what is cheap or what you've been comfortable with in the past. Anticipate your future growth and requirements. Engineer flexibility. Learn from your mistakes. And don't accept mediocrity or down right awful software. You've also got to find the correct blend of ingredients. Find solutions that work together as easily as possible. In my college software engineering classes we learned that sometimes buying in the more expensive off-the-shelf solution that did the job was often cheaper over the long term than developing internally or reconfiguring something cheaper. Be open to those alternatives.
Good communication: This is a 3 way process including the owner/management, staff and the customer. The business needs to listen to the customer to know what service to provide. Don't just expect the mountain to come to you. It doesn't work like that for us mere mortals. The owner/management must communicate their vision to the staff. And the owner/management needs to listen to the staff because they are the eyes and ears of the business. If the business has recruited well, then they have hired experts. Use that internal expertise and develop the business using well founded knowledge instead of pipe dreams founded on misinterpretations of breakfast seminar.
Keep it simple: I'm all for being creative. That little script here or a scheduled job there can be the difference between being OK and looking like a genius to your customer. Building an enterprise out of a cobweb of that is pure nuts. Imagine an administrator planning on managing many hundreds of mission critical Windows servers via scripts?!?!? Talk about a nightmare. I can install SCOM to do that in 3 days and have a 100% completely managed infrastructure with reporting, in-depth control and trust in the solution. Sometimes doing the MacGuyver thing is cool. Sometimes you've got to step back and thing bigger and simpler. It's almost like a state of mind where you let your focus drift back from the detail while keeping it in vision.
The customer: There is no business if there is no customer. That customer might be the business you work for - they can "leave you" by outsourcing your job if you don't cut the mustard. And they still might just do that if management has a brainfart to cut costs, e.g. a company laying off 15 staff and replacing them with 77 consultants :-). It might be a traditional customer who will leave you or never sign a contract if what you provide isn't what they want or need.

I'm going to add a couple of more bullet points to this:

The owner/management: The latest series of Kitchen Nightmares showed how a restaurant can fail if the owner is not up to the job. This applies equally in IT. I've worked for a few companies in the last few years and I've seen how the owner/management can steer a boat towards the inevitable fall over a waterfall. Whether it was those who locked themselves away in an office and surfing the web and chatting with friends about pipe dreams, those who think they know everything there is to know and put their fates in the hands of a consulting firm that's being screwing up and ripping them off for years while ignoring advice to the contrary or those companies that split IT into competing factions where work is not done because it falls between the cracks. One thing that was common was that their IT infrastructures were 100% s***. I was brought in and either given nothing to do or they didn't want to listen. Unless the captain wants to work, the boat will never reverse course. It's more than just talking. You've got to get down and get your hands dirty. Decisions need to be made. Work must be done. Changes need to start at the top. Sorry! I've also had the opposite experience where the owner/management are completely committed and passionate about the success of the enterprise. I'm lucky to be there right now. I enjoy starting work in the morning for the first time since 2005. That energy feeds from the top through the staff.
Staff: You must recruit well. I guess I see three types of administrator/engineer. There's the 10:00-16:00 person who comes in to to a job. They don't care about the job. They have no passion. They don't learn. Anything new is work for consultants or contractors. Anyway, "I just want to be doing XYSTSJ instead of this. This is just to pay the bills". Those folks are in the vast majority in the business. They produce s*** and the infrastructure is s***. There is the hard slogger. This is the person who comes in 100% committed to working their best. They want to learn and they are good, honest people. There only downfall is that despite their great work ethic, things can be done a little bit more efficiently. Then there is the lazy admin. Lazy; isn't that bad? Not necessarily. I class myself as a lazy admin. I'd rather have someone or something else do the hard work for me. Such as? Why not let the network look after itself? It is possible, you know. It's not just marketing. I've done the Dynamic Systems Initiative and Optimised Infrastructure thing before. A team of 3 of us ran 170+ globally located servers and were doing 3 hours a day of troubleshooting. We could focus the rest of the time on more interesting work and on projects that added value to the competitiveness of the business. In this case, being "lazy" was a good thing.
Passion/Commitment: Being in this business mean spending more time than just 7.5 hours a day a the office. To be the best you've got to be flexible and work that little bit more. All of the alpha geeks I know read IT blogs, books, etc after work. They go to evening seminars. The best owners/managers I've worked for make personal sacrifices for the sake of the business and their staff, e.g. coming in on that evening to help out an engineer who is working on a severity 1 issue or being in the office when there is a maintenance weekend, even if they have nothing to do. This is a key to success.

So that's what I've learned from the restaurant business over the last series of Ramsays' show and how it applies to our business. I just found a YouTube video where he gives 5 keys to success. They seem to apply to our business from what I've seen over the last few years:

Passion: Everything you do as a leader influences the staff.
Commitment: Don't get into business if you're seeing it as a hobby. This isn't a way to get yourself away playing golf, cruising around town in a fast car or playing Playstation all day long. This is work, not just for your employees but also for yourself.
Competition: Know them. Strengths, weaknesses opportunities and threats - good old SWOT analysis from the marketing classes back in college. Knowing your competition means you can beat them. The best American Football players spend more time studying their competition in film rooms probably more than they'll be in the gym or on the practice field. Be able to offer something more, better or different.
Creative Input: Don't just rest on your laurels. Always look for something new. Invest time and money in R&D. If you don't, you will stagnate and your competition will leap ahead of you.
The Customer is king: I've seen a company in a dominant position become the most hated company in their industry. Awful sales ("sell them everything and we'll break the bad news to them once the contract is signed"), atrocious support, failing operations and a laissez faire attitude from management turned off the market from them. Their name became synonymous with everything that you could do wrong. Ramsay makes an intersting point. Some chefs cook for other chefs. That's wrong because you're forgetting to cook for taste. In our business, don't do something because it's cool or it's geeky. Do something because it makes your job easier or because it adds something of value for the customer.

Those who know me will find it a little funny that I find so much in common with a chef who's famous for having a bad temper. I don't see any similarities at all :-)