Busy Project Time.........

Just when you think it's all going to be quiet and maybe it will be a good time to get those niggling little tasks out of the way and to be able to sit down and write some decent blog articles someone comes up with the idea of decommissioning a server room to save on power. So now I'm involved in a project that requires the relocation of about 4TB of data to another filer, including updating and moving the servers that use the filer data....

Yes, it's going to be a busy few months.

And a project manager just asked me if I needed any help installing IIS...... Sometimes I'd rather be doing anything else than working in IT.

Some DNS Tips

Several times in just the past week I've had to deal with DNS entries that have made things a touch more painful than they should have been so I thought it might be time for me to jot down a few notes on how DNS should be configured to save IS people's sanity!

First up the DNS servers themselves. You should always have a primary and secondary which generally, speaking are two different DNS servers at your ISP's location. If two are not available you should consider switching ISP's. Personally, I use three. Two from my ISP and one from OpenDNS. This way, should the ISP change for any reason and/or should access be denied to the ISP's DNS servers I've got a third, totally separate service available to me.

Next up, A records. These should always point to the IP address of the server in question and they should always use the hostname of the server. Sure, this can lead to some unfriendly names but it's really handy to know the proper hostname of the server. If you want to use something 'pretty' then use CNames. When you create the A record make sure the PTR record is also created in the reverse look up zone. This way, when you are trying to work out what physical server a CName is all you have to do is a reverse lookup against the IP address.

MX Records should also have two internal/DMZ based mail servers which they can deliver to and a third at the ISP which can retry delivery to your internal servers at a later date.

These are simple tips and they (or variants of them) can be found as best practice advice for standard DNS configurations.

Understanding your environment

A practical demonstration of why understanding your environment is vital occurred a few evenings ago when some NetApp filer\domino work went wrong. A little bit of background first, domino data is stored on a NetApp filer which is shared using nfs. This is mounted by the domino server and it all (most of the time) works.

For some reason this particular server running Domino (let's call him Bob) was showing high i/o stats, although the server itself was responding fine. The filer (Nutkins) wasn't reporting any problems but it was deemed that Nutkins had to be at fault. There are a lot of connections to Nutkins after all and in fairness the mount point is living in an aggregate that is unbalanced in terms of i/o profile so the decision was made to create a new aggregate decided for Bob. Simple enough to do. For those not filer aware an aggregate is a collection of physical disks. In giving Bob his own aggregate it dedicated 8 spindles to the Domino data. More than enough to remove any i/o bottleneck.

Now, Nutkins itself has a very cool piece of technology called snapmirror. A snapmirror was duly setup and Nutkins began copying the data to its new home.

So, the big evening arrives. The paperwork is signed (in blood, naturally). The changes authorised, the servers poised....... A hush descends and the commands to stop Domino are typed into Bob......... and Domino promptly hangs.

Red flag 1 - when a manager says "oh, it always does that. Just issue kill -9 and everything will be fine, well except that a few databses might be corrupt" it's probably time to start worrying. However, the final snapmirror is initiated and the last 140mb of changes are copied (in 22 seconds no less, not even enough time to get a cup of tea). The snapmirror is then quiesed and broken. This makes the destination for the snapmirror writable. Over to the unix admin and a few key clicks later the export is mounted and Bob was started.........
Or not. Seems that a small fact was missed. Bob not only has data stored on Nutkins but also has a local directory for crash dump logs.

Red-flag 2 - when Bob's admin doesn't know the configuration of Bob's setup it is probably time to start panicking. Anyway, a tappety-tap of the keyboard and the directory is created. Oh, lets stop and start Bob hoping red flag 1 doesn't pop up. Mr. Unix issues the command and on the screen "server shutdown. Bob_stop not found". Ok, so did it shut down or not? Ps -ef | grep lotus and nope, nothing running. Red flag 1 avoided! So, start Bob and..... Nothing. Not happy. Hmmm. Time to fail back, something isn't understood\not working.. So Mr. Unix does his stuff and...... No Bob. Seems red flag 1 corrupted the data then the final snapmirror copied corrupt data. Also seems that the shutdown script has at least one bug in it which causes a loop to fail when the script is executed.

Anyway, to cut a long story short we backed out and made the change a few days later. There are several lessons learnt here mostly revolving around documentation, standarisation and knowing your environment. I'll leave it as an excercise to the reader to work out the rest!

AD Find

AD Find is the second of the two tools I managed to find in the same week. This little tool weighs in at just 700K for the download and about 2mb for the actual file. This tool does exactly what it says, it finds things in Active Directory. The clever part about it is it's possible to say exactly what you want to get back and the format it should be in.
As an example, a few weeks back I had the issue with Bindview not liking non-ASCII characters.

Now, the version of Bindview that's being used where I work is a very old NT4 only aware application which means it will update the SAMAccountName attribute but not the display name.

This isn't a problem as there is a workflow from an HR application which deals with all of that, all bindivew should be doing is delegated group permissions (and yes, I know it's much easier in AD but thats a war story for another time).

Anyway, I was curious to know how many SAMAccountNames didn't match up with display names so I used ADFind to display the CN, Samaccountname, mail, firstname and lastname fields in a CSV format which could then be processed by a filer in Excel. Much quicker than messing around with the native Active Directory tools.

AD Explorer from Sysinternals

Sometimes it's possible to stumble upon a tool and wonder just how you would have gotten a task accomplished without it. Last week I had the good fortune to stumble upon two such applications right at the time when I needed them most. I did consider buying a lottery ticket that evening!

The first one is AD Explorer and it's from sysinternals and it's exactly what it says, a explorer tool for Active Directory. It allows viewing, searching and editing of the AD in ways that are far superior to Active Directory Users and Computers. I suspect the only thing that AD users and computers can do (or do better) that this tool cannot are password changes, logon hour restrictions and limiting logon ID's to specific computers.

One very nice feature this tool has is the ability to take a snapshot of an Active Directory and compare it to another snapshot. Doing this shows just how many changes occur in the AD in just a few days. It's also a great way to see how many differences accumulate between your production and test active directory environments.

Overall this is a fantastic tool and one I'll be using when the MS technotes require delving into some obscure key via ADSIEdit. I'll also be using it in place of tools like Softerras LDAP browser unless I need to something LDAP specfic.

Why Total Cost of Ownership is a fallacy

If I have one more potential supplier try and sell me something on the lie that it will "reduce TCO" I will not only scream but I will beat them to death with a CAT 5 cable.

Total Cost of Ownership (TCO) is one of those almost unmeasurable values that seems to have pride of place in the salespersons portfolio. How do they KNOW a new system (with it's associated equipment, licensing and training costs) will work out cheaper than the old one?
The idea is that newer systems have better support so rather than training someone in an older system and maybe having to buy in more expensive skills more legacy systems it works out cheaper to upgrade or replace with the latest model.

I don't disagree that for some systems which are truly legacy such the old DOS or OS/2 application may well work out cheaper in the long run but the one thing that will truly reduce TCO?

Understand your systems.

Take time to test and document the fixes.

Use your call logging system as a knowledge base.

These three tips alone will truly reduce TCO.

VMWare course

For much of this week I'm on a VMWare course for the second half of my VMWare training. This part of the course is titled Deploy, Secure and Analyse. The course itself is to prepare me for a server consolidation project that the company I work for is kicking off.
The project invovles several VMWare clusters, a Hitachi SAN and blades. Lots of flashing lights and new technology to break support.