Google Search » Database Underground | Sean McCown » July 2006

July 25, 2006 | Comments: (0)

Are Your DBAs Working for You?

TAGS: General Database

OK, for a while now I've been on this kick about how companies ought to look out for their people, and recognize the talent they have and keep and nurture it. So, this one is for the IT managers out there, but DBAs should read it too just to know what they're being graded against.

Here's the problem:
You have a DBA, or even a team of DBAs on staff. Your main production system goes down, and it's now time for them to spring into action. After assessing the problem, they decide that the DB is completely shot and needs to be restored from backup.

Here's your quiz:
Are they telling the truth, or do they just not really know what they're doing? Could the system have been brought back online much faster? Was it really necessary to restore and possibly lose the latest transactions? Did you lose any transaction, and if so, is it because you just couldn't help it, or because they didn't set it up correctly to recover from disaster?

There are other questions to ask yourself, but you get the idea.

Now, most of you probably don't know the answers to any of these questions, and it's partially your fault. More on that in a minute.

In the example I gave above, it could have really gone either way. Here are just some of the things that could have happened.

1. DB could be in suspect mode, in which case it could be as simple as a drive outage, but it could easily be that the DB outgrew the drive in the middle of a huge transaction and there wasn't enough space to rollback. Does this require a restore? No.

2. DB could have corruption. In this case, it really depends on the level and cause of the corruption. It also depends on how long it's been corrupt. There are ways to fix the corruption both with and without data loss. However, in this case restoring could actually make the problem worse because you could restore the corruption instead of fixing it.

3. If you restore for whatever reason, did your DBAs think ahead and test the backups? Did they leave you in a position to be able to backup the tail of the log so you don't lose transactions?

I think I've made my point.

So how is it partially your fault that you're in a bad position?
The hows and whys are really important, and it's important for you to not just accept your DBA's word. When there's a situation, ask for details. Ask them what the root cause of the problem was, and how it could have been avoided. Get them to explain everything to you, and if you don't understand, don't stop until you do. I do a post mortem after every outage. I've got a form I use that outlines the players, when it happened, when the DBAs were informed, when it got fixed, what the cause was, how it can be avoided, etc. This is very important in understanding your environment and can help you recognize major shortcomings in your plan.

You should also insist that your DBAs go through practice runs of recovering from a disaster. Make them restore the system from the ground up. Make them test their backups and their restore scripts. It's not only necessary for seeing where you stand as a company, but it's also to see where your DBAs stand in their professional development.

I find that most DBAs don't push themselves to study or practice to help prepare for their next disaster. It's important to familiarize yourself with the techniques involved in recovering systems, and if you don't know them, ask someone. There are plenty of forums, MVPs, books, etc out there to get this knowledge from.

Hell, one of my favorite methods of all time of learning techniques I don't know is to surf the newsgroups. There are newsgroups for every DB out there, and if you just cruise them a couple times a week, or for 30mins/day, you'll be amazed at what you'll come up with. Somebody posts with a question you've never heard... just pretend like it's happening in your environment, and research it and try to find the answer yourself. You can also see what the other users are saying about it. You can't think of every situation yourself, so the newsgroups come in handy for giving you ideas. So once a day I would go into a newsgroup and pick a thread. I would then pretend that it was something someone had asked me at work, and I would try to solve it myself. You'll be surprised how helpful that is.

Ok, that's all I've got this time. I hope I helped some of you managers realize where your risk lies and how to assess how good of a job your DBAs are doing. Push them to become better, and make them explain their actions. I find that quite often DBAs make decisions without clear goals in mind. You have to know what your goal is before you know if you've achieved it.

OK, this time I'm really done... honest.

Posted by Sean McCown on July 25, 2006 08:15 PM

July 24, 2006 | Comments: (0)

A Glimpse into My Mailbox

TAGS: General Database

You know, one of the things that has really surprised me is how many people are tired of the same things I am. Since I started the series on how companies treat their IT staff, my mailbox has been full of people urging me on and telling me their horror stories. It's been well received to say the least.

I just wanted to say thanks to all of you who have written me and supported me during this series. A lot of you are right... I do seem to be the only one sometimes who fights these kinds of fights. I remember at the editor's workshop here at the magazine earlier this year, they joked about changing my title to 'Database Lightening Rod'. I don't mind being the one to champion these causes because I really believe in them.

I was also surprised to walk around TechED and find out how many people read my blogs. It's always nice to find out you're being read, but I had no idea it was as extensive as it is. I'm not bragging, I'm actually going somewhere with this.

If you all remember, right before TechED I posted a blog called Benchmark Chicanery. I wasn't sure how well it would be received, but I wrote it anyway, fully expecting some blowback from the industry. The problem was I believed what I wrote was true, and I felt you guys has the right to know what was actually going on.
Well, at TechED, I had several vendors, SQL Server MVPs, Microsoft developers and managers, and even other journalists come up to me and tell me how pleased they were with that post. One MVP told me that he had read that press release from Idera and was very impressed. The trouble is he didn't look past it, or bother to do any of the math. When he read my blog post on the topic, he went back and took a closer look and found I was right. While all these guys were praising my efforts, I tried to get them to join me in their own blogs so I at least wouldn't be the only one out there speaking out against that press release. They all said they'd love to, but just couldn't. But they were all glad I did... whatever guys.

Anyway, I've gotten some really nice email lately, and I just wanted to share some of this with you guys before I get back to business.

Oh, and btw... I may have an update on the benchmark chicanery piece. I've got a lead and if it pans out I'll let you guys know right away.

Posted by Sean McCown on July 24, 2006 07:19 PM

July 17, 2006 | Comments: (0)

Can YOU Pick a Good DBA?

TAGS: General Database

I've tried to get off this topic, but I keep getting dragged back in. Our own Tom Yager wrote a few weeks ago about companies not respecting the IT talent they have in-house, and it got me going again.
Here's a quick list of my recent posts that are on the same topic:
Outsourcing IT Staff
Database Apprenticeship
Important Apprenticeships

It is a problem that companies don't recognize their key talent. My last company had several excellent members in their IT staff and systematically dropped each one of them for the cheaper, less experienced models. I think the problem is two-fold. First, companies just don't care. Most companies still view their IT staff with the same eye they use for their janitorial staff. They really don't see the difference. To them, IT is simply a fancy form of record keeping that pretty much anybody could do. One DBA is just as good as the next. And if you get one that's certified, then he's surely better than all the rest. I originally started getting certifications because there were several companies who wouldn't even look at me unless I had the MCDBA after my name. This is a lot like the company about 3yrs ago who wouldn't hire me because my degree was in French linguistics instead of IS. The sad part is the hiring manager wanted me. I outshined every candidate they had, but he couldn't get the company to lift the restriction that every IT job had to have a degree in IS.... umm, OK. I even had those nice little Microsoft letters after my name. Other companies still won't hire you unless you have any kind of college degree. This is the one I'd like someone to explain to me. Like I said, my degree is in French linguistics. Can any of you guess how many times that skillset comes up in my DBA work? Go ahead, take a guess. I have a friend who does very well for himself in IT as a Cisco guy. His degree is actually in horticulture. And yet, he tells me that often times, his degree is the first thing that companies look at.

So it appears that ANY degree is all that most companies need. Because a degree in horticulture certainly means that you're qualified for a Cisco job. Just like French linguistics easily qualifies me for DBA work. This is the same mentality that made them think Michael Brown would be a good choice to lead FEMA. I mean, why not? The guy did after all run horse shows.

OK, getting away from degrees for a minute, let's look at certifications.
I'm going to tell you something that may come as a BIG surprise, but I want you to listen. This is especially for hiring managers and CIOs. Certifications don't mean a thing. In fact, they're practically worthless.

Let me tell you something about the MCDBA cert. It's so incredibly easy to cheat it's not even funny. There are several sites out there where you can download every question on the exam and ace it without any trouble. You don't even have to understand the question. All you have to do is memorize the answers. This is what a lot of MCDBAs have done. Hell, every developer I know has his DBA cert... big deal. They still don't even know the basics. I honestly can't count the number of times someone bragged to me they were certified, and then couldn't even prove the simplest of skills.
Trust me on this one... the only thing a cert means is that the guy had an internet connection and $100 burning a hole in his pocket.

The 2nd part of the problem is that companies don't know how to spot good talent. My last company hired me after what amounted to a very simple, rather pathetic tech screening. It didn't take me long to see that the guy interviewing me really didn't know that much. To him I was the best DBA on the planet, and they pretty much hired me on the spot. The tech screening lasted all of like 10mins and we really only talked about basic backup and restore ops. And the problem is he said I was the only one who answered those question right (that's a topic for another post). The problem here is that these managers simply don't know what makes a good DBA. Actually, a lot of DBAs don't either, but again, that's for another time.

It's not entirely their fault though. They're doing the best they can. They don't know anything about DBs, so how do they pick a good DBA? Think about it... what would you do if you were in charge of hiring a new accounting manager? What questions would you ask, and how would you verify that the answers were even right? Sure, they can sound right on the surface, but how do you know that he has good judgement? Hiring managers have the same problem with hiring IT staff. They have no idea whether or not their candidates really know Cisco or are just fresh out of class. What can they do, look at the resume? What good is that? It's easy to fake a resume, so that doesn't mean anything either. I don't even look at resumes anymore. I just give my candidates a tech screen and that's all I go by. I don't care if their certified, degreed, or have experience on paper. All I care about is whether they know SQL or not.

It's basically those two factors that lead to companies not having any respect for their IT staff's abilities. They consistently fail to recognize where the true talent is, and throw away people who have worked hard for them, and who have concistently produced for them.

This also leads to the next insult bestowed on IT. There's a huge movement for professional development where IT staff is encouraged, and in some cases even required to go back to school and get a business degree. Apparently, we're so useless that we now have to be experts in both business and in IT. And all of this while the business user is allowed to remain as stupid about technology as he likes. I have seen some companies offer computer training for their users, but I've really been hit in the face recently with the real solid push for IT guys to become business people too.

I've been asked easily more than a dozen times how much I know about the business of a prospective company, yet I'm sure it wouldn't go over very well for me to ask a hiring manager how much he knows about SQL because I don't want to be plagued with stupid questions and unreasonable requests. For some reason, it's necessary for me to know the company's business but it's not necessary for them to know mine.

Anyway, I guess what it boils down to is companies really need to start looking at who's important in their IT staff and try to do what it takes to keep them. These are the people who know your business model and the histroy of your servers and applications, and you can't just throw them away for someone cheaper and expect to keep doing business like you have been. Change your mentality about IT. Once you start seeing us as the viable part of the business we are, then you'll not only start keeping us around longer, but you'll start recruiting better talent to begin with.
Good Luck.

Posted by Sean McCown on July 17, 2006 07:42 AM