Share your IT war stories

Got amazing tales, real-life experiences, lessons learned the hard way, or war stories from the trenches where IT and business intersect? Perhaps you've been inspired by our Off the Record column and want to share your story, as well. Spill your guts in comments below, or via e-mail. We will conceal your identity here on the blog, and ask that you do the same for the company and colleagues you write about. (If we spotlight your story at the top of our home page, we'll send you a $25 American Express gift cheque for your troubles.)

---

May 06, 2008 | Comments: (0)

How to make a DBA blush

As the person who keeps the local instant messenger gateway server running, I have worked with our DBAs to ensure the database is backed up and regularly purged of old records. With the occasional update and patch, it has been running more or less problem-free for about six months. I usually only have to touch it when HR or Legal needs to pursue a issue. The oldest records are supposed to be purged weekly to keep the size down. A week or so ago, one of the DBAs called and said the purge wasn't running successfully and he was seeing data corruption. He exported the database and tried to restore it. Still corrupt. He tried to restore from an earlier backup. Still corrupt. He even tried to truncate it without any success. As a last ditch effort, the DBA looked through the export file to see if the data might be salvageable. He came back a few minutes later and asked if I knew what people did in IM. He seemed a little embarrassed. I explained that the system popped up a notice every 15 minutes to remind people that their IM conversations were being logged. In theory, that notification... more April 29, 2008 | Comments: (0)

Don't do this in Windows

Our own Test Center guru Brian Chee encountered quite a few setbacks in a recent installation of Office Communications Server. His blog, Geeks in Paradise, often offers insight -- both amusing and useful -- into the working life of an IT professional. We're posting an example of those insights here in Off the Record this week not only to give you a taste of Brian's candor but also to provide some tips in case you're planning your own foray into Office Communications Server land. As Windows ages and evolves, it's sometimes too easy to forget those little gotchas that creep up and become the bane of sysadmins. My scenario is working on the Interop iLabs (technology demonstration areas) on Unified Communication. I'm responsible for bringing up Office Communications Server so that our group can get both voice and presence status to transit a gateway to a Jabber server through a federation interface. But I lost a bunch of time on little nit-picky issues as I burned the midnight oil rushing to meet our ship deadline.... So for the newbies, take note and save some bookmarks. For the old timers, have a good chuckle and remember the pain you went through... more April 22, 2008 | Comments: (0)

Congratulations, you have been integrated

I work as a senior manager for a small high-technology engineering firm. We were recently acquired and integration is now being inflicted upon us. One of the early integration targets was corporate travel. For the first time since the genesis of Travelocity, Expedia, Orbitz, and their ilk, we have a corporate travel agent whose use is mandated for all travel. If there is anything more useless to a bunch of brilliant engineers with Web access and smartphones than an old-fashioned travel agent, it's hard to imagine what it is. Our accounting and timekeeping systems are now being integrated. By integration, the parent company means ditching our systems and force-fitting us into their less appropriate systems. The timekeeping affects the line engineers the most. We are subject to total time accounting and therefore everyone must record their time daily. Our old system was junked in the name of "integration" because a different system that was already in use by the parent company could import time charges into their accounting system directly. Our old system required one person to perform two steps, both of which were almost completely automated. The cost for removing these two steps is the almost complete lack of... more April 15, 2008 | Comments: (0)

Our security is secretly secure

I make no bones about being a bigot when it comes to routing gear. I like the company that has the bridge on the box. I also manage the world wide WAN resources of a little company that had a few billion dollars in sales last year. Even converted to euros, that's a chunk of change. We bought a division that is outsourced to one of the big three-letter acronym companies. There is a firewall between the parent company and the new division until the turnover process is completed. As the day for turnover approached, we started asking for details on firewalls, routers, and switches. We were inundated with pictures, spreadsheets, and procedure documents but no actual useful information. I think this particular group adhered to the baffle-them-with-bull-stuff rule. In one of our weekly time-waisting conference calls, feeling incredibly frustrated at the impending turnover with nary a solid piece of documentation, the outsourcers finally agreed to run some commands on the firewall and routers and send us the output. Being the wise guy that I am, I told them I only needed one command executed. Mind you, on this call are people who supposedly work in networking for a living.... more April 08, 2008 | Comments: (0)

Laptop follies

I did tech support for a company with a few dozen account managers scattered throughout the U.S. Most of them had many years of experience in a business with a multitude of nuance and detail that had to be attended to with the initial contract, and on a continuing basis. Millions of dollars a month were at stake with each account. Most of the account managers were salespeople at heart. Top management had a long history of encouraging their sometimes outsized ideas of entitlement. Hotshots were hired. Their sense of entitlement, and lack of technical ability, may have been larger all of the others combined. In fairness, most of them were intelligent, conscientious, and appreciated the extra effort it took for us to support them by phone and e-mail. We gave thanks for Apple Remote Desktop often, as it saved us hours of trying to control wayward mouse clicks and explaining, for the tenth time, how to get to the Apple menu. Then there were the others. One was a little ditzy, but those in the know reported she was good at her job. Unfortunately, she was one of those who did not believe that her job included a minimum... more April 01, 2008 | Comments: (0)

Bait and switch IT

My company, U, acquired another company, V. The entire IT department at V was outsourced to Company TLA (Three Letter Acronym). In the process of trying to discoveri the extent of the network we had recently acquired, we asked TLA reps for diagrams and any other relevant information such as network size, IP addresses, and numbers of PC desktops. Let's just say that my dog leaves more legible information on a fire hydrant than what we got from the TLA people. Most of the diagrams looked more like a game of pick-up-sticks than networking. I was unaware you could cross that many lines in one drawing or that you could intentionally loop traffic through a firewall more than once. We decided they were either brilliant geniuses or blooming morons. You can guess which camp I put them in. One particular site was particularly sparse on information. When we asked for additional material, we were told an investigation would be required. Several days, even a week passed before we got the results back. The diagrams and information were minimally improved but it came with a nice cover letter detailing the deficiencies of this site and how it was not up to... more March 25, 2008 | Comments: (0)

How to keep your servers moist

I've done a lot of contracting but this one tops them all. I worked for a company as a contractor to design and oversee the building and migration of a new server room. This was a major move up for them and they wanted the works with waterless fire suppression UPS, generator cooling, and so on. They liked the design and all the features, and planned to go ahead with it. After the demolition was done on the space and the new work started, the CIO left his position and the CFO stepped in for the interim. He had no understanding of IT. Based on recommendations from other sources, he decided the company could save money by replacing the fire suppression system with good old standard sprinklers. And that they could reduce the fire rating required for the room, thereby saving even more money. Yes, my blood ran cold, too. I protested the changes and explained the risks and issues but he would have none of it. He'd made his decision. Well, I get paid either way so his wishes were carried out. The project was completed on time and well under budget. A bonus was paid to the CFO... more March 18, 2008 | Comments: (0)

How fast can you move a datacenter?

We had been with our datacenter provider for a little over two years and were planning to move out. We believe in partnering with our vendors so our intended move was no surprise to them. We agreed to go month to month rather than sign up for a 6-month extension. Everything seemed good; we had plans to upgrade our power, cooling, fire suppression, and build the room. We were on track to move in July, but in February got notice that they colocation facility was kicking us out with 45 days notice. Luckily the team I have is top notch and rather than panic, we aggressively planned. We came up with contingencies for power (we didn't have enough) but could retire some systems, replace some older 21-inch CRT monitors, and virtualize a lot of systems in the short amount of time we had. We didn't have cooling but were able to secure temporary cooling units in time for the move. We needed backups but luckily had a really good relationship with our vendor who agreed to expedite new hardware for us. We met with our executives and explained the risks: limited redundancy on power and cooling, none on our... more March 11, 2008 | Comments: (0)

Finding my own level of incompetence

I am temperamentally a lone or small-group software developer, and have always generated frequent -- usually weekly -- builds for my customer to review. I was so successful at this at one company that I was promoted to manager. I had taken on the contract development of a DOS application for PCs in C from a regular customer. There was just too much work for me to deliver it on time by myself so I hired a programming manager -- let's call him Bill -- and he hired three more C programmers and a technical writer. He had a great track record, so I gave him the management of the project. I took on the database design and the assembly language programming, but otherwise tried to stay out of Bill's way and manage the business. We started work in the summer. Our first deliverable build was due in October. I booked a flight to the customer's site for myself and Bill two months ahead of time. I was very impressed by the progress everyone was reporting until it got to 90 percent and stayed there. Uh, oh. And where was the integrated build? Not to worry, said Bill: Everything was... more March 04, 2008 | Comments: (0)

Manual? What manual?

We have all had our own personal "learning experiences." I experienced one back in 1991 when I was the second-shift computer operator responsible for backing up our Novell servers, including payroll. My job was to run the backups and any other little jobs that I could do, such as run reports, change printer ribbons, and the like. It was a great job for a college student learning about computers and IT. I'd been doing it for 6 months when we got a new version of backup software. Upgrades seemed easy enough and I was looking for more to do, so I offered to do it after hours. The install went fine, so fine in fact, that I didn't even need to read the manual. That night the backups worked perfectly. I was proud of the way I handled it and pretty excited that my first real IT project went so smoothly. All was good for three months or so, when I got a call from the first shift admin, Al. "Hey, did backups finish OK last night?" Al asked. I knew they had worked since I had watched them, so I replied, "Yup sure did." I looked at the log... more February 26, 2008 | Comments: (0)

Readers: Keep your pants on!

Last week's Off the Record showcased a junior IT guy caught in a tough situation. Readers sounded off with their own advice. On the one hand, there's your pride; on the other, your job -- and your trousers. Last week's Off the Record told the story of a young fellow in IT support just trying to make his way in the datacenter when he accidentally pushed the wrong button. To make a long story short, if he released the button, a critical server would have shut down while people were using it, and he'd be history. Yet he couldn't reach the help line some 10 feet away. And so he took off his pants and attempted to lasso the phone, dragging it to him. Well, you get the picture. (For the down-and-dirty details, read “Don’t lift a finger.”) Our IT readers, problem-solvers by nature, had plenty of alternative solutions for this fellow. Some of them were pretty creative. Practicality goes a long way, as John writes: "Why not use his belt to loop around the rack and possibly tie the button down?" Or, for that matter, shoelaces would work, too. If irony is what you seek, check out Sachin's advice... more February 19, 2008 | Comments: (0)

Don't lift a finger

This incident dates back to 1998 when I was an employee of a very large IT company on the North Side of Dublin. My position there was as a junior member of the IT support team: a team supporting 650 users. One day one of the other guys on the team asked me to head down to one of the server rooms and shut down a particular server. It was a noncritical server, so there was no issue with doing this, at least I didn't think there was. A brief amble later I arrived at my final destination, and entered the server room. It seemed more like a scene from a NASA film with huge fans and blinking lights everywhere. The Server I was to shut down was called "Dub06." It was connected along with 5 other servers to a single keyboard, mouse and monitor via a switch box. I selected the corresponding button on the switch box to give me control over Dub06. All fine there. I then told the computer to shut down, and after about two minutes I got the message saying "It is now safe to switch of your computer." Grand, I thought. I reached over... more February 12, 2008 | Comments: (0)

Nothing cheeky about safety

The incident you are about to read about is entirely true in every single respect. There is no exaggeration anywhere, as any would be pointless. About four years ago, I was contracting to a large insurance firm in their IT department in City Centre Dublin (Dawson Street, as it happens, for those who wish to guess what the firm is). Anyway, my boss, Harry for the sake of this account, had asked me to go to Server Room Two and repatch some of the panels, as the cables were in an awful state. After about an hour on the job, alone in the server room, I started to get a little bored and so decided to go get a cup of coffee. I began to untangle myself from the cables around my feet and in the process turned around in order to make my exit from the room. Unfortunately, untangling myself suddenly became a problem as my foot slipped through a gap in the floor. I instantly realized my foot was jammed and so I bent down to attempt to free it. This is where things went haywire. During the action of bending down, my bum extended toward the patch... more February 05, 2008 | Comments: (0)

Avoiding dot-bomb 2.0

With an increasing wave of Web-based applications in development, it may be worth remembering some of the follies from the dot-com boom era so that we don't repeat some of their mistakes in the new enthusiasm. [Submit your own tales of IT woe! Email them to offtherecord@infoworld.com and receive a $50 American Express gift cheque for your trouble.] I was a VP of development at a site trying to attract a specific demographic slice, one that we all thought would be highly valued by advertisers and individuals alike. With several tens of millions of dollars from investors, we had a huge staff, lots of partner contracts, and a completely unscalable architecture. And calling it architecture is kind; it was a hodgepodge of tools and hand-coding that meant implementing anything took inordinate time and human resources. I came in after launch and, having worked at more traditional companies, tried to bring some process order to the chaos. The young wunderkinds who largely staffed the company believed in the popular Silicon Valley mantra that the old rules didn't apply on the Web, and they were determined to do the opposite of "established" technology and business practice. Little things such as budget approval... more January 29, 2008 | Comments: (0)

Black plasma

Nobody challenged the loud, overconfident marketing exec who thought an outdated idea was hip A high-profile exec was hired to do marketing, and find or create new ways to drum up business. He was smart, fairly good looking, and thought highly of every word he spoke. Although a little bombastic, he made sense, or appeared to, often enough to have some credibility. He had not been in our business in many years, but apparently recognized it as an easy entry to bigger and better things. He hired assistants. He bought new lights and office accessories. He hired a fancy-pants and probably very expensive cohort to help him hatch new and brilliant schemes. He presumed to know better than those with more time in the business than he had birthdays. His clothes were expensive, his German cars were new and large. His grandiose house was known to all who would listen, and a few who would rather they hadn't. He was the Big Cheese. He didn't care what kind, as long as it was rare and expensive. It turned out to be a little stinkier than anticipated. After the assistants, all of whom adopted his self-important airs, were outfitted with the... more January 22, 2008 | Comments: (0)

Naivete and cheapness kill a launch

The dot-com heyday of excess may be over, but dreams of bootstrapping a Web-based business are not. I was part of a small team hoping to launch a demographically targeted job- and consultant-matching site in 2007. The principals, myself included, were not technical pros, so we hired a firm to do the database and Web development for the service. The firm was recommended by people we know, and I had worked with one of the firm's principals a decade earlier. We did the project spec in what we thought was the right way: a detailed specification, mockups of the UI to show the flow of information and options for the users, so there'd be a context for what was being architected and then coded. The firm took the specs and got to work, answering questions along the way on various details and noting implications we hadn't seen -- what we expected from a professional firm. Six months later, some components were available for testing, and they mostly worked. Except the lead developer forgot a key fact: the matches were many to many, not one to one, and his database architecture didn't anticipate that. Whoops. Time for an architectural redo. But... more January 15, 2008 | Comments: (0)

A classic dot-com tale of woe

This is one of those cautionary tales from the dot-com boom. Do we really need another, you ask? Well, there's always room for a classic story of hubris, terrific marketing, and a total misunderstanding of the importance of infrastructure. In 1999, I was involved in a startup consumer Web site that decided to launch before the technology was fully baked. The thinking -- remember those days?-- was that we didn't want anyone to get the jump on our brilliant idea. So our little company put its effort into a PR campaign and a glorified signup page to capture users, which at the time were valued at about $1,000 a pop. Just think, if we got a million of 'em, we'd be a billion-dollar company. [ Share your own tales of IT woe by sending them to offtherecord@infoworld.com ] We had a great team of developers, although in our shoestring operation, most were moonlighting (which may indicate the level of faith they had in the CEO). They used the occasion to develop an elegant, JSP-based site, which was fairly cutting edge at the time. A free version of WebSphere was the app server of choice. To keep costs under control, a... more January 08, 2008 | Comments: (0)

Running cable without a clue

Back in the bad old days, sloppy network cabling practices caused intermittent network problems that were painful to solve One day we got a call from somebody having trouble with her mainframe connection. I had started the installation of this workstation some 8 months previous, and then somebody else completed it while I was on vacation. All of our cross-connects were made on 66 blocks; there was no structured cabling in any building. Token-ring was pretty robust in that respect; it usually ran the full 100 meters with no issues. It was the 90s, by the way, which explains why the entity I worked for was still doing all token-ring networking. It wasn't until much later that IBM even acknowledged the existence of Ethernet -- and our CIO wore Blue underwear. One of our campuses had been built partly in the 1940s, and the newest portion in the late 60s. All of the inter-building was AT&T-style voice-grade stuff. No fiber existed. We started having a major problem with the network in one entire wing. It would periodically go completely bonkers, lots of lost connections, time-outs, and slowdowns. Then it would mysteriously stop. Even after we threw everything we had at... more December 18, 2007 | Comments: (0)

The best of "Off the Record"

From the ridiculous to the sublime, reader tales cover the gamut of real-life IT experiences It's no easy feat choosing the "best" of Off the Record. One tech's poison is frequently another one's honey. Who's to say which InfoWorld reader told a more amusing story? Who can determine which tale was more rife with snafus? The only thing everybody can agree on is that the more things change in this Dilbertesque working world, the more they stay the same. Clueless bosses, annoying coworkers, and incredible situations show up in the Off the Record posts weekly -- and never fail to entertain us. Here's a sampler of the workplace insanity we enjoyed reading about this year. Go ahead -- try to decide which one is best. Lame bosses. Irascible dictators and delusional oligarchs played a significant role in the stories submitted, which surprises just about nobody. Who can resist the allure of talking smack about bosses who can't defend themselves? Especially when they deserve it. From micromanaging maniacs to former soldiers, IT sees its share of irritating leadership. Maybe the most egregious errors come from bosses who don't know much about technology and wouldn't know a good idea if it kicked... more December 11, 2007 | Comments: (0)

Zinc whisker infestation

Tiles are tiles, right? What's a little galvanic corrosion among friends? I work at a small company (about 300 employees), owned by one of the largest in the world. Yes, of course the parent company is brutish, inept, inefficient and overall, a PITA to deal with. Their senior management, however, naturally thinks otherwise, and does not like, to put it mildly, to have its questionable practices challenged. Our very small datacenter has a raised floor, for which we needed some new tiles. The datacenter manager did the research, recommended a product and a vendor we had previously used, and requested the purchase be made. Mr. Pinchpenny, who works in another state and has never seen or met any of us, was in charge of purchasing. He decided to save a few bucks and forced us to use tiles that were languishing in a company warehouse. We gamely installed them over a weekend, lamenting the fact that we had to mix so many ugly and beat-up tiles into our formerly pristine floor. Several months later, a couple of our servers suddenly stopped in their tracks. Bad power supplies. We thought it odd that more than one at a time would go... more December 04, 2007 | Comments: (0)

Ignore these specs immediately

Coordinating the activity of 30 different truck depots is no picnic -- especially when management pays no attention to the plans it commissioned for a new system In the 80s, I was a super keen graduate trainee at a large logistics company specializing in liquid movements in tanks. Our depot was chosen to assist the computer guys in designing a new system that would centralize all depot planning. The goal was to allow management to coordinate all routing, planning, and invoicing procedures, thereby running the 30 or so depots more efficiently. Theoretically, every depot would know where everyone else's trucks were so that work could be assigned accordingly. We spent six months writing specifications, explaining how tanks were dedicated to specific products working for specific customers and highlighting where we could share resources. The project demanded lots of time and energy, but we were excited about it. So you can imagine our surprise when IT then dumped a second-hand general freight system into the mix, completely ignoring all the information we'd given them. The result, after some £4 million spent (nearly bankrupting the company): one very slow delivery note printer in the corner of the office. The inter-depot notification system... more November 27, 2007 | Comments: (0)

Data flows faster downhill

It was the peak of the dotcom boom and I worked for a travel agency that was buying out Mom-and-Pop travel shops from Seattle to Miami -- mid-1999, if I remember correctly. I was part of the traveling network team that was responsible for acquisition network and server reviews and employee evaluations of said acquisitions. I was sent off to our recently acquired sister site in Seattle, a mega travel company that had about 10 smaller sites and a supposedly superior sys admin -- or so I was informed. Eric was the only IT guy for this company, and he had just gotten his CCNA AND MCSE. It was my job to review Eric and the site to determine whether he should stay or go and what other changes might be necessary. Eric walked me through the impressive datacenter (or server room at the time), which was a pristine room reflective of a man who took pride in his work. Cables were run in cable trays, zip ties kept them bunched neatly, and each rack was arranged in the same fashion, hinting at consistency and forethought. The site was not typical of a Mom-and-Pop shop with Kmart network gear... more November 20, 2007 | Comments: (0)

Choosing the best code for the job

Once upon a time when handheld PCs were larger than the original Motorola "brick" cell phone, a project was to be coded in BASICA under DOS 3.3. Why? Because the Fortune 500 company requesting the project was too cheap to do it right. The guy coding the stuff was being asked to make the BASICA program do automatic edits. Did I mention that BASICA was slower than molasses in January? All my coworkers were mini and mainframe programmers by trade. One day, I and the guy in charge of coding this project were on a break when he lamented how difficult BASICA was to work with. What they wanted was brutally simple to whip out in any real language. I happened to dabble in dBase and Clipper in my spare time and offered to bring him a prototype the next day. He loaded the compiled EXE on the lowly 386SX handheld, and it was nearly everything they'd asked him to do. I gave him the source code, and I compiled his edits the next night. He picked up Clipper coding in about two hours by looking at my original code and a manual I'd let him borrow. He'd wasted four... more November 13, 2007 | Comments: (0)

The tech who almost killed Saddam

Everyone has stories of the Novell server that ran for 12 years and was found only after a Y2K upgrade. I, on the other hand, had a coworker with Saddam in his gun sights Bill was about 26 when I met him in 2001, and I instantly learned that he was a mediocre technician. He was an IT cowboy (a.k.a. "Network Administrator") and once managed to reset an entire corporate file server system with his user account owning and having all exclusive right to all files. His peculiarities led him to believe that hubs were the most secure network devices (a whole other story) since they don't usually have user names and passwords, thus not allowing anyone to sniff traffic. He had set up an intricate 20 interlinked hubs that ran the corporate network for the large Fortune 500 company we worked for. This guy managed to find a few people that would listen to him at any time and wasted more time drinking coffee and shooting the breeze than IT work. His attempts to try and take over our weekly meetings with "I can top that" stories grew tiresome pretty quickly. His stories never really related to the IT... more November 06, 2007 | Comments: (0)

No photos, please

As the newly hired tech specialist of a 500+ store chain of fast food restaurants, it was my responsibility to work with food scientists, designers, and management to research new methods of providing food safely and quickly into the customers' hands. As new items were added to the menu, we needed to modify existing kitchens to accommodate new equipment. I was tasked with visiting a local unit to confirm some measurements, so before I left HQ, I asked the department secretary if I could borrow the Polaroid camera (this was the 80s) to snap some pictures to help me in this retrofit. Before she could answer me, a voice rang out from one of the nearby cubicles "You can't take flash pictures in the restaurants!" It was Tony, a long time employee of the R&D department. "Why not?" I asked. "Because ..." Tony said, "if you use the Flash from the camera in the restaurant, the computer-operated cash registers will shut down!" Being the new guy, I didn't want to question his sanity, but I politely responded with, "Huh?" Tony explained to me that for as long as they had had the new computer registers (3 years), every time a... more October 30, 2007 | Comments: (0)

Computer function 101

When troubleshooting, it's a good idea to know your audience I was a national support specialist for a major equipment manufacturer. Thirty minutes before quitting time on a Monday afternoon, a call was sent to my desk from a major customer who had a complete system failure. Their system supported 300+ users and everything had stopped. I found myself on the phone with both a man and a woman who told me that everything had been fine, but then suddenly the system had died. There had been no error messages or error lights on the main processor unit. I sent them into the next room to check for fault lights on the disk drives (these old drives were the size of dishwashers and used three-phase 220V power). But there were no fault lights. I asked the woman to type in various diagnostic commands into the system console. In each case, she told me nothing happened. After concluding that there was no way to possibly salvage the state of the machine, I had her press the reset button on the front of the CPU, knowing full well that this meant as many as 300 people would lose their work and require... more October 23, 2007 | Comments: (0)

Mystery solved: The case of the very bad Monday

Symptoms can be misleading. Sometimes the best diagnosis comes from examining the bigger picture. July 1987: My first day on the National Support desk for Wang Labs in Canada, and I guess the rest of the team wanted to provide me with a trial by fire. I was assigned to work with a specific customer who had an intermittent problem where his departmental mini-computer would suddenly slow to a crawl every Monday morning at some time between 9:30 a.m. and 10 a.m. every week, and suddenly start up again some 10 to 15 minutes later. Over the course of the next several weeks, we studied every aspect of what happened on Monday mornings. Folks came in and logged on, checked their e-mail; certain users went into applications, others went for coffee. The rate of use was not significant, and there was normally much more activity happening by 10:30 or midafternoon. Moreover, there was much more activity on Tuesday through Thursday, but the machines experienced no slowdown on those days. All we could tell was that the system seemed to be suddenly doing a very large quantity of system-related disk I/O and no one could move forward while this was occurring.... more October 16, 2007 | Comments: (0)

My kingdom for a RAID array!

Hardware was only one of many costs to consider when it came time to purchase more disk space. But that was the only one management focused on. In the early 90s, I worked as the IT manager for a hardware distributor on the East coast. They were a fairly good-size company with a couple hundred employees and multiple branch offices spread throughout the country. At the time, we were operating on an IBM RS/6000 server that was quickly running out of disk space. Because of a lack of drive bays in the main system, it was necessary to purchase an external array. Being the ever-paranoid IT manager that I was, I thought, "Great! Now I can get that RAID 5 array I've been dreaming of!" (I would have preferred a mirrored array, but disk was very expensive back then.) So I did my research and wrote a proposal to my boss recommending an external RAID array from EMC that would have hot-swappable hard drives, super-fast dual-channel SCSI interface (remember this was the early 90s -- SCSI was fast back then!), redundant power supplies, and many other fine features. The price tag for this hardware was around $60,000. Considering the benefits... more October 09, 2007 | Comments: (0)

Password disaster in the making

What's worse: passwords taped to the monitor or a boss who assigns them all? My contract ended yesterday, and today I am a happier and feel much more comfortable in my chair. This gig was for the local hospice organization, which falls under HIPAA because of the patient information it retains. It also maintains a database of donors and family members of the deceased. I started working for the organization last year providing solutions to help the growing office, which has 16 desktops and a small (five-year-old Win2000) server. I started off updating software and bringing all of the computers up to Win2000 SP4, adding a second anti-virus program (Symantec was not licensed for all of their computers) and anti-malware software. I noticed that some of the troubles stemmed from not having enough CALs (Client Access Licenses) and some very badly managed Active Directory permissions. It seemed like each time I fixed an Active Directory issue, something else would "break." In time, I was able to fix the permissions, clean and protect the computers, and get a new DSL modem. The only thing that I was not able to do was get the Director to allow Active Directory to manage... more October 02, 2007 | Comments: (0)

Night-shift horror a screamer

Virtually unsupervised, the "new guy" got a thorough initiation when faced with a night-shift full of spun-down drives Back in the glory days of mainframe computing, the University I worked at had an IBM timeshare machine. Connected to it was more than the usual number of disk drives because there were so many users. In those days, a disk drive was about the size of a washing machine and they were so expensive that they had removable disk packs. These drives were usually located in rows and columns on the floor, something like a checkerboard to minimize cable lengths. This was colloquially called a disk farm. Because of the way our machine room was configured, the disk farm was located on the floor directly below the main computer. The unmounted disk packs were stored in the same area on shelves along the walls When a user wanted data that was not online, he would go downstairs, spin down an appropriate disk drive, remove the 40-lb. disk pack, mount the new disk pack, spin the drive up and tell the OS it was now available. The machine-room staff performed this procedure numerous times a day, all three shifts (we ran 24/7).... more September 25, 2007 | Comments: (0)

Label this tape 'trouble'

Neither voodoo nor sabotage was behind the mysterious Monday tape overload problem. Nevertheless, it took a year to find a solution. Years ago, I ran a small VAX network for an engineering company. 8mm tape was just emerging as a backup technology, and I eagerly went for it. The high-density tapes we used would do an entire system backup on one tape, with room to spare, in a relatively short time. We could back up daily, unattended. What a nice change from mounting a dozen 9-track tapes and babysitting the process over several hours every weekend. Everything worked fine until one Tuesday morning when I came down to take out the Monday backup tape and replace it with Tuesday's. The backup program was asking for a second tape for Monday's backup. Oh, no! Our files had finally exceeded the capacity of one tape! This was a bad situation, because we had only a single-tape deck and putting a second tape in after the workday was underway would be useless. So I spent that day looking for files I could delete (there were always plenty) to get it back down to one tape's worth. I must have done well; Tuesday's backup... more September 18, 2007 | Comments: (0)

Zapped: Ungrounded PC a shocker

Who knew that terra firma would have been the culprit behind one client's PC connectivity problems? In the late 80s I had a customer with various PCs connected to an IBM mainframe controller via coax. We had a maintenance contract on all of the PC hardware that covered every component, and all but one of the PCs worked just fine. The 3270 emulation screen would just lock up and stop communicating very intermittently to add a layer of complexity. Because this customer was a long drive away from our office, the contract dictated next day service, and so I had asked the dispatchers to let me know when it failed again so I could drop anything I was working on and go there immediately. The problem perplexed me but it had also become one that I was determined to figure out. The final time I was called out for this problem, the customer was asked to leave the PC alone in the failure state. When I arrived, sure enough, the emulation screen was locked up. I tried swapping the cable with another port on the controller – it still failed. Over time, I had swapped the 3270 coax card with... more September 11, 2007 | Comments: (0)

Off the Record -- now with swag!

InfoWorld's Off the Record blog takes a look back at its most popular posts to announce bigger payoffs and a nifty T-shirt for new submissions One of InfoWorld.com's more popular blogs is the one that's authored by you, our readers. To recognize your contributions properly, we're upping our payment to $50 per accepted submission, and we're giving away a T-shirt that will tell the world that your story went live at InfoWorld.com. What are we looking for? Here's a sample of the blog posts that stimulated the most interest. Tell us your real-world IT tale (or horror story) and you could generate some interest, too. --Eds. The ghost who sabotaged the mainframe The story really starts back in December of 1971. I had just started as the new Director of Data Processing for a large nonprofit in New York City. My predecessor, Ernie (I've changed his name), had held the position for many years, and only his wife's insistence that he retire had convinced him to leave. It was an extremely difficult decision for him. The mainframe we were running at the time was a Honeywell Model 120 with 64K of main memory, three 1200 BPI tape drives, a printer... more September 04, 2007 | Comments: (0)

We don't need no stinkin' UPS

Just to prove that the server had its own circuit breaker, Joe went ahead and, well, broke it I had been working as the Junior Network Administrator for about 6 months at a large grocery reseller in 1995, and budget season was coming up. Our group had submitted the initial budget and now the series of managers responsible for approving the budget wanted to tour the datacenter to see what they had bought last year. Joe, the senior network administrator, had the duty of leading the tour. All of the managers came out of the mainframe era of the company, and the entire PC Network thing was new to them. Joe grew up on Netware, and he was about to show off his pride and joy. We worked at world headquarters and supported about 750 local users. The previous year, Joe had purchased two high-end Netware servers, splitting the load between them. Joe showed the management group the primary server, starting with the council screen. He pointed out the uptime, indicating that it had been online since about 3 weeks after it was purchased. He explained the processor, the memory, and the external drive arrays. This was an impressive server... more August 28, 2007 | Comments: (0)

D'oh! Donuts may get you fired

Management didn't like desktop shortcuts or donuts and I liked both. It was a recipe for trouble. I worked the help desk under a contracting company for the FDIC when they transitioned from Windows 95 to new Dell desktops running Windows XP. Many of their users were utilizing desktop shortcuts to get to various applications and files. We had an introductory teleconference meeting to discuss strategy and approach for this transition. Those managing it made it resoundingly clear that we would not be copying desktop shortcuts from the old machines to the new ones. I advised the team against that course of action, explaining how it would lead to a lot of customer dissatisfaction and help desk calls. And for such a small issue, it seemed logical to minimize the level of change for the users – let them keep as much that was familiar as we could. But the management team would have none of it. In fact, they got downright irrational, claiming that “many of those shortcuts are broken, anyway.” (Never mind the majority that were not.) This response was delivered in a very dismissive and unmistakably condescending tone of voice. I could tell it wouldn’t be worth... more August 21, 2007 | Comments: (0)

Detail-obsessed CEO misses the big picture

Our chief preferred to micro-manage the small stuff than deal with substantive issues I thought I had found my dream CIO job when I scored an interview at a key firm in the financial services industry. My subsequent first-round interview meeting with the CEO and her management team went very well. A week later while en route to a client, I received a frantic call from my wife. Apparently, the CEO had been calling my house trying to locate me. The CEO sounded anxious, demanding to be put in touch with me immediately. So, I called the CEO directly and she asked how soon I could meet with the interview team again. As the CEO put it, what exact time could I be over at her office, that day? Wow, I thought. Was an offer on the table? That was my first taste of this CEO's over-reactive, micro-managing style. After that second interview, she asked if I would consider a different role in their firm, that of managing their distribution facility in New Jersey. I was a little disappointed about not getting the CIO role (they indicated that they found someone with Six Sigma experience), but the salary offer was... more August 14, 2007 | Comments: (0)

The game with no winner

A coworker's need to get ahead sabotaged my career from that point forward In 1993, I took a job with an insurance broker. With the public use of the Internet looming, they didn't even have a real network. Having been a midrange developer, I learned about PC servers, the Blue Screen Of Death, and everything else related to our needs. I oversaw infrastructure growth from barely there with 25 employees to 70 employees, each with a computer, a nine-office WAN, e-mail that Boss said we would never use, voice over IP, and a host of other particulars. I was pleased but overwhelmed. I took my case to Boss and got approval for a network administrator, who I will call Doom. He was young and relatively inexperienced, yet I saw potential. I offered a generous salary because Boss's vision was to have a small IT staff -- and that required motivated workers. Doom and I got along great. After many successes, things were clicking. When the BlackBerry arrived, we got approval for morning meetings at the coffee shop. Living near each other, we would meet, find our table, open our laptops, and keep up on the office while discussing work --... more August 07, 2007 | Comments: (0)

Another screw-up? You're promoted!

It took three years for upper management to squeeze out the lemon they'd hired Two mergers ago, the CFO of the company I worked for hired a new Director of IT, who would report to the CFO. The new IT director (let's call him Griff) had a command-and-control style of management and liked to use manufacturing analogies to describe everything. He soon began irritating the executive board, terrorizing staff, and alienating vendors. He buried the staff in so much admin work that we couldn't get much done. What we could get done was never good enough, and everything was always someone else's fault. He was what is commonly referred to as a "seagull" manager: fly in, make a lot of noise, crap all over everything, then leave. At one meeting with a vendor in which Griff commented that he did not believe in courting a relationship with any particular vendor. After the meeting, the vendor phoned me to ask "what is up with this guy?" I explained that he was relatively new to this company, but the vendor declined any further involvement with the project. Then I told Griff about the call (I figured he was entitled to know for... more July 31, 2007 | Comments: (0)

Panic mode

Lack of knowledge + heat of the moment = bad decisions I started in the industry in the early eighties and worked as a technical salesperson for a local micro-computer company. We had a client that purchased a 286-based server running SCO Xenix to run a FoxBASE application in a multi-user environment ... pretty hot stuff in those days. The application had been written by a local consultant -- let's call him Dwayne -- who knew FoxBASE in single-user mode on DOS but didn't know Xenix and didn't really want to. Dwayne, who was running behind on his schedule to get the application done and fully taking advantage of the multi-user system, hired Herman, a college student, to do some coding for him. [Ever worked with a technology that seemed to defy all logic or simply drove you up the wall? Submit your story to offtherecord@infoworld.com] About four weeks after the system was up and running, Herman was on-site doing some coding and the two office admins came into his office. They told him that the system was locked up and that they couldn't continue their data entry. Their boss was exceptionally impatient and demanding, so they wanted to know... more July 24, 2007 | Comments: (0)

Covet my code

Sturdy code can be a thing of beauty. Unless you prefer money and sex. I was working in an accounting department in the mid-1980s which had just acquired a completely new computer system, and the accounting application had been written from scratch. Of course it was full of bugs, and the serious logic errors got fixed rather quickly. However, the reporting scripts were never touched, and the reports were awful. In my bookkeeping role I was under strict orders from the manager and the accountant not to touch the code, even though I had been given administrative access. The manager had little accounting experience, and treated the code (outsourced and produced by a large company you've all heard of) as a black box. It inspired a certain fear in him, I think. In any event, the department operated under the notion that the code was the code, and we all just had to live with it. For the accountant, code was the furthest thing from his mind. He was having an affair with a woman in the department, so we assumed that was his primary distraction. Because I was so desperate to generate better reports, I began to edit code,... more July 17, 2007 | Comments: (0)

The war of the hidden motives

Sometimes systems don't work because people have a vested interest in keeping them broken Three years ago, I landed what I thought was a dream job. I had been working as a freelance Web developer, and my impressive portfolio proved useful when the largest newspaper publisher in my South American country advertised a vacancy in the Web department. The company's print publications had driven good traffic to its online counterparts; about 40 percent of the whole country's news-related traffic was ours. Nevertheless, in the eyes of this enterprise, the Internet was nothing but a necessary burden. I saw it as a diamond in the rough, and I couldn't wait to polish it. My first step was to improve and redesign all the obsolete technology in use. I discovered that the hierarchy I worked under was not standard or even logical -- most of the things my department did went unsupervised, and most of our requests were not approved. I learned quickly that our job was to exist and do as little as possible. My direct boss didn't help matters much. She was a newscaster with a megawatt smile and no training in any type of technology. There was no way...