Off The Record | Anonymous

Ooops, wrong switch

Anonymous — 2008-07-15T03:00:00-08:00

The foot patrol officer should have been an ally. Instead, he was our biggest security problem. READ MORE

Admin pretends college IT security exists

Anonymous — 2008-07-08T03:00:00-08:00

In 2005, I attended a public gifted school. The school's computer admin decided we needed an update in the software used to select our classes, so he created a simple program where you type in your Social Security number as the password, then select the classes. Well, some friends and I got curious one day and decided to look into the folder where the software was, and voila – we found the Public folder. It didn't contain much, but we located a simple .txt file that contained an alphabetical list of names and Social Security numbers. We decided to take this file, so we could have some fun with some of our more paranoid friends by walking up and greeting them as "Number ...." Word quickly got out. Rumors spread that we had obtained a bar-code scanner and had used it to scan student IDs when the students weren't looking. Some even suggested we hacked the server. Eventually, school administrators found out who had copies of the file and started to threaten expulsion. We all came clean. We even showed them how we did it. Some people were amused but others weren't. Some students demanded we be sent to the... READ MORE

Back-end trouble

Anonymous — 2008-07-01T03:00:00-08:00

In the late 90s I worked for a midsize ISP in the midwestern United States. Like any ISP at that time, we had dialup service, which meant we had a consumer-oriented tech support number. A friend of one of my co-workers was hired for dialup tech support. We'll call him Jake. He had some computer experience but no specific qualifications to do more than what he was hired to do. Jake proved himself fairly quickly, and within a few months, he had risen through the ranks and was managing the tech support department. Everybody was pretty impressed with him overall, since it was rare for anyone with a clue or any ambition to work for dialup support. Jake's prior job was a bit more ... active than this one, and the fact that he sat in a chair all day answering the phone started to manifest itself physically. The poor guy must have gained 30 or 40 pounds in a matter of a few months. Perils of the job, I guess. I was the systems lead at that time. One day, the NOC contacted me to let me know our main user server was down. This affected shell accounts, personal... READ MORE

Sure, that software's been tested

Anonymous — 2008-06-24T03:00:00-08:00

In the late '80s, I worked at the R&D facility for a PC software company. At that time, it was the largest PC database software maker and the third largest PC software company behind Microsoft and Lotus. Following the phenomenal success of their previous version, the company was working hard to put out the next version. Four years in the making, it was still undergoing serious feature creep as the marketing wonks demanded ever more features to be grafted onto the basic framework. Since the CEO of the company's only previous experience was marketing soap (the kind that makes bubbles), he was ill-equipped to determine the quality of the product. In fact, rumor had it that he couldn't even start it up. Despite this, he was among the highest-paid CEOs in the country. As a result, company ended up shipping the product while it was still full of bugs, which effectively destroyed the company when the product became the laughingstock of the industry. The whole story is a cautionary tale about software development and the pitfalls in it. Two senior developers took to handing the CEO a copy of The Mythical Man-Month each time they saw him. He would thank... READ MORE

Lies at the laser printer

Anonymous — 2008-06-17T03:00:00-08:00

A company I used to work for printed its invoices on a laser printer using preprinted forms. It was one of my staff members' responsibilities to make sure these were printed and delivered to accounting every morning, so they could be mailed out. I came in one morning to find this employee in complete panic. He was ranting about the printer being broken, that every invoice had printed wrong and he had no idea how to fix the problem. Part of the reason for his panic was that the accounting manager was a raving lunatic if she didn't receive the invoices by 8 in the morning, and quite obviously she was not going to get them by that time this morning. I calmly asked my employee to show me what the problem was. He handed me an invoice and it was printed upside down. I asked him if he had loaded the invoice with forms this morning and he had answered in the affirmative. I started laughing and told him he had put them in upside down when he loaded them in the printer. He yelled at me "NO, I DIDN'T!" I said yes, he had, and from there, our... READ MORE

The tale of the opportunist and some code

Anonymous — 2008-06-10T03:00:00-08:00

In the mid '90s, I was working at a regional ISP, covering about half a dozen states as a systems engineer. Everything in the company ran on trouble tickets. Support issues, customer provisioning, even the sales people used the trouble ticket system to handle their order processing. The ticketing system was a proprietary monster, coded from scratch by a single developer. It was a massive blob of C++, 30-line SQL queries, and X11/Motif code. It was tailored exactly to our company, our customer data, and our workflow. Everything was in this system. Customer data, billing, tech support issues, and sales orders. It worked so well that I've missed using it ever since I left. It did everything but brew our coffee. One day this developer's desktop workstation died. The next day, we found out that the only copy of the source code for the ticketing system was on his hard drive, and it was gone. For the next few days, the developer tried frantically to recover the data. At that time, we didn't back up workstations to tape, so we had no stored copy to recover. It seemed like we were between a rock and a hard place. Then came... READ MORE

Is it hot in here or is it me?

Anonymous — 2008-06-03T03:00:00-08:00

Like a lot of IT groups we don't manage the physical plant (building, physical security, power, and such). We are lucky enough to work very closely with our facilities department, though, and have them set up with our paging system. When our temperature sensors register high temperatures, the facilities department gets paged along with us. We also have a monitor set up in the guard shack that shows the temperature so they can monitor it too. One night all the pagers went off with an overtemp alarm. While we were still scrambling, however, it magically cleared. A few admins lived close enough to check on things anyway. When they got into the datacenter, the heat was overpowering: at least 85 degrees, not counting the hot spots. The temperature sensor read a perfect 62 degrees. What could cause such a discrepancy? One of the main 30-ton AC units had failed and a temporary portable put into service. When the 3-ton portable didn't make the temperature alarm go away, the probe was moved in front of the running unit to "fix" the problem. Now before you all start saying how facilities departments just don't understand, we had a similar problem with a... READ MORE

I can't fix stupid

Anonymous — 2008-05-27T03:00:00-08:00

I was called to work on a Windows 2003 server for a customer. When I got on site, they explained that the original server setup was an 80GB hard drive with 2 partitions. They set the system partition at 8GB and used the rest for storage. Not too bad, except that they loaded Exchange, Domain controller, Citrix, QuickBooks, and several other applications on the 8GB partition. A few months down the road, however, and the partition was full. That is the point in our story where I enter. They called to see what could be done. After researching and trying several cloning programs, I decided there was just no way to increase the system partition in Windows 2003 server. I asked the customer to do a full backup and explained that I would have to rebuild the domain, Exchange, Citrix, blah, blah, blah. The customer gave me carte blanche to do what was necessary. I spent a rather lengthy time reloading and rebuilding this server. I simply used the original 80GB hard drive for the OS and installed a second hard drive for data files. (Stick with me, this is pretty good) I built the new domain, Exchange, QuickBooks (you... READ MORE

Captain Obvious and his ship of fools

Anonymous — 2008-05-20T03:00:00-08:00

I work for a company that acquired two of its competitors about a year apart. The first purchase was a company much like us that I'll call GoodCo. The integration was not without issue, but overall, it went smoothly. And so I was lulled into a false sense of security when I heard about the acquisition of the other company that we'll call BadCo. Because we had just integrated GoodCo’s network the year before, I was not too worried about repeating the process with BadCo's. That is, until I learned BadCo had outsourced its entire IT department to a large TLA (three-letter acronym) company. Well, the honeymoon was over with this one. Nothing came easily with the BadCo integration. Instead of a couple of techie guys discussing the best way to redistribute routes and coming to a gentleman's agreement, BadCo had a process that required paperwork for all manner of inane minutiae and nothing was free. It was all about billable hours, of course. If the TLA had been good, the fees would have been acceptable, but TLA was not good. They just didn't grasp the obvious. We spent days troubleshooting the simplest problems. Once, when we couldn't get people... READ MORE

What is it you do again?

Anonymous — 2008-05-13T03:00:00-08:00

In the late 1990s I worked for a Midwestern manufacturer. We ran a very small IT shop for the size of our corporation (8 people for a company grossing a third of a billion dollars annually, across 10 locations), and all of us in IT were expected to wear many hats. I was a programmer, analyst and project manager when I wasn't network administrator and tech support. From our corporate office, I was directly responsible for five factories around the Great Lakes area, and I also helped out the MIS staff at our factory just down the road. A year before, I had responded to a Friday plant disaster that left inventory and manufacturing untouched but destroyed the office space and all computers. Thanks to spare equipment and offsite backup tapes, our Monday morning shipments went out on time, saving the company hundreds of thousands of dollars. This event opened the purse strings a bit, and I was allowed to purchase a backup server to use in case one of the other factories had a similar disaster. Like many companies at that time, we decided to jump on the ERP bandwagon. All the upper-level execs were involved, as we were... READ MORE

How to make a DBA blush

Anonymous — 2008-05-06T03:00:00-08:00

As the person who keeps the local instant messenger gateway server running, I have worked with our DBAs to ensure the database is backed up and regularly purged of old records. With the occasional update and patch, it has been running more or less problem-free for about six months. I usually only have to touch it when HR or Legal needs to pursue a issue. The oldest records are supposed to be purged weekly to keep the size down. A week or so ago, one of the DBAs called and said the purge wasn't running successfully and he was seeing data corruption. He exported the database and tried to restore it. Still corrupt. He tried to restore from an earlier backup. Still corrupt. He even tried to truncate it without any success. As a last ditch effort, the DBA looked through the export file to see if the data might be salvageable. He came back a few minutes later and asked if I knew what people did in IM. He seemed a little embarrassed. I explained that the system popped up a notice every 15 minutes to remind people that their IM conversations were being logged. In theory, that notification... READ MORE

Don't do this in Windows

Anonymous — 2008-04-29T03:00:00-08:00

Our own Test Center guru Brian Chee encountered quite a few setbacks in a recent installation of Office Communications Server. His blog, Geeks in Paradise, often offers insight -- both amusing and useful -- into the working life of an IT professional. We're posting an example of those insights here in Off the Record this week not only to give you a taste of Brian's candor but also to provide some tips in case you're planning your own foray into Office Communications Server land. As Windows ages and evolves, it's sometimes too easy to forget those little gotchas that creep up and become the bane of sysadmins. My scenario is working on the Interop iLabs (technology demonstration areas) on Unified Communication. I'm responsible for bringing up Office Communications Server so that our group can get both voice and presence status to transit a gateway to a Jabber server through a federation interface. But I lost a bunch of time on little nit-picky issues as I burned the midnight oil rushing to meet our ship deadline.... So for the newbies, take note and save some bookmarks. For the old timers, have a good chuckle and remember the pain you went through... READ MORE

Congratulations, you have been integrated

Anonymous — 2008-04-22T03:00:00-08:00

I work as a senior manager for a small high-technology engineering firm. We were recently acquired and integration is now being inflicted upon us. One of the early integration targets was corporate travel. For the first time since the genesis of Travelocity, Expedia, Orbitz, and their ilk, we have a corporate travel agent whose use is mandated for all travel. If there is anything more useless to a bunch of brilliant engineers with Web access and smartphones than an old-fashioned travel agent, it's hard to imagine what it is. Our accounting and timekeeping systems are now being integrated. By integration, the parent company means ditching our systems and force-fitting us into their less appropriate systems. The timekeeping affects the line engineers the most. We are subject to total time accounting and therefore everyone must record their time daily. Our old system was junked in the name of "integration" because a different system that was already in use by the parent company could import time charges into their accounting system directly. Our old system required one person to perform two steps, both of which were almost completely automated. The cost for removing these two steps is the almost complete lack of... READ MORE

Our security is secretly secure

Anonymous — 2008-04-15T03:00:00-08:00

I make no bones about being a bigot when it comes to routing gear. I like the company that has the bridge on the box. I also manage the world wide WAN resources of a little company that had a few billion dollars in sales last year. Even converted to euros, that's a chunk of change. We bought a division that is outsourced to one of the big three-letter acronym companies. There is a firewall between the parent company and the new division until the turnover process is completed. As the day for turnover approached, we started asking for details on firewalls, routers, and switches. We were inundated with pictures, spreadsheets, and procedure documents but no actual useful information. I think this particular group adhered to the baffle-them-with-bull-stuff rule. In one of our weekly time-waisting conference calls, feeling incredibly frustrated at the impending turnover with nary a solid piece of documentation, the outsourcers finally agreed to run some commands on the firewall and routers and send us the output. Being the wise guy that I am, I told them I only needed one command executed. Mind you, on this call are people who supposedly work in networking for a living.... READ MORE

Laptop follies

Anonymous — 2008-04-08T03:00:00-08:00

I did tech support for a company with a few dozen account managers scattered throughout the U.S. Most of them had many years of experience in a business with a multitude of nuance and detail that had to be attended to with the initial contract, and on a continuing basis. Millions of dollars a month were at stake with each account. Most of the account managers were salespeople at heart. Top management had a long history of encouraging their sometimes outsized ideas of entitlement. Hotshots were hired. Their sense of entitlement, and lack of technical ability, may have been larger all of the others combined. In fairness, most of them were intelligent, conscientious, and appreciated the extra effort it took for us to support them by phone and e-mail. We gave thanks for Apple Remote Desktop often, as it saved us hours of trying to control wayward mouse clicks and explaining, for the tenth time, how to get to the Apple menu. Then there were the others. One was a little ditzy, but those in the know reported she was good at her job. Unfortunately, she was one of those who did not believe that her job included a minimum... READ MORE