The Deep End | Paul Venezia | TAG: Problems

MORE ENTRIES

TAG: Problems

July 23, 2008 | Comments: (0)

In another San Francisco Chronicle article today, there are some quotes and summarizations from this morning's bail hearing:

A judge refused today to lower the $5 million bail for a San Francisco computer engineer accused of hijacking the city's network, after prosecutors said he had rigged the system to melt down during routine maintenance.
Prosecutor Conrad del Rosario said Childs had arranged the system so that key programs were held in temporary memory files that would evaporate when the network was shut down during routine maintenance or any unexpected power failure.

The city had scheduled a shutdown for regular maintenance last Saturday, but experts caught the problem in time and transferred data to permanent files, del Rosario said.

"He had a malicious intent to destroy the entire network," the prosecutor said.

I think I know now why his bail wasn't reduced, but it doesn't really have much basis in reality, and again, I have to question the city's take on events.

Last Friday, I ran a story that described how Childs had refrained from saving configurations to flash in remote-site routers, ostensibly for security reasons. This isn't something that I would consider to be appropriate, but from what I can tell, Childs was quite paranoid about network security, and thought differently. My source detailed having this discussion with Childs, and the eventual outcome was that Childs agreed that disabling password recovery on those routers would suffice, and would prevent problems if power was lost. This is exactly what the city claims to be "malicious intent to destroy the entire network". Frankly, a CCIE-level engineer has a million far more insidious ways to "destroy the entire network" than simply not saving configurations to flash.

Again, I'm not in San Francisco, I haven't seen this network with my own eyes, but this still seems to be blown way, way out of proportion.

And as far as the city "shutting down the network for regular maintenance" last Saturday, I can guarantee that no network administrator worthy of the title would dream of powering off a network device that is working perfectly, but to which they cannot log into. Further, unless that administrator was going to be physically replacing the router or switch or adding hardware to the router or switch, there would be no reason to power off the routers during "routine" maintenance. And if that were the case, there's no way that they would be able to complete that maintenance without being able to log into the device.

And perhaps most curious, if "experts caught the problem in time and transferred data to permanent files", then they had to have administrative access to those devices, or current copies of the configurations of those devices. But that can't be possible given the other statements made by the prosecution -- last Saturday, they didn't have the passwords, and they've publicly stated that they don't have the configurations.

It just doesn't make any sense. They can't have it both ways.

I really wish that someone with the technical expertise to understand what's happening in San Francisco would be involved in this investigation, and would provide some accurate technical information. My theories and suppositions from 3,000 miles away aren't nearly enough.

Posted by Paul Venezia on July 23, 2008 05:07 PM

July 23, 2008 | Comments: (0)

Childs' motion for reduced bail has been denied

Terry Childs remains in jail for lack of $5 million bail. Apparently, the city claims that there are still three networks that remain "locked", and bullets were found in Childs' home during the police search. The city also brought up his 25-year-old felony conviction.

But to keep him on $5 million bail, there has to be more to this story. Has to be.

Posted by Paul Venezia on July 23, 2008 02:30 PM

July 23, 2008 | Comments: (0)

Still not getting it right

The San Francisco Chronicle ran a story today on Mayor Newsom's intervention in the Terry Childs case. Unfortunately, they continued the spate of inaccuracies surrounding this case:

"But there was a snag, Ballard said - the code that Childs supplied to Newsom didn't function immediately. Newsom had to call back the attorney, who provided more information, and the system started working, officials say."

It's reasonably likely that the reason it didn't immediately work was that there were ACLs on the vty consoles for each router, and they had to telnet/ssh in from a specific subnet. But I think it should be made perfectly clear that the system was always working. Nowhere have I read that network was anything less than fully operational throughout this event. All that occurred here was some clarification on the necessary requirements to access the router and switch consoles. It has no bearing on the normal operation of the network, the data, applications, or services provided by that network.

Perhaps the best way to describe this whole thing is to imagine that the city hired an engineer to build a very high-powered engine for a car. When the engineer was finished, he looked around and felt that nobody else had the requisite knowledge to maintain the normal operation of the engine, so in order to make sure nobody messed with it, he locked the hood and kept the only key. The car would continue to be fully operational, but only he would be able to access the engine compartment. What Newsom did, then, was to get the key to unlock the hood, and had to call back to get more information on how to use the key.

This is a very, very important distinction.

Posted by Paul Venezia on July 23, 2008 02:30 PM

July 23, 2008 | Comments: (0)

The network as art

On Sunday, I wrote a blog post titled "Distillation" in which I said:

"It's quite difficult to accurately convey the stress and effort required to build and maintain large complex networks to those with no real frame of reference. I've done it for years, building networks for city governments, universities, hospitals, and private companies. At some point, a network moves beyond "straightforward" complexity, and almost becomes a work of art. Whether it's a clever iBGP VPN failover for a large MPLS-based WAN, an OSPF-based ISDN dialback configuration, or a novel method of route injection through a third-party cloud, there are instances where network architects and admins need to color outside the lines to provide a needed service or measure of redundancy. It's at this point that the proverbial wheat is separated from the chaff in terms of network administration."

I've felt this way about several of the networks that I've built in the past -- they transcended the mundane and became basically a work of art. Terry Childs also felt this way, because he applied for and received a copyright in June 2007 on the configuration of the FiberWAN as technical artistry. This would back up my contention that Childs' felt that what he had created couldn't be understood or maintained by anyone else. After all, would Picasso let anyone else work on one of his paintings?

More information coming to light shows just how in the dark his managers really are. In the arrest warrant, several key details are presented as evidence of malfeasance on the part of Childs. These include a detailed description of an analog modem and a DSL modem that were discovered in a network cabinet that he had built, and another analog modem attached to a desktop PC that he had installed. The description in the arrest warrant introduces these devices as evidence that Childs had added backdoors to the FiberWAN. Further on, the inspector describes an event in which Childs' pager was taken from him, and shortly thereafter, the pager went off with a message described as having come "from one of the routers on the network". This event was presented as evidence that Childs "still had administrative access to the network", and was probably a very important "fact" that helped convince the judge to sign the warrant.

The fact that this information is in the arrest warrant underscores the fact that the city truly doesn't understand anything about this case. According to Childs, one of the modems was in place to perform dialback services to provide him with emergency access to the network, and was installed following an outage event that was extended due to the lack of such access. Further, it was installed with the full knowledge of his managers. Also according to Childs, the other analog modem was hooked up to a desktop system running What's Up Gold, a network monitoring tool. This modem was used solely to send warning messages to Childs' pager when problems occurred on the network -- it's more than likely that this is the same modem that called Childs' pager after he had surrendered it to his management.

I find it deeply disturbing that both the inspector that prepared the arrest warrant and affidavit, and the "expert" brought in to help the city with this situation did not understand the actual purpose of these items, and yet are apparently still involved in the investigation of this case. I find Childs' description of these two modems and their purpose to be far more realistic than the description in the arrest warrant affidavit.

The DSL modem is slightly more curious. If it was connected to a raw pair (or a BANA circuit), where's the other end? If it was connected to an ISP, providing Internet access or a path through which to access the network, I find it hard to believe that nobody else knew about it. After all, unless Childs was paying for that circuit from his own pocket, the bills had to go somewhere, and ostensibly, somebody had to sign off on it. More information is needed on that one.

Unfortunately, it appears I might have an answer to some of my questions from my post on Sunday, namely "If the FiberWAN network is as complex as it appears to be, are there CCIE-level forensic networking experts employed or contracted by the San Francisco police department?" The answer would be that no, there aren't. The people tasked with investigating this case appear to be woefully ignorant, and lack basic understanding of how enterprise networks are constructed and maintained. This isn't necessarily a knock on the SFPD -- there's no realistic expectation that they should have this level of expertise on staff. They should, however, contract with skilled engineers that can provide that service for them.

I suppose I should be surprised, but I'm not. Now that Mayor Newsom has swept in with his superhero cape and retrieved the passwords from Childs, I do hope that at some point I'll be able to interview Terry himself, and get a more detailed version of the technical details of this case, since it certainly appears that they are being pushed aside.

Posted by Paul Venezia on July 23, 2008 09:40 AM

July 23, 2008 | Comments: (0)

This?! This is the DNS flaw?

I casually read Halvar Flake's post speculating on the nature of the DNS flaw this evening. Everyone and their brother appears to be in panic mode over this, but I was blown away by the simplicity. Halvar might have missed a small detail or two, but apparently, he got it more or less correct. But there must be more to it than this, right?

If not, then could it be that such an obvious flaw has been overlooked for more than twenty years because it's so ridiculously simple? Everyone that should have known and/or fixed this missed it due to it's simplicity, yet someone with an inquiring mind yet little knowledge of DNS can figure it out because they have no prior experience with the protocol? If you had detailed this to me a few months ago, I would have probably said that there's no way it would work, simply because it's so, well, simple. Surely, it couldn't be possible. But apparently it is, and using the dig test from dns-oarc.net, Neal Krawetz found that quite a few provider DNS servers are still unpatched.

Frankly, I'm not sure whether I should be shocked or start looking for Alan Funt hiding behind a plant.

Posted by Paul Venezia on July 23, 2008 12:29 AM

July 22, 2008 | Comments: (0)

Some interesting points on the Childs case

As I mentioned in my post on Sunday, my inbox has been quite busy recently. I've received several notes from past colleagues of Terry Childs, some who worked with him well before he was employed by the City of San Francisco, some more recently. Each one of them portray him in a positive light, and universally refer to him as a gifted network engineer.

Other emails offer some other interesting points of view. I received note from Richard Childers that definitely struck a chord. In pondering this situation, he reflected that most organizations actually demand an above-and-beyond attitude from their top network architects and admins:

"... search Craigslist's 'Jobs' section for the keyword "ownership". I see 674 references to the word, the majority of them in the IT-related industries.
Sure, it's a buzzword, but it's also a way of life for many IT professionals. We are paid to TAKE OWNERSHIP. We get bonuses for seeing problems and fixing them - also known as BEING PROACTIVE."

His point is well taken. He also offers some justification for withholding sensitive information from management:

"I think it is perfectly acceptable to resist turning that information over to someone who is going to keep it in an unsecured spreadsheet, on an unsecured laptop that she carries home with her, on BART. Security is only as good as its weakest link - and I'm guessing that insuring the city government's security was Childs' job description. He probably held himself to a high standard - and wanted his management to do the same."

To me this might be taking things a bit too far, but without actually being there, who can really say? If an admin gives in to a management demand for sensitive network login information, and that information is subsequently leaked and used to compromise the system, we all know who has the responsibility of fixing everything in the aftermath, but who gets the blame?

Childers also takes me to task:

"It would be nice if you stood apart from the crowd and STOPPED hedging your bets by saying things to make it clear that you're NOT defending this poor guy, and just make it your JOB to get in touch with him and to tell HIS side of the story."

I've tried to arrange for an interview with Terry, but his attorney isn't letting anyone talk to him, apparently. As for me, I'm still trying to get all the information. While the details are certainly clearer this week than they were last week, I'm not certain that we've heard the whole story yet. For instance, the city still needs to come up with a statement on this case that makes some technical sense. Until then, I don't think that taking a firm position one way or the other is necessarily a good idea.

On another note, I'm in the process of putting together a post with more information on the details of this case, including pertinent information on the specifics of the criminal complaint against Childs, and his responses. It certainly makes for interesting reading.

Posted by Paul Venezia on July 22, 2008 03:09 PM

July 20, 2008 | Comments: (0)

Distillation

My story on Terry Childs and the San Francisco City network "hostage situation" ran late in the day on Friday. Since then I've received numerous emails, some offering more information, some thanking me for providing some clarity to what is still a murky subject, and some asking more questions than I have answers for.

In order to keep things together in my mind, and perhaps for those looking for information on this story, I'm going to attempt to relay what I know so far, including some new information, some hypotheses, and -- unfortunately -- some more questions.

Now that we have some background on Terry Childs, he seems somewhat familiar. In my life as an enterprise consultant, I've seen more than a few IT admins that fit his mold -- extremely intelligent individuals that lack the social tools or the will to conform within a highly political environment. The best thing that could possibly happen for people of this nature is to be left alone, trusted implicitly, and basically protected from the political machinations that seem to consume some organizations. This type of employee can be extremely valuable, but also tend to have short fuses, very little patience for the mundane, and are a high risk for burnout.

In fact, I must admit that I may share some of his traits, including the reluctance to allow marginal admins access to core network resources. While some may think that this is hubris or arrogance, in my case, it's based on experience. There are few things more frustrating than to complete a very complex network implementation only to have another admin blow it up through ignorance or incompetence. It's happened to me many times.

It's quite difficult to accurately convey the stress and effort required to build and maintain large complex networks to those with no real frame of reference. I've done it for years, building networks for city governments, universities, hospitals, and private companies. At some point, a network moves beyond "straightforward" complexity, and almost becomes a work of art. Whether it's a clever iBGP VPN failover for a large MPLS-based WAN, an OSPF-based ISDN dialback configuration, or a novel method of route injection through a third-party cloud, there are instances where network architects and admins need to color outside the lines to provide a needed service or measure of redundancy. It's at this point that the proverbial wheat is separated from the chaff in terms of network administration.

I get the feeling that Terry Childs subscribed to this theory, and at some point felt that there was nobody else that could be trusted to understand what he had built. As the lone CCIE, he was apparently well beyond his peers in skills and knowledge, and felt that he, and only he, could handle what he had created. Of course, this should have been noticed by his colleagues and dealt with by his superiors, but it obviously wasn't. If it was widely known that he was the only one with the logins to the FiberWAN network, then I find it very disturbing that this particular problem hadn't been solved well before now.

Also at issue is the way that the city dealt with the problem once it finally noticed. Generally speaking, you don't have your highest-level network administrator jailed for computer tampering if no actual damage has been done. Even a civil proceeding seems harsh given the circumstances, but to bring criminal charges? Unless there are large parts of this story still to be told, that seems like a very extreme measure to take for what appears to be simple insubordination. I would also think it unadvisable to make public statements claiming "millions of dollars" in damages and consistently confusing the network with the services, applications and data that ride on top of it.

To reiterate the technical details as I understand them, the city's network is functioning normally. There are some number of network devices that cannot be accessed by administrators, but all of the applications, data, and services that the network supports are fully operational. I want to make this perfectly clear: The actions of Terry Childs do not appear to have caused any disruption in normal operation of any city resources. Obviously the lack of access is a significant problem, but it is not impairing the normal functions of the city at this time.

However, I have received some more information that may change a few minds. It may be the case that Childs did in fact modify the logins for other routers and switches within the city network, not just those under his direct purview. If this is true, and he did this in order to prove a point or in retribution for some perceived wrong, then the case may take on a different light. I have no hard proof of this one way or the other, and my original source has been silent, presumably due to the publicity surrounding this case and this story.

This brings me to my questions. I have quite a few:

Who's investigating this case? Has the network been declared a crime scene? Can a Cisco switch configuration be entered into evidence? If so, who is providing the required information to the police? If the FiberWAN network is as complex as it appears to be, are there CCIE-level forensic networking experts employed or contracted by the San Francisco police department? Who's inspecting ACLs for backdoors? Who's poring through routing tables looking for clues, or evidence of wrongdoing? How extensive was Childs' reach?

There are only around sixteen thousand CCIEs in the world. Since Cisco split the certification into different technologies, we might assume that there are maybe ten thousand individuals in the world that could match Terry Childs' switching and routing skills. Are any of them actively involved in the investigation of this case? If Childs has in fact been offering to give the city access to the network since last Tuesday, as his lawyer claims, why hasn't this happened by now?

In short, how on Earth does the city of San Francisco plan on prosecuting Terry Childs? How much will it cost for them to get rid of their most advanced networking expert?

Posted by Paul Venezia on July 20, 2008 09:00 AM

July 18, 2008 | Comments: (0)

Notes from the (San Francisco City IT Department) Underground

This week, I wrote a few posts about Terry Childs and the City of San Francisco's "hostage situation". Based on the smattering of information made public, there seemed to be much more -- and much less -- to this story than city officials were divulging. I postulated that this sounded like a network-level issue, and that the fix couldn't possibly cost millions of dollars, especially if all we're talking about are logins and passwords to the routers and switches that make up the city's FiberWAN network.

Apparently, I was correct.

In a private email referencing those posts, and speaking under a condition of anonymity, someone with intimate knowledge of San Francisco's IT department offered to tell me what they knew. So far, I am reasonably certain that this person is who they say they are, and the information they have given me appears to be accurate.

I'm writing the story now, with much more information on the specifics and technical details of this case than have been released anywhere, to my knowledge. As soon as the story is posted, I will update this entry with a direct link to the story.

Update: Here's the full report: Why San Francisco's net admin went rogue

Posted by Paul Venezia on July 18, 2008 01:47 PM

July 17, 2008 | Comments: (0)

The criminal, digital divide

So after I wrote about the San Francisco network hostage situation yesterday, I started thinking a bit more about this situation. Based on all the data I've found in the press, it certainly appears that Terry Childs changed the passwords on some number of network devices within the city's network. If there's more to this story (such as the rumors that he was a DBA and this was an Oracle sabotage job), then we move into a completely different ballpark. As it stands now, using the information publicly available, what Childs has done could be considered a juvenile prank, not an attempt to sabotage the network and cause real damage.

Again, we're assuming here, but even if we remove the specifics and make this a hypothetical case, there are many, many miles between changing the passwords on the core and edge switches and, say, dropping a dozen databases.

Unfortunately, to the public at large, there isn't much of a difference. To a normal computer user, the phrases "he maliciously altered the AAA mechanisms in the city's network to prevent access" and "he issued queries damaging to city data repositories" are basically the same thing. Of course, they're miles apart in damage done, but to folks who struggle with spyware and anti-virus tools (and sit on juries) they might as well be the same thing.

In a comment on my previous post on this issue, a user name 'der golem' summed it up nicely:

"Okay, I won't pretend I understand everything you wrote here, but there is something really alluring and provocative about tech speak."

"Alluring" might not have been the word I would have chosen, but the point is that the law deals with common crimes like theft with offense levels. There's petty larceny and grand larceny, for instance. What Childs did may actually violate any number of other laws, possibly even anti-terrorism laws since it involves a city government. If all he did was change those passwords, then it's likely that he'll be charged with crimes that don't match the events, simply because the case centers around a computer network.

Sten, another commenter on that post, had another point of view (and one that's quite common):

"Entelleghent [sic] mangers always love exaggerating the actual proportions - it's a management trick they call "risk management" - you pretend to have a huge problem; if the problem is small and solved fast - you're a genius hero; if the problem turned out to be complex - you can say 'I told you so'"

This is definitely true -- underpromising and over-delivering aren't bad things, necessarily, but for city government officials to do so publicly, while the man accused of the crimes is in jail isn't really appropriate.

Again, this is all speculation and hypotheticals since I don't have enough information on the specifics of the Childs case to come to any meaningful conclusion. I would love to have more information on this case, however. If anyone has anything more detailed than what's been released to the press, I'd love to hear it.

Given the facts known, Childs certainly did something he shouldn't have, but unless he dropped a logic bomb in the network, it's barely a bump in the road.

If you really wanted to make a point and mess up the network, there are many better ways to do it. You could place a box near the core somewhere that randomly swaps bits in the datastream. That would certainly cause problems, but would also be discovered quickly.

Better yet, write a few database queries that randomly swap numbers and letters in various database fields. If that script started out slow and then grew in scope over days and weeks, it's likely that by the time the problem was discovered, most of the backups would already be tainted, and anything using that database would be basically unusable. For a municipal government, the data loss and time required to fix that problem would be significant, to be sure. Most or all criminal and tax records would be compromised and chaos would ensue. Interestingly, I wrote about this very scenario several years ago. The cost to fix problems like that would carry a heavy pricetag, indeed. Maybe even millions of dollars.

Don't get me wrong -- I'm not defending Childs' actions in any way, shape, or form, I'm just pointing out that there's a world of difference between letting the air out of a car's tires and wiring a bomb to the ignition switch.

Posted by Paul Venezia on July 17, 2008 07:45 AM

July 16, 2008 | Comments: (0)

An outside scoop on San Francisco's network lockout

If you haven't heard, San Francisco is being held hostage. At least, the city's new network is being held hostage. It seems that Terry Childs, a disgruntled network admin took it upon himself to lock out all the other admins from "the city's new FiberWAN network," and is currently hanging out in jail, holding the keys to San Francisco's kingdom.

There have been many articles written about this event, and they all share an obscene lack of detail. The "network" as used in these pieces could be interpreted as just about anything from one or more servers, the network switches and routers, some storage servers, or any combination thereof. This quote from an IDG news item unfortunately doesn't offer much clarity: "The new FiberWAN handles city payroll files, jail bookings, law enforcement documents, and official e-mail for San Francisco." There are an awful lot of moving parts in that description. We obviously don't know what part(s) they're talking about. Thus, it's terribly difficult to draw a clear picture of what's actually transpired.

A clue as to the actual nature of the lockout has come from statements that it might cost the city "millions of dollars" to unlock the system.

Millions? Really?

Unless Childs managed to install BIOS or kernel-level disk encryption on all the servers or stuff M80s in the server drive bays, there's no way that the cost of "unlocking" the network would run into the millions of dollars. Since officials are talking publicly about bringing in Cisco experts to undo the damage, it may be safe to assume that what Childs did was change the login to some or all of the routers and switches running the network. Now, being a veteran Cisco network architect, I can tell you that there's no way that a network of this size should have been built with only local passwords on the switches and routers. Using TACACS+ or RADIUS to control admin logins isn't just a good idea, it's the only way to handle authentication on a network of this size. Perhaps one of these methods was in use, but Childs modified the configurations to use only local logins. We can't know for sure, but we can speculate.

In any event, if we're talking about resetting the passwords on Cisco switches and routers that are physically accessible, then we're talking about a much, much smaller problem. It takes a few minutes to powercycle a Cisco router or switch, break the boot, change the configuration register (0x2142), reboot the switch and restore the configuration (don't forget to reset the confreg!). I could probably write an expect script to do just that in 20 minutes. Multiply the time required to either do it manually or run a script by the number of switches and routers affected, and you have an estimate of the hours required to undo the damage. My guess would be that with a few knowledgeable folks with laptops and a brief set of instructions, the whole network could be "fixed" in a matter of a day or so. There would be downtime, of course, but it shouldn't be terribly significant.

As to the claims that he might have installed a back door into the system, I can't speculate. There are too many unknowns, and apparently too many clueless people talking to the press to get any real idea of what that might mean. And the claims of him installing a "tracing system to monitor communications related to his personnel case", well, that could be as simple as a keylogger or a SNORT box in the right place, or as complex as custom code running on a server somewhere. My guess is the former.

Granted, there are a lot of assumptions leading me to the conclusion that city officials are playing Chicken Little, but based on everything I've read so far, they're certainly making a mountain out of a molehill. Heck, I'd fly out there and fix the whole thing myself for only $500,000. What a bargain!

Posted by Paul Venezia on July 16, 2008 01:44 PM

July 16, 2008 | Comments: (0)

An outside scoop on San Francisco's network lockout

A clue as to the actual nature of the lockout has come from statements that it might cost the city "millions of dollars" to unlock the system.

Millions? Really?

Posted by Paul Venezia on July 16, 2008 01:44 PM

July 09, 2008 | Comments: (0)

...and justice for all

If you've been following my tale of Google AdSense woe, you know that just under a month ago, Google disabled my AdSense account for no apparent reason. I didn't even notice for four days, as I was traveling at the time and buried in the midst of a major network core restructuring project, but when I did notice, I filled out their dispute form. Several weeks later, I was notified that my appeal was denied, and I was still persona non grata. I filled out another dispute form following that rejection, but had resigned myself to a lifetime bereft of the opportunity to use Google's AdSense service, since they apparently have a one-strike and you're out policy.

So imagine my surprise when I received this today:

Hello,

After thoroughly re-reviewing your AdSense account, we've decided to reinstate your account. However, there will be a delay before ads start running on your website, as it may take up to 48 hours before all of our servers are informed of the change.

We've also applied a credit of $45.40 to your earnings for this month, reflecting your valid earnings prior to the account disabling. You'll be able to see your finalized earnings for this month when they're posted to your account's Payment History page during the first week of next month. [...]

Apparently, I'm now back in Google's good graces, and they've even seen fit to replace the (admittedly small) balance in my AdSense account. Sometimes, justice does prevail.

After this fiasco, I'm wondering what I can do to not run afoul of Google's fraud detection again. I'm still completely in the dark as to why this happened in the first place, so I'm not really sure how to prevent an unknown event from occurring again, but I'm thinking about adding some code to my sites to prevent the Google ads from appearing if I'm either logged into the site or visiting the site from my normal external IP address. This will obviously add some overhead to the sites, but if it provides some protection against having my account disabled again, it may be worth it.

This trip to the darker side of Google has certainly been educational. I'd never given much thought to how Google deals with click fraud in the past, but it's definitely been on my mind the past month, and it does seem to be a significant challenge. I'm guessing that Google flags large numbers of ad clicks from a single IP address or range of IPs within the same subnet, and compares them against other actions from that IP, such as logging into a Google account, be that a Google Analytics, GMail, or AdSense account. It may possibly geolocate the offending IPs and compare them to the geolocation of a known IP that has logged into the Google AdSense account in the past, and then punts to a human to make the final determination. This process would certainly not be foolproof, especially with superproxies, tor, open proxies on the Internet, and the myriad other ways that IP addresses can be masked, but those problems would seem to pale in comparison to those introduced by malware, or over-eager legitimate software, like the AVG pre-clicking debacle I discussed earlier this week.

I wonder what chaos might ensue if a virus or piece of malware bent on "clicking" every Google ad it sees became prevalent. I suppose that dealing with those problems is part of the cost to be the boss. If my adventures are any indication, it's certain that Google is quite vigilant in protecting their advertisers -- indeed, perhaps a little overzealous.

Posted by Paul Venezia on July 9, 2008 10:21 AM

July 07, 2008 | Comments: (0)

Who's clicking who?

I read this article on The Register with interest this weekend. It seems that the latest version of AVG anti-virus has implemented a "feature" that clicks links on Web pages for you, scanning the resulting pages for malware. In theory this might be considered a good idea. In practice it's a terrible idea. The concept is that by clicking all those links for you (and apparently they're limited to search results, but who really knows?), AVG can better protect the user from malware-laden links. The obvious problem is that AVG uses standard browser identification strings to do this, so each click is indistinguishable from an actual user click. Thus, when using AVG 8, you litter logfiles with fake clicks, and cause bandwidth utilization to rise on sites that you aren't even visiting. Website statistics become relatively useless since they're not accurately showing user actions, and perhaps more importantly, it may be that using this tool and visiting your own site can cause clicks on your own Google ads, unbeknownst to you. Further, other users that visit your sites may be clicking on all the ads as well, even though they're not actually clicking them.

I can't yet verify that this is true, however, and AVG has apparently announced that they will stop this practice, but there are millions of installations of AVG out there that will continue performing this operation until they're updated. There may be other applications out there doing the same thing, but with a smaller install base and thus haven't received attention.

If you've been following my Google AdSense account suspension saga, you know that I have no idea why my account was disabled, since I never violated any of their rules. They won't tell me what their reasoning was behind the account suspension, nor will they disclose any information about it whatsoever.

My guess is that my account was suspended for fraudulent clicks. Presumably, that would be a person clicking their own ads to drive up their AdSense revenue. But with "features" like this example in AVG, it seems that no user intervention is required to click those ads. In fact, the user never knows it's happening. If tools like this are clicking every link and supplying a valid browser ID string, Google's AdWords model goes out the window, as there's no way to accurately determine a user-generated click versus an automated click. Thus, advertisers are paying for clicks that never reach the user, and it's entirely possible that AdSense accounts will be disabled even though the owner of that account has nothing to do with it.

I'm trying to figure out if there's a moral to this story, but I'm coming up short. Maybe "Damned if you do, damned if you don't".

Posted by Paul Venezia on July 7, 2008 10:05 AM

July 01, 2008 | Comments: (0)

Google speaks, sort of

A day after I posted the last update in this sorry saga, Google deigned to communicate with me:

Hello,

Thanks for providing us with additional information. However, after
thoroughly reviewing your account data and taking your feedback into
consideration, we've re-confirmed that your account poses a significant
risk to our advertisers. For this reason, we're unable to reinstate your
account. Thank you for your understanding.

As a reminder, if you have any questions about your account or the actions
we've taken, please do not reply to this email. You can find more
information by visiting
https://www.google.com/adsense/support/bin/answer.py?answer=57153.

Sincerely,

The Google AdSense Team

This is obviously a form email, and has nothing of substance -- like an actual reason for the account suspension. Also, under the same account name, I'm a Google advertiser with AdWords. Apparently, I pose a significant risk to myself as well. Also, Google has been very keen on charging me for the AdWords ads, and has taken the money my sites earned before my account suspension. Talk about getting hit coming and going.

Posted by Paul Venezia on July 1, 2008 01:36 PM

June 30, 2008 | Comments: (0)

Google's response, or lack thereof

It's been 18 days since Google decided to disable my AdSense account. Having no Earthly idea why they did this, I've followed their instructions and filled out all the forms, wondered aloud about why they might have done this, and postulated that it might be stupidly simple for anyone to cause this to happen to others. It's all been for naught, I'm afraid.

Although the dispute form I filled out claimed a 48-hour response time, I've heard nothing from them. I gave them my phone number, email address, carrier pigeon route ID... everything I could think of -- but nary a whisper.

It appears that I have no recourse at this point. None. They've disabled my account permanently, as in a life-time ban -- apparently I'll never be able to use AdSense again for the rest of my days.

They've taken the money earned from my sites in the month or so since I received the only payment they ever sent me, and haven't given me any justification, asked any questions, offered any insight, or made any attempt at contacting me whatsoever.

It's a lot like a mugging.

Posted by Paul Venezia on June 30, 2008 01:32 PM

June 19, 2008 | Comments: (0)

Musings on AdSense and DoS attacks

So 72 hours have elapsed since I contacted Google about the abrupt suspension of my AdSense account, with no response. I've gone through the logs on the servers that run the three sites I had AdSense ads on, and the statistics are underwhelming. Two of those sites have had a few hundred visits this month combined, and the third isn't much farther ahead. There was nothing suspicious in any of the logs (and remember, I use Google Analytics for all these sites, so they know this as well). I used those ads to defray the cost of hosting those sites. They certainly didn't pay for the hosting -- those sites have always run in the red. But it's all small potatoes.

So I started thinking about the whole picture. Since I've done nothing to cause this action, others must have (or Google's got some serious click detection coding issues) -- but how?

I cannot log back into my AdSense account, but I'm sure that someone can verify this for me -- you can take your AdSense invocation code and drop it just about anywhere, on any page, and it will work. I vaguely recall a setting in AdSense that will allow you to limit the sites that can display your ads, but the default is that this function is disabled. Thus, anyone can copy the JavaScript AdSense code from your site and use it on theirs. Common sense would say that this shouldn't be much of a concern because if someone were to do that, they wouldn't get paid for those ad impressions and clicks. However, if someone were to do this and start running up the clicks and impressions on their own site using your Google AdSense code, from what I've seen, you'd likely have your AdSense account suspended permanently.

Thus, this is a highly effective DoS attack that hits anyone using AdWords directly in the pocketbook -- permanently -- as in "you'll never use Google AdSense ever again, ever".

Someone please tell me I'm wrong here... I can't verify this without creating another AdSense account, and I'd prefer to wait and see what comes from my dispute for before I do anything involving AdSense. But I don't think that I'm wrong. So all it takes is some chucklehead clicking all the ads on your site repeatedly, or pulling the JavaScript code from your own pages and putting it on theirs (and doing the same) to get you a lifetime ban from Google AdSense. That's it. Sounds like an easy way to hurt your competition, or as retribution for some perceived wrong.

So much like eBay's feedback system's flaws were shown by the seedy side of humanity in the form of shoddy sellers giving retaliatory feedback, Google's got AdSense problems. Given that AdSense is the financial core of their business, I'd think that they'd want to protect it (and their users) a bit more than this. Oh, and one more thing -- when your account gets disabled, any earnings between your last check (or in my case, my only check) and the time the account is disabled disappear completely. The icing on the cake, so to speak.

Posted by Paul Venezia on June 19, 2008 09:05 AM

June 18, 2008 | Comments: (0)

Do no evil, continued

So it's been six days since Google disabled my AdSense account without warning, and 48 hours since I filled out their response form. I've heard nothing from them in that time. Based on a comment on my original post, I started digging around to see what other folks were running into, and found adsenseaccountdisabled.org, a site that has some more information on these occurrences, and forums for those that have been hit by it. Apparently, based on the text of the email I received, I've been flagged for violating their T&C for landing pages. Of course, this is all news to me, as I haven't made any changes to any ads on any sites in four months, and prior to that, in almost a year. Again, I'm completely flummoxed as to why they would have disabled my account.

adsenseaccountdisabled.org doesn't paint a rosy picture of the future, either:

Can I rejoin AdSense?

The short answer is “No, never as yourself again”. Google made it clear in the Disabled Account FAQ,

We understand your concern about the actions taken against your account. Please know that our actions are the result of careful investigation by our team of dedicated specialists, taking into account the interests of our advertisers, publishers, and users. Though you may be disappointed with our decision, we are unable to reinstate your account.

Please also note that publishers disabled for invalid click activity are not allowed any further participation in AdSense. For this reason, these publishers may not open new accounts.

So apparently, there's little to no recourse.

So let's open this up. Google, please tell me why you disabled my account. If you're feeling generous, tell me why there seems to be a growing number of people using your service that suddenly find themselves in the same boat. You must have some justification for your actions, so let's hear it. Give me at least an inkling of why you chose to do this -- some proof, some information, something to explain this. I know that I haven't done anything overt or covert, and from reading the T&C, I'm reasonably sure that I haven't violated any of the terms. If somehow there is a problem with a site I have, let me know, and I'll be more than happy to fix it.

This is all small potatoes -- these sites generate maybe $10-$20 a month in ad revenue collectively -- but it's the principle that's important. Google has every right to deny service to whomever they choose, but generally speaking, they should tell them why, and give them a fair chance to respond. Instead, they appear to be using a zero-tolerance policy coupled with mandatory capital punishment.

I'm all ears.

Posted by Paul Venezia on June 18, 2008 10:06 AM

June 16, 2008 | Comments: (0)

Do no evil?

Like many, many folks the world over, I've taken to using Google as my one-stop shop for Web elements. Ads through AdSense, advertising through AdWords, and analytics through Google Analytics. The tools are very well designed, fast, and provide just about everything I need for the small collection of sites that collectively handle maybe a few thousand hits a day.

In fact, just two weeks ago, I received my first check from Google for $124 -- representing over a year of their ads displayed on my low-traffic sites.

So imagine my surprise when my account was disabled last Thursday. From the email:

While going through our records recently, we found that your AdSense account has posed a significant risk to our AdWords advertisers. Since keeping your account in our publisher network may financially damage our advertisers in the future, we've decided to disable your account.

Please understand that we consider this a necessary step to protect the interests of both our advertisers and our other AdSense publishers. We realize the inconvenience this may cause you, and we thank you in advance for your understanding and cooperation.

Wow. I've received a smackdown from Google. I should be chastened. The problem is that I have absolutely no idea what they're talking about.

This happened last week, apparently. I can't log into my AdSense account, and I haven't logged in for weeks, so I'm completely in the dark there. Google won't tell me what has happened, so I can only guess that this is all a big mistake, or someone's gaming me. After all, how hard would it be to write a script that transits one or more sites running Google ads on the same account, and clicks all the ad links? Not terribly challenging. What would the end result be? Google would shut down the account, apparently. Now, I have no idea if that's what happened here, but when you're erroneously accused of click fraud, you start to think about things like that.

I've filled out Google's dispute form, and we'll see how that goes. They cautioned me that it may be 48 hours or more before I get a response. Judging by when they disabled the account, the supposed malfeasance occurred while I was in the middle of a road trip. I was orchestrating major surgery on a large network -- two nights of Cisco Catalyst 6509 core replacements. Believe me, I didn't have time to bother with nonsense like this. Heck, I didn't even notice the account was disabled for four days.

Regardless of the outcome of this dispute resolution, I realize now that I'm not going to be so Google-gung-ho in the future. For the past few years, using Google's tools for Web development, advertising and monitoring has seemed to be a no-brainer. Easy, simple, fast... who wouldn't use them?

I think I'm starting to realize the answer to that question -- what happens when they decide they don't like you?

Posted by Paul Venezia on June 16, 2008 11:06 AM

June 08, 2008 | Comments: (0)

It's way too early for this...

My nifty plan of a sleep-in Sunday morning was destroyed by the puppy, unfortunately, leading me to traipse around the neighborhood with bedhead, Birkenstocks, and a bathrobe at 7am. So much for that.

Once I had corralled the dog, I came back in to grab a cup of coffee, and check my email. I found a love note from Microsoft Live/Hotmail, informing me that they had rejected a legitimate email from me to a Hotmail recipient for policy reasons. I suppose this is a better reaction than simply eating the email and not telling anyone, but barely. Their SMTP 5.1.1 error contained a link to contact them, so I did. That link (http://postmaster.live.com/) contains some interesting wording and interesting links, including this statement:

Q. Does Windows Live Hotmail operate an “allow list” that I can get on?

A. No. An allow list is essentially a “free pass” that allows e-mails from certain senders to bypass junk e-mail filters and other precautions. Windows Live Hotmail evaluates all inbound e-mail for malicious content. You can find out more about our filtering processes here.

There are a bevy of oddly-constructed pages that detail the myriad steps to be taken to actually send email to a Hotmail account. Then, there's the feedback form, where you can detail why you should be allowed to send email to Hotmail and Microsoft Live accounts. This feedback form assumes that you're running a mailing list (I'm not) and that you have a valid MX/DNS reverse record for the sending server.... except that this isn't possible when dealing with virtual domains on a single server, or multiple domains hosted behind a single egress IP. You can see the form right here.

Now, recall that I've already had to write custom destination rules for my mailserver to ship all Hotmail/Microsoft Live mail through my provider's SMTP gateway so that they wouldn't simply delete it. Now, they're rejecting even that email. How can you possibly justify this? Why bother providing the service at all if you're going to regularly change the rules and continue to delete legitimate inbound email?

For the record, the servers I run are not, and have never been involved in a bulk email/spam operation. Ever. Not once. They're not in any DNSBLs, or any other unsavory place. They are connected to the Internet via a business circuit, not a residential DSL or cable connection. The volume of mail from these servers to Hotmail/Microsoft Live is probably 0.25 messages per day.

Yet they can't send to Hotmail, even through a large volume, legitimate relay.

To further the absurdity, after filling out the Microsoft Junk Mail Reporting form I mentioned above, you get a page containing a ticket number, and this text:

"To make sure that you can receive a reply from Microsoft, add the "microsoft.com" domain to your e-mail "safe list". If you do not receive a response in your "inbox" within 24 hours, check your "bulk mail" or "junk mail" folders."

Oh, I see. Microsoft won't run an allow list, but I have to put them on mine to make sure that their response actually gets to me? It's a good thing that I didn't enter my Live.com or Hotmail.com address in the form -- we've already determined that receiving mail sent to those accounts is at best hit-or-miss. Again, it's painfully clear that Microsoft Live/Hotmail is simply not worth the time. Get Gmail, and get thee behind me, Hotmail.

Posted by Paul Venezia on June 8, 2008 07:40 AM

April 14, 2008 | Comments: (0)

ComCastastrophe Resolved

A few months ago, I wrote about a ComCast Business circuit that had yet to be installed, despite the fact that the contract had been signed and initial fees paid six months prior. It was a sad tale, and one without resolution -- until Friday.

Yes, on Friday, April 11th, just over eight months after the contract was signed, the circuit went live. Although ComCast had claimed several times during those months that the circuit would be installed "in two weeks", the last time they made this claim, they actually delivered -- two weeks to the day after hearing those words yet again, packets were passed, source routing was configured, and there was much rejoicing.

I've long thought that cable and DSL services make for solid overflow Internet bandwidth for many companies. While the tried-and-true T1 is still the best bet for reliable, synchronous bandwidth, they're too expensive for basic Internet browsing these days, with the relatively low cost of high-bandwidth cable and DSL circuits. Many firewalls can support multiple egress links, and many switches can handle the source routing tasks necessary to use two different circuits -- heck, in the absence of either of those, a really low-spec Linux box can handle the source routing -- so it makes all kinds of sense to go this route.

Of course, that's assuming that it doesn't take the better part of a year to install, but I would hope that is the exception and not the rule.

Posted by Paul Venezia on April 14, 2008 09:29 AM

April 14, 2008 | Comments: (0)

How you know if your IT department is doing it right

This one's easy. A good IT department is generally kinda bored.

When the infrastructure has been designed and built correctly and the telemetry is just right, IT doesn't have much to do except keep an eye on things and work on new projects. Sure, there are always break/fix scenarios, but those are par for the course. Unless there's a major project underway, it's the "insanely busy" IT departments that are the cause for worry, not the ones playing Nerf football. As a consultant, I've been involved in projects with hundreds of different companies and seen just about every form IT can take. This theme crosses all boundaries.

My theory of good IT is that the best network and system administrators are the laziest. When presented with a problem that will require lots of small modifications to lots of moving parts, they will always opt to write some code to automate the process. This generally takes less time than the manual effort, and the resulting code can be reused in the future. In many cases, this will require that the admin learn a new language, or at least be able to think abstractly in order to address the problem. Those that opt to do everything manually still get the job done, but with plenty of wasted effort and no long-term gains. To turn a phrase, they're generally too busy mopping the floor to turn off the faucet.

For instance, given a task to migrate from one firewall platform to another, there are many, many admins that would simply re-create all ACLs and rules in the new firewall. This is obviously error-prone and will take a long time if there are many rules. "Lazy" admins will write some perl to parse the config from the original firewall and generate valid code for the new platform. I've done this many times -- even published perl code to migrate PIX firewall rules from conduits to ACLs.

The best admins will design a system that will be more difficult and may take slightly longer to implement in the beginning, but will all but eliminate problems later. Those are the admins you're looking for.

When presented with a new project or new requirements, the better IT shops will look for open-source solutions or frameworks and adapt them to their needs rather than look for something they can buy that may not be as adaptable, but might be simpler to implement. That's not to say that commercial products are never used, but the first course of action isn't to spend lots of money, it's to research what's out there. These shops also don't generally use consultants since there's no real need for them. These are also the IT shops that tend to have the highest admin-to-user ratios, and the lowest overall cost.

Of course, there are downsides to the "lazy IT" method. The main problem is that the "lazy" approach doesn't play well with non-technical executives. The issue is that a well-designed and implemented infrastructure makes everything look easy. Modifications, additions, and tweaks become simple if the foundation is solid, though they can lead to disaster if the foundation is poor. In the right environment, major projects can be implemented with great speed and competency -- but giving the impression to those outside of the IT department the idea that anyone can do it.

Regardless of the stability and performance of the IT infrastructure, there are many that believe that unless the IT staff is red-faced and sweating, they're not doing their jobs. This can lead to staffing cuts, which then cause major problems when those that were most capable of maintaining a stable infrastructure are let go since "they weren't doing anything". New, cheaper staff are bought in and the stability and resiliency of the network infrastructure soon begins to falter. But those new admins sure seem to be working hard, running in circles trying to keep the roof from collapsing. I've seen this happen far too many times. Quite often, I've been the consultant brought in at a high hourly rate to perform CPR and stop the bloodshed.

To executives that lack a concrete grasp of how IT should work, a solid IT department needs to be presented as the best insurance policy available. After all, those insurance premiums don't do anything unless they're needed, but what happens if you stop paying them?

Posted by Paul Venezia on April 14, 2008 09:12 AM

March 29, 2008 | Comments: (0)

How not to deal with fraud

My friend Desmond recently got an invoice from Dell for two DLP projectors. He hadn't ordered any DLP projectors. They weren't shipped to his house, either. Instead, they went to Roosevelt Island, NY (NB: Roosevelt Island? Really?). Obviously he was perplexed and called Dell. Well, he tried anyway. While American Express handled this with aplomb, Dell, well, didn't. In fact, they didn't seem to care.

What makes this more interesting is that Desmond is the IT Director for a sizable technology company that has done significant business with Dell for many years. As he notes "When I order something from Dell for my business, they will not ship to any address than the addresses on my credit card. Why did Dell do it for this order?"

Suffice it to say, this is a very poor example of how to deal with a very troubling issue. Here's the full post.

Posted by Paul Venezia on March 29, 2008 11:08 AM

March 27, 2008 | Comments: (0)

Technology White Papers

InfoWorld Technology Marketplace

» Technology White Papers Library

Technology White Papers by Topic

Technology White Papers E-mail Alert