- Optimize the slowest thing, take 2
- Optimize the slowest thing
- TopCoder: developer productivity and code quality through competition
- Pair programming at a distance
- Is "professional software developer" an oxymoron?
- DSM tool offers high developer productivity
- Fear of bugs
- TDD with Silverlight
- Agile is as Agile does
- UI Design: Beauty, or Consistency?
July 02, 2008 | Comments: (0)
Optimize the slowest thing, take 2
My posting Optimize the slowest thing last week drew quite a bit of feedback, which made me think about the issue again. Let's start with the feedback.
DCrosby pointed out that "optimize the slowest thing" is a tenet of Theory of Constraints. He pointed to http://www.goldratt.com/ and the book The Goal: A Process of Ongoing Improvement. I wasn't previously familiar with the Theory of Constraints.
Brian said:
I don't completely agree with this. There is logic to it, but that logic is limited in scope.
My projects always try to encompass certain performance strategies from the beginning. The key is not to go overboard, but make at least a first-pass effort.The problems with reserving all optimization work to the end are:
- You work against the well-known principle that ANY design factor, introduced early enough, is easy and cheap to incorporate. Leave it too late and those factors become orders of magnitude more expensive to implement;
- Once the project is working, business realities often force attentions to switch to other projects. In the real world your "working" system may never get optimized;
- When multiple factors all contribute to poor performance in roughly equal magnitude, spending time only on the slowest thing can cause you to return to optimization work again, and again, and again, and... ;
- Moore's law makes it tempting to simply throw more hardware at the problem. This "solution" can be the design intent, in other words. There are too many example of this extant to list. However in the real world, upgrades tend to come at widely spaced intervals of years, not continuously. The net result is unsatisfied clients, also for years.
So as a backfilling effort, I generally agree with your strategy. However I wouldn't want that to be my sole performance strategy.
I answered:
I agree with you in practice. As I wrote the post, I kept thinking of times when I did optimize early on because of long experience as to where the bottlenecks were likely to be. And then I felt a little guilty about them.
Is knowing what you're doing cheating? :)
Brian responded:
From where I stand, knowing what you're doing is NEVER cheating, so long as appropriate disclosure guidelines are followed!
I was working years ago in a programming environment and had rote learned all the traditional optimization techniques. Most programmers did the same thing. Finally a technical author wrote an article about benchmarks he had performed on those techniques.
They all worked, but it turned out that most were minor techniques, barely worth using. Just a very small group of optimizations accounted for 95+% of the real world performance improvements.
It was a revelation!
This made me think of two famous stories. One was a classic optimization blunder made as I recall by Jeff Richter: he got the order wrong when traversing a big two-dimensional array in a sample application, and the performance of a simple nested loop suddenly became something like a thousand times slower than it should have been. Jeff, to his credit, turned this mistake into a very interesting article.
The other was a story from the 1960s, when a hot consultant came in to optimize a huge, badly-understood IBM mainframe application. He made a bunch of changes, charged a fortune, and left.
Years later, the application crashed. The crash dumps pointed back to some of the code the consultant had changed. When the programmers looked at it carefully, they discovered that the code never could have run without crashing. In other words, the expensive consultant had optimized, at great expense, at least one section of code that had never been called.
There are a bunch of lessons here, and there's a bunch more to say about it, but it's your turn: Let's hear more about your optimization experiences, good and bad, and what lessons you took from them.
Posted by Martin Heller on July 2, 2008 08:45 AM
June 26, 2008 | Comments: (0)
Yesterday I was writing the back cover copy for The C# Programming Language, Third Edition. As a bonus, I got to read the book in draft form. This is a great book, even better than the previous editions because it's annotated. I'm not sure when it'll be available, but you can pre-order it from Amazon.
Disclosure: I used to be Series Editor for the Microsoft .NET Development Series, so you'll find my name in previous editions of this book. I'm not doing that any more, although I do still take on occasional writing, editing, and reviewing work for those folks.
One of the annotations struck me, and I'd like to share a little bit of it with you. (I'll try to stay within the limits of Fair Use.) It's by Eric Lippert, and it annotates a section of the specification that talks about the instantiation overhead of classes and structs:
Always remember that it makes little sense to optimize anything other than the slowest thing. If your program is not gated on heap allocations in the first place then spending any time deciding whether to use structs or classes for performance reasons is not an effective use of time. Find the slowest thing, and then optimize that.
Find the slowest thing, and then optimize that. Great advice. It reminds me of the chemical principle of the rate-limiting step. It's also something that's easy to forget when you're designing a program that you want to run fast. Don't try to optimize it before you write it. First make it run correctly, then make it run fast.
You can find another take on this, possibly a bit extreme, over at Tim Bray's WideFinder 2 project. The latest posting on that is here.
Posted by Martin Heller on June 26, 2008 08:02 AM
May 09, 2008 | Comments: (0)
TopCoder: developer productivity and code quality through competition
Wednesday I had lunch with Mike Morris, Senior Vice President of Software Development at TopCoder. That's his picture at the left. He had an interesting story to tell.
I was vaguely familiar with TopCoder through their programming contests. I was never interested enough to sign up for their site, and had no idea what kind of business model they had.
Mike filled me in. He and the other founders of TopCoder had been at Tallán, a custom software development and consulting company. Tallán grew to something over 600 developers, was acquired by CMGI in CMGI's brief heyday, and then spun off again.
"At Tallán, the quality started to deteriorate as we grew," said Morris. "When we founded TopCoder, we want to figure out a way to build a software factory that would scale."
What they came up with was a community of coders built around programming competitions. Initially, TopCoder supplied the prizes: eventually, sponsors and patrons supported that part of the endeavor.
That's only half the model, though. The other half is that they use the competitions to build components and applications for clients, and to screen developers for employers. What clients and employers? Morris mentioned a few, and the site lists many more: Caliper, Direct Energy, ABB, the Salk Institute, Stage Stores, AOL, Borders, Burlington Coat Factory, IMS, VeriSign, BT, Lilly, and the NSA.
That's right, No Such Agency. I could imagine why the signal intelligence arm of the NSA would be interested in TopCoder, and I was right. "They do a great job of recruiting," said Morris. "They're restricted to U.S. citizens, but they recruit many of our best coders."
The methodology used for development at TopCoder is quite rigorous, but too complicated to cover here. Morris explains it in a series of movies online.
The TopCoder business model isn't outsourcing: it's built on global competition, and controlled by metrics and peer review. It's also market-driven. "I can tell from the metrics we have on each coder what the probability is that a component will be completed successfully by the entrants," said Morris. "If the probability is too low, we up the ante to attract better developers. There can be a factor of 20 difference in productivity between a mediocre coder and a top coder, but you don't have to pay the top coders twenty times as much."
TopCoder will be hosting an in-person competition among 120 finalists at the Mirage in Las Vegas next week, the TopCoder Open. There's $260,000 in prize money at stake.
Posted by Martin Heller on May 9, 2008 08:36 AM
May 04, 2008 | Comments: (0)
Pair programming at a distance
One of the drawbacks of working by myself is that I can't usually do pair programming. This is usually offset by the increased concentration and "flow" that comes from being by myself and undisturbed for long periods, but sometimes I just need another pair of eyes to look at my code.
Last Tuesday was one of those times. No sooner had I published my little blog entry on "clans" than I started to hear bug reports that traced back to some ActiveX code I'd written for PC Pitstop. It wasn't straightforward: two people reported "IE aborted in an unusual manner" errors, but over 8,000 people had run it successfully. I'd missed a rare case.
After looking at the relevant code until my eyes started to cross, I gave Dave Methvin a call. "Dave, do you have a minute? I think that I'm insufficiently paranoid to fix this bug."
Each in our separate offices in different states, we opened up the same Visual Studio project checked out from our version control repository, and looked at the same code. When either of us saw something we didn't understand, we'd point the other at the line of code. It was almost like pair programming, but without the physical presence. In some ways it was better, because we each had our own mouse, keyboard, and screen.
"Look at line 129. What happens if that pointer is null?"
"How could it be null? It's set in line 128."
"Suppose there were no items in the list. It's unlikely, but it's still possible."
"You're right. I'll add an 'if' test right there. OK, it compiles and I've checked it in: update your copy and have a look."
We went on like that for about an hour, and convinced ourselves we'd found every possible problem in what was, after all, not very much code. Then I rebuilt the control, ran all my regression tests, and promoted the control to production.
The ending was happy: after that, there were no more reported "IE aborted" errors, even from the people who had reported them before.
Do you have your own ways of collaborating at a distance?
Posted by Martin Heller on May 4, 2008 12:01 PM
April 21, 2008 | Comments: (0)
Is "professional software developer" an oxymoron?
I've been reading Emergent Design: The Evolutionary Nature of Professional Software Development by Scott Bain (Addison-Wesley, 2008, $49.99, 0-321-50936-6). I'm finding it a thoughtful book, which makes a good case for the adoption of patterns, refactoring, and test-driven development.
On the other hand, Bain leads with the premise that software development is not currently a profession. Why is that? It's not a matter of being paid for the work: it's because software development is too hard, too unpredictable, too chaotic.
Of course, those are the things that make it fun. Bain admits that as well.
I think it's no stretch to accept Bain's point that medicine is a profession. Bain cites some of the things that go with medicine -- lengthy training, a specialized language, a professional organization, peer review, standards and practices -- as things that define a profession as opposed to a job.
Bain says that software development is by nature a professional activity, and should be conducted as a professional activity. He also says that we're not yet conducting it as a professional activity.
What do you think? Is the phrase "professional software developer" an oxymoron?
Posted by Martin Heller on April 21, 2008 07:36 AM
April 17, 2008 | Comments: (0)
DSM tool offers high developer productivity
I should have known better than to write about a vendor demo, as I did on Monday for Telelogic Rhapsody. Now everybody wants to give me a demo. As the organ-grinder's song from The Threepenny Opera goes, "Oh, the line forms at the right, dears, now that Mackie's back in town."
This morning's demo was a Web meeting, since the vendor, MetaCase, is in Jyväskylä, Finland. My Plantronics DSP headset worked well, the voice quality was good, and the VoIP delay wasn't all that bad as long as I didn't interrupt Dr. Juha-Pekka Tolvanen, CEO of MetaCase.
Dr. Tolvanen's starting point is the proposition that Domain-Specific Modeling (DSM) offers something like a 7x productivity improvement over C++ and Java, even when those languages are enhanced by UML modeling like Telelogic's. He cited several customer quotes in support of that: Panasonic experienced a "5-fold productivity increase when compared to standard development methods"; Nokia reported module development decreasing from 2 weeks to 1 day, i.e. a 10x gain. Other customer quotes supported different perceived values: EADS cited the improved quality of generated code because of the design rules; DENSO claimed "MetaEdit+ has eliminated our need to outsource software development activities."
Dr. Tolvanen showed me a demo of a DSM for building Enterprise applications for Nokia smart phones running on Symbian (see figure at left). This particular application does conference registration. The underlying DSM knows about the capabilities and limitations of Symbian, and the code generator emits Python.
Where does the high productivity come in? Only one or two hard-core developers in the company work on the code generation; most developers (hundreds at Nokia, if I understood correctly) work on applications in a high-level design view like the diagram editor shown in the figure, or an alternative view like a matrix or table.
I've played around with Microsoft's DSL tools, and found them to require a lot of work -- weeks -- to build an effective model. I assumed that was the state of the art. Apparently not.
Here's the money quote, from Laurent Safa of Matsushita, speaking about MetaEdit+:
"I could define a domain-specific language in about six hours -- design, testing and one failed trial included."
MetaCase can be reached by email at info@metacase.com and on the Web at www.metacase.com.
Posted by Martin Heller on April 17, 2008 12:05 PM
April 02, 2008 | Comments: (0)
I used to give a talk at trade shows and seminars called Bugoholics Anonymous. I'd start with "Hi, my name is Martin, and I'm a bugoholic." The audience would invariably respond with "Hi, Martin." I'd apologize for the lack of a coffeepot (unless there actually was one in the room), and go on to talk about how I had no power over bugs, but bugs had power over me.
Then, of course, I'd present a twelve-step program for learning to deal with bugs. The talk always went over well, and often filled the venue. It wasn't just for the entertainment value, though.
What was the serious content? First, that errors in software are inevitable: even if the software is perfect when it goes out the door, the environment will change, and break assumptions that you hadn't even known were changeable and hadn't realized that you'd built into the software.
At the time, a commonly broken assumption was "memory pointers can't change." In 16-bit Windows, memory pointers could change if they weren't locked, and the best practice was to unlock memory you weren't actively using to allow Windows memory management to work.
The second piece of serious content in that talk was that you should welcome bug reports, not fear them. A good tester is a pearl beyond price. A developer who views each bug report as an opportunity to improve what is inherently imperfect code is going to be much better at debugging than one who views each bug report as a personal insult that has to be caused by user error, or as the saying goes, "PIBKAC."
Do you believe that your code is perfect? How do you react emotionally to a bug report?
Posted by Martin Heller on April 2, 2008 08:59 AM
March 28, 2008 | Comments: (0)
Microsoft has not only released the source code for the Silverlight controls, they have released almost 2,000 of their own unit tests -- and a unit testing framework that works with Silverlight.
More on Brad Abrams' blog...
I think this is seriously cool. What do you think?
Posted by Martin Heller on March 28, 2008 02:25 PM
March 04, 2008 | Comments: (0)
Scott Ambler reported at SD West that a February survey with about 600 respondents found 69% were utilizing agile development practices. "The bad news is, this is the exact same number we had last year," Ambler said. This, he said, leads him to speculate that "agile has peaked."
A similar survey at Methods and Tools reported different trends. Some of these can be explained by differences in the question asked, the times of the surveys, and the population surveyed. The question was: "At what stage is the agile approach (XP, Scrum, FDD, ...) adoption at your location?"
| 2008 | 2005 | |
| Not aware | 13% | 26% |
| Not using | 13% | 16% |
| Investigating | 14% | 14% |
| Analyzed and rejected | 4% | 3% |
| Pilot projects | 8% | 4% |
| Partial implementation (adoption of some agile practices) | 17% | 17% |
| Partial deployment (some projects are using this approach) | 14% | 12% |
| Deployed (all new projects are using this approach) | 17% | 8% |
The analysis of this survey compares it to two other surveys, VersionOne's and Ambler's:
When compared to other surveys performed in 2007 on agile adoption, the results of the Methods & Tools poll are equivalent. The adoption rate of participants is slightly inferior with 56% (48% without pilot projects) compared with 73% for the VersionOne survey and 69% for the Scott Ambler survey. Our percentage of organizations having deployed agile approaches in all projects stands at 17%, which is very close to the 15% rate that VersionOne presents for 100% agile adoption.
The real question about all of these surveys is whether the respondents actually know what they're talking about. As was recently discussed on LinkedIn, lots of organizations may label their software development practices "agile" because it's trendy, without actually implementing many (or any) of the practices recommended for any agile development approach.
What do you think?
VersionOne 2nd Annual “State of Agile Development” Survey (2007) | Scott Ambler Surveys (2007) | Scott Ambler Surveys Presented in Dr. Dobb’s (2007)
Posted by Martin Heller on March 4, 2008 06:40 AM
January 01, 2008 | Comments: (0)
UI Design: Beauty, or Consistency?
For years, one of the big selling points of Windows was its consistency: once you've learned how to use one Windows application, you've learned a lot about using most Windows applications. You could say essentially the same thing about Mac OS, KDE, and GNOME, and for that matter about Java/JWT and Java/Swing applications.
If the user expects to see File, Edit, View, and Help menus, your application will be easier to learn if those menus are present. If the user expects to see a toolbar and is familiar with a specific set of toolbar icons, then you can improve your application's usability by meeting those expectations: you're providing instant familiarity by adhering to standards.
The objections to these standards, which I most often hear from designers, are that they lead to ugly, boring applications. But often, the alternatives that they deem beautiful and exciting turn out to be hard for users to learn. I have sometimes heard myself sneering at these attempts as "eye candy," although I do appreciate attractive visuals.
In its day, Visual Basic led to a run of horribly designed user interfaces, as well as some very nice ones, by giving designers more power. Flash has done the same for Web sites, and I fear that Windows Presentation Framework will be taking junky eye candy to new levels on Windows.
What's your take?
Posted by Martin Heller on January 1, 2008 02:40 PM
November 27, 2007 | Comments: (0)
My discussion of software surprises on Saturday reminded me about the Law or Principle of Least Astonishment, also called the Law or Rule of Least Surprise. This certainly goes back to the early days of Unix, if not before: it is discussed in The Art of Unix Programming, by Eric Raymond, which is online here.
I'm convinced that the originator of the name of this law had a Physics background. The name is strikingly similar to the Law of Least Action, which is the basis of Lagrangian and Hamiltonian mechanics.
Tracking down the origins of the Law of Least Surprise reminded me that I had on my bookshelf at work signed copies of The Tao of Programming and The Zen of Programming, given to me by the author, Geoffrey James, when we were in the same Tai Ji Quan class some years ago.
The entire English text of The Tao of Programming is actually online. Half the pleasure of the book is the Chinese on the facing pages and the layout; that has been lost in the online version. Even though I've studied Chinese and still have my dictionaries, I've never bothered to translate that Chinese -- and I've never asked Geoff what I'd find if I did. I've also never asked whether he minded the online version of his book.
The passage on the 'Law of Least Astonishment' follows:
A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity.
A program should follow the 'Law of Least Astonishment'. What is this law? It is simply that the program should always respond to the user in the way that astonishes him least.
A program, no matter how complex, should act as a single unit. The program should be directed by the logic within rather than by outward appearances.
If the program fails in these requirements, it will be in a state of disorder and confusion. The only way to correct this is to rewrite the program.
Posted by Martin Heller on November 27, 2007 07:45 AM
November 17, 2007 | Comments: (0)
Danny Sabbah on Business Users as Programmers
In an interview with Chris Kanaracus of IDG News Service, Danny Sabbah of IBM Rational (whom I interviewed earlier this year) said he has little use for a recent trend: Tools and platforms that supposedly make it easy for business users to do some development.
"I don't believe any of that stuff," he said. "I've never met a business user that can use any type of professional development tool. Period." Business users are better off working within more familiar environments, such as spreadsheets and word processing programs, he said.
I wonder what the Microsoft Popfly people and other groups working on end-user and non-professional programming tools have to say about that.
Posted by Martin Heller on November 17, 2007 08:04 AM
November 05, 2007 | Comments: (0)
Can SCM Make Your Software More Reliable?
No developer in his or her right mind builds significant software in this day and age without some sort of version control or software configuration management (SCM). Free climbing (without a belay rope) and free tightrope walking (without a net) may get your pulse rate going (even if they verge on the suicidal), but it's hard to see any value at all to losing your software development efforts to an accident.
Now, one SCM system is pretty much like another, right? Not according to Damon Poole of AccuRev.
Consider the common scenario where you've branched an application to allow for maintenance of version 1 by one team while a second team builds version 2. Suppose version 1 gets 20 bug fixes before it's time for version 2 to go into alpha test. How are you going to know whether all 20 fixes were properly merged into version 2?
I thought that the SCM tools I use (Subversion and Visual Source Safe) could handle this, but I was wrong; I had to keep track of bug fixes by branch manually. AccuRev can track merges automatically, however, according to Poole.
Suppose you were a company that built many different products based on a shared software code base -- a common situation for embedded systems makers that churn out a new product revision every month, but have to maintain their existing products in the field for years. That branching scenario just became a nightmare, because the number of branches just became very large, and the importance of tracking merges just went way up. Missing a patch in one product out of hundreds can make that product unreliable; if it's a product that people depend on, that unreliability could become a major issue.
So maybe SCM can make your software more reliable.
AccuRev's site is here. In addition to the usual sales pitch and white papers, it offers a free 5-user, 30-day trial.
Posted by Martin Heller on November 5, 2007 01:45 PM
October 18, 2007 | Comments: (0)
Department of Amplification: Ben Chelf on Boolean Satisfiability (SAT)
Coverity's PR counsel just forwarded me a message from Ben Chelf, clarifying his note posted Wednesday:
Thanks for posting Ben Chelf's comments so quickly.
Upon reading the posting, Ben realized that some of your readers may be confused by this statement:
Boolean Satisfiability is the same type of technology that Synopsys, Cadence, and other EDA tools providers leverage in verifying chip designs. This new technology, which is called SAT, allows the analysis, not only of all the paths through the code but also all of the potential values of the variables in any given path.
The technology actually isn't new, it's a new application of the technology Synopsys, Cadence, and other EDA tool providers have been using to find hardware defects. What is new is its application to software development.
Here's a statement that might clear this up:
Synopsys, Cadence, and other EDA tools providers leverage Boolean Satisfiability (SAT) in verifying chip designs through sophisticated SAT-solvers that can solve formulas on millions of variables. The application of SAT-solvers to software analysis allows the analysis to cover not only all the paths through the code but also all the potential values of the variables in any given path.
Thanks again!
Posted by Martin Heller on October 18, 2007 11:02 AM
October 17, 2007 | Comments: (0)
Ben Chelf on Reliability in Complex Software Systems
Last month when I was covering ESC I asked how we can guarantee the reliability of complex embedded software systems. Ben Chelf, CTO of Coverity just emailed me a response that I think you might find interesting:
Martin,
I noticed your recap on the ESC conference. I was in Boston presenting two topics, one at ESC on the application of next generation static analysis to embedded software and one at SD Best Practices on next generation software development tools. Quite a few of your comments resonated with me. You talk about the increasing complexity in automobiles in terms of number of control units. Another thing to think about is the software code that controls those units. In fact, there are more than 10 million lines of code on a premium automobile these days. The complexity of a system like that is mind-boggling, and certainly now it's impossible for any individual or even group of individuals to keep straight in their head all at once. That's why reliability is so hard right now and your question is dead on -- how do we guarantee reliability?
The good news is there is a reasonable model for the folks developing all this software. Conventional static and dynamic analysis have been attacking this problem with some success over the past few years, but the opportunity to innovate within this space is still there. For example, if we look to chip designers, we see there are tools for chip design that are based on rigorous verification and static analysis for detecting defects in the hardware designs. The price of failure in sending a bad design for physical fabrication is tremendous and therefore, the tools are necessarily very sophisticated. It's time for us to bring that sophisticated level of analysis into the software development world.
Static analysis for software is a concept that is not new. There have been companies that have provided static analysis solutions for a long time (most notably in a tool called Lint). But in the past, the biggest problem for all of these tools for software has been "false positives" -- where the tool reports something that just is not correct. Part of the reason for these false positives has to do with some fundamental limitations in the tools that followed Lint.
Eliminating false positives is no easy task because software systems are so much more complex than chips. Today, companies are using sophisticated new technology to perform Path Simulation statically through dataflow analysis to cover 100% of potential paths through the code, in addition to leveraging Boolean Satisfiability for analyzing code. Boolean Satisfiability is the same type of technology that Synopsys, Cadence, and other EDA tools providers leverage in verifying chip designs. This new technology, which is called SAT, allows the analysis, not only of all the paths through the code but also all of the potential values of the variables in any given path. This level of precision means the latest static analysis tools can find very sophisticated defects at a tremendously low false positive rate (under 15%). And they can do all this over millions of lines of code in a matter of hours so developers get to spend their time implementing new features instead of tracking down those nasty bugs.
To see how far we’ve come with this challenge, maybe I can add some historical context. A number of your readers are probably familiar with Stephen Johnson, the creator of Lint, here’s what he had to say about false positives and software defects:
“Lint tries to give information with a high degree of relevance. Messages of the form ‘xxx might be a bug’ are easy to generate, but are acceptable only in proportion to the fraction of real bugs they uncover. If this fraction of real bugs is too small, the messages lose their credibility and serve merely to clutter up the output, obscuring the more important messages.” Stephen Johnson, 7/26/1978.
So, as all your readers know, finding software defects is not a new problem – and clearly false positive results are not either. Of course, static analysis is not a silver bullet, and software development organizations still need to test their code and that will without a doubt uncover even more defects. But by leveraging static analysis the moment the code is written, thousands of developer hours will be saved by letting the computer start to do the debugging for them...
- Ben Chelf, Coverity CTO
Posted by Martin Heller on October 17, 2007 10:16 AM
September 28, 2007 | Comments: (0)
Static Path Analysis and Embedded Systems
In my posting about embedded systems on the 19th, I asked how companies could guarantee the reliability of complicated embedded systems. One set of generic guidelines for this comes from Barry Boehm of USC: see, for example, the Software Reduction Top 10 List.
I know of three vendors addressing this in the embedded space: Coverity, Klocwork, and Parasoft. One technique that all three of these vendors use is static path analysis.
In response to my question about guaranteeing reliability, I got this email from Gwyn Fisher, CTO of Klocwork:
There are software assurance compliance standards that require mission critical systems (e.g. avionics) to provide validation that they've inspected every code path, normally through a combination of manual and automated techniques. This is about as close to a reliability guarantee that exists today, from a software assurance perspective. This is expensive but can be effective for small systems, however, for large, multi-million LOC embedded systems it becomes a more difficult problem to solve. It will always be hard to guarantee reliability for systems of this size and complexity but organizations are engaging in aggressive risk reduction strategies through the use of advanced code analysis techniques. These complement traditional testing techniques and code review methods and provide feasible and cost-effective means that allow an automated tool to at least investigate all code paths for a wide variety of potentially debilitating errors (including difficult to test areas of a system such as error handling routines). The tool makes decisions on which paths are feasible or not, hence worthy of further inspection, enabling the right balance between scalability and intelligent code and defect coverage. If a system can be made free of programming errors through this type of automation, then further along in the development process is subject to rigorous runtime testing, organizations can go a long way to dramatically reducing the risk of field failure and increasing the reliability of business and mission critical embedded systems.
Most developers spend the majority of their time on the true path. Most designers spend most of their time laying out what should happen on the true path. Peer code reviews focus on reviewing the main paths in a system. Most QA organizations spend most of their time testing that the true path does what it’s supposed to do. What about the rest of the system?
Static analysis brings the potential for agnostic coverage:
- Test the false path just as well as the true path
- Validate that the false path doesn’t lead to failure, even in the most “corner” of edge cases
- Ignore a developer’s / designer’s instinct to rigidly peer review only the most “core” of the available code paths – review all feasible paths at all timesThis should be a strong notion for embedded software, since field failure can be very expensive. With current testing techniques it's very difficult to test "edge cases", so static analysis enhances testing coverage for embedded software.
Posted by Martin Heller on September 28, 2007 07:54 AM
August 10, 2007 | Comments: (0)
Second Life Responds: A Conversation with Joe Miller
In response to my Friday posting on August 3rd about reliability and voice in Second Life (SL), Joe Miller, Linden Lab’s VP for product and platform development, called me on the phone the following Monday.
If you look at Joe's bio you'll see that he's "responsible for extending the Second Life platform to reliably support very large member communities." Joe is also personally interested in getting voice working in SL.
We discussed the problems I had encountered with voice in Second Life, batted around a few possible scenarios that might explain what happened, and nailed down the time, region, and local conditions so that Joe could have someone check the logs. Maybe some good will come of my venting after all.
We also discussed how Second Life goes about trying to achieve the balance between reliability and features that I was talking about in my blog post. Joe said that he agrees that reliability is a big question for users, but he made the point that they've been making a lot of progress. The majority of in-house Second Life developers -- Joe said 68% of them -- work on reliability, stability, and performance; major new features are usually either acquired or contracted out.
Voice was contracted out, if I understood Joe correctly. WindLight, which "simulates the ways that sunlight is scattered by the atmosphere under different climatic conditions, such as fog or haze", and Nimble, a cloud simulator, were both acquired from WindwardMark Interactive. (The five founders of WindwardMark now work for Linden Lab in Boston, but WindwardMark still exists.)
Developers at Linden Lab are organized into studios, and a couple of the studios concentrate on fixing bugs. The Blacklight studio is where all new developers at Linden work, and Blacklight has the job of fixing the top bugs each week. According to Joe, the bug closure statistics are moving in the right direction.
It certainly sounds like Linden Lab is doing the right things. Nevertheless, there have been several operational problems at SL this week, which affected the availability of the servers: these are disclosed on the Official Linden Blog. SL probably isn't at one nine reliability yet, as far as I can tell, but I hope it gets there, and soon.
Posted by Martin Heller on August 10, 2007 02:25 PM
May 30, 2007 | Comments: (0)
Managing Iterative Software Development Projects
I took Managing Iterative Software Development Projects, by Kurt Bittner and Ian Spence (Addison-Wesley, 2007, 672pp, $49.99, ISBN-10: 0-321-26889-X) home to read over the Memorial Day weekend, and curled up with it on the couch Saturday. My 13-year-old daughter asked what I was reading, so I held it up to show her.
"Managing Iterative Software Development Projects?"
"Yes."
"And you're reading it straight through?"
"Yes."
"That's sick."
"Sick" can be taken a number of ways coming from her, but based on her tone of voice I don't think it was a compliment. To be fair to myself, a science fiction novel and the latest issue of The New Yorker were also on my reading pile at the time.
Reading this book reminded me of why I have always practiced iterative, incremental software development: it's because I was trained as a physicist, not a computer scientist. My formative programming efforts (in the 1960s and 70s) were subservient to mathematical and scientific goals, so I almost automatically fell into a pattern of applying the scientific method to programming: hypothesis, experiment, conclusion, revised hypothesis... I would start with a clear goal for an iteration, and clear criteria for determining whether the program was operating correctly. The source code I wrote became the hypothesis, and running the test cases the experiment.
Computer Science was an emerging discipline at the time. When I was a Physics graduate student at Brown in the 1970s, computer science was taught out of the Applied Mathematics department; now, of course, it is a thriving department in its own right. I taught computer science courses at the Boston University Corporate Education Center for several years in the early 1990s; I have never actually taken a computer science course.
But back to Managing Iterative Software Development Projects. I found this book valuable, in that it crystallized a lot of practices that I'd learned from decades of trial and error experience. Why tackle the hardest parts of a project in an early iteration? Bittner and Spence make it explicit in Chapter 1: you do proof of concepts early to reduce your development risk. Why give the customer frequent builds to review? To reduce requirements misunderstandings early.
It was a holiday weekend, so I didn't finish the entire book: the temptation was too great to start reading the science fiction novel on my pile, and once I started that I was loath to put it down. It's Eifelheim, by Michael Flynn, and I'm not surprised to find that it was nominated for a Hugo award this spring.
I usually like to read physical books, but you can also read parts of Managing Iterative Software Development Projects for free online at Google and Addison-Wesley and Safari. You can also read all of this book and other Addison-Wesley and O'Reilly books online with a monthly Safari subscription.
Posted by Martin Heller on May 30, 2007 06:00 AM
May 21, 2007 | Comments: (0)
Jason Jones, formerly of Why Smalltalk, and now of SlickEdit, wrote the other day to tell me about a new programmer's blog, "Hello, World." My first reaction was ennui, on the assumption that it would be company fluff, but then I took the trouble to read the nine entries that have already been posted.
Sure enough, the oldest entries were basically sales pitches for SlickEdit, but then they warmed up a bit. Starting at about the end of April, the developer bloggers at SlickEdit posted some interesting essays: What is a Power Programmer?, How to Write an Effective Design Document, and do { … } while (false);, among others.
But wait, there's more!
Two days later, Earl Beede wrote to let me know that he, Steve McConnell, and Matt Peloquin are now blogging and leading discussions at Construx Conversations. Basically, Construx is Steve McConnell's company; Steve is the author of some of the most influential books about software development; Steve founded the company "to provide industry-leading support for software development best practices;" and the Construx Conversations site is "dedicated to discussing and advancing software development best practices." It's just getting off the ground, but it looks interesting.
Posted by Martin Heller on May 21, 2007 06:00 AM
TOP STORIES
ADDITIONAL RESOURCES
- Remote Access: Maintain Security and Decrease the Burden on IT
- Beyond AntiVirus: Symantec Endpoint Protection
- What Every Enterprise Needs to Know About VDI
- Monitor the core and troubleshoot the access layer
- Help Simplify Virtualization
- Solution for Open Virtualization Provides Server Consolidation
