Sunday, September 05, 2010

A Real-Life Short-Short Story About Software Craftsmanship

One day, at the end of my team's daily stand-up meeting, one person said he was cleaning out his garage. He'd found a couple of books that his previous employer gave to all new hires. And that now he would like to get rid of those books to anyone interested.

One of the books was Software Craftsmanship by Pete McBreen.

One of our managers laughed, saying, "Oh, so now you're all going to be craftsmen."

Several other people laughed, too.

After they were done laughing, I said that I thought it was a good book and I recommended reading it.

There was silence.

Monday, July 26, 2010

Does "Software Engineering" Need to be a Profession?

[This is Part 2 of a review of Steve McConnell's book Professional Software Development. Read Part 1 here.]

In proposing that software development should become a true profession, McConnell points out that for a something to be considered a profession, it must have

  • A requirement for extensive learning and training.
  • A code of ethics imposing standards higher than those normally tolerated in the marketplace.
  • A disciplinary system for professionals who breach the code.
  • A primary emphasis on social responsibility over strictly individual gain, and a corresponding duty of its members to behave as members of a disciplined and honorable profession.
  • A prerequisite of a license prior to admission to practice.

In discussing licensing, McConnell notes

Licensing is a mandatory, legal process that is intended to protect the public, and is typically administered by jurisdictions (states, provinces, and territories).

And

One of the consequences of being a professional engineer is that you can be held personally liable for the work your company performs under your signature. Courts in the United States have held that only members of a profession can be found guilty of malpractice. Doctors, lawyers, and architects can be guilty of malpractice. Garbage truck drivers, short order cooks, and computer programmers cannot be guilty of malpractice because, legally, they aren't considered to be professionals.

No individual engineer will be required to be licensed, but some companies eventually will be required to employ licensed engineers.

Professional engineers in these companies will review software engineering work and sign off on the software their companies deliver. To those companies, employing professional engineers will be a legal necessity. If software companies follow other engineering disciplines, the company that hires a professional engineer will pay for liability insurance as part of the professional engineer's employment package, which will minimize that disadvantage of getting your professional engineering license.

Professional engineers will gain other benefits. The professional engineers who put their signature and reputation on the line for their companies will have final say over methodology choices, design approaches, and other decisions that affect the quality of the software for which they might be held liable. Without professional standing, your boss can come to you and demand that you commit to an unrealistic schedule, make shortsighted design compromises, or sacrifice quality in order to get the software out the door. With a well-defined profession—consisting of education, a code of ethics, and licensing—you will be able to say, "The standards of my profession don't allow me to shortchange quality in this situation. I could lose my license or get sued." You will have a professional and legal basis for standing up to unenlightened managers, marketers, and customers—a basis that is sorely lacking today.

Wow. In some ways, it sounds great to have a legal basis for saying no to bad development practices. In other ways, it sounds very scary to be subject to that liability.

Mostly, though it makes me wonder if software development truly needs to have "standards higher than those normally tolerated in the marketplace" and "a primary emphasis on social responsibility over strictly individual gain." It makes me want to know more about what happened to make it generally agreed that in engineering, medicine, and law, public good can't be ensured by the marketplace alone.

Because right now, one of the arguments for high standards in software development is that it's a good business decision. "The only way to go fast is to go well," as Uncle Bob Martin says. Or, we want to make sure we don't take on the wrong kind of technical debt, because that will be bad for the company in the long run.

But for a profession, these high standards don't have to be justified solely in marketplace terms. Instead, they have been accepted by society as being important for its own public interest.

In discussing how engineering came to be a profession, McConnell says:

Historically, professional engineering has been established in response to threats to public safety. Although we take the safety of modern bridges for granted, in the 1860s American bridges were failing at the rate of 25 or more per year. Bridge failures and the loss of life they caused precipitated creation of a stricter engineering approach to bridge design and construction.

I wonder what sort of disaster would need to happen for society to demand higher standards in software development. Or would a slow build-up of buggier and buggier software be enough? How many times do you need to lose your data, have your computer crash, or just plain not be able to figure out how to do what you want in a user interface before you've had enough? And if you've had enough, can you buy a competitor's software instead, or is it just as buggy?

Can the problems of today's software development be fixed by the marketplace, or do they require regulation? Does software development need to be a profession?

Sunday, July 25, 2010

Book Review: Professional Software Development

I can understand how many people sound as though they were disappointed in Steve McConnell's Professional Software Development: Shorter Schedules, Better Projects, Superior Products, Enhanced Careers. It's not a "practical" book—like Code Complete, Rapid Development, or Software Estimation—that you can immediately put into practice and improve your software's code or schedules.

But whether or not you agree with McConnell's conclusions about how to "fix" software development by turning it into a profession—like engineering, medicine, or law—it does a good job highlighting some of the ongoing struggles in developing software.

Many other reviews point out that the stratification and specialisation advocated by McConnell seem out of touch with agile development, and I agree. On the other hand, agile development encourages "generalising specialists," not straight-up generalists.

Other reviewers have also pointed out that the ladder-like professional development program promoted by McConnell also seems overly-regimented, and I agree with that, too. On the other hand, if career progression depends on taking specific classes, publishing articles, or participating in conferences, that might be better than not being allowed time for any learning or professional development at all, which happens at some companies.

[To be continued in "Does 'Software Engineering' Need to be a Profession?"]

Saturday, July 10, 2010

Book Review: Growing Object-Oriented Software, Guided by Tests

Recently, I read Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pryce. Amongst excellent discussions of other TDD topics, it sheds a lot of light on good ways to use mock objects.

Personally, I can remember reading Martin Fowler's Mocks Aren't Stubs quite a while ago and concluding that I'd be a "classicist."

Some time later, I started working with mocks as a way to isolate unit tests from "slow" dependencies, such as databases, trying to make them run more quickly. I didn't have much success, though, because I was still writing my tests in a classicist style.

This book helped open my eyes to how the "mockist" style really works, and why it can be a good thing. I absolutely recommend it to any programmer who is doing TDD (or writing unit tests), but who still feels like when it comes to being a mockist, they just don't get it.

I also think a great complement to the book is the video of J. B. Rainsberger's presentation Integration Tests Are A Scam.

As a note on how to read the book, I got a bit bogged down in the middle section, with the worked example. I ended up skipping to the final section and came back to the middle after. Reading it like that worked just fine. I also think it could be interesting to alternate chapters of the middle section with those of the final section.

Saturday, April 24, 2010

Do Unit Tests and Quality Code Result in Fewer Bugs?

At the end of my team's last release, I set out to prove that unit tests and quality code lead to fewer bugs.

Here are the tools I used.

Jira

The first thing to know is that the team I'm on keeps its features in Jira, and when we commit code to Subversion, we include the Jira ID in the comments.

Below, you'll see how this helped me relate code to features.

The second thing to know is when we find bugs in a feature during its development, in Jira we link those bugs back to the parent feature.

This allowed me to measure how many bugs had been reported for each feature at release.

SvnPlot

There's an open source project called svnplot that allows you to export the information stored in svn to a SQLite database. To do this, follow the instructions in Step 1 of the Quick Start.

You end up with a couple tables that include rows for each revision, including the comment and the files that were added, removed, and changed.

ANTLR

I used ANTLR C# grammar to help map the files changed to classes.

NCover

I used NCover to measure unit test coverage of the added and changed classes.

NDepend

And finally, I used NDepend to measure code quality of those classes, as well as to query the coverage metrics.

Results

This is the part where I'd love to show some fancy graphs proving my point. Unfortunately, the results weren't exactly conclusive.

I ended up dropping the idea of trying to correlate code quality to number of bugs altogether, because I just ended up with a mess.

The results for how unit test coverage related to bugs is sort-of presentable, at least, but they don't show a nice correlation:

Feature LOC Changed / Bug Average Coverage (%)
1 521 65
2 290 38
3 165 0
4 140 54
5 125 21
6 76 25
7 74 77
8 59 30
9 2 80

I'd like to think I'll try again another day, but given those results, I just don't know. I can come up with many reasons why things didn’t come out the way I'd hoped:

  • The amount of change for each feature was very diverse. Some features involved lots of code changes and others involved very few, so that the features may not really be comparable in this way.
  • Some features involved mostly changing UI code, where the coverage can't be high, because we don't currently have a good way to unit test our UI code.
  • Some features, especially Feature 9, had high test coverage, but the unit tests were very low on actual assertions.
  • There's just not enough data.

In the end, if my argument to the business is that unit tests and code quality are important because they reduce the number of bugs, results like these won't help me make my case.

So, if you know of more persuasive results—either your own or stuff out there in the literature—I'd love to hear about it :-)

Monday, March 08, 2010

Using the Windows Findstr Command to Find Text, Files and Folders

Currently, I like the findstr command better than other ways to search files in Windows.

Prerequisite: Open Command Window Here
For XP, it's part of Microsoft PowerToys for Windows XP.

For Vista and Windows 7, apparently you hold down the hold down the shift key and right click.

Finding File Contents
Once you have a command window open at the directory you want to search within, you  can do something like this, which will return all lines containing "MyText" in all subdirectories of the current one, case-insensitive, and only searching files that contain printable characters:

findstr /s /i /p "MyText" *.*

It turns out that "MyText" is treated as a regular expression, which is great. But if you don't want it treated as a regular expression, you can use the /l or /c: options.

Finding File & Folder Names
Instead of giving findstr files to search, you can pipe text to it from another command, so to search for file or folder names, you can give it the results of the dir command to search:

dir /s /b | findstr /i "MyFileName.txt"

Combining the Two
Say you want to find all the lines containing "MyText", except you don't want those that are part of a subversion directory. You can do this, where the /v option means only return lines that don't match:

findstr /s /i /p "MyText" *.* | findstr /v /l ".svn"

Thursday, February 18, 2010

Software Development Career Paths: Up or Out?

Continued from Software-Development Career Paths: Organizational Quality.

Moving Up
For this option, mostly I can only repeat the traditional oft-heard worries: your tech skills will get rusty and you'll be stuck in middle-management forever.

On the other hand, I don't think developers who like to code should dismiss moving up out of hand. If you get a chance, try it. Who knows, you might like it.

There's a nice discussion about staying technical as a manager over on Rands In Repose: Technicality. If you keep your technical skills sharp, then hopefully if it turns out that you hate being a manager, you can still go back to being a developer.

Moving Out
Lately I have a new perspective on moving out, thanks to the software craftsmanship movement. How many books and articles have you read lately that use the word "journeyman?" There's a reason the word "jouney" is in there.

In parts of Europe… spending time as a journeyman, moving from one town to another to gain experience of different workshops, was an important part of the training of an aspirant master. [Wikipedia, Journeyman]

Although I don't think it's a perfect book, Apprenticeship Patterns by Dave Hoover and Adewale Oshineye is currently the best reference I know for thinking about different ways to approach a career as a software craftsman from the perspective of an individual developer rather than that of a manager.

Another Option: Stay Small, Maybe Even Really Small
Keep in mind that not all organizations force you to choose between coding and managing. Smaller companies often don't have "Eiffel Tower corporate hierarchies." If you're moving around, you might find more satisfaction sticking with smaller companies, especially since that way you're more likely to personally provide value to customers, as described by Cameron Purdy in his great article Eight Theses.

Don't forget that you can go it on your own. Recently at DDD8, Liam Westley gave a talk about owning your own software development company. Of course, the talk assumed that you already "have commercial ideas, clients, or sales leads." I have no idea how to get to that point, but the fact that Liam was there giving the talk is a reminder that people actually do manage it.

Staying Motivated
Whether you literally become a journeyman by moving among companies or you stay for a long time at a single company, if you don't want to move up, you may need to look for non-traditional ways to keep yourself up-to-date technically and otherwise generally motivated, especially if you find yourself occasionally working at places that treat you as an interchangeable coding machine.

Apprenticeship Patterns gives some ideas about how to do this. Obviously, there are always things like learning new languages, participating in either local or virtual developer communities, and in open source projects.

Interestingly, all of those things follow a general leadership-followership pattern that's pointed out in The Art of Followership:

[An] example, which is becoming increasingly common in developed countries, occurs with employees who have extensive professional education. These employees can often perform their work assignments without relying on technical guidance from their hierarchical superiors. Their education often includes a strong socialization component, instilling in them a desire for autonomous, self-controlling behavior and the tendency to look to their professional colleagues for support and recognition. This often occurs with university professors who are highly educated and tenured. These individuals often view their university as a place to practice their teaching and research skills with little interference by administrators. They work with and obtain feedback from colleagues, they read academic journals for up-to-date information on their field, and they derive intrinsic satisfaction from their publications and recognition at academic conferences. Similar tendencies exist in firms that provide consulting and accounting services. In these instances, professional education and socialization can replace or substitute for a leader's direction and supportive behavior. [Chapter 3, Three Perspectives on Followership by Howell and Mendez]

I don't think the software development profession has necessarily quite reached this point yet, but it's an interesting perspective to keep in mind. Ideally, you can get "support and recognition" from your "hierarchical superiors" as well as your peers. Anything else is draining. However, in a pinch, don't forget to look out as well as up to get the sustenance you need to nurture your craft.

Tuesday, February 16, 2010

Software-Development Career Paths: Organizational Quality

A developer-friend of mine, Tom Wessels, recently asked the computer-age-old question:

I just want to be a programmer. But can I?

He follows up the question with a great quote from the great and still-relevant book Peopleware by DeMarco and Lister, which points the career path encouraged, or at least tolerated, by many organizations: If you're not on your way up, you'd best be on your way out.

To be fair, DeMarco and Lister follow up this description with a contrasting one, "The Mentality of Permanence:"

Over the years, we have been privileged to work and consult for a few companies with extraordinarily low turnover. You won't be surprised to learn that low turnover is not the only good thing about these companies. Indeed, they seem to excel at many or most of the people-conscious qualities discussed in these pages. They are the best…. In the best organizations, the short term is not the only thing that matters. What matters more is being the best. And that's a long-term concept.

DeMarco and Lister go on to describe ways in which these companies invest in their employees, fostering a culture of learning, professional improvement, permanence, and organizational excellence.

To answer the question, I think you need to look at your career along a few different dimensions. One of which is the quality of the organizations at which you will work.

It might be possible, to get a job at one of these "best organizations" and be comfortable for some time with one of the options Tom presents:

You reach a point where you just accept that you're at a good company, you get to write code in a somewhat enjoyable fashion, and you leave it at that.

Hooray! Yes, I think it would be good for everyone to at least have that option for a few years at some point in their careers.

But a couple things stand in the way.

First, it can be hard to get a job at an organization like that, not only because its bar to entry might be pretty high, but also because from the outside, it can be hard even to identify which companies are truly the best. Or maybe you just don't happen to live near the offices of any of those best organizations and aren't able to move.

Second, nothing lasts forever, including "being the best." I'm not sure the great organizations DeMarco and Lister use to demonstrate their "mentality of permanence" are still considered to be the best now that it's 10-20 years later: Reader's Digest, Hewlett-Packard, Pacific Bell? Even the lean software development movement's favourite company, Toyota, has recently shown signs of falling off the "being the best" pedestal.

In short, the mentality of permanence may not be permanent. If you're lucky enough to find yourself working for one of these best organizations, that's super. Enjoy it while it lasts, or for as long as you're comfortable staying. But don't be shocked if it doesn't last forever.

Which brings us to the next dimension. Given that you might find yourself working at lower-quality, "up or out" organizations during your career, should you move up or should you move out?

Continued in Software Development Career Paths: Up or Out?