Guest post by Fatal Exception’s Neil McAllister discusses why code analysis and similar metrics provide little insight into what really makes an effective software development team, in the wake of a new scorecard system employed at IBM. “Code metrics are fine if all you care about is raw code production. But what happens to all that code once it’s written? Do you just ship it and move on? Hardly – in fact, many developers spend far more of their time maintaining code than adding to it. Do your metrics take into account time spent refactoring or documenting existing code? Is it even possible to devise metrics for these activities?” McAllister writes, “Are developers who take time to train and mentor other teams about the latest code changes considered less productive than ones who stay heads-down at their desks and never reach out to their peers? How about teams that take time at the beginning of a project to coordinate with other teams for code reuse, versus those who charge ahead blindly? Can any automated tool measure these kinds of best practices?”
I find I work in busts … I usually work for 10 minutes really hard … then spend 25 minutes just chatting and making tea.
When it comes to the crunch though I can crank out code as fast as the next guy.
I also find solutions to most of my problems while making myself a Tea … or when I am sat in the pub on a Wednesday evening playing pool.
I don’t know if this is just me, or the fact that I usually work on one person projects, but I spend like 30 minutes furiously writing code, then 3 hours tweaking it until it actually runs like it should, or run at all in many cases.
Maybe it is because I have no formal training, I am not sure.
I am similar in that regard. I think it is because I moved from a Traditional Engineering disicipline … then moved to software engineering .. I don’t really have a hardcore coder background.
Others generally comment generally I think quite oddly and I tend to make connection between like playing Snooker and Jeet Kune Do …
I think many managers also have problems with me at first, because I am a bit of a maverick. I want to do my job, I just do it my own way rather then the way they want me to. I also tend to self manage.
We have an obsession with making sure all decisions are based on numbers that permeates our culture. It’s like we’ve created a culture of management that cannot make any decision without someelse putting the numbers in front of them… well then what is the point of having management at that point? Just have the computer do the final step and compare the two numbers.
Ultimately, any complex task is going to be impossible to measure via any cheap means or expert opinion. Ultimately, you’ll have better luck with peer review, professional programs…
I can’t tell you who/what makes the best teacher… but I guarantee I can tell you after speaking to their fellow teachers. Ditto for doctors, nurses, engineers, lawyers …
This is all thrown off of course if you are basing their performance review or who gets laid off on it
Of course software developers don’t have the luxury of some kind of professional association or proper training path (residency, articling, accreditation…)
You’d probably get a decent view of software developers by having a group of known good people (developers, product manager…) do random interviews with teams, review developers, review code, ask questions…
Amen!
I worked at a Fortune 500 company for a while and they forced you to use THEIR internal troubleshooting system regardless of whether or not you already knew how to solve the issue. You were FIRED if you didn’t use the system.
I had 100% success rate, 15% troubleshooting guide use rate (they called it something else…), and came highly recommended from my peers and managers while also having received praise from most all clients.
None of that mattered, though… I was still penalized rather heavily on the ‘scoreboard’ and fell below the ‘target performance’ metric. I worked there a total of three weeks – I have the luxury of no mouths to feed but my own, I have no intentions of placating idiots with big fancy pieces of paper from big fancy buildings when I know more than they’ll ever know…
[ Yes, I was fired 😉 – first time for everything! ]
–The loon
Probably not fired for “not using the system” but probably for being difficult to manage (as in making the managers job difficult).
Many companies tends to frown upon that, just so you know….
While you are absolutely right, I would rather not fall in line and go along with the rest of the people in the cubicle farm.
Make waves, all the interesting people do.
Yes, I was fired for a combination of reasons – they were listed (verbally):
1. Failing to use required tools
2. Management didn’t like you because you didn’t use the required tools and they couldn’t monitor you
3. Complaining that the workflow required 3 monitors
4. Complaining about Internet Explorer 6’s slow performance and lack of tabs
In the end, though, only ONE manager did any complaining (I had six… and was, myself, a 2nd level manager…LOL!) – and only because I was deemed a threat to his might. Co-workers complained and he was relocated and I was -eventually- offered my job back, which I didn’t take.
Always more to the story 😉
–The loon
looncraz,
Yep, instead of recognizing your good work, some bean counter had to justify their job by firing you even if it hurt the origination as a whole.
I too have been fired, but it was for the incompetence of another full time developer who found it beneficial to point the finger at me as a scape goat to the management. They paid me less than half of what they owed. That small company is no longer in existence.
I agree with Yamin that developers can usually recognize each other’s strengths and weaknesses, but we also live in a time when politics are needed to get ahead and stepping on each other is often better rewarded than hard work. I don’t think there is any possible combination of metrics that would fix this.
I’d just add for emphasis, developers should not just be ‘analyzed’ by other developers, but by all people in the organization. Testers, product managers, customer support, sales, other groups…
Working in a large organization is complex and hard work or talented work is not all that matters.
“Of course software developers don’t have the luxury of some kind of professional association or proper training path (residency, articling, accreditation…)”
I am not saying these things could not use improvement, but its not as if they do not exist at all…
professional association… What about the ACM or IEEE ?
proper training path (residency, articling, accreditation…)… What about a B.S. in CS, Masters of CS, Masters of Software Engineering, MCAD/MCSD, Sun Certified Professional (SCP)?
Those aren’t proper training paths because:
1. They are not mandatory. So you will never end up with a professional culture as it can always be undercut by untrained people just good enough to appear to do the immediate job.
2. They do no include a significant on the job mentorship component. It is not just ‘can you write code’. But can you communicate, handle clients, ethics, behave yourself as a professional… formal passing on of real life practice and methods built into the profession
If anything the formal residency style mentorship is significantly more important than any educational credential.
“it might be best simply to forget about the idea of measuring developer productivity and rely instead on tried and true methods. That’s right: A highly effective, productive developer workforce is often the result of high-quality, effective management. Unfortunately, nobody has developed a metric for that yet. Coincidence?”
he has some valid ideas, but he is also ranting. he is attacking logic-based decisions with emotional fire. fury even!
make your points, but we must still keep TRYING to get better. we can’t shut it all down and work on intuition. we (organizations of people) must TRY to find processes to better ourselves. even if we keep failing!
To his point a bit, I also find that a lot of organizations start to run down the road of “performance metrics” periodically and it always seems that we start spending more time working on stuff to “keep track of metrics” instead of the actual work itself.
If I’m spending between 30% and 200% of my time updating all the tools and systems with my breakdowns of how much time I actually spent working on tasks, it doesn’t seem all that efficient to me… The alternatives: lines of code, bugs fixed, scm checkins, are all so easy to game, they’re not really useful either.
intangible,
“To his point a bit, I also find that a lot of organizations start to run down the road of ‘performance metrics’ periodically and it always seems that we start spending more time working on stuff to ‘keep track of metrics’ instead of the actual work itself.”
I had one employer who implemented a system to measure every single minute of work and log what we’re doing at all moments. While it’s a quasi-reasonable thing to want to have, it’s terribly inefficient and invasive in practice. Their system sent out emails informing most of the employees that they didn’t put in the required 40 hours, which is insulting given that many of us were there more than 50 hours, obviously we just didn’t account for every minute of every day. We complained that often times developers help each other out by talking to one another even when we’re not always on the same projects, management responded that the only time we’d be given credit for is what got logged in the time sheet (the system didn’t allow us to record time for items which weren’t officially assigned to us).
In the end, the system under-emphasizes important results and just encourages abuse: rounding up time, leaving the clock running, etc. I wouldn’t be surprised if many developers are fabricating the numbers all together for their time quota.
What ever happened to treating software engineers as professionals?
Edited 2011-11-18 19:46 UTC
He is not ranting at all.
It’s actually a very good point. One I think you should look at again.
They haven’t found any good metrics for management? Why is that… because it is a skilled job… just like developers.
And the argument that we should *keep trying* is kind of pointless. It’s great to *keep trying* in an academic setting. The problem is that a bad metric is worse than having no metrics.
For example, if you ran a software company and used lines of code as a measure of performance… you would end up with a poorer product than if you just let people do their job.
Of course we should always progress and try and innovate in everything… including process. However how much you adopt of ‘new things’ in current systems is always a challenge as you risk breaking what works and even making things worse.
That is not progress at all.
Given the poor state of developer metrics, I certainly wouldn’t think it progress to put in anything into a company right now. That’s a step backwards. To put it simply… the best metrics we have are peer review.
he is totally ranting, AND peer review is the best. BAM!
Back when I was working for Storagetek (during the Sun takeover) we had a Sun Fellow come and talk to us. I was lucky to talk to him one on one and he had some very interesting things to say that completely turned my view of this matter around 180.
The mangement process is not meant to serve the developer, it serves the company. Another way of putting it is its role is to create a champion team. And a champion team always beats a team of champions.
Mangement of the developer might lower the individuals productivity but by helping the larger group work together total relevant productivity is increased. This becomes more important the larger the team gets.
Of course like all things a bad manager/process/metrics only screws things up, but then a bad anything (developer?) will screw things up too.
“And a champion team always beats a team of champions.”
No one here would argue that a good team is essential. A team of champions might not be better than a well oiled team.
I really don’t see the relevance of what you say in relation to the origin
This is why you include a broad spectrum of people to survey (Tests, product managers, fellow developers..) You will get a pretty good grasp of what makes the best team. You don’t just measure how good a developer is by how quickly he can solve some technical problem.
Good management, and I have worked for some good managers, understand this and do build great teams.
This reminds me of a story about Bill Atkinson, who wrote many of the low-level graphics functions in the original Mac OS.
The management at Apple had just put in place a developer productivity metric based on “lines of code”. A developer had to fill out the number of lines of code they had written at the end of the week.
Bill Atkinson had taken a week to rewrite some graphics functions. The rewrite had sped up those functions by a massive amount, cleaned up the code to make it more maintainable, and also had the side-effect of being 2000 lines of code shorter than previously.
When it came time to fill out the “number of lines of code written” form at the end of the week, he thought for a few moments and then wrote “-2000”.
Shortly after that, they stopped asking him to fill out the form.
We have a system that we use to track our project time, and I just make shit up about what I did for the day.
4 hours for operational maintenance, 2 hours for this and 2 hours for that.
Everyone fakes this stuff, I can almost guarantee it.