Quick feedback. More testable code. Finding serious problems early. What’s not to like? Michael Kelly pleads the case for unit testing.
Should We Be Doing More Unit Testing?
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
Wrote 12 tests tonight on a php5 project….it’s very nice to have a bug free api when adding tiers.
PHPUnit 2.2.0 by Sebastian Bergmann.
…………
Time: 0.02682900428772
OK (12 tests)
..from all the new buzzwords and new trends that I learned in the past few years, UT is the one that really made a huge difference on by daily programming. But it takes some time to really get in the TDD mindset, at least a couple of months.
But it is no less valid now than it was whenever it was created as a term and part of a process.
If you do small unit tests at a fine level of detail, and build on that in layers, it will become more readily apparent when you’ve added cycles into the dependency structure of the application, since once you’ve done that, you can’t test small bits of functionality at a time without having all kinds of hacks of the code to stub out things you don’t want to test. What’s important is that these tests are done at the same time as the code they test is implemented, so there’s no latency between the two, so the implementation being tested can be fixed before the cost gets too high.
Now, imagine if Linux and the various desktop environments (Gnome, KDE) did this at module/class level, as appropriate… the desired results would be a unit test system that’s many times the built complexity of the final application, but you would quickly be able to point to where something wasn’t working as expected: is it a small bit of functionality, or a larger combination of small bits of functionality when integrated?
Of course, there are certain types of things that are notoriously difficult to create unit and integration tests for, as running a test is of little value if the output isn’t analyzed for correctness, and that can take quite awhile, depending on what’s tested, and often must be done by hand.
So SW types call it “unit” testing. Um didn’t it used to be just called “module” testing 20yrs ago and is “unit” better in any respect.
As a HW-SW type I think we always used considerable “module” testing for real pay HW work but much less in quickie or undermanned work. Verification effort level depends more on how cross connected the module is (kinda like how many shared global vars). The debug technique is like violently shaking stuff till bits stop dropping off. Its a good job that todays PCs are so fast, the test cycles even on compiler sized projects can be less than a sec. Go back 20yrs, and it would have been hours/days, but probably way less ambitious.
I wonder how relevant the 100 or so newish words that I see popping up like refactoring are to HW and what the equivalent is if there is one. The reason for asking is that today most HW is designed with special purpose programming languages often using same dev environment as SW uses.
I can see why so much SW might be written without it though, the temptation is too great. I hope that most of the SW I actually use gets UT but I have to wonder.
I wonder if iso9000 has anything to say about this?
its about balance. why write 3000 lines of unit testing code to test a 12 line class? only when its critical. else its wasted effort.
and anyway – far too little thought is given to *meaningful* tests .. tests which coveny a lot of information during th e test .. one well crafted test is better than 15 bad ones…
I often hear statically typed languages being criticised, among other things with the argument that unit testing takes care of the safety provided by static typing without the `hassle’ of static typing.
Personally I feel unit testing and static typing indeed approach the same problem (finding/avoiding errors early at development-time rather than run-time), but they solve this for a different class of errors. Roughly speaking, I think unit testing tests for logic errors, while static typing tests for more lowlevel programming errors.
What do you think?
(Note that i’m mainly focussing on `proper’ static typing as found in most functional languages, for example, not so much on the outdated typesystems of C and Java)
Everytime someone mentions Unit Testing people say “but hey, I’ve been testing my sw/hw for decades, this ain’t new”, but when people say “unit testing is new” what they really mean is Test-Driven Development (TDD).
People have been testing stuff for eons but the technique of create a test -> see it fail -> create the code that makes the test pass is relatively new, at least on the public perception.
I am very skeptical about the “buzzword of the day”, such as XML, AOP and even some UML stuff, when they are presented as silver bulletts, but I can’t see why someone would ditch the concept of TDD, it’s very intuitive and from my personal experience the gains are HUGE compared to a relatively low overhead. For instance when it comes to refactoring, I can’t conceive how it can be done reliably without some kind of unit testing; it’s hard to guess all the side effects that a change in a module might cause in the rest of the system.
Bottom line is, if you can’t think on a proper test for the code you’re about to write, that means you haven’t understood what you gotta do yet, so it’s not a bad idea to think a bit more.
Of course that, if not used appropriatedly used, UT can make things worst, but then again, a bad programmer is a bad programmer and will mess up with or without UT or any other development methodology!
For instance when it comes to refactoring, I can’t conceive how it can be done reliably without some kind of unit testing; it’s hard to guess all the side effects that a change in a module might cause in the rest of the system.
Before anyone else points it out: yeah I know that if you have really modular and orthogonal components you shouldn’t have side effects as long as you don’t mess with the interfaces, but hey, we don’t live in a perfect world
It is well known in computer science for about 3 decades that testing is an inadequate bugfixing strategy. This is because many, often not obvious, inputs are not being tested, which are in practice the situations that cause bugs. Testing all possible inputs is often impossible.
In other words, automated testing is no replacement for programmers being able to write code that they know is correct. Aiding and encouraging programmers to do this is the big challenge to write better software.
The main thing that is stopping unit testing from being universal is that it is tedious, and (like many other advantageous programming techniques) seems to make things slower in the short-term.
Full design docs, software process metrics and unit testing are all techniques that pay off in the long term, but they are like the programming equivalent of brussel sprouts… all programmers know they are “good for us”, but we avoid them if we can, because they simply aren’t pleasurable.
It is only when it is too late, and you are snowed-under with bug reports, that you realize that you should have eaten your sprouts.
I don’t mean to troll by this question, but what is the difference of Unit testing and Regression tests (eg those tests executed when doing “make test” in ffmpeg)?
In other words, automated testing is no replacement for programmers being able to write code that they know is correct
UT is certainly no replacement to good coding, but I don’t think people expect it to be..but the question is, is it a good addition to good coding?
And BTW how do you really _know_ your code is correct, if not testing it? The only thing that TDD adds to the table is to make testing a required, “official” part of the development proccess, instead of an afterthought.
Sorry for the missing bold tag.
UT is good. But i came to hate it. I was employed in a company where the main developer, and the 2 bosses were … like crazy into unit testing. Did you do the unit testing? Have the tests passed? Did it worked? It was like … if you have done the unit testing, and the tests worked, then …the code must be good. Nothing far from the truth.
Slowed development. When UT were broken, I often spent nights debugging other people’s tests. No, not the code, you heard me: THE TESTS. You got bugs in your program, you got bugs in your tests, you got bugs everywhere. And you don’t have time. And progress is being made at a very slow rate. And there is no progress when you believe that unit testing is all that you have to do, without caring for what you test, and how. They often wanted me to test Swing components. That is the most absurd request i ever heard in my life.
So, to sum it all up: UT sucks big time, and I hate it. It doesn;t solve you problems. The code is not perfect if it passes UT, the bugs are still there. It doesn’t mean anything. A waste of time.
That sounds more like “${technique} sucks when your bosses have missed the point of it.”. You said yourself that you were given absurd requests, which suggest that your bosses were suffering from Pointy Haired Boss Syndrome.
Unittests when used properly is IMHO a great way to improve the quality of the code in question, but it is a bit odd at the begging.
The C2 wiki contains a lot of interesting information on that subject:
http://c2.com/cgi/wiki?TestFirstDesign
Regression tests are simply tests that can be run again and again to make sure that old things haven’t broken. They can be done by hand or be automated.
Unit tests are specific to testing small modules of code. They can also be used as regression tests, but not all regression tests can be considered Unit tests. For instance, there is also integration testing (to be sure all those small parts work together correctly).
Unit tests are way overrated. Too often have they been a crutch for passing bad or just plain wrong code, unnecessarily added to development time and costs, or used to sell more patterns and ‘best-practices’ for problems that didn’t exist but were artificially created. Easy to market, but mostly superficial and useless. Analyze and fix the problem, not create a new problem and fix it.
Unit tests can certainly be useful and part of a solution, but they are not the answer nor solution.
Slowed development. When UT were broken, I often spent nights debugging other people’s tests. No, not the code, you heard me: THE TESTS.
Looks like the unit tests were not done properly then..it usually involves a couple of lines of code, where you call one specific function in your system…if you start putting “logic” inside UTs, that’s certainly very bug-prone.
As it was mentioned before, a bad programmer is a bad programmer and will mess up anyway, with or without UT. That means maybe you wouldn’t lose your nights trying to fix a hairy UT, but it doesn’t mean the bugs wouldn’t be in your system, they just would be undetected.
Unit tests can certainly be useful and part of a solution, but they are not the answer nor solution.
As well as OO itself is not a complete solution, etc..it’s all part of a development framework.
I agree with Xaignar about the “a bit odd in the beginning” part, I remember that in the first couple of months UT was not “natural” for me, and I felt slowed down. But I decided to insist on it, until it became natural, and now I save a lot of time, because once I can come up with a test, that means I understood the problem, with means I can make the code with a _goal_ in mind, instead of simply start coding early and think about the problem while coding (which sometimes might mean a whole morning going on the wrong direction).
But if anyone believes UT is a silver bullet, forget it. Nothing is.
Three pages without ever defining what a, “unit”, is, how it’s to be tested or even what kind of test you might run. Doesn’t give any real arguments either way, an empty read providing no valuable information except maybe a couple links. Developers testing their code in functional units, now there’s a new and exciting concept, HAH, “only found in the agile world”. Snork!
Good unit tests improve the aroma of the code.
Nothing like a working example, with dummy data, to communicate to the maintenance coder WTF was going on.
I have been using TDD for a few years now, and can’t imagine developing without them. When someone asks me to review their code or worse yet try to refactor it, unless they have any tests (unit or functional), I pass on it and ask them to put some in. If I have a mound of *small* (less than 10-15 lines, for easier reading/maintenance) tests, I can go deep into the code, mess with it, and see what I affected and what broke and why (it’s like having a massive QA dept with one click). Of course if the tests are bad, this doesn’t work right? That’s why you write the tests BEFORE you write the code and have them reviewed by others BEFORE you code up some meaningless mess. Then there is no “I thought this code was supposed to do xyz, not abc???”. Everyone knows what the code is supposed to be doing.
Another nice side effect of unit testing is the ability to diving into someone else’s code: it’s easier to take a look at the unit tests to figure out the intention behind some module, what it’s supposed to do and what it’s not supposed to do.
There’s a common sense that it’s much easier to do something (be it programming or anything else) if you first set the goals, and then go for it. Unit testing is this piece of common sense formalized. Think what you want to do first, then implement it.
Use a language with well-defined semantics and prove the correctness of each part of your program. Programmers need a math education, especially discreet math and logic.
Programming is not like a natural science that needs to “test” its hypotheses. Programs are constructed, not discovered. If they are constructed carefully then they will work properly, barring hardware failure.
Industry computer science is immature, so making correct programs is hard because the tools in the workplace are designed by uneducated people. Interfacing a program with these tools whose behavior is neither well specified nor predictable leads to the need for testing.
In the future I am confident that the field will mature and find its place as a branch of discreet math.
I wonder (no troll intended), how many commercial apps have you developed? Do you have training on discreet math and logic, and a piece of bug-free code to back up your theory?
Did you or someone else prove this theory of yours mathematically, or is it wishful thinking?
Your argument seems to be “in a perfect world, we would have perfect software”, but the problem is that the first part will never happen, to forget about it.
Sometimes people need to take “shortcuts” (not allowed in your mathematical approach) not because they’re lazy but because there’s a need to compromise performance x robustness, user experience x validation, and stuff like that.
Try to make tools for “educated people” and then make bug-free programs on top of it and return when you’re done.
> how many commercial apps have you developed
A few. I worked for
– TIE Commerce writing&debugging business to business data translation software
– West Group creating a web interface for tracking sales performance
– UW Condor Project Invented and implemented a functional programming language based on the lambda-calculus to manipulate distributed-computing information.
– some others
> Do you have training on discreet math and logic
Yes. I have taken 3 undergraduate logic courses (proof theory, model theory, etc) with a 4.0 GPA.
The point isn’t me, however, it is this idea of software correctness.
> and a piece of bug-free code to back up your theory?
No. I’m working toward that goal, one day at a time. It’s a lot of work, but I think this is where computer science worthy of the name is headed.
> Did you or someone else prove this theory of yours
> mathematically, or is it wishful thinking?
Yes. It’s wishful thinking. I don’t know exactly how to go about mathematically proving predictions like, “In the future I am confident that the field will mature and find its place as a branch of discreet math.”
> Your argument seems to be “in a perfect world, we would
> have perfect software”
Not exactly. My argument is that software is not a mysterious thing; we make it. There are alternatives to poking it to see if it works. Draw on the thousands of years of math experience and learn to use mathematical techniques.
> Sometimes people need to take “shortcuts”
Yeah, I hear you. Those people’s software will have bugs. It’s a tradeoff, and maybe it’s good enough for their purposes. Good for them.
> Try to make tools for “educated people” and then make
> bug-free programs on top of it and return when you’re
> done.
I’ll do my best, chief.