TDD: What's the Point?

What's the point of TDD?  This might sound outrageous, but IMHO, the point of TDD is not to exhaustively test your code as your developing it.  It's not to test every corner case, every possible input/output scenario imaginable.  It's not even -- gulp -- to achieve 100% code coverage. 

What?!  Blasphemy! 

Yeah, yeah.  100% coverage is a worthy goal, and if you're doing TDD right, you'll get pretty close anyway.  But still, 100% coverage isn't the point.

The point of TDD is to force you to design and write your code from an outside-in perspective, rather than an inside-out. It therefore also ensures that it is testable, well modularized, free of inappropriate foreign entanglements, and is most likely to work the way you need it to.  The focus is on the use of the class, not the implementation of the class. 

When you're not doing TDD, you're thinking more (sometimes exclusively) about what the class needs, and how it will do what it will do.  When you do TDD, you think more about how the class is used, what it's user's need, and what the rules are.

You write better code that way.  So TDD is a way of getting yourself to write better code.

So TDD is an approach to development -- not an additional task added to development.  It isn't overhead.  That brings me to the next topic.

Sabotage Your Code with Time Bombs

Sometimes you need some code for a short period of time, after which
it needs to be removed.  I've been having to do that a lot in the past
few weeks.  If I'm an average developer, then the average developer
has, at one time or another, done this by putting in comments like:

// XXX HACK FIXME TODO remove this!!!

The problem with this is you write it and then forget about it.  It
gets lost in the bazillions of other auto-generated TODO comments.
Granted, an above average developer doesn't tolerate all those
auto-generated TODO comments.  But then, by definition, most of the
developers in the world are average.  Like me.

The point is, it gets forgotten, and many times you never go back to
make the fix.  Well in a case I ran into this week, I had to be
absolutely certain that this time, it couldn't slip through the
cracks.

The solution?  TimeBomb.

It looks like this:

TimeBomb.explodeAt(11, 1, 2010, "Remove the following line or else...");

The explodeAt method checks the current system date, and if it's past
the date you've given it, will throw a java.lang.RuntimeException with
the message you provide in the last argument.

It's easy to ignore a TODO comment. Not so easy to ignore a broken app.

Oh, and I'm not showing the implementation, because if you can't
implement that, then you aren't (or shouldn't be) writing Java code to
begin with.

Murphy's Law for Business Application Data

If a system lacks data integrity constraints, the data will lack integrity.  If it permits illogical, incorrect, unreasonable, or even theoretically impossible states to exist, then illogical, incorrect, unreasonable and theoretically impossible states will exist in the data.

Corollary: The more someone insists that a data condition will never happen, the more assured you can be that it already has.

Estimates, Targets, Quotes: Semantics!

I just had a brief conversation with a project manager on the way back from the coffee hole.  

Part of the problem with waterfall projects is that we have to "estimate" the delivery date and cost at the beginning of the project.  So we take mostly wild guesses and give estimates that we only learn later are unreasonable -- or at least pretty inaccurate.  [Dilbert's project uncertainty principle hits the mark here.] But these estimates set the customer's expectation.  They then get (understandably) bent out of shape when things go down the way they usually do in the real world: late and over budget. 

We usually call our wild guesses "estimates," which most non-technical folks expect to be something like the estimate we get from our car mechanic.  Oddly, the estimate we get from our mechanic is actually more like a quote than an estimate.  Most of the time, the "estimate" is exactly what we pay.

And that's pretty easy for a mechanic to do.  Car parts have been standardized, and the mechanic has done the same thing hundreds of times.  If every car had a custom-designed engine, mechanics would suffer the same difficulty we do.

But to the point, people get upset because they expect our estimates to be equivalent to car mechanic estimates.  So I'm wondering whether -- if you are forced to work in a waterfall world -- would it help to refuse to use the word estimate?  

How about "target" instead?  When you're tumbling in a waterfall, that's more like what you've got -- a target.  You hit it or not depending on the flow and the wind.

Rubout

Our three-year-old found a large pencil eraser last night.  She started rubbing it furiously on my head.  It started to make me wonder.  Was she trying to erase me?

Wily Code

Ok, so it all started when I encountered this curiously laborious little line of code in a client's app:

String s = String.valueOf(" ");

As I said, it's quite a laborious way to initialize a string to a space. Never mind that the code in question didn't actually need that space for anything. I thought it was quaint, and wondered what other creative idioms we might employ to initialize a string to a single space character.

So Eric Galluzzo and I came up with some "brilliant ideas." Most of the samples will actually work if you compile and run them.

New

String space = new String(MessageFormat.format(
 new String(new byte[]{123,48,125}), new String(new byte[]{32})));

Symmetrical

/*/
/*/; String space = "" + " " + ""  /* | *\ "" + " " + "" = space String ;/*/;
/*/
/*/

Bees!

String space = "b\bb\bb\bb\bb\bb\bb\bb\bb\bb\bb\bb\b ";
  // only works if you write this to something that will actually execute the backspace.

Binary-Encoded Space

String charCode = "100000";
        int chr = Integer.parseInt(charCode.substring(5)) 
         + (Integer.parseInt(charCode.substring(4, 5)) << 1)
         + (Integer.parseInt(charCode.substring(3, 4)) << 2)
         + (Integer.parseInt(charCode.substring(2, 3)) << 3)
         + (Integer.parseInt(charCode.substring(1, 2)) << 4)
         + (Integer.parseInt(charCode.substring(0, 1)) << 5);
        String space = String.valueOf((char)chr);

Bit Twiddler

String space = String.valueOf((char) (32&1 + 32&2 + 32&4 + 32&8 + 32&16 + 32&32); 

ASCII Art

String space = (
                  "#####                               " +                            
                  "#     # #####    ##    ####  ###### " +
                  "#       #    #  #  #  #    # #      " +     
                  " #####  #    # #    # #      #####  " +
                  "      # #####  ###### #      #      " +
                  "#     # #      #    # #    # #      " +
                  " #####  #      #    #  ####  ######. "  ).replaceAll("#", "").trim().replaceAll(".", " ");

Around-The-World

// get space from China
    HttpClient client = new HttpClient();
    GetMethod get = new GetMethod("http://www.china.cn");
    client.executeMethod(get);
    String response = get.getResponseBodyAsString();
    String space = response.charAt(response.indexOf(" ")) + "";
    
    // or if you're feeling more poetic, get your space from space at nasa.gov

Wait for space to come to you

DatagramSocket udpSock = new DatagramSocket(68);  // get data from DHCP requests on the network
 DatagramPacket packet = new DatagramPacket(new byte[5120], 5120);
 boolean found = false;
 while (!found) { 
   udpSock.receive(packet);
   byte[] data = packet.getData();
   Arrays.sort(data);
   int indx = Arrays.binarySearch(data, 32);
   if (indx >= 0) {
  String space = String.valueOf((char) data[indx]);
  found = true;
          }
        }

 // disclaimer: your network security group might not like you sniffing the wire, so although
        // this code works, I recommend you don't ever use it, even though it's an irresistible way
        // to initialize a string ;-)

Java Intermediate Language (IL)

0  ldc  [16]
    2  astore_1 [space]

Perl programmer

String exp = "/( {1})/";
Pattern.compile(exp)... ah forget it, I can do this in Perl without even typing anything !!

Vendor-Specific

/*
 * Thanks to Eric Galluzzo for reminding us of the importance of vendor-specific implementations!
 * I wouldn't doubt that someone, somewhere is doing this...
 */ 

import com.vendor.string.String;
import com.vendor.string.StringInitializer;
import com.vendor.string.StringUsingApplication;
 

public class MyApplication extends StringUsingApplication {

    private String myString;

    @Override
    protected void initializeStrings() {
        myString = StringInitializer.getInstance().initializeString( " " );
    }
}

String-Definition Domain Specific Language

# file space.stringdef
 define string {
    immutable
    encoding UTF-8
    locale en-us
    initialize with \s
 }

// file SomeClass.java
String space = StringDSLEngine.loadString("space.stringdef");
    
// implementation of StringDSLEngine is left to the reader as an exercise.

Enterprise Architecture

It has been recommended, in keeping with corporate architectural strategies and best practices, that the new string initialization service be designed to be as simple as possible, while leveraging key synergies across business unit infrastructure.  We have achieved this goal, and, as required by the SOA governance board, provide for all critical SLAs.

    All applications going forward will be required to use the string initialization service, as an enterprise best practice, to ensure optimal flexibility, uniformity and reliability.

    Initial development and deployment is projected to see an initial $2 million investment, it is also projected to produce $10 Million ROI per annum.

    Among key benefits of this architecture are the new character database that allows seamless support for new character encodings, a vital capability if new character sets are ever introduced.  This uniquely positions the organization to respond quickly as Unicode is retired in favor of the Intergalactic Character Set (IGCS).

I Love This Code

Just saw the following at a client site...

if (name != null){
  person.setName(name);
}else if (name == null){
  person.setName(null);
}

I love it.  It could only be improved by being even more explicit.

if (name.equalsIgnoreCase("Allen")) {
  person.setName("Allen");
} else if (name.equalsIgnoreCase("Bob") {
  person.setName("Bob");
} else.
   /* omitting lines in this post */
} else if (name.equalsIgnoreCase("Zuzu") {  // hand cramp at line  1087667556223
  person.setName("Zuzu:);

} else if (name == null) {
  person.setName(null);
}

The only drawback I can see to this approach is that it is open to modification under new first names, otherwise, it is brilliantly self-documenting!  ;-)

String Literals

I was just working on some some rather messy Java code. It was doing
something pretty silly, like this:

String str = "hello" + " world," + " how " + "are" + "you?"

Of course, sometimes you want to do this, like when you have to split
your string literal across multiple lines. You might do this when
your down the unfortunate path of hard coding SQL statements -- bleh
-- or maintaining old code that does the same.

That made me wish Java had C++ string literal handling. In C++, you can write

string str = "select * "
"from nowhere"
"where "
"something = nothing"

and the compiler will concatenate all those literals for you into a
single string. Much nicer than having all those concatenation
operators floating around.

Even better, though, would be support for other languages' multi-line
string literals -- like Groovy, Scala, Python (I think) etc. So even
better is:

def str = """ select *
from nowhere
where
something = nothing """

Logic in the Data Layer

I was just listening to a conversation between two developers. One of
them said, "there should be *no logic* in the stored procedure or the
DAO..." [emphasis mine]

I hear this a lot, and I think it's wrong because it is too broad. I
agree that there should be no *business* logic in those places, but we
have to be clear about the difference between business logic, a.k.a,
business rules, and other kinds of rules or logic.

In the conversation I just heard, the logic being discussed was not a
business rule, it was a rule about how data gets stored, and what
happens in the database when a particular operation occurs. This is
data logic, and it is perfectly proper for it to be in a stored
procedure or DAO.

In general, any logic or rules that relate specifically to the data
itself -- how it is stored, where it is stored, how it is aggregated
and moved -- is data logic, and belongs to the data layer. After all,
what does my application really care about how the data gets stored,
retrieved or deleted? And remember, logical relationships between
entities are not business rules.

I admit that the line can sometimes get blurry, but let us not make
such foolish blanket statements such as "no logic of any kind here."

It's the Perimeter, Stupid

Some years ago I allowed myself to be lulled into a false sense of security by people I worked with.  When I would raise a security point, they would counter it with the perimeter argument:  sure we're putting sensitive data on the wire unencrypted, but it's our private wire, inside the perimeter. 

I bought it.  After all, if the network folks have sufficiently secured the corporate network, then we don't have to worry very much about packet sniffing.  They hold the perimeter, so as long as our traffic stays inside the firewall, there's no need to add additional protection.  This mentality is very pervasive among corporate developers.  Why?  Well I think partly because we want to believe it.  It makes our lives easier. 

But if we're honest (and not totally ignorant), we're forced to acknowledge that even if there were no such thing as the inside threat, this argument is embarrassingly, obviously, wildly wrong.

Last post I mentioned that someone once asked me, with respect to the horrendous security situation we're in, "how did it get this way?"

This is exactly how.  Naive, pollyannaish assumptions about the environment our applications are running in and the kind of people who will be abusing them.  These assumptions are at the heart of nearly every grave security problem, from the wide-open network protocols of the internet, to unencrypted account data, to the XSS vulns in our apps, to insecure default OS configurations.

I'm at least as guilty as anyone else, having bought into the perimeter fallacy some time ago.But I've seen the light again in recent years.  We can't keep this up or we're in serious trouble.


The other day, I was working on an application that needed to make use of a service over HTTP.  The payload would contain sensitive data, even social security numbers at times.  When working in the development environment, the developer of the service forwarded me a URL with the http schema. 

I pointed this out another developer, who rightly interrupted me to emphatically insist that use HTTPS. Right on.  But even this developer was not all that worried about it, citing the perimeter fallacy.


Now don't get me wrong, I perfectly understand the dilemma.  Developers don't want to think about security.  It's generally interesting only to developers who happen to be interested in security anyway.  Developers are far more interested in spending their brain power creating useful new things, solving problems at hand.

This is why I think that we need to start taking security out of the hands of developers, so that developers won't have to care about security in order to create relatively secure applications.  How to do that?  That's for another time.  I've already gone on too long.

In the meantime, we must start insisting on more secure development practices in our organizations.  We're negligent if we don't.

The Internet Must Die

The Internet Must Die.  It must cease to be what it is and be reborn. 

A few months ago I had a conversation with a man who had left an IT career to enter religious life.  At one point in the conversation he asked me about the state of security in the industry.

"Absolutely horrible," I told him.  "We are living in the equivalent of the wild west." 

After hearing some details he was astonished.  "How did it get this way?" he asked.

How it got this way is simple.  Getting into trouble is always a very easy thing to do.  Getting out of it is the hard part. 

There are two primary reasons that our systems are so insecure.

First, nearly everything is on the net (even things that don't need to be), and nearly every protocol on which the net operates is intrinsically insecure.

ARP, BGP, DNS, DHCP, TCP, IP...  The list goes on.  These are insecure protocols, designed for a pollyannaish garden of Eden, where even the snake is uneager to exploit vulnerabilities.

A foundation so weak cannot support a robust, secure network.  But it is precisely this foundation, this set of protocols, that makes the internet what it is.

It therefore must cease to be what it is.  It must be replaced by, or morphed into, a network built on a new set of secure protocols. It must die and be reborn.  Hence the title of this post.

But I said there are two primary reasons why things are so bad, and the second one is actually a bigger problem.  Most of software that is running on the net (web apps, or other apps running on devices that are online), were also developed -- and continue to be developed -- with the same naive outlook as that of the protocol designers.  The same pollyannaish Eden.

Every developer I know (myself included) has been guilty of neglecting to harden our apps, if not routinely, then more than once.  And even to the extent that sometimes we developers actually pay attention to security, the average developer, in my experience, does not have adequate knowlege and skill to avoid, detect and prevent vulnerabilities anyway.

Yeah, yeah, I know, most of us don't want to have to spend time thinking about security.  We just want to get the app working. But we can't afford to take that attitude anymore.  [In my next post, I'll take a look at one fallacy that is partly to blame for our lax disposition toward security.]

But if we continue to be lax, if we refuse to start thinking about how we secure our apps and actually do so for every app we build, it will not be long before software developers will need malpractice insurance, because we're going to start getting sued.

It will not be long before companies are going to start being held liable for their apps that get hacked and start spreading trojans and botnets.  I'm only surprised it hasn't happened already.

HIDS

I've been working with some host-based IDS software on my (Linux) laptop lately.  It had been bothering me for quite a while that I didn't have visibility into my system.  If I were ever to suspect I had been owned, how would I know what files had changed on my drive?  Especially on one of the Windows boxes I help support (mom and dad), AV's failure to detect anything is not proof that you're clean.  New derivatives won't be seen by AV.  And even when you know with certainty that you've been infected, how can you account for everything that's been altered?  How can you confidently recover?  You can't.

Without HIDS, you're totally blind.

Actually, even with HIDS, even though you're not totally blind, you're not omniscient either.  But some is better than none.  And so I've toyed around with two HIDS on my laptop: tripwire and OSSEC.

OSSEC is nice because it runs on multiple platforms -- you can use it in Linux, Windows (and Mac too, I think).  It also offers more features than simple file checksumming, including a firewall and some AV capability.  One big problem I've had so far is that I can't figure out how to get it to give me alerts in any way other than email (and the documentation is fairly sparse).  Email alerts would actually be great, but so far I can't get them to work.  So I have to manually inspect the alert log.  Not good.

This is still work in progress, so I might eventually figure it out.

Tripwire was the first hids I tried, and because of the problems with OSSEC, I'm keeping it going.  The problem with tripwire is that it is a bit cumbersome.  I have it set up to run periodically as a cron job, which works nicely, and it works smoothly.  The problem is that I have so many software updates (usually at least 2 security updates per week) that it is turning into a lot of work to keep the tripwire database updated. 

Another problem I'm having with tripwire is that it keeps adding files that I think it shouldn't be -- despite the rules I've configured (probably incorrectly) -- leading to false positives.  Log files, for example.  I really don't care if they've changed.  We want them to change constantly.

All of these changes and false positives increase the risk that I could miss an illegitimate file diff and have it accepted into the tripwire checksum db as a legitimate file state.  Then I find myself guarding malice.

I'll keep toying with this, and might give some other hids a try.  So far, it's better than nothing, but I'm not as happy as I'd hoped to be with the experience.

Oracle Packages and Stored Procs

Today I needed to look at an Oracle package, but the only client tool I had access to was Squirrel Sql.  

You can see the source code for a package (or stored proc or function) with by querying either of these tables: all_source, user_source, dba_source.

select text from all_source where type='PACKAGE BODY' and name ='MYPACKAGE'

or

to see the entire package

select text from all_source where name='MYPACKAGE'

More Windows Stuff

Not sure where this came from, but found it in my notes...

The following command-line outputs the list of running processes (with the complete command-line arguments used for each process) to a text file:
    [This only works for XP Pro, not XP Home]
    Click Start, Run and type CMD

    Type the command given below exactly:

    WMIC /OUTPUT:C:\ProcessList.txt PROCESS get Caption,Commandline,Processid

    or

    WMIC /OUTPUT:C:\ProcessList.txt path win32_process get Caption,Processid,Commandline

    Now, open the file C:\ProcessList.txt. You can see the details of all the processes in that file.

More command line stuff

I ran across this again today.  Posting it here for the usual reasons...

When opening a command prompt (cmd.exe) console, then it checks the following STRING values in the registry to see if any commands should be executed:

[HKEY_CURRENT_USER \SOFTWARE \Microsoft \Command Processor]
[HKEY_LOCAL_MACHINE \SOFTWARE \Microsoft \Command Processor]
Autorun = "prompt [%computername%]$S$P$G && COLOR 0A && CD C:\"

Note the commands above will change the command prompt and color and change the default path/directory when opening the command prompt to the root of the C-drive.

Note to specify several commands separate them with &&, or use a batch file like a Autoexec.bat.

Note the cmd has a switch to disable the execution of the Autorun: cmd /d

Sed on Windows?

I needed sed today on Windows, and though there are versions of it available, or cygwin would work, I don't have privileges to install software on my client's machine.

Well, at least I had groovy.  So I did this:

new File(args[0]).eachLine { line -> println(line.replaceAll(args[1], args[2])) }

  And in a .bat file, this:

echo off
IF /i "%1" == "/?" (
echo [path expression] [match regex] [replace string] [output dir]
) ELSE ( 
for %%f in (%1) DO (echo converting "%%f" to "%4\%%~nxf") && (groovy replaceAll.groovy "%%f" "%2" %3)  > "%4\%%~nxf"
)

Yeah, I could have done the entire thing in groovy, but I've never used 'for' in cmd.exe before, and I just wanted to try it.

I'll probably not do it again, unless I have no other choice.  Anyway, it's here for future reference in case I ever have to use it again.

PVS Award Winner

This PVS award goes to the developer of this code (sanitized to avoid NDA violations).  While trying to troubleshoot an issue one day, the stack trace was always null.  Huh?  null stack trace?  How does that happen? 

The following class is in a shared library jar, and is used as the base class for many different application-specific exception classes in several apps.
 

public class MYException extends Exception {

...

    public MyException(String message, Throwable cause) {
        super(message);
        initializeCause(cause);
    }

    public Throwable initializeCause(Throwable cause)  throws IllegalArgumentException IllegalStateException {
        // ... some code here that conditionally throws those exceptions depending on what cause is, followed by this gem... \\
        mCause = cause;
        mStackTrace = null;  // clear a stack trace
        return this;
    }
...
    public void printStackTrace() { printStackTrace(System.err); }

    public void printStackTrace(PrintStream stream) {
        if (mStackTrace != null) {
            super.printStackTrace();
            if (mCause != null) {
                stream.print("Caused by: ");
                mCause.PrintStackTrace();
            }
        } else {
            stream.print(mStackTrace);
        }
    }

So printStackTrace always prints null.

When it's the best you've got...

I use mostly Linux and Mac, but I'm frequently forced to use Windows at client sites.  The Windows cmd.exe is pretty lame, but here are some tips to do a few useful things.


start /b  - run something asynchronously

findstr  - poor man's grep

dir /S   - recursive directory list

dir /S *.java  - can do it with file name patterns

dir /S | findstr

dir /b  - bare names

dir /b /S [pattern] | cmd     -- this will open all the files that match the pattern (as long as an editor is registered for the extension)

fc -- not as good as diff, but can get you something when you've got nothing

systeminfo

systeminfo | findstr Time:
(if you shut down your laptop every night when you go home, this shows you how long you've been on the clock)

driverquery

%ERRORLEVEL% - returns the error code of the most recently used command.

%HOMEPATH%  - best we can do without ~/

set  - shows environment variables,  'set p' shows only variables starting with 'p' etc.,  set aslo sets variables.

%RANDOM% - returns random int between 0 and 32767
(for when you need to make a decision. nice coin toss)

reg  - display and work with registry entries, might have to combine with runas

pushd/popd  - push/pop directories for quick in and out

sort  - sort files or pipe standard output to it

tasklist  - same list as taskmanager, but on the command line, useful with...

taskkill - to kill one of those frequently misbehaving windows processes

sc  - control and interact with services

Reluctant Blogger

For years I refused to start a blog. At first it seemed the epitome of narcissism. Incidently, I now think the most narcissistic things on the web are Twitter and Facebook. But anyway I now have some good reasons to keep a blog.