Horray, The Vendor’s code is improving!

I’m happy to say the code is getting much better. This presents some new information to my thinking about off-shoring…

Along those lines, while I was at OOPSLA (I know, I didn’t post anything from there yet — getting to it), I talked with a developer from Holland whose company outsources regularly to Romania. Apparently, he gets much better code than our initial batch — he wasn’t worried about code quality, and even talked about leaving things for his developers to finish designing, so they don’t get bored. This sounds like a much healtier working relationship. I plan on emailing him soon to talk more.

Outsourcing as Bargain-Basement Software

Two trains of thought have been merging lately: the poor quality of the outsourced project I was working on, and “The Source Code is the Design” (see both the original article at developer.* and the XP Wiki page discussing it).

The gist of “The Source Code is the Design” is that coding is a design activity, not a construction activity. High-level design (UML, specs, etc) still has to be done, but when you’re coding, you’re still doing design work, making design decisions that affect the eventual outcome. If business people and software managers want quality software, they should understand this, and avoid hiring code grunts. This makes sense to me.

On the other train of thought, I still see poor quality software being produced by off-shore programmers. I’m speaking entirely from my own experience, and the experience of people I know personally and have worked with. We’d have done a much better job. We regularly had to hobble the initial design because of pushback by The Vendor, and the code they delivered was poor beyond belief. It works, yes, but I hope never to have to plumb its depths. It violates most of our standards, and many basic principles of programming, period. Although it does work now. After some thumping and kicking.

These two trains of thought come together at (I think) a disappointing conclusion. Track one: management has to wake up about how it thinks about programming, if it wants quality software. Track two: management is paying for cheaper, lower-quality software. Conclusion: great quality isn’t what’s wanted.

This first made me think of the shift from custom work to manufacturing. But with manufacturing, the final product is generally better than a custom one. Not at first, maybe, but once the production line has had the kinks worked out, the product is consistent, and warranties protect against lemons.

It’s closer to buying low-quality products that are just good enough for how you intend to use them. I could buy a Mercedes, but if I drive a Kia, with the left-over cash I can do stuff that’s more important to me than the experience of driving a sexy, finely-tuned sports car. Tools are a better analogy, since they’re bought primarily for utility. Whether you buy a top-of-the-line model or the cheapest depends on how sensitive you are to its quality, how long you’ll use it, and how important it is to do quality work. This is why professionals and hobbyists have nice gear, and amateurs and casual users have cheap stuff: pros have their reputation and customer satisfaction on the line, and hobbyists are spending their free time using it. I like to program, so I have a nice laptop. I rarely need a chainsaw, so I have a cheap, sissy little 14-inch electric one (in fact, I broke it over a month ago, and still haven’t fixed it).

So if you’re trying to decide which software package to buy, you want to know how much it costs, how good it is, and how good it has to be for you. If you’re trying to sell a software package, you need to know how good your product is, how it compares to the competitors (both on quality and cost), and how good your customers need it to be. And then you need to sell yourself. Right now, U.S. IT departments are pricey, and don’t advertise themselves very well. Off-shore IT shops are cheaper, and have great PR. Guess who’s getting the business? There are other factors that make business people hesitate (just google “outsourcing risks”), but as those problems are solved, U.S. shops will have to get better at selling themselves, showcasing what they can do for the business.

I have some ideas how my IT department could do better: our business customers think we’re too slow, and too expensive. I’ve suggested we try some agile projects, and loosen the stifling framework we work under. IT management sees it the other way: tighten the process so it’s easy to integrate with cheap off-shore vendors. I haven’t made much headway yet, but I think agile’s emphasis on just-good-enough software and tight feedback loops might be the ticket to lowering cost and showing the business just how fast we can be.

What Indians call the ‘@’ symbol

One of The Vendor’s guys-on-site just stopped by to ask me to email him some things. He just came onto the project, and I didn’t know his email address yet, so as I prepared the email, he dictated his address to me. As he spelled out each letter, I typed it in, until he said something that sounded just like “adder-oit”. I asked him to repeat it twice, and both times, it sounded just like “adder-oit”. Oh shit, I thought, maybe it’s one of those letters that has a weird translation, and he can’t remember what it is in English. Like ‘eegreck’ in French means the letter ‘y’. He realized I was confused, leaned over, and typed ‘@’.

“Oh!” I said, “you mean, ‘adroit’? Is that the name you use for the ‘at’ symbol? We just call it ‘at’.” I figured it was one of those fancy names for symbols, like ‘ampersand’ for ‘&’ — easy to forget, and not used all that often in the States. [Remember, Indians typically learn to speak British English, not American, so it would make sense that they learned some things like that. At least, in my version of the world it does.]

“No,” he explained, “we call it ‘at the rate’.” He had to say it a few times for me to get it, but suddenly, I had a mental image of markets, selling stuff: “3 @ $1 ea”, that kind of thing. After a laugh, he sent me some background info on @, which I Googled: here’s the original.

Why do Americans tell themselves they’ll survive off-shoring by being more creative than the rest of the world?

I’ve been reading and thinking more and more about off-shoring. One thing that keeps coming up is the idea that Americans are generally more mentally flexible, or something. People talking about off-shoring say “yes, you’ll lose your mundane jobs, your grunt-work. But you’ll excel by being creative! That’s what Americans are good at!” Two pieces of the idea:

  • We value trouble-makers. People who say “screw off,” and go do it their own way. Cowboys, mavericks, smart-alecks, rebels, scrappy little kids. Clever, resourceful people. Iconoclasts. Main benefit: we feel it’s a Good Thing to charge off and try something, no matter how stupid the idea seems. We take risks, and sometimes that stupid idea pays off. Go figure.
  • We don’t have a rigid class system. We believe, as a kind of core cultural value, that any poor bum with talent has a chance of using his talent to get rich. Call it the American dream. Main benefit: we believe that anyone can be rich/famous/president/CEO, so why not you? Why can’t you be C-E-freakin’-O?

These qualities make us skeptical of old ideas, and eager to embrace new ones. I suppose both of those points go back to how the country was founded: a bunch of poor people said “screw off” (or were kicked out, whatever), went to a different continent, and gradually became a really successful country. At least, that’s the gist of the story as we learn it (I have no interest in talking about how much of that is exactly accurate). But no matter where these values come from, they make us more mentally flexible, more creative. That’s the idea, at least.

In contrast, the Chinese, the Indians, the Japanese all have very rigid societies. They value stability. Respect. Honor. Protocol. The vendor liason I worked with had worked in both Japan and the US, and said that as you move from East to West, the culture becomes less rigid, less formal. Apparently in Japanese meetings, only the two people of the highest rank actually talk, while everyone else listens. The underlings silently shake their heads to indicate whether they agree. In this kind of culture, there’s no “thinking outside the box” (you know you were just waiting for me to say that). The liason said that India was nicely situated between Japan and the US: not-too-rigid, not-too-loose. Able to deal with both cultures. [I guess this is a different kind of flexibility.]

On my off-shored project, I only saw two bits of evidence of this. 1) The Indians were hesitant to criticize our ideas at first: until they became comfortable with us, they erred on the polite side…but that’s hardly being inflexible. 2) — the real piece. They were very reluctant to figure out any problems we threw at them. Any part of the design we left for them to solve, no matter how trivial, they asked us to fill in. [This after repeatedly being told by management, “these guys are geniuses — lean on them, learn from them. Working with them will improve us.” Whatever.] Whenever we expected them to take a pro-active approach to things, to apply creativity or intelligence, we had problems. It seemed like they preferred to grind things out, code-monkey-style, without thinking about it. This puzzles me, because that stuff is the part that I like about my job — figuring things out, solving problems.

Now this all sounds very one-sided to me. Why is it only Americans can be creative? What kind of crock are we telling ourselves? I’d love to hear other perspectives on this — but spare me the “non-US programmers can be innovative, too, jackass!” I know they can. Has anyone seen evidence of this? Any evidence against it? Other than “oh, such-and-such Indian company is very innovative, if you look at their homepage…”? I’m looking for stories from people who actually worked on off-shored projects, and were amazed at the creativity and talent of the off-shore team.

For a good example of this idea, check out The New Face of the Silicon Age from Wired.

Help, I’ve been outsourced, and The Vendor’s code is a mess!

I’ve been working as tech. lead, going full-speed on a project since February. Last May, we learned it would be outsourced to The Vendor in India. Since my team has never outsourced anything before, we suddenly had to figure out how the whole thing would work (of course, there’s no extra time). We already did a decent amount of design work…would The Vendor repeat that work? Or would they work from what we provided them? How would they tie in to our build process? How about our internal services that use our own SOA framework? [We use proprietary, built-in-house frameworks for anything. We’re just now (yes, now) moving off our proprietary MVC framework to the latest, greatest, cutting-edge Struts 1.1 MVC framework. w00t!]

Anyway, it was a hard summer, trying to figure all this new stuff out, finish the design, and not get all gloomy over my career going to India (hence the relative quiet here for the past months). I have some more posts coming about the whole experience, but for now, I wanted to say something about the quality of the code we’re starting to see from The Vendor.

It sucks. The code absolutely sucks. I’ve occasionally seen code this bad from very junior programmers with not much talent. People keep saying “but surely they just haven’t polished it up yet!” To them I say, the mess it’s in makes it, oh, three times harder to understand then if they’d just keep it somewhat clean to begin with. It’s not that I’m expecting a 5-lane highway and getting a dirt road, it’s that I’m expecting a dirt road and getting the thickest, nastiest, buggiest (har har) jungle undergrowth that almost totally prohibits movement. We’re not supposed to code at all on this project, or fix any bugs (the contract says The Vendor will do that), but I’ve had to whip out my trusty refactoring machete and cut some trails through the thicket just to understand the mess. Even the liason seems embarrassed.

Now I admit I like clean code, but I am not being a code priss here. I like to see evidence of thought. I like to see (say) naming conventions. Consistent capitalization would at least be nice. I like to see computer resources being used intelligently. I see none of that here. I’m not pouting that my job is being outsourced, and taking it out on the poor guys in India. This code is far below anything I’d tolerate from my team members — even a junior programmer I’m mentoring.

I don’t want to believe the programmers in India are brain-dead. I imagine, since this was a fixed-bid project that they underestimated, and since it’s their first contract with my employer, that they were eager to impress, and now find themselves trying to do something impossible. [We all know how sales over-promises, and engineering suffers.] I know for a fact that they’re working around 70 hour weeks, and I feel for them. Maybe there are a bunch of coders there running amok, just trying to get the thing thrown together and shipped. But then I think about some of the specific examples I’ve seen, and it almost defies belief. One good example will give a picture. The paraphrased code:

List counties = LoadList("countries");
session.save ("states", counties);

Yes, you read that right. Load a list of countries, store it in a variable called “counties”, and store it in session as “states”. Is it a list of counties? countries? states? Who knows? [I know, I know — maybe “counties” is just a typo of “countries”, but we do deal with counties. If this was the only problem, I’d give them the benefit of the doubt.]

What I really think is going on here (now that I finally get to it) is that the coders in India have a wide talent margin. I think there are coders there who’ve been coding for (maybe) three months longer than they’ve been employed. It’s been said elsewhere, and this isn’t a new thought, but coders in India are in huge demand, and the barrier to entry is very low. So many of them need to make a living, and with demand so high, software is easy money. It’s like the dot-com bubble here, when people who had no business coding were able to make scads of money. My advice for anyone working on a project that’s been off-shored is beware of the newbie programmers — and you’ll probably have some on your project. The Vendor will have some slick, talented people on their team who know what they’re doing, but for every great programmer there’s at least one dud.