Managing Offshore Programming Projects

When meeting with potential clients of my company, Dabble Communications, I describe our use of proven software development methodologies: we create detailed requirements documents, we build mock-ups of the system, and so on. Just as their eyes begin to glaze over I say, "We turn the design over to a programming team in India or Romania." People suddenly perk up at this point: "Really?", "How does that work?", "That's amazing.", "Do they speak English?". The use of offshore programming by US firms is a huge business. In 1998 and 1999, India alone exported $3.9 billion in computer software and services to the US. Nevertheless, many companies and individuals have not encountered the practice directly. For the uninitiated, I will attempt to demystify the process. For others who are leveraging this incredible resource and, like us, are struggling to perfect the process, I offer our experience, which includes projects ranging in size from 100 hours to 1500 hours. The typical project size is 400 to 800 hours of effort, and the development tools used include Java, Active Server Pages, Delphi, Visual Basic, and C++.

I should point out that this article is about using offshore teams for programming only. Dabble staff members are responsible for gathering requirements from the client, designing the software, developing the specifications, testing the software, and implementing the software. We believe that this is the best way to leverage offshore talent for software development. The design phase of the software development process is the most difficult one and the most important. Other difficult aspects of a software project are managing the project, ensuring the quality of the software, and managing the client relationship. As such, we can offer our clients the most value by focusing our energies in these areas. In general, our clients would not experience the level of quality or customer service that they receive from us if they worked directly with an offshore team. First, designing software is difficult and it's a rare skill for any firm whether the firm is local or offshore. Second, the communication, cultural, and distance challenges would make it very difficult for an offshore team from India or Eastern Europe to design a software application for a US client. Third, as much as I believe in the power of distributed teams, there is no substitute for face-to-face meetings when designing software. Brainstorming meetings, user interviews, and focus groups are much more efficient and effective in the real world than in the virtual world.

In many ways, the use of offshore programming is no different than having a team of programmers across the hall. In both situations, it pays to have a thorough design with meticulous attention to detail. Well written, comprehensive documentation, a technical architecture developed by an experienced engineer, and client-reviewed mock-ups lead to efficient software development. This is well known and described fully in books such as Steve McConnell's Rapid Development. The difference is that short-changing the design phase is likely to have a larger negative impact on offshore development projects than on projects where the entire team is in one location. Hacking out an application is never a good idea, but recovery may be easier when the programmer is in the next cubicle. If I'm managing a software project with an on-site programming team and I neglect to document the functionality of a button, the programmer can say, "Hey Jeff, how is this button supposed to work?" If the programmer is halfway around the world, this oversight typically means that there will be at least one working day before the programmer knows the answer.

Nevertheless, for software architects who are in the habit of creating thorough specifications, it is not particularly difficult to make the leap to offshore development. In a typical project, the offshore team will ask a few questions about the specification at the start of programming. The programmers will crank out the software and may ask a few more questions along the way. We generally ask for weekly status updates from the programming team, but the initial code delivery may take six weeks or more. When the first code arrives, it is generally as good as initial code deliveries from local programming teams with which I have worked. With all but two of the offshore projects that we've undertaken, the first code deliveries have resulted in the satisfaction of seeing our software designs come to life. I will describe the two that did not go so well later in the article. If you are an experienced project manager and you already provide your programming teams with complete specifications, you should have no fear in tackling offshore programming projects. Of course, you have to have a good team and we've experience good firms as well as bad ones.

Selecting an Offshore Partner

What is the most difficult part of a software project? I would argue that estimating time frames and hours of effort is the hardest thing to do in any project. However, finding the right programming team is almost as difficult.

To choose an outsourcing partner, we look for international technical certifications such as ISO-9001 and SEI-CMM Level 3 or 4. This doesn't necessarily tell you that the firm can deliver high-quality software, but it at least tells you that it understands quality assurance concepts and has gone through a process to implement company-wide standards. We look at work product, and we call references. These are two important steps that should not be shortchanged. We ask for code samples and review the code for commenting, naming conventions, well-structured SQL statements, and just plain beautiful code. We value programmers who take pride in their work and invest time in code aesthetics: consistent indenting, nicely formatted procedure headers, and lots of white space. Beautiful code not only makes the code easier to read for maintenance, it is also a sign of programmers who take pride in their work and are meticulous and detail oriented.

The size of your company and the size of your projects will drive the type of relationship you establish with an offshore company. If you expect a steady flow of work using consistent programming languages and operating systems, it may be ideal to have an offshore team dedicated exclusively to your company. A dedicated team will give you more continuity. Our projects have required a range of programming languages and platforms. Thus, we have not established any dedicated teams.

Should you work with a large firm or a small firm? Having worked with companies ranging in size from 20 programmers to 800 programmers, we found that the overall size of a company is not as important as the quality of the few people that you work with on your project. The fact that you chose a fast-growing firm with a great reputation will not help you overcome a situation where you are assigned a poor project manager or inexperienced programmers. While a large organization may be better able to quickly replace individuals that don't meet your needs, a smaller firm may be hungrier for your business and may try harder.

Other factors to consider when selecting an offshore partner include turnover, corporate focus, and corporate culture. It seems that high turnover is commonplace in India. There are a tremendous number of technical firms and programmers. The firms compete for the best talent and people tend to jump from firm to firm. One company I talked to has resigned itself to high programmer turnover. Instead, they tout their low project-manager turnover. Their philosophy, which is somewhat valid, is that the individual programmers are less important than the project managers. Adding to the turnover problem is the fact that the ultimate goal for many programmers is to get to the United States. Frequently, programmers will change firms or do whatever they believe will improve their chances of getting staffed on a US project.

Our experience suggests that turnover is less of a factor in other countries. On the Romanian teams we have worked with consistency is the norm. The programmers there tend to find a firm that they like and stay. From what I've heard, the same is true with Russia.

In terms of corporate focus and culture, consider a firm's commitment to offshore development. Some firms take offshore projects and also place programmers on site at companies throughout the world. When evaluating such a firm, try to determine their primary focus. Ideally, you want to find a firm that is committed to the offshore model. Other firms provide both medical transcription services and programming services. Where are they devoting their energies? Are they trying hard to attract and retain the best programmers or are they programming on the side as they build their transcription business?

Testing the Waters

No matter how much due diligence is performed, starting work with an offshore firm requires a leap of faith. If possible, start with a small project that your in-house staff can take over in the event of a disaster. For example, We recently tried a new Indian firm by giving them a small site enhancement project that was about 100 hours of programming. The firm did such a poor job with the programming and with meeting deadlines that we dismissed the idea of a long-term partnership. I personally did some programming to patch up holes in the enhancements.

Most of our projects have gone smoothly: the offshore teams have understood our designs immediately, and they have asked relevant questions whenever there were holes in the specifications. But a recent project underscored the importance of a strong offshore project manager. Your local software designers will know the system inside and out. You need a counterpart on the offshore team. That is, you need somebody who knows the system as well as you who can communicate your vision of the application to the programmers. This particular project required about 1400 hours of development effort and was split into two phases. We hired a company that we had worked with before and we were comfortable with their staff and their processes. The first phase went according to our expectations and the quality and timeliness was consistent with their prior work. The second phase was delivered late, there was a lack of communication about the progress, and there were many problems with the initial code deliveries. What changed? There were four programmers on the project and one project leader. Three out of the four programmers from phase one worked on phase two as well. But the end of phase one, the original project leader was transferred to the United States to work on another project.

The first sign of trouble was in getting a schedule for delivery of the phase-two code. I asked for a schedule several times via e-mail and emphasized the importance of meeting the project deadline. Each time, I was assured that the deadline would be met and that a schedule would be delivered soon. During these communications, I learned that the project manager had been replaced. The office manager told me that the new project manager needed to get up to speed and then he would let me know the delivery schedule. As the deadline neared, I called the project manager and he assured me verbally that the deadline would be met. You have to follow your instinct when getting assurances like these. If something is not right and it seems like the deadline will never be met, assume it won't, regardless of any promises you may be getting. In phase one, all of the e-mails came from the project leader. In phase two, I suddenly started getting e-mails from the programming team. I rarely heard from the project manager. We now know that this is a bad sign.

The team did not meet their deadline. They delivered the code a few days late, and it was woefully short of meeting our specifications. Features were implemented incorrectly, features were missing, and the web page layouts were sloppy. Our quality assurance staff worked hard to identify and document all the problems. We were identifying and documenting things that should not have been problems at all, resulting in increased testing costs.

As the pressure from the client grew to deliver a working system, we went through several quick iterations of coding, implementing, testing, and reporting problems. Our staff and the offshore team worked late into the evenings and on weekends to get the system working. We had significant communication problems. We use a bug tracking system to manage bug reports. The offshore team flagged bugs as fixed that, in fact, were not fixed. This wasn't malicious. They did not understand what the bugs were. It was almost a twilight zone experience: I felt like the team did not understand English. For example, phase two required that two screens from phase one be significantly updated. There was no way to confuse the phase one screens with the mocked-up phase two screens. The screens were not updated. This was submitted to the bug tracker and flagged as fixed by the team. I said, "Bug number 56 is not fixed. Here is the link to the mock-up of the phase two screens. Compare them to the screens you delivered." Their response was "We do not understand what the problem is." Finally, I pasted four screen shots into a Word document with comments, "Here is what the screens looks like now. Here is what the screens are supposed to look like." At last, that did the trick.

To address some of the other problems, I had lengthy instant messaging chat sessions with the programmers. This was incredibly helpful. Essentially, I had to take over the role of offshore project leader and explain things directly to the programming team. In retrospect, I should have stepped into this role a little sooner. If you can identify a project leader problem early in a project, I recommend replacing that person in order to avoid taking on that role yourself. If you are nearing the end of a project and have a tight deadline, I recommend taking over the reins as soon as possible to minimize delays. New Delhi, India is 10.5 hours ahead of Chicago. At about 10:00 or 10:30 p.m. Chicago time, the Indian team is just starting their workday. I'm a night person, so this worked out well for me. I was able to have chat sessions with them during their morning. During this crisis period, I spent hours at a time going through the functionality and describing features in great detail using examples. I didn't leave a topic until I was convinced that the programmer fully understood the feature. The night before one critical code delivery, I slept in front of my computer with the volume turned up in order to hear any incoming chat requests. The project was delivered late and our credibility with the client was damaged.

Leveraging the Time Zone Difference

There are advantages and disadvantages to the time difference. One advantage is that programming and quality assurance efforts can be staggered. When the offshore team sends new code at the end of their workday that means the code arrives at our office in the early morning. We can install the software and test it while the programmers get some sleep. We can deliver our initial test results to the offshore team before they start work the next day. This staggered approach keeps the project moving ahead quickly during the debugging phase. The biggest drawback is that critical programming problems cannot be addressed immediately. For example, I had a meeting scheduled with a client to demonstrate the latest version of the software. There was one key area that had a couple problems, so the programming team was working on the code during the prior day. The code was waiting for us in the morning, but the team failed to include one of the program files. By the time we realized the file was missing, it was 10:00 p.m. in New Delhi. I called our account manager in India on his cell phone, but he was unable to locate a programmer to help us at that late hour. In this particular case I was able to hack a temporary solution for the demonstration, but this might not always be possible.

Some people are interested in "round-the-clock" programming. Theoretically, programmers in the US could work with offshore programmers in shifts to allow programming on a project to continue twenty-four hours a day. Although nice in theory, it seldom works in practice. The time required to transition a programmer's work to another programmer every day makes the process inefficient. Consider the situation where a programmer in the US and a programmer in India are sharing the programming effort for a single module. At the start of each workday, the programmer in the US or India has to study and understand the changes made by his counterpart during the previous shift. Because of this inefficient use of time, round-the-clock programming is rarely attempted.

Managing Effectively

Once you have a quality offshore team in place, how do you make sure your projects go smoothly? In general, the practices that make on-site projects go smoothly still apply; however, there are some unique factors involved in managing an offshore project. Early in a project, we involve the offshore team in the technical architecture decisions. Such decisions may include the choice of programming language, the selection of a database, whether to use an application server, and so on. This is smart for two reasons. First, it gives you a better architecture. The offshore team has experience and knowledge that should be tapped. Second, it gets the offshore team invested in the project at an early stage.

We provide a coding standards document for every project. Our coding standards document defines the commenting style, the naming conventions, and the code formatting style. Moreover, we repeatedly state that carefully written, well-commented code is important to us and that we want the team to take the time to do a nice job with the code. Once the code is delivered, we look through it to make sure that our standards were followed.

Communication is a unique challenge. First and foremost, your documentation and mock-ups need to be thorough. For post-specification communication, we have found that simple, tried-and-true tools such as e-mail and instant messaging work quite well. It is wise to keep things simple. Tools such as videoconferencing and online whiteboards may add some value, but are not essential. Phone calls can be difficult. In my experience, there are latency problems in phone calls to India resulting in a half-second delay between the time you say something and the time the person on the other end hears you. The latency problem combined with heavy accents can make phone calls seem almost worthless. I'd much rather have an instant messaging conversation with an offshore partner. A side benefit of instant messaging is that you can save the chats and refer back to them later. On the negative side, e-mail and instant messaging lack the warm-and-fuzzy attributes that can help unite a team and make people feel connected. We haven't found this to be a serious problem, but we intend to experiment with video e-mail and videoconferencing to supplement e-mail and instant messaging.

Patience, praise, and empathy are important. These are important behaviors and practices with any software project, but it is easier to overlook these factors when your communication is through the impersonal mediums of e-mail and chat. We don't hesitate to praise our teams when we honestly admire the work they have done. Likewise, we try to show some empathy when things are not going so well and the team is working hard to make things better. Patience is essential. You have to be prepared to explain your specifications and explain them again if, for whatever reason, the team is not grasping your vision. In general, you should think of the offshore team as your partners rather than subcontractors. One of the reasons that our projects have gone smoothly is that all of our software designers are experienced programmers. We know what it is like to be in their shoes so we can empathize, and we can give them what they need to get their jobs done.

As I mentioned earlier, one tool that we use is a web-based bug-tracking tool. Our quality assurance people submit bugs to the database by e-mail or through a web browser. The bug report is stored in a database and forwarded to all team members via e-mail. As the bugs are addressed, they are moved from an "incoming" folder to "in-progress", "fixed", or "trash" folders. For testing, we implement the software on a Windows or Unix test server. We give our offshore teams Telnet or pcAnywhere access to the test server. In general, we install the software and manage the database ourselves. Therefore, the remote access is not used by our offshore team very often, but it can prove useful if there is some unique problem occurring on our test server that the team cannot replicate on their own server. We use a version control system that runs over the Internet, but we have not asked our offshore teams to check code in and out directly. Rather, we receive the code delivery, compile, install, and check it into our version control system. Typically, the offshore team has their own version control system that they use during development.

To keep a project on track, it is important that you meet your own deadlines for delivering specifications to the programming team. When working on a multi-phase project where you are delivering specifications over time, missing target delivery dates can have both expected and unexpected consequences. The expected consequence is that the delay may negatively impact the project completion date. The less obvious consequence is that you may lose one or more team members. The offshore firm has to maximize the utilization of its resources and cannot afford to have key employees idle or working at less than full capacity. The change in project managers described earlier was partially due to the fact that we were about one month late delivering our phase two specification. In retrospect, I would have asked the team to explain the potential consequences of delivering the specification late. We may have been able to speed up our delivery or deliver a partial specification on time.

We have found that it is important to review the code itself upon the first code delivery. It is not enough to simply test the functionality. Occasionally, we catch problems in the code such as database connections that are not pooled or inefficient SQL statements. Catching these problems early can save time and trouble later in the project. If the initial code delivery is going to take four weeks or more, it might be useful to request a partial code delivery after two weeks strictly for the sake of reviewing the code.

Our clients do not communicate directly with the offshore team. There is no need for this direct communication and it can lead to confusion. The Dabble software designers are responsible for the software. As such, they can address all questions, concerns, and suggestions from the client. At the same time, we do not attempt to hide the fact that we use offshore programming. In fact, it is a key part of a marketing message, and we sell it as a benefit for our clients.

Code Quality and Legal Issues

On occasion, we encounter skepticism about our use of offshore programming. You may experience some of the same kind of skepticism when trying to implement an offshore programming practice in your own organization. When we first started working with offshore teams, some people told me that the quality of the software could not be controlled. We have proven this to be false. Good project management skills and a thorough software design are the keys to high quality software whether the programming team is down the hall or in another hemisphere. Moreover, careful selection of the programming teams and testing the waters with small projects greatly improves your likelihood of a successful project. Some potential clients have the misconception that we are "farming out" the project as if Dabble was primarily a sales organization. Nothing could be further from the truth. We add tremendous value by focusing on the most difficult aspects of software development. Programming is not quite a commodity, but it is close. There are good programmers and bad programmers and there can be a vast difference in productivity from one programmer to another; however, there are many programming teams throughout the world that can efficiently deliver a well-programmed system to meet the specifications for any given project. Consider Dell computers. They don't make hard drives or microprocessors or motherboards. They find the best companies around the world that make those components and they put them together in a way that meets their customers' needs. Dell is successful because they focus on the most difficult aspects of the personal computer business: designing and assembling high quality systems and providing great customer service.

Legal concerns can be an obstacle. What recourse do you have if your offshore partner steals your intellectual property? We have contracts with all of our offshore partners, but we have not yet encountered a situation that raised a legal problem. I think it is a legitimate concern especially for software projects where the software itself has a very high intellectual property value. Some offshore firms, such as Global Advance, incorporate in the United States to give their US clients peace of mind.

Although there are some risks and challenges, the advantages make offshore programming worthy of careful consideration. Give it a try. It may forever change the way you think about building software.