Making Distributed Development Easier

Recently my life took a swing in an unexpected direction. I was provided the opportunity to work remotely from the comforts of my own home two states away from Velir's offices. The transition from the office environment to working remotely was a challenge. I needed resources both locally to myself, and at Velir's offices. I would say that overall, to find and implement the tools and configurations that would allow me to work on certain projects effectively, took about a month. This post entails adventures from my trip down the Distributed Development rabbit hole and where I ended up.

Distributed Development

In Distributed Development, employees work from various locations and worksites. Through the use of email, the internet, and other communication devices, in this day and age, Distributed Development has become a very popular method of bringing people together on a range of projects. This method is different from outsourcing and subcontracting, where the company pays others to produce work for them. Rather in the Distributed Development model, team members are employees of the company that owns the project. If you want to see a clear cut case of success in Distributed Development, then take a look at the story of how Stack Overflow was created, and see how it affected the overall project. Distributed Development is also a well-known word around companies like Microsoft, whose reach is global and because of this, practical Distributed Development becomes essential to the survival of globally reaching projects.

From my experience on the topic, there are a lot of issues that arise when you move your working environment away from the resources that it needs (centralized resources). These resources can be anything from database connectivity to network connectivity, in terms of specific IP addresses, SMTP servers, or even auth services. This post seeks to layout a subset of challenges that a developer would face when attempting to work remotely, and what he or she can do to subdue those beasts.

The Environment

In a development environment where you have a centralized location for resources, the best option for accessing those resources, aside from being on its network, is to use a VPN solution. Logging into the VPN makes your computer seem like it's on the remote network and it gives you access to the remote resources. So is that it? Does VPN solve all of our issues with Distributed Development? While VPN is the first step to accessing resources, there are several issues that crop up because of the nature of Distributed Development. Let's examine a project scenario that I am currently working with.

Say that project X is a web-based Sitecore project that needs access to a database server. Project X requires access to Client A's staging services for user authentication and access to other web services. Since this is a staging service on the client's side, and because they are security mindful, they use an IP address filter to limit access to their staging services. It is clear that working remotely on Project X at this company's primary location, and with this IP address limitation, can cause a myriad of problems.

The Problems

Here is a list of issues that will and can arise when dealing with a project like this:

1. Connecting to large databases over a VPN is not practical for a Sitecore solution. The limited bandwidth on these databases can cause the server to run slowly and can cause connection issues due to timeouts.
2. Client A's use of IP address filtering causes the following issues from outside the company's network:
   • Access to any resources from outside the company's network will be denied. This means requests for authentication will come back invalid. Trying to access any staging website or Sitecore back-end will be blocked.
   • If the client is hosting the code repository on a system like TFS (Microsoft's Team Foundation Server), then the IP address filtering will prevent the remote developer from being able to check-out/check-in code, or to be able to grab the latest version of the code from the repository, or even make any other type of request (like getting history, merging, branching...)

These issues can cripple the ability of remote developers to do any meaningful work and render them unable to perform their jobs.

The Solutions

The solutions for working on this type of environment are overall very easy to implement given the availability of proper development hardware.

Database

The requirement for a database can be overcome simply by installing a database server on the machine that will be performing the work remotely. The biggest issue with this solution is that the developer has to be diligent in making updates to the database. This can be done either though database backups, or through the Sitecore Packaging system. As long as the database can be kept up to date without too much hassle, this solution allows the developer to run the database resources locally and to be able to load the website and Sitecore back-end. Note however that you need to have at least the standard version of the MS Sql server to run databases greater than 2 GB. For some Sitecore instances this might be an issue if you are using the MS Sql Express version.

IP Filtering: Proxy Auto Config

Dealing with IP address filtering can be a challenge. The obvious solution to this is to use a Proxy Server on the company network over the VPN connection. This combination of Proxy Server and VPN is powerful. It allows you to not only get access to resources on the remote network, but also allows you to view the rest of the world as though you actually were part of the remote network. All of your traffic is filtered to the remote network, and then out to the rest of the world, giving the impression that you are coming directly from the remote network. If developers push all of the network traffic through the VPN however, this can cause network congestion and other issues. So often connecting to the remote network's Proxy Server is not allowed through a VPN connection. The solution to this issue would be to selectively push traffic through the remote network's Proxy Server so that IP address filtered resources (such as client web services) can be accessed properly.

Introducing the Proxy Auto Configuration. On my machine in the default website of IIS I created a file called proxy.pac, which is a file that contains a function to determine what to do with the connection. I have a proxy setup at Velir to handle two connections. The first is web. The proxy is on port . This internal proxy is only available when connected to the VPN. Using the proxy auto configuration script, I can redirect certain requests through specific connections. For example, let's pretend that the proxy is located at the IP address 192.168.222.222 on port 8123.

function FindProxyForURL(url, host) {
  if (shExpMatch(url,"http://www.google.com*")) {
    return "PROXY 192.168.222.222:8123";
  }
  return "DIRECT";
}

In this example, you can see the use of the function shExpMatch, which takes the URL and a pattern to match on (see the references section for a Wikipedia article on PAC files). I found it sufficient to use the domain with a wild card character after it. Once the URL is matched, you send it to the proxy of your choosing. The last statement is a fall through where if nothing is matched, the "DIRECT" connection is selected. Once the script has been written, you need to add the mime type to IIS to make it accessible.

Once the mime type is setup, hit the URL http://localhost/proxy.pac on your browser to download the file. Once the file is hosted, you can now add it to your VPN connection information. To do this, you can access the connection settings by going to your control panel > Internet Options. Click on the "Connections" tab. Here you will see a list of Dial-up and Virtual Private Networks. For these settings, we only need to apply the Proxy Auto Config to the VPN connection that we want to work with. Click on that connection and then click on settings.

Check the box for "Use automatic configuration script" and set the address to the one you setup for the PAC file.

Now when you connect to your VPN connection, these new settings will take effect and use the proxy auto config to selectively push connections to a proxy server on the VPN network. This method works with any proxy location on any port, so applications like Visual Studios, that may need to connect to a TFS server, can be setup to use the auto config to pass any TFS traffic through the VPN proxy.

IP Filtering: Web Services in your Web Application

Another issue with IP Filtering that you will run into is trying to access a client's web services. For Web Applications running in IIS, the Proxy Auto Config will not work. IIS does not pay attention to this setting. There is however another way that you can configure a proxy server for a web application in the web.config file. This is called the Default Proxy element.

In the system.net section of the web config, you can set a "defaultProxy" which takes a proxy and module object as well as a bypasslist object. I found it easy enough to just use the bypassonlocal. If you notice, the proxyaddress value is the same that we used for connections in the PAC file. These settings force IIS to use a proxy for any non-local connections, like calls to remote web services.

The Result

While it can be tricky to have an effective Distributed Development model, the tools exist to facilitate this growing development trend. These tools allow developers to integrate their systems seamlessly with remote resources, thus increasing their productivity and expanding the availability of good programmers. For me, the reasons for working remotely were personal in nature. If my development productivity was hampered because of projects and resource limitations, I would no longer be a valuable employee, but instead a liability. I am happy to say that I have found a balance which allows me to be as effective of a programmer-developer while working from home, as I would be if I worked from the office.