Engineering teams that use common tools throughout the entire software development lifecycle are substantially more efficient and productive.

Having the same version of your tooling (compiler, linters, etc.) for the local environment, continuous-integration environments, and build environments ensures you can catch issues sooner, fix problems once, and generally have lower overhead.

This post explores our two-year journey as a remote-first team aiming for efficiency, starting with GitHub Codespaces and a recent shift from cloud-based development environments to environments on our laptops.

Prodvana started with a couple of shell scripts and homebrew on our laptops. After a month of early MVP work, we prepared for more engineers to join. Culturally, we felt it was important for people to value speed and not waste cycles setting up and debugging their development environment. This meant moving towards a repeatable solution.  

Our Requirements

• Natively supports arm64 architecture / provides a fast-enough emulated experience on an M1 Pro with 32GB of memory.

• Setting up a new engineer takes minutes

• It is easy to keep the dev tooling and environment in sync

• Easy to spin up a development environment (and easy to re-create it)

Docker Compose was a clear winner for providing an easy way to stamp out a development environment that matched the containerized environment our production workloads run in.

We hit a few snags as we started using Docker Compose with Docker Desktop for Mac. Docker Desktop uses a Linux virtual machine under the hood, so any container images we use must be compatible with Linux/Arm64; otherwise, the VM must emulate the more common AMD64 architecture.

We started seeing issues with slow disk IO when using mounts between the host Mac filesystem and Docker, another artifact of the VM. This hit us extensively with our frontend development server, which would rebuild and reload when source files were modified. We saw tens of seconds of delays, which made it quite painful to iterate quickly.

We would spend more time futzing with a fragile and slow development environment than actually working on the product, so we decided to look into remote development options.

Enter Codespaces

At the top of the list was GitHub’s Codespaces offering. Codespaces uses the Microsoft-developed dev containers specification to define container-based development environments with VS Code. While some other tools are adopting it, it’s still most well supported by VS Code (there is also work on a reference CLI implementation here).

Since we already used GitHub for code hosting and CI, we decided to proceed with Codespaces. By doing so, we removed the arm64 variable from the equation (Codespace instances are amd64) and avoided the Mac-to-Linux VM complications introduced by Docker Desktop. 

A nice side effect of Codespaces being based on dev containers is that we also have a migration path if we ever need to move off the platform. 

Migrating to Codespaces was straightforward and took minimal time. We could easily onboard new engineers and keep the dev environment up to date by simply maintaining a Dockerfile like we would for our production images. Aside from reliability issues in 2023, we have found it to be a solid tool.

Until it wasn't.

Local Redux

As Prodvana’s service matured, we started experiencing growing pains with Codespaces, starting with hitting machine resource limits. We maintain many end-to-end tests and keep a local development environment that closely mimics the production workloads. For example, we run a kind (Kubernetes in Docker) instance in our dev environment to fully exercise the bits of Prodvana that interact with Kubernetes.

Running our dev environment, kind, language servers, linters, and regularly rebuilding Docker images during development quickly required us to upgrade to larger Codespace instances. We were primarily limited by memory, but even after increasing the instance sizes, we could still not avoid performance issues when building Docker images. 

The root cause of this is Codespace disk IO. The primary disk is on network-attached storage (the instances run in Azure), and if you have IO-heavy workloads, you can quickly exceed your IO budget, resulting in IOPS slowing to a crawl.

To address this, we tried a couple of tricks. The first was mounting the Docker cache directories on a ramfs. This helped reduce build times but put memory pressure on the Codespaces. 

Next, based on a tip from GitHub after a few conversations with GitHub and Microsoft engineers, we modified our dev container’s Docker runArgs to mount a temporary directory from the underlying Azure VM’s SSD drive into the dev container. This way, the Docker builds within the dev container use a faster and (slightly) more predictable SSD for I/O.

# /mnt on the host VM is an SSD mount.
"initializeCommand": "mkdir -p /mnt/pvn-docker /mnt/pvn-cache",
"runArgs": [ "--init", 
            "--privileged",
            "--mount", 
            "type=bind,source=/mnt/pvn-docker,target=/var/lib/docker", 
            "--mount", "type=bind,source=/mnt/pvn-cache,target=/mnt/pvn-cache"

Ultimately, our Codespaces cost grew too high in pure dollar spend for us to find the simplicity of the setup worth the investment. We decided to test the portability of dev containers back to local during our February company hack-a-thon offsite.

The theory was pretty sound, but we found a few gotcha’s that we walk through below in case you’re considering a similar journey.

Moving GitHub Codespaces to Local

CPU Architecture 

Our engineers use ARM-based Apple processors, so we had to update our Docker files to ensure that any piece of software installed did so in an architecture-aware way (this way, we could also maintain Codespaces compatibility). Luckily, over the two years since we first started with Codespaces, all the tools we use have started releasing macOS/arm64 and Linux/arm64 builds.

In a handful of places, we use images fetched from upstream container registries — we found in a few places this required explicitly passing DOCKER_DEFAULT_PLATFORM to docker-compose to ensure the correct image architecture was selected.

Filesystem Performance

Docker Desktop on Mac has improved significantly over the past few years, especially in disk IO performance when using the VirtioFS option, which is available as of Docker Desktop 4.6.  This improvement to disk performance virtually eliminated the propagation delays we had experienced prior. 

Docker Desktop Replacement

Informal testing of Colima and OrbStack as Docker Desktop alternatives showed that, anecdotally, OrbStack is slightly easier on system resources locally. It works as a drop-in replacement and reduces the performance tuning needed. For example, it does the same VirtioFS option that Docker Desktop does for faster disk IO.

Secrets

In Codespaces, we use Github Secrets to inject development secrets (used for end-to-end testing things like Auth0, Slack integrations, etc.). Moving to local development, we did not want to lose the ease of use this provided. It was untenable to go back to the days of manually sharing a .env file with new engineers or the confusion it causes the team when someone adds a new variable and forgets to let everyone know.

We use 1Password for password management, and their command-line tool worked perfectly to inject our dev secrets. We do this by using the userEnvProbe feature of dev containers, which allows you to inject environment variables dynamically.

Hooking into this provides a very slick experience—when a new dev container is started, 1Password prompts the user to authenticate, and then the secrets are injected. The friction here is minimal compared to Codespaces and was one of the most significant fears we had when working with local development.

It took us 2-3 days to productionize this setup, thus moving us back to a local development environment! It’s been a surprisingly smooth transition. The most significant downside has been reduced battery life (expected now that we’re utilizing the computing power on our laptops).

Next Steps

Given how seamless the transition back to local development has been, our list of future enhancements has been relatively small:

1. Upgrade Remix to take advantage of the new development server. The version of the Remix dev server we’re currently using has a known memory leak, which causes the server to crash frequently (this happened in Codespaces as well, but less frequently).

2. Try Jetpack’s Devbox, which is a Nix-backed development shell. Since it doesn’t use containerization to provide a hermetic build of packages, it would eliminate the outer container (the one where the editor and dev tools live) and run natively on the host MacOS.

3. Use volume mounts to avoid mounting the repo in from MacOS to further improve IO performance.

Our Findings

Starting with a cloud development environment reduced the work needed to run our development environment and allowed us to grow into a more complex setup. 

Given the ubiquity of arm64 builds for Mac and Linux now and new options like Devbox and DevPod, we would likely have tried local development earlier if we had started today.

If you find what you learned and the approach useful, please subscribe to our blog!