Building software can be a frustrating experience. In the times of ancient history, there was only one real choice. The build system that the original developer set up. Did they use pkg-config to tell your app where their libraries are? Did they put their headers in a spot where the compiler can find it or do we have to configure paths for these things? The result was basically to run the ./configure script over and over, and each time it stops with an error, go fix whatever it's complaining about. Building software was often a huge rabbit hole, colloquially known as "dependency hell", that wastes everyone's time by not focusing on their actual goals. Docker and Make is an incredibly useful combination of software that makes things much easier.
Docker gives us the power to run complex build systems with native dependencies using minimal commands to achieve it. Make of course ties things together and gives us an easy to remember top level command for building our projects.
I have found that using this combination minimizes the amount of software that I have to install on my desktop. I don't need NodeJS nor NPM. Don't need Python. Don't need gcc. Don't have to install or build interim tools for transforming code. Because all these things are in some public Docker container already that we can use on demand.
Side note about this blog post. There is of course many systems you could use to trigger the Docker commands described here. Make is not the end-all-be-all of build systems. However our needs here are simple and Make is ubiquitous. It only needs to make the commands simpler and detect if anything changed.
Let's have an example. One day you're browsing Reddit and a post comes up in r/haskell about a new tool someone wrote that can turn your grocery list into a working program that makes you dinner. It can output Go language code which is perfect because then you can integrate dinner with your other project, grocery-list-as-a-service which is written in Go.
Your team of a dozen eager programmers can't wait to hear about your decisions about direction of the project. Every single one of them will cry tears of joy while figuring out how to install and run Haskell, grocery2dinner, and golang to build the app that they all have to work on.
Enter Docker. The grocery2dinner authors have helpfully provided a container that has grocery2dinner in it's system PATH. And of course, Google has done the same for go. Now we can skip the bits about installing Haskell, grocery2dinner and Golang dependencies locally and just get straight to the juicy bit about using the tools we want to use.
Docker in a build mode
There is a straightforward Docker configuration that can mount the current working directory, run a command, write output back to the working directory.
$ docker run --rm -v $(pwd):/app -w /app -it image command args...
Let's review this.
- "--rm" tells Docker to remove this container when we're done. We don't need to keep the runtime image around after running, we just want to run the command for the output.
- "-v" argument is a "bind mount" volume. We are mounting the current directory in the host machine to /app inside the container we are about to run. $(pwd) does not work in all circumstances as discussed in more detail below.
- "-w" specifies the working directory inside the container so that we don't have to `cd /app` before doing work.
- "-it" is optional. It runs an interactive terminal which produces more colorful output during builds and of course can stop and ask you questions. If your build is entirely non-interactive and/or you plan on running this on a server (e.g. continuous integration) you probably want to leave this off.
- "image" is the image to run as listed on Dockerhub.
You may want to make an alias or script version of this for yourself, but let's create one that works well inside our Makefile.
The Makefile
cygpath = $(shell which cygpath)
wd = $(shell $(if ${cygpath}, ${cygpath} -am ., realpath .))
run = docker run --rm -v "${wd}:/app" -w /app -it
Whoa whoa hold up, what's this?!
It illustrates a minor issue with the basic Docker build command outlined above. $(pwd) doesn't always do the right thing. On my main day-to-day terminal Cygwin, $(pwd) returns a path like `/home/fletch/ifeellikechickentonight`. But Cygwin doesn't run Docker Desktop. Windows does. So a docker command with a bind mount is expecting Windows paths. Not Unix-y ones.
What I'm doing here is checking for Cygwin by testing for the `cygpath` utility. If it exists, we get the current working directory from it in Windows "mixed mode" path. Otherwise we use the common `realpath` utility to get the working directory. It would be nice if we could just pass a "." to Docker but this results in an error about "volume name is too short." Attempting to use the --mount option to mount "." as "src" results in "invalid mount config for type 'bind': invalid mount path: '.' mount path must be absolute." So we're currently stuck with figuring out the right absolute path to use.
This is the thing most likely to go wrong when deploying this setup.
A more concrete example
Let's abandon the theoretical grocery service and look at a more concrete example. The tool ANTLR is a parser generator for describing a language or text format with a grammar notation and turning it into code that parses input in that format.
ANTLR is built in Java but can produce a parser in Java, Javascript, C#, C++, Go, PHP, Python 2 or 3, and Swift. If we're outputting to any of those languages except Java, we really don't care about Java. There's a Docker image called "vlinder/antlr4" that we can use here.
parser: MyParser.g4 MyLexer.g4
${run} vlinder/antlr4 antlr4 -Dlanguage=JavaScript -o parser MyLexer.g4 MyParser.g4
We can now run:
$ make parser
It will run `antlr4` and produce Javascript output in the ./parser/ directory.
Make is smart if we tell it the names of our source files and the output directory as above. It should skip the "parser" target if the output directory is newer than the source files.
Create a new NPM project
Next let's build an NPM project to run our parser. Before we dive in to the Makefile again, we're going to need to create a new NPM project. No problem!
$ docker run --rm -v $(pwd):/app -w /app -it node /bin/bash
root@c63097a0b098:/app# npm init
(... answer questions ...)
root@c63097a0b098:/app# ls
package.jsonroot@c63097a0b098:/app# npm i antlr4 lodash
( ... )
root@c63097a0b098:/app# exit
Cool! Now we have a NodeJS project that can run our parser.
Let's create a new Makefile target for it:
node_modules: package.json
${run} node:13 npm install
ANTLR4 Javascript output currently needs NodeJS version 13 so we load that. This target will replace node_modules if it does not exist, so it can be added to a project's .gitignore file if desired and recreated at any time. Also because we've put package.json as the source and node_modules as the output, Make should recognize when package.json has changed and run this target only when package.json is newer.
Completed solution
Let's tie it all together.
cygpath = $(shell which cygpath)
wd = $(shell $(if ${cygpath}, ${cygpath} -am ., realpath .))
run = docker run --rm -v "${wd}:/app" -w /app -it
all: node_modules parser
parser: MyParser.g4 MyLexer.g4
${run} vlinder/antlr4 antlr4 -Dlanguage=JavaScript -o parser MyLexer.g4 MyParser.g4
node_modules: package.json
${run} node:13 npm install
shell:
${run} node:13 /bin/bash
test: parser node_modules
${run} node:13 npm test
.PHONY: all shell test
Note that we have added an "all" rule here. By defining it first in the Makefile, Make will use this target if none is specified.
We can now run:
$ make
Which will build the whole project.
We can also:
$ make parser
$ make node_modules
$ make shell
$ make test
The "make test" and "make shell" commands are handy for day to day usage. "make shell" can be used to easily add new NPM dependencies for example. Note that "make shell" requires the "-it" flags in the run command as specified above. If you leave "-it" off of the general run command, you could just insert it after ${run} in the Makefile target.
The .PHONY entry in the Makefile specifies targets that are not connected directly to files. The "parser" and "node_modules" targets both are connected to directories on disk. If the output directory is newer, the target would not be run. .PHONY tells make that these three targets "all", "shell" and "test", are not related to any files on disk but just contain commands to run. Credit goes to /u/pathslog on Reddit for pointing out this was missing from the original post.
We can of course run our program in a NodeJS container as well.
$ docker run --rm -v "${wd}:/app" -w /app -it node:13 node index.js
It may be wise to make a run.sh script with this command.
Results
Using only Docker and Make along with images found on Docker Hub, we have achieved quite a lot.
A parser that needs Java without installing Java. A NodeJS project that needs Node and NPM without installing Node and NPM.
A zero-installation development environment that re-creates itself at the drop of a hat.
A clean desktop.
Dependencies: Docker Desktop and `make` in a terminal. The end.
Contact us for professional Docker consulting services.