I've never really known what to do with my personal site. Over the years it's been a dumping ground for links to different projects, and played host to various half-hearted attempts at blogging. But it's never really had much in the way of an actual purpose.
I decided to start afresh and relaunch this site with more of a focus. After speaking to the guys from Techzing, I'm going to hunker down and focus my efforts on learning Go really well, with the aim of possibly doing some consultancy work around it in the future. So over the coming months and maybe even years, I hope to create a lot of useful content for anyone else doing the same.
Because it's also full redesign of the site, I'll do a little colophon.
The site is now just static content, although I use Sass for stylesheets and Markdown for writing blog posts (both of which are compiled on my local machine before publication). Some custom Go code handles the routing and templating, and it's all hosted on Heroku.
For development I used Ubuntu as my operating system, Sublime Text as my editor, Git for version control, and Dropbox for real-time backups.
So with the mandatory first new post out of the way, I'm looking forward to doing a lot more with this site in the future!
I've recently moved the site you're reading right now from a Sinatra application to an (almost) static one served by Go. While it's fresh in my head, here's an explanation of principles behind creating and serving static sites with Go.
Let's begin with a simple but real-world example: serving vanilla HTML and CSS files from a particular location.
Start by creating a directory to hold the project:
$ mkdir static-site
$ cd static-site
Along with an app.go file to hold our code, and some sample HTML and CSS files in a static directory.
First we use the FileServer function to create a handler which responds to all HTTP requests with the contents of a given FileSystem. For our FileSystem we're using the static directory relative to our application, but you could use any other directory on your machine (or indeed any object that implements the FileSystem interface). Next we use the Handle function to register our FileServer as the handler for all requests, and launch the server listening on port 3000.
It's worth pointing out that in Go the pattern "/" matches all request paths, rather than just the empty path.
Go ahead and run the application:
$ go run app.go
Listening...
And open localhost:3000/example.html in your browser. You should see the HTML page we made with a big red heading.
Almost-Static Sites
If you're creating a lot of static HTML files by hand, it can be tedious to keep repeating boilerplate content. Let's explore using the Template package to put shared markup in a layout file.
At the moment all requests are being handled by our FileServer. Let's make a slight adjustment to our application so it only handles request paths that begin with the pattern /static/ instead.
Notice that because our static directory is set as the root of the FileSystem, we need to strip off the /static/ prefix from the request path before searching the FileSystem for the given file. We do this using the StripPrefix function.
{{define "title"}}A templated page{{end}}
{{define "body"}}
<h1>Hello from a templated page</h1>
{{end}}
If you've used templating in other web frameworks or languages before, this should hopefully feel familiar.
Go templates – in the way we're using them here – are essentially just named text blocks surrounded by {{define}} and {{end}} tags. Templates can be embedded into each other, as we do above where the layout template embeds both the title and body templates.
First we've added the html/template and path packages to the import statement.
We've then specified that all the requests not picked up by the static file server should be handled with a new serveTemplate function (if you were wondering, Go matches patterns based on length, with longer patterns take precedence over shorter ones).
In the serveTemplate function, we build paths to the layout file and the template file corresponding with the request. Rather than manual concatenation we use filepath.Join, which has the advantage joining paths using the correct separator for your OS.
Importantly, because the URL path is untrusted user input, we use filepath.Clean to sanitise the URL path before using it.
(Note that even though filepath.Join automatically runs the joined path through filepath.Clean, to help prevent directory traversal attacks you need to manually sanitise any untrusted inputs before joining them.)
We then use the ParseFiles function to bundle the requested template and layout into a template set. Finally, we use the ExecuteTemplate function to render a named template in the set, in our case the layout template.
Restart the application:
$ go run app.go
Listening...
And open localhost:3000/example.html in your browser. If you look at the source you should find the markup from both templates merged together. You might also notice that the Content-Type and Content-Length headers have automatically been set for us.
Lastly, let's make the code a bit more robust. We should:
Send a 404 response if the requested template doesn't exist.
Send a 404 response if the requested template path is a directory.
Send a 500 response if the template.ParseFiles or template.ExecuteTemplate functions throw an error, and log the detailed error message.
File: app.go
package main
import (
"html/template"
"log"
"net/http"
"os"
"path/filepath"
)
func main() {
fs := http.FileServer(http.Dir("static"))
http.Handle("/static/", http.StripPrefix("/static/", fs))
http.HandleFunc("/", serveTemplate)
log.Println("Listening...")
http.ListenAndServe(":3000", nil)
}
func serveTemplate(w http.ResponseWriter, r *http.Request) {
lp := filepath.Join("templates", "layout.html")
fp := filepath.Join("templates", filepath.Clean(r.URL.Path))
// Return a 404 if the template doesn't exist
info, err := os.Stat(fp)
if err != nil {
if os.IsNotExist(err) {
http.NotFound(w, r)
return
}
}
// Return a 404 if the request is for a directory
if info.IsDir() {
http.NotFound(w, r)
return
}
tmpl, err := template.ParseFiles(lp, fp)
if err != nil {
// Log the detailed error
log.Println(err.Error())
// Return a generic "Internal Server Error" message
http.Error(w, http.StatusText(500), 500)
return
}
if err := tmpl.ExecuteTemplate(w, "layout", nil); err != nil {
log.Println(err.Error())
http.Error(w, http.StatusText(500), 500)
}
}
If you found this post useful, you might like to subscribe to my RSS feed.
One of the first things I missed when learning Go was being able to route HTTP requests to handlers based on the pattern of a URL path, like you can with web frameworks like Sinatra and Django.
Although Go's ServeMux does a great job at routing incoming requests, it only works for fixed URL paths. To support pretty URLs with variable parameters we either need to roll a custom router (or HTTP request multiplexer in Go terminology), or look to a third-party package.
In this post we'll compare and contrast three popular packages for the job: Pat, Routes and Gorilla Mux. If you're already familiar with them, you might want to skip to the benchmarks and summary.
Pat
Pat by Blake Mizerany is the simplest and lightest of the three packages. It supports basic pattern matching on request paths, matching on request method (GET, POST etc), and the capture of named parameters.
The syntax for defining URL patterns should feel familiar if you're from the Ruby world – named parameters start with a colon, with the remainder of the path matched literally. For example, the pattern /user/:name/profile would match a request to /user/oomi/profile, with the name oomi captured as a parameter.
It's worth pointing out that behind the scenes Pat uses a custom algorithm for pattern matching, rather than a regular expression based approach like the other two packages. In theory this means it should be more a little more optimised for the task at hand.
Let's take a look at a sample application using Pat:
$ mkdir pat-example && cd pat-example
$ touch app.go
$ go get github.com/bmizerany/pat
In the main function we start by creating a new HTTP request multiplexer (or mux for short) with Pat. Then we add a rule to the mux so that all GET requests which match the specified pattern are routed to the profile function.
Next we use the Handle function to register our custom mux as the handler for all incoming requests in Go's DefaultServeMux.
Because we're only using a single handler in this code, an alternative approach would be to skip registering with the DefaultServeMux, and pass our custom mux directly to ListenAndServe as the handler instead.
When a request gets matched, Pat adds any named parameters to the URL RawQuery. In the profile function we then access these in the same way as a normal query string value.
Pat also provides a couple of other nice touches, including redirecting paths with trailing slashes. Here's the full documentation.
Routes
Routes by Drone provides a similar interface to Pat, with the additional benefit that patterns can be more tightly controlled with optional Regular Expressions. For example, the two patterns below are both valid, with the second one matching if the name parameter contains lowercase letters only:
/user/:name/profile
/user/:name([a-z]+)/profile
Routes also provides a few other nice features, including:
Built-in routing for a static files.
A before filter, so specific code can be run before each request is handled.
Helpers for returning JSON and XML responses.
Basic usage of Routes is almost identical to Pat:
$ mkdir routes-example && cd routes-example
$ touch app.go
$ go get github.com/drone/routes
Gorilla Mux is the most full-featured of the three packages. It supports:
Pattern matching on request paths, with optional regular expressions.
Matching on URL host and scheme, request method, header and query values.
Matching based on custom functions.
Use of sub-routers for easy nested routing.
Additionally the matchers can be chained together, giving a lot of potential for granular routing rules if you need them.
The pattern syntax that Gorilla uses is slightly different to the other packages, with named parameters surrounded by curly braces. For example: /user/{name}/profile and /user/{name:[a-z]+}/profile.
Let's take a look at an example:
$ mkdir gorilla-example && cd gorilla-example
$ touch app.go
$ go get github.com/gorilla/mux
Fundamentally there's the same thing going on here as in the previous two examples. So although the syntax looks a bit different I won't dwell on it – the Gorilla documentation does a fine job of explaining it if it's not immediately clear.
Relative Performance
I ran two different sets of benchmarks on the packages. The first was a stripped-down benchmark to look at their performance in isolation, and the second was an attempt at profiling a more real-world use case.
In both tests I measured the number of successful requests across a ten second period, and took the average over 50 iterations, all running on my local machine.
For the 'stripped-down' benchmark, requests were simply routed to a handler that returned a 200 status code and message. Here are the code samples and results:
In this test the best performing package appeared to be Pat by a large margin. It handled around 30% more requests than Routes and Gorilla Mux, which were very evenly matched.
In the second benchmark requests were routed to a handler which accessed a named parameter from the URL, and then merged it with a HTML template read from disk. Here are the code samples and results:
In this benchmark the performance difference between the three packages was negligible.
Although it's always dangerous to draw conclusions from just one set of tests, it does point toward the overall performance impact of a router being much smaller for higher-latency applications, such as those with a lot of file system or database access in the handlers.
Summary
Pat would appear to be a good choice for scenarios where performance is important, you have a low-latency application, and only require simple pattern-based routing.
If you're likely to be validating a lot of parameter input with regular expressions in your application, then it probably makes sense to skip Pat and use Routes or Gorilla Mux instead, with the expressions built into your routing patterns.
For higher-latency applications, where there appears to be less of an overall impact due to router performance, Gorilla Mux may be a wise choice because of the sheer number of options and the flexibility it provides. Although I haven't looked at it in detail, larger applications with a lot of URL endpoints may also get a performance benefit from using Gorilla's nested routing too.
If you found this post useful, you might like to subscribe to my RSS feed.
Processing HTTP requests with Go is primarily about two things: ServeMuxes and Handlers.
A ServeMux is essentially a HTTP request router (or multiplexor). It compares incoming requests against a list of predefined URL paths, and calls the associated handler for the path whenever a match is found.
Handlers are responsible for writing response headers and bodies. Almost any object can be a handler, so long as it satisfies the Handler interface. In lay terms, that simply means it must have a ServeHTTP method with the following signature:
ServeHTTP(http.ResponseWriter, *http.Request)
Go's HTTP package ships with a few functions to generate common handlers, such as FileServer, NotFoundHandler and RedirectHandler. Let's begin with a simple but contrived example:
$ mkdir handler-example
$ cd handler-example
$ touch main.go
In the main function we use the http.NewServeMux function to create an empty ServeMux.
We then use the http.RedirectHandler function to create a new handler. This handler 307 redirects all requests it receives to http://example.org.
Next we use the ServeMux.Handle function to register this with our new ServeMux, so it acts as the handler for all incoming requests with the URL path /foo.
Finally we create a new server and start listening for incoming requests with the http.ListenAndServe function, passing in our ServeMux for it to match requests against.
Go ahead and run the application:
$ go run main.go
Listening...
And visit http://localhost:3000/foo in your browser. You should find that your request gets successfully redirected.
The eagle-eyed of you might have noticed something interesting: The signature for the ListenAndServe function is ListenAndServe(addr string, handler Handler), but we passed a ServeMux as the second parameter.
We were able to do this because ServeMux also has a ServeHTTP method, meaning that it too satisfies the Handler interface.
For me it simplifies things to think of a ServeMux as just being a special kind of handler, which instead of providing a response itself passes the request on to a second handler. This isn't as much of a leap as it first sounds – chaining handlers together is fairly commonplace in Go.
Custom Handlers
Let's create a custom handler which responds with the current local time in a given format:
type timeHandler struct {
format string
}
func (th *timeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
tm := time.Now().Format(th.format)
w.Write([]byte("The time is: " + tm))
}
The exact code here isn't too important.
All that really matters is that we have an object (in this case it's a timeHandler struct, but it could equally be a string or function or anything else), and we've implemented a method with the signature ServeHTTP(http.ResponseWriter, *http.Request) on it. That's all we need to make a handler.
Let's embed this in a concrete example:
File: main.go
package main
import (
"log"
"net/http"
"time"
)
type timeHandler struct {
format string
}
func (th *timeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
tm := time.Now().Format(th.format)
w.Write([]byte("The time is: " + tm))
}
func main() {
mux := http.NewServeMux()
th := &timeHandler{format: time.RFC1123}
mux.Handle("/time", th)
log.Println("Listening...")
http.ListenAndServe(":3000", mux)
}
In the main function we initialised the timeHandler in exactly the same way we would any normal struct, using the & symbol to yield a pointer. And then, like the previous example, we use the mux.Handle function to register this with our ServeMux.
Now when we run the application, the ServeMux will pass any request for /time straight on to our timeHandler.ServeHTTP method.
For simple cases (like the example above) defining new custom types and ServeHTTP methods feels a bit verbose. Let's look at an alternative approach, where we leverage Go's http.HandlerFunc type to coerce a normal function into satisfying the Handler interface.
Any function which has the signature func(http.ResponseWriter, *http.Request) can be converted into a HandlerFunc type. This is useful because HandleFunc objects come with an inbuilt ServeHTTP method which – rather cleverly and conveniently – executes the content of the original function.
If that sounds confusing, try taking a look at the relevant source code. You'll see that it's a very succinct way of making a function satisfy the Handler interface.
Let's reproduce the timeHandler application using this technique:
File: main.go
package main
import (
"log"
"net/http"
"time"
)
func timeHandler(w http.ResponseWriter, r *http.Request) {
tm := time.Now().Format(time.RFC1123)
w.Write([]byte("The time is: " + tm))
}
func main() {
mux := http.NewServeMux()
// Convert the timeHandler function to a HandleFunc type
th := http.HandlerFunc(timeHandler)
// And add it to the ServeMux
mux.Handle("/time", th)
log.Println("Listening...")
http.ListenAndServe(":3000", mux)
}
In fact, converting a function to a HandlerFunc type and then adding it to a ServeMux like this is so common that Go provides a shortcut: the ServeMux.HandleFunc method.
This is what the main() function would have looked like if we'd used this shortcut instead:
Most of the time using a function as a handler like this works well. But there is a bit of a limitation when things start getting more complex.
You've probably noticed that, unlike the method before, we've had to hardcode the time format in the timeHandler function. What happens when we want to pass information or variables from main() to a handler?
A neat approach is to put our handler logic into a closure, and close over the variables we want to use:
The timeHandler function now has a subtly different role. Instead of coercing the function into a handler (like we did previously), we are now using it to return a handler. There's two key elements to making this work.
First it creates fn, an anonymous function which accesses ‐ or closes over – the format variable forming a closure. Regardless of what we do with the closure it will always be able to access the variables that are local to the scope it was created in – which in this case means it'll always have access to the format variable.
Secondly our closure has the signature func(http.ResponseWriter, *http.Request). As you may remember from earlier, this means that we can convert it into a HandlerFunc type (so that it satisfies the Handler interface). Our timeHandler function then returns this converted closure.
In this example we've just been passing a simple string to a handler. But in a real-world application you could use this method to pass database connection, template map, or any other application-level context. It's a good alternative to using global variables, and has the added benefit of making neat self-contained handlers for testing.
You might also see this same pattern written as:
func timeHandler(format string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
tm := time.Now().Format(format)
w.Write([]byte("The time is: " + tm))
})
}
Or using an implicit conversion to the HandlerFunc type on return:
func timeHandler(format string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
tm := time.Now().Format(format)
w.Write([]byte("The time is: " + tm))
}
}
The DefaultServeMux
You've probably seen DefaultServeMux mentioned in lots of places, from the simplest Hello World examples to the Go source code.
It took me a long time to realise it isn't anything special. The DefaultServeMux is just a plain ol' ServeMux like we've already been using, which gets instantiated by default when the HTTP package is used. Here's the relevant line from the Go source:
var DefaultServeMux = NewServeMux()
The HTTP package provides a couple of shortcuts for working with the DefaultServeMux: http.Handle and http.HandleFunc. These do exactly the same as their namesake functions we've already looked at, with the difference that they add handlers to the DefaultServeMux instead of one that you've created.
Additionally, ListenAndServe will fall back to using the DefaultServeMux if no other handler is provided (that is, the second parameter is set to nil).
So as a final step, let's update our timeHandler application to use the DefaultServeMux instead:
File: main.go
package main
import (
"log"
"net/http"
"time"
)
func timeHandler(format string) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
tm := time.Now().Format(format)
w.Write([]byte("The time is: " + tm))
}
return http.HandlerFunc(fn)
}
func main() {
// Note that we skip creating the ServeMux...
var format string = time.RFC1123
th := timeHandler(format)
// We use http.Handle instead of mux.Handle...
http.Handle("/time", th)
log.Println("Listening...")
// And pass nil as the handler to ListenAndServe.
http.ListenAndServe(":3000", nil)
}
If you found this post useful, you might like to subscribe to my RSS feed.
I wrote a short Bash script to automatically reload Go programs.
The script acts as a light wrapper around go run, stopping and restarting it whenever a .go file in your current directory or $GOPATH/src folder is saved. I've been using it mainly when developing web applications, in the same way that I use Shotgun or Guard when working with Ruby.
#!/bin/bash
# Watch all *.go files in the specified directory
# Call the restart function when they are saved
function monitor() {
inotifywait -q -m -r -e close_write --exclude '[^g][^o]$' $1 |
while read line; do
restart
done
}
# Terminate and rerun the main Go program
function restart {
if [ "$(pidof $PROCESS_NAME)" ]; then
killall -q -w -9 $PROCESS_NAME
fi
echo ">> Reloading..."
go run $FILE_PATH $ARGS &
}
# Make sure all background processes get terminated
function close {
killall -q -w -9 inotifywait
exit 0
}
trap close INT
echo "== Go-reload"
echo ">> Watching directories, CTRL+C to stop"
FILE_PATH=$1
FILE_NAME=$(basename $FILE_PATH)
PROCESS_NAME=${FILE_NAME%%.*}
shift
ARGS=$@
# Start the main Go program
go run $FILE_PATH $ARGS &
# Monitor the /src directories in all directories on the GOPATH
OIFS="$IFS"
IFS=':'
for path in $GOPATH
do
monitor $path/src &
done
IFS="$OIFS"
# Monitor the current directory
monitor .
Usage
The only dependency for this script is inotify-tools, which is used to monitor the filesystem for changes.
$ sudo apt-get install inotify-tools
Once you've downloaded (or copy-pasted) the script, you'll need to make it executable and move it to /usr/local/bin or another directory on your system path:
For anyone new to building web applications with Go, it's important to realise that all incoming HTTP requests are served in their own Goroutine. This means that any code in or called by your application handlers will be running concurrently, and there is a risk of race conditions occurring.
In case you're new to concurrent programming, I'll quickly explain the problem.
Race conditions occur when two or more Goroutines try to use a piece of shared data at the same time, but the result of their operations is dependent on the exact order that the scheduler executes their instructions.
As an illustration, here's an example where two Goroutines try to add money to a shared bank balance at the same time:
Instruction
Goroutine 1
Goroutine 2
Bank Balance
1
Read balance ⇐ £50
£50
2
Read balance ⇐ £50
£50
3
Add £100 to balance
£50
4
Add £50 to balance
£50
5
Write balance ⇒ £150
£150
6
Write balance ⇒ £100
£100
Despite making two separate deposits, only the second one is reflected in the final balance because the two Goroutines were racing each other to make the change.
The Go blog describes the downsides:
Race conditions are among the most insidious and elusive programming errors. They typically cause erratic and mysterious failures, often long after the code has been deployed to production. While Go's concurrency mechanisms make it easy to write clean concurrent code, they don't prevent race conditions. Care, diligence, and testing are required.
Go provides a number of tools to help us avoid data races. These include Channels for communicating data between Goroutines, a Race Detector for monitoring unsynchronized access to memory at runtime, and a variety of 'locking' features in the Atomic and Sync packages. One of these features are Mutual Exclusion locks, or mutexes, which we'll be looking at in the rest of this post.
Creating a Basic Mutex
Let's create some toy code to mimic the bank balance example:
import "strconv"
var Balance = ¤cy{50.00, "GBP"}
type currency struct {
amount float64
code string
}
func (c *currency) Add(i float64) {
// This is racy
c.amount += i
}
func (c *currency) Display() string {
// This is racy
return strconv.FormatFloat(c.amount, 'f', 2, 64) + " " + c.code
}
We know that if there are multiple Goroutines using this code and calling Balance.Add() and Balance.Display(), then at some point a race condition is likely to occur.
One way we could prevent a data race is to ensure that if one Goroutine is using the Balance variable, then all other Goroutines are prevented (or mutually excluded) from using it at the same time.
We can do this by creating a Mutex and setting a lock around particular lines of code with it. While one Goroutine holds the lock, all other Goroutines are prevented from executing any lines of code protected by the same mutex, and are forced to wait until the lock is yielded before they can proceed.
In practice, it's more simple than it sounds:
import (
"strconv"
"sync"
)
var mu = &sync.Mutex{}
var Balance = ¤cy{50.00, "GBP"}
type currency struct {
amount float64
code string
}
func (c *currency) Add(i float64) {
mu.Lock()
c.amount += i
mu.Unlock()
}
func (c *currency) Display() string {
mu.Lock()
amt := c.amount
mu.Unlock()
return strconv.FormatFloat(amt, 'f', 2, 64) + " " + c.code
}
Here we've created a new mutex and assigned it to mu. We then use mu.Lock() to create a lock immediately before both racy parts of the code, and mu.Unlock() to yield the lock immediately after.
There's a couple of things to note:
The same mutex variable can be used in multiple places throughout your code. So long as it's the same mutex (in our case mu) then none of the chunks of code protected by it can be executed at the same time.
Holding a mutex lock doesn't 'protect' a memory location from being read or updated. A non-mutex-locked line of code could still access it at any time and create a race condition. Therefore you need to be careful to make sure all points in your code which are potentially racy are protected.
Because our mutex is only being used in the context of a currency object, it makes sense to anonymously embed it in the currency struct (an idea borrowed from Andrew Gerrard's excellent
10 things you (probably) don't know about Go slideshow). If you look at a larger codebase with lots of mutexes, like Go's HTTP Server, you can see how this approach helps to keep locking rules nice and clear.
We've also made use of the defer statement, which ensures that the mutex gets unlocked immediately before a function returns. This is common practice for functions that contain multiple return statements, or where the return statement itself is racy.
Read Write Mutexes
In our bank balance example, having a full mutex lock on the Display() function isn't strictly necessary. It would be OK for us to have multiple reads of Balance happening at the same time, so long as nothing is being written.
We can achieve this using RWMutex, a reader/writer mutual exclusion lock which allows any number of readers to hold the lock or one writer. Depending on the nature of your application and ratio of reads to writes, this may be more efficient than using a full mutex.
Reader locks can be opened and closed with RLock() and RUnlock() like so:
Taking inspiration from the Rails layouts and rendering guide, I thought it'd be a nice idea to build a snippet collection illustrating some common HTTP responses for Go web applications.
In this post I'm going to run through a start-to-finish tutorial for building a online contact form in Go. I'll be trying to explain the different steps in detail, as well as outlining a sensible pattern for processing that can be extended to other forms.
Let's begin the example by creating a new directory for the application, along with a main.go file for our code and a couple of vanilla HTML templates:
<h1>Confirmation</h1>
<p>Your message has been sent!</p>
Our contact form will issue a POST request to /, which will be the same URL path that we use for presenting the form. This means that we'll need to route requests for the same URL to different handlers based on the HTTP method.
There are a few ways of achieving this, but we'll use Pat – a third-party routing library which I've talked about before. You'll need to install it if you're following along:
$ go get github.com/bmizerany/pat
Go ahead and create a skeleton for the application:
File: main.go
package main
import (
"github.com/bmizerany/pat"
"html/template"
"log"
"net/http"
)
func main() {
mux := pat.New()
mux.Get("/", http.HandlerFunc(index))
mux.Post("/", http.HandlerFunc(send))
mux.Get("/confirmation", http.HandlerFunc(confirmation))
log.Println("Listening...")
http.ListenAndServe(":3000", mux)
}
func index(w http.ResponseWriter, r *http.Request) {
render(w, "templates/index.html", nil)
}
func send(w http.ResponseWriter, r *http.Request) {
// Validate form
// Send message in an email
// Redirect to confirmation page
}
func confirmation(w http.ResponseWriter, r *http.Request) {
render(w, "templates/confirmation.html", nil)
}
func render(w http.ResponseWriter, filename string, data interface{}) {
tmpl, err := template.ParseFiles(filename)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
if err := tmpl.Execute(w, data); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
This is fairly straightforward stuff so far. The only real point of note is that we've put the template handling into a render function to cut down on boilerplate code.
If you run the application:
$ go run main.go
Listening...
And visit localhost:3000 in your browser you should see the contact form, although it doesn't do anything yet.
Now for the interesting part. Let's add a couple of validation rules to our contact form, display the errors if there are any, and make sure that the form values get presented back if there's an error so the user doesn't need to retype them.
One approach to setting this up is to add the code inline in our send handler, but personally I find it cleaner and neater to break out the logic into a separate message.go file:
$ touch message.go
File: message.go
package main
import (
"regexp"
"strings"
)
type Message struct {
Email string
Content string
Errors map[string]string
}
func (msg *Message) Validate() bool {
msg.Errors = make(map[string]string)
re := regexp.MustCompile(".+@.+\\..+")
matched := re.Match([]byte(msg.Email))
if matched == false {
msg.Errors["Email"] = "Please enter a valid email address"
}
if strings.TrimSpace(msg.Content) == "" {
msg.Errors["Content"] = "Please write a message"
}
return len(msg.Errors) == 0
}
So what's going on here?
We've started by defining a new Message type, consisting of the Email and Content values (which will hold the data from the submitted form), along with an Errors map to hold any validation error messages.
We then created a Validate method that acts on a given Message, which summarily checks the format of the email address and makes sure that the content isn't blank. In the event of any errors we add them to the Errors map, and finally return a true or false value to indicate whether validation passed successful or not.
This approach means that we can keep the code in our send handler fantastically light. All we need it to do is retrieve the form values from the POST request, create a new Message object with them, and call Validate(). If the validation fails we'll then want to reshow the contact form, passing back the relevant Message object.
File: main.go
...
func send(w http.ResponseWriter, r *http.Request) {
msg := &Message{
Email: r.FormValue("email"),
Content: r.FormValue("content"),
}
if msg.Validate() == false {
render(w, "templates/index.html", msg)
return
}
// Send message in an email
// Redirect to confirmation page
}
...
As a side note, in this example above we're using the FormValue method on the request to access the POST data. We could also access the data directly via r.Form, but there is a gotcha to point out – by default r.Form will be empty until it is filled by calling ParseForm on the request. Once that's done, we can access it in the same way as any url.Values type. For example:
Anyway, let's update our template so it shows the validation errors (if they exist) above the relevant fields, and repopulate the form inputs with any information that the user previously typed in:
Still, our contact form is pretty useless unless we actually do something with it. Let's add a Deliver method which sends the contact form message to a particular email address. In the code below I'm using Gmail, but the same thing should work with any other SMTP server.
File: message.go
package main
import (
"fmt"
"net/smtp"
"regexp"
"strings"
)
...
func (msg *Message) Deliver() error {
to := []string{"someone@example.com"}
body := fmt.Sprintf("Reply-To: %v\r\nSubject: New Message\r\n%v", msg.Email, msg.Content)
username := "you@gmail.com"
password := "..."
auth := smtp.PlainAuth("", username, password, "smtp.gmail.com")
return smtp.SendMail("smtp.gmail.com:587", auth, msg.Email, to, []byte(body))
}
The final step is to head back to our main.go file, add some code to call Deliver(), and issue a 303 redirect to the confirmation page that we made earlier:
If mapping form data to objects is something that you're doing a lot of, you may find Gorilla Schema's automatic decoder useful. If we were using it for our contact form example, the code would look a bit like this:
Additionally, Goforms appears to be a promising idea, with a fairly slick Django-like approach to dealing with forms. However, the existing validation options are fairly limited and the library doesn't seem to be under active development at the moment. It's still worth a look though, especially if you're thinking of rolling something a bit more generic for your form handling.
If you found this post useful, you might like to subscribe to my RSS feed.
Often in web applications you need to temporarily store data in-between requests, such as an error or success message during the Post-Redirect-Get process for a form submission. Frameworks such as Rails and Django have the concept of transient single-use flash messages to help with this.
In this post I'm going to look at a way to create your own cookie-based flash messages in Go.
We'll start by creating a directory for the project, along with a flash.go file for our code and a main.go file for an example application.
$ mkdir flash-example
$ cd flash-example
$ touch flash.go main.go
In order to keep our request handlers nice and clean, we'll create our primary SetFlash() and GetFlash() helper functions in the flash.go file.
File: flash.go
package main
import (
"encoding/base64"
"net/http"
"time"
)
func SetFlash(w http.ResponseWriter, name string, value []byte) {
c := &http.Cookie{Name: name, Value: encode(value)}
http.SetCookie(w, c)
}
func GetFlash(w http.ResponseWriter, r *http.Request, name string) ([]byte, error) {
c, err := r.Cookie(name)
if err != nil {
switch err {
case http.ErrNoCookie:
return nil, nil
default:
return nil, err
}
}
value, err := decode(c.Value)
if err != nil {
return nil, err
}
dc := &http.Cookie{Name: name, MaxAge: -1, Expires: time.Unix(1, 0)}
http.SetCookie(w, dc)
return value, nil
}
// -------------------------
func encode(src []byte) string {
return base64.URLEncoding.EncodeToString(src)
}
func decode(src string) ([]byte, error) {
return base64.URLEncoding.DecodeString(src)
}
Our SetFlash() function is pretty succinct.
It creates a new Cookie, containing the name of the flash message and the content. You'll notice that we're encoding the content – this is because RFC 6265 is quite strict about the characters cookie values can contain, and encoding to base64 ensures our value satisfies the permitted character set. We then use the SetCookie function to write the cookie to the response.
In the GetFlash() helper we use the request.Cookie method to load up the cookie containing the flash message – returning nil if it doesn't exist – and then decode the value from base64 back into a byte array.
Because we want a flash message to only be available once, we need to instruct clients to not resend the cookie with future requests. We can do this by setting a new cookie with exactly the same name, with MaxAge set to a negative number and Expiry set to a historical time (to cater for old versions of IE). You should note that Go will only set an expiry time on a cookie if it is after the Unix epoch, so we've set ours for 1 second after that.
Let's use these helper functions in a short example:
File: main.go
package main
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/set", set)
http.HandleFunc("/get", get)
fmt.Println("Listening...")
http.ListenAndServe(":3000", nil)
}
func set(w http.ResponseWriter, r *http.Request) {
fm := []byte("This is a flashed message!")
SetFlash(w, "message", fm)
}
func get(w http.ResponseWriter, r *http.Request) {
fm, err := GetFlash(w, r, "message")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if fm == nil {
fmt.Fprint(w, "No flash messages")
return
}
fmt.Fprintf(w, "%s", fm)
}
$ curl -i --cookie-jar cj localhost:3000/set
HTTP/1.1 200 OK
Set-Cookie: message=VGhpcyBpcyBhIGZsYXNoZWQgbWVzc2FnZSE=
Content-Type: text/plain; charset=utf-8
Content-Length: 0
$ curl -i --cookie-jar cj --cookie cj localhost:3000/get
HTTP/1.1 200 OK
Set-Cookie: message=; Expires=Thu, 01 Jan 1970 00:00:01 UTC; Max-Age=0
Content-Type: text/plain; charset=utf-8
Content-Length: 26
This is a flashed message!
$ curl -i --cookie-jar cj --cookie cj localhost:3000/get
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Content-Length: 17
No flash messages
You can see our flash message being set, retrieved, and then not passed with subsequent requests as expected.
Additional Tools
If you don't want to roll your own helpers for flash messages, or need them to be 'signed' to prevent tampering, then the Gorilla Sessions package is a good option. Here's the previous example implemented with Gorilla instead:
package main
import (
"fmt"
"github.com/gorilla/sessions"
"net/http"
)
func main() {
http.HandleFunc("/set", set)
http.HandleFunc("/get", get)
fmt.Println("Listening...")
http.ListenAndServe(":3000", nil)
}
var store = sessions.NewCookieStore([]byte("a-secret-string"))
func set(w http.ResponseWriter, r *http.Request) {
session, err := store.Get(r, "flash-session")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
session.AddFlash("This is a flashed message!", "message")
session.Save(r, w)
}
func get(w http.ResponseWriter, r *http.Request) {
session, err := store.Get(r, "flash-session")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
fm := session.Flashes("message")
if fm == nil {
fmt.Fprint(w, "No flash messages")
return
}
session.Save(r, w)
fmt.Fprintf(w, "%v", fm[0])
}
If you found this post useful, you might like to subscribe to my RSS feed.
When you're building a web application there's probably some shared functionality that you want to run for many (or even all) HTTP requests. You might want to log every request, gzip every response, or check a cache before doing some heavy processing.
One way of organising this shared functionality is to set it up as middleware – self-contained code which independently acts on a request before or after your normal application handlers. In Go a common place to use middleware is between a ServeMux and your application handlers, so that the flow of control for a HTTP request looks like:
In this post I'm going to explain how to make custom middleware that works in this pattern, as well as running through some concrete examples of using third-party middleware packages.
The Basic Principles
Making and using middleware in Go is fundamentally simple. We want to:
Implement our middleware so that it satisfies the http.Handler interface.
Build up a chain of handlers containing both our middleware handler and our normal application handler, which we can register with a http.ServeMux.
I'll explain how.
Hopefully you're already familiar with the following method for constructing a handler (if not, it's probably best to read this primer before continuing).
In this snippet we're placing our handler logic (a simple w.Write) in an anonymous function and closing-over the message variable to form a closure. We're then converting this closure to a handler by using the http.HandlerFunc adapter and returning it.
We can use this same approach to create a chain of handlers. Instead of passing a string into the closure (like above) we could pass the next handler in the chain as a variable, and then transfer control to this next handler by calling it's ServeHTTP() method.
This gives us a complete pattern for constructing middleware:
You'll notice that this middleware function has a func(http.Handler) http.Handler signature. It accepts a handler as a parameter and returns a handler. This is useful for two reasons:
Because it returns a handler we can register the middleware function directly with the standard ServeMux provided by the net/http package.
We can create an arbitrarily long handler chain by nesting middleware functions inside each other. For example:
Run this application and make a request to http://localhost:3000. You should get log output similar to this:
$ go run main.go
2014/10/13 20:27:36 Executing middlewareOne
2014/10/13 20:27:36 Executing middlewareTwo
2014/10/13 20:27:36 Executing finalHandler
2014/10/13 20:27:36 Executing middlewareTwo again
2014/10/13 20:27:36 Executing middlewareOne again
It's clear to see how control is being passed through the handler chain in the order we nested them, and then back up again in the reverse direction.
We can stop control propagating through the chain at any point by issuing a return from a middleware handler.
In the example above I've included a conditional return in the middlewareTwo function. Try it by visiting http://localhost:3000/foo and checking the log again – you'll see that this time the request gets no further than middlewareTwo before passing back up the chain.
Understood. How About a Proper Example?
OK, let's say that we're building a service which processes requests containing a XML body. We want to create some middleware which a) checks for the existence of a request body, and b) sniffs the body to make sure it is XML. If either of those checks fail, we want our middleware to write an error message and to stop the request from reaching our application handlers.
File: main.go
package main
import (
"bytes"
"net/http"
)
func enforceXMLHandler(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Check for a request body
if r.ContentLength == 0 {
http.Error(w, http.StatusText(400), 400)
return
}
// Check its MIME type
buf := new(bytes.Buffer)
buf.ReadFrom(r.Body)
if http.DetectContentType(buf.Bytes()) != "text/xml; charset=utf-8" {
http.Error(w, http.StatusText(415), 415)
return
}
next.ServeHTTP(w, r)
})
}
func main() {
finalHandler := http.HandlerFunc(final)
http.Handle("/", enforceXMLHandler(finalHandler))
http.ListenAndServe(":3000", nil)
}
func final(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("OK"))
}
This looks good. Let's test it by creating a simple XML file:
$ cat > books.xml
<?xml version="1.0"?>
<books>
<book>
<author>H. G. Wells</author>
<title>The Time Machine</title>
<price>8.50</price>
</book>
</books>
And making some requests using cURL:
$ curl -i localhost:3000
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Content-Length: 12
Bad Request
$ curl -i -d "This is not XML" localhost:3000
HTTP/1.1 415 Unsupported Media Type
Content-Type: text/plain; charset=utf-8
Content-Length: 23
Unsupported Media Type
$ curl -i -d @books.xml localhost:3000
HTTP/1.1 200 OK
Date: Fri, 17 Oct 2014 13:42:10 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8
OK
Using Third-Party Middleware
Rather than rolling your own middleware all the time you might want to use a third-party package. We're going to look at a couple here: goji/httpauth and Gorilla's LoggingHandler.
The goji/httpauth package provides HTTP Basic Authentication functionality. It has a SimpleBasicAuth helper which returns a function with the signature func(http.Handler) http.Handler. This means we can use it in exactly the same way as our custom-built middleware.
Gorilla's LoggingHandler – which records Apache-style logs – is a bit different.
It uses the signature func(out io.Writer, h http.Handler) http.Handler, so it takes not only the next handler but also the io.Writer that the log will be written to.
Here's a simple example in which we write logs to a server.log file:
In a trivial case like this our code is fairly clear. But what happens if we want to use LoggingHandler as part of a larger middleware chain? We could easily end up with a declaration looking something like this:
We can clean this up by creating a constructor function (let's call it myLoggingHandler) with the signature func(http.Handler) http.Handler. This will allow us to nest it more neatly with other middleware:
If you're interested, here's a gist of the three middleware handlers from this post combined in one example.
As a side note: notice that the Gorilla LoggingHandler is recording the response status (200) and response length (2) in the logs. This is interesting. How did the upstream logging middleware get to know about the response body written by our application handler?
It does this by defining it's own responseLogger type which wraps http.ResponseWriter, and creating custom responseLogger.Write() and responseLogger.WriteHeader() methods. These methods not only write the response but also store the size and status for later examination. Gorilla's LoggingHandler passes responseLogger onto the next handler in the chain, instead of the normal http.ResponseWriter.
(I'll probably write a proper tutorial about this sometime, in which case I'll add a link here!)
Additional Tools
Alice by Justinas Stankevičius is a clever and very lightweight package which provides some syntactic sugar for chaining middleware handlers. At it's most basic Alice lets you rewrite this:
In my eyes at least, that code is slightly clearer to understand at a glance. However, the real benefit of Alice is that it lets you to specify a handler chain once and reuse it for multiple routes. Like so:
I've written a package for chaining context-aware handlers in Go, called Stack. It was heavily inspired by Alice.
What do you mean by 'context-aware'?
If you're using a middleware pattern to process HTTP requests in Go, you may want to share some data or context between middleware handlers and your application handlers. For example you might want to:
Use some middleware to create a CRSF token, and later render the token to a template in your application handler. Or perhaps...
Authenticate a user in one middleware handler, and then pass the user details to a second middleware handler which checks if the user is authorised to access the resource.
There are a few packages that can help with this. Matt Silverlock has written a good article about some of the different approaches and tools – I won't rehash it here, instead I recommend giving it a read.
Why make another package?
Because none of the existing tools seemed ideal – at least to me. Gorilla Context is simple and very flexible, but relies on a global context map and you remembering to clear the context after each request. (It's still my favourite though). Goji provides request-scoped context, which is good, but it's part of a larger package and ties you into using the Goji router. The same is true of Gocraft/web, which also relies on reflection tricks under the hood that I struggle to wrap my head around.
I realised that the only time you need to worry about context is when you're chaining handlers together. So I looked at my favorite tool for chaining handlers, Alice, and began adapting that to create Stack.
I wanted the package to:
Do a simple job, and then get out of the way.
Provide a request-scoped context map.
Let you create stackable, reusable, handler chains in the Alice style.
Be as type-safe at compile time as it possibly could be.
Be simple to understand and non-magic.
Operate nicely with existing standards. In particular:
The handler chain must satisfy the http.Handler interface, so it can be used with the http.DefaultServeMux.
It should be compatible with the func(http.Handler) http.Handler pattern commonly used by third-party middleware packages.
package main
import (
"fmt"
"github.com/alexedwards/stack"
"github.com/goji/httpauth"
"net/http"
)
func main() {
// Setup goji/httpauth, some third-party middleware
authenticate := stack.Middleware(httpauth.SimpleBasicAuth("user", "pass"))
// Create a handler chain and register it with the DefaultServeMux
http.Handle("/", stack.New(authenticate, tokenMiddleware).Then(tokenHandler))
http.ListenAndServe(":3000", nil)
}
func tokenMiddleware(ctx stack.Context, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Add a value to Context with the key 'token'
ctx["token"] = "c9e452805dee5044ba520198628abcaa"
next.ServeHTTP(w, r)
})
}
func tokenHandler(ctx stack.Context) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Retrieve the token from Context and print it
fmt.Fprintf(w, "Token is: %s", ctx["token"])
})
}
This is the first in a series of tutorials about persisting data in Go web applications.
In this post we'll be looking at SQL databases. I'll explain the basics of the database/sql package, walk through building a working application, and explore a couple of options for cleanly structuring your code.
In this post I'll be using Postgres and the excellent pq driver. But all the code in this tutorial is (nearly) exactly the same for any other driver or database – including MySQL and SQLite. I'll point out the very few Postgres-specific bits as we go.
$ go get github.com/lib/pq
Basic usage
Let's build a simple Bookstore application, which carries out CRUD operations on a books table.
If you'd like to follow along, you'll need to create a new bookstore database and scaffold it with the following:
CREATE TABLE books (
isbn char(14) NOT NULL,
title varchar(255) NOT NULL,
author varchar(255) NOT NULL,
price decimal(5,2) NOT NULL
);
INSERT INTO books (isbn, title, author, price) VALUES
('978-1503261969', 'Emma', 'Jayne Austen', 9.44),
('978-1505255607', 'The Time Machine', 'H. G. Wells', 5.99),
('978-1503379640', 'The Prince', 'Niccolò Machiavelli', 6.99);
ALTER TABLE books ADD PRIMARY KEY (isbn);
Once that's done, head over to your Go workspace and create a new bookstore package directory and a main.go file:
$ cd $GOPATH/src
$ mkdir bookstore && cd bookstore
$ touch main.go
Let's start with some code that executes a SELECT * FROM books query and then prints the results to stdout.
File: main.go
package main
import (
_ "github.com/lib/pq"
"database/sql"
"fmt"
"log"
)
type Book struct {
isbn string
title string
author string
price float32
}
func main() {
db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore")
if err != nil {
log.Fatal(err)
}
rows, err := db.Query("SELECT * FROM books")
if err != nil {
log.Fatal(err)
}
defer rows.Close()
bks := make([]*Book, 0)
for rows.Next() {
bk := new(Book)
err := rows.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price)
if err != nil {
log.Fatal(err)
}
bks = append(bks, bk)
}
if err = rows.Err(); err != nil {
log.Fatal(err)
}
for _, bk := range bks {
fmt.Printf("%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price)
}
}
There's a lot going on here. We'll step through this bit-by-bit.
The first interesting thing is the way that we import the driver. We don't use anything in the pq package directly, which means that the Go compiler will raise an error if we try to import it normally. But we need the pq package's init() function to run so that our driver can register itself with database/sql. We get around this by aliasing the package name to the blank identifier. This means pq.init() still gets executed, but the alias is harmlessly discarded (and our code runs error-free). This approach is standard for most of Go's SQL drivers.
Next we define a Book type – with the struct fields and their types aligning to our books table. For completeness I should point out that we've only been able to use the string and float32 types safely because we set NOT NULL constraints on the columns in our table. If the table contained nullable fields we would need to use the sql.NullString and sql.NullFloat64 types instead – see this Gist for a working example. Generally it's easiest to avoid nullable fields altogether if you can, which is what we've done here.
In the main() function we initialise a new sql.DB object by calling sql.Open(). We pass in the name of our driver (in this case "postgres") and the connection string (you'll need to check your driver documentation for the correct format). It's worth emphasising that the sql.DB object it returns is not a database connection – it's an abstraction representing a pool of underlying connections. You can change the maximum number of open and idle connections in the pool with the db.SetMaxOpenConns() and db.SetMaxIdleConns() methods respectively. A final thing to note is that sql.DB is safe for concurrent access, which is very convenient if you're using it in a web application (like we will shortly).
From there we follow a standard pattern that you'll see often:
We fetch a resultset from the books table using the DB.Query() method and assign it to a rows variable. Then we defer rows.Close() to ensure the resultset is properly closed before the parent function returns. Closing a resultset properly is really important. As long as a resultset is open it will keep the underlying database connection open – which in turn means the connection is not available to the pool. So if something goes wrong and the resultset isn't closed it can
rapidly lead to all the connections in your pool being used up. Another gotcha (which caught me out when I first began) is that the defer statement should come after you check for an error from DB.Query. Otherwise, if DB.Query() returns an error, you'll get a panic trying to close a nil resultset.
We then use rows.Next() to iterate through the rows in the resultset. This preps the first (and then each subsequent) row to be acted on by the rows.Scan() method. Note that if iteration over all of the rows completes then the resultset automatically closes itself and frees-up the connection.
We use the rows.Scan() method to copy the values from each field in the row to a new Book object that we created. We then check for any errors that occurred during Scan, and add the new Book to the bks slice we created earlier.
When our rows.Next() loop has finished we call rows.Err(). This returns any error that was encountered during the interation. It's important to call this – don't just assume that we completed a successful iteration over the whole resultset.
If our bks slice has been filled successfully, we loop through it and print the information about each book to stdout.
If you run the code you should get the following output:
$ go run main.go
978-1503261969, Emma, Jayne Austen, £9.44
978-1505255607, The Time Machine, H. G. Wells, £5.99
978-1503379640, The Prince, Niccolò Machiavelli, £6.99
Using in a web application
Let's start to morph our code into a RESTful-ish web application with 3 routes:
GET /books – List all books in the store
GET /books/show – Show a specific book by its ISBN
POST /books/create – Add a new book to the store
We've just written all the core logic we need for the GET /books route. Let's adapt it into a booksIndex() HTTP handler for our web application.
File: main.go
package main
import (
_ "github.com/lib/pq"
"database/sql"
"fmt"
"log"
"net/http"
)
type Book struct {
isbn string
title string
author string
price float32
}
var db *sql.DB
func init() {
var err error
db, err = sql.Open("postgres", "postgres://user:pass@localhost/bookstore")
if err != nil {
log.Fatal(err)
}
if err = db.Ping(); err != nil {
log.Fatal(err)
}
}
func main() {
http.HandleFunc("/books", booksIndex)
http.ListenAndServe(":3000", nil)
}
func booksIndex(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
http.Error(w, http.StatusText(405), 405)
return
}
rows, err := db.Query("SELECT * FROM books")
if err != nil {
http.Error(w, err.Error(), 500)
return
}
defer rows.Close()
bks := make([]*Book, 0)
for rows.Next() {
bk := new(Book)
err := rows.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price)
if err != nil {
http.Error(w, err.Error(), 500)
return
}
bks = append(bks, bk)
}
if err = rows.Err(); err != nil {
http.Error(w, err.Error(), 500)
return
}
for _, bk := range bks {
fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price)
}
}
So how is this different?
We use the init() function to set up our connection pool and assign it to the global variable db. We're using a global variable to store the connection pool because it's an easy way of making it available to our HTTP handlers – but it's by no means the only way. Because sql.Open() doesn't actually check a connection, we also call DB.Ping() to make sure that everything works OK on startup.
In the booksIndex hander we return a 405 Method Not Allowed response for any non-GET request. Then we have our data access logic. This is exactly the same as the earlier example, except that we're now returning proper HTTP errors instead of exiting the program. Lastly we write the books' details as plain text to the http.ResponseWriter.
Run the application and then make a request:
$ curl -i localhost:3000/books
HTTP/1.1 200 OK
Content-Length: 205
Content-Type: text/plain; charset=utf-8
978-1503261969, Emma, Jayne Austen, £9.44
978-1505255607, The Time Machine, H. G. Wells, £5.99
978-1503379640, The Prince, Niccolò Machiavelli, £6.99
Querying a single row
For the GET /books/show route we want to retrieve single book based on its ISBN, with the ISBN being passed in the query string like:
/books/show?isbn=978-1505255607
We'll create a new bookShow() handler for this:
File: main.go
...
func main() {
http.HandleFunc("/books", booksIndex)
http.HandleFunc("/books/show", booksShow)
http.ListenAndServe(":3000", nil)
}
...
func booksShow(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
http.Error(w, http.StatusText(405), 405)
return
}
isbn := r.FormValue("isbn")
if isbn == "" {
http.Error(w, http.StatusText(400), 400)
return
}
row := db.QueryRow("SELECT * FROM books WHERE isbn = $1", isbn)
bk := new(Book)
err := row.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price)
if err == sql.ErrNoRows {
http.NotFound(w, r)
return
} else if err != nil {
http.Error(w, err.Error(), 500)
return
}
fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price)
}
Once again the handler starts again by checking that it's dealing with a GET request.
We then use the Request.FormValue() method to fetch the ISBN value from the request query string. This returns an empty string if there's no parameter found, so we check for that and issue a 400 Bad Request response if it's missing.
Now we get to the interesting bit: DB.QueryRow(). This method is similar to DB.Query, except that it fetches a single row instead of multiple rows.
Because we need to include untrusted input (the isbn variable) in our query we take advantage of placeholder parameters, passing in the value of our placeholder as the second argument to DB.QueryRow() like so:
db.QueryRow("SELECT * FROM books WHERE isbn = $1", isbn)
Behind the scenes, db.QueryRow (and also db.Query() and db.Exec()) work by creating a new prepared statement on the database, and subsequently execute that prepared statement using the placeholder parameters provided. This means that all three methods are safe from SQL injection when used correctly . From Wikipedia:
Prepared statements are resilient against SQL injection, because parameter values, which are transmitted later using a different protocol, need not be correctly escaped. If the original statement template is not derived from external input, injection cannot occur.
The placeholder parameter syntax differs depending on your database. Postgres uses the $N notation, but MySQL, SQL Server and others use the ? character as a placeholder.
OK, let's get back to our code.
After we've got a row from DB.QueryRow() we use row.Scan() to copy the values into a new Book object. Note how any errors from DB.QueryRow() are deferred and not surfaced until we call row.Scan().
If our query returned no rows, our call to row.Scan() will return an error of the type sql.ErrNoRows. We check for that error type specifically and return a 404 Not Found response if that's the case. We then handle all other errors by returning a 500 Internal Server Error.
If everything went OK, we write the book details to the http.ResponseWriter.
Give it a try:
$ curl -i localhost:3000/books/show?isbn=978-1505255607
HTTP/1.1 200 OK
Content-Length: 54
Content-Type: text/plain; charset=utf-8
978-1505255607, The Time Machine, H. G. Wells, £5.99
If you play around with the ISBN value, or issue a malformed request you should see that you get the appropriate error responses.
Executing a statement
For our final POST /books/create route we'll make a new booksCreate() handler and use DB.Exec() to execute a INSERT statement. You can take the same approach for an UPDATE, DELETE, or any other action that doesn't return rows.
In the booksCreate() handler we check we're dealing with a POST request, and then fetch the request parameters using request.FormValue(). We verify that all the necessary parameters exist, and in the case of price use the strconv.ParseFloat() to convert the parameter from a string into a float.
We then carry out the insert using db.Exec(), passing our new book details as parameters just like we did in the previous example. Note that DB.Exec(), like DB.Query() and DB.QueryRow(), is a variadic function, which means you can pass in as many parameters as you need.
The db.Exec() method returns an object satisfying the sql.Result interface, which you can either use (like we are here) or discard with the blank identifier.
The sql.Result() interface guarantees two methods: LastInsertId() – which is often used to return the value of an new auto increment id, and RowsAffected() – which contains the number of rows that the statement affected. In this code we're picking up the latter, and then using it in our plain text confirmation message.
It's worth noting that not all drivers support the LastInsertId() and RowsAffected() methods, and calling them may return an error. For example, pq doesn't support LastInsertId() – if you need that functionality you'll have to take an approach like this one.
Let's try out the /books/create route, passing our parameters in the POST body:
$ curl -i -X POST -d "isbn=978-1470184841&title=Metamorphosis&author=Franz Kafka&price=5.90" localhost:3000/books/create
HTTP/1.1 200 OK
Content-Length: 58
Content-Type: text/plain; charset=utf-8
Book 978-1470184841 created successfully (1 row affected)
Using DB.Prepare()
Something you might be wondering is: Why aren't we using DB.Prepare()?
As I explained a bit earlier, we kinda are behind the scenes. All of DB.Query(), DB.Exec() and DB.QueryRow() set up a prepared statement on the database, run it with the parameters provided, and then close (or deallocate) the prepared statement.
But the downside of this is obvious: we have 3 round trips to the database with each HTTP request, whereas if we set up prepared statements with DB.Prepare() – possibly in the init() function – we could have only one round trip each time.
But the trade-off isn't that simple. Prepared statements only last for the duration of the current database session. If the session ends, then the prepared statements must be recreated before being used again. So if there's database downtime or a restart you'll need to recreate the prepared statements.
For a web application where latency is critical it might be worth the effort to setup monitoring for your database, and reinitialise the prepared statements after an outage. But for an application like this where latency isn't that important, using DB.Query()et al is clear and effective enough.
At the moment all our database access logic is mixed in with our HTTP handlers. It's probably a good idea to refactor this for easier maintainability and DRYness as our application grows.
But this tutorial is already pretty long, so I'll explore some of the options for refactoring our code in the next post – Practical Persistence in Go: Organising Database Access (coming soon!)
Additional tools
The Sqlx package by Jason Moiron provides some additions to the standard database/sql functionality, including support for named placeholder parameters and automatic marshalling of rows into structs.
If you're looking for something more ORM-ish, you might like to consider Modl by the same author, or gorp by James Cooper.
The null package by can help make managing nullable values easier, if that's something you need to do a lot of.
Lastly, I found the tutorials at go-database-sql.org to be clear and helpful. Especially worth reading is the surprises and limitations section.
If you found this post useful, you might like to subscribe to my RSS feed.
In the context of a web application what would you consider a Go best practice for accessing the database in (HTTP or other) handlers?
The replies it got were a genuinely interesting mix. Some people advised using dependency injection, a few espoused the simplicity of using global variables, others suggested putting the connection pool pointer into x/net/context.
Me? I think the right answer depends on the project.
What's the overall structure and size of the project? What's your approach to testing? How is it likely to grow in the future? All these things and more should play a part when you pick an approach to take.
So in this post I'll take a look at four different methods for organising your code and structuring access to your database connection pool.
Global variables
The first approach we'll look at is a common and straightforward one – putting the pointer to your database connection pool in a global variable.
To keep code nice and DRY, you'll sometimes see this combined with an initialisation function that allows the connection pool global to be set from other packages and tests.
I like concrete examples, so let's carry on working with the online bookstore database and code from my previous post. We'll create a simple application with an MVC-like structure – with the HTTP handlers in main and a models package containing a global DB variable, InitDB() function, and our database logic.
package main
import (
"bookstore/models"
"fmt"
"net/http"
)
func main() {
models.InitDB("postgres://user:pass@localhost/bookstore")
http.HandleFunc("/books", booksIndex)
http.ListenAndServe(":3000", nil)
}
func booksIndex(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
http.Error(w, http.StatusText(405), 405)
return
}
bks, err := models.AllBooks()
if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
for _, bk := range bks {
fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price)
}
}
File: models/db.go
package models
import (
"database/sql"
_ "github.com/lib/pq"
"log"
)
var db *sql.DB
func InitDB(dataSourceName string) {
var err error
db, err = sql.Open("postgres", dataSourceName)
if err != nil {
log.Panic(err)
}
if err = db.Ping(); err != nil {
log.Panic(err)
}
}
File: models/books.go
package models
type Book struct {
Isbn string
Title string
Author string
Price float32
}
func AllBooks() ([]*Book, error) {
rows, err := db.Query("SELECT * FROM books")
if err != nil {
return nil, err
}
defer rows.Close()
bks := make([]*Book, 0)
for rows.Next() {
bk := new(Book)
err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price)
if err != nil {
return nil, err
}
bks = append(bks, bk)
}
if err = rows.Err(); err != nil {
return nil, err
}
return bks, nil
}
If you run the application and make a request to /books you should get a response similar to:
$ curl -i localhost:3000/books
HTTP/1.1 200 OK
Content-Length: 205
Content-Type: text/plain; charset=utf-8
978-1503261969, Emma, Jayne Austen, £9.44
978-1505255607, The Time Machine, H. G. Wells, £5.99
978-1503379640, The Prince, Niccolò Machiavelli, £6.99
Using a global variable like this is potentially a good fit if:
All your database logic is contained in the same package.
Your application is small enough that keeping track of globals in your head isn't a problem.
Your approach to testing means that you don't need to mock the database or run tests in parallel.
For the example above using a global works just fine. But what happens in more complicated applications where database logic is spread over multiple packages?
One option is to have multiple InitDB calls, but that can quickly become cluttersome and I've personally found it a bit flaky (it's easy to forget to initialise a connection pool and get nil-pointer panics at runtime). A second option is to create a separate config package with an exported DB variable and import "yourproject/config" into every file that needs it. Just in case it isn't immediately understandable what I mean, I've included a simple example in this gist.
Dependency injection
The second approach we'll look at is dependency injection. In our example, we want to explicitly pass a connection pool pointer to our HTTP handlers and then onward to our database logic.
In a real-world application there are probably extra application-level (and concurrency-safe) items that you want your handlers to have access to. Things like pointers to your logger or template cache, as well as the database connection pool.
So for projects where all your handlers are in the same package, a neat approach is to put these items into a custom Env type:
type Env struct {
db *sql.DB
logger *log.Logger
templates *template.Template
}
… and then define your handlers as methods against Env. This provides a clean and idiomatic way of making the connection pool (and potentially other items) available to your handlers. Here's a full example:
File: main.go
package main
import (
"bookstore/models"
"database/sql"
"fmt"
"log"
"net/http"
)
type Env struct {
db *sql.DB
}
func main() {
db, err := models.NewDB("postgres://user:pass@localhost/bookstore")
if err != nil {
log.Panic(err)
}
env := &Env{db: db}
http.HandleFunc("/books", env.booksIndex)
http.ListenAndServe(":3000", nil)
}
func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
http.Error(w, http.StatusText(405), 405)
return
}
bks, err := models.AllBooks(env.db)
if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
for _, bk := range bks {
fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price)
}
}
package models
import "database/sql"
type Book struct {
Isbn string
Title string
Author string
Price float32
}
func AllBooks(db *sql.DB) ([]*Book, error) {
rows, err := db.Query("SELECT * FROM books")
if err != nil {
return nil, err
}
defer rows.Close()
bks := make([]*Book, 0)
for rows.Next() {
bk := new(Book)
err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price)
if err != nil {
return nil, err
}
bks = append(bks, bk)
}
if err = rows.Err(); err != nil {
return nil, err
}
return bks, nil
}
Or using a closure…
If you don't want to define your handlers as methods on Env an alternative approach is to put your handler logic into a closure and close over the Env variable like so:
File: main.go
package main
import (
"bookstore/models"
"database/sql"
"fmt"
"log"
"net/http"
)
type Env struct {
db *sql.DB
}
func main() {
db, err := models.NewDB("postgres://user:pass@localhost/bookstore")
if err != nil {
log.Panic(err)
}
env := &Env{db: db}
http.Handle("/books", booksIndex(env))
http.ListenAndServe(":3000", nil)
}
func booksIndex(env *Env) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
http.Error(w, http.StatusText(405), 405)
return
}
bks, err := models.AllBooks(env.db)
if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
for _, bk := range bks {
fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price)
}
})
}
Dependency injection in this way is quite a nice approach when:
All your handlers are contained in the same package.
There is a common set of dependencies that each of your handlers need.
Your approach to testing means that you don't need to mock the database or run tests in parallel.
Again, you could still use this general approach if your handlers and database logic are spread across multiple packages. One way to achieve this would be to setup a separate config package exporting the Env type and close overconfig.Env in the same way as the example above. Here's a basic gist.
Using an interface
We can take this dependency injection example a little further. Let's change the models package so that it exports a custom DB type (which embeds *sql.DB) and implement our database logic as methods against the DB type.
The advantages of this are twofold: first it gives our code a really clean structure, but – more importantly – it also opens up the potential to mock our database for unit testing.
Let's amend the example to include a new Datastore interface, which implements exactly the same methods as our new DB type.
type Datastore interface {
AllBooks() ([]*Book, error)
}
We can then use this interface instead of the direct DB type throughout our application. Here's the updated example:
File: main.go
package main
import (
"fmt"
"log"
"net/http"
"bookstore/models"
)
type Env struct {
db models.Datastore
}
func main() {
db, err := models.NewDB("postgres://user:pass@localhost/bookstore")
if err != nil {
log.Panic(err)
}
env := &Env{db}
http.HandleFunc("/books", env.booksIndex)
http.ListenAndServe(":3000", nil)
}
func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
http.Error(w, http.StatusText(405), 405)
return
}
bks, err := env.db.AllBooks()
if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
for _, bk := range bks {
fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price)
}
}
package models
type Book struct {
Isbn string
Title string
Author string
Price float32
}
func (db *DB) AllBooks() ([]*Book, error) {
rows, err := db.Query("SELECT * FROM books")
if err != nil {
return nil, err
}
defer rows.Close()
bks := make([]*Book, 0)
for rows.Next() {
bk := new(Book)
err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price)
if err != nil {
return nil, err
}
bks = append(bks, bk)
}
if err = rows.Err(); err != nil {
return nil, err
}
return bks, nil
}
Because our handlers are now using the Datastore interface we can easily create mock database responses for any unit tests:
package main
import (
"bookstore/models"
"net/http"
"net/http/httptest"
"testing"
)
type mockDB struct{}
func (mdb *mockDB) AllBooks() ([]*models.Book, error) {
bks := make([]*models.Book, 0)
bks = append(bks, &models.Book{"978-1503261969", "Emma", "Jayne Austen", 9.44})
bks = append(bks, &models.Book{"978-1505255607", "The Time Machine", "H. G. Wells", 5.99})
return bks, nil
}
func TestBooksIndex(t *testing.T) {
rec := httptest.NewRecorder()
req, _ := http.NewRequest("GET", "/books", nil)
env := Env{db: &mockDB{}}
http.HandlerFunc(env.booksIndex).ServeHTTP(rec, req)
expected := "978-1503261969, Emma, Jayne Austen, £9.44\n978-1505255607, The Time Machine, H. G. Wells, £5.99\n"
if expected != rec.Body.String() {
t.Errorf("\n...expected = %v\n...obtained = %v", expected, rec.Body.String())
}
}
Request-scoped context
Finally let's look at using request-scoped context to store and pass around the database connection pool. Specifically, we'll make use of the x/net/context package.
Personally I'm not a fan of storing application-level variables in request-scoped context – it feels clunky and burdensome to me. The x/net/context documentation kinda advises against it too:
Use context Values only for request-scoped data that transits processes and APIs, not for passing optional parameters to functions.
That said, people do use this approach. And if your project consists of a sprawling set of packages – and using a global config is out of the question – it's quite an attractive proposition.
Let's adapt the bookstore example one last time, passing context.Context to our handlers using the pattern suggested in this excellent article by Joe Shaw.
In this post I'm going to be looking at using Redis as a data persistence layer for a Go application. We'll start by explaining a few of the essential concepts, and then build a working web application which highlights some techniques for using Redis in a concurrency-safe way.
This post assumes a basic knowledge of Redis itself (and a working installation, if you want to follow along). If you haven't used Redis before, I highly recommend reading the Little Book of Redis by Karl Seguin or running through the Try Redis interactive tutorial.
Installing a driver
First up we need to install a Go driver (or client) for Redis. A list of available drivers is located at http://redis.io/clients#go.
Throughout this post we'll be using the Radix.v2 driver. It's well maintained, and I've found it's API clean and straightforward to use. If you're following along you'll need to go get it:
$ go get github.com/mediocregopher/radix.v2
Notice that the Radix.v2 package is broken up into 6 sub-packages (cluster, pool, pubsub, redis, sentinel and util). To begin with we'll only need the functionality in the redis package, so our import statements should look like:
As an example, let's say that we have an online record shop and want to store information about the albums for sale in Redis.
There's many different ways we could model this data in Redis, but we'll keep things simple and store each album as a hash – with fields for title, artist, price and the number of 'likes' that it has. As the key for each album hash we'll use the pattern album:{id}, where id is a unique integer value.
So if we wanted to store a new album using the Redis CLI, we could execute a HMSET command along the lines of:
127.0.0.1:6379> HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8
OK
To do the same thing from a Go application, we need to combine a couple of functions from the Radix.v2 redis package.
The first is the Dial() function, which returns a new connection (or in Radix.v2 terms, client) to our Redis server.
The second is the client.Cmd() method, which sends a command to our Redis server across the connection. This always returns a pointer to a Resp object, which holds the reply from our command (or any error message if it didn't work).
It's quite straightforward in practice:
File: main.go
package main
import (
"fmt"
// Import the Radix.v2 redis package.
"github.com/mediocregopher/radix.v2/redis"
"log"
)
func main() {
// Establish a connection to the Redis server listening on port 6379 of the
// local machine. 6379 is the default port, so unless you've already
// changed the Redis configuration file this should work.
conn, err := redis.Dial("tcp", "localhost:6379")
if err != nil {
log.Fatal(err)
}
// Importantly, use defer to ensure the connection is always properly
// closed before exiting the main() function.
defer conn.Close()
// Send our command across the connection. The first parameter to Cmd()
// is always the name of the Redis command (in this example HMSET),
// optionally followed by any necessary arguments (in this example the
// key, followed by the various hash fields and values).
resp := conn.Cmd("HMSET", "album:1", "title", "Electric Ladyland", "artist", "Jimi Hendrix", "price", 4.95, "likes", 8)
// Check the Err field of the *Resp object for any errors.
if resp.Err != nil {
log.Fatal(resp.Err)
}
fmt.Println("Electric Ladyland added!")
}
In this example we're not really interested in the reply from Redis (all successful HMSET commands just reply with the string "OK") so we don't do anything with the *Resp object apart from checking it for any errors.
In such cases, it's common to chain the check against Err like so:
When we are interested in the reply from Redis, the Resp object comes with some useful helper functions for converting the reply into a Go type we can easily work with. These are:
Resp.Bytes() – converts a single reply to a byte slice ([]byte)
Resp.Float64() – converts a single reply to a Float64
Resp.Int() – converts a single reply to a int
Resp.Int64() – converts a single reply to a int64
Resp.Str() – converts a single reply to a string
Resp.Array() – converts an array reply to an slice of individual Resp objects ([]*Resp)
Resp.List() – converts an array reply to an slice of strings ([]string)
Resp.ListBytes() – converts an array reply to an slice of byte slices ([][]byte)
Resp.Map() – converts an array reply to a map of strings, using each item in the array reply alternately as the keys and values for the map (map[string]string)
Let's use some of these in conjunction with the HGET command to retrieve information from one of our album hashes:
File: main.go
package main
import (
"fmt"
"github.com/mediocregopher/radix.v2/redis"
"log"
)
func main() {
conn, err := redis.Dial("tcp", "localhost:6379")
if err != nil {
log.Fatal(err)
}
defer conn.Close()
// Issue a HGET command to retrieve the title for a specific album, and use
// the Str() helper method to convert the reply to a string.
title, err := conn.Cmd("HGET", "album:1", "title").Str()
if err != nil {
log.Fatal(err)
}
// Similarly, get the artist and convert it to a string.
artist, err := conn.Cmd("HGET", "album:1", "artist").Str()
if err != nil {
log.Fatal(err)
}
// And the price as a float64...
price, err := conn.Cmd("HGET", "album:1", "price").Float64()
if err != nil {
log.Fatal(err)
}
// And the number of likes as an integer.
likes, err := conn.Cmd("HGET", "album:1", "likes").Int()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s by %s: £%.2f [%d likes]\n", title, artist, price, likes)
}
It's worth pointing out that, when we use these helper methods, the error they return could relate to one of two things: either the failed execution of the command (as stored in the Resp object's Err field), or the conversion of the reply data to the desired type (for example, we'd get an error if we tried to convert the reply "Jimi Hendrix" to a Float64). There's no way of knowing which kind of error it is unless we examine the error message.
If you run the code above you should get output which looks like:
$ go run main.go
Electric Ladyland by Jimi Hendrix: £4.95 [8 likes]
Let's now look at a more complete example, where we use the HGETALL command to retrieve all fields from an album hash in one go and store the information in a custom Album struct.
File: main.go
package main
import (
"fmt"
"github.com/mediocregopher/radix.v2/redis"
"log"
"strconv"
)
// Define a custom struct to hold Album data.
type Album struct {
Title string
Artist string
Price float64
Likes int
}
func main() {
conn, err := redis.Dial("tcp", "localhost:6379")
if err != nil {
log.Fatal(err)
}
defer conn.Close()
// Fetch all album fields with the HGETALL command. Because HGETALL
// returns an array reply, and because the underlying data structure in
// Redis is a hash, it makes sense to use the Map() helper function to
// convert the reply to a map[string]string.
reply, err := conn.Cmd("HGETALL", "album:1").Map()
if err != nil {
log.Fatal(err)
}
// Use the populateAlbum helper function to create a new Album object from
// the map[string]string.
ab, err := populateAlbum(reply)
if err != nil {
log.Fatal(err)
}
fmt.Println(ab)
}
// Create, populate and return a pointer to a new Album struct, based on data
// from a map[string]string.
func populateAlbum(reply map[string]string) (*Album, error) {
var err error
ab := new(Album)
ab.Title = reply["title"]
ab.Artist = reply["artist"]
// We need to use the strconv package to convert the 'price' value from a
// string to a float64 before assigning it.
ab.Price, err = strconv.ParseFloat(reply["price"], 64)
if err != nil {
return nil, err
}
// Similarly, we need to convert the 'likes' value from a string to an
// integer.
ab.Likes, err = strconv.Atoi(reply["likes"])
if err != nil {
return nil, err
}
return ab, nil
}
Running this code should give an output like:
$ go run main.go
&{Electric Ladyland Jimi Hendrix 4.95 8}
Using in a web application
One important thing to know about Radix.v2 is that the redis package (which we've used so far) is not safe for concurrent use.
If we want to access a single Redis server from multiple goroutines, as we would in a web application, we must use the pool package instead. This allows us to establish a pool of Redis connections and each time we want to use a connection we fetch it from the pool, execute our command on it, and return it too the pool.
We'll illustrate this in a simple web application, building on the online record store example we've already used. Our finished app will support 3 functions:
Method
Path
Function
GET
/album?id=1
Show details of a specific album (using the id provided in the query string)
POST
/like
Add a new like for a specific album (using the id provided in the request body)
GET
/popular
List the top 3 most liked albums in order
If you'd like to follow along, head on over into your Go workspace and create a basic application scaffold…
$ cd $GOPATH/src
$ mkdir -p recordstore/models
$ cd recordstore
$ touch main.go models/albums.go
$ tree
.
├── main.go
└── models
└── albums.go
…And use the Redis CLI to add a few additional albums, along with a new likes sorted set. This sorted set will be used within the GET /popular route to help us quickly and efficiently retrieve the ids of albums with the most likes. Here's the commands to run:
HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8
HMSET album:2 title "Back in Black" artist "AC/DC" price 5.95 likes 3
HMSET album:3 title "Rumours" artist "Fleetwood Mac" price 7.95 likes 12
HMSET album:4 title "Nevermind" artist "Nirvana" price 5.95 likes 8
ZADD likes 8 1 3 2 12 3 8 4
We'll follow an MVC-ish pattern for our application and use the models/albums.go file for all our Redis-related logic.
In the models/albums.go file we'll use the init() function to establish a Redis connection pool on startup, and we'll repurpose the code we wrote earlier into a FindAlbum() function that we can use from our HTTP handlers.
File: models/albums.go
package models
import (
"errors"
// Import the Radix.v2 pool package, NOT the redis package.
"github.com/mediocregopher/radix.v2/pool"
"log"
"strconv"
)
// Declare a global db variable to store the Redis connection pool.
var db *pool.Pool
func init() {
var err error
// Establish a pool of 10 connections to the Redis server listening on
// port 6379 of the local machine.
db, err = pool.New("tcp", "localhost:6379", 10)
if err != nil {
log.Panic(err)
}
}
// Create a new error message and store it as a constant. We'll use this
// error later if the FindAlbum() function fails to find an album with a
// specific id.
var ErrNoAlbum = errors.New("models: no album found")
type Album struct {
Title string
Artist string
Price float64
Likes int
}
func populateAlbum(reply map[string]string) (*Album, error) {
var err error
ab := new(Album)
ab.Title = reply["title"]
ab.Artist = reply["artist"]
ab.Price, err = strconv.ParseFloat(reply["price"], 64)
if err != nil {
return nil, err
}
ab.Likes, err = strconv.Atoi(reply["likes"])
if err != nil {
return nil, err
}
return ab, nil
}
func FindAlbum(id string) (*Album, error) {
// Use the connection pool's Get() method to fetch a single Redis
// connection from the pool.
conn, err := db.Get()
if err != nil {
return nil, err
}
// Importantly, use defer and the connection pool's Put() method to ensure
// that the connection is always put back in the pool before FindAlbum()
// exits.
defer db.Put(conn)
// Fetch the details of a specific album. If no album is found with the
// given id, the map[string]string returned by the Map() helper method
// will be empty. So we can simply check whether it's length is zero and
// return an ErrNoAlbum message if necessary.
reply, err := conn.Cmd("HGETALL", "album:"+id).Map()
if err != nil {
return nil, err
} else if len(reply) == 0 {
return nil, ErrNoAlbum
}
return populateAlbum(reply)
}
Something worth elaborating on is the pool.New() function. In the above code we specify a pool size of 10, which simply limits the number of idle connections waiting in the pool to 10 at any one time. If all 10 connections are in use when an additional pool.Get() call is made a new connection will be created on the fly.
When you're only issuing one command on a connection, like in the FindAlbum() function above, it's possible to use the pool.Cmd() shortcut. This will automatically get a new connection from the pool, execute a given command, and then put the connection back in the pool.
Here's the FindAlbum() function re-written to use this shortcut:
Alright, let's head over to the main.go file and set up a simple web server and HTTP handler for the GET /album route.
File: main.go
package main
import (
"fmt"
"net/http"
"recordstore/models"
"strconv"
)
func main() {
// Use the showAlbum handler for all requests with a URL path beginning
// '/album'.
http.HandleFunc("/album", showAlbum)
http.ListenAndServe(":3000", nil)
}
func showAlbum(w http.ResponseWriter, r *http.Request) {
// Unless the request is using the GET method, return a 405 'Method Not
// Allowed' response.
if r.Method != "GET" {
w.Header().Set("Allow", "GET")
http.Error(w, http.StatusText(405), 405)
return
}
// Retrieve the id from the request URL query string. If there is no id
// key in the query string then Get() will return an empty string. We
// check for this, returning a 400 Bad Request response if it's missing.
id := r.URL.Query().Get("id")
if id == "" {
http.Error(w, http.StatusText(400), 400)
return
}
// Validate that the id is a valid integer by trying to convert it,
// returning a 400 Bad Request response if the conversion fails.
if _, err := strconv.Atoi(id); err != nil {
http.Error(w, http.StatusText(400), 400)
return
}
// Call the FindAlbum() function passing in the user-provided id. If
// there's no matching album found, return a 404 Not Found response. In
// the event of any other errors, return a 500 Internal Server Error
// response.
bk, err := models.FindAlbum(id)
if err == models.ErrNoAlbum {
http.NotFound(w, r)
return
} else if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
// Write the album details as plain text to the client.
fmt.Fprintf(w, "%s by %s: £%.2f [%d likes] \n", bk.Title, bk.Artist, bk.Price, bk.Likes)
}
If you run the application:
$ go run main.go
And make a request for one of the albums using cURL. You should get a response like:
$ curl -i localhost:3000/album?id=2
HTTP/1.1 200 OK
Content-Length: 42
Content-Type: text/plain; charset=utf-8
Back in Black by AC/DC: £5.95 [3 likes]
Using transactions
The second route, POST /likes, is quite interesting.
When a user likes an album we need to issue two distinct commands: a HINCRBY to increment the likes field in the album hash, and a ZINCRBY to increment the relevant score in our likes sorted set.
This creates a problem. Ideally we would want both keys to be incremented at exactly the same time as a single atomic action. Having one key updated after the other opens up the potential for data races to occur.
The solution to this is to use Redis transactions, which let us run multiple commands together as an atomic group. To do this we use the MULTI command to start a transaction, followed by the commands (in our case a HINCRBY and ZINCRBY), and finally the EXEC command (which then executes our both our commands together as an atomic group).
Let's create a new IncrementLikes() function in the albums model which uses this technique.
File: models/albums.go
...
func IncrementLikes(id string) error {
conn, err := db.Get()
if err != nil {
return err
}
defer db.Put(conn)
// Before we do anything else, check that an album with the given id
// exists. The EXISTS command returns 1 if a specific key exists
// in the database, and 0 if it doesn't.
exists, err := conn.Cmd("EXISTS", "album:"+id).Int()
if err != nil {
return err
} else if exists == 0 {
return ErrNoAlbum
}
// Use the MULTI command to inform Redis that we are starting a new
// transaction.
err = conn.Cmd("MULTI").Err
if err != nil {
return err
}
// Increment the number of likes in the album hash by 1. Because it
// follows a MULTI command, this HINCRBY command is NOT executed but
// it is QUEUED as part of the transaction. We still need to check
// the reply's Err field at this point in case there was a problem
// queueing the command.
err = conn.Cmd("HINCRBY", "album:"+id, "likes", 1).Err
if err != nil {
return err
}
// And we do the same with the increment on our sorted set.
err = conn.Cmd("ZINCRBY", "likes", 1, id).Err
if err != nil {
return err
}
// Execute both commands in our transaction together as an atomic group.
// EXEC returns the replies from both commands as an array reply but,
// because we're not interested in either reply in this example, it
// suffices to simply check the reply's Err field for any errors.
err = conn.Cmd("EXEC").Err
if err != nil {
return err
}
return nil
}
We'll also update the main.go file to add an addLike() handler for the route:
File: main.go
func main() {
http.HandleFunc("/album", showAlbum)
http.HandleFunc("/like", addLike)
http.ListenAndServe(":3000", nil)
}
...
func addLike(w http.ResponseWriter, r *http.Request) {
// Unless the request is using the POST method, return a 405 'Method Not
// Allowed' response.
if r.Method != "POST" {
w.Header().Set("Allow", "POST")
http.Error(w, http.StatusText(405), 405)
return
}
// Retreive the id from the POST request body. If there is no parameter
// named "id" in the request body then PostFormValue() will return an
// empty string. We check for this, returning a 400 Bad Request response
// if it's missing.
id := r.PostFormValue("id")
if id == "" {
http.Error(w, http.StatusText(400), 400)
return
}
// Validate that the id is a valid integer by trying to convert it,
// returning a 400 Bad Request response if the conversion fails.
if _, err := strconv.Atoi(id); err != nil {
http.Error(w, http.StatusText(400), 400)
return
}
// Call the IncrementLikes() function passing in the user-provided id. If
// there's no album found with that id, return a 404 Not Found response.
// In the event of any other errors, return a 500 Internal Server Error
// response.
err := models.IncrementLikes(id)
if err == models.ErrNoAlbum {
http.NotFound(w, r)
return
} else if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
// Redirect the client to the GET /ablum route, so they can see the
// impact their like has had.
http.Redirect(w, r, "/album?id="+id, 303)
}
If you make a POST request to like one of the albums you should now get a response like:
$ curl -i -L -d "id=2" localhost:3000/like
HTTP/1.1 303 See Other
Location: /album?id=2
Date: Thu, 25 Feb 2016 17:08:19 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8
HTTP/1.1 200 OK
Content-Length: 42
Content-Type: text/plain; charset=utf-8
Back in Black by AC/DC: £5.95 [4 likes]
Using the Watch command
OK, on to our final route: GET /popular. This route will display the details of the top 3 albums with the most likes, so to facilitate this we'll create a FindTopThree() function in the models/albums.go file. In this function we need to:
Use the ZREVRANGE command to fetch the 3 album ids with the highest score (i.e. most likes) from our likes sorted set.
Loop through the returned ids, using the HGETALL command to retrieve the details of each album and add them to a []*Album slice.
Again, it's possible to imagine a race condition occurring here. If a second client happens to like an album at the exact moment between our ZREVRANGE command and the HGETALLs for all 3 albums being completed, our user could end up being sent wrong or mis-ordered data.
The solution here is to use the Redis WATCH command in conjunction with a transaction. WATCH instructs Redis to monitor a specific key for any changes. If another client modifies our watched key between our WATCH instruction and our subsequent transaction's EXEC, the transaction will fail and return a nil reply. If no client changes the value before our EXEC, the transaction will complete as normal. We can execute our code in a loop until the transaction is successful.
File: models/albums.go
package models
import (
"errors"
"github.com/mediocregopher/radix.v2/pool"
// Import the Radix.v2 redis package (we need access to its Nil type).
"github.com/mediocregopher/radix.v2/redis"
"log"
"strconv"
)
...
func FindTopThree() ([]*Album, error) {
conn, err := db.Get()
if err != nil {
return nil, err
}
defer db.Put(conn)
// Begin an infinite loop.
for {
// Instruct Redis to watch the likes sorted set for any changes.
err = conn.Cmd("WATCH", "likes").Err
if err != nil {
return nil, err
}
// Use the ZREVRANGE command to fetch the album ids with the highest
// score (i.e. most likes) from our 'likes' sorted set. The ZREVRANGE
// start and stop values are zero-based indexes, so we use 0 and 2
// respectively to limit the reply to the top three. Because ZREVRANGE
// returns an array response, we use the List() helper function to
// convert the reply into a []string.
reply, err := conn.Cmd("ZREVRANGE", "likes", 0, 2).List()
if err != nil {
return nil, err
}
// Use the MULTI command to inform Redis that we are starting a new
// transaction.
err = conn.Cmd("MULTI").Err
if err != nil {
return nil, err
}
// Loop through the ids returned by ZREVRANGE, queuing HGETALL
// commands to fetch the individual album details.
for _, id := range reply {
err := conn.Cmd("HGETALL", "album:"+id).Err
if err != nil {
return nil, err
}
}
// Execute the transaction. Importantly, use the Resp.IsType() method
// to check whether the reply from EXEC was nil or not. If it is nil
// it means that another client changed the WATCHed likes sorted set,
// so we use the continue command to re-run the loop.
ereply := conn.Cmd("EXEC")
if ereply.Err != nil {
return nil, err
} else if ereply.IsType(redis.Nil) {
continue
}
// Otherwise, use the Array() helper function to convert the
// transaction reply to an array of Resp objects ([]*Resp).
areply, err := ereply.Array()
if err != nil {
return nil, err
}
// Create a new slice to store the album details.
abs := make([]*Album, 3)
// Iterate through the array of Resp objects, using the Map() helper
// to convert the individual reply into a map[string]string, and then
// the populateAlbum function to create a new Album object
// from the map. Finally store them in order in the abs slice.
for i, reply := range areply {
mreply, err := reply.Map()
if err != nil {
return nil, err
}
ab, err := populateAlbum(mreply)
if err != nil {
return nil, err
}
abs[i] = ab
}
return abs, nil
}
}
Using this from our web application is nice and straightforward:
File: main.go
func main() {
http.HandleFunc("/album", showAlbum)
http.HandleFunc("/like", addLike)
http.HandleFunc("/popular", listPopular)
http.ListenAndServe(":3000", nil)
}
...
func listPopular(w http.ResponseWriter, r *http.Request) {
// Unless the request is using the GET method, return a 405 'Method Not
// Allowed' response.
if r.Method != "GET" {
w.Header().Set("Allow", "GET")
http.Error(w, http.StatusText(405), 405)
return
}
// Call the FindTopThree() function, returning a return a 500 Internal
// Server Error response if there's any error.
abs, err := models.FindTopThree()
if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
// Loop through the 3 albums, writing the details as a plain text list
// to the client.
for i, ab := range abs {
fmt.Fprintf(w, "%d) %s by %s: £%.2f [%d likes] \n", i+1, ab.Title, ab.Artist, ab.Price, ab.Likes)
}
}
One note about WATCH: a key will remain WATCHed until either we either EXEC (or DISCARD) our transaction, or we manually call UNWATCH on the key. So calling EXEC, as we do in the above example, is sufficient and the likes sorted set will be automatically UNWATCHed.
Making a request to the GET /popular route should now yield a response similar to:
$ curl -i localhost:3000/popular
HTTP/1.1 200 OK
Content-Length: 147
Content-Type: text/plain; charset=utf-8
1) Rumours by Fleetwood Mac: £7.95 [12 likes]
2) Nevermind by Nirvana: £5.95 [8 likes]
3) Electric Ladyland by Jimi Hendrix: £4.95 [8 likes]
I’ve just released SCS, a session management package for Go 1.7+.
Its design leverages Go’s new context package to automatically load and save session data via middleware.
Importantly, it also provides the security features that you need when using server-side session stores (like straightforward session token regeneration) and supports both absolute and inactivity timeouts. The session data is safe for concurrent use.
A simple example
SCS is broken up into small single-purpose packages for ease of use. You should install the session package and your choice of session storage engine from the following table:
$ go get github.com/alexedwards/scs/session
$ go get github.com/alexedwards/scs/engine/memstore
Usage is then simple:
File: main.go
package main
import (
"io"
"net/http"
"github.com/alexedwards/scs/engine/memstore"
"github.com/alexedwards/scs/session"
)
func main() {
// Create the session manager middleware, passing in a new storage
// engine instance as the first parameter.
sessionManager := session.Manage(memstore.New(0))
// Set up your HTTP handlers in the normal way.
mux := http.NewServeMux()
mux.HandleFunc("/put", putHandler)
mux.HandleFunc("/get", getHandler)
// Wrap your handlers with the session manager middleware.
http.ListenAndServe(":4000", sessionManager(mux))
}
func putHandler(w http.ResponseWriter, r *http.Request) {
// Use the PutString helper to add a new key and associated string value
// to the session data. Helpers for other types are included.
err := session.PutString(r, "message", "Hello from a session!")
if err != nil {
http.Error(w, err.Error(), 500)
}
}
func getHandler(w http.ResponseWriter, r *http.Request) {
// Use the GetString helper to retrieve the value associated with the
// "message" key.
msg, err := session.GetString(r, "message")
if err != nil {
http.Error(w, err.Error(), 500)
return
}
io.WriteString(w, msg)
}
You should be able to verify that the session data is being across requests with curl:
$ curl -i -c cookies localhost:4000/put
HTTP/1.1 200 OK
Set-Cookie: scs.session.token=uts3FRcCMOIXpyx5uZx28Y54uUFRHxgtYhbgD4epeI4; Path=/; HttpOnly
Date: Tue, 30 Aug 2016 17:37:12 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8
$ curl -i -b cookies localhost:4000/get
HTTP/1.1 200 OK
Date: Tue, 30 Aug 2016 17:37:21 GMT
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Hello from a session!
Over the past few years I've built up a collection of snippets for validating inputs in Go. There's nothing new or groundbreaking here, but hopefully they might save you some time.
The snippets assume that the data to validate is stored as strings in r.Form, but the principles are the same no matter where the data has come from.
You can verify that a value for the "foo" field has been submitted with:
if r.Form.Get("foo") == "" {
fmt.Println("error: foo is required")
}
For checkbox and select inputs this will ensure that at least one item has been checked.
Blank text
If you have the HTML form:
<input type="text" name="foo">
You can verify that a value for the "foo" field isn't blank (i.e. contains whitespace only) with the strings.TrimSpace function:
import "strings"
···
if strings.TrimSpace(r.Form.Get("foo")) == "" {
fmt.Println("error: foo must not be blank")
}
Min and max length (bytes)
If you have the HTML form:
<input type="text" name="foo">
You can verify that the "foo" field contains a certain number of bytes with the builtin len function:
l := len(r.Form.Get("foo"))
if l < 5 || l > 10 {
fmt.Println("error: foo must be between 5 and 10 bytes long")
}
Min and max length (number of characters)
If you have the HTML form:
<input type="text" name="foo">
You can verify that the "foo" field contains a certain number of characters with the utf8.RuneCountInString function. This is subtly different to checking the number of bytes. For example, the string "Zoë" contains 3 characters but is 4 bytes long because of the accented character.
import "unicode/utf8"
···
l := utf8.RuneCountInString(r.Form.Get("foo"))
if l < 5 || l > 10 {
fmt.Println("error: foo must be between 5 and 10 characters long")
}
Starts with, ends with and contains
If you have the HTML form:
<input type="text" name="foo">
You can verify that the "foo" field starts with, ends with, or contains a particular string using the functions in the strings package:
import "strings"
···
// Check that the field value starts with 'abc'.
if !strings.HasPrefix(r.Form.Get("foo"), "abc") {
fmt.Println("error: foo does not start with 'abc'")
}
// Check that the field value ends with 'abc'.
if !strings.HasSuffix(r.Form.Get("foo"), "abc") {
fmt.Println("error: foo does not end with 'abc'")
}
// Check that the field value contains 'abc' anywhere in it.
if !strings.Contains(r.Form.Get("foo"), "abc") {
fmt.Println("error: foo does not contain 'abc'")
}
Matches regular expression pattern
If you have the HTML form:
<input type="text" name="foo">
You can verify that the "foo" field matches a particular regular expression using the regexp package. For example, to check that it matches the pattern ^[a-z]{4}\.[0-9]{2}$ (four lowercase letters followed by a period and two digits):
import "regexp"
···
// Pre-compiling the regular expression and storing it in a variable is more efficient
// if you're going to use it multiple times. The regexp.MustCompile function will
// panic on failure.
var rxPat = regexp.MustCompile(`^[a-z]{4}.[0-9]{2}$`)
if !rxPat.MatchString(r.Form.Get("foo")) {
fmt.Println("error: foo does not match the required pattern")
}
Note that because the dot character has a special meaning in regular expressions, we escaped it using the \ character so it is interpreted as a literal period character instead. In the example above we also used a raw string for the regular expression. If you use an interpreted string (i.e. a string surrounded by double quotes), you need to escape the backslash too because that's the escape character for interpreted strings. So you would need to write:
var rxPat = regexp.MustCompile("^[a-z]{4}\.[0-9]{2}$")
If you're not familiar with regular expressions then this guide from Mozilla is a good explanation.
Unicode character range
If you have the HTML form:
<input type="text" name="foo">
You can verify that the "foo" field only contains characters in a certain unicode range using the regexp package. For example, to check that it contains only Cyrillic characters in the two unicode blocks 0400 - 04FF (Cyrillic) and 0500 - 052F (Cyrillic Supplementary):
import "regexp"
···
// Use an interpreted string and the \u escape notation to create a regular
// expression matching the range of characters in the two unicode code blocks.
var rxCyrillic = regexp.MustCompile("^[\u0400-\u04FF\u0500-\u052F]+$")
if !rxCyrillic.MatchString(r.Form.Get("foo")) {
fmt.Println("error: foo must only contain Cyrillic characters")
}
Email validation
If you have the HTML form:
<input type="email" name="foo">
You can sanity check that the "foo" field contains an email address using the regexp package. Choosing a regular expression to use for email validation is a contentious topic, but as a starting point I would suggest the pattern recommended by the W3C and Web Hypertext Application Technology Working Group.
In addition, the email addresses have a practical limit of 254 bytes. Putting those together, a decent sanity check is:
import "regexp"
···
var rxEmail = regexp.MustCompile("^[a-zA-Z0-9.!#$%&'*+\\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$")
e := r.Form.Get("foo")
if len(e) > 254 || !rxEmail.MatchString(e) {
fmt.Println("error: foo is not a valid email address")
}
Note that we have to use an interpreted string for the regular expression because it contains a backtick character (which means we can't use a raw string).
URL validation
If you have the HTML form:
<input type="url" name="foo">
You can verify that the "foo" field contains a valid URL by first parsing it with the url.Parse function. This will take a URL string, break it into it's component pieces, and store it as a url.URL struct. You can then sanity check the component pieces as necessary.
For instance, to check that a URL is absolute (i.e has both a scheme and host) and that the scheme is either http or https:
import "net/url"
···
// If there are any major problems with the format of the URL, url.Parse() will
// return an error.
u, err := url.Parse(r.Form.Get("foo"))
if err != nil {
fmt.Println("error: foo is not a valid URL")
} else if u.Scheme == "" || u.Host == "" {
fmt.Println("error: foo must be an absolute URL")
} else if u.Scheme != "http" && u.Scheme != "https" {
fmt.Println("error: foo must begin with http or https")
}
You can verify that the "foo" field contains an integer by parsing it with the strconv.Atoi function. You can then sanity check the integer as necessary.
For instance, to check that an integer is a multiple of 5 between 0 and 100:
import "strconv"
···
n, err := strconv.Atoi(r.Form.Get("foo"))
if err != nil {
fmt.Println("error: foo must be an integer")
} else if n < 0 || n > 10 {
fmt.Printf("error: foo must be between 0 and 100")
} else if n%5 != 0 {
fmt.Println("error: foo must be an multiple of 5")
}
You can verify that the "foo" field contains an integer by parsing it with the strconv.ParseFloat function. You can then sanity check the float as necessary.
For instance, to check that an float is a between 0 and 1:
import "strconv"
n, err := strconv.ParseFloat(r.Form.Get("foo"), 64)
if err != nil {
fmt.Println("error: foo must be a float")
} else if n < 0 || n > 1 {
fmt.Printf("error: foo must be between 0 and 1")
}
You can verify that the "foo" field contains a valid date by parsing it with the time.Parse function. It will return an error if the date is not real: any day of month larger than 31 is rejected, as is February 29 in non-leap years, February 30, February 31, April 31, June 31, September 31, and November 31.
If you're not familiar with time.Parse, it converts strings with a given format into a time.Time object. You specify what the format is by passing the reference time (Mon Jan 2 15:04:05 -0700 MST 2006) as the first parameter, laid-out in the format you want. If you are expecting a date in the format YYYY-MM-DD then the reference time is 2006-01-02.
You can then use the various functions in the time package to sanity check the date as necessary.
For instance, to check that an date is valid and between 2017-01-01 and 2017-12-31:
import "time"
···
d, err := time.Parse("2006-01-02", r.Form.Get("foo"))
if err != nil {
fmt.Printf("error: foo is not a valid date")
} else if d.Year() != 2017 {
fmt.Printf("error: foo is not between 2017-01-01 and 2017-12-31")
}
You can verify that the "foo" field contains a valid datetime for a specific location by parsing it with the time.ParseInLocation function. This will return an error if the date is not real: any day of month larger than 31 is rejected, as is February 29 in non-leap years, February 30, February 31, April 31, June 31, September 31, and November 31.
For instance, to check that an datetime for the "Europe/Vienna" timezone is valid and between 2017-01-01 00:00 and 2017-12-31 23:59:
import "time"
···
// Load the users local time zone. This accepts a location name corresponding
// to a file in your IANA Time Zone database.
loc, err := time.LoadLocation("Europe/Vienna")
if err != nil {
···
}
d, err := time.ParseInLocation("2006-01-02T15:04:05", r.Form.Get("foo"), loc)
if err != nil {
fmt.Printf("error: foo is not a valid datetime")
} else if d.Year() != 2017 {
fmt.Printf("error: foo is not between 2017-01-01 00:00:00 and 2017-12-31 23:59:00")
}
The time zone database needed by LoadLocation may not be present on all systems, especially non-Unix systems. LoadLocation looks in the directory or uncompressed zip file named by the ZONEINFO environment variable, if any, then looks in known installation locations on Unix systems, and finally looks in $GOROOT/lib/time/zoneinfo.zip.
Radio, Select and Datalist (one-in-a-set) validation
You can check that the submitted value for the "foo" field is one of a known set like this:
set := map[string]bool{"wibble": true, "wobble": true, "wubble": true}
if !set[r.Form.Get("foo")] {
fmt.Printf("error: foo not match 'wibble', 'wobble' or 'wubble'")
}
To validate these, and make sure that all values sent by the form are either
wibble, wobble or wubble, we need to access the underlying form data directly and range over each value:
You can check that the submitted values for the "foo" field are part of a known set like this:
set := map[string]bool{"wibble": true, "wobble": true, "wubble": true}
for _, f := range r.Form["foo"] {
if !set[f] {
fmt.Printf("error: foo does not match 'wibble', 'wobble' or 'wubble'")
break
}
}
Single checkboxes
Sometimes you might have a single checkbox, and you want to verify that it has been checked. A common example is an "I accept the terms" checkbox on a form.
If you have the HTML form:
<input type="checkbox" name="foo" value="checked"> I accept the terms
You can verify that it has been checked like this:
if r.Form.Get("foo") != "checked" {
fmt.Println("foo must be checked")
}
If you're running a HTTP server and want to rate limit user requests, the go-to package to use is probably Tollbooth by Didip Kerabat. It's well maintained, has a good range of features and a clean and clear API.
But if you want something simple and lightweight – or just want to learn – it's not too difficult to roll your own middleware to handle rate limiting. In this post I'll run through the essentials of how to do that.
If you'd like to follow along, you'll need to install the x/time/rate package. This provides a token bucket rate-limiter algorithm (which is also used by Tollbooth behind the scenes).
$ go get golang.org/x/time/rate
Then create a demo directory containing two files: limit.go and main.go.
$ mkdir ratelimit-demo
$ cd ratelimit-demo
$ touch limit.go main.go
Let's start by making a global rate limiter which acts on all the requests that a HTTP server receives.
Open up the limit.go file and add the following code:
File: ratelimit-demo/limit.go
package main
import (
"net/http"
"golang.org/x/time/rate"
)
var limiter = rate.NewLimiter(2, 5)
func limit(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if limiter.Allow() == false {
http.Error(w, http.StatusText(429), http.StatusTooManyRequests)
return
}
next.ServeHTTP(w, r)
})
}
In this code we've used the rate.NewLimiter() function to initialize and return a new rate limiter. Its signature looks like this:
A Limiter controls how frequently events are allowed to happen. It implements a "token bucket" of size b, initially full and refilled at rate r tokens per second.
Or to describe it another way – the limiter permits you to consume an average of r tokens per second, with a maximum of b tokens in any single 'burst'. So in the code above our limiter allows 2 tokens to be consumed per second, with a maximum burst size of 5.
In the limit middleware function we call the global limiter's Allow() method each time the middleware receives a HTTP request. If there are no tokens left in the bucket Allow() will return false and we send the user a 429 Too Many Requests response. Otherwise, calling Allow() will consume exactly one token from the bucket and we pass on control to the next handler in the chain.
It's important to note that the code behind the Allow() method is protected by a mutex and is safe for concurrent use.
Let's put this to use. Open up the main.go file and setup a simple web server which uses the limit middleware like so:
File: ratelimit-demo/main.go
package main
import (
"net/http"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", okHandler)
// Wrap the servemux with the limit middleware.
http.ListenAndServe(":4000", limit(mux))
}
func okHandler(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("OK"))
}
Go ahead and run the application…
$ go run *.go
And if you make enough requests in quick succession, you should eventually get a response which looks like this:
$ curl -i localhost:4000
HTTP/1.1 429 Too Many Requests
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Thu, 21 Dec 2017 19:25:52 GMT
Content-Length: 18
Too Many Requests
Rate limiting per user
While having a single, global, rate limiter is useful in some cases, another common scenario is implement a rate limiter per user, based on an identifier like IP address or API key. In this post we'll use IP address as the identifier.
A conceptually straightforward way to do this is to create a map of rate limiters, using the identifier for each user as the map key.
At this point you might think to reach for the new sync.Map type that was introduced in Go 1.9. This essentially provides a concurrency-safe map, designed to be accessed from multiple goroutines without the risk of race conditions. But it comes with a note of caution:
It is optimized for use in concurrent loops with keys that are stable over time, and either few steady-state stores, or stores localized to one goroutine per key.
For use cases that do not share these attributes, it will likely have comparable or worse performance and worse type safety than an ordinary map paired with a read-write mutex.
In our particular use-case the map keys will be the IP address of users, and so new keys will be added to the map each time a new user visits our application. We'll also want to prevent undue memory consumption by removing old entries from the map when a user hasn't been seen for a long period of time.
So in our case the map keys won't be stable and it's likely that an ordinary map protected by a mutex will perform better. (If you're not familiar with the idea of mutexes or how to use them in Go, then this post has an explanation which you might want to read before continuing).
Let's update the limit.go file to contain a basic implementation. I'll keep the code structure deliberately simple.
File: ratelimit-demo/limit.go
package main
import (
"net/http"
"sync"
"golang.org/x/time/rate"
)
// Create a map to hold the rate limiters for each visitor and a mutex.
var visitors = make(map[string]*rate.Limiter)
var mtx sync.Mutex
// Create a new rate limiter and add it to the visitors map, using the
// IP address as the key.
func addVisitor(ip string) *rate.Limiter {
limiter := rate.NewLimiter(2, 5)
mtx.Lock()
visitors[ip] = limiter
mtx.Unlock()
return limiter
}
// Retrieve and return the rate limiter for the current visitor if it
// already exists. Otherwise call the addVisitor function to add a
// new entry to the map.
func getVisitor(ip string) *rate.Limiter {
mtx.Lock()
limiter, exists := visitors[ip]
mtx.Unlock()
if !exists {
return addVisitor(ip)
}
return limiter
}
func limit(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Call the getVisitor function to retreive the rate limiter for
// the current user.
limiter := getVisitor(r.RemoteAddr)
if limiter.Allow() == false {
http.Error(w, http.StatusText(429), http.StatusTooManyRequests)
return
}
next.ServeHTTP(w, r)
})
}
Removing old entries from the map
There's one problem with this: as long as the application is running the visitors map will continue to grow unbounded.
We can fix this fairly simply by recording the last seen time for each visitor and running a background goroutine to deletes old entries from the map (and therefore free up memory as we go).
File: ratelimit-demo/limit.go
package main
import (
"net/http"
"sync"
"time"
"golang.org/x/time/rate"
)
// Create a custom visitor struct which holds the rate limiter for each
// visitor and the last time that the visitor was seen.
type visitor struct {
limiter *rate.Limiter
lastSeen time.Time
}
// Change the the map to hold values of the type visitor.
var visitors = make(map[string]*visitor)
var mtx sync.Mutex
// Run a background goroutine to remove old entries from the visitors map.
func init() {
go cleanupVisitors()
}
func addVisitor(ip string) *rate.Limiter {
limiter := rate.NewLimiter(2, 5)
mtx.Lock()
// Include the current time when creating a new visitor.
visitors[ip] = &visitor{limiter, time.Now()}
mtx.Unlock()
return limiter
}
func getVisitor(ip string) *rate.Limiter {
mtx.Lock()
v, exists := visitors[ip]
if !exists {
mtx.Unlock()
return addVisitor(ip)
}
// Update the last seen time for the visitor.
v.lastSeen = time.Now()
mtx.Unlock()
return v.limiter
}
// Every minute check the map for visitors that haven't been seen for
// more than 3 minutes and delete the entries.
func cleanupVisitors() {
for {
time.Sleep(time.Minute)
mtx.Lock()
for ip, v := range visitors {
if time.Now().Sub(v.lastSeen) > 3*time.Minute {
delete(visitors, ip)
}
}
mtx.Unlock()
}
}
func limit(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
limiter := getVisitor(r.RemoteAddr)
if limiter.Allow() == false {
http.Error(w, http.StatusText(429), http.StatusTooManyRequests)
return
}
next.ServeHTTP(w, r)
})
}
Some more improvements…
For simple applications this code will work fine as-is, but you may want to adapt it further depending on your needs. For example, it might make sense to:
Check the X-Forwarded-For or X-Real-IP headers for the IP address, if you are running your server behind a reverse proxy.
Port the code to a standalone package.
Make the rate limiter and cleanup settings configurable at runtime.
Remove the reliance on global variables, so that different rate limiters can be created with different settings.
Switch to a sync.RWMutex to help reduce contention on the map.
There are a lot of good tutorials which talk about Go's sql.DB type and how to use it to execute SQL database queries and statements. But most of them gloss over the SetMaxOpenConns(), SetMaxIdleConns() and SetConnMaxLifetime() methods — which you can use to configure the behavior of sql.DB and alter its performance.
In this post I'd like to explain exactly what these settings do and demonstrate the (positive and negative) impact that they can have.
Open and idle connections
I'll begin with a little background.
A sql.DB object is a pool of many database connections which contains both 'open' and 'idle' connections. A connection is marked open when you are using it to perform a database task, such as executing a SQL statement or querying rows. When the task is complete the connection becomes idle.
When you instruct sql.DB to perform a database task, it will first check if any idle connections are already available in the pool. If one is available then Go will reuse the existing connection and mark it as open for the duration of the task. If there are no idle connections in the pool when you need one then Go will create a new additional connection and 'open' it.
The SetMaxOpenConns method
By default there's no limit on the number of connections that can be open at the same time. But you can implement your own limit via the SetMaxOpenConns() method like so:
// Initialise a new connection pool
db, err := sql.Open("postgres", "postgres://user:pass@localhost/db")
if err != nil {
log.Fatal(err)
}
// Set the maximum number of concurrently open connections to 5. Setting this
// to less than or equal to 0 will mean there is no maximum limit (which
// is also the default setting).
db.SetMaxOpenConns(5)
In this example code the pool now has a maximum limit of 5 concurrently open connections. If 5 connections are already open and another new connection is needed, then the application will be forced to wait until one of the 5 open connections is freed up and becomes idle.
To illustrate the impact of changing MaxOpenConns I ran a benchmark test with the maximum open connections set to 1, 2, 5, 10 and unlimited. The benchmark executes parallel INSERT statements on a PostgreSQL database and you can find the code in this gist. Here's the results:
For this benchmark we can see that the more open connections that are allowed, the less time is taken to perform the INSERT on the database (3129633 ns/op with 1 open connection compared to 531030 ns/op for unlimited connections — about 6 times quicker). This is because the more open connections that there are, the less time (on average) the benchmark code needs to wait for an open connection to be freed and made idle (ready for use) again.
The SetMaxIdleConns method
By default sql.DB allows a maximum of 2 idle connections to be retained in the connection pool. You can change this via the SetMaxIdleConns() method like so:
// Initialise a new connection pool
db, err := sql.Open("postgres", "postgres://user:pass@localhost/db")
if err != nil {
log.Fatal(err)
}
// Set the maximum number of concurrently idle connections to 5. Setting this
// to less than or equal to 0 will mean that no idle connections are retained.
db.SetMaxIdleConns(5)
In theory, allowing a higher number of idle connections in the pool will improve performance because it makes it less likely that a new connection will need to be established from scratch — therefore helping to save resources.
Lets take a look at the same benchmark with the maximum idle connections is set to none, 1, 2, 5 and 10 (and the number of open connections is unlimited):
When MaxIdleConns is set to none, a new connection has to be created from scratch for each INSERT and we can see from the benchmarks that the average runtime and memory usage is comparatively high.
Allowing just 1 idle connection to be retained and reused makes a massive difference to this particular benchmark — it cuts the average runtime by about 8 times and reduces memory usage by about 20 times. Going on to increase the size of the idle connection pool makes the performance even better, although the improvements are less pronounced.
So we should you maintain a large idle connection pool? The answer is it depends on the application.
It's important to realise that keeping an idle connection alive comes at a cost — it takes up memory which can otherwise be used for both your application and the database.
It's also possible that if a connection is idle for too long then it may become unusable. For example, MySQL's wait_timeout setting will automatically close any connections that haven't been used for 8 hours (by default).
When this happens sql.DB handles it gracefully. Bad connections will automatically be retried twice before giving up, at which point Go will remove the connection from the pool and create a new one. So setting MaxIdleConns too high may actually result in connections becoming unusable and more resources being used than if you had a smaller idle connection pool (with fewer connections that are used more frequently). So really you only want to keep a connection idle if you're likely to be using it again soon.
One last thing to point out is that MaxIdleConns should always be less than or equal to MaxOpenConns. Go enforces this and will automatically reduce MaxIdleConns if necessary. This Stack Overflow post nicely describes why:
There is no point in ever having any more idle connections than the maximum allowed open connections, because if you could instantaneously grab all the allowed open connections, the remain idle connections would always remain idle. It's like having a bridge with four lanes, but only ever allowing three vehicles to drive across it at once.
The SetConnMaxLifetime method
Let's now take a look at the SetConnMaxLifetime() method which sets the maximum length of time that a connection can be reused for. This can be useful if your SQL database also implements a maximum connection lifetime or if — for example — you want to facilitate gracefully swapping databases behind a load balancer.
You use it like this:
// Initialise a new connection pool
db, err := sql.Open("postgres", "postgres://user:pass@localhost/db")
if err != nil {
log.Fatal(err)
}
// Set the maximum lifetime of a connection to 1 hour. Setting it to 0
// means that there is no maximum lifetime and the connection is reused
// forever (which is the default behavior).
db.SetConnMaxLifetime(time.Hour)
In this example all our connections will 'expire' 1 hour after they were first created, and cannot be reused after they've expired. But note:
This doesn't guarantee that a connection will exist in the pool for a whole hour; it's quite possible that the connection will have become unusable for some reason and been automatically closed before then.
A connection can still be in use more than one hour after being created — it just cannot start to be reused after that time.
This isn't an idle timeout. The connection will expire 1 hour after it was first created — not 1 hour after it last became idle.
Once every second a cleanup operation is automatically run to remove 'expired' connections from the pool.
In theory, the shorter ConnMaxLifetime is the more often connections will expire — and consequently — the more often they will need to be created from scratch.
To illustrate this I ran the benchmarks with ConnMaxLifetime set to 100ms, 200ms, 500ms, 1000ms and unlimited (reused forever), with the default settings of unlimited open connections and 2 idle connections. These time periods are obviously much, much shorter than you'd use in most applications but they help illustrate the behaviour well.
In these particular benchmarks we can see that memory usage was more than 3 times greater with a 100ms lifetime compared to an unlimited lifetime, and the average runtime for each INSERT was also slightly longer.
Exceeding connection limits
Lastly, this article wouldn't be complete without mentioning what happens if you exceed a hard limit on the number of database connections.
As an illustration, I'll change my postgresql.conf file so only a total of 5 connections are permitted (the default is 100)...
max_connections = 5
And then rerun the benchmark test with unlimited open connections...
BenchmarkMaxOpenConnsUnlimited-8 --- FAIL: BenchmarkMaxOpenConnsUnlimited-8
main_test.go:14: pq: sorry, too many clients already
main_test.go:14: pq: sorry, too many clients already
main_test.go:14: pq: sorry, too many clients already
FAIL
As soon as the hard limit of 5 connections is hit my database driver (pq) immediately returns a sorry, too many clients already error message instead of completing the INSERT.
To prevent this error we need to set the total maximum of open and idle connections in sql.DB to comfortably below 5. Like so:
// Initialise a new connection pool
db, err := sql.Open("postgres", "postgres://user:pass@localhost/db")
if err != nil {
log.Fatal(err)
}
// Set the number of open and idle connection to a maximum total of 3.
db.SetMaxOpenConns(2)
db.SetMaxIdleConns(1)
Now there will only ever be a maximum of 3 connections created by sql.DB and the benchmark should run without any errors.
But doing this comes with a big caveat: when the open connection limit is reached, any new database tasks that your application needs to execute will be forced to wait until a connection becomes free.
For some applications that behavior might be fine, but for others it might not. For example, in a web application it could arguably be better to immediately log the error message and send a 500 Internal Server Error to the user, rather than having their HTTP request hang and potentially timeout while waiting for a free connection.
In summary
For most applications using SetMaxOpenConns()to limit the maximum of open connections will have a negative impact on performance, but it may make sense to do so if your database is operating in a particularly resource-constrained environment.
If your application is bursty or regularly executing more than 2 database tasks concurrently then there will probably be a positive performance impact from increasing the size of the idle connection pool via SetMaxIdleConns(). But be aware that making it too large may be counter-productive. Don't go crazy with it, and benchmarking real-world load is probably a good idea.
For most applications implementing a connection lifetime limit via SetConnMaxLifetime() will have a negative impact on performance. But if your database itself enforces a short connection lifetime then it makes sense to match this in sql.DB to avoid the overhead of trying-and-retrying bad connections.
If you prefer for your application to wait for a connection to be freed up (rather than return an error) when your database's hard limit of connections is reached then you should explicitly set both SetMaxOpenConns() and SetMaxIdleConns().
A nice feature of Go's http.FileServer is that it automatically generates navigable directory listings, which look at bit like this:
But for certain applications you might want to prevent this behavior and disable directory listings altogether. In this post I’m going to run through three different options for doing exactly that:
Before http.FileServer generates a directory listing it checks for the existence of an index.html file in the directory root. If an index.html file exists, then it will respond with the contents of the file instead.
So it follows that a simple way to disable directory listings is to add a blank index.html file to your root static file directory and all sub-directories, like so:
This is a good-enough solution if you can't (or don't want to) make any changes to your Go application itself.
But it's not perfect. You'll need to remember to add a blank index.html file for any new sub-directories in the future, and many people — myself included — would argue that a 403 Forbidden or 404 Not Found status would be more appropriate than sending the user an empty 200 OK response.
Using middleware
Both of these imperfections can be addressed if we take a different approach and implement some custom middleware to intercept requests before they reach the http.FileServer.
Essentially, we want the middleware to check if the request URL ends with a / character, and if it does, return a 404 Not Found response instead of passing on the request to the http.FileServer. Here's a basic implementation:
This approach would result in a user getting responses like these:
$ curl -i http://localhost:4000/static/img/
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 13 Mar 2018 12:46:20 GMT
Content-Length: 19
404 page not found
$ curl -i http://localhost:4000/static/img
HTTP/1.1 301 Moved Permanently
Location: /static/img/
Date: Tue, 13 Mar 2018 12:46:55 GMT
Content-Length: 43
Content-Type: text/html; charset=utf-8
<a href="/static/img/">Moved Permanently</a>.
To me, this feels like a cleaner and easier-to-maintain way to disable directory listings than using blank index.html files. But again, it's still not perfect.
Firstly, requests for any directories without the trailing slash will be 301 redirected only to receive a 404 Not Found response. It's extra, unnecessary, requests for both the client and server to deal with.
Secondly, if one of your directories does contain an index.html file then it won't ever be used. For example, if you have the directory structure...
... any request to http://localhost:4000/static/css/ will result in a 404 Not Found response instead of returning the contents of the /static/css/index.html file.
$ curl -i http://localhost:4000/static/css/
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 13 Mar 2018 12:51:09 GMT
Content-Length: 19
404 page not found
Using a custom filesystem
The final option we're going to look at is creating a custom filesystem and passing that to your http.FileServer.
There are a couple of approaches described by Brad Fitzpatrick and George Armhold you might want to consider, but I would personally suggest doing something like this:
In this code we're creating a custom neuteredFileSystem type which embeds the standard http.FileSystem. We then implement an Open() method on it — which gets called each time our http.FileServer receives a request.
In our Open() method we Stat() the requested file path and use the IsDir() method to check whether it's a directory or not. If it is a directory we then try to Open() any index.html file in it. If no index.html file exists a os.ErrNotExist error will be returned (which in turn will be transformed into a 404 Not Found response by http.Fileserver). Otherwise, we just return the file and let http.FileServer do its thing.
Putting this to use with the directory structure...
$ curl -i http://localhost:4000/static/img/
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 13 Mar 2018 16:53:21 GMT
Content-Length: 19
404 page not found
$ curl -i http://localhost:4000/static/img
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 13 Mar 2018 16:53:22 GMT
Content-Length: 19
404 page not found
$ curl -i http://localhost:4000/static/css/
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 37
Content-Type: text/html; charset=utf-8
Last-Modified: Tue, 13 Mar 2018 12:49:00 GMT
Date: Tue, 13 Mar 2018 16:53:27 GMT
<h1>This is my custom index page</h1>
This is now working pretty nicely:
All requests for directories (with no index.html file) return a 404 Not Found response, instead of a directory listing or a redirect. This works for requests both with and without a trailing slash.
The default behavior of http.FileServer isn't changed any other way, and index.html files work as per the standard library documentation.
Earlier this year AWS announced that their Lambda service would now be providing first-class support for the Go language, which is a great step forward for any gophers (like myself) who fancy experimenting with serverless technology.
So in this post I'm going to talk through how to create a HTTPS API backed by AWS Lambda, building it up step-by-step. I found there to be quite a few gotchas in the process — especially if you're not familiar the AWS permissions system — and some rough edges in the way that Lamdba interfaces with the other AWS services. But once you get your head around these it works pretty well.
There's a lot of content to cover in this tutorial, so I've broken it down into the following seven steps:
I'm keeping the API deliberately simple to avoid getting bogged-down in application-specific code, but once you've grasped the basics it's fairly clear how to extend the API to support additional routes and actions.
Setting up the AWS CLI
Throughout this tutorial we'll use the AWS CLI (command line interface) to configure our lambda functions and other AWS services. Installation and basic usage instructions can be found here, but if you’re using a Debian-based system like Ubuntu you can install the CLI with apt and run it using the aws command:
Next we need to set up an AWS IAM user with programmatic access permission for the CLI to use. A guide on how to do this can be found here. For testing purposes you can attach the all-powerful AdministratorAccess managed policy to this user, but in practice I would recommend using a more restrictive policy. At the end of setting up the user you'll be given a access key ID and secret access key. Make a note of these — you’ll need them in the next step.
Configure the CLI to use the credentials of the IAM user you've just created using the configure command. You’ll also need to specify the default region and output format you want the CLI to use.
$ aws configure
AWS Access Key ID [None]: access-key-ID
AWS Secret Access Key [None]: secret-access-key
Default region name [None]: us-east-1
Default output format [None]: json
(Throughout this tutorial I'll assume you're using the us-east-1 region — you'll need to change the code snippets accordingly if you're using a different region.)
Creating and deploying an Lambda function
Now for the exciting part: making a lambda function. If you're following along, go to your $GOPATH/src folder and create a books repository containing a main.go file.
$ cd ~/go/src
$ mkdir books && cd books
$ touch main.go
Next you'll need to install the github.com/aws-lambda-go/lambda package. This provides the essential libraries and types we need for creating a lambda function in Go.
$ go get github.com/aws/aws-lambda-go/lambda
Then open up the main.go file and add the following code:
File: books/main.go
package main
import (
"github.com/aws/aws-lambda-go/lambda"
)
type book struct {
ISBN string `json:"isbn"`
Title string `json:"title"`
Author string `json:"author"`
}
func show() (*book, error) {
bk := &book{
ISBN: "978-1420931693",
Title: "The Republic",
Author: "Plato",
}
return bk, nil
}
func main() {
lambda.Start(show)
}
In the main() function we call lambda.Start() and pass in the show function as the lambda handler. In this case the handler simply initializes and returns a new book object.
Lamdba handlers can take a variety of different signatures and reflection is used to determine exactly which signature you're using. The full list of supported forms is…
… where the TIn and TOut parameters are objects that can be marshaled (and unmarshalled) by Go's encoding/json package.
The next step is to build an executable from the books package using go build. In the code snippet below I'm using the -o flag to save the executable to /tmp/main but you can save it to any location (and name it whatever) you wish.
$ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books
Important: as part of this command we're using env to temporarily set two environment variables for the duration for the command (GOOS=linux and GOARCH=amd64). These instruct the Go compiler to create an executable suitable for use with a linux OS and amd64 architecture — which is what it will be running on when we deploy it to AWS.
AWS requires us to upload our lambda functions in a zip file, so let's make a main.zip zip file containing the executable we just made:
$ zip -j /tmp/main.zip /tmp/main
Note that the executable must be in the root of the zip file — not in a folder within the zip file. To ensure this I've used the -j flag in the snippet above to junk directory names.
The next step is a bit awkward, but critical to getting our lambda function working properly. We need to set up an IAM role which defines the permission that our lambda function will have when it is running.
For now let's set up a lambda-books-executor role and attach the AWSLambdaBasicExecutionRole managed policy to it. This will give our lambda function the basic permissions it need to run and log to the AWS cloudwatch service.
First we have to create a trust policy JSON file. This will essentially instruct AWS to allow lambda services to assume the lambda-books-executor role:
Make a note of the returned ARN (Amazon Resource Name) — you'll need this in the next step.
Now the lambda-books-executor role has been created we need to specify the permissions that the role has. The easiest way to do this it to use the aws iam attach-role-policy command, passing in the ARN of AWSLambdaBasicExecutionRole permission policy like so:
$ aws iam attach-role-policy --role-name lambda-books-executor \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Note: you can find a list of other permission policies that might be useful here.
Now we're ready to actually deploy the lambda function to AWS, which we can do using the aws lambda create-function command. This takes the following flags and can take a minute or two to run.
--function-name
Thethat name your lambda function will be called within AWS
--runtime
The runtime environment for the lambda function (in our case "go1.x")
--role
The ARN of the role you want the lambda function to assume when it is running (from step 6 above)
--handler
The name of the executable in the root of the zip file
So there it is. Our lambda function has been deployed and is now ready to use. You can try it out by using the aws lambda invoke command (which requires you to specify an output file for the response — I've used /tmp/output.json in the snippet below).
If you're following along hopefully you've got the same response. Notice how the book object we initialized in our Go code has been automatically marshaled to JSON?
Hooking it up to DynamoDB
In this section we're going to add a persistence layer for our data which can be accessed by our lambda function. For this I'll use Amazon DynamoDB (it integrates nicely with AWS lambda and has a generous free-usage tier). If you're not familiar with DynamoDB, there's a decent run down of the basics here.
The first thing we need to do is create a Books table to hold the book records. DynanmoDB is schema-less, but we do need to define the partion key (a bit like a primary key) on the ISBN field. We can do this in one command like so:
The next thing to do is update our Go code so that our lambda handler can connect to and use the DynamoDB layer. For this you'll need to install the github.com/aws/aws-sdk-go package which provides libraries for working with DynamoDB (and other AWS services).
$ go get github.com/aws/aws-sdk-go
Now for the code. To keep a bit of separation create a new db.go file in your books repository:
$ touch ~/go/src/books/db.go
And add the following code:
File: books/db.go
package main
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
)
// Declare a new DynamoDB instance. Note that this is safe for concurrent
// use.
var db = dynamodb.New(session.New(), aws.NewConfig().WithRegion("us-east-1"))
func getItem(isbn string) (*book, error) {
// Prepare the input for the query.
input := &dynamodb.GetItemInput{
TableName: aws.String("Books"),
Key: map[string]*dynamodb.AttributeValue{
"ISBN": {
S: aws.String(isbn),
},
},
}
// Retrieve the item from DynamoDB. If no matching item is found
// return nil.
result, err := db.GetItem(input)
if err != nil {
return nil, err
}
if result.Item == nil {
return nil, nil
}
// The result.Item object returned has the underlying type
// map[string]*AttributeValue. We can use the UnmarshalMap helper
// to parse this straight into the fields of a struct. Note:
// UnmarshalListOfMaps also exists if you are working with multiple
// items.
bk := new(book)
err = dynamodbattribute.UnmarshalMap(result.Item, bk)
if err != nil {
return nil, err
}
return bk, nil
}
And then update the main.go to use this new code:
File: books/main.go
package main
import (
"github.com/aws/aws-lambda-go/lambda"
)
type book struct {
ISBN string `json:"isbn"`
Title string `json:"title"`
Author string `json:"author"`
}
func show() (*book, error) {
// Fetch a specific book record from the DynamoDB database. We'll
// make this more dynamic in the next section.
bk, err := getItem("978-0486298238")
if err != nil {
return nil, err
}
return bk, nil
}
func main() {
lambda.Start(show)
}
Save the files, then rebuild and zip up the lambda function so it's ready to deploy:
$ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books
$ zip -j /tmp/main.zip /tmp/main
Re-deploying a lambda function is easier than creating it for the first time — we can use the aws lambda update-function-code command like so:
$ aws lambda invoke --function-name books /tmp/output.json
{
"StatusCode": 200,
"FunctionError": "Unhandled"
}
$ cat /tmp/output.json
{"errorMessage":"AccessDeniedException: User: arn:aws:sts::account-id:assumed-role/lambda-books-executor/books is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:us-east-1:account-id:table/Books\n\tstatus code: 400, request id: 2QSB5UUST6F0R3UDSVVVODTES3VV4KQNSO5AEMVJF66Q9ASUAAJG","errorType":"requestError"}
Ah. There's a slight problem. We can see from the output message that our lambda function (specifically, the lambda-books-executor role) doesn't have the necessary permissions to run GetItem on a DynamoDB instance. Let's fix that now.
Create a privilege policy file that gives GetItem and PutItem privileges on DynamoDB like so:
As a side note, AWS has some managed policies called AWSLambdaDynamoDBExecutionRole and AWSLambdaInvocation-DynamoDB which sound like they would do the trick. But neither of them actually provide GetItem or PutItem privileges. Hence the need to roll our own policy.
Let's try executing the lambda function again. It should work smoothly this time and return information about the book with ISBN 978-0486298238.
So our lambda function is now working nicely and communicating with DynamoDB. The next thing to do is set up a way to access the lamdba function over HTTPS, which we can do using the AWS API Gateway service.
But before we go any further, it's worth taking a moment to think about the structure of our project. Let's say we have grand plans for our lamdba function to be part of a bigger bookstore API which deals with information about books, customers, recommendations and other things.
There's three basic options for structuring this using AWS Lambda:
Microservice style — Each lambda function is responsible for one action only. For example, there are 3 separate lambda functions for showing, creating and deleting a book.
Service style — Each lambda function is responsible for a group of related actions. For example, one lambda function handles all book-related actions, but customer-related actions are kept in a separate lambda function.
Monolith style — One lambda function manages all the bookstore actions.
Each of these options is valid, and theres some good discussion of the pros and cons here.
For this tutorial we'll opt for a service style, and have one books lambda function handle the different book-related actions. This means that we'll need to implement some form of routing within our lambda function, which I'll cover later in the post. But for now…
Go ahead and create a bookstore API using the aws apigateway create-rest-api command like so:
Again, keep a note of the root-path-id value this returns.
Now we need to create a new resource under the root path — specifically a resource for the URL path /books. We can do this by using the aws apigateway create-resource command with the --path-part parameter like so:
Again, note the resource-id this returns, we'll need it in the next step.
Note that it's possible to include placeholders within your path by wrapping part of the path in curly braces. For example, a --path-part parameter of books/{id} would match requests to /books/foo and /books/bar, and the value of id would be made available to your lambda function via an events object (which we'll cover later in the post). You can also make a placeholder greedy by postfixing it with a +. A common idiom is to use the parameter --path-part {proxy+} if you want to match all requests regardless of their path.
But we're not doing either of those things. Let's get back to our /books resource and use the aws apigateway put-method command to register the HTTP method of ANY. This will mean that our /books resource will respond to all requests regardless of their HTTP method.
Now we're all set to integrate the resource with our lambda function, which we can do using the aws apigateway put-integration command. This command has a few parameters that need a quick explanation:
The --type parameter should be AWS_PROXY. When this is used the AWS API Gateway will send information about the HTTP request as an 'event' to the lambda function. It will also automatically transform the output from the lambda function to a HTTP response.
The --integration-http-method parameter must be POST. Don't confuse this with what HTTP methods your API resource responds to.
Note that the --statement-id parameter needs to be a globally unique identifier. This could be a random ID or something more descriptive.
Alright, let's try again:
$ aws apigateway test-invoke-method --rest-api-id rest-api-id --resource-id resource-id --http-method "GET"
{
"status": 502,
"body": "{\"message\": \"Internal server error\"}",
"headers": {},
"log": "Execution log for request test-request\nThu Apr 05 11:12:53 UTC 2018 : Starting execution for request: test-invoke-request\nThu Apr 05 11:12:53 UTC 2018 : HTTP Method: GET, Resource Path: /books\nThu Apr 05 11:12:53 UTC 2018 : Method request path: {}\nThu Apr 05 11:12:53 UTC 2018 : Method request query string: {}\nThu Apr 05 11:12:53 UTC 2018 : Method request headers: {}\nThu Apr 05 11:12:53 UTC 2018 : Endpoint response body before transformations: {\"isbn\":\"978-0486298238\",\"title\":\"Meditations\",\"author\":\"Marcus Aurelius\"}\nThu Apr 05 11:12:53 UTC 2018 : Endpoint response headers: {X-Amz-Executed-Version=$LATEST, x-amzn-Remapped-Content-Length=0, Connection=keep-alive, x-amzn-RequestId=48d29098-38c2-11e8-ae15-f13b670c5483, Content-Length=74, Date=Thu, 05 Apr 2018 11:12:53 GMT, X-Amzn-Trace-Id=root=1-5ac604b5-cf29dd70cd08358f89853b96;sampled=0, Content-Type=application/json}\nThu Apr 05 11:12:53 UTC 2018 : Execution failed due to configuration error: Malformed Lambda proxy response\nThu Apr 05 11:12:53 UTC 2018 : Method completed with status: 502\n",
"latency": 211
}
So unfortunately there's still an error, but the message has now changed:
Execution failed due to configuration error: Malformed Lambda proxy response
And if you look closely at the output you'll see the information:
Endpoint response body before transformations: {\"isbn\":\"978-0486298238\",\"title\":\"Meditations\",\"author\":\"Marcus Aurelius\"}
So there's some definite progress here. Our API is talking to our lambda function and is receiving the correct response (a book object marshalled to JSON). It's just that the AWS API Gateway considers the response to be in the wrong format.
This is because, when you're using the API Gateway's lambda proxy integration, the return value from the lambda function must be in the following JSON format:
So to fix this it's time to head back to our Go code and make some alterations.
Working with events
The easiest way to provide the responses that the AWS API Gateway needs is to install the github.com/aws/aws-lambda-go/events package:
go get github.com/aws/aws-lambda-go/events
This provides a couple of useful types (APIGatewayProxyRequest and APIGatewayProxyResponse) which contain information about incoming HTTP requests and allow us to construct responses that the API Gateway understands.
type APIGatewayProxyRequest struct {
Resource string `json:"resource"` // The resource path defined in API Gateway
Path string `json:"path"` // The url path for the caller
HTTPMethod string `json:"httpMethod"`
Headers map[string]string `json:"headers"`
QueryStringParameters map[string]string `json:"queryStringParameters"`
PathParameters map[string]string `json:"pathParameters"`
StageVariables map[string]string `json:"stageVariables"`
RequestContext APIGatewayProxyRequestContext `json:"requestContext"`
Body string `json:"body"`
IsBase64Encoded bool `json:"isBase64Encoded,omitempty"`
}
type APIGatewayProxyResponse struct {
StatusCode int `json:"statusCode"`
Headers map[string]string `json:"headers"`
Body string `json:"body"`
IsBase64Encoded bool `json:"isBase64Encoded,omitempty"`
}
Let's go back to our main.go file and update our lambda handler so that it uses the signature:
Essentially, the handler will accept a APIGatewayProxyRequest object which contains a bunch of information about the HTTP request, and return a APIGatewayProxyResponse object (which is marshalable into a JSON response suitable for the AWS API Gateway).
File: books/main.go
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
"os"
"regexp"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
)
var isbnRegexp = regexp.MustCompile(`[0-9]{3}\-[0-9]{10}`)
var errorLogger = log.New(os.Stderr, "ERROR ", log.Llongfile)
type book struct {
ISBN string `json:"isbn"`
Title string `json:"title"`
Author string `json:"author"`
}
func show(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
// Get the `isbn` query string parameter from the request and
// validate it.
isbn := req.QueryStringParameters["isbn"]
if !isbnRegexp.MatchString(isbn) {
return clientError(http.StatusBadRequest)
}
// Fetch the book record from the database based on the isbn value.
bk, err := getItem(isbn)
if err != nil {
return serverError(err)
}
if bk == nil {
return clientError(http.StatusNotFound)
}
// The APIGatewayProxyResponse.Body field needs to be a string, so
// we marshal the book record into JSON.
js, err := json.Marshal(bk)
if err != nil {
return serverError(err)
}
// Return a response with a 200 OK status and the JSON book record
// as the body.
return events.APIGatewayProxyResponse{
StatusCode: http.StatusOK,
Body: string(js),
}, nil
}
// Add a helper for handling errors. This logs any error to os.Stderr
// and returns a 500 Internal Server Error response that the AWS API
// Gateway understands.
func serverError(err error) (events.APIGatewayProxyResponse, error) {
errorLogger.Println(err.Error())
return events.APIGatewayProxyResponse{
StatusCode: http.StatusInternalServerError,
Body: http.StatusText(http.StatusInternalServerError),
}, nil
}
// Similarly add a helper for send responses relating to client errors.
func clientError(status int) (events.APIGatewayProxyResponse, error) {
return events.APIGatewayProxyResponse{
StatusCode: status,
Body: http.StatusText(status),
}, nil
}
func main() {
lambda.Start(show)
}
Notice how in all cases the error value returned from our lambda handler is now nil? We have to do this because the API Gateway doesn't accept error objects when you're using it in conjunction with a lambda proxy integration (they would result in a 'malformed response' errors again). So we need to manage errors fully within our lambda function and return the appropriate HTTP response. In essence, this means that the return parameter of error is superfluous, but we still need to include it to have a valid signature for the lambda function.
Anyway, save the file and rebuild and redeploy the lambda function:
As a side note, anything sent to os.Stderr will be logged to the AWS Cloudwatch service. So if you've set up an error logger like we have in the code above, you can query Cloudwatch for errors like so:
Go ahead and give it a try using curl. It should work as you expect:
$ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=978-1420931693
{"isbn":"978-1420931693","title":"The Republic","author":"Plato"}
$ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=foobar
Bad Request
Supporting multiple actions
Let's add support for a POST /books action. We want this to read and validate a new book record (from a JSON HTTP request body) and then add it to the DynamoDB table.
Now that the different AWS services are hooked up, extending our lambda function to support additional actions is perhaps the most straightforward part of this tutorial, as it can be managed purely within our Go code.
First update the db.go file to include a new putItem function like so:
And then update the main.go function so that the lambda.Start() method calls a new router function, which does a switch on the HTTP request method to determine which action to take. Like so:
