Building a Form Validation and Processing API in Go

Sun, Dec 1, 2019

head

At Lakesite.Net, we love using JAMStack (Javascript APIs and Markup) because it simplifies the developer experience, allows for a performant and scalable site, at an overall lower cost. Business web sites typically require marketing “who we are,” “what we do,” “why should you pay us to do it?” style information. This doesn’t require any programming, but does require markup, style and content. Removing complexity is desirable. However, you’ll likely want to process contact forms.

There are APIs and services which will process contact forms, such as Mailchimp. So, other than developing a micro-service API for fun, why should we do this?

  1. You want to run your own API for a JAMStack without relying on others.
  2. You want to retain a copy of contact information without relying on others.
  3. You presume to be more responsible with your customer’s data than a third party.
  4. You want to use an open source solution.
  5. You’re interested in providing your own API service.

Let’s create our own API. It should provide an endpoint to submit contact form data. The endpoint will validate form fields, save the data to a database, and e-mail the contact information. We want to accept a confirmation or thanks page to redirect to after valid form submission, and receive messages if form fields are invalid, so we can let users know if, for example, their email is invalid.

I. Dependencies

The following dependencies will be used in our project to simplify the process. We want to minimize the amount of code we actually write and keep our API clean, concise, well documented and tested.

  1. The Gorilla web toolkit’s schema package will handle filling our contact structure with form values.
  2. Alex Saskevich’s govalidator will handle validating the contact structure.
  3. ls-fibre will handle the web API service.
  4. ls-config will handle configuration settings.
  5. Cobra commander will be used to handle command line arguments (path to configuration file, versioning, etc).
  6. Thomas Pelletier’s go-toml will be used to process our configuration file.

II. Structure

Let’s call our project zefram, for Zefram Cochrane, from Star Trek’s First Contact. Our eventual file structure will look like the following:

zefram/
    bin/          - Executable and bundled files from 'make'.
    cmd/root.go   - Cobra commander command line handler.
    docs/         - Documentation.
    pkg/
      api/        - API routes and handlers.
      mail/       - Mail convenience package.
      manager/    - Main manager package.
      models/     - Database related structures and models.
    config.toml   - Application configuration.
    main.go       - Application entry point.
    Makefile      - Makefile to build app
    README.md     - Basic README with references to documentation and usage.

The full source is available and MIT licensed. We’ll provide an overview of how things fit together in a general way, and then review how we perform validation, save the data, and e-mail the contact.

III. Boilerplate

Let’s dive right in with a summary of how we get to the actual work.

zefram/main.go references cmd.Execute:

1func main() {
2	cmd.Execute()
3}

The root command for the project is defined as such:

zefram/cmd/root.go:

 1rootCmd = &cobra.Command{
 2  Use:   "zefram -c [config.toml]",
 3  Short: "run zefram",
 4  Long:  `run zefram with config.toml as a daemon`,
 5  Run: func(cmd *cobra.Command, args []string) {
 6    ms := &manager.ManagerService{}
 7    if config == "" {
 8      config = "config.toml"
 9    }
10    ms.Init(config)
11    ms.Daemonize()
12  },
13}

We create a new manager service, which provides an interface between the command line and the web service API. We require a configuration file, by default config.toml, then initialize the application with any configuration options we’ve provided. We’ll cover some of the initialization logic later. Finally we tell the manager service to daemonize the app so it runs continuously and processes requests.

The manager’s Daemonize function determines the host and IP for the service to listen on. We create a new API instance with an ls-fibre WebService, an app configuration, and a database configuration (and connection). Next we setup routes for our application. Finally, we run the web service.

zefram/pkg/manager/manager.go:

 1// Daemonize sets up the web service and defines routes for the API.
 2func (ms *ManagerService) Daemonize() {
 3        address := config.Getenv("ZEFRAM_HOST", "127.0.0.1") + ":" + config.Getenv("ZEFRAM_PORT", "7990")
 4				ws := service.NewWebService("zefram", address)
 5				api := api.NewAPI(
 6					ws,           // web service
 7					ms.Config,    // app configuration
 8					ms.DBConfig,  // database connection and configuration
 9				)
10        api.SetupRoutes()
11        api.WebService.RunWebServer()
12}

The api package has a function to setup routes. The route we care about will be posting data to /contact, which we define:

zefram/pkg/api/routes.go:

1// SetupRoutes defines and associates routes to handlers.
2func (api *API) SetupRoutes() {
3        api.WebService.Router.HandleFunc("/zefram/api/v1/contact/", api.ContactHandler).Methods("POST")
4}

Our ContactHandler should accept POST requests which contain:

  1. First name
  2. Last name
  3. Email
  4. Message
  5. Add to mailing list?

Email should be required and must be a valid email address. The other fields are optional and require no further validation. We should return a json response for invalid email or missing email field information, otherwise, we should return a status ‘contact made’. In either case we should return 200 OK, unless we have an internal error.

Our contact model handles the data:

zefram/pkg/models/contact.go

1// Contact holds the minimal data for a web contact.
2type Contact struct {
3  gorm.Model
4  FirstName string
5  LastName string
6  Email string `gorm:"type:varchar(100);"valid:"email"`
7  Message string
8  AddMailing bool `gorm:"default:false;"`
9}

We also initialize our DB (presently using sqlite3) and run the necessary migration to create a contact table. The Migrate command is called before we Daemonize, when we call Init with our app configuration. Init parses our config file and initializes our app, zefram, which populates a DBConfig structure and calls Migrate:

zefram/pkg/models/dbconfig.go:

 1func (db *DBConfig) Init() {
 2	if db.Driver == "sqlite3" {
 3		db.Connection, _ = gorm.Open("sqlite3", db.Path)
 4	} else {
 5		// handle connections with other drivers
 6	}
 7}
 8
 9func (db *DBConfig) Migrate() {
10	if db.Connection != nil {
11		db.Connection.AutoMigrate(&Contact{})
12	}
13}

For now we’ll focus on a simple sqlite3 database which only requires a driver type of sqlite3 and a path to the database. Our config.toml is simple:

zefram/config.toml:

1[zefram]
2apikey   = "secretkey"
3dbdriver = "sqlite3"
4dbpath   = "zefram.db"
5mailto   = "andy"
6mailfrom = "zefram@lakesite.net"

IV. Handling Contacts

Our API’s ContactHandler needs to do a few things.

First, we parse the form data. If we can’t, we return an error with a Bad Request status code.

zefram/pkg/api/api.go: ContactHandler breakdown

// parse the form
err := r.ParseForm()
if err != nil {
  api.WebService.JsonStatusResponse(w, "Error parsing form data.", http.StatusBadRequest)
}

Now we need to map the form data into our contact structure, using Gorilla’s scheme library and decoder:

// create a new contact
c := new(model.Contact)

// using a new decoder, decode the form and bind it to the contact
decoder := schema.NewDecoder()
decoder.Decode(c, r.Form)

Next we need to validate the structure, using govalidator. The only field we really care about being proper is the contact email address.

// validate the structure:
_, err = valid.ValidateStruct(c)
if err != nil {
  api.WebService.JsonStatusResponse(w, fmt.Sprintf("Error: %s", err.Error()), http.StatusBadRequest)
  return
}

With our form data bound to a structure and validated, we can proceed to save the structure to our database using gorm. At this point, if we do encounter any errors, we should return an Internal Server Error response.

// insert the contact structure
if dbc := api.DBConfig["zefram"].Connection.Create(c); dbc.Error != nil {
  api.WebService.JsonStatusResponse(w, fmt.Sprintf("Error: %s", dbc.Error.Error()), http.StatusInternalServerError)
  return
}

Next we’ll send an e-mail per our configuration.

// send an email
mailfrom, _ := api.GetAppProperty("zefram", "mailfrom")
mailto, _ := api.GetAppProperty("zefram", "mailto")
subject := "Contact from: " + c.Email
body := "First Name: " + c.FirstName + "\nLast Name: " + c.LastName + "\nMessage: \n\n" + c.Message
err = mail.LocalhostSendMail(mailfrom, mailto, subject, body)

if err != nil {
  api.WebService.JsonStatusResponse(w, fmt.Sprintf("Error: %s", err.Error()), http.StatusInternalServerError)
  return
}

Finally, we return http.StatusOK and a Contact made message:

// Return StatusOK with Contact made:
api.WebService.JsonStatusResponse(w, "Contact made", http.StatusOK)

zefram/pkg/api/api.go ContactHandler in full:

 1// ContactHandler handles POST data for a Contact.
 2func (api *API) ContactHandler(w http.ResponseWriter, r *http.Request) {
 3	// parse the form
 4	err := r.ParseForm()
 5	if err != nil {
 6		api.WebService.JsonStatusResponse(w, "Error parsing form data.", http.StatusBadRequest)
 7		return
 8	}
 9
10	// create a new contact
11	c := new(model.Contact)
12
13	// using a new decoder, decode the form and bind it to the contact
14	decoder := schema.NewDecoder()
15	decoder.Decode(c, r.Form)
16
17	// validate the structure:
18	_, err = valid.ValidateStruct(c)
19	if err != nil {
20		api.WebService.JsonStatusResponse(w, fmt.Sprintf("Error: %s", err.Error()), http.StatusBadRequest)
21		return
22	}
23
24	// insert the contact structure
25	if dbc := api.DBConfig["zefram"].Connection.Create(c); dbc.Error != nil {
26		api.WebService.JsonStatusResponse(w, fmt.Sprintf("Error: %s", dbc.Error.Error()), http.StatusInternalServerError)
27		return
28	}
29
30	// send an email
31	mailfrom, _ := api.GetAppProperty("zefram", "mailfrom")
32	mailto, _ := api.GetAppProperty("zefram", "mailto")
33	subject := "Contact from: " + c.Email
34	body := "First Name: " + c.FirstName + "\nLast Name: " + c.LastName + "\nMessage: \n\n" + c.Message
35	err = mail.LocalhostSendMail(mailfrom, mailto, subject, body)
36
37	if err != nil {
38		api.WebService.JsonStatusResponse(w, fmt.Sprintf("Error: %s", err.Error()), http.StatusInternalServerError)
39		return
40	}
41
42	// Return StatusOK with Contact made:
43	api.WebService.JsonStatusResponse(w, "Contact made", http.StatusOK)
44}

V. Testing Form Submission

Using curl, we can submit some data to our /contact/ endpoint to test:

    $ curl -d "email=test@notvalid&message=This%20is%20a%20test" -X POST http://localhost:7990/api/zefram/v1/contact/

We get an expected bad response, because test@notvalid is not a valid message:

    $ curl -d "email=test@notvalid&message=This%20is%20a%20test" -X POST http://localhost:7990/zefram/api/v1/contact/
    "Error: Email: test@notvalid does not validate as email"
    $

Now if we use a valid e-mail:

    $ curl -d "email=test@lakesite.net&message=This%20is%20a%20test" -X POST http://localhost:7990/zefram/api/v1/contact/
    "Contact made"
    $

Note: our config.toml has a mailto value of andy, which is not valid e-mail. For testing purposes, the validation for mailto was temporarily removed, so we can deliver mail to a local user.

VI. Further work

Zefram requires further work to support mysql and postgresql, and further API endpoints to support export and import (in JSON format) of data. Zefram should handle different contact points, for example an inquiry about a specific product or service.

Zefram currently assumes you’re sending mail from localhost, which is fine if you have a system configured for this. You may instead have an ephemeral cloud instance which probably won’t be accepted by Google or other mail services that require, at a bare minimum, forward and reverse DNS to match. So we’ll want to use another function to send authenticated mail. This should be broken out into another library.

Other code used in zefram should be further abstracted. We really only care about the implementation details in api.go, where we handle contact and inquiries.

We can generalize our manager service and have it handle the boiler plate work of:

  1. Accept a project configuration.
  2. Setup app configurations per project.
  3. Setup a web service configuration, using config and environment variables.
  4. Setup a database service if desired and run migrations (if required).
  5. Setup routes for the web service configuration using an API configuration.
  6. Run the web service if desired.

We can extend ls-config to handle toml configuration management, and possibly hold database configuration options, but we need the connection and ORM accessible. We’ll want to generalize pkg/api/utils.go and pkg/manager/utils.go and include this in ls-config.

We should generalize the mail package as a convenience tool for us to use in other projects. We could also make use of ls-config here.

Finally, in both the code we abstract into libraries and in our project, we need full test coverage. It’s OK to prototype an initial solution, but before we use this in production, we must make sure the specification is covered by tests. Afterward, any new feature we deliver should have a test developed first and then the minimal solution will be implemented to ensure the test passes.

comments powered by Disqus