# Content Central

It's a rare privilege to be paid to create a large, business-critical web application from the ground up. I've been fortunate to do just that at Avalara, building a content management system called Content Central. Although it's only available to a few hundred employees, I'm as proud of the application as if it were used by millions. I certainly didn't built it single-handedly; at least 9 engineers have contributed significant changes. However, as the technical lead on this project from day 1, I'm in a unique position to tell the story of this middle-aged enterprise web application.

# Background and History

Avalara's flagship offering, AvaTax, calculates how much tax should be charged on a transaction at a certain time and place. To do that, the Content Research department translates the ever-changing tax regulations around the world into data that AvaTax can use. Besides tracking many strange and complicated tax scenarios and fees, they maintain the geographical boundaries of tax jurisdictions too.

In Avalara's early growth years, the customer-facing systems were rightfully prioritized, while internal systems were not. For certain types of content, researchers transcribed tax data into enormously wide and deep tables in Excel files, which were reviewed and transformed into database update scripts every month. The manpower needed to make it all work month after month was both impressive and heartbreaking. Eventually it was inescapable that it was time to pay down the technical and process debt in order to improve accuracy and scale. Content Central was conceived as a web application to replace and modernize the content process.

When I joined the project, I was allowed a few months to gather information, sit with the researchers and learn their process and pains, and create static prototypes. In terms of getting feedback on a proposal, I'm not a fan of wireframes and presentations. I believe that it's well worth the effort to make a clickable prototype that people can actually interact with in their own browser. I'm convinced that those prototypes were instrumental in communicating at a level that would not have been possible with wireframes or mockups.

After more than a year and a half of research, development, beta testing, and a big organizational change, we celebrated a significant milestone when the first batch of U.S. sales tax rates curated in Content Central was published to AvaTax. In the following months and years we added more capabilities and "on-boarded" more teams in a familiar cadence: research, propose, train, test, deploy, review, repeat.

It's not enough to just build and launch a web app and enable logins. That's the easiest part, by far. I have learned that there are many links in the chain of of a successful application, such as:

  • Old data must be cleaned and imported accurately and repeatably.
  • The new application must to export data at least as well as (and hopefully much better than) the old system.
  • There must be little to no expectation of changes to upstream and downstream systems. Every external dependency is a potential project-killer.
  • End-users must have training, confidence, and motivation to use the new application.
  • The application has to demonstrate quantifiable value, much sooner than you would expect.
  • There must be a definitive moment when people start using the new system and stop using the old system, like Cortez scuttling his fleet or Caesar crossing the Rubicon.

...and so on. Each of those could be chapters in a book.

# Technical Stack

Content Central is a web application with a static front-end, REST API, and a relational database hosted in AWS. The following explains what is under the hood right now, along with some thoughts of what I might do differently today.

# Front End

Content Cental's front-end started with AngularJS, but now it is a Vue application. Check out this article for more detail about Vue. The application is split up into mini-apps using Vue's multi-page build option. That is an artifact of when we ported the application from Angular to Vue over a few months, section by section. If this project was being built today, however, I'd prefer to make it fully SSR, possibly using Nuxt. That's because we use Vue's router and state management, but the multi-page nature of Content Central sometimes causes problems.

We are using the Vuetify UI framework, an open-source implementation of Google's Material Design for Vue. While it's possible (and fun!) to create a complete design system, I think we made the right call given our constraints. Customizing a well-structured and high-quality framework like Vuetify has been a major productivity boost. I think a good alternative might be BootstrapVue, if for no other reason than the ubiquity of Bootstrap itself.

The front-end code is written using ECMAScript 6, transpiled by Babel to JavaScript. That's a common and good default for Vue apps, but if I was starting a project all over again I'd try to use TypeScript. Regardless, the fact that the Vue CLI uses Webpack under the hood gives a lot of good options here.

We use Pug markup in our single file component templates because it's a little cleaner than HTML. The CSS preprocessor is Stylus, but I think when we upgrade to Vuetify 2 we'll be switching over to SASS. I don't have strong feelings about Pug or Stylus, but I appreciate that they make messy HTML and CSS a little cleaner.

We use map tiles from Mapbox to display maps, but we don't use the Mapbox GL library. Instead, we use OpenLayers thanks to the vastly superior geometry editing tools. We also use Turf.js to manipulate GeoJSON in both the client and the server.

# Back End

The API is Express on Node. The process manager is PM2. Using the Sequelize ORM, we serve up a standard RESTful API. For production, the front end assets static assets are built and bundled with the API in a Docker image, which is then deployed to a cluster with Terraform.

This is the part of the stack that would be the most different, if I were starting this project now. Instead of Express, I'd investigate using Fastify or possibly micro. I don't think I'd use an ORM again, I'd rather use node-postgres directly. A GraphQL endpoint with Hasura or PostGraphile might have met 80% of our needs, but we still would have needed REST endpoints for the other 20%. I'd look at ways to host and deploy the static portion of Content Central on a CDN outside of Node/Docker, per the JAMstack best practices.

The database is PostgreSQL because we make heavy use of PostGIS. Redis helps out with some caching and session management. We experimented with ArangoDB at one point, but PostgreSQL has proven itself time and again. I have nothing but high praise for PostgreSQL and PostGIS, especially the documentation.

# Infrastructure

Content Central is hosted on AWS, but only uses ECS, RDS for PostgreSQL, and Elasticache for Redis. It could easily run on Heroku with almost no changes.

Avalara's internal GitLab instance provides centralized source control and review. We have partial CI/CD using Jenkins, but we're hoping to replace Jenkins with GitLab's native solution soon. Atlassian products (JIRA/Confluence) are used for bug tracking, project management, and documentation. I'd prefer if we just used GitLab for everything in the interests of fewer moving parts, but that is unlikely to happen at Avalara.

# War Stories

TODO: add specific experiences

  • Real-time vector tiles
  • Editing boundaries with OpenLayers + Turf.js
  • Authenticating with SAML; JWT vs. sessions
  • Uplift from Angular to Vue
  • UX design process: discovery > proposal > prototype > development > documentation
  • CI/CD with Jenkins, Teraform, GitHub