Scaling Multiple Website for Growing Businesses

For businesses that grow to acquire a network of websites, the limitations of existing development processes and technology are hit very quickly. What once worked at a small scale is no longer suitable at this larger scale.

You need a scalable architecture and development processes which will allow you to grow with confidence and without having to scale your IT team.

Indicative Signs of Poor Scalability

There are some telltale signs that your architecture is wrong, which can manifest as some, all or more than the following list:

Slow movement from your development team
Fear over deploying changes & downtime during deployments
Culture of not altering websites or implementing new features
Dependence on a single or ‘rockstar’ developer

There are many reasons that this can happen, although usually these problems stem from a leadership or cultural issue within the organization; while we are fixing the technical side effects, it is important to take a moment to reflect on and adjust the culture going forward. Try to stimulate innovation and education within the people in your business to make sure your next hurdle is leaped.

Establishing your Scalability Goals

The goals of a scalable solution may be the same as our list, or you may have one or two more difference that apply specifically to your use case, overall your goals should be to:

Develop faster
Deploy without fear
Centralise common components
Implement a professional development process

Turning your Goals into an Action Plan

We have to solve the technical problems in order as each tier of your technology stack defines what is capable for the next level sitting on top of it.

Find suitable hosting
Migrate your websites
Redefine the development process
Centralise your features
Continue scaling

1. Suitable Hosting

We need suitable hosting to enable us to use advanced features which are a requirement of the refined development process.

Your hosting is the first issue to address, if you are still using shared hosting it is time to move on, you have a wealth of choices to move forward with. Better hosting does not necessarily mean more cost, although this may be a factor.

Shared hosting provides a lot of hand holding, you get high reliability with low performance and low flexibility. However going forward we require high performance and complete control to be in charge of our own reliability.

Virtual Private Hosting (VPS) gives you a bare minimum server where you must install your own software without support from the hosting provider and this is our recommendation. Sometimes this can work out cheaper or minimally more expensive.

Hosting Styles

Monolithic: This approach gives you just one large server to put all of your websites on, it is often the most cost effective but also the most dangerous as problems end up being shared across all of your websites.
Clustered: A cheaper alternative is to have 2 or 3 medium sized servers which each hold a few of your websites each. Your risk is now distributed into multiple parts reducing the likelihood of a total site-wide outage.
Individual: The most distributed approach takes away the risk of shared problems but is a trade off with more maintenance required since there are more servers to take care of.

Depending on how many websites are in your network will affect the cost of each solution and therefore naturally make one or the other more attractive to pursue. You should also consider that you will keep scaling so keep in mind how your costs may change over time.

SSH Access

The main reason for moving sites is to gain SSH access to the server which will allow us later to set up automated tools for deploying website changes. We can also reduce our security risk by removing FTP access which has a much weaker authentication process compared to SSH.

2. Migration of Websites

The migration of your websites from your previous hosting to your new servers must be run alongside the redefinition of the development process. This is because we are deploying websites in a totally new shape.

FTP will no longer be an option, neither will directly connecting to the Database because these are far too exposed to the public and carry a large risk.

Server Lockdown & Firewall

The biggest change that your development team will experience is the requirement to use SSH to make any changes to the websites or database and very rarely will this even happen manually.

SSH is a secure shell protocol which requires keys with a default of 1,024 bits which can optionally be 2,048 or 4,096.

Keeping your server locked down means your database and files can never be accessed with just a password and are only available locally when in a SSH session. This can be achieved through the use of a firewall.

3. Redefine the Development Process

The automation, centralisation and architecture will be taking a leap forward, this requires a few changes to the process of the development team within your organisation.

Version Control (GIT)

GIT Version Control is the core system responsible for tracking all changes to your code. This will centralise all websites away from just existing as a collection of files on a live site and allow multiple environments to exist such as: Live sites, Development sites and Release sites.

GIT makes it possible for multiple developers to work on the same project without conflicting changes and it is this tracking of file changes that allows features to be launched automatically with zero human input for much safer deployments.

There is some learning curve to take GIT on but your developers will either already know of GIT or already know how to use it in full due to the huge popularity of this solution.

Continuous Integration (Circle CI)

Continuous Integration is the system that will take changes from your developers and automatically implement them on the live website without human input, this is where your safety net really starts to take effect.

Your CI system will be connected to your GIT repository. It will constantly monitor and wait for any changes that you developers commit. Those changes will automatically trigger the CI system to start pushing the changes to the appropriate environment.

Package Systems & Bundlers (NPM, Composer, Webpack)

Package systems are the modern way to develop, instead of reinventing the wheel making your own features or spending time downloading and packing libraries into your projects we can specify every library we want in a list which can be automatically compiled.

This servers a secondary purpose, or in our case it is actually the primary purpose. In the next step where we centralise all of our common features we will actually package this up into a library. By doing that we can allow each of your websites to include the centralised codebase.

Consulting

If appropriate consulting and training should be delivered to your development team. This is important to give them the confidence to implement the new process as it should be performed to maximise the competence of the solution you are receiving.

4. Centralise Your Features

You will find one of the large battles you have faced is having multiple websites with identical or close to identical features. When your developers change one site, they have to copy and paste this change across to all of the others, worrying about potential differences between the sites.

Centralising your features is going to be the longest part of your refactoring project.

Creating The Centraliser Project

We will need to build a separate project which will act as a library of all common features and settings. This project will then be included into each of the websites.

Functionality is grouped into 4 categories: CMS, Utilities, Components, Models.

CMS

You will need to create standardised features to support the specific CMS or Framework that your website is built on, this can be to configure those systems into a the best practices for your goals and could involve:

Standard configuration of all CMS/Frameworks
Utilities for building faster platform specific features, i.e. for WordPress: Custom Post Types, Sidebars, Meta boxes, Taxonomies, Shortcode, Widgets
Prebuild common admin dashboard pages & meta boxes
Raise the capability of your CMS/Framework to Implement MVC software architecture
Create common supplementary libraries
Introduce new platform specific functionality

Utilities

There are various parts of your system that do not depend on your CMS or Framework, it is great practice to build things in an abstracted way so that you are never fully tied to any CMS or Framework, platform agnostic features may include:

Formatting: numbers / currency / phone numbers / emails
Geography: Country lists, county lists, postcode lists, geocoding
IP & Browser information
Mailing system
3rd party library or API integrations

Components

Components are the individual features that we will be centralising. Over time this is what should grow the most and be specific to your websites. As such only a few examples are shown:

Contact Form
Payment Form
Cookie Notification Bar
Email Templates
Google Maps
Testimonials Slider
Meet the team listing
Job listings

Models

Models are the definitions for common entities, such as a common shape, or providing a critical feature for all websites or the centralisers. These look more like behind the scenes utilities that are there to support the architecture of your solution overall and encourage further scalable growth. Examples as such are:

Controllers for common pages or as a starting point
AJAX endpoints
API integrations
Config adapter
Enums
Cookies

5. Continue Scaling

While the solution listed above is extremely powerful, versatile and battle tested across many large scale businesses, it is important to not just copy the solution otherwise the next time a different scaling issue outside the scope of this article comes your way you will not be prepared to deal with it.

This solution should not be treated as a silver bullet, you must be ready to adapt and think on your feet or you will find yourself in this situation again.