For businesses that grow to acquire a network of websites, the limitations of existing development processes and technology are hit very quickly. What once worked at a small scale is no longer suitable at this larger scale.
You need a scalable architecture and development processes which will allow you to grow with confidence and without having to scale your IT team.
There are some telltale signs that your architecture is wrong, which can manifest as some, all or more than the following list:
There are many reasons that this can happen, although usually these problems stem from a leadership or cultural issue within the organization; while we are fixing the technical side effects, it is important to take a moment to reflect on and adjust the culture going forward. Try to stimulate innovation and education within the people in your business to make sure your next hurdle is leaped.
The goals of a scalable solution may be the same as our list, or you may have one or two more difference that apply specifically to your use case, overall your goals should be to:
We have to solve the technical problems in order as each tier of your technology stack defines what is capable for the next level sitting on top of it.
We need suitable hosting to enable us to use advanced features which are a requirement of the refined development process.
Your hosting is the first issue to address, if you are still using shared hosting it is time to move on, you have a wealth of choices to move forward with. Better hosting does not necessarily mean more cost, although this may be a factor.
Shared hosting provides a lot of hand holding, you get high reliability with low performance and low flexibility. However going forward we require high performance and complete control to be in charge of our own reliability.
Virtual Private Hosting (VPS) gives you a bare minimum server where you must install your own software without support from the hosting provider and this is our recommendation. Sometimes this can work out cheaper or minimally more expensive.
Depending on how many websites are in your network will affect the cost of each solution and therefore naturally make one or the other more attractive to pursue. You should also consider that you will keep scaling so keep in mind how your costs may change over time.
The main reason for moving sites is to gain SSH access to the server which will allow us later to set up automated tools for deploying website changes. We can also reduce our security risk by removing FTP access which has a much weaker authentication process compared to SSH.
The migration of your websites from your previous hosting to your new servers must be run alongside the redefinition of the development process. This is because we are deploying websites in a totally new shape.
FTP will no longer be an option, neither will directly connecting to the Database because these are far too exposed to the public and carry a large risk.
The biggest change that your development team will experience is the requirement to use SSH to make any changes to the websites or database and very rarely will this even happen manually.
SSH is a secure shell protocol which requires keys with a default of 1,024 bits which can optionally be 2,048 or 4,096.
Keeping your server locked down means your database and files can never be accessed with just a password and are only available locally when in a SSH session. This can be achieved through the use of a firewall.
The automation, centralisation and architecture will be taking a leap forward, this requires a few changes to the process of the development team within your organisation.
GIT Version Control is the core system responsible for tracking all changes to your code. This will centralise all websites away from just existing as a collection of files on a live site and allow multiple environments to exist such as: Live sites, Development sites and Release sites.
GIT makes it possible for multiple developers to work on the same project without conflicting changes and it is this tracking of file changes that allows features to be launched automatically with zero human input for much safer deployments.
There is some learning curve to take GIT on but your developers will either already know of GIT or already know how to use it in full due to the huge popularity of this solution.
Continuous Integration is the system that will take changes from your developers and automatically implement them on the live website without human input, this is where your safety net really starts to take effect.
Your CI system will be connected to your GIT repository. It will constantly monitor and wait for any changes that you developers commit. Those changes will automatically trigger the CI system to start pushing the changes to the appropriate environment.
Package systems are the modern way to develop, instead of reinventing the wheel making your own features or spending time downloading and packing libraries into your projects we can specify every library we want in a list which can be automatically compiled.
This servers a secondary purpose, or in our case it is actually the primary purpose. In the next step where we centralise all of our common features we will actually package this up into a library. By doing that we can allow each of your websites to include the centralised codebase.
If appropriate consulting and training should be delivered to your development team. This is important to give them the confidence to implement the new process as it should be performed to maximise the competence of the solution you are receiving.
You will find one of the large battles you have faced is having multiple websites with identical or close to identical features. When your developers change one site, they have to copy and paste this change across to all of the others, worrying about potential differences between the sites.
Centralising your features is going to be the longest part of your refactoring project.
We will need to build a separate project which will act as a library of all common features and settings. This project will then be included into each of the websites.
Functionality is grouped into 4 categories: CMS, Utilities, Components, Models.
You will need to create standardised features to support the specific CMS or Framework that your website is built on, this can be to configure those systems into a the best practices for your goals and could involve:
There are various parts of your system that do not depend on your CMS or Framework, it is great practice to build things in an abstracted way so that you are never fully tied to any CMS or Framework, platform agnostic features may include:
Components are the individual features that we will be centralising. Over time this is what should grow the most and be specific to your websites. As such only a few examples are shown:
Models are the definitions for common entities, such as a common shape, or providing a critical feature for all websites or the centralisers. These look more like behind the scenes utilities that are there to support the architecture of your solution overall and encourage further scalable growth. Examples as such are:
While the solution listed above is extremely powerful, versatile and battle tested across many large scale businesses, it is important to not just copy the solution otherwise the next time a different scaling issue outside the scope of this article comes your way you will not be prepared to deal with it.
This solution should not be treated as a silver bullet, you must be ready to adapt and think on your feet or you will find yourself in this situation again.