PWSLab leverages GitLab and Jenkins based runners depending on the project tech stack. In situations where you don't use your Runners 24/7, wanting to have a cost-effective and scalable solution, we have developed an ability to automatically spin up and down VMs, still making sure your builds/jobs in pipelines are processed immediately.
This ability is currently supported on PWSLab instances or PWSLab Runners hosted on Amazon Web Services (AWS) only.
PWSLab CI/CD allows splitting your pipeline jobs across multiple machines. By default, each new runner node requires some setup work to provision and attach it to the PWSLab instance, but we used automation by provisioning a single Spawner/Manager machine and let that machine decide how much capacity is required and then spin up or down further instances as required.
Creating the spawner/manager
First off, we need a spawner machine. This runs 24/7 and checks that PWSLab CI/CD has enough capacity to run the jobs currently in the queue as the pipelines are triggered/scheduled in the projects. It doesn’t run any jobs itself.
We use Ubuntu 16.04 LTS for the spawner machine's internal tooling, so an EC2 instance (t2.micro is enough which is included in the AWS free tier.) Setting up VPCs and related subnets is out of the scope of this article (will be configured by our Support team). Then we need to install a bunch of software on our machine to set it up.
Docker Machine is a handy bit of software that allows one host running Docker to spin up and provision other machines running Docker. Read more.
You’ll need an AWS Access Key
tied to a user with permission to scale (EC2) and update the cache (via S3). Create a new user with policies
for EC2 (AmazonEC2FullAccess) and S3 (AmazonS3FullAccess). To be more secure, you can disable the console login for that user. Keep the tab open or copy-paste the security credentials in an editor as they will be required by our Support team while setting up the real-time CI/CD Runners for your instance.
Save up to 90 percent on EC2 costs using Spot Instances
Spot Instances are magic. They allow us to bid for unused capacity in the AWS infrastructure and often can mean that EC2 costs can be dramatically lower. We’re currently seeing discounts of around 85 percent on our EC2 bills due to using Spot Instances. There is (of course) a downside. If our bid price for VMs is exceeded, then our instances shut down with only a few minutes notice.
But, as long as our bid is high enough, this isn’t an issue. Pricing in the spot market is insanely complex but in ap-south-1 at least, prices for m4.large and xlarge instances appear to have been static for months so a bid 10-20 percent higher than the current spot price appears to be a safe bet. Just keep your eyes peeled. The current spot price for an m4.xlarge instance is $0.026. We’ve set our maximum price at $0.03 to give us some wiggle room. At time of writing, the on-demand price is $0.232. The numbers speak for themselves.
Note: Spot pricing can vary significantly between instance sizes, regions and even availability zones in the same region. This guide assumes that spot pricing won’t vary massively or we set a good buffer above the current spot price to avoid outages.
This configuration tells PWSLab Spawner/Manager that instead of just spawning new EC2 instances at full price, that it should request AWS Spot Instances at the current spot price, setting a maximum bid that it should not exceed per hour, in USD (regardless of what currency you’re billed in. We’re billed in INR, but Spot Instances are still calculated in USD).
The maximum bid is whatever you’re comfortable paying. We tend to set it close to the on-demand price because we’re looking for any discount. As long as we’re not paying more than we otherwise would, it’s fine with us. Your financial constraints may affect your decisions differently.
Update: From October, AWS will charge in seconds, rather than hours used, making the potential savings even higher for unused partial hours.
We’d love to see how you get along with this so please let us know. You can contact us at email@example.com
. We've been testing this internally (as beta) since last 4 months, it saved us time and money and that’s never a bad thing.
Making the Spot price bids, choosing the right set of instances, PWSLab Spawner setup - is all configured by PeerXP Support. Raise a Premium DevOps Support Ticket here to get the Real-time CI/CD runner configuration done for your PWSLab instance.
Also, let us know if the article is helpful!