Learn How to Declutter Your Dev and Testing GCP Environments

Experimenting with Google Cloud, testing new ideas and creating lots of cloud assets for POCs can create a lot of clutter. Leaving it there can get expensive. Also, this mess imposes the cost of understanding your environment as you continue to work with the project.

What should you do? Blast it!

If you are a customer of DoiT International, you can use Sandbox Projects, which are self-contained and get deleted to clear away the clutter.

A Short DoiT Pitch: If you buy your GCP services through DoiT, you pay no more than you would otherwise. You will also get free consulting from architects like me, as well as access to leading cost-control tools.

But if you’re not yet a DoiT customer, or if your dev projects have assets that you prefer to keep around, while still deleting other clutter, take a look at a Cloud Blaster, a new open source project I will talk about in this article. Simply put, Cloud Blaster safely finds and deletes unwanted assets.

Cloud Blaster vs Safe Scrub

Safe Scrub is a project I have written in Bash. It essentially does the same thing. Feel free to read this blog post for more information.

The advantages of Safe Scrub:

• Safe Scrub gives you the safety of not actually deleting anything. It just outputs a Bash script with a simple list of delete statements. You can then review and run it at your discretion.
• Since Safe Scrub is pure Bash, you may have more confidence as you see the code that is running, without a compilation step.
• Safe Scrub supports more asset types than Cloud Blaster (for now).

Cloud Blaster has its own safety features, as you’ll see below. It also supports the most common asset types, with the capability of adding more.

Compared to Safe Scrub, Cloud Blaster supports more complexity, since it is in Kotlin rather than Bash. This makes it easier to add new asset types and features, robustly handling special cases. Cloud Scrub is at the limit of the complexity that can be accommodated in Bash, so it’s unlikely to go further.

Cloud Blaster Use Case: Same as Safe Scrub

The use case is intended for development and informal testing projects, where you want to start afresh at the end of the day or before a new test run.

It is less likely to be useful for production or staging projects, or even team testing projects. For these, you should be using Terraform or other Infrastructure as Code (IaC) which tracks the assets that were created. You can then delete those assets when needed.

Safety First

To keep things safe and secure, Cloud Blaster has these features.

1. The first step, the Lister, doesn't delete any assets. It just lists assets in a file you can review.
2. The Lister requires you to explicitly state a project. It does not implicitly use your gclouddefault project.
3. The Lister can be filtered. You can specify a regular expression per asset type so that only certain assets are listed or skipped over.
4. After running the Lister, you review the list of assets for deletion and add a comment line # Ready to delete to the top. This makes sure you do not forget the review stage. (If you like to live dangerously, write a script to add this comment between the Lister and Deleter steps.)
5. Finally, run the Deleter, which deletes exactly the assets listed in the file.

Instructions

To get the code, git clone or download it as a zip file.

See the README for details and options. To summarize briefly:

• You optionally edit list-filter.yaml to whitelist or blacklist assets that you want to include or exclude from the deletion list.
• Run the Lister from the command-line.
• Review the output fileasset-list.txt and add a comment # Ready to delete.
• Run the Deleter.

To see other options, run ./lister.sh -h or ./deleter.sh -h.

See the Cloud Blaster README for more

Features

Cloud Blaster now supports common important asset types that are set up and torn down in typical development and QA. These include:

• Google Compute Engine instances, disks, firewalls and addresses
• Google Cloud PubSub topics and subscriptions
• Google Kubernetes Engine regional and zonal clusters
• Google Cloud Operations log metrics
• Google App Engine services and versions
• Google Cloud Functions
• Cloud Run services
• Cloud SQL instances
• Google Cloud Storage buckets

Adding Features

If you want more asset types or new features, please submit an issue at GitHub or add support for the asset type and submit a pull request.

It’s easy!

Kotlin’s concise syntax and a base deleter class mean that you can easily add a deleter for a new asset type. The body of the PubSub Topic deleter, for example, is only seven lines. Since writing the first draft of this article, I added one for Cloud SQL and tested it — in just 15 minutes and 13 lines of code.

Here’s how:

• Uncomment the asset type in asset-types.properties and specify the deleter class here if needed. Instructions are at the top of that file.
• Add the asset type to list-filter.yaml. (We keep two separate config files, because only one is meant to be user-editable. If you forget to edit this file, a clear error message will remind you.) Optionally add a default filter as in the Firewallexample in the file.
• Implement a subclass of BaseDeleteralongside the existing deleter classes. You can use those as examples of how to call the various Google Cloud APIs.

Testing

Cloud Blaster has an integration test that sets up assets and then deletes them, checking the output of each step. For details, see the README.

Other Projects and Approaches

• Safe Scrub is an older Bash project that does the same thing.
• Travis CI GCloud Cleanup and Bazooka also delete GCE assets.
• Cloud Nuke does this for AWS.
• Sandbox Projects are available to DoiT International customers at no charge.

To stay connected, follow us on the DoiT Engineering Blog, DoiT Linkedin Channel and DoiT Twitter Channel. To explore career opportunities, visit https://careers.doit.com.