Blog

The hidden costs of Google Cloud Datastore

Datastore export costs don’t show up in Google Cloud Monitoring. Here’s how to set up real-time alerts to better keep track of export costs.

Google Cloud Datastore monitoring, available in GCP Operations (formerly Stackdriver) as well as the App Engine Dashboard, shows metrics for your Datastore operations, allowing you to track where cloud costs or infrastructure load comes from.

1 laccckob6rseynehwfeceg
Reveal the hidden costs of Google Cloud Datastore

Or at least, mostly.

0 s5otpyghceq4wp17
Datastore Monitoring in GCP Operations Dashboard during Export
0 sppmwrqwai2l3qlg
Datastore Monitoring in GCP Operations Dashboard during Export

Read operations which are part of Datastore Export, are not reflected in the monitoring. You can see that in the two images to the left.

Compare these to the visualization in the dashboard for normal Datastore operations. (See the Appendix at the end of the post.)

So, if you do a massive backup using Export, you will pay for, but not easily see the source of, the costs. Worse, since Exports are usually set up to run periodically, using a cron job or the equivalent, massive Exports may be happening frequently and without your knowledge.

You can track costs in GCP Billing or with reOptimize, but that lags by a day and does not show the precise source of the costs.

Set up real-time alerts for when Datastore is in use

Instead, look for an ExportEntities entry in the GCP logs. Find it using this advanced Log Viewer filter (or go straight to it here).

You can even set up an alert on this log query to remind you that your Datastore is in (potentially massive) use.

The steps to do this are just the usual ones for a Log-based Metric.

  • Go to the Log Viewer query.
  • Select “Create Metric”.
  • Give it a name like DatastoreExport, and leave the other default values (Counter Metric). Click “Create Metric.” You will now be taken to the Metrics page.
  • Click the three vertical dots on the far right of your metric as in this screenshot, and choose “Create alert from metric.”
0 qiuocyj4ychyf bj
Log Metrics; choose to Create alert

You may have to close the condition panel now, as the auto-filled values are buggy (and show an error indication).

  • Give your alerting policy a name, for example, exporting-alert.
  • Click “Add Condition” to (again) open the condition panel.
  • Give a Target of your new log-based metric (logging/user/DatastoreExport), and a trigger condition of “is above 0”. Click “Add”.
0 rnai y peei3era
Creating an alerting policy
  • Click “Add notification channel,” choose “Email”, and enter your address, and click “Add.”

You will now get an email when Export runs.

Appendix

These dashboard images show the normal operation of Monitoring — Datastore operations are correctly presented.

0 n1sz83vdxef71gjs
Datastore Monitoring in GCP Operations Dashboard during normal operations
0 ebeabwjju3hcd4y0
Datastore Monitoring in GCP Operations Dashboard during normal operations
0 j byiirtmdmdfnxf
Datastore billing status in AppEngine Dashboard during both normal operations and Export
0 hczuuid2rnxl430w
Discovering the (possibly cron-triggered) Export, that incurred the cost

Subscribe to updates, news and more.

Related blogs