- Home
- Docs
- Cloud Functions
- Documentation
- Guides
Retrying Event-Driven Functions
This document describes how to enable retrying for event-driven functions [background functions and CloudEvent functions]. Automatic retrying is not available for HTTP functions.
Note: the type of event-driven function you use is determined by the runtime your function will run on.Semantics of retry
Cloud Functions guarantees at-least-once execution of an event-driven function for each event emitted by an event source. However, by default, if a function invocation terminates with an error, the function will not be invoked again, and the event will be dropped. When you enable retries on an event-driven function, Cloud Functions will retry a failed function invocation until it completes successfully, or the retry window [by default, 7 days] expires.
Why event-driven functions fail to complete
On rare occasions, a function might exit prematurely due to an internal error, and by default the function might or might not be automatically retried.
More typically, an event-driven function may fail to successfully complete due to errors thrown in the function code itself. Some of the reasons this might happen are as follows:
- The function contains a bug and the runtime throws an exception.
- The function cannot reach a service endpoint, or times out while trying to reach the endpoint.
- The function intentionally throws an exception [for example, when a parameter fails validation].
- When functions written in Node.js return a rejected promise or pass a non-null value to a callback.
In any of the above cases, the function stops executing by default and the event is discarded. If you want to retry the function when an error occurs, you can change the default retry policy by setting the "retry on failure" property. This causes the event to be retried repeatedly for up to multiple days until the function successfully completes.
Enabling and disabling retries
To enable or disable retries, you can either use the gcloud command-line tool or the Cloud Console. By default, retries are disabled.
Using the gcloud command-line tool
To enable retries via the gcloud command-line tool, include the --retry flag when deploying your function:
gcloud functions deploy FUNCTION_NAME --retry FLAGS...To disable retries, re-deploy the function without the --retry flag:
gcloud functions deploy FUNCTION_NAME FLAGS...Using the Cloud Console
You can enable or disable retries in the Cloud Console as follows:
Go to the Cloud Functions Overview page in the Cloud Platform Console.
Click Create function. Alternatively, click an existing function to go to its details page and click Edit.
Fill in the required fields for your function.
Ensure the Trigger field is set to an event-based trigger type, such as Cloud Pub/Sub or Cloud Storage.
Expand the advanced settings by clicking More.
Check or uncheck the box labeled Retry on failure.
Best practices
This section describes best practices for using retries.
Use retry to handle transient errors
Because your function is retried continuously until successful execution, permanent errors like bugs should be eliminated from your code through testing before enabling retries. Retries are best used to handle intermittent/transient failures that have a high likelihood of resolution upon retrying, such as a flaky service endpoint or timeout.
Set an end condition to avoid infinite retry loops
It is best practice to protect your function against continuous looping when using retries. You can do this by including a well-defined end condition, before the function begins processing. Note that this technique only works if your function starts successfully and is able to evaluate the end condition.
A simple yet effective approach is to discard events with timestamps older than a certain time. This helps to avoid excessive executions when failures are either persistent or longer-lived than expected.
For example, this code snippet discards all events older than 10 seconds:
Node.js
functions/tips/index.js
View on GitHub Feedback
/**
* Background Cloud Function that only executes within
* a certain time period after the triggering event
*
* @param {object} event The Cloud Functions event.
* @param {function} callback The callback function.
*/
exports.avoidInfiniteRetries = [event, callback] => {
const eventAge = Date.now[] - Date.parse[event.timestamp];
const eventMaxAge = 10000;
// Ignore events that are too old
if [eventAge > eventMaxAge] {
console.log[`Dropping event ${event} with age ${eventAge} ms.`];
callback[];
return;
}
// Do what the function is supposed to do
console.log[`Processing event ${event} with age ${eventAge} ms.`];
// Retry failed function executions
const failed = false;
if [failed] {
callback['some error'];
} else {
callback[];
}
};
Python
functions/tips/main.py
View on GitHub Feedback
from datetime import datetime, timezone
# The 'python-dateutil' package must be included in requirements.txt.
from dateutil import parser
def avoid_infinite_retries[data, context]:
"""Background Cloud Function that only executes within a certain
time period after the triggering event.
Args:
data [dict]: The event payload.
context [google.cloud.functions.Context]: The event metadata.
Returns:
None; output is written to Stackdriver Logging
"""
timestamp = context.timestamp
event_time = parser.parse[timestamp]
event_age = [datetime.now[timezone.utc] - event_time].total_seconds[]
event_age_ms = event_age * 1000
# Ignore events that are too old
max_age_ms = 10000
if event_age_ms > max_age_ms:
print['Dropped {} [age {}ms]'.format[context.event_id, event_age_ms]]
return 'Timeout'
# Do what the function is supposed to do
print['Processed {} [age {}ms]'.format[context.event_id, event_age_ms]]
return # To retry the execution, raise an exception here
Go
functions/tips/infinite_retries/infinite_retries.go
View on GitHub Feedback
// Package tips contains tips for writing Cloud Functions in Go.
package tips
import [
"context"
"fmt"
"log"
"time"
"cloud.google.com/go/functions/metadata"
]
// PubSubMessage is the payload of a Pub/Sub event.
// See the documentation for more details:
// //cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
type PubSubMessage struct {
Data []byte `json:"data"`
}
// FiniteRetryPubSub demonstrates how to avoid inifinite retries.
func FiniteRetryPubSub[ctx context.Context, m PubSubMessage] error {
meta, err := metadata.FromContext[ctx]
if err != nil {
// Assume an error on the function invoker and try again.
return fmt.Errorf["metadata.FromContext: %v", err]
}
// Ignore events that are too old.
expiration := meta.Timestamp.Add[10 * time.Second]
if time.Now[].After[expiration] {
log.Printf["event timeout: halting retries for expired event '%q'", meta.EventID]
return nil
}
// Add your message processing logic.
return processTheMessage[m]
}
Java
functions/concepts/retry-timeout/src/main/java/functions/RetryTimeout.java
View on GitHub Feedback
import com.google.cloud.functions.BackgroundFunction;
import com.google.cloud.functions.Context;
import com.google.events.cloud.pubsub.v1.Message;
import com.google.gson.Gson;
import java.time.Duration;
import java.time.ZoneOffset;
import java.time.ZonedDateTime;
import java.util.logging.Logger;
public class RetryTimeout implements BackgroundFunction {
private static final Logger logger = Logger.getLogger[RetryTimeout.class.getName[]];
private static final long MAX_EVENT_AGE = 10_000;
// Use Gson [//github.com/google/gson] to parse JSON content.
private static final Gson gson = new Gson[];
/**
* Background Cloud Function that only executes within
* a certain time period after the triggering event
*/
@Override
public void accept[Message message, Context context] {
ZonedDateTime utcNow = ZonedDateTime.now[ZoneOffset.UTC];
ZonedDateTime timestamp = ZonedDateTime.parse[context.timestamp[]];
long eventAge = Duration.between[timestamp, utcNow].toMillis[];
// Ignore events that are too old
if [eventAge > MAX_EVENT_AGE] {
logger.info[String.format["Dropping event with timestamp %s.", timestamp]];
return;
}
// Process events that are recent enough
// To retry this invocation, throw an exception here
logger.info[String.format["Processing event with timestamp %s.", timestamp]];
}
}
C#
functions/concepts/TimeBoundedRetries/Function.cs
View on GitHub Feedback
using CloudNative.CloudEvents;
using Google.Cloud.Functions.Framework;
using Google.Events.Protobuf.Cloud.PubSub.V1;
using Microsoft.Extensions.Logging;
using System;
using System.Threading;
using System.Threading.Tasks;
namespace TimeBoundedRetries
{
public class Function : ICloudEventFunction
{
private static readonly TimeSpan MaxEventAge = TimeSpan.FromSeconds[10];
private readonly ILogger _logger;
// Note: for additional testability, use an injectable clock abstraction.
public Function[ILogger logger] =>
_logger = logger;
public Task HandleAsync[CloudEvent cloudEvent, MessagePublishedData data, CancellationToken cancellationToken]
{
string textData = data.Message.TextData;
DateTimeOffset utcNow = DateTimeOffset.UtcNow;
// Every PubSub CloudEvent will contain a timestamp.
DateTimeOffset timestamp = cloudEvent.Time.Value;
DateTimeOffset expiry = timestamp + MaxEventAge;
// Ignore events that are too old.
if [utcNow > expiry]
{
_logger.LogInformation["Dropping PubSub message '{text}'", textData];
return Task.CompletedTask;
}
// Process events that are recent enough.
// If this processing throws an exception, the message will be retried until either
// processing succeeds or the event becomes too old and is dropped by the code above.
_logger.LogInformation["Processing PubSub message '{text}'", textData];
return Task.CompletedTask;
}
}
}
Ruby
functions/tips/infinite_retries/app.rb
View on GitHub Feedback
require "functions_framework"
FunctionsFramework.cloud_event "avoid_infinite_retries" do |event|
# Use the event timestamp to determine the event age.
event_age_secs = Time.now - event.time.to_time
event_age_ms = [event_age_secs * 1000].to_i
max_age_ms = 10_000
if event_age_ms > max_age_ms
# Ignore events that are too old.
logger.info "Dropped #{event.id} [age #{event_age_ms}ms]"
else
# Do what the function is supposed to do.
logger.info "Handling #{event.id} [age #{event_age_ms}ms]..."
failed = true
# Raise an exception to signal failure and trigger a retry.
raise "I failed!" if failed
end
end
PHP
functions/tips_infinite_retries/index.php
View on GitHub Feedback
/**
* This function shows an example method for avoiding infinite retries in
* Google Cloud Functions. By default, functions configured to automatically
* retry execution on failure will be retried indefinitely - causing an
* infinite loop. To avoid this, we stop retrying executions [by not throwing
* exceptions] for any events that are older than a predefined threshold.
*/
use Google\CloudFunctions\CloudEvent;
function avoidInfiniteRetries[CloudEvent $event]: void
{
$log = fopen[getenv['LOGGER_OUTPUT'] ?: 'php://stderr', 'wb'];
$eventId = $event->getId[];
// The maximum age of events to process.
$maxAge = 10; // 10 seconds
// The age of the event being processed.
$eventAge = time[] - strtotime[$event->getTime[]];
// Ignore events that are too old
if [$eventAge > $maxAge] {
fwrite[$log, 'Dropping event ' . $eventId . ' with age ' . $eventAge . ' seconds' . PHP_EOL];
return;
}
// Do what the function is supposed to do
fwrite[$log, 'Processing event: ' . $eventId . ' with age ' . $eventAge . ' seconds' . PHP_EOL];
// infinite_retries failed function executions
$failed = true;
if [$failed] {
throw new Exception['Event ' . $eventId . ' failed; retrying...'];
}
}
Distinguish between retriable and fatal errors
If your function has retries enabled, any unhandled error will trigger a retry. Make sure that your code captures any errors that should not result in a retry.
Node.js
functions/tips/index.js
View on GitHub Feedback
/**
* Background Cloud Function that demonstrates
* how to toggle retries using a promise
*
* @param {object} event The Cloud Functions event.
* @param {object} event.data Data included with the event.
* @param {object} event.data.retry User-supplied parameter that tells the function whether to retry.
*/
exports.retryPromise = event => {
const tryAgain = !!event.data.retry;
if [tryAgain] {
throw new Error['Retrying...'];
} else {
console.error['Not retrying...'];
return Promise.resolve[];
}
};
/**
* Background Cloud Function that demonstrates
* how to toggle retries using a callback
*
* @param {object} event The Cloud Functions event.
* @param {object} event.data Data included with the event.
* @param {object} event.data.retry User-supplied parameter that tells the function whether to retry.
* @param {function} callback The callback function.
*/
exports.retryCallback = [event, callback] => {
const tryAgain = !!event.data.retry;
const err = new Error['Error!'];
if [tryAgain] {
console.error['Retrying:', err];
callback[err];
} else {
console.error['Not retrying:', err];
callback[];
}
};
Python
functions/tips/main.py
View on GitHub Feedback
from google.cloud import error_reporting
error_client = error_reporting.Client[]
def retry_or_not[data, context]:
"""Background Cloud Function that demonstrates how to toggle retries.
Args:
data [dict]: The event payload.
context [google.cloud.functions.Context]: The event metadata.
Returns:
None; output is written to Stackdriver Logging
"""
# Retry based on a user-defined parameter
try_again = data.data.get['retry'] is not None
try:
raise RuntimeError['I failed you']
except RuntimeError:
error_client.report_exception[]
if try_again:
raise # Raise the exception and try again
else:
pass # Swallow the exception and don't retry
Go
functions/tips/retry.go
View on GitHub Feedback
// Package tips contains tips for writing Cloud Functions in Go.
package tips
import [
"context"
"errors"
"log"
]
// PubSubMessage is the payload of a Pub/Sub event.
// See the documentation for more details:
// //cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
type PubSubMessage struct {
Data []byte `json:"data"`
}
// RetryPubSub demonstrates how to toggle using retries.
func RetryPubSub[ctx context.Context, m PubSubMessage] error {
name := string[m.Data]
if name == "" {
name = "World"
}
// A misconfigured client will stay broken until the function is redeployed.
client, err := MisconfiguredDataClient[]
if err != nil {
log.Printf["MisconfiguredDataClient [retry denied]: %v", err]
// A nil return indicates that the function does not need a retry.
return nil
}
// Runtime error might be resolved with a new attempt.
if err = FailedWriteOperation[client, name]; err != nil {
log.Printf["FailedWriteOperation [retry expected]: %v", err]
// A non-nil return indicates that a retry is needed.
return err
}
return nil
}
Java
functions/concepts/retry-pubsub/src/main/java/functions/RetryPubSub.java
View on GitHub Feedback
import com.google.cloud.functions.BackgroundFunction;
import com.google.cloud.functions.Context;
import com.google.events.cloud.pubsub.v1.Message;
import com.google.gson.Gson;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.logging.Logger;
public class RetryPubSub implements BackgroundFunction {
private static final Logger logger = Logger.getLogger[RetryPubSub.class.getName[]];
// Use Gson [//github.com/google/gson] to parse JSON content.
private static final Gson gson = new Gson[];
@Override
public void accept[Message message, Context context] {
String bodyJson = new String[
Base64.getDecoder[].decode[message.getData[]], StandardCharsets.UTF_8];
JsonElement bodyElement = gson.fromJson[bodyJson, JsonElement.class];
// Get the value of the "retry" JSON parameter, if one exists
boolean retry = false;
if [bodyElement != null && bodyElement.isJsonObject[]] {
JsonObject body = bodyElement.getAsJsonObject[];
if [body.has["retry"] && body.get["retry"].getAsBoolean[]] {
retry = true;
}
}
// Retry if appropriate
if [retry] {
// Throwing an exception causes the execution to be retried
throw new RuntimeException["Retrying..."];
} else {
logger.info["Not retrying..."];
}
}
}
C#
functions/concepts/Retry/Function.cs
View on GitHub Feedback
using CloudNative.CloudEvents;
using Google.Cloud.Functions.Framework;
using Google.Events.Protobuf.Cloud.PubSub.V1;
using Microsoft.Extensions.Logging;
using System;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
namespace Retry
{
public class Function : ICloudEventFunction
{
private readonly ILogger _logger;
public Function[ILogger logger] =>
_logger = logger;
public Task HandleAsync[CloudEvent cloudEvent, MessagePublishedData data, CancellationToken cancellationToken]
{
bool retry = false;
string text = data.Message?.TextData;
// Get the value of the "retry" JSON parameter, if one exists.
if [!string.IsNullOrEmpty[text]]
{
JsonElement element = JsonSerializer.Deserialize[data.Message.TextData];
retry = element.TryGetProperty["retry", out var property] &&
property.ValueKind == JsonValueKind.True;
}
// Throwing an exception causes the execution to be retried.
if [retry]
{
throw new InvalidOperationException["Retrying..."];
}
else
{
_logger.LogInformation["Not retrying..."];
}
return Task.CompletedTask;
}
}
}
Ruby
functions/tips/retry/app.rb
View on GitHub Feedback
require "functions_framework"
FunctionsFramework.cloud_event "retry_or_not" do |event|
try_again = event.data["retry"]
begin
# Simulate a failure
raise "I failed!"
rescue RuntimeError => e
logger.warn "Caught an error: #{e}"
if try_again
# Raise an exception to return a 500 and trigger a retry.
logger.info "Trying again..."
raise ex
else
# Return normally to end processing of this event.
logger.info "Giving up."
end
end
end
PHP
functions/tips_retry/index.php
View on GitHub Feedback
use Google\CloudFunctions\CloudEvent;
function tipsRetry[CloudEvent $event]: void
{
$cloudEventData = $event->getData[];
$pubSubData = $cloudEventData['message']['data'];
$json = json_decode[base64_decode[$pubSubData], true];
// Determine whether to retry the invocation based on a parameter
$tryAgain = $json['some_parameter'];
if [$tryAgain] {
/**
* Functions with automatic retries enabled should throw exceptions to
* indicate intermittent failures that a retry might fix. In this
* case, a thrown exception will cause the original function
* invocation to be re-sent.
*/
throw new Exception['Intermittent failure occurred; retrying...'];
}
/**
* If a function with retries enabled encounters a non-retriable
* failure, it should return *without* throwing an exception.
*/
$log = fopen[getenv['LOGGER_OUTPUT'] ?: 'php://stderr', 'wb'];
fwrite[$log, 'Not retrying' . PHP_EOL];
}
Make retryable event-driven functions idempotent
Event-driven functions that can be retried must be idempotent. Here are some general guidelines for making such a function idempotent:
- Many external APIs [such as Stripe] let you supply an idempotency key as a parameter. If you are using such an API, you should use the event ID as the idempotency key.
- Idempotency works well with at-least-once delivery, because it makes it safe to retry. So a general best practice for writing reliable code is to combine idempotency with retries.
- Make sure that your code is internally idempotent. For example:
- Make sure that mutations can happen more than once without changing the outcome.
- Query database state in a transaction before mutating the state.
- Make sure that all side effects are themselves idempotent.
- Impose a transactional check outside the function, independent of the code. For example, persist state somewhere recording that a given event ID has already been processed.
- Deal with duplicate function calls out-of-band. For example, have a separate clean up process that cleans up after duplicate function calls.
Next steps
- Deploying Cloud Functions.
- Calling Cloud Pub/Sub Trigger Functions.
- Calling Cloud Storage Trigger Functions.
- Cloud Functions with Cloud Pub/Sub Tutorial.
- Cloud Functions with Cloud Storage Tutorial.