You can embed etcd directly into a Go application, which sounds like a neat trick to avoid managing a separate etcd cluster for development or small deployments, but it’s actually a fundamentally different beast than using etcd as a standalone service.
Here’s how you might run etcd embedded in a Go app:
package main
import (
"context"
"log"
"net"
"net/http"
"os"
"time"
"go.etcd.io/etcd/server/v3/embed"
)
func main() {
// Create a temporary directory for etcd data
dataDir, err := os.MkdirTemp("", "etcd-embed-")
if err != nil {
log.Fatalf("failed to create temp dir: %v", err)
}
defer os.RemoveAll(dataDir) // Clean up the temp dir
// Configuration for the embedded etcd server
cfg := embed.Config{
// Use a unique name for the embedded etcd member
Name: "embed-etcd-member",
// Specify the directory where etcd will store its data
Dir: dataDir,
// Bind etcd to a specific IP and port for client communication
ClientURLs: []string{"http://127.0.0.1:2379"},
// Bind etcd to a specific IP and port for peer communication (if clustering)
// For a single embedded instance, this can be the same as ClientURLs or a different one
PeerURLs: []string{"http://127.0.0.1:2380"},
// Set a heartbeat interval for cluster health checks
HeartbeatInterval: 100 * time.Millisecond,
// Set a timeout for leader elections
ElectionTimeout: 1000 * time.Millisecond,
// Configure logging
Logger: "zap",
// Disable the client security (for simplicity in this example)
// In production, you'd configure TLS here.
ACMEProvider: "none",
}
// Start the embedded etcd server
etcdServer, err := embed.Start(&cfg)
if err != nil {
log.Fatalf("failed to start embedded etcd: %v", err)
}
defer etcdServer.Close()
log.Println("Embedded etcd started successfully.")
// Wait for etcd to be ready
select {
case <-etcdServer.Server.ReadyNotify():
log.Println("Embedded etcd server is ready.")
case <-time.After(60 * time.Second):
log.Fatal("timed out waiting for embedded etcd to become ready")
}
// Now you can interact with the etcd server using its client API
// For example, you could create an etcd client here:
// cli, err := clientv3.New(clientv3.Config{
// Endpoints: cfg.ClientURLs,
// DialTimeout: 5 * time.Second,
// })
// if err != nil {
// log.Fatalf("failed to create etcd client: %v", err)
// }
// defer cli.Close()
// Keep the application running
select {}
}
The most surprising true thing about embedding etcd is that you’re not just running a smaller version of the standalone etcd; you’re essentially giving your Go application direct control over the entire lifecycle of an etcd cluster member, including its data persistence and network interfaces, as if it were a library.
Let’s see this in action. Imagine your Go application needs a reliable key-value store for configuration or service discovery, but you want to avoid the operational overhead of managing a separate etcd cluster. You can spin up an embedded etcd instance within your application’s process.
package main
import (
"context"
"log"
"net"
"net/http"
"os"
"time"
clientv3 "go.etcd.io/etcd/client/v3"
"go.etcd.io/etcd/server/v3/embed"
)
func main() {
dataDir, err := os.MkdirTemp("", "etcd-embed-")
if err != nil {
log.Fatalf("failed to create temp dir: %v", err)
}
defer os.RemoveAll(dataDir)
cfg := embed.Config{
Name: "embed-etcd-member",
Dir: dataDir,
ClientURLs: []string{"http://127.0.0.1:2379"},
PeerURLs: []string{"http://127.0.0.1:2380"},
Logger: "zap",
ACMEProvider: "none",
}
etcdServer, err := embed.Start(&cfg)
if err != nil {
log.Fatalf("failed to start embedded etcd: %v", err)
}
defer etcdServer.Close()
select {
case <-etcdServer.Server.ReadyNotify():
log.Println("Embedded etcd server is ready.")
case <-time.After(60 * time.Second):
log.Fatal("timed out waiting for embedded etcd to become ready")
}
// Create a client to interact with the embedded etcd
cli, err := clientv3.New(clientv3.Config{
Endpoints: cfg.ClientURLs,
DialTimeout: 5 * time.Second,
})
if err != nil {
log.Fatalf("failed to create etcd client: %v", err)
}
defer cli.Close()
log.Println("Successfully connected to embedded etcd.")
// Use the client to put and get data
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
_, err = cli.Put(ctx, "mykey", "myvalue")
cancel()
if err != nil {
log.Fatalf("failed to put key: %v", err)
}
log.Println("Successfully put 'mykey' with value 'myvalue'.")
ctx, cancel = context.WithTimeout(context.Background(), 5*time.Second)
resp, err := cli.Get(ctx, "mykey")
cancel()
if err != nil {
log.Fatalf("failed to get key: %v", err)
}
for _, kv := range resp.Kvs {
log.Printf("Retrieved 'mykey': %s", kv.Value)
}
// Keep the application running so etcd stays alive
select {}
}
This code snippet demonstrates the core idea: you configure etcd’s essential parameters (like Dir for data storage, ClientURLs and PeerURLs for networking) directly within your Go application’s configuration. When embed.Start(&cfg) is called, your application spawns an etcd server process within its own memory space. You then use the standard etcd client library to communicate with this embedded server, just as you would with a standalone cluster.
The mental model here is that your Go application hosts the etcd cluster member. It’s responsible for its lifecycle, its data directory, and its network bindings. This is powerful because it tightly couples your application’s state and its configuration store. You control when etcd starts and stops, and its data is managed alongside your application’s data.
The key levers you control are the embed.Config fields. Name identifies the member, Dir is crucial for persistence (if you omit it or don’t clean it up, etcd will resume from its last state), ClientURLs is how your application (and other clients) will talk to it, and PeerURLs are for clustering. HeartbeatInterval and ElectionTimeout are standard etcd tuning parameters that become directly configurable in your app. Security settings like ACMEProvider (for TLS) are also managed here.
What most people don’t realize is that when you embed etcd, you are bypassing the entire etcd daemon orchestration. You’re not dealing with systemd units, Docker Compose files, or Kubernetes StatefulSets. Your Go application is the etcd operator for its own instance. This means that if your Go application crashes, the embedded etcd instance crashes with it. There’s no independent process to restart.
The next concept you’ll likely grapple with is how to manage multiple embedded etcd instances for fault tolerance, which usually means moving back towards a clustered etcd setup, potentially still managed by your Go application but with more complex configuration.