The EC2 serial console is your last-ditch effort to debug an EC2 instance that’s completely unresponsive, even to SSH. It’s like having a direct keyboard and monitor attached to your server, but over a network.
Here’s how you can use it to revive a stubborn instance.
First, you need to enable access to the serial console for your AWS account. This is a security measure, so it’s not on by default.
aws ec2 enable-serial-console-access
This command doesn’t output anything if successful, but it flips the switch in your account settings. Without this, any attempt to connect will just fail with a generic permission error.
Next, you need to set up an IAM policy that grants specific IAM users or roles permission to connect to the serial console. This is crucial for security; you don’t want just anyone getting console access.
Here’s an example IAM policy. Make sure to replace i-0123456789abcdef0 with the actual ID of your instance.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:StartSerialConsoleSession",
"ec2:SendSerialConsoleSessionCommand",
"ec2:DescribeSerialConsoleSessions"
],
"Resource": "arn:aws:ec2:us-east-1:123456789012:instance/i-0123456789abcdef0"
},
{
"Effect": "Allow",
"Action": "ec2:DescribeInstances",
"Resource": "*"
}
]
}
Attach this policy to the IAM user or role that needs serial console access. The ec2:DescribeInstances action is needed so the user can see the instance they are trying to connect to.
Now, let’s actually connect to the serial console. You’ll use the AWS CLI for this.
aws ec2 start-serial-console-session --instance-id i-0123456789abcdef0
This command initiates a session. If successful, it will output a SerialConsoleSession object with a SessionId. Keep this SessionId handy.
The output will look something like this:
{
"SerialConsoleSession": {
"SessionId": "t-0123456789abcdef0",
"InstanceId": "i-0123456789abcdef0",
"State": "connected",
"StreamUrl": "wss://console-serial-port.us-east-1.amazonaws.com/...",
"DestinationUrl": "..."
}
}
The SessionId is what you’ll use to interact with the session. The StreamUrl is a WebSocket URL, which you can’t directly use with the AWS CLI. You’ll use a separate command to send commands.
To send commands to the instance, you use send-serial-console-session-command.
aws ec2 send-serial-console-session-command --instance-id i-0123456789abcdef0 --session-id t-0123456789abcdef0 --command "ls -l /tmp"
This command sends the ls -l /tmp command to the instance. The output you get back will be the standard output and standard error from that command executed on the instance.
If you want to get a more interactive session, you can use a tool like websocat (or wscat if you have it installed) to connect to the StreamUrl obtained from start-serial-console-session.
First, get the StreamUrl:
aws ec2 start-serial-console-session --instance-id i-0123456789abcdef0 --output json | jq -r '.SerialConsoleSession.StreamUrl'
Then, pipe that URL into websocat:
websocat <paste_stream_url_here>
Once connected via websocat, you’ll see the boot messages, login prompts, or any output your instance is generating on its console. You can then type commands directly as if you were at a physical console.
Common use cases for the serial console include:
- Fixing GRUB/bootloader issues: If your instance won’t boot past the initial bootloader, you can often interact with GRUB via the serial console to select a different kernel or repair its configuration.
- Resetting passwords: If you’ve lost root access and can’t SSH in, you can often boot into a rescue mode or use a live CD environment (if you’ve pre-configured your AMI for this) to reset the root password.
- Troubleshooting network configuration: If a bad network configuration is preventing SSH, you can use the serial console to correct it.
- Debugging kernel panics: If your instance is experiencing a kernel panic, the serial console will often display the panic message, providing clues to the underlying issue.
- Rebooting into single-user mode: For in-depth system maintenance or recovery, you might need to reboot into single-user mode. This is achievable via serial console interaction with the bootloader.
Remember, the serial console is a powerful tool but requires careful handling. Mistakes made here can render your instance even more inaccessible. Always have a backup plan and ensure your IAM policies are as restrictive as necessary.
If you successfully fix your instance and can SSH into it again, you might find yourself needing to manage the lifecycle of your EC2 instances more effectively, which often leads to exploring tools like AWS Systems Manager Run Command or AWS OpsWorks.