Curious about what your traffic in your Virtual Private Cloud (VPC) can tell you? Of course. Fortunately, AWS has just launched its VPC Flow Logs that provides data from the network interfaces in your VPCs. This new functionality captures accepted, rejected or all traffic flow information for the network interfaces in that resource. The network interfaces include the VPC itself, a subnet, or an Elastic Network Interface (ENI) in your account. Relevant network traffic is sent to Amazon CloudWatch Logs and can be retrieved and viewed there. And the cost? It’s free for VPC customers, except for applicable charges for logs stored in CloudWatch Logs.
Quick refresher – CloudWatch Logs was introduced almost a year ago to monitor and troubleshoot applications and systems in just about real-time, based on existing log files in your environment. CloudWatch Logs monitor your logs for specific phrases, values or patterns, and triggers alarms when specific thresholds are passed. Since CloudWatch is independent from your infrastructure, its metrics are available even when the instances themselves have been terminated.
VPC Flow Logs collects information about permitted and denied traffic, based on security group and network ACL rules. This data assists with network monitoring of entering and exiting traffic and drills down to specific ports, protocols, etc. Flow Logs also aid in troubleshooting situations; for example, to discover why specific traffic is not reaching an instance. Ascertaining that traffic is not received can lead to refinement of overly restrictive security group rules.
VPC Flow Log records are visible in the console after collection and processing that takes about 10 minutes. Multiple flow logs can send data to the same log group in CloudWatch Logs. This feature can be managed via the AWS Management Console, the CLI and the AWS SDK.
The new VPC Flow Logs are agentless, in contrast to the CloudWatch Logs that are agent-based and installed on Amazon Elastic Compute Cloud (EC2) instances. The CloudWatch agents are only able to provide data on network flows that are visible to the instance, and some overhead is required on each instance. The CloudWatch Logs console or API cannot be used to create log streams for your network interfaces.
Flow Log Records
Flow Log records contain the following values specifying the source, destination, and protocol, etc. for an Internet protocol (IP) flow:
Note that Flow Logs will not include:
- Traffic to Amazon DNS servers, including queries for private hosted zones
- Windows license activation traffic for licenses provided by Amazon
- Requests for instance metadata
- DHCP requests or responses
More information about VPC Flow Logs is available in the AWS documentation.