Amazon Cloud Service (AWS)
Containers/Elastic Container Service (ECS)
1.1. ECS: Enabling Amazon ECS Exec
Problem
note
Default Amazon ECS Exec is disabled in the ECS cluster and we can not execute the command in the container to debug the application, eg: healthcheck, log, etc.
Solution
tip
Since 2021-03-16, Amazon ECS Exec is available for all ECS clusters https://aws.amazon.com/blogs/containers/new-using-amazon-ecs-exec-access-your-containers-fargate-ec2/
Preparation
Step 1: Update IAM Trust Relationship
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Step 2: Update IAM Role Policies
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel"
],
"Resource": "*"
}
]
}
Step 3: Update ECS Task Definition
- Access ECS and Update Task Definition
- Ensure you add the following JSON snippet under containerDefinitions to enable
initProcessEnabled
{
"family": "foo-service",
"containerDefinitions": [
{
...
"volumesFrom": [],
"linuxParameters": {
"initProcessEnabled": true
},
...
"healthCheck": {
"command": [
"CMD-SHELL",
"curl -f http://localhost/healthcheck || exit 1"
],
"interval": 90,
"timeout": 10,
"retries": 3,
"startPeriod": 10
},
"systemControls": []
}
]
}
Step 4: Enable Execution Command in ECS Cluster
- After updating the task definition, we need to update the ECS service to enable the execution command.
{
export ECS_CLUSTER_NAME='foo-cluster'
export ECS_SERVICE_NAME='foo-service'
export AWS_REGION='us-east-2'
aws ecs update-service \
--cluster $ECS_CLUSTER_NAME \
--service $ECS_SERVICE_NAME \
--region $AWS_REGION \
--enable-execute-command \
--force-new-deployment
}
- Check if the execution command is enabled, the output result in showing that is
true
{
export ECS_CLUSTER_NAME='foo-cluster'
export ECS_SERVICE_NAME='foo-service'
export AWS_REGION='us-east-2'
aws ecs describe-services \
--cluster $ECS_CLUSTER_NAME \
--service $ECS_SERVICE_NAME \
--region $AWS_REGION \
| jq '.services[0].enableExecuteCommand'
}
Execution
- Access the ECS console and click on the ECS cluster and select the ECS service to get ECS task arn
{
export ECS_CLUSTER_NAME='foo-cluster'
export AWS_REGION='us-east-2'
aws ecs describe-services \
--cluster $ECS_CLUSTER_NAME \
--service $ECS_SERVICE_NAME \
--region $AWS_REGION \
| jq '.services[0].events[].message'
}
- Execute the command to access the container
{
export ECS_CLUSTER_NAME='foo-cluster'
export ECS_TASK_ARN='arn:aws:ecs:us-east-2:123456789012:task/foo-cluster/1234567890123456789'
export AWS_REGION='us-east-2'
aws ecs execute-command \
--cluster $ECS_CLUSTER_NAME \
--task $ECS_TASK_ARN \
--container foo-container \
--interactive \
--command "/bin/bash"
}
- If we get the error message, so need to install the SessionManagerPlugin
danger
SessionManagerPlugin is not found. Please refer to SessionManager Documentation here: http://docs.aws.amazon.com/console/systems-manager/session-manager-plugin-not-found
Reference
- NEW – Using Amazon ECS Exec to access your containers on AWS Fargate and Amazon EC2
- A Step-by-Step Guide to Enabling Amazon ECS Exec
- Install the Session Manager plugin on macOS
Containers/Amazon Elastic Kubernetes (EKS)
Cloud Formation/Cloud Development Kit (CDK)
1.1. CDK: Init Development Environment
Problem
note
We need to create a new CDK project and initialize the development environment. In development environment will use dedicated VPC, Subnet, Security Group, etc. to avoid conflict with the existing environment. Also with private and public subnets, we can deploy the application in a secure way.
What and why CDK? https://www.pluralsight.com/blog/cloud/what-is-aws-cdk-cloud-development
Solution
tip
1.1.1 Preparation
- Current version of CDK
{
nvm use v18.16.0
cdk version
# 2.125.0 (build 5e3c3c6)
}
1.1.2 Folder Structure
- Separated folder for each region and each environment, example environment
developmentin regionus-east-2, in future if we have new environment, examplestaging, will create new folder under regionus-east-2too. Also same with AWS components, it's separated folder too, examplenetworking-content-delivery/vpc
Folder Structure
├── README.md
├── bin
│ ├── mimic.d.ts
│ └── mimic.ts
├── cdk.context.json
├── cdk.json
├── jest.config.js
├── lib
│ ├── us-east-2
│ │ └── development
│ │ ├── compute
│ │ │ ├── ec2
│ │ │ │ ├── bastion
│ │ │ │ │ ├── mimic.d.ts
│ │ │ │ │ └── mimic.ts
│ │ │ │ └── verifyVpcEndpoint
│ │ │ │ ├── mimic.d.ts
│ │ │ │ └── mimic.ts
│ │ │ └── elb
│ │ │ ├── mimic.d.ts
│ │ └── networking-content-delivery
│ │ └── vpc
│ │ ├── mimic.d.ts
│ ├── us2-development-stack.d.ts
│ ├── us2-development-stack.ts
│ └── utils
│ ├── index.d.ts
│ ├── index.ts
│ └── interfaces
│ └── development.ts
├── package-lock.json
├── package.json
├── test
│ ├── mimic.test.d.ts
│ └── mimic.test.ts
└── tsconfig.json
1.1.3 Init New VPC
-
Create a new VPC with public and private subnets, also with NAT Gateway and Internet Gateway
- Naming convention for the VPC, Subnet, Security Group, etc. is using the following format:
tagProjectName-tagEnvironment-tagComponentName
- Naming convention for the VPC, Subnet, Security Group, etc. is using the following format:
btin/lib/us-east-2/development/networking-content-delivery/vpc/mimic.ts
// list of tasks need to do to create VPC
...
private initialize() {
this.createRole();
this.createVpcLogGroup();
this.createVpc();
this.enableFlowLogs()
this.createVpcGatewayEndpoint()
}
...
private createVpcLogGroup() {
const logGroup = new logs.LogGroup(this.scope, 'Us2DevelopmentVpcLogGroup', {
logGroupName: `${this.props.suffixName}/aws/vpc/vpc-flow-logs`,
retention: logs.RetentionDays.ONE_MONTH,
removalPolicy: RemovalPolicy.DESTROY
});
new CfnOutput(this.scope, 'Us2DevelopmentVpcLogGroupName', { value: logGroup.logGroupName });
new CfnOutput(this.scope, 'Us2DevelopmentVpcLogGroupArn', { value: logGroup.logGroupArn });
this.logGroup = logGroup;
}
private createRole() {
const role = new iam.Role(this.scope, 'Us2DevelopmentVpcFlowLogRole', {
assumedBy: new iam.ServicePrincipal('vpc-flow-logs.amazonaws.com'),
roleName: `${this.props.suffixName}-vpc-flow-log-role`
});
new CfnOutput(this.scope, 'Us2DevelopmentVpcFlowLogRoleArn', { value: role.roleArn });
new CfnOutput(this.scope, 'Us2DevelopmentVpcFlowLogRoleName', { value: role.roleName });
this.role = role;
}
private createVpc() {
const vpc = new ec2.Vpc(this.scope, 'Us2DevelopmentVpc', {
vpcName: `${this.props.suffixName}-vpc`,
maxAzs: this.props.vpcMaxAzs,
natGateways: this.props.vpcNatGateways,
enableDnsHostnames: true,
enableDnsSupport: true,
ipAddresses: ec2.IpAddresses.cidr(this.props.vpcCidr),
subnetConfiguration: [
{
cidrMask: this.props.vpcSubnetMask,
name: `${this.props.tagProjectName}-${this.props.tagEnvironment}-public`,
subnetType: ec2.SubnetType.PUBLIC,
},
{
cidrMask: this.props.vpcSubnetMask,
name: `${this.props.tagProjectName}-${this.props.tagEnvironment}-private`,
subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS
}
]
});
new CfnOutput(this.scope, 'Us2DevelopmentVpcId', { value: vpc.vpcId });
new CfnOutput(this.scope, 'Us2DevelopmentVpcCidr', { value: vpc.vpcCidrBlock });
new CfnOutput(this.scope, 'Us2DevelopmentVpcPublicSubnets', { value: vpc.publicSubnets.map(subnet => subnet.subnetId).join(',') });
new CfnOutput(this.scope, 'Us2DevelopmentVpcPrivateSubnets', { value: vpc.privateSubnets.map(subnet => subnet.subnetId).join(',') });
this.vpc = vpc;
}
private enableFlowLogs() {
const flowLog = new ec2.FlowLog(this.scope, 'Us2DevelopmentVpcFlowLog', {
resourceType: ec2.FlowLogResourceType.fromVpc(this.vpc),
destination: ec2.FlowLogDestination.toCloudWatchLogs(this.logGroup, this.role)
});
new CfnOutput(this.scope, 'Us2DevelopmentVpcFlowLogId', { value: flowLog.flowLogId });
this.flowLog = flowLog;
}
private createVpcGatewayEndpoint() {
const vpcEndpoint = new ec2.GatewayVpcEndpoint(this.scope, 'Us2DevelopmentVpcGatewayEndpoint', {
service: ec2.GatewayVpcEndpointAwsService.S3,
vpc: this.vpc,
subnets: [
{ subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS }
]
});
new CfnOutput(this.scope, 'Us2DevelopmentVpcGatewayEndpointId', { value: vpcEndpoint.vpcEndpointId });
}
Containers/Nat
1.1. Nat: Nat Gateway and Nat Instances
note
Nat Gateway and Nat Instances are used to allow the private subnet to access the internet. The main difference between Nat Gateway and Nat Instances is the performance and the cost. Nat Gateway is managed by AWS and it's more reliable and faster than Nat Instances. Nat Instances are EC2 instances that we need to manage by ourselves.
Reference: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-comparison.html
1.1.1 Comparison
| Attribute | NAT GateWay | Nat Instance |
|---|---|---|
| Availability | Highly available. NAT gateways in each Availability Zone are implemented with redundancy. Create a NAT gateway in each Availability Zone to ensure zone-independent architecture. | Use a script to manage failover between instances. |
| Bandwidth | Scale up to 100 Gbps. | Depends on the bandwidth of the instance type. |
| Maintenance | Managed by AWS. You do not need to perform any maintenance. | Managed by you, for example, by installing software updates or operating system patches on the instance. |
| Performance | Software is optimized for handling NAT traffic. | A generic AMI that's configured to perform NAT. |
| Cost | Charged depending on the number of NAT gateways you use, duration of usage, and amount of data that you send through the NAT gateways. | Charged depending on the number of NAT instances that you use, duration of usage, and instance type and size. |
| Type and size | Uniform offering; you don’t need to decide on the type or size. | Choose a suitable instance type and size, according to your predicted workload. |
| Public IP addresses | Choose the Elastic IP address to associate with a public NAT gateway at creation. | Use an Elastic IP address or a public IP address with a NAT instance. You can change the public IP address at any time by associating a new Elastic IP address with the instance. |
| Private IP addresses | Automatically selected from the subnet's IP address range when you create the gateway. | Assign a specific private IP address from the subnet's IP address range when you launch the instance. |
| Security groups | You cannot associate security groups with NAT gateways. You can associate them with the resources behind the NAT gateway to control inbound and outbound traffic. | Associate with your NAT instance and the resources behind your NAT instance to control inbound and outbound traffic. |
| Network ACLs | Use a network ACL to control the traffic to and from the subnet in which your NAT gateway resides. | Use a network ACL to control the traffic to and from the subnet in which your NAT instance resides. |
| Flow logs | Use flow logs to capture the traffic. | Use flow logs to capture the traffic. |
| Port forwarding | Not supported. | Manually customize the configuration to support port forwarding. |
| Bastion servers | Not supported. | Use as a bastion server. |
| Traffic metrics | View CloudWatch metrics for the NAT gateway. | View CloudWatch metrics for the instance. |
| Timeout behavior | When a connection times out, a NAT gateway returns an RST packet to any resources behind the NAT gateway that attempt to continue the connection (it does not send a FIN packet). | When a connection times out, a NAT instance sends a FIN packet to resources behind the NAT instance to close the connection. |
| IP fragmentation | Supports forwarding of IP fragmented packets for the UDP protocol. Does not support fragmentation for the TCP and ICMP protocols. Fragmented packets for these protocols will get dropped. | Supports reassembly of IP fragmented packets for the UDP, TCP, and ICMP protocols. |
1.1.2 Nat Gateway
1.1.3 Nat Instances
In this case, we will focus into Nat Instances to alternative solution for Nat Gateway in private subnet.