My PODs are stuck creating

General Discussions
,
1

Hello,

I have created a self managed cluster with AWS EC2 instances using kubadm manually.
I have used Calico network.

Everything was successfully installed however, my PODs are not created. They are stuck creating.

Below is the error. Please help.

Warning FailedCreatePodSandBox 49m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container “05fadc6a0a55aa22fde992dea9851fb169597f45238c9c6e916f69e3878e3bbf” network for pod “cust-db-mgmt-5d567dc6d4-567vd”: networkPlugin cni failed to set up pod “cust-db-mgmt-5d567dc6d4-567vd_sai” network: add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused”, failed to clean up sandbox container “05fadc6a0a55aa22fde992dea9851fb169597f45238c9c6e916f69e3878e3bbf” network for pod “cust-db-mgmt-5d567dc6d4-567vd”: networkPlugin cni failed to teardown pod “cust-db-mgmt-5d567dc6d4-567vd_sai” network: del cmd: error received from DelNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused”]
Normal SandboxChanged 3m56s (x211 over 49m) kubelet Pod sandbox changed, it will be killed and re-created.

Thanks,
Prasad

2

Hi,

Appreciate your quick help please, as I have a blocker here.

I don’t understand why this is reported.
transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused

Thanks
Prasad

3

Looking at you output, sounds like a miss permission configuration, maybe role policy or security group…

4

Thank you very much Weyder.
I think I have used the required policies and sec group.Infact, I have opened “ALL” for the sc. group for my both master and worker EC2 instances.

Kindly review the below and let me know if I am missing anything.

Thanks in advance.

Below are the policies attached to the master node role:
ElasticLoadBalancingFullAccess
AdministratorAccess
kubernetes-cluster-master-iam-policy
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“autoscaling:DescribeAutoScalingGroups”,
“autoscaling:DescribeLaunchConfigurations”,
“autoscaling:DescribeTags”,
“ec2:DescribeInstances”,
“ec2:DescribeRegions”,
“ec2:DescribeRouteTables”,
“ec2:DescribeSecurityGroups”,
“ec2:DescribeSubnets”,
“ec2:DescribeVolumes”,
“ec2:CreateSecurityGroup”,
“ec2:CreateTags”,
“ec2:CreateVolume”,
“ec2:ModifyInstanceAttribute”,
“ec2:ModifyVolume”,
“ec2:AttachVolume”,
“ec2:AuthorizeSecurityGroupIngress”,
“ec2:CreateRoute”,
“ec2:DeleteRoute”,
“ec2:DeleteSecurityGroup”,
“ec2:DeleteVolume”,
“ec2:DetachVolume”,
“ec2:RevokeSecurityGroupIngress”,
“ec2:DescribeVpcs”,
“elasticloadbalancing:AddTags”,
“elasticloadbalancing:AttachLoadBalancerToSubnets”,
“elasticloadbalancing:ApplySecurityGroupsToLoadBalancer”,
“elasticloadbalancing:CreateLoadBalancer”,
“elasticloadbalancing:CreateLoadBalancerPolicy”,
“elasticloadbalancing:CreateLoadBalancerListeners”,
“elasticloadbalancing:ConfigureHealthCheck”,
“elasticloadbalancing:DeleteLoadBalancer”,
“elasticloadbalancing:DeleteLoadBalancerListeners”,
“elasticloadbalancing:DescribeLoadBalancers”,
“elasticloadbalancing:DescribeLoadBalancerAttributes”,
“elasticloadbalancing:DetachLoadBalancerFromSubnets”,
“elasticloadbalancing:DeregisterInstancesFromLoadBalancer”,
“elasticloadbalancing:ModifyLoadBalancerAttributes”,
“elasticloadbalancing:RegisterInstancesWithLoadBalancer”,
“elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer”,
“elasticloadbalancing:AddTags”,
“elasticloadbalancing:CreateListener”,
“elasticloadbalancing:CreateTargetGroup”,
“elasticloadbalancing:DeleteListener”,
“elasticloadbalancing:DeleteTargetGroup”,
“elasticloadbalancing:DescribeListeners”,
“elasticloadbalancing:DescribeLoadBalancerPolicies”,
“elasticloadbalancing:DescribeTargetGroups”,
“elasticloadbalancing:DescribeTargetHealth”,
“elasticloadbalancing:ModifyListener”,
“elasticloadbalancing:ModifyTargetGroup”,
“elasticloadbalancing:RegisterTargets”,
“elasticloadbalancing:SetLoadBalancerPoliciesOfListener”,
“iam:CreateServiceLinkedRole”,
“kms:DescribeKey”
],
“Resource”: [
“*”
]
}
]
}

Below are the policies attached to the master node role:
ElasticLoadBalancingFullAccess
AdministratorAccess
kubernetes-cluster-worker-iam-policy
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“ec2:DescribeInstances”,
“ec2:DescribeRegions”,
“ecr:GetAuthorizationToken”,
“ecr:BatchCheckLayerAvailability”,
“ecr:GetDownloadUrlForLayer”,
“ecr:GetRepositoryPolicy”,
“ecr:DescribeRepositories”,
“ecr:ListImages”,
“ecr:BatchGetImage”
],
“Resource”: “*”
}
]
}

=========================================================================================================================================

5

Thank you very much Weyder.
I think I have used the required policies and sec group.Infact, I have opened “ALL” for the sc. group for my both master and worker EC2 instances.

Kindly review the below and let me know if I am missing anything.

Thanks in advance.

Below are the policies attached to the master node role:
ElasticLoadBalancingFullAccess

AdministratorAccess

kubernetes-cluster-master-iam-policy

{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“autoscaling:DescribeAutoScalingGroups”,
“autoscaling:DescribeLaunchConfigurations”,
“autoscaling:DescribeTags”,
“ec2:DescribeInstances”,
“ec2:DescribeRegions”,
“ec2:DescribeRouteTables”,
“ec2:DescribeSecurityGroups”,
“ec2:DescribeSubnets”,
“ec2:DescribeVolumes”,
“ec2:CreateSecurityGroup”,
“ec2:CreateTags”,
“ec2:CreateVolume”,
“ec2:ModifyInstanceAttribute”,
“ec2:ModifyVolume”,
“ec2:AttachVolume”,
“ec2:AuthorizeSecurityGroupIngress”,
“ec2:CreateRoute”,
“ec2:DeleteRoute”,
“ec2:DeleteSecurityGroup”,
“ec2:DeleteVolume”,
“ec2:DetachVolume”,
“ec2:RevokeSecurityGroupIngress”,
“ec2:DescribeVpcs”,
“elasticloadbalancing:AddTags”,
“elasticloadbalancing:AttachLoadBalancerToSubnets”,
“elasticloadbalancing:ApplySecurityGroupsToLoadBalancer”,
“elasticloadbalancing:CreateLoadBalancer”,
“elasticloadbalancing:CreateLoadBalancerPolicy”,
“elasticloadbalancing:CreateLoadBalancerListeners”,
“elasticloadbalancing:ConfigureHealthCheck”,
“elasticloadbalancing:DeleteLoadBalancer”,
“elasticloadbalancing:DeleteLoadBalancerListeners”,
“elasticloadbalancing:DescribeLoadBalancers”,
“elasticloadbalancing:DescribeLoadBalancerAttributes”,
“elasticloadbalancing:DetachLoadBalancerFromSubnets”,
“elasticloadbalancing:DeregisterInstancesFromLoadBalancer”,
“elasticloadbalancing:ModifyLoadBalancerAttributes”,
“elasticloadbalancing:RegisterInstancesWithLoadBalancer”,
“elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer”,
“elasticloadbalancing:AddTags”,
“elasticloadbalancing:CreateListener”,
“elasticloadbalancing:CreateTargetGroup”,
“elasticloadbalancing:DeleteListener”,
“elasticloadbalancing:DeleteTargetGroup”,
“elasticloadbalancing:DescribeListeners”,
“elasticloadbalancing:DescribeLoadBalancerPolicies”,
“elasticloadbalancing:DescribeTargetGroups”,
“elasticloadbalancing:DescribeTargetHealth”,
“elasticloadbalancing:ModifyListener”,
“elasticloadbalancing:ModifyTargetGroup”,
“elasticloadbalancing:RegisterTargets”,
“elasticloadbalancing:SetLoadBalancerPoliciesOfListener”,
“iam:CreateServiceLinkedRole”,
“kms:DescribeKey”
],
“Resource”: [
“*”
]
}
]
}

Below are the policies attached to the master node role:

ElasticLoadBalancingFullAccess

AdministratorAccess

kubernetes-cluster-worker-iam-policy

{

“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“ec2:DescribeInstances”,
“ec2:DescribeRegions”,
“ecr:GetAuthorizationToken”,
“ecr:BatchCheckLayerAvailability”,
“ecr:GetDownloadUrlForLayer”,
“ecr:GetRepositoryPolicy”,
“ecr:DescribeRepositories”,
“ecr:ListImages”,
“ecr:BatchGetImage”
],
“Resource”: “*”
}
]
}