Office Hours for 21 November

General Discussions
1

It’s that time again! Kubernetes Office Hours is our monthly livestream where Kubernetes developers answer user questions live on the air. We’ll use this thread as a place to collect questions, so if you’re stuck then this would be a good time to ask for help!

This week we’ll have Matt Farina from Helm during the morning session, so if you have Helm questions feel free to pile 'em up.

And as usual if we discuss your question on line you’ll automatically be entered in a raffle to win a Kubernetes tshirt.

Check out the link for more info and subscribe to this thread if you want to be notified of new activity.

2

Just a reminder to everyone that tomorrow will be the office hours, feel free to link your unanswered question here or give us a link.

3

Alright here’s the video from the first session:

And here are the notes, they are not as useful by themselves but act as a agenda item as it’s the order in which we ask the questions. I’ll leave this post as a wiki if someone wants to help timestamp these!

Office Hours 2018-11-21

Your hosts: Bob Killen, Jorge Castro, Puja Abbassi, Joel Speed, and featuring Matt Farina.

Name: Sagar
Source: PM to Jorge
Question: HI jorge, does minikube version: v0.30.0 support docker version 18.09.0?? I was trying to install minikube on my ubuntu virtual machine, but it says that it doesn’t support docker version 18.09.0
Answer: Are you using vm-driver=none? Current default for minikube ISOs is 18.06.1, what is the exact error message? Could potentially roll your own ISO.

Name: jimcm
Source: Office Hours Slack
Question: Hi, I’d like to submit a question for office hours. I’m a newbie kubernetes user, and am having a problem getting kubernetes up and running on my single node cluster. Problem description here: https://stackoverflow.com/questions/53309671/coredns-in-crashloopbackoff-kubernetes-1-11 . I followed the debugging steps at Troubleshooting - Kubernetes but that didn’t help solve my problem. My question: what are suggested next steps for debugging? e.g. how to turn on additional tracing/logging, how to snoop traffic on the pod network, etc.

Answer: It looks like it was deployed with kubeadm, but generally referencing the API server should be the host’s IP and not a clusterIP service? Can turn up debugging and take a look.

Jorge: ask ilya to take a look

Name: Simon Gottschlag
Source: Office Hours Slack
Question: Question for Matt during office hours: We are using Ansible for most of the things we do. We’ve been seeing some issues and read about more when it comes to the helm module for Ansible. Do you see that module as something that is recommended for production or is it deprecated? If we shouldn’t use the module, how do you recommend using Helm with Ansible? Thanks!
Answer: Get Darren a tshirt.

Name: Brian
Source: Office Hours Slack
Question: Question: I’ve used managed clusters and local single node clusters. Now I want a 3 node cluster (primarily targeted at kubeflow/app dev) what is the suggested path of least resistance? Is Centos 7 vm’s (master: “memsize”: “8192”, “vcpus”: “8”; slaves half that) configured with kubespray a good path? Thanks for having office hours!
Answer:

Jorge: Find a cluster-API person, here’s a good demo: https://www.youtube.com/watch?v=F8ZTNFkfYOE&feature=youtu.be

Name: benji
Source: Office Hours Slack
Question: Could you guys describe why Federationv1 has been stopped and how Federationv2 is diffrent/better ?
Answer: https://old.reddit.com/r/kubernetes/comments/9moz0r/what_is_wrong_with_federation_v1_and_what_makes/e7hmsb8/

Check out: https://github.com/bookingcom/shipper

Name: Anand Singh Kunwar
Source: Office Hours Slack
Question: Anyone knows a way to get a secret in environment variables in a statefulset where each secret/key pair corresponds to one of the ordinal indices of the statefulset pod?
Answer:

K8Crypto [9:27 AM]
Speaking of ENVs, is there a reason why some pods in same namespace see other pod envs ?
Are those coming from Secrets?

Anand Singh Kunwar [9:27 AM] So you have use an initContainer as well to create another env variable?

https://kubernetes.io/docs/concepts/services-networking/service/#discovering-services

Name: jimangel
Source: Office Hours Slack
Question: Hey Matt, looking for an elevator pitch of what’s coming in helm v3, ETA, and any major roadblocks / how we can help?
Answer:

Name: eduard-t
Source: Office Hours Slack
Question: what is your experience with bringing internet traffic into a baremetal cluster? Currently using keepalived vip pointing to a ingress controller nodeport service. Is there a better way to do this?
Answer:

Name: darren
Source: Office Hours Slack
Question: What are some of the “must have” applications you use to help manage / interact with your kubernetes clusters? Just to give an example of ones I find really useful:
• https://github.com/wercker/stern - for tailing logs from multiple containers
• https://www.telepresence.io/ - for joining your kubernetes network to your local machine
We’re on the cusp of delivering our first k8s cluster our developers and would like suggestions for making it easier on them.

We use chartmuseum for hosting our helms…is there any way to force overwrite of a chart with the same version? We use the push plugin for helm and I’m not sure if that functionality exists.
Answer:

Name: K8Crypto
Source: Office Hours Slack
Question: Speaking of ENVs, is there a reason why some pods in same namespace see other pod envs ?
Are those coming from Secrets?
Answer: Services - Kubernetes

Name: Simon Gottschlag
Source: Office Hours Slack
Question: Question: Can I use Istio as “Network Policy” between PODs? I want to limit what services a POD can access. I’m able to do it using ServiceEntry for egress traffic, but not sure about inter-pod communication
Answer:

Name: darren
Source: Office Hours Slack
Question: What are some of the “must have” applications you use to help manage / interact with your kubernetes clusters? Just to give an example of ones I find really useful:
• https://github.com/wercker/stern - for tailing logs from multiple containers
• https://www.telepresence.io/ - for joining your kubernetes network to your local machine
We’re on the cusp of delivering our first k8s cluster our developers and would like suggestions for making it easier on them.

Name: benji
Source: Office Hours Slack
Question: Follow up about the fedv2 question: I installed a Federation with kubefed2 but unlike federationv1 there is no context for the federation itself , will this be implemented in the future?
Answer:

Name: Mark
Source: Office Hours Slack
Question: Question: I am really wondering what the best course of action is to orchestrate global kubernetes clusters. We have global presence and managing cluster by cluster seems fine for a few clusters, but not for 20+ global clusters. Any ideas? With global presence we mean multiple isolated clusters in several AWS regions.
Answer:

Name: eduard-t
Source: Office Hours Slack
Question: Hi! what is your experience with bringing internet traffic into a baremetal cluster? Currently using keepalived vip pointing to a ingress controller nodeport service. Is there a better way to do this?
Answer:

Name: Simon Gottschlag
Source: Office Hours
Question: Question: How are you handling id_tokens with --token=? We are using powershell where we extract the token manually ($idtoken = Get-AdfsIdToken) and then run it with --token=$idtoken. It works, but feels clonky. Any better way of handling this?
Answer:

Name: Snowcrash
Source: Office Hours Slack
Question: I’ve been struggling with this problem:
kube-system: Pod Warning FailedScheduling default-scheduler no nodes available to schedule pods for days. There’s a StackOverflow post about it here: https://stackoverflow.com/questions/53381739/kube-system-pod-warning-failedscheduling-default-scheduler-no-nodes-available-t
Answer:

Talk to errordeveloper about eksctl

Name: Nosfert
Source: Office Hours Slack
Question: Question: First, do you think that there is an issue in having replicas defined in the yml? The usecase that we have seen is that we do re-deploys by using the whole file…(this might be wrong/bad) Any and all replica changes that have occurred will then be reseted to the default in the file (if a replica set is defined).
If yes/no, please explain how you would do it.
Answer:

4

And here’s the video for the 2nd session and notes:

Office Hours 2018-11-21 (West Coast Edition)

Your hosts: Bob Killen, Jorge Castro, Jeffrey Sica, Mario Loria, Chad Moon

Name: xqc1
Source: #office-hours
Question: Question: What are best practices for monitoring Kubernetes with Prometheus? Isn’t it somewhat a problem that when the cluster fails the prometheus running inside it might fail too? Would it be possible to monitor multiple clusters from a special monitoring cluster or how do people actually manage that problem?
Answer: prometheus operator + aggregate out, possibly store with M3/Thanos
Jorge: reach out to prometheus operator people.

Name: sanwar
Source: #office-hours
Question: I’m trying to deploy sentry through helm with helm install --name sentry stable/sentry .
The postgresql pod which is bundled with this is failing withe the error
initdb: could not change permissions of directory "/var/lib/postgresql/data/pgdata": Operation not permitted
Would anyone be able to help with resolving this ?


Answer:

Name: Evesy
Source: #office-hours
Question: :wave: Hello once again – So looking at running Kafka in Kubernetes, and struggling with external access. Clients need to connect to the kafka service and then to a designated broker – How can I make these individual statefulset pods (behind a headless service) externally accessible (GKE fwiw)?
Answer: https://itnext.io/exposing-statefulsets-in-kubernetes-698730fb92a1

Name: mitchellmaler
Source: #office-hours
Question: When looking at multi region deployments would it better to just have multiple clusters and deploy to each one or have single large clusters with nodes (pools) in different regions.
Answer:

Name: Dan Manners
Source: #office-hours
Question: Hello! This is definitely more of an opinion quesiton at this point. I’m using EKS and looking to multi-tenant services for various clients. Do you feel that Kubernetes is mature enough at this point in time to support multi-tenancy in a safe/secure way between namespaces and various CNI’s, or would you recommend separate clusters entirely?
Answer: 11 Ways (Not) to Get Hacked - Kubernetes

Name: Evesy
Source: #office-hours
Question: I can float one more question – Does anyone have an exact understanding on helm + rollbacks. It seems like a failed upgrade can leave the cluster in a state where helm has created resources but is not tracking them – Forward fix and helm fails because the resource it tries to create already exists, rollback and it doesn’t clean up those newer resources.
Answer:

Name: soggy
Source: #office-hours
Question: Something that comes up occasionally: How have y’all handled certificate rotations for the root K8s certificate?
Answer:

Name: Dan Manners
Source: #office-hours
Question: While on the topic of certificates, have any of you leveraged Hashicorp Vault and the PKI backend for certificates, or do you have a preferred toolset for PKI management?
Answer:

Name: Evesy
Source: #office-hours
Question: Does anyone bother with priority class on pods, especially if you have cluster autoscaling at your disposal?
Answer:

Name: Dan Manners
Source: #office-hours
Question: Any highly recommended classes/conferences/sessions you’d recommend to check out at Kubecon?
Answer:

Name: Dave
Source: #kubernetes-users
Question: Hello, Is there a quick and dirty way to convert a node to a master in kubernetes? I added this to the labels: node-role.kubernetes.io/master: "" and this to the specs: ` taints:

Name: ondrej
Source: #kubernetes-users
Question: hey guys, what part of kubernetes is responsible for telling nginx-lego what service nginx should try to direct the traffic?
Answer:

Name: techcanuck
Source: #kubernetes-users
Question: Question on kubectl/POD deployment - has anyone ever seen it when PODS just STOP deploying in deployments? I can’t provision anything new (and with no errors to the controllers) - scaling up/down has ZERO effect. Any thoughts? Been at this one for hours and can’t see any resource constraints or other that would suggest this. Even new deployments state 1 desired, 0 available. They just don’t start :disappointed:
Answer:

Name: David
Source: #kubernetes-novice
Question:
With distributed block storage solutions does the disk speed e.g. raid sas or ssd matter? Or does the software layer in between make their speed similar? (edited)
Answer:

Name: raine
Source: #kubernetes-users
Question: raine [12:22 PM]
what does updating the image of a deployment typically look like, like when you want to start running a new version of an image? if I use kubectl set image, my yaml file won’t be in sync with the current state anymore, and I can’t apply it in case there are other changes later. not sure how to automate this
Answer:

Name: gsantovena
Source: #kubernetes-users
Question: hello everyone, is there a way to know how/when/why a node changed its status from “Ready” to “Ready,SchedulingDisabled”?
Answer:

Name: xdexter
Source: #kubernetes-users
Question:
Hello, my raills application has 2 important files, database.yml and configuration.yml, i don’t want store this files in git repo for security reasons, what other solution i have in Kubernetes? Using secrets stored in namespace?
Answer:

Name: bradley.hession
Source: #kubernetes-users
Question: I’m seeing high latency for PATCH operations on my API server(s) - disks are fine(low iowait), network fine, cpu 10%.
All other API latencies seem fine
what can i do to troubleshoot this further?
Answer:

Name: artem
Source: #kubernetes-users
Question: Hi, what is canonical way for code running in pod to learn pods IP address ? I can see IP with ifconfig inside pod and kuberctl describe pod but I do not see that IP in environment variables.
Answer:

Name: Evesy
Source: #office-hours
Question: Does anyone bother with priority class on pods, especially if you have cluster autoscaling at your disposal?
Answer:

Name: Dan Manners
Source: #office-hours
Question: Any highly recommended classes/conferences/sessions you’d recommend to check out at Kubecon?
Answer: