Machine Learning in onprem Kubernetes

I am Newbie on ML , we wanted to try out Machine learning workloads on Kubernetes

how to install ML containers on K8S
we need a small dataset , perform training and show predictions

any blog / page will help

Personally I like tensor flow. There are premade tensorflow containers already here and there is plenty of knowledge already floating out there.

The container images are HUGE though, if you ever decide to refactor for the size, look into building your own container images later. I’ve had some success with the python:3-alpine images.

Thank you @protosam

any ML dataset you recommend to try out training and prediction

Check out keras.

Im a little late to this, but maybe check out kubeflow. I was looking at it just today myself.

Guessing you saw that post over on Reddit about people exposing the Kubeflow in an irresponsible way too? :smile:

1 Like

I haven’t seen that but I could definitely understand that. A lot of the examples just script exposing it via ingress, so I could see that being a big security hole lol.

Do you have a link to the post you are referring to? Would like to check it out.

It’s this reddit post. The article is mostly talking about crypto mining attacks in conjunction with kubeflow. It’s probably harder to detect than against most workloads because ML is CPU intensive to begin with. So your fancy prometheus fire alarms probably wont go off. The TLDR is not to expose your dashboards.

1 Like

oh wow!! LOL cutting corners didn’t seem to pay off well. And worse so in the cloud. jack that compute cost WAY up. thanks for the link!