Hi hannesknutsson:
It is hard for me to provide any deal-breaker advice on all the questions you provide, but at least regarding etcd, in the documentation they answer the What is the maximum cluster size (for etcd):
Theoretically, there is no hard limit. However, an etcd cluster probably should have no more than seven nodes. Google Chubby lock service, similar to etcd and widely deployed within Google for many years, suggests running five nodes. A 5-member etcd cluster can tolerate two member failures, which is enough in most cases. Although larger clusters provide better fault tolerance, the write performance suffers because data must be replicated across more machines.
As Kubernetes supports clusters up to 5000 nodes Best Practices for Large Clusters, this would rule-out the option to set an “homogeneous” cluster (with all nodes having simultaneously control plane and worker roles, if every node hosts an etcd member).
It is possible to do so for small clusters (3 to 5 (or even 7) nodes) before the chattiness of the etcd quorum process becomes a problem.
Etcd clusters also need highly performant storage, so it may be better to provide specialised resources to the etcd nodes while the regular control plane nodes have less optimised (and probably cheaper) resources.
From the operations point of view, removing the etcd component from the control plane nodes will make them stateless (as far as I know), so they would be easier to maintain…
I hope it helps you understand a little bit better the need for separating the control plane and etcd “nodes” if you still thinking about it.
Best regards,
Xavi