Service Account Access control

Is there a way we can restrict a pod not to use a particualr service account, we give all the access to a service account and any pods running in the namespace can simply pretend themselves as using that service account. so even if we give control w.r.t service account level, any deployment can use the service account in the same namespace and get all the access. How can we solve this.?