Migrate from LoadBalancer services to Ingress resources
Currently, CiviInstances are exposed to the internet through LoadBalancer services created in pkg/reconciler/civiinstance/civiinstance.go. However, this is not cost effective as each LoadBalancer svc will spin up a new load balancer instance in whatever cloud provider is being used. Going forwards, we should use Ingress resources, with the nginx ingress controller providing the ingress gateway. Note: all of the tooling integrates with the nginx ingress, so we should use it here.
What this will involve is:
- For each CiviInstance, during reconciliation make the Service without specifying the type as LoadBalancer
- After making the service, create the Ingress resource and point it at the service
- Make sure that the external-dns and cert-manager annotations are set on the ingress resources, and not on the service