Upstream Connect Error or Disconnect/Reset Before Headers

The error message “upstream connect error or disconnect/reset before headers” indicates that the connection between the client and the server was closed before the server could send a response.

The error message “protocol error” indicates that there was an issue with the protocol used to communicate between the client and the server for the health check endpoint.

This could be caused by a number of issues, such as a misconfigured firewall (if any), a network issue, or a problem with the server itself.

Generally speaking, to troubleshoot this issue, you can try the following steps:

1. Check the firewall settings to ensure that the port used by the container app is open.
2. Check the network settings to ensure that there are no issues with the network.
3. Check the logs of the container app to see if there are any errors or issues that could be causing the problem.
4. Try accessing the container app from a different network or location to see if the issue is specific to your current network.

I was facing the same issue, then I read about protocol selection. I realised the name of the port in the service definition should start with for example http-. This fixed the issue for me. And . if you face the issue still you might need to look at the tls-check for the pods and resolve it using destinationrules and policies.

One more reason to get this issue is that the k8s service is not able to connect with the container because it may be exposed to different port. Validate the port exposed in container. eg:

K8s service Port: ports: – name: http port: 8080 protocol: TCP targetPort: 8080

Container running inside pod should also exposed to 8080

0.6 so now available. If you are encountering this issue with a version early then Kubeflow 0.6 please try upgrading to 0.6.

Per the discussion in this thread we think this might have been due to an issue with envoy and/or Ambassador.

With 0.6 we’ve switched to ISTIO and are using much newer versions of envoy. So we need to see if this problem still reoccurs and then we can try to track it down.