Use connection draining (sending all new requests to your desired environment) and giving time for blue clients to fall off.
You can set the max-lifetime of your WebSocket (connection draining period has to be longer than max if that kind of reliability is required)
Otherwise would just handle the client-side. If WebSocket drops initiate a new connection through your AWS ELB to a healthy server. Do not keep any state on your ephemeral ELB backends. This will also work when scaling down on AWS.
If you want to learn more, check out this AWS Certification Course by Intellipaat.