First of all, RLPx is a suite of protocols and not just a single protocol. On one hand it maintains the p2p overlay network with the means of a modified version of the kademlia node discovery protocol using UDP messages.
The first time two peers meet each other they should perform a two-phase handshake (i.e. two different handshakes). The first handshake pertains to the exchange of some cryptographic secrets that are used for the subsequent communications between them. The subsequent messages are therefore encrypted and authenticated.
In the second handshake, they exchange their capabilities, i.e. which devp2p subprotocols they support. It could be considered a part of devp2p wire protocol (the Hello message). The documentation says that:
the ÐΞVp2p nodes communicate by sending messages using RLPx (this time using TCP messages). I list some devp2p wire subprotocols that are widely used:
eth, the ethereum wire subprotocol used to exchange the information about the blockchain
whisper
les the light ethereum subprotocol is the protocol designed for the light clients
...If you want you can write your own subprotocol https://github.com/ethereum/go-ethereum/wiki/Peer-to-Peer.
Therefore RLPx messages are encrypted and authenticated. On top of them, you can communicate with different protocols. We can say that RLPx is the transport layer protocol and devp2p and its subprotocols are simply application-level protocols.
If you want to be an expert in your field, then enroll in Blockchain Certification by Intellipaat.