Introduction to Parsing WebSocket Protocol Binary from a TCP Socket while Using WebSocket Python
Parsing WebSocket protocol binary from a TCP Python socket can provide more control and insight into WebSocket communication. While libraries like WebSockets in Python handle most of the work, manual parsing allows for custom optimizations and troubleshooting specific scenarios.
Affiliate: Experience limitless no-code automation, streamline your workflows, and effortlessly transfer data between apps with Make.com.
It also gives developers a deeper understanding of the protocol and the ability to handle unique or edge cases that standard implementations may not cover. In certain applications, having this level of control can help when dealing with custom or unconventional use cases.
Furthermore, manual parsing can be critical in security-sensitive environments where deep packet inspection and validation are needed. Understanding how WebSocket frames are structured and parsed also provides the flexibility to integrate with low-level systems and protocols that may not be compatible with existing WebSocket Python libraries.
This approach can also be handy for educational purposes, providing hands-on experience in how the protocol operates at a fundamental level. Mastering these concepts can lead to better decision-making when implementing WebSocket communication in various projects, including those requiring high-performance optimizations.
Why Parse WebSocket Protocol Manually When Python Libraries Already Exist?
Understanding the underlying WebSocket protocol helps debug, optimize, and handle custom use cases where standard libraries may not fit. Manual parsing allows developers to gain granular control over the connection, identify potential security issues, and create unique features that would be challenging with pre-built libraries.
Certain situations may require manual parsing, such as adding custom headers, handling unexpected WebSocket behavior, or integrating with a custom TCP server. Additionally, manual parsing is useful in embedded systems or resource-constrained environments where full-fledged libraries might not be feasible. In many embedded applications, using lighter and more efficient solutions is necessary, and manual parsing offers the lean control needed to optimize memory and CPU usage.
Moreover, understanding the WebSocket protocol from the ground up enables developers to contribute to designing and improving existing tools and even innovate new approaches that better fit specific use cases. Whether enhancing security, adding customized behavior, or fitting WebSocket functionality into existing complex systems, knowing how to parse WebSocket frames manually provides the versatility often needed in cutting-edge software development.
Using Internal Websockets Python Library API for Parsing Instead of Manual Decoding
Writing WebSocket manual parsing requires a thorough understanding of all the WebSocket RFC standards and keeping them up to date. The WebSocket protocol constantly evolves, and new versions or extensions are introduced, making manual implementation challenging and time-consuming. Developers must stay current with the latest standards and ensure their implementation follows them, which can be difficult.
While manual decoding provides more flexibility and control over the protocol, it has significant overhead. Instead, we will use a much simpler approach that takes less time to implement by leveraging the internal API of the WebSockets Python library. The WebSockets library is well-maintained and reliable. It has regular updates, ensuring it remains compatible with the latest WebSocket standards. This library is tested by numerous users in real-world environments, making it a robust choice for implementing WebSocket communication.
Using the internal API of the WebSockets Python library saves time and effort compared to manual parsing. However, using the internal API of a package is less common, as it can lead to future complexities. Internal APIs may change without notice, introducing breaking changes that require maintenance. Additionally, internal components can make the code dependent on specific library versions, leading to compatibility issues when upgrading to newer versions. Despite these potential complexities, leveraging an existing library is often the most practical approach for typical WebSocket use cases.
What is WebSocket Protocol? Understanding Before Using WebSocket Python
Overview of WebSocket Protocol
WebSocket is a communication protocol. This particular protocol provides full-duplex communication over a single TCP connection. Unlike HTTP, it establishes a persistent connection that allows data to flow with low latency.
The WebSocket is lightweight and very efficient, minimizing the overhead of traditional HTTP communication. By enabling a constant open line of communication, WebSocket allows real-time data transfer, making it perfect for applications that require live updates.
The WebSocket protocol’s simplicity in terms of packet structure and control messages helps keep it efficient while providing the necessary capabilities for reliable and secure data transmission. The WebSocket API is widely supported across browsers, which is a good choice for web developers who want to implement real-time features without the complications of managing multiple request-response cycles.
Maintaining open connections with low latency is particularly beneficial for applications like financial trading platforms, where data changes rapidly, and any delay can lead to losses or missed opportunities.
When and Why WebSocket Protocol is Used
WebSocket is ideal for real-time applications like chat, online gaming, and live updates like stock tickers. Unlike HTTP, which relies on repeated requests to update data, WebSocket maintains a connection that allows instant two-way communication, reducing latency and network overhead.
WebSocket is also widely used for collaborative editing tools, notifications, and other use cases where data changes rapidly and needs to be pushed to multiple clients simultaneously. This ability to push real-time updates makes it more efficient and responsive than traditional HTTP methods.
Collaborative applications, such as Google Docs-style real-time editors, rely heavily on the ability to instantly share updates among users without polling for changes. This efficiency extends to IoT (Internet of Things) devices, where a lightweight protocol is needed to minimize battery and data usage.
Additionally, gaming applications that require synchronous multiplayer interaction benefit significantly from WebSocket’s persistent and low-latency communication channels. The gaming experience becomes smoother and more immersive when the delay between a player’s actions and the server’s response is minimal.
Difference between HTTP and WebSocket
HTTP and WebSocket are communication protocols that serve different purposes and significantly differ in operation. Earlier versions of HTTP operated by sending a single request from the client to the server and then receiving a single response, after which the connection would be closed. Each new request required creating a new connection, adding significant overhead.
HTTP is a request-response protocol that relies on the client requesting the server, which then responds. This works for traditional applications. Still, it introduces latency when frequent updates are needed, as the client must repeatedly request new information.
WebSocket establishes a persistent connection. This connection allows for real-time, two-way communication. Once established, both sides can send data any time without repeated handshakes, making it much faster for scenarios requiring constant updates, such as chat or gaming. WebSocket reduces the overhead associated with repeatedly opening and closing connections, which is a limitation of HTTP.
The latest versions of HTTP, such as HTTP/1.1, HTTP/2, and HTTP/3, have introduced multiplexing. This means that multiple requests and responses to be sent over the same connection, similar to WebSocket. However, HTTP still uses text-based communication, with extensive headers that add extra data to each message. After the initial protocol upgrade from HTTP, WebSocket uses a binary format, where only one byte is needed to indicate if it is a request (masked) or response (unmasked) and whether to use compression.
This compact frame format makes WebSocket significantly more efficient. Unlike HTTP, where compression needs to be explicitly handled, WebSocket can compress messages internally within the protocol, reducing the message size and increasing speed. This internal compression is a crucial advantage, particularly for real-time applications where reducing data size and minimizing latency is vital.
Advantages of WebSocket vs. HTTP for Real-Time Communication
WebSocket offers reduced latency, efficient resource usage, and a persistent connection, making it ideal for live and interactive applications. HTTP requires repeated polling, which increases latency and server load, while WebSocket provides instant updates as data changes.
HTTP polling is a technique in which the client continuously sends requests to the server to inquire about new data. This process can be inefficient, as each request requires creating a new connection and receiving a response from the server, even if no new data is available.
HTTP keep-alive helps mitigate this by keeping the connection open, allowing multiple requests and responses to be sent over the same connection without repeatedly creating new ones. However, each polling request still requires a response from the server, unlike WebSocket, where the server can push updates instantly without waiting for a client request.
This difference results in a much smoother experience for end users, especially in applications where milliseconds matter, such as online gaming or financial trading platforms. The continuous connection of WebSocket eliminates the need for multiple handshakes, significantly reducing the time required for communication.
WebSocket also uses significantly fewer resources, as the server does not need to keep responding to each poll request as it does with HTTP. This reduced resource consumption is significant in environments with thousands of connected clients, such as stock trading platforms or online auctions, where keeping track of the state and delivering timely updates to all clients can be challenging with HTTP.
WebSocket’s efficiency becomes even more evident in mobile device scenarios, where battery and data usage are critical factors. Since WebSocket keeps connections open without continuous polling, it conserves energy and reduces data consumption compared to traditional HTTP requests.
How does WebSocket Protocol Work? Understanding Before WebSocket Python Implementation
WebSocket Handshake: Protocol Upgrade from HTTP
The WebSocket connection starts with an HTTP handshake. The client uses the connection to send an HTTP request to the server, asking to upgrade to the WebSocket protocol. Once the server accepts the upgrade, the connection switches to a persistent WebSocket connection.
This initial handshake uses HTTP, which makes WebSocket compatible with existing infrastructure, such as firewalls and proxies. It helps ensure a smooth transition to a WebSocket connection. A standard HTTP request ensures that WebSocket can be used in environments where only HTTP is permitted.
The upgrade request includes specific headers that indicate the intent to use the WebSocket protocol, and the server must respond with a particular status code and headers to confirm the switch. Once upgraded, all further communication occurs over WebSocket frames, allowing efficient, low-latency data transfer.
The seamless handshake process ensures that developers can implement WebSocket communication with minimal disruption to the existing network setup, a significant advantage in enterprise environments.
HTTP Request Example for WebSocket Protocol Upgrade
The client initiates a WebSocket connection with an HTTP request. Below is an example of a WebSocket upgrade request:
GET /chat HTTP/1.1
Host: example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
Sec-WebSocket-Protocol: soap, wamp
The “Upgrade” and “Connection” headers request switching the protocol to WebSocket. “Sec-WebSocket-Key” is a random string the server will use to generate a response key (generated with the help of magic static GUID, hashed with SHA1, and encoded with base64). The “Sec-WebSocket-Version” header indicates the WebSocket protocol version the client supports.
This handshake ensures the client and server agree on the protocol’s use. Including these headers ensures that intermediaries, like proxies, understand the intent to change the communication mode to WebSocket.
The handshake also plays a vital role in ensuring security by including a challenge-response mechanism, where the server responds with a derived key to prove it understands the protocol and the specific connection request.
Sec-WebSocket-Protocol Header Explanation
The “Sec-WebSocket-Protocol” header is used to indicate the subprotocols that the client wants to use. This allows both sides of the connection (the client and the server) to agree on a specific subprotocol that will be used for communication after establishing the WebSocket connection. This is helpful when multiple types of communication need to occur over the same WebSocket connection, allowing both parties to select a mutually supported protocol.
For example, Sec-WebSocket-Protocol: soap, wamp indicates that the client supports both the soap and wamp subprotocols. The server will respond with one of these protocols if it supports it, ensuring that the client and server are aligned in interpreting the messages. This ensures the WebSocket communication follows the rules of the chosen subprotocol, facilitating better interoperability between different systems.
The subprotocol list in the websocket handshake can be taken from the IANA subprotocols registry list.
Example of HTTP Response for Protocol Upgrade in WebSocket Python
Once the server accepts the upgrade, it sends an HTTP response like this:
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
The “101 Switching Protocols” status code confirms the upgrade. The “Sec-WebSocket-Accept” header contains a hashed value based on “Sec-WebSocket-Key” to verify the upgrade request.
This response means the server successfully switches the protocols, and communication can proceed using WebSocket frames. The Sec-WebSocket-Accept header value is derived from the client’s Sec-WebSocket-Key using a predefined algorithm, which helps prevent unauthorized attempts to hijack a connection.
This handshake process adds a layer of security. It ensures that the client and server are fully synchronized before the WebSocket connection begins.
Examples of WebSocket Text Frame before WebSocket Python Implementation
WebSocket communication consists of frames, which include data and control frames. Here is an example of an unmasked (response) text frame:
\x81\x05Hello
The first byte (\x81) indicates a text frame. The second byte (\x05) represents the payload length, followed by the actual data (Hello). Control frames like ping and pong are used to maintain the connection. Frames may be fragmented to support large messages, and the FIN bit determines whether the frame is the final fragment in a message.
This fragmentation feature allows WebSocket to handle messages that exceed the maximum frame size, ensuring that large payloads are transmitted efficiently without blocking other communication.
Understanding the structure of WebSocket frames is crucial when implementing low-level control over data transmission, as it allows developers to optimize how and when data is sent, especially in environments with limited bandwidth.
Example of a Closing Frame Before Understanding WebSocket Python Implementation
A closing frame is used to terminate the connection gracefully. It starts with:
\x88
The first byte of the WebSocket frame (\x88) indicates a closing frame. The connection is closed after sending or receiving this frame. The closing handshake helps ensure the client and server know the connection is being terminated, preventing data loss or corruption.
Properly handling closing frames is critical in maintaining a clean and predictable shutdown process, especially in applications where reliability is vital, such as financial services or industrial control systems. When a connection is closed without the closing handshake, data might be lost or corrupted, leading to inconsistent states in real-time applications.
WebSocket Opcodes and Their Meanings. Before WebSocket Python Implementation
WebSocket uses opcodes to identify frame types:
0x0: Continuation frame0x1: Text frame0x2: Binary frame0x8: Connection close0x9: Ping0xA: Pong
Each opcode defines the purpose of the frame, whether it’s for data or connection control. For example, 0x9 (ping) is used to check if the other side is still responsive, while 0xA (pong) is used to reply to a ping frame.
Understanding opcodes helps debug and implement advanced features, such as custom ping-pong mechanisms, to keep the connection alive. Different opcodes allow the WebSocket protocol to differentiate between control and data messages.
This ensures that essential control signals, like connection termination, are handled appropriately even when large amounts of data are transmitted.
WebSocket Frame Masking and Compression, Before Using WebSocket Python
Masked vs. Unmasked Frames
Clients must mask frames sent to the server, indicated by a masking key in the frame. Servers do not mask their frames. The masking key is a 4-byte value applied to the payload using XOR, ensuring that each message appears different even if the content is the same.
Deflate Compression: Enabled vs. Disabled
WebSocket optionally supports compression using the “permessage-deflate” extension. When enabled, frames are compressed before being sent, reducing bandwidth usage.
This feature is handy for applications transmitting large amounts of text data, such as chat applications, where reducing the size of messages can significantly improve performance. Compression can be negotiated during the handshake, allowing the client and server to determine whether to use it.
The benefits of compression are particularly evident when there are repeated patterns in the data being sent, as deflate compression can dramatically reduce the size of such messages, leading to faster transmission and less network congestion. However, balancing compression benefits with CPU overhead is essential, as compressing and decompressing data can increase processing time, especially on resource-constrained devices.
Creating a WebSocket Client and Server in WebSocket Python
Before we use the internal API of the websockets Python library, we will show an official way of using it to send/receive messages.
First, we need to download and install Python and then install the package:
pip install websockets
Note: Our code relies on at least websockets python package version 13.1.
WebSocket Python Client Example Using WebSockets Library
Here’s an example of a WebSocket client in Python using websockets library:
# client.py
import asyncio
import websockets
async def test_websocket():
uri = "ws://example.com:443"
async with websockets.connect(uri) as websocket:
messages = [
"Message 1: Hello, WebSocket server!",
"Message 2: How are you?",
"Message 3: Goodbye!"
]
for msg in messages:
print(f"Sending to server: {msg}")
await websocket.send(msg)
response = await websocket.recv()
print(f"Received from server: {response}")
print("Connection closed by server.")
asyncio.run(test_websocket())Let’s walk through the code in client.py, a simple WebSocket client implemented using Python’s asyncio and websockets libraries.
Import Statements
import asyncio
import websockets- asyncio: This built-in Python library is intended to build concurrent code using the async/await syntax. It provides support for asynchronous tasks, I/O operations, and event loops.
- websockets: This third-party library in Python provides a WebSocket API, allowing the client to establish and communicate over a WebSocket connection to a server. The WebSocket protocol provides full-duplex communication between client and server.
Function Definition
async def test_websocket():- This function is defined as async, meaning it’s an asynchronous coroutine. This enables it to run non-blocking operations, like I/O, without halting the entire program. The coroutine can be paused and resumed, making it ideal for handling WebSocket communications.
Define the WebSocket Server URI
uri = "wss://example.com:443"- uri: This is the URI of the WebSocket server the client will connect to.
- wss:// indicates a secure (SSL/TLS) WebSocket connection. If it were ws://, it would represent an unencrypted WebSocket connection.
- example.com is a placeholder domain name. In a real application, you’d replace this with the actual domain of the WebSocket server.
- 443 is the port number, the default port for HTTPS (and WSS when using secure WebSockets).
Establish WebSocket Connection
async with websockets.connect(uri) as websocket:- websockets.connect(uri): This is an asynchronous function provided by the websockets library. It initiates a connection to the WebSocket server specified by uri.
- async with: This is an asynchronous context manager. When using async with, the program ensures that the WebSocket connection will be properly opened at the start and closed automatically at the end of the with block. This means that the resources will be properly cleaned up.
Sending and Receiving Messages
messages = [
"Message 1: Hello, WebSocket server!",
"Message 2: How are you?",
"Message 3: Goodbye!"
]The client will send a list of three messages to the WebSocket server.
for msg in messages:
print(f"Sending to server: {msg}")
await websocket.send(msg)
response = await websocket.recv()
print(f"Received from server: {response}")- The for loop iterates over the messages list.
- print(f”Sending to server: {msg}”): Before sending each message, the client prints it to the console for logging purposes.
- await websocket.send(msg): This line sends the current message (msg) to the server using the WebSocket connection. await is used here because this operation is asynchronous and might take time (e.g., due to network latency).
- response = await websocket.recv(): This line waits for a message from the server. recv() is a non-blocking call that will receive the server’s response asynchronously. The received message is stored in the response variable.
- print(f”Received from server: {response}”): The received response from the server is printed to the console.
Closing the Connection
print("Connection closed by server.")Once all messages are sent and responses are received, the connection will be closed automatically when the program exits the async with block. A final message is printed to indicate that the connection has been closed.
Running the Coroutine
asyncio.run(test_websocket())asyncio.run(test_websocket()): This is how you start the asynchronous function test_websocket. asyncio.run() sets up the event loop, runs the coroutine (the asynchronous function), and blocks until the coroutine completes.
This is the main entry point of the script and ensures that the event loop runs until test_websocket() finishes all its asynchronous operations (i.e., sending and receiving messages over the WebSocket connection).
Complete Flow
- The WebSocket client connects to a server at wss://example.com:443.
- It sends three predefined messages to the server one by one:
“Message 1: Hello, WebSocket server!”
“Message 2: How are you?”
“Message 3: Goodbye!” - The client waits for and prints the server’s response for each message.
- Once all messages are exchanged, the connection is closed automatically, and the client prints “Connection closed by the server.”
WebSocket Python Server Example Using WebSockets Library
This is a WebSocket Python example of a server using websockets library:
# server.py
import asyncio
import websockets
async def echo(websocket):
for _ in range(3):
message = await websocket.recv()
print(f"Received message from client: {message}")
response = f"Server echoes: {message}"
await websocket.send(response)
print(f"Sent response to client: {response}")
print("Closing connection")
await websocket.close()
async def main():
async with websockets.serve(echo, '192.168.1.100', 443):
print("WebSocket server started on wss://192.168.1.100")
await asyncio.Future() # Run forever
asyncio.run(main())Let’s walk through the code in server.py, which sets up a simple WebSocket server that echoes messages back to the client. The server listens on a specific IP address and port using the asyncio and websockets libraries.
Import Statements
import asyncio
import websocketsSame as with client.py above.
Echo Function
async def echo(websocket):- This function is defined as async, a coroutine that can perform asynchronous (non-blocking) operations, such as sending and receiving messages.
- websocket: This parameter represents the WebSocket connection between the server and a client. It’s an instance of a WebSocket object that allows you to send and receive messages.
Receiving and Echoing Messages
for _ in range(3):
message = await websocket.recv()
print(f"Received a message from client: {message}")
response = f"Server echoes: {message}"
await websocket.send(response)
print(f"Sent response to client: {response}")- for _ in range(3): This loop ensures that the server processes exactly three messages from the client. _ is used as a placeholder variable because we don’t care about the actual value of the loop counter.
- message = await websocket.recv(): This line waits to receive a message from the client. recv() is an asynchronous operation, meaning the server will pause here and wait for a message without blocking the entire program.
- print(f”Received a message from client: {message}”): This prints the received message to the server console for logging purposes.
- response = f”Server echoes: {message}”: The server creates a response string that echoes the received message back to the client, prefixed with “Server echoes: “.
- await websocket.send(response): This line sends the echo response back to the client through the WebSocket. Since sending data might take time (e.g., due to network latency), this operation is asynchronous and uses await.
- print(f”Sent response to client: {response}”): This logs the sent response to the server console.
Closing the Connection
print("Closing connection")
await websocket.close()- print(“Closing connection”): After processing three messages, the server logs that it’s closing the connection.
- await websocket.close(): This closes the WebSocket connection. The await ensures that the connection is closed asynchronously, allowing the server to handle other tasks while waiting for the closure to complete.
Main Function
async def main():
async with websockets.serve(echo, '192.168.1.100', 443):
print("WebSocket server started on wss://192.168.1.100")
await asyncio.Future() # Run foreverasync def main(): This is the main coroutine that sets up and runs the WebSocket server.
Starting the WebSocket Python Server
async with websockets.serve(echo, '192.168.1.100', 443):- websockets.serve(echo, ‘192.168.1.100’, 443): This line starts a WebSocket server that listens on the IP address 192.168.1.100 and port 443.
- echo: This function will be called whenever a client connects. When a client connects, the echo coroutine will handle the communication.
- ‘192.168.1.100’: This is the IP address where the server will listen for incoming connections. In this case, it is a local IP address typically used in a private network.
- 443: This is the port number. Port 443 is generally reserved for secure HTTPS traffic (and WebSocket connections over SSL/TLS, wss://).
- async with: This is an asynchronous context manager. It ensures that the WebSocket server will be properly initialized when entering the block and properly closed when exiting it. While the server is running, the code block inside the async with a statement will be executed.
Logging Server Start and Running Forever
print("WebSocket server started on wss://192.168.1.100")This prints a console message indicating that the WebSocket Python server has started and is listening for incoming connections on 192.168.1.100:443. Note the use of wss:// (WebSocket Secure), which implies that SSL/TLS would be used. A real setup would require proper SSL/TLS certificates to secure the connection.
await asyncio.Future() # Run forever- await asyncio.Future(): This is used to keep the server running indefinitely. asyncio.Future is a low-level construct representing a placeholder for a result that hasn’t been computed yet.
- In this case, asyncio.Future() creates a “never-ending” future that will never resolve, so the server stays up indefinitely. This is necessary because, without this line, the server would start and immediately shut down after serving one request.
- In practice, you could replace await asyncio.Future() with more advanced logic to allow for controlled shutdowns or other long-running tasks.
Running the Main Function
asyncio.run(main())asyncio.run(main()): This starts the main() coroutine. It sets up the asyncio event loop and runs main() until completion. In this case, main() contains an infinite loop (await asyncio.Future()), so the server will run forever.
Complete Flow
- The main() function is executed, starting the WebSocket server on 192.168.1.100:443.
- The server waits for incoming client connections. When a client connects, the echo function is invoked to handle communication.
- The echo function processes exactly three messages from the client:
It receives a message, prints it to the console, creates an echo response, and sends it back to the client. - After three messages are processed, the connection is closed.
- The server continues to run indefinitely while ready to accept new connections from other clients.
WebSocket Python Client and Server Example Explanation
The above is a simple example of WebSocket Python execution of a client and a server. The client sends three messages; the server receives three and responds to each. The real-world implementation of WebSocket will be more complex and depend on your needs.
Working with a Custom TCP Server in WebSocket Python
Using a Custom TCP Python Server with WebSocket
Handling WebSocket connections in a custom TCP server requires manually implementing the WebSocket handshake and framing protocol. This allows for the integration of WebSocket communication into an existing TCP-based solution.
It involves managing the initial HTTP handshake, upgrading the protocol, and then processing WebSocket frames, which requires understanding the structure of frames and how to decode them.
This approach allows developers to create a tightly integrated solution that combines WebSocket functionality with other protocols or custom application logic, offering more flexibility than relying solely on high-level libraries.
Parsing WebSocket Messages with a Custom TCP Server
To parse WebSocket messages manually, handle the initial HTTP handshake, read frame headers, and decode the payload. This involves identifying opcodes, masking keys, and managing fragmented frames.
Manually parsing frames allows for customization, such as adding specific headers or handling proprietary data formats, and provides more visibility into the communication process. It also enables precise control over data handling, including error detection and correction, which is particularly important in mission-critical systems.
Parsing frames at this level can also help integrate WebSocket with legacy systems that do not natively support it, ensuring smooth and reliable data exchange across diverse platforms.
Example of Simple TCP Client and TCP Server in Python
It’s essential to understand how TCP works and that you get binary (bytes) data from the socket. The binary data is what you would want to parse using the websocket Python parser.
Simple TCP Client Example in Python
Below is an example of a simple TCP client using Python’s socket library:
import socket
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(('localhost', 65432))
client_socket.sendall(b'Hello, server')
response = client_socket.recv(1024)
print(f'Received: {response.decode()}')
client_socket.close()This client connects to a server on localhost and sends a simple message. It then waits for a response from the server and prints it to the console. The “socket” library allows for low-level control of the TCP connection, enabling custom protocols or modifications to be implemented as needed. This setup can be the foundation for implementing more complex communication protocols like WebSocket.
Import the socket module
import socketThe socket module in Python provides access to the underlying network interface and protocols for network communication. It allows us to create a socket responsible for communication between a client and a server (sending/receiving data).
Create a socket object
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)- socket.socket() creates a new socket object.
- socket.AF_INET specifies the address family. AF_INET is the family used for IPv4 addresses.
- socket.SOCK_STREAM specifies the socket type. SOCK_STREAM means the socket will use TCP, a connection-oriented protocol (i.e., it ensures reliable data transmission).
- This line initializes a TCP socket for the client to communicate with the server.
Connect to the server
client_socket.connect(('localhost', 65432))- client_socket.connect() attempts to establish a connection to a server.
- ‘localhost’ is the server address, which refers to the local machine (loopback address 127.0.0.1).
- 65432 is the port number on which the server listens for incoming connections.
- This line tells the client socket to connect to a server on the same machine (localhost) at port 65432. This assumes a server is already running on this port and is accepting TCP connections.
Send a message to the server
client_socket.sendall(b'Hello, server')- client_socket.sendall() sends data to the connected server.
- b’Hello, server’ is a byte string, as required by the socket’s sendall() method. The b prefix in b’Hello, server’ converts the string to bytes, which is necessary because socket communication happens using raw byte streams. This is how TCP works.
- sendall() ensures that all data is sent (it will retry sending if necessary until all the data is sent).
- This line sends the message “Hello, server” in bytes from the client to the server.
Receive a response from the server
response = client_socket.recv(1024)- client_socket.recv(1024) receives data from the server.
- 1024 specifies the maximum number of bytes to receive in one call. This value is arbitrary but limits the size of the message to 1024 bytes.
- The received data is stored in the variable response as a byte string.
- This line waits for and receives a response from the server (up to 1024 bytes). It blocks execution until data is received.
Print the server’s response
print(f'Received: {response.decode()}')- response.decode() converts the byte string received from the server into a human-readable string using the default UTF-8 encoding.
- f’Received: {response.decode()}’ is a formatted string incorporating the decoded response into the output.
- This line prints the decoded response received from the server to the console.
Close the client socket
client_socket.close()- client_socket.close() closes the connection and releases any resources associated with the socket.
- This line closes the socket, ending the client and server communication session. It is a good practice to close sockets when you’re done with them to free up system resources.
Overall Flow
- The client socket is created and set up for TCP communication.
- It connects to a server at localhost on port 65432.
- The client sends the message “Hello, server” to the server.
- The client waits for a response (up to 1024 bytes).
- It prints the server’s response to the console after decoding it.
- Finally, it closes the socket and terminates the connection.
Example of Expected Output
If the server responds with “Hello, client,” the printed output will be:
Received: Hello, client
This code will only work if there’s a corresponding server running on the same machine (localhost) listening on port 65432. The server would need to accept the connection, receive the message from the client, send a response, and then close the connection.
Simple TCP Server Example in Python
Here is a simple TCP server example:
import socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(('localhost', 65432))
server_socket.listen()
conn, addr = server_socket.accept()
with conn:
print(f'Connected by: {addr}')
while True:
data = conn.recv(1024)
if not data:
break
conn.sendall(data)This server listens for incoming connections, accepts them, and echoes back any data it receives, creating a simple echo server. This basic implementation demonstrates the concept of handling client-server communication using TCP, which is the foundation for more complex protocols like WebSocket. By controlling the connection at this level, developers can directly manage connection details such as timeouts, retries, and custom packet handling, which is not easily achievable with higher-level abstractions.
Import the socket module
import socketThe socket module provides low-level network communication capabilities, allowing you to work with different network protocols. In this case, the server will use TCP sockets for communication.
Create a server socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)- socket.socket() creates a new socket object.
- socket.AF_INET specifies the address family. AF_INET is used for IPv4 addresses.
- socket.SOCK_STREAM specifies the socket type, which is TCP (a connection-oriented protocol).
- This line creates a TCP server socket, which will be used to listen for and accept client connections.
Bind to an address and port
server_socket.bind(('localhost', 65432))- server_socket.bind() binds the socket to a specific network address and port to listen for incoming connections.
- (‘localhost’, 65432):
‘localhost’ means the server only listens on the local machine (IP address 127.0.0.1).
65432 is the port number that the server will listen on for client connections. - This line binds the server to the specified address (localhost) and port (65432). Clients must connect to this address and port to communicate with the server.
Listen for incoming connections
server_socket.listen()- server_socket.listen() puts the server socket into listening mode, where it waits for incoming client connection requests.
- By default, the listen() method allows for a backlog of pending connections (i.e., how many connections can wait in line to be accepted). This number is typically set to a reasonable default (e.g., 5) if not specified.
- This line enables the server to start accepting connection requests from clients. It essentially makes the server “open for business.”
Accept a client connection
conn, addr = server_socket.accept()- server_socket.accept() waits for an incoming client connection. When a client connects, it returns:
conn: a new socket object representing the connection between the server and the client. You will use conn to communicate with this specific client.
addr: the client’s address (a tuple containing the client’s IP address and port). - This line accepts a connection from a client. It blocks execution until a client connects. Once a client is connected, the server can communicate with the client using the conn socket.
Handle the connection using a context manager
with conn:- The with conn: statement is a context manager that ensures the connection will be closed when the code inside the with statement finishes, even if an error occurs.
- This ensures the client connection (conn) is correctly closed once the communication ends. It simplifies resource management and is safer than manually managing the socket’s lifecycle.
Print the address of the connected client
print(f'Connected by: {addr}')- f’Connected by: {addr}’ is a formatted string that includes the address (addr) of the connected client.
- This prints a message to the console showing the client’s IP address and port number connected to the server. This helps confirm that a client has successfully connected.
Receive and echo data in a loop
while True:
data = conn.recv(1024)
if not data:
break
conn.sendall(data)- Receive Data:
conn.recv(1024) reads up to 1024 bytes of data from the client.
The server waits (blocks) until data is received from the client.
If the client sends no data (i.e., the connection is closed), recv() returns an empty byte string (b”), and the server breaks out of the loop. - Check for Empty Data:
if not data: check if no data was received (i.e., the connection is closed). If so, it breaks the loop and stops the communication. - Send Data Back:
conn.sendall(data) returns the received data to the client. sendall() ensures that all data is sent (similar to the client-side sendall()). - This loop continually receives data from the client. It echoes it back to the client until the client sends no data (i.e., the connection is closed). It implements a simple echo server, where whatever message the client sends to the server is returned (echoed) back to the client.
Implicit connection close
After the “with conn:” block finishes, the connection (conn) is automatically closed because of the context manager. The server is still running and listening for new connections, but the current client’s communication session has ended.
Overall Flow
- The server creates a socket, binds to localhost on port 65432, and listens for connections.
- When a client connects, the server accepts the connection and enters a loop where it:
Receives data from the client.
Echoes the received data back to the client.
The loop breaks when the client closes the connection or sends no data. - The connection is automatically closed once the loop exits, but the server remains active and can accept new client connections.
Example of Expected Behavior
If a client (like the one in the previous question) sends “Hello, server” to the server, the server will echo “Hello, server” back to the client. The client will then receive the echoed message and print it.
This is the foundation of a simple TCP-based echo server, which can handle a single client simultaneously. You would typically use threading or asynchronous I/O for multiple clients.
Main Websocket Python Functions to Manage Handshakes and Parse Frames
Here are the Python functions to manage WebSocket handshake and manage and parse WebSocket frames:
from typing import Union, Generator
import logging
from websockets.server import ServerProtocol
from websockets.extensions.permessage_deflate import PerMessageDeflate, ServerPerMessageDeflateFactory
from websockets.http11 import Request
from websockets.frames import Frame, Opcode
from websockets.protocol import OPEN
from websockets.streams import StreamReader
from websockets.exceptions import InvalidHeaderValue, ProtocolError, PayloadTooBig
class WebsocketParseWrongOpcode(Exception):
pass
class WebsocketFrameParser:
def __init__(self):
# Instantiate the permessage-deflate extension.
# If a frame uses 'deflate,' then the 'permessage_deflate' should be the same object during parsing of
# several messages on the same socket. Each time 'PerMessageDeflate' is initiated, the context changes
# and more than one message can't be parsed.
self.permessage_deflate_masked = PerMessageDeflate(
remote_no_context_takeover=False,
local_no_context_takeover=False,
remote_max_window_bits=15,
local_max_window_bits=15,
)
# We need separate instances for masked (frames from client) and unmasked (frames from server).
self.permessage_deflate_unmasked = PerMessageDeflate(
remote_no_context_takeover=False,
local_no_context_takeover=False,
remote_max_window_bits=15,
local_max_window_bits=15,
)
def parse_frame_bytes(
self,
data_bytes: bytes
):
# Define the read_exact function
def read_exact(n: int) -> Generator[None, None, bytes]:
return reader.read_exact(n)
# Helper function to run generator-based coroutines
def run_coroutine(coroutine):
try:
while True:
next(coroutine)
except StopIteration as e:
return e.value
except Exception as e:
raise e # Re-raise exceptions to be handled by the caller
# Function to parse frames
def parse_frame(mask: bool, deflate: bool):
try:
if mask:
# Decide whether to include permessage-deflate extension
extensions = [self.permessage_deflate_masked] if deflate else []
else:
extensions = [self.permessage_deflate_unmasked] if deflate else []
# Use Frame.parse to parse the frame
frame_parser = Frame.parse(
read_exact,
mask=mask, # Client frames are masked
max_size=None,
extensions=extensions
)
current_frame = run_coroutine(frame_parser)
except EOFError as e:
# Not enough data to parse a complete frame
raise e
except (ProtocolError, PayloadTooBig) as e:
print("Error parsing frame:", e)
raise e
except Exception as e:
print("Error parsing frame:", e)
raise e
return current_frame
def process_frame(current_frame):
if current_frame.opcode == Opcode.TEXT:
message = current_frame.data.decode('utf-8', errors='replace')
return message, 'TEXT'
elif current_frame.opcode == Opcode.BINARY:
return current_frame.data, 'BINARY'
elif current_frame.opcode == Opcode.CLOSE:
return None, 'CLOSE'
elif current_frame.opcode == Opcode.PING:
return None, 'PING'
elif current_frame.opcode == Opcode.PONG:
return None, 'PONG'
else:
raise WebsocketParseWrongOpcode("Received unknown frame with opcode:", current_frame.opcode)
# Create the StreamReader instance
reader = StreamReader()
masked = is_frame_masked(data_bytes)
deflated = is_frame_deflated(data_bytes)
# Feed the data into the reader
reader.feed_data(data_bytes)
# Parse and process frames
frame = parse_frame(masked, deflated)
parsed_frame, frame_opcode = process_frame(frame)
# This is basically not needed since we restart the 'reader = StreamReader()' each function execution.
# # After processing, reset the reader's buffer
# reader.buffer = b''
result: dict = {
'is_deflated': deflated,
'is_masked': masked,
'frame': parsed_frame,
'opcode': frame_opcode
}
return result
def create_byte_http_response(
byte_http_request: Union[bytes, bytearray],
enable_logging: bool = False
) -> bytes:
"""
Create a byte HTTP response from a byte HTTP request.
Parameters:
- byte_http_request (bytes, bytearray): The byte HTTP request.
- enable_logging (bool): Whether to enable logging.
Returns:
- bytes: The byte HTTP response.
"""
# Set up extensions
permessage_deflate_factory = ServerPerMessageDeflateFactory()
# Create the protocol instance
protocol = ServerProtocol(
extensions=[permessage_deflate_factory],
)
# At this state, the protocol.state is State.CONNECTING
if enable_logging:
logging.basicConfig(level=logging.DEBUG)
protocol.logger.setLevel(logging.DEBUG)
protocol.receive_data(byte_http_request)
events = protocol.events_received()
event = events[0]
if isinstance(event, Request):
# Accept the handshake.
# After the response is sent, it means the handshake was successful, the protocol.state is State.OPEN
# Only after this state can we parse frames.
response = protocol.accept(event)
return response.serialize()
else:
raise ValueError("The event is not a Request object.")
def create_websocket_frame(
data: Union[str, bytes, bytearray],
deflate: bool = False,
mask: bool = False,
opcode: int = None
) -> bytes:
"""
Create a WebSocket frame with the given data, optionally applying
permessage-deflate compression and masking.
Parameters:
- data (str, bytes, bytearray): The payload data.
If str, it will be encoded to bytes using UTF-8.
- deflate (bool): Whether to apply permessage-deflate compression.
- mask (bool): Whether to apply masking to the frame.
- opcode (int): The opcode of the frame. If not provided, the BINARY and TEXT will be determined based on the data type.
Example:
from websockets.frames import Opcode
Opcode.TEXT, Opcode.BINARY, Opcode.CLOSE, Opcode.PING, Opcode.PONG.
Returns:
- bytes: The serialized WebSocket frame that is ready to be sent.
"""
# Determine the opcode if not provided
if opcode is None:
if isinstance(data, str):
opcode = Opcode.TEXT
elif isinstance(data, (bytes, bytearray)):
opcode = Opcode.BINARY
else:
raise TypeError("Data must be of type str, bytes, or bytearray.")
else:
if not isinstance(opcode, int):
raise TypeError("Opcode must be an integer.")
if not isinstance(data, (str, bytes, bytearray)):
raise TypeError("Data must be of type str, bytes, or bytearray.")
# Encode string data if necessary
if isinstance(data, str):
payload = data.encode('utf-8')
else:
payload = bytes(data)
# Create the Frame instance
frame = Frame(opcode=opcode, data=payload)
# Set up extensions if deflate is True
extensions = []
if deflate:
permessage_deflate = PerMessageDeflate(
remote_no_context_takeover=False,
local_no_context_takeover=False,
remote_max_window_bits=15,
local_max_window_bits=15,
)
extensions.append(permessage_deflate)
# Serialize the frame with the specified options
try:
frame_bytes = frame.serialize(
mask=mask,
extensions=extensions,
)
except Exception as e:
raise RuntimeError(f"Error serializing frame: {e}")
return frame_bytes
def is_frame_masked(frame_bytes):
"""
Determine whether a WebSocket frame is masked.
Parameters:
- frame_bytes (bytes): The raw bytes of the WebSocket frame.
Returns:
- bool: True if the frame is masked, False otherwise.
"""
if len(frame_bytes) < 2:
raise ValueError("Frame is too short to determine masking.")
# The second byte of the frame header contains the MASK bit
second_byte = frame_bytes[1]
# The MASK bit is the MSB (most significant bit) of the second byte
mask_bit = (second_byte & 0x80) != 0 # 0x80 is 1000 0000 in binary
return mask_bit
def is_frame_deflated(frame_bytes):
"""
Determine whether a WebSocket frame is deflated (compressed).
Parameters:
- frame_bytes (bytes): The raw bytes of the WebSocket frame.
Returns:
- bool: True if the frame is deflated (compressed), False otherwise.
"""
if len(frame_bytes) < 1:
raise ValueError("Frame is too short to determine deflation status.")
# The first byte of the frame header contains the RSV1 bit
first_byte = frame_bytes[0]
# The RSV1 bit is the second most significant bit (bit 6)
rsv1 = (first_byte & 0x40) != 0 # 0x40 is 0100 0000 in binary
return rsv1We use it in our atomicshop package.
WebSocket Python Frame Parsing Function Explanation
The WebSocket Python frame parsing function can parse client frames (masked) on the server side and server frames (unmasked) on the client side. This will be our primary purpose since we use it specifically on the server side to parse client frames.
Imports
The imports section is relevant for all the functions in the code above.
from typing import Union, Generator
import logging
from websockets.server import ServerProtocol
from websockets.extensions.permessage_deflate import PerMessageDeflate, ServerPerMessageDeflateFactory
from websockets.http11 import Request
from websockets.frames import Frame, Opcode
from websockets.protocol import OPEN
from websockets.streams import StreamReader
from websockets.exceptions import InvalidHeaderValue, ProtocolError, PayloadTooBigfrom typing import Union, Generator:
- typing is a standard Python library used for type hinting. It helps clarify what types are expected in function arguments and return values.
- Union: Used to specify that a value can be one of several types. For example, Union[int, str] would mean a value can be either an integer or a string.
- Generator: Type hint that indicates a function is a generator, which returns values lazily (using the yield keyword) instead of returning a final result all at once.
import logging: The logging module is used to log messages that track events during the execution of a program. These messages can be written to the console or files, depending on how the logging system is configured.
from websockets.server import ServerProtocol:
- ServerProtocol: A class from the websockets library that implements the server-side WebSocket protocol. It manages connections with WebSocket clients.
from websockets.extensions.permessage_deflate import PerMessageDeflate, ServerPerMessageDeflateFactory:
- PerMessageDeflate: An extension that compresses and decompresses WebSocket messages using the “permessage-deflate” algorithm. This helps reduce the size of messages.
- ServerPerMessageDeflateFactory: A factory class that creates instances of PerMessageDeflate for server-side use. The factory handles setting up the compression for the WebSocket connection.
from websockets.http11 import Request:
- Request: Represents an HTTP/1.1 request used during the WebSocket handshake process. Before a WebSocket connection is established, an HTTP upgrade request is made, and this Request class represents that part of the process.
from websockets.frames import Frame, Opcode:
- Frame: Represents a WebSocket frame. WebSocket data is transmitted over the network as frames, encapsulating the payload (data) and the control information (like message type).
- Opcode: Represents the operation code of a WebSocket frame. For example, an opcode can indicate whether the frame contains text, binary data, or a control message (like closing the connection or ping/pong messages).
from websockets.streams import StreamReader:
- StreamReader: A class that reads data from the WebSocket connection in a stream-like fashion. It allows asynchronous reading of incoming messages, which can be handled in chunks rather than all at once.
from websockets.exceptions import InvalidHeaderValue, ProtocolError, PayloadTooBig:
- InvalidHeaderValue: This exception is raised when a WebSocket handshake request contains a header with an invalid value.
- ProtocolError: Raised when there is an error related to the WebSocket protocol, such as violating the protocol rules.
- PayloadTooBig: Raised when a WebSocket message’s payload (the data) is too large. WebSocket servers usually impose size limits to avoid memory exhaustion.
Breakdown of each import
typing.Union and typing.Generator:
- These are just type hints that provide clearer information on what types are expected in functions.
- They aren’t related to WebSocket logic but help ensure that code is self-documenting and easier to maintain.
logging: The logging module is used to track important events or errors. It will help developers debug or monitor the status of the WebSocket server.
websockets.server.ServerProtocol: This class manages a WebSocket connection on the server side. It handles the initial handshake, frame reading/writing, and message processing.
websockets.extensions.permessage_deflate: These imports allow for message compression using the permessage-deflate extension, an optimization used in many WebSocket implementations to reduce bandwidth usage.
websockets.http11.Request: WebSocket connections start with an HTTP request (called a WebSocket handshake). The Request class represents this HTTP request before the WebSocket connection is established.
websockets.frames.Frame and websockets.frames.Opcode: WebSocket communication occurs via frames, and the opcode helps indicate the type of frame (data or control frame, text, binary, etc.). These are essential for low-level handling of the WebSocket data.
websockets.streams.StreamReader: The WebSocket server will need to read data streams from the client. StreamReader is a utility for that purpose. It handles reading WebSocket frames or data chunks asynchronously.
websockets.exceptions: These exceptions handle potential errors related to WebSocket communications:
- InvalidHeaderValue: Thrown when there’s a problem with the HTTP headers during the WebSocket handshake.
- ProtocolError: A generic error that indicates a violation of the WebSocket protocol.
- PayloadTooBig: Thrown when a WebSocket message exceeds the allowed size limits.
Custom Exception
class WebsocketParseWrongOpcode(Exception):
passWebsocketParseWrongOpcode: A custom exception class that will be raised when a WebSocket frame contains an unexpected or unrecognized opcode (which is the operation type of the frame). OpCodes indicate the type of data being transmitted (text, binary, close, etc.).
Class Constructor (__init__)
class WebsocketFrameParser:
def __init__(self):WebsocketFrameParser: This is the main class that handles the parsing of WebSocket frames. The constructor sets up the deflate extension that will handle compressed messages.
self.permessage_deflate_masked = PerMessageDeflate(
remote_no_context_takeover=False,
local_no_context_takeover=False,
remote_max_window_bits=15,
local_max_window_bits=15,
)- self.permessage_deflate_masked: This is an instance of the PerMessageDeflate class. It is used to handle frames from the client on the server, which are always masked.
- remote_no_context_takeover=False: This flag controls whether or not the remote compression context (used by the server) is reset after each message. Setting this to False means the context is reused across messages, which can improve compression efficiency. The server and the client should turn this attribute on or off to work together.
- local_no_context_takeover=False: This flag behaves similarly for the local (client-side) context.
- remote_max_window_bits=15: This sets the maximum window size (2^15 bytes) for the server-side compression.
local_max_window_bits=15: This sets the maximum window size for the client-side compression.
self.permessage_deflate_unmasked = PerMessageDeflate(
remote_no_context_takeover=False,
local_no_context_takeover=False,
remote_max_window_bits=15,
local_max_window_bits=15,
)self.permessage_deflate_unmasked: This is another instance of the PerMessageDeflate class, used for frames from the server on the client, which are always unmasked. The parameters here are the same as those for permessage_deflate_masked, but this will handle the other side of the communication.
We use two different instances because if you want to parse both masked and unmasked messages on the same thread, you would need different instances of the PerMessageDeflate class. The same instance can’t parse both masked and unmasked frames.
parse_frame_bytes Method
def parse_frame_bytes(self, data_bytes: bytes):- This core method parses a stream of bytes (data_bytes) into a WebSocket frame.
- data_bytes: bytes: A bytes object representing the raw WebSocket frame data to be parsed.
def read_exact(n: int) -> Generator[None, None, bytes]:
return reader.read_exact(n)- read_exact(n: int): This function acts as a generator to read exactly n bytes from the byte stream. It uses the StreamReader instance (reader) to perform the reading.
- n: int: The number of bytes to read from the stream.
def run_coroutine(coroutine):
try:
while True:
next(coroutine)
except StopIteration as e:
return e.value
except Exception as e:
raise e- run_coroutine(coroutine): This helper function runs a generator-based coroutine (e.g., one like Frame.parse). It steps through the generator using next() until it reaches the end. At this point, it captures the value of StopIteration (which indicates the end of the coroutine) and returns it.
- If the coroutine raises any other exceptions during execution, those are re-raised.
def parse_frame(mask: bool, deflate: bool):- parse_frame(mask: bool, deflate: bool): This function handles the actual parsing of a WebSocket frame based on whether it’s masked or deflated.
- mask: bool: A boolean that indicates if the frame is masked (client frames are always masked).
- deflate: bool: A boolean indicating if the frame uses compression (deflation).
try:
if mask:
extensions = [self.permessage_deflate_masked] if deflate else []
else:
extensions = [self.permessage_deflate_unmasked] if deflate else []- Depending on whether the frame is masked or unmasked, this part decides which PerMessageDeflate instance to use (either the one for masked frames or unmasked frames).
- extensions: A list of extensions applied to the WebSocket frame. If the frame is deflated, the corresponding PerMessageDeflate instance is included in this list. Otherwise, the list is empty.
frame_parser = Frame.parse(
read_exact,
mask=mask,
max_size=None,
extensions=extensions
)frame_parser = Frame.parse(…): This line uses the Frame.parse method (likely from the websockets library) to parse a frame. It uses:
- read_exact: The function defined earlier to read the exact number of bytes from the stream.
- mask=mask: Whether the frame is masked (True for client frames).
- max_size=None: The maximum size of the frame (set to None, meaning no size limit is enforced here).
- extensions=extensions: The list of extensions to apply (such as PerMessageDeflate for deflation).
current_frame = run_coroutine(frame_parser)current_frame = run_coroutine(frame_parser): The Frame.parse method is a coroutine (generator) that needs to be run. This line uses the run_coroutine helper to step through the generator and get the parsed frame as the current_frame.
except EOFError as e:
raise eEOFError: If the stream ends before the whole frame is read (i.e., there aren’t enough bytes to complete the frame), this exception is raised and re-raised here to signal an incomplete frame.
except (ProtocolError, PayloadTooBig) as e:
print("Error parsing frame:", e)
raise eProtocolError, PayloadTooBig: These exceptions are caught and logged. These might occur if:
- The frame violates the WebSocket protocol (e.g., invalid frame format).
- The frame’s payload exceeds the maximum allowable size.
except Exception as e:
print("Error parsing frame:", e)
raise eGeneral Exception Handling: Any other exceptions are caught, logged, and re-raised.
return current_frameReturn current_frame: The fully parsed current_frame is returned if no exceptions were raised.
process_frame Function
def process_frame(current_frame):
if current_frame.opcode == Opcode.TEXT:
message = current_frame.data.decode('utf-8', errors='replace')
return message, 'TEXT'
elif current_frame.opcode == Opcode.BINARY:
return current_frame.data, 'BINARY'
elif current_frame.opcode == Opcode.CLOSE:
return None, 'CLOSE'
elif current_frame.opcode == Opcode.PING:
return None, 'PING'
elif current_frame.opcode == Opcode.PONG:
return None, 'PONG'
else:
raise WebsocketParseWrongOpcode("Received unknown frame with opcode:", current_frame.opcode)This function processes a parsed WebSocket frame by checking its opcode and handling it accordingly:
- TEXT (Opcode.TEXT): If the frame is a text frame, it decodes the data as UTF-8 and returns the message along with the opcode type (‘TEXT’).
- BINARY (Opcode.BINARY): If the frame contains binary data, the raw binary data is returned along with ‘BINARY.’
- CLOSE (Opcode.CLOSE): If the frame is close, it returns None with the ‘CLOSE’ opcode.
- PING (Opcode.PING) and PONG (Opcode.PONG): These are control frames used to keep the WebSocket connection alive. They return None with the respective opcode.
- Unknown Opcode: If the opcode is not recognized, the custom exception WebsocketParseWrongOpcode is raised with the invalid opcode.
reader = StreamReader()reader = StreamReader(): This creates an instance of StreamReader that handles reading from the byte stream (data_bytes).
masked = is_frame_masked(data_bytes)
deflated = is_frame_deflated(data_bytes)- masked: A boolean value determined by calling the is_frame_masked function (assumed to be defined elsewhere), which checks if the frame is masked.
- deflated: A boolean value, determined by calling is_frame_deflated, checks if the frame uses the deflate extension for compression.
reader.feed_data(data_bytes)reader.feed_data(data_bytes): This feeds the raw WebSocket frame data (data_bytes) into the StreamReader instance for processing.
frame = parse_frame(masked, deflated)
parsed_frame, frame_opcode = process_frame(frame)- frame = parse_frame(masked, deflated): Calls parse_frame to parse the frame based on whether it is masked or deflated.
- parsed_frame, frame_opcode = process_frame(frame): After parsing the frame, it is processed by the process_frame function, which returns the parsed data and the frame type (opcode).
result: dict = {
'is_deflated': deflated,
'is_masked': masked,
'frame': parsed_frame,
'opcode': frame_opcode
}result: A dictionary containing details about the parsed frame:
- is_deflated: Whether the frame was deflated (compressed).
- is_masked: Whether the frame was masked (client frames).
- frame: The actual data in the frame (e.g., text, binary, or None for control frames).
- opcode: The type of frame (TEXT, BINARY, CLOSE, etc.).
return resultreturn result: The dictionary containing the parsed frame details is returned to the caller.
Summary of Key Functions and Parameters
- parse_frame_bytes(data_bytes: bytes): Takes raw WebSocket frame bytes and parses them.
– Parameters:
data_bytes: Raw frame bytes from the WebSocket.
– Returns:
A dictionary with information about the frame (whether it’s masked, deflated, its contents, and its opcode). - parse_frame(mask: bool, deflate: bool): Parses a WebSocket frame based on whether it’s masked or deflated.
– Parameters:
mask: Whether the frame is masked.
deflate: Whether the frame uses compression. - process_frame(current_frame): Processes a parsed frame and returns its contents based on its opcode.
– Parameters:
current_frame: The frame parsed from the WebSocket data. - read_exact(n: int): Reads exactly n bytes from the stream.
- run_coroutine(coroutine): Helper to run generator-based coroutines like Frame.parse.
WebSocket Python Frame Creation Function Explanation
The WebSocket Python frame creation function will create a frame on the client side (masked) to send to a WebSocket server or on the server side (unmasked) to send to the client. The constructs and serializes a WebSocket frame to be sent over a WebSocket connection. The function optionally applies compression and masking to the frame and chooses an appropriate opcode based on the data type.
Function Definition
def create_websocket_frame(
data: Union[str, bytes, bytearray],
deflate: bool = False,
mask: bool = False,
opcode: int = None
) -> bytes:Parameters:
- data (Union[str, bytes, bytearray]): The content (payload) of the WebSocket frame. It can be:
A string (str), which will be encoded as bytes using UTF-8.
A byte sequence (bytes or bytearray). - deflate (bool, default = False): Indicates whether to apply permessage-deflate compression (optional WebSocket extension).
- mask (bool, default = False): Specifies whether the frame should be masked. Masking is generally used for WebSocket frames from clients to servers.
- opcode (int, optional): Defines the type of the WebSocket frame. It will be determined based on the data type if not provided. Opcodes include:
Opcode.TEXT for text data,
Opcode.BINARY for binary data,
Opcode.CLOSE, Opcode.PING, and Opcode.PONG for control frames.
Return Type: The function returns the serialized WebSocket frame as bytes, ready to be transmitted.
Opcode Handling
if opcode is None:
if isinstance(data, str):
opcode = Opcode.TEXT
elif isinstance(data, (bytes, bytearray)):
opcode = Opcode.BINARY
else:
raise TypeError("Data must be of type str, bytes, or bytearray.")
else:
if not isinstance(opcode, int):
raise TypeError("Opcode must be an integer.")
if not isinstance(data, (str, bytes, bytearray)):
raise TypeError("Data must be of type str, bytes, or bytearray.")- If an opcode is not provided, it is inferred based on the data type:
1. The TEXT opcode is used if data is a str (since WebSocket text frames carry string data).
2. The BINARY opcode is used if data is bytes or bytearray.
3. If data is unsupported, it raises a TypeError. - If an opcode is provided, it checks that it’s an integer and that data is of a valid type (str, bytes, or bytearray). If not, it raises a TypeError.
Payload Encoding
if isinstance(data, str):
payload = data.encode('utf-8')
else:
payload = bytes(data)- If data is a string, it is encoded into bytes using UTF-8 (since WebSocket frames require the payload to be in binary form).
- If data is already in bytes or bytearray form, it is converted to bytes directly (ensuring a consistent data type for the payload).
Frame Creation
frame = Frame(opcode=opcode, data=payload)A Frame instance is created using the opcode determined earlier and the payload (in bytes). This Frame class is presumably imported from a WebSocket library and represents a WebSocket frame.
Optional Deflate Compression
extensions = []
if deflate:
permessage_deflate = PerMessageDeflate(
remote_no_context_takeover=False,
local_no_context_takeover=False,
remote_max_window_bits=15,
local_max_window_bits=15,
)
extensions.append(permessage_deflate)- If deflate is True, the function adds permessage-deflate compression to the frame by creating an instance of PerMessageDeflate. This sets up a WebSocket extension that compresses messages, improving bandwidth efficiency.
- Various parameters (remote_no_context_takeover, local_no_context_takeover, remote_max_window_bits, local_max_window_bits) control the compression behavior (how the compression context is maintained and the size of the compression window).
- The PerMessageDeflate extension has been added to the extensions list and will be passed on to the frame serialization process.
Frame Serialization
try:
frame_bytes = frame.serialize(
mask=mask,
extensions=extensions,
)
except Exception as e:
raise RuntimeError(f"Error serializing frame: {e}")- The frame.serialize() method converts the frame (with its opcode, payload, mask, and extensions) into a series of bytes, ready to be transmitted over a WebSocket connection.
- The mask argument is passed, determining whether the frame should be masked (typically done by clients in WebSocket communication).
- The extensions list is passed, containing the PerMessageDeflate object if compression was enabled.
- If serialization fails, it raises a RuntimeError, encapsulating any exception encountered during the process.
Return Value
return frame_bytesFinally, the serialized WebSocket frame (in bytes) is returned.
Summary
The create_websocket_frame function constructs a WebSocket frame with the given data, optionally compresses the frame, and optionally applies masking. It supports text and binary data, automatically selects the appropriate WebSocket opcode when necessary, and returns the frame in a serialized form ready for transmission. The function handles various edge cases, including incorrect data types or invalid opcodes, and throws appropriate exceptions when errors occur.
This code relies on external websockets for Python package classes or functions (Opcode, Frame, PerMessageDeflate).
WebSocket Python Frame Masking Type Determination Function Explanation
The function is_frame_masked(frame_bytes) is designed to determine whether a WebSocket frame is “masked.” In WebSocket protocol, the data frames exchanged between the client and server can be masked (from client to server) or unmasked (from server to client). This function explicitly checks that condition by analyzing the header of a WebSocket frame. The function can be used without the websockets external Python package.
Function Signature
def is_frame_masked(frame_bytes):- Parameters:
frame_bytes: This parameter is expected to be a bytes object representing the raw bytes of a WebSocket frame. - Returns:
The function returns a boolean value (True or False):
True if the frame is masked.
False if the frame is unmasked.
Check the length of the frame
if len(frame_bytes) < 2:
raise ValueError("Frame is too short to determine masking.")- The function checks if frame_bytes has fewer than 2 bytes.
- The WebSocket frame header must have at least 2 bytes, where:
1. The first byte is the FIN bit and the opcode.
2. The second byte contains the length and the MASK bit. - If frame_bytes is too short (i.e., less than 2 bytes), the function raises a ValueError, indicating that it can’t determine if the frame is masked.
Access the second byte of the frame
second_byte = frame_bytes[1]The second byte of the WebSocket frame (at index 1) contains important metadata about the frame:
- The most significant bit (MSB) of this byte (bit 7) indicates whether the frame is masked (the “MASK” bit).
- This byte’s lower 7 bits (bits 0–6) indicate the payload length (or part of it).
Extract and check the MASK bit
mask_bit = (second_byte & 0x80) != 0 # 0x80 is 1000 0000 in binary- The bitwise AND operation (second_byte & 0x80) isolates the second byte’s most significant bit (the MASK bit).
- 0x80 is a hexadecimal representation of the binary value 1000 0000. This value ensures that all bits except the MSB are ignored in the second byte.
- The frame is masked if the MSB (MASK bit) is set to 1. Otherwise, it is not.
- The result of second_byte & 0x80 is compared to 0 to determine whether the MASK bit is set:
1. If the result is non-zero, the MASK bit is set (i.e., the frame is masked).
2. If the result is zero, the MASK bit is not set (i.e., the frame is unmasked).
Return the result
return mask_bitThe function returns the value of mask_bit:
- True if the frame is masked.
- False if the frame is not masked.
WebSocket Masking Context
In the WebSocket protocol:
- Client-to-server frames must always be masked, meaning the MASK bit is set to 1.
- Server-to-client frames are generally not masked, meaning the MASK bit is 0.
Summary
The function checks if the input frame_bytes is at least 2 bytes long. It reads the second byte of the WebSocket frame, which contains the MASK bit in the most significant position (bit 7).
Using a bitwise operation, the function extracts the MASK bit and checks whether it is set. Lastly, It returns True if the MASK bit is set (indicating the frame is masked) and False otherwise.
Example
Let’s say we have a WebSocket frame with the following header:
frame_bytes = b'\x81\xfe' # Binary: 1000 0001 1111 1110- The first byte (0x81) is not relevant for masking.
- The second byte (0xfe or 1111 1110 in binary) has the MASK bit set to 1 (because the MSB is 1), meaning the frame is masked.
The function will return True for this input.
Another example:
frame_bytes = b'\x81\x7e' # Binary: 1000 0001 0111 1110In this case, the second byte (0x7e or 0111 1110 in binary) has the MASK bit set to 0, meaning the frame is unmasked. The function will return False for this input.
WebSocket Python Frame Deflation Determination Function Explanation
The function is_frame_deflated(frame_bytes) determines whether a WebSocket frame is deflated (compressed). This is useful when working with WebSocket frames that may be compressed using extensions like permessage-deflate, which can compress the data sent in the frame. This function can also be used without the external websockets Python package.
Let’s break down the function in detail.
Function Signature
def is_frame_deflated(frame_bytes):Parameters:
- frame_bytes: This is a bytes object representing the raw WebSocket frame data.
Returns a boolean (True or False):
- True if the frame is deflated (compressed).
- False if the frame is not deflated.
Check the length of the frame
if len(frame_bytes) < 1:
raise ValueError("Frame is too short to determine deflation status.")- This step ensures that the input frame_bytes has at least 1 byte.
- The WebSocket frame header starts with the first byte containing the RSV1 bit that indicates deflation.
- If frame_bytes is too short (i.e., less than 1 byte), the function raises a ValueError because it can’t determine whether the frame is deflated.
Access the first byte of the frame
first_byte = frame_bytes[0]The first byte of the WebSocket frame contains several important flags:
- FIN bit (bit 7) – indicates if this is the final frame in a sequence.
- RSV1, RSV2, and RSV3 bits (bits 6, 5, 4) are reserved for extensions such as compression (RSV1 is relevant for deflation).
- Opcode (bits 3 to 0) – specifies the type of frame (e.g., text, binary, ping, etc.).
Extract and check the RSV1 bit (which indicates compression)
rsv1 = (first_byte & 0x40) != 0 # 0x40 is 0100 0000 in binary- The RSV1 bit (bit 6) indicates whether the frame is compressed (deflated) when using the WebSocket extension for compression, such as permessage-deflate.
- RSV1 is the second most significant bit in the first byte.
- The function performs a bitwise AND operation (first_byte & 0x40) to isolate the RSV1 bit.
- 0x40 is a hexadecimal value that represents the binary value 0100 0000.
- This operation zeroes out all bits except bit 6 (RSV1), the only bit we’re interested in.
The frame is compressed if the RSV1 bit is set (1). The frame is not compressed if it is not set (0). The comparison (first_byte & 0x40) != 0 checks whether the RSV1 bit is set to 1 or not:
- If the result of the AND operation is non-zero, the RSV1 bit is set (deflated).
- If the result is zero, the RSV1 bit is not set (not deflated).
Return the result
return rsv1The function returns the value of rsv1:
- True if the RSV1 bit is set (indicating that the frame is compressed).
- False if the RSV1 bit is not set (indicating that the frame is not compressed).
WebSocket Deflation Context
In the WebSocket protocol, compression can be enabled through extensions such as permessage-deflate. When compression is in use, the RSV1 bit (in the first byte of the frame header) is set to 1 to indicate that the frame’s payload is compressed.
- RSV1: This bit indicates that the frame’s payload is compressed using a specific compression method.
- RSV2 and RSV3 are reserved for future use and are typically set to 0 unless another extension uses them.
Summary
The function checks that the frame_bytes input has at least 1 byte. It reads the first byte of the WebSocket frame header.
The function isolates and checks the RSV1 bit (bit 6) using a bitwise operation to determine if the frame is deflated (compressed). Finally, it returns True if the RSV1 bit is set (indicating the frame is compressed) and False if the RSV1 bit is not set (indicating the frame is not compressed).
Example
Suppose we have a WebSocket frame with the following header:
frame_bytes = b'\xc1\x7e' # Binary: 1100 0001 0111 1110- The first byte is 0xc1 (binary: 1100 0001).
- The RSV1 bit is the second most significant bit (bit 6); in this case, it is 1, indicating that the frame is deflated (compressed).
The function will return True for this input.
Another example:
frame_bytes = b'\x81\x7e' # Binary: 1000 0001 0111 1110- The first byte is 0x81 (binary: 1000 0001).
- The RSV1 bit (bit 6) is 0, indicating that the frame is not compressed.
The function will return False for this input.
TCP Server with WebSocket Python Parsing Example
Here is an oversimplified Python example of a TCP server with the above WebSocket Python functions:
import socket
import threading
from typing import Union
# Importing classes and methods from the provided code
from websockets.frames import Opcode
from websockets.exceptions import InvalidHeaderValue
def start_server(host: str, port: int):
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind((host, port))
server_socket.listen(5)
print(f"Server listening on {host}:{port}")
try:
while True:
client_socket, address = server_socket.accept()
print(f"Connection established with {address}")
client_handler = threading.Thread(
target=handle_client, args=(client_socket,)
)
client_handler.start()
except KeyboardInterrupt:
print("Shutting down server...")
server_socket.close()
def handle_client(client_socket: socket.socket):
try:
# Perform WebSocket handshake
request_data = client_socket.recv(1024)
response_data = create_byte_http_response(request_data)
client_socket.sendall(response_data)
# WebSocket connection established
frame_parser = WebsocketFrameParser()
while True:
# Receive WebSocket frame from the client
frame_data = client_socket.recv(1024)
if not frame_data:
break
# Parse WebSocket frame
parsed_frame = frame_parser.parse_frame_bytes(frame_data)
opcode = parsed_frame['opcode']
frame_content = parsed_frame['frame']
if opcode == 'TEXT':
print(f"Received message: {frame_content}")
response_frame = create_websocket_frame(
data=f"Echo: {frame_content}", opcode=Opcode.TEXT, mask=False
)
client_socket.sendall(response_frame)
elif opcode == 'CLOSE':
print("Connection closing...")
break
elif opcode == 'PING':
print("Received PING, sending PONG...")
pong_frame = create_websocket_frame(
data=b"", opcode=Opcode.PONG, mask=False
)
client_socket.sendall(pong_frame)
elif opcode == 'PONG':
print("Received PONG")
else:
print(f"Unsupported opcode: {opcode}")
except (InvalidHeaderValue, ConnectionResetError) as e:
print(f"Error: {e}")
finally:
client_socket.close()
print("Connection closed.")
if __name__ == "__main__":
start_server(host="127.0.0.1", port=8765)This server listens for incoming frames, parses them, and then echoes them back. It demonstrates how a server can handle WebSocket frames over a TCP connection, allowing developers to manage the communication manually. The script provides the foundational understanding needed to handle WebSocket frames in a custom server environment, offering the ability to fine-tune communication, add specific features, and perform advanced troubleshooting.
Implementing WebSocket parsing in a custom server can be particularly useful in environments where integration with other communication protocols is needed or complete control over the connection is required for security and reliability reasons.
Considerations: The above code is a simple example that does not consider security and complex situations. Don’t forget that WebSocket is a Full-Duplex communication protocol.
Partial TCP Client with Websocket Python Parsing
We can also use the parsing class “WebsocketFrameParser” on the client side. Here’s an example:
import socket
import time
from typing import Union
from websockets.frames import Opcode
def start_client(host: str, port: int):
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect((host, port))
print(f"Connected to server at {host}:{port}")
try:
# Perform WebSocket handshake
# You should perform your own WebSocket Handshake Request mechanism since we haven't tested it.
handshake_request: bytes
# Send the handshake to the server.
client_socket.sendall(handshake_request)
# Receive handshake response from the server
response_data = client_socket.recv(1024)
print(f"Received handshake response:\n{response_data.decode('utf-8')}")
# Send a text message to the server
message = "Hello, Server!"
frame_data = create_websocket_frame(data=message, mask=True)
client_socket.sendall(frame_data)
print(f"Sent message: {message}")
# Receive a response frame from the server
frame_data = client_socket.recv(1024)
# If you have a while loop, the parser class should be above it and not inside the loop. Since it holds the instances of the deflate extension classes.
frame_parser = WebsocketFrameParser()
parsed_frame = frame_parser.parse_frame_bytes(frame_data)
print(f"Received from server: {parsed_frame['frame']}")
# Send a close frame to the server
close_frame = create_websocket_frame(data=b"", opcode=Opcode.CLOSE, mask=True)
client_socket.sendall(close_frame)
print("Sent CLOSE frame to server")
except Exception as e:
print(f"Error: {e}")
finally:
client_socket.close()
print("Connection closed.")
if __name__ == "__main__":
start_client(host="127.0.0.1", port=8765)It’s essential to understand that this is just a practical example. An actual TCP client with WebSocket parsing will be much more complex. In addition, we didn’t test the WebSocket handshake request mechanism much, so it is not here. You will need to implement one yourself. However, the websockets Python package has the necessary functions.
Reference WebSocket Python HTTP Handshake
Here’s an example Python code you can try to generate an HTTP handshake request of a WebSocket protocol:
import logging
from websockets.extensions.permessage_deflate import ClientPerMessageDeflateFactory
enable_logging = True
# Set up extensions
permessage_deflate_factory = ClientPerMessageDeflateFactory()
# Parse the WebSocket URI.
wsuri = parse_uri('ws://example.com/websocket')
# Create the protocol instance
protocol = ClientProtocol(
wsuri,
extensions=[permessage_deflate_factory],
)
if enable_logging:
logging.basicConfig(level=logging.DEBUG)
protocol.debug = True
# Perform the handshake and emulate the connection and request sending.
request = self.protocol.connect()
self.protocol.send_request(request)
_ = self.protocol.data_to_send()
# At this state, the protocol.state is State.CONNECTINGSince we didn’t need to implement this anywhere, there is no working guarantee. This can be used as a starting point reference for testing purposes. Because there is a random key generated in this HTTP request, which is then used by the server to generate a valid HTTP handshake response, use it with caution security-wise.
Conclusion
We covered the WebSocket protocol, framing, handshake, and creating clients and servers using Python. Manual parsing offers insights into WebSocket communication.
The ability to implement custom features, handle unique edge cases, and optimize performance are key reasons to learn about the manual parsing of WebSocket frames. Understanding the underlying mechanisms of the WebSocket protocol also provides the foundation for building highly reliable and efficient communication systems.
As more applications demand real-time interactivity, knowing how to work with WebSocket from the ground up gives developers the tools to meet these demands with precision and creativity.