Reverse Engineering
My biggest personal project (~2 years)
Date: 2022 - End 2023
Overview
This large-scale personal project, spanning over three years, allowed me to develop strong foundations in IT areas such as reverse engineering, data processing, web developpement, data management and automation. By reverse-engineering a game client and building a distributed system from scratch, I developed a comprehensive infrastructure including a Python proxy server, a data parsing system, and both MySQL and MongoDB databases. The project integrates automation through ADB, custom bots, and webhooks, with a focus on real-time data processing and event-driven actions across multiple platforms.
Key Features
- Reverse Engineering: Reverse engineering a game client and managing the communication with a Python server.
- Packet Injection: After traffic mirroring, a custom Packet Injection System allows for modification or addition of packets in real-time, enabling automation and control of game events.
- Data Processing: Handling large volumes of data, managing databases, and parsing complex structures in real-time.
- Python Proxy: Utilizes a Python proxy to mirror traffic, sending data into a queue system for further processing.
- Data Parsing & Dispatching: A Python-based parser processes and structures data, then dispatches it to corresponding components.
- Database Management: Integration with both MySQL and MongoDB databases for robust data storage and access.
- Website Development: A LAMP stack-based website displays processed data including user stats, in-game chat, and player statistics, with dynamic maps and visualizations.
- Automation with ADB: Local Python program at the client level, automated control of the game client through ADB, allowing for automated gameplay based on stored data and predefined actions/reactions.
- Distributed Automation System: The automation system integrates with local and remote bots, ensuring seamless interactions and event handling.
- Discord Bots: Developed multiple Python-based Discord bots that retrieve and push data to both MySQL and MongoDB databases. The bots handle real-time interactions, generating images, maps, and event-driven messages.
Technologies Used
Proxy & Parser Backend
| Name | Badge | Purpose |
|---|---|---|
| Python |  | Main language used for reverse engineering and data processing |
| socketserver |  | Python Library used to create a server to handle the communication between the game Client and the Server |
| named semaphore |  | System used to send information from the proxy to the parser |
Hosting & Database
| Name | Badge | Purpose |
|---|---|---|
| LAMP Stack |  | Full-stack solution for web development |
| Apache2 |  | Web server for hosting the PHP application |
| MySQL |  | Database system for storing and managing data |
| PHP |  | Backend for database management and dynamic content generation |
Remote Database
| Name | Badge | Purpose |
|---|---|---|
| MongoDB |  | Database system for storing and managing data |
| PyMongo |  | Python Library used to interact with the MongoDB database |
| MongoDB PHP Library |  | PHP Library used to interact with the MongoDB database |
Web Development
| Name | Badge | Purpose |
|---|---|---|
| HTML5 |  | Structure of the website |
| CSS3 |  | Design and layout of the website |
| JavaScript |  | Core logic of the website and simulations |
| Bootstrap |  | Website design and responsiveness |
| Plotly |  | Dynamic graphs for data visualization |
| ChartJs |  | Graphs for interactive results display |
| Leaflet |  | Open-source JavaScript library for mobile-friendly interactive maps |
Automation
| Name | Badge | Purpose |
|---|---|---|
| ADB |  | Android Debug Bridge for automating actions on the game client |
| Pure Python ADB |  | Python Library used to interact with the ADB server |
Communication & Bots
| Name | Badge | Purpose |
|---|---|---|
| Discord.py |  | Python Library used to interact with the Discord API |
| Discord Slash Commands |  | Python Library used to interact with the Discord API for slash commands |
| Webhooks |  | Triggering external actions from internal data events |
Hosting & Version Control
| Name | Badge | Purpose |
|---|---|---|
| OVH |  | Hosting provider for the web application |
| Cloudflare |  | Web infrastructure and website security company |
| GitHub |  | Version control and collaboration platform for the project |
Helper Website
| Name | Badge | Purpose |
|---|---|---|
| Editor X |  | Website builder used to create the helper website |
Challenges and Learnings
This project presented numerous technical challenges, providing invaluable learning opportunities in the areas of reverse engineering, automation, data flow management, and system integration.
Reverse Engineering: One of the most significant challenges was understanding and deconstructing the communication protocols between the game client and the server. This involved reverse engineering the structure of the network packets exchanged in real time. Through extensive traffic mirroring and packet analysis, I developed a deep understanding of how the game's internal state was represented and transmitted. This step-by-step deconstruction of the game's communication protocol was essential for implementing automation and custom interactions.
Data Flow/ Big Data Management: Managing data flow between various components of the system—such as the proxy, queue system, parser, and dispatcher—was critical to ensuring the smooth operation of the distributed system. I had to design efficient pipelines that handled real-time data processing, while ensuring that the data passed through each stage without bottlenecks or corruption. This required a robust architecture to manage the high-frequency data flows and the synchronization between systems.
Automation with ADB: A key feature of the project was automating gameplay through Android Debug Bridge (ADB). I developed an advanced system capable of interacting with the mobile game client to perform automated tasks. Synchronizing these tasks with the injected packets and data processing workflows added complexity, particularly in maintaining the accuracy and responsiveness of the automated actions without disrupting the game client.
Integration with Discord: Integrating automation tools with Discord bots allowed me to extend the system’s functionality beyond the game client. I implemented multiple bots that interact with MySQL and MongoDB databases, automate complex interactions, and provide real-time feedback to users through Discord channels. These bots not only provided information but also performed actions that helped manage and control different parts of the system, bridging the gap between the user interface and backend processing.
Data Visualization: Creating dynamic and interactive visualizations was essential for monitoring the system and analyzing gameplay data. I used web technologies such as Leaflet for geographical data visualization and Plotly/ChartJS for generating rich, interactive plots. These tools allowed me to present complex data in an intuitive format, facilitating both real-time monitoring and retrospective analysis of the automated systems and their outputs.
Web Development: Building a full web application was a significant challenge. I used a LAMP stack to create a dynamic website that could visualize processed data and real-time events. This required integrating data from multiple sources, ensuring data consistency, and providing a responsive user interface that could handle a wide range of user interactions. The website served as a central hub for monitoring and controlling the system, providing a user-friendly interface for both administrators and end-users. This project allowed me to learn for the first time how to build a website from scratch, from the design to the backend, and the database management to managing the deployment and the hosting on my own server but also while ensuring the security of the website.
System Scalability: Designing for scalability was a core requirement of the project. The system was built to support multiple databases, bots, and automation scripts that interact with live users, while also being flexible enough to handle future expansions. Ensuring that each component could scale independently was a challenge, especially when coordinating interactions between different elements like the proxy server, local scripts, databases, and remote systems. Scalability was achieved through careful planning, modular system design, and rigorous testing.
Packet Injection: Another significant challenge was mastering packet injection. Once I had reverse-engineered the packet structures, I built a custom system to inject packets into the game's communication stream. The challenge here was ensuring that the injected packets were timed correctly and processed seamlessly by the game client without causing crashes or unexpected behavior. This involved not only technical knowledge but also a deep understanding of how the game’s mechanics reacted to external inputs.
Through these challenges, I gained a deep understanding of network protocols, distributed system architecture, website developpement and automation techniques. This project significantly improved my skills in reverse engineering, data management, system scalability, and the integration of complex systems across multiple platforms.