Refactor: add ppadb instead of pyautogui#8
Conversation
* feat: Update ADB actions and testing scripts; remove obsolete tests and enhance functionality * chore: Remove compiled Python bytecode files from __pycache__ * feat: Update game end handling to click "OK" button and adjust reset logic * docs: Simplify installation instructions by consolidating package installations and remove the PyAutoGUI requirements --------- Co-authored-by: Brody Dai <138156236+krazyness@users.noreply.github.com>
| return False | ||
|
|
||
| try: | ||
| self.device.shell(f"input touchscreen swipe {x} {y} {x} {y} 500") |
There was a problem hiding this comment.
note: I'm using touchscreen swipe instead of tap because it allows us to controll the duration of the input, sometime tap command lift the finger too fast and the card is not applied
|
Have you been able to run multiple instances simultaneously? |
No, it requires more modification to the repository. I've looked a bit into it yesterday night and pytorch has a multiprocessing package which could suit for our needs, but I think we should split the PRs especially with such radical changes, First release this PR if you think it is mature enough and after that release a PR to scale the training process |
|
Can confirm this works... Tested myself. Just wanted to say that this project works perfectly on macOS as well, as I have tested it on there. I wasn't able to download the pyadb-reborn library, so I went for the original, less frequently updated one. |
I'm currently testing pytorch multiprocessing, It looks to work perfectly fine with fine tunning |
I haven't tested on MacOS, try to add adb in your env path, then try |
I had to install the adb packages with home brew, and after that it worked. Is it intentional that the the terminal says pressing okay at the end of the game, but never actually does? |
Definitely not, it should tap on the "ok" button |
|
Before multi instance training is added we should update the training logic. Right now the bot is learning to play the game the wrong way. I don't know how easy it would be to add card type detection and counters, as well as cycle counting, but if that was implemented alongside multiple people running multi instance training the bot would get pretty decent at cr much faster. Also later on if possible elixr management would be huge. |
I would love to help the implémentation of it, I'm not new to python but I'm really new to AI, I've thought about changing the state, implementing HP tower vision. I'm definitely capable of doing that |
I've never even heard of roboflow before so I'm not too sure what AI stuff is going on with it either. I wonder if it would be possible to add card recognition and then feed human played games into the epsilon instead of adding hardcoded counters. If you just feed the bot all of the correct actions with positive reinforcement for a scenario it would improve learning speed and skill immensely. Kind of like a neural net. That would be pretty complicated though and probably not entirely feasible. If it did work perfectly you could have a bot with top 1% skill 😀 |
* feat: Update ADB actions and testing scripts; remove obsolete tests and enhance functionality * chore: Remove compiled Python bytecode files from __pycache__ * feat: Update game end handling to click "OK" button and adjust reset logic * docs: Simplify installation instructions by consolidating package installations and remove the PyAutoGUI requirements * feat: Fix the click ok button after the game, Add timing decorator for performance measurement and apply it to relevant methods * feat: Refactor screenshot handling and update device dimensions for better adaptability * feat: Enhance click_ok_button method to support multiple confidence levels for improved button detection
|
RuntimeError: Error(s) in loading state_dict for DQN: |
My bad I didn't mean to push this commit on this PR, I've forked this repository and trying a lot of things, from multiprocessing training to making the AI aware of its deck and its associated cost and masking unavailable actions, |
All good. |
There was a problem hiding this comment.
Pull Request Overview
This PR refactors the bot's input and screenshot mechanisms by replacing PyAutoGUI with Android Debug Bridge (ADB) for direct communication with the BlueStacks emulator. This enables headless operation and eliminates dependencies on screen positioning and window management.
Key changes include:
- Replaced PyAutoGUI-based screen capture and input with ADB commands for direct device interaction
- Added comprehensive ADB setup and testing infrastructure with automated device discovery
- Updated coordinate systems from screen-relative to device-relative positioning
Reviewed Changes
Copilot reviewed 10 out of 40 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| Actions.py | Complete refactor replacing PyAutoGUI with ADB client for screenshots, clicks, and template matching |
| setup_adb.py | New script for automated ADB setup, device connection, and configuration validation |
| utils.py | New utilities for ADB screenshot capture and performance timing decorators |
| env.py | Updated to use ADB-based actions and removed PyAutoGUI dependencies |
| test_adb_actions.py | Comprehensive test suite for validating all ADB-based functionality |
| README.md | Updated installation instructions to include ADB setup requirements |
| elixir_verification.py | Removed PyAutoGUI-based elixir verification script |
| if device_id: | ||
| cmd.extend(["-s", device_id]) | ||
| else: | ||
| cmd.append("-d") # Default to the only connected USB device |
There was a problem hiding this comment.
The -d flag assumes there's exactly one USB device connected, but BlueStacks typically connects via TCP. This could cause the command to fail if no USB devices are present. Consider using a more specific device selection or handling the case where no devices match the -d filter.
| cmd.append("-d") # Default to the only connected USB device | |
| def get_first_adb_device(): | |
| """Return the first available device ID from adb devices, or None if none found.""" | |
| try: | |
| result = subprocess.run( | |
| [ADB_PATH, "devices"], | |
| stdout=subprocess.PIPE, | |
| stderr=subprocess.PIPE, | |
| check=True, | |
| timeout=2, | |
| encoding="utf-8" | |
| ) | |
| lines = result.stdout.strip().splitlines() | |
| # Skip the first line ("List of devices attached") | |
| for line in lines[1:]: | |
| if not line.strip(): | |
| continue | |
| parts = line.split() | |
| if len(parts) >= 2 and parts[1] == "device": | |
| return parts[0] | |
| return None | |
| except Exception as e: | |
| print(f"Error listing adb devices: {e}") | |
| return None | |
| def take_screenshot(device_id=None): | |
| """Take screenshot from specified device or the default device""" | |
| # Build the command | |
| cmd = [ADB_PATH] | |
| # Add device selection if specified, otherwise pick the first available device | |
| if device_id: | |
| cmd.extend(["-s", device_id]) | |
| else: | |
| selected_device = get_first_adb_device() | |
| if not selected_device: | |
| print("No ADB devices found. Please connect a device or specify a device_id.") | |
| return None | |
| cmd.extend(["-s", selected_device]) |
| confidences = [0.8, 0.6, 0.4] | ||
|
|
||
| # Define the region for the OK button in device coordinates | ||
| ok_button_region = (0, self.HEIGHT / 2, self.WIDTH, self.HEIGHT) |
There was a problem hiding this comment.
Division of integers by 2 may result in float values, but the region tuple should contain integer coordinates. Use integer division (self.HEIGHT // 2) instead.
| ok_button_region = (0, self.HEIGHT / 2, self.WIDTH, self.HEIGHT) | |
| ok_button_region = (0, self.HEIGHT // 2, self.WIDTH, self.HEIGHT) |
| ("Card Play", self.test_card_play), | ||
| ("Template Matching", self.test_template_matching), | ||
| ("Swipe Functionality", self.test_swipe), | ||
| ("Detect Game End", self.actions.detect_game_end), |
There was a problem hiding this comment.
The detect_game_end method is called directly without wrapping in a lambda or method, which means it will be executed immediately during list construction rather than when the test runs. This should be lambda: self.actions.detect_game_end() or a proper test method.
| ("Detect Game End", self.actions.detect_game_end), | |
| ("Detect Game End", lambda: self.actions.detect_game_end()), |
| pyautogui.moveTo(1611, 831, duration=0.2) | ||
| pyautogui.click() | ||
| print("All cards are Unknown, clicking at center and skipping move.") | ||
| self.actions._click(640, 400) # Click at center of screen in device coordinates |
There was a problem hiding this comment.
Hard-coded coordinates (640, 400) should be derived from device dimensions or defined as constants. Consider using self.actions.device_width // 2, self.actions.device_height // 2 or similar calculations based on the device's actual dimensions.
| self.actions._click(640, 400) # Click at center of screen in device coordinates | |
| self.actions._click(self.actions.WIDTH // 2, self.actions.HEIGHT // 2) # Click at center of screen in device coordinates |
| # Card name to position mapping (will be updated during detection) | ||
| self.current_card_positions = {} | ||
| try: | ||
| self.device.shell(f"input touchscreen swipe {x} {y} {x} {y} 300") |
There was a problem hiding this comment.
The hard-coded duration value of 300 should be extracted as a constant or parameter. This magic number makes the code less maintainable and the duration isn't obvious from the method signature.
| self.device.shell(f"input touchscreen swipe {x} {y} {x} {y} 300") | |
| self.device.shell(f"input touchscreen swipe {x} {y} {x} {y} {self.DEFAULT_CLICK_DURATION}") |
| return count | ||
| else: | ||
| return 0 | ||
|
|
There was a problem hiding this comment.
The bounds checking inside the loop could be moved outside since elixir_y is constant and screenshot_np.shape doesn't change during iteration. Consider validating dimensions once before the loop starts.
|
Don't listen to Copilot, I accidentally clicked on the "review" button for Copilot. I'll take a look at it myself. |
The hardest thing is to create a robust dataset that allows the AI to label entities on the screen with high accuracy, I took a look at roboflow available dataset, and I've fond some with around 3k unlabeled images, labeling image is a nightmare to do from what i've seen |
Do you just have to manually label each image? 2 people could probably do it in a month, or something with chatgpt api that auto labels. If labeling meant fully working bot learning from human gameplay I would do it. |
|
Agree with this. Would definitely put my time into this if it meant a very capable bot 😅On 9 Aug 2025, at 1:29 pm, craftguy1243 ***@***.***> wrote:craftguy1243 left a comment (krazyness/CRBot-public#8)
wrong way. I don't know how easy it would be to add card type detection and counters, as well as cycle counting, but if that was implemented alongside multiple people running multi instance training the bot would get pretty decent at cr much faster. Also later on if possible elixr management would be huge.
I would love to help the implémentation of it, I'm not new to python but I'm really new to AI, I've thought about changing the state, implementing HP tower vision. I'm definitely capable of doing that
I've never even heard of roboflow before so I'm not too sure what AI stuff is going on with it either. I wonder if it would be possible to add card recognition and then feed human played games into the epsilon instead of adding hardcoded counters. If you just feed the bot all of the correct actions with positive reinforcement for a scenario it would improve learning speed and skill immensely. Kind of like a neural net. That would be pretty complicated though and probably not entirely feasible. If it did work perfectly you could have a bot with top 1% skill 😀
The hardest thing is to create a robust dataset that allows the AI to label entities on the screen with high accuracy, I took a look at roboflow available dataset, and I've fond some with around 3k unlabeled images, labeling image is a nightmare to do from what i've seen
Do you just have to manually label each image? 2 people could probably do it in a month, or something with chatgpt api that auto labels. If labeling meant fully working bot learning from human gameplay I would do it.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>
|
|
I'm very new to AI and I'm unsure if 3k images will be enough to make a robust enough recognition model |
Share the link I'll help. I think the best path to take would be watching real people play as it would be fairly easy for the bot to just constantly watch top games from tv royale or you playing. There really isn't a ruleset you can use to train a cr bot that will be good because of the millions of edge cases. |
This pull request refactors the bot's input and screenshot mechanisms to eliminate the PyAutoGUI dependency.
Key Changes:
Input & Screenshots: The bot now uses the Android Debug Bridge (ADB) to send inputs and capture screenshots from the emulator. This removes all constraints related to screen size and window positioning.
Headless Operation: The training process can now run headlessly, freeing up the user's mouse and monitor. This is a significant step towards scalability, allowing us to run multiple emulators simultaneously to accelerate the bot's learning.
Image Updates: The previous training images were captured via Windows screenshots, leading to inconsistencies. This PR updates all training images to be captured directly from a Bluestacks instance, ensuring data quality.