Project Ace
- Sam Blakeman
- Jun 18
- 40 min read
Updated: Jun 19
Introduction
In this blog post I will outline some of my experiences and thoughts from working at Sony AI over the last 6 years on Project Ace. This will serve as a bit of a personal record of what I experienced but I also hope it may help others facing similar challenges. The focus will be specifically on the area that I worked on - Reinforcement Learning for control. Everything I write here is a reflection of my own personal thoughts and does not represent Sony in any way. I can also assure you that every word here was written by me and will remain untouched by large language models.
Before delving into the technical journey, I think it is worth contextualizing Project Ace and explaining why we did what we did. During Project Ace we had the challenge of trying to compete with professional table tennis players with an autonomous robot. To keep the competition fair we had to adhere to official International Table Tennis Federation (ITTF) rules, which meant we could not resort to alterations to the game, such as markings on the ball, custom rackets, illegal serves (must be a vertical throw of a specific height behind the table) or a reduced play area.
Why choose such a seemingly arbitrary challenge? On the surface it may not be clear why competing with professionals at table tennis is an interesting problem to tackle. In recent years, we have perhaps been de-sensitized to seeing robots running and doing backflips or autonomous systems driving cars or flying drones. However, we have never seen an autonomous system compete with human professionals in a competitive sport that involves direct interaction between two competitors. Why is this fundamentally different? To compete with humans at the pinnacle of sport, an autonomous system must have several properties:
Move at the boundaries of human reaction times and speed in the physical world
Sense the world around it at extremely low latency and with high fidelity
Select actions in a highly unpredictable environment with a near infinite number of possible incoming observations and outgoing actions.
Operate safely in close proximity to humans with minimal time to avoid collision
Be able to respond and deal with actions from humans deliberately trying to expose failures in the system (i.e. adversarial attacks)
Learn from extremely limited data on professional opponents (does not have the luxury of years of playing top players in the real world)
To combine all of these properties into a single system that performs reliably is an extremely difficult challenge that I believe no robotics application has been able to achieve until now.
Simplifying the Learning Problem for a First Prototype
When I first joined the project out of my PhD, I did not know what the goal of the project was. All I knew was that it involved robotics and that Reinforcement Learning (RL) may be useful for solving the problem. I was drawn to the project because it represented a completely new challenge. My PhD was in computational neuroscience and had largely been about applying RL to video games. Project Ace would require me to learn about the issues of applying RL to a real-world physical system and the differences between cognitive and physical intelligence (partly motivated by my love for sport).
You can imagine then that once I joined the project and learnt about its purpose, it took me a while to fully appreciate the gravity of the task and the challenges that lay before us. Indeed, I joined in the first year of the project when a lot of the groundwork still needed to be done. Initially (and for what turned out to be the whole project), I was tasked with the problem of ‘robot control’ i.e. learning how to map observations of the ball to robot commands such that we return the ball in a competitive manner.
In the very first few months of brainstorming how to solve the ‘control’ problem, we started off with ‘floating rackets’ in simulation that could be directly controlled via velocity commands, which were learnt using RL (Video 1). This led to some nice rallies in simulation and in many ways was similar to the work I had done in my PhD on video games. However, it became painfully clear that we were side-stepping several key problems that we would have to solve in order to achieve our true goal: (1) our simulation needed to be physically accurate and represent uncertainty, (2) we needed to take into account the physical capabilities of the robot and (3) the returns we learnt needed to be of a professional level. While obvious, this was already a huge learning curve for me, I had been used to having standard video game benchmarks and then focusing on algorithmic ideas to improve performance. Here suddenly there was a large shift from algorithmic ideas, to designing the environment in which the algorithm would be implemented. There was no point having the fanciest RL algorithm if it did not learn in an environment that faithfully captured the behaviour of the ball, the uncertainty of the observations we would receive and the response of the hardware given its physical limitations.
Video 1: Our very first prototype in simulation, using Reinforcement Learning to learn a policy that output velocity commands for a floating racket.
We therefore changed our approach quite substantially and focused on a (then recent) paper by Tebbe et al. (2021) called ‘Sample-Efficient Reinforcement Learning in Robotic Table Tennis’. This really went to the other end of the spectrum; focusing much more on learning in a physically plausible environment and simplifying the learning algorithm as much as possible. In this approach the problem was decomposed as follows:
The flight of the ball was predicted using an Extended Kalman Filter (EKF). This was used to predict the state of the ball (position, velocity and spin) as it crossed the edge of the table in front of the robot
This predicted state was used as an input to a neural network policy that mapped it to a desired hitting velocity and orientation of the racket. This policy was learnt using an actor-critic architecture, where the critic explicitly predicted the landing state of the ball and the height of the ball as it crossed the net (after the robot had hit the ball). The reward was simply a differentiable function of these two quantities and so the gradient could be used to update the policy as usual.
The Reflexxes Library was used to compute the desired motion of the robot such that it satisfied the robot kinematics and the desired hitting state of the racket calculated in the previous step.
From an RL/learning perspective this simple approach had several interesting implications. The learning problem was reduced to a supervised learning problem where the critic takes the predicted ball state and racket state as input and outputs the resulting ball landing position and height over the net. The reward was then computed as a function of those explicit critic outputs such that it remained differentiable for policy optimization. As each episode was a single hit of the ball, the learning can actually be framed as a bandit problem i.e. a Markov Decision Process with no time-component. This drastically simplifies the learning problem as the critic does not need to reason about future rewards based on future actions produced by the policy. The problem of dealing with changing ball observations and predictions is not dealt with by the learning algorithm but instead by querying the policy repeatedly to get new racket state targets during execution (learning always used the last prediction for updates). Similarly, the problem of dealing with uncertainty of ball observations is off-loaded from the policy to the EKF, which must infer the state of the ball and then make a subsequent prediction. One of the perks of this simplification of the learning problem is that the approach becomes highly data-efficient and afforded learning directly in the real-world using only several hundred shots.
Figure 1 - Our initial control approach on Project Ace. (1) An Extended Kalman Filter was used to predict the future flight of the ball. From this prediction the position, velocity and spin of the ball was extracted as the ball crossed the far edge of the table (on the robots side). (2) This predicted ball state was given as input to a neural network policy that output the desired racket rotation and velocity in order to return the ball to the table (position was fixed based on the prediction). (3) With the desired racket position, velocity and rotation a racket trajectory was generated that would intercept the ball at the correct point in time. (4) The racket trajectory was converted into joint commands using inverse kinematics. During learning the process is only executed once however during real-world execution it is repeated in an open-loop.
With a variation of this approach (see Figure 1) we could have co-operative rallies with the robot (Video 2)! For me, this part of the project was a great lesson in more classical robotics approaches such as motion planning and state estimation. It also demonstrated how decomposing the problem into its specific sub-components can enable fast progress and a fully working prototype.
Video 2 - A co-operative rally using the control approach from Figure 2. The robot explicitly hits the ball on a plane perpendicular to the edge of the table based on the predicted flight of the ball. The learning algorithm focuses on learning the correct racket velocity and rotation at this point in order to return the ball.
However, as you can see from the video we were still a long way from competitive play at the level of human professionals. Indeed, this was in a co-operative setting where we were actively trying to play within the capabilities of the robot to have as long a rally as possible. It quickly became clear that by splitting the problem into these different sub-components and simplifying the learning problem we had several key issues:
Constraining the robot to hit on a single plane at the edge of the table was extremely limiting and not making the most of the kinematics of the robot
As the learning problem did not reason over time, it could not learn to make ‘preparatory’ moves based on uncertainty. In other words, the system was always acting ‘greedily’ and not taking into account that predictions changed over time from the EKF.
The approach was highly sensitive to the accuracy of the ball predictions. If early on in the fight of the ball, the prediction was bad, then the robot would start to initiate a motion that was very different to the one required later on. In some cases this could lead to un-recoverable motions and failed returns.
The Switch to Reinforcement Learning
After these initial results it felt natural to ask the question; can we use RL to (1) learn how to deal with the changing observations over time and the uncertainties associated with them and (2) remove the hard-coded constraints such as an explicit hitting plane? RL represents a very natural framework for solving these problems because it is designed to reason about future outcomes and make decisions that may forgo immediate reward for higher reward in the future. This can naturally occur if we no longer treat each shot/episode as a bandit problem (with a single action) but instead as a full Markov Decision Process such that the policy produces sequential actions during the flight of the ball. I was therefore tasked with increasing the complexity of the learning problem to allow for sequential decision making and to reduce the reliance on explicit sub-components such as EKFs and supervised critic models (Figure 2).
Figure 2 - On one end of the spectrum we have the previously described bandit problem (left), where we assume we have accurate predictions of the flight of the ball and can therefore act on the final prediction in a greedy manner. Actions are always relative to the terminal state (i.e. hitting-state) and we can perform re-optimization of this action during inference based on new ball predictions. Stars represent actions and circles represent re-optimizations. On the other end (right) we have a Markov Decision Process where each action is chosen based on the current state of the ball and we learn a policy that reasons over uncertainty about future states. This is still updated at a high frequency but in a closed-loop fashion.
The naive approach to RL control would be to allow the policy to output either joint or racket states every N milliseconds until the robot hits the ball and use a motion planner to move between those states. However, when we tried this for the first time on our robot we ran into several key problems:
The learnt motions tended to be very un-natural (aesthetically un-appealing) and jerky from a kinematics point of view
A lot of the time the policy would request joint/racket states that were not feasible given the current state of the robot. This led to inefficient exploration and non-linear tracking of the policy output, making the learning problem more difficult.
The size of the state-action space was so vast that the exploration problem was extremely difficult. The probability of hitting a ball, let alone returning it, was extremely low in the initial phase of learning. This meant that the learnt solution was highly seed-dependent and this made benchmarking and development difficult.
To address these issues we focused our attention on a paper called ‘LASER: Learning a Latent Action Space for Efficient Reinforcement Learning’ by Allshire et al. (2021) (Figure 3). The premise of this paperis to learn a latent space that encodes different motion trajectories, which an RL agent can then learn to combine to solve a downstream task. Importantly, this is done in a state-dependent manner such that from any given robot state, the latent space encodes all feasible trajectories given the initial training data. Another way to view this is that we first learn an action manifold from prior data and then the RL policy learns to navigate this manifold to achieve some downstream task based on incoming observations of the environment. This solves many of the above problems because it allows for (1) the recombination of desirable trajectories that are natural and less jerky, (2) the encoding of mostly feasible trajectories given the current robot state and (3) the simplification of the exploration problem by focusing on trajectories from the initial dataset.
Figure 3 - Figure from ‘LASER: Learning a Latent Action Space for Efficient Reinforcement Learning’ by Allshire et al. (2021). The top row of images shows exploration of a robot arm trying to open a door using a learnt latent space as the action space. These explorative motions stay on the learnt manifold represented by the latent space. The bottom row of images shows the same but with a naive (e.g. velocity-based) control space that does not follow the learnt latent space. The use of a latent space focuses explorative motions on trajectories that are useful and efficient for the task at hand.
From a practical point of view, we used the approach as follows:
Generate Primitive Library: We used a formulation of Model Predictive Control (MPC) to generate a library of smooth low-jerk robot motions that we deemed useful for competitive table tennis. This was done by generating different volumes of interest and then sampling different racket states within those volumes (Video 3). This allowed us to cover different types of shot such as forehands, backhands and blocks, as well as different velocities and spin types. A short video of this primitive generation process can be found below:
Video 3 - Generating motion primitives in simulation in order to form a dataset for latent space training. For each volume (in green) a range of racket positions, velocities and rotations were sampled as end-point targets. Smooth motions were then generated to satisfy these end points using a form of Model Predictive Control (MPC). End-points were designed to cover classic table tennis shots such topspin, backspin and block shots.
Learn Latent Action Space: With the offline motions generated, we learnt a latent encoding of the different motions in a state-dependent manner. To achieve this we first chunked the data into 60ms trajectory segments, which would later define the frequency of the RL problem. We then used a Variational Autoencoder (VAE) architecture to compress these segments into a low-dimensional latent space. Crucially the current robot state was given as an additional input to the decoder such that the latent space only encoded feasible segment trajectories based on the current robot state. More details of the exact formulation we used can be seen in Figure 4. One crucial ingredient for ensuring the learnt latent space was an effective action space for the RL agent was autoregressive training, which was scheduled over time. This meant sampling a sequence of 5 trajectory segments and using the output of each decoding as input to the next prediction. Gradients could then be accumulated across predictions and backpropagated through time. This ensured that the latent space remained temporally consistent and was able to remain on the data manifold despite its own prediction errors.
Figure 4 - Variational Autoencoder (VAE) architecture used to learn a latent space from the generated primitive library. Each trajectory from the primitive library is segmented into 60ms segments. The robot starting state (st) and end state (st+1) of a sampled segment is given as input to the encoder. The encoder then produces N gaussian distributions, which are sampled from to generate the latent embedding z. The decoder takes the latent embedding and the robot starting state (st), and predicts the joint velocities for the whole segment (vt). There are four key losses present during training: (1) KL loss to bound the latent space and encourage disentanglement, (2) Constraint loss to encourage sequences of joint velocities to respect the derivative limits of the robot joints, (3) Velocity profile loss to encourage the predicted joint velocities to match those of the original segment, (4) Next state loss to encourage the result of applying the predicted sequence of joint velocities to match the end state of the original segment. We chose not to provide the original joint velocities of the segment as input to the encoder because this did not allow for auto-regressive training, which requires joint velocity trajectories not in the original dataset. Furthermore it was crucial that the decoder was state-dependent such that z only encoded velocity trajectories based on the current state of the robot.
Train RL Algorithm: Once the latent space had been learned, the decoder was frozen and became part of the environment, converting actions from the RL algorithm (Soft Actor-Critic - SAC) to state-dependent joint velocity trajectories (Figure 5). These velocities were then transformed to a position trajectory and a feasibility filter based on quadratic programming was used to track the resulting trajectory on the robot. The RL policy observed the robot’s racket state, the robot’s joint state, and the ball state relative to the robot and racket. The reward function was simply to return the ball to the other side of the table with the highest velocity possible. To aid in the initial exploration of the policy, we used N episodes at the start of training to sample from the original primitives and simulated ball trajectories that passed through the end point of the primitive. We could then get the corresponding trajectory of latent values from the encoder and add it to the replay buffer for training (along with any achieved rewards).
Figure 5 - The learnt decoder from the VAE is frozen as used as part of the environment to translate the actions from an RL algorithm (Soft Actor-Critic) to sequences of joint velocity commands. The joint velocity commands (vt) are converted to position commands (pt) and then run through a feasibility filter before being sent to the robot. This procedure is run in a closed-loop such that the state of the robot after executing the velocity commands is given as input to the RL policy and the decoder on the next time-step. The RL policy also observes the ball state relative to both the racket and the robot base.
Deploy on Robot: To be able to utilize the policy learnt in the previous step we had to implement several practical features so that we could deploy it successfully on the robot in the real world. Firstly, we needed to ensure that any trajectory we sent to the robot would not collide with itself or the table. To achieve this we always calculated a full reset trajectory from the robot state at the end of the decoded trajectory to a safe initial position. The decoded trajectory was then only sent to the robot if both itself and the resulting reset trajectory were collision-free. If one of these were not collision-free then the reset trajectory from the previous time-step was sent to the robot. This ensured that the commands being sent to the robot were always collision-free. Secondly, the computational budget for running the policy, decoder, feasibility filter, reset planner and collision checking, was fixed to 5ms. As the robot control frequency was higher than 200Hz this meant that we could not run our approach in the main process used to control the robot. Instead, our approach ran in a separate process and the policy received its observations 5ms before its actions were retrieved and executed. Crucially, both the reset behaviour and staggered observations were present during the training of the policy, which encouraged it to learn to avoid collisions and deal with old observations.
Overall, this formulation allowed us to focus the RL on the exact problem we wished to solve; how to act under uncertainty to chain desirable motions together that account for future rewards (Video 4). From a personal perspective, this was one of the most enjoyable times on the project. Working in a small team of 3 with very talented colleagues on a completely different approach to the problem was incredibly exciting and rewarding. There were lots of chances to read papers, discuss novel improvements and quickly see the results on the robot. It also moved us from co-operative rallies to finally being able to beat the majority of the people in the office in a competitive setting.
Video 4 - Example of the LASER approach on the real robot in a rally scenario. The Reinforcement Learning policy has learnt to recombine trajectory segments from the original motion primitives in order to return the ball.
Despite being able to use LASER to demonstrate the effectiveness of RL for the problem of robotic table tennis, we were still a long way from playing at a professional level. In particular, the use of a pre-trained latent space raised several key issues:
Deciding which motions to include in the dataset for the latent space was difficult and relied upon our own intuition. Naturally, this meant that some motions were probably not useful while other important motions were not present in the dataset.
Reactivity appeared to be limited. This was likely because the dataset consisted of whole smooth low-jerk trajectories. Therefore the ability to suddenly switch between motions with a high-jerk response was unlikely to be encoded in the state-dependent latent space.
High-fidelity control also seemed limited. By compressing motions into a latent space (that is heavily regularized to be close to a unit gaussian) we naturally lose fine-grained information about the low-level motion. This is critical in table tennis where motions need to be extremely precise in order to return the ball at a competitive level.
Interestingly, very similar work has recently been published but in the domain of tennis using humanoid robots (Zhang et al. 2026). In this work, they follow a very similar process with a few smart improvements. Most notably, they learn a state-dependent prior over latent values for the final latent space. The RL actions are then used as a scalar multiplier for the prior standard-deviation. This is useful because the latent space is only regularized to a unit-gaussian but by definition it cannot satisfy this because it needs to encode information about the different possible motions given the current robot state. By learning the prior and expressing the action as a function of that prior, the RL policy can more faithfully follow the learnt manifold and avoid sampling out-of-distribution latent values. In addition, the authors allow the RL policy to learn corrective joint commands for the wrist joint of the robot directly. This is important because it allows for high-fidelity control of the racket when hitting the ball, while still conferring many of the advantages of using a latent space for the other joints.
A Bitter Lesson - Reducing Human Bias
It was at this point in the project where for the first time I really felt the full weight of ‘The Bitter Lesson’ by R. Sutton i.e. including our own human knowledge into autonomous systems eventually becomes the bottleneck. The latent space work outlined above used human domain-knowledge, in the form of the dataset of motions, to train the latent space. We had used our own intuition to overcome issues around exploration and smooth desirable motions. As a result we could never reach the true ceiling of performance, we needed a more general approach that learnt from the ground up with minimal intervention or guidance. Only then could we hope to perform highly reactive, precise control that utilized the full capabilities of the robot.
It was around this time that my colleagues with expertise in Optimal Control proposed a novel method for providing our RL algorithm with an action space that was bijective and explicitly encoded state-dependent feasibility. In short, the approach worked as follows:
The RL policy output a 2D action between -1 and 1 for each joint of the robot, which is equivalent to the standard tanh transformation used in SAC.
For each joint, the 2D action was mapped to a feasible position and velocity waypoint 31.25ms into the future based on the current state of the robot. This used a novel mapping method that ensured the mapping from 2D abstract action to waypoint was feasible, bijective and computationally efficient. A waypoint was defined as feasible if the desired position and velocity was achievable based on the current state of the robot without violating position, velocity, acceleration and jerk limits.
With the waypoint selected, a trajectory from the current robot state to the desired waypoint was generated using N cubic splines. The trajectory satisfied the desired waypoint exactly and was optimized to minimize jerk along the whole trajectory for smooth motion.
Crucially, this new approach gave us a way to allow the RL algorithm to learn how to control the robot with no human-bias, while still ensuring smooth feasible motions and a simplified exploration space due to the state-dependent sampling i.e. no exploration was wasted on infeasible motions. Around the same time, we also received our new custom-designed robot from the hardware team. This robot had been explicitly designed for table tennis motions and also included an XY-stage that allowed us to cover the whole table tennis court. This switch naturally had an impact on the exploration of our RL problem and likely allowed for more natural motions under random exploration.
Initial comparisons in simulation on our new robot hardware demonstrated that this new approach, which we termed Feasible Action for Optimal Control (FAOC), led to higher asymptotic rewards compared to LASER therefore confirming the bitter lesson. Another useful benefit of the switch to FAOC was that we did not need to generate data and pre-train a latent space before learning the RL policy. This meant we could iterate much quicker and also focus on improvements on the RL side, which were general and scalable.
Once published you will be able to find a full academic paper on FAOC at the following link: <TBD>.
Figure 6 - Graphical representation of the mapping performed by FAOC. Left: The action space is represented as a 2D hyper-cube for each joint, which is achieved by applying the tanh function to the output of the policy. Right: FAOC then maps this to a 2D polytope, where one dimension is the desired joint position (q*T) and the other is the desired joint velocity (q.*T). This mapping is bijective and ensures that every point in the 2D polytope represents a feasible desired joint state based on the current state of the robot.
Scaling Up and the Last 20% Rule
Armed with a more general solution for RL-based control and the final hardware we would use to compete against professional table tennis players, we switched our focus to scaling up our approach and optimizing the performance. The nature of the work from this point (approximately the final 1-2 years) represented quite a significant change to me. No longer were we exploring many different directions in parallel and working on research prototypes to demonstrate effectiveness. Instead, the entire team focused on a single direction and tried to improve every component of the system. As a result, we were able to go from lower amateur to professional level in a very short space of time. I believe this was only possible because we worked in a unified direction and because we had a general RL framework that allowed us to scale and improve in an unbounded way. This period on the project reminds of the common phrase “the final 20% of progress requires 80% of the work”. It really required the sustained effort of everyone in the team to make sure every part of the system was optimized.
From an RL for control perspective, the scaling up involved many different streams of work. Below I will summarize some of the most impactful streams:
Cloud-based training - We switched from training on local machines to cloud-based training. This meant we could run many more instances of the environment in parallel to aid the exploration problem. Equally, it meant we could run more experiments at once, increasing development speed and democratizing training for other members in the team.
Universal Value Function Approximators (Schaul et al. 2015) - We expanded our policies to be goal-conditioned such that they could demonstrate a wide variety of behaviours within a single policy. We provided explicit XY position goals on the opponents side of the table and rewarded the policy based on the distance of the return to that goal. Furthermore we provided a spin goal (-1 - backspin, 0 - free spin, 1 - topspin) whereby the reward was a function of the spin goal multiplied by the amount of y-spin the policy returned the ball with. With these changes our policies went from just trying to return with maximum velocity to demonstrating a range of skills and behaviours commonly seen in table tennis matches. While relatively simple, when randomly sampled these skills were enough to keep our human opponents guessing on every shot. We also found that by encouraging the policy to learn skills such as topspin, it served as an exploration bonus and increased the velocity of the robots' return in general.
Replay Buffer Augmentation - For training we used the off-policy algorithm Soft-Actor Critic (Haarnoja et al. 2018). One of the primary benefits of this approach was that we could employ different forms of replay buffer augmentation to stabilize and improve learning. Firstly, as we were using goal-conditioning, we used Hindsight Experience Replay (HER - Andrychowicz et al. 2017) to re-label goals during training. New transitions were added to the replay buffer with the desired goal re-labelled as the achieved goal and the reward recomputed accordingly. Secondly, as both our robot and the game of table tennis are symmetrical along the XZ-plane, we added additional transitions to the replay buffer with all observations and actions mirrored in the XZ-plane. This helped to speed-up learning and reduce bias in the policies where performance on one side of the court was better than on the other. Finally, we used event tables (Kompella et al. 2022) to stratify the sampling of minibatches during gradient updates. This involves manually defining events that each have a corresponding replay buffer. When an event is experienced the N transitions leading up to that event are added to their corresponding replay buffer. Stratified sampling is then used during minibatch construction so that gradients corresponding to the different events are consistently used. This helps to reduce variance across seeds and ensure the policy does not forget rare (under the current policy) but important events e.g. collisions late on in training.
SimbaV2 (Lee et al. 2025) - A recent stream of work has focused on scaling up RL to larger and larger networks so that it can achieve the kinds of performance gains observed in supervised learning. To explore whether larger networks could improve performance for our problem we implemented several components from SimbaV2. As proposed by SimbaV2, we switched to a distributional critic using the same formulation as C51 (Bellemare et al. 2017). We also incorporated the hyperspherical normalization across both features and weights. These changes allowed us to scale up the sizes of our policy and critic networks by ~2x and ~8x respectively, resulting in a large gain in performance.
Aside from these changes to the fundamental components of our RL training, we also put in substantial effort to improve the transfer of our policies from simulation to the real world (commonly referred to as sim2real). Indeed, through this work we were able to use policies trained purely in simulation for competitive games in the real world against human opponents. This work can generally be summarized as follows:
Physics Modelling - An entire sub-team on the project was dedicated to improving the key physics models in our training simulation (aerodynamics, ball-table contact and ball-racket contact). The modelling of these physics interactions required the use of analytical models as well as learnt residual models. This part of the project was very incremental in nature and is potentially one of the most limiting factors for applying our work to other domains. If years of experience and work is required to precisely model every domain we want to train robots for, then it is unlikely that the autonomous robotic revolution will truly take off. In the case of table tennis, we had to continuously collect data from the robot as it improved such that we could model the physics behind increasingly competitive play. This kind of ‘on-policy’ physics learning would lend itself well to model-based and real-world learning, however see below for the practical limitations of these approaches.
Privileged Actor-Critic - As mentioned previously, one of the main motivations for switching to RL-based control was its ability to reason about uncertainty in the state observation and pick actions based on future outcomes. Initially on the project we tried to model the uncertainty of the ball observation explicitly (e.g. with an Extended Klaman Filter) and then pass this to the policy. However, we ultimately found that it was best to let the policy learn about uncertainty directly rather than receive an intermediate approximation, which is susceptible to increasing the sim2real gap. We therefore trained the policy in simulation with a history of raw, noisy sensor measurements as input and the critic with the ground-truth ball state (Eschmann et al. 2024, Geles et al. 2024, Vasco et al. 2024). In this way, the policy is forced to learn how to select actions under uncertainty while the critic can provide accurate gradients for policy learning. The critic is not required in the real-world so we do not need to estimate the ground-truth state of the ball at inference time. To help the policy implicitly learn the ground-truth state of the ball, we trained it to reconstruct the ground-truth state during training as an auxiliary loss.
Sensor Modelling - As the policy received a history of raw sensor measurements as input, the problem of good sim2real transfer was shifted to accurately modelling the sensor properties of the real-world in simulation. We found this to be a substantially more tractable problem than trying to learn a separate explicit state estimator and then model that in simulation based on its properties in the real-world. In essence we took the ground-truth ball state values from the simulator and applied the following transformations to mimic the real-world sensors; latency, dropout, precision (noise), accuracy (bias). By matching these transformations to real-world data we could substantially improve the transfer of policies from simulation to the real-world. Similarly it gave us an easy and effective pipeline for translating improvements from real-world perception to new control policies by adjusting the transformations based on new data.
Initial State Distributions - Finally, a key part of our problem was ensuring that the policy was trained on the correct data distribution for professional-level table tennis, especially considering the opponents would be actively trying to find weaknesses of Ace. If we only consider individual shots and assume accurate physics models, then this means finding the optimal set of initial ball states. To achieve this we resorted to a data-driven approach very similar to Abeyruwan et al. 2023, whereby we iteratively collected data from robot-human games and added the shots to our training data. The collected datasets were modelled using Kernel Density Estimation (KDE) such that we could sample from them during training (and check if the resulting sample was legal or not). In the end, we had a set of different KDE models representing different levels of competitiveness which we then sampled from using fixed mixing ratios.
Through a combination of these fundamental RL and sim-to-real improvements we were ultimately able to achieve our goal of beating professional table tennis players (Video 5). I think over time we all become quite desensitized to what we would see every week; an autonomous system competing in high-level sport with a single neural network policy at the heart of it, producing control commands every 32ms. I guess this is often the case with new technologies, it is not long before we become accustomed to it and adjust our expectations of the world. However, every now and then I do still marvel that the same fundamental concept of RL, which I first learnt about many years ago in the context of a mouse finding a piece of cheese in a maze, is now also capable of professional-level sport in the real world.
Video 5 - Example game of Ace against professionally ranked F. Igarashi (March 2026). This video highlights the substantial gains in performance made by optimizing our final approach and minimizing the sim2real gap.
What Didn’t Work?
While on the surface the technical story of our work can look deliberate and logical, there were many streams of work that did not feature in the final system. I believe it is important to discuss why this was the case and what prevented these pieces of work from providing the benefits we hoped they would. In particular, there were four areas I was involved in that did not deliver on their promise: hierarchical RL for tactics, real-world fine-tuning, curriculum learning and model-based RL.
Hierarchical RL - Previously I mentioned that we randomly sampled goals (in the form of aiming positions and spin direction) to give as input to the policy in order to add variability to the robots playstyle. A natural extension to this approach is to pursue a hierarchical system that learns to propose a goal for each shot in order to maximize the probability of winning the point. Naturally this can again be framed as an RL problem whereby the policy observes the incoming shot and produces a goal for the lower level ‘skill’ policy to execute. While this framing appears straight-forward the crux of the problem is that we require a reward signal to train this top level ‘tactics’ policy. The obvious choice for this reward signal is whether the robot won the point or not. However, to have access to such a signal we need an opponent that the robot can play in simulation. We explored many different forms of simulated opponent, from models that use behaviour cloning on human data to the robot playing against itself. In all cases, we could demonstrate that a tactics policy providing goals to a skills policy could beat a skills policy that just used random sampling. Despite this positive result, we could never prove that it held for playing against humans in the real-world. In fact, human opponents often rated the random sampling as more competitive due to its unpredictability. This really hits at the heart of the overall problem; humans are constantly adapting and changing during play and we found it extremely difficult to model this non-stationarity in simulation. Training signal aside, there was also difficulty in deciding on what the goal interface should look like between the tactics and skills policies. A wider variety of goals affords the tactics policy more options for winning the point but also makes learning good tactical decisions more critical as not all skills will be competitive in all scenarios e.g. a drop-shot only makes sense when the opponent is far from the table. Conversely, a small choice of goals that are all highly competitive means the tactics policy is less necessary and has a weaker learning signal to follow. Ultimately, we found it best to just have a handful of highly competitive skills (high velocity or spin) and simply use random sampling to make the play less predictable.
Real-World Fine-Tuning - Towards the end of the project I spent a lot of time working on ways to fine-tune policies in the real-world. For much of the project I was extremely skeptical that we could beat human-professionals in a zero-shot manner i.e. only training in simulation. To this day, the return rate of our policies is lower in the real-world than in simulation suggesting there still exists a difference between the two scenarios. I therefore wanted to explore whether we could continue training in the real-world to recover this lost performance and account for the phenomena that we are unable to model in simulation. To test different algorithmic ideas we explored sim2sim scenarios where the policy had to continue learning in simulation but under a change in physics parameters. With this approach we identified several algorithmic ideas for efficient fine-tuning including: DroQ (Hiraoka et al, 2021), mixed replay buffers (Julian et al., 2020, Kumar et al., 2022), progressive networks (Rusu et al., 2016) and warm-starts (Zhou et al., 2025). Having found promising algorithmic approaches, we created a real-world setup that would autonomously fire balls from a ball cannon and perform intermittent learning on the data collected (Video 6). Initial results, using the LASER approach outlined above, were promising. We could show that learning in both sim2sim and the real-world was relatively data efficient, with noticeable improvement after just a few hundred shots (Figure 7). However, it was around this time that we switched to FAOC and began to scale up our approach to achieve competitive play. This meant wide-ranging changes to components such as the physical robot, the action space formulation, the distribution of initial states during training and the horizon of the RL problem. Similarly, more work was put into improving the simulation, which meant a reduction in the sim2real gap and a weaker fine-tuning signal. After all these changes, we noticed a substantial drop in fine-tuning efficiency, with several thousand shots required to improve the policy. Unfortunately, this led to a very practical limitation, for any policy we wished to use we would need to invite professional level players beforehand to collect data for fine-tuning. These players would need to play against the robot for several days and no subsequent changes to the system could be made. If we had access to several robots and a pool of professional table tennis players then perhaps real-world fine-tuning could have been possible. In reality, we did not have the infrastructure in place to make this a priority. We did explore off-policy approaches to fine-tuning, using data collected from other policies in past games to improve a policy trained in simulation. However, the results of this were disappointing and learning was made difficult by the fact that a lot of our historical data did not reflect the state of our current system e.g. in terms of hardware or perception. Through investigating real-world fine-tuning, I definitely learnt valuable lessons about the need to prioritize the right infrastructure beforehand and also how algorithmic innovation comes second to reliable and extensive data collection.
Video 6 - The automated learning setup used to explore real-word fine-tuning approaches. The ball cannon would fire N shots, after which there would be a pause while the policy was updated with G gradient steps. If a ball was stuck on the table it would be automatically detected and fans would be triggered to blow the ball off the table. Angled fabric was used below the table to funnel all the balls under the table and back to the ball cannon.
Figure 7 - The top row shows example return locations for a policy before and after fine-tuning in the real world using the setup in the video. The bottom row shows the return rate over the course of fine-tuning for approximately 1000 shots. The return rate shows a noticeable improvement after just 100 shots in the real world.
Curriculum Learning - During the scaling up part of the project, I explored many different curriculum learning approaches for increasing the complexity of the ball shot distribution over the course of learning. In theory, we hoped this would bootstrap learning and also allow the agent to learn how to return shots that weren’t in the initial ball cannon distributions we had extracted from real-world gameplay (i.e. be more robust to adversarial attacks). Most notably we tried particle-based (Lee et al. 2020) and self-play (Silver et al. 2018) approaches for updating the initial ball state distribution over time. When it came to evaluating these policies in the real world however, we saw no observable benefit of these approaches against highly skilled human opponents. This was a surprising finding but several theories for why this might be the case include: (1) the initial ball state distributions from real-world play already cover the distribution required for competitive gameplay, (2) the mix of beginner and professional shots in our initial distributions is already enough to bootstrap learning and (3) our use of an off-policy RL algorithm (SAC) and event-tables means that the current data distribution has a marginal effect on learning dynamics.
Model-Based RL - The final stream of work that we explored was model-based RL and in particular DreamerV3 (Hafner et al. 2023) and TD-MPC (Hansen et al. 2022). In both cases, we found the results of such approaches were disappointing compared to straight-forward model-free RL. In general, I believe this was due to several reasons: (1) The learnt latent models naturally had errors that would compound when simulating over multiple time steps. For our problem of robotic table tennis extremely precise models are required to accurately return the ball to the table. For example, small errors around action selection at the point of contact cannot be accounted for with subsequent actions, unlike other problems such as locomotion. (2) The increased computational costs for training the models was a practical limitation as we had access to many low-power CPU machines as opposed to single powerful GPU machines. This naturally lended itself to learning in a model-free way with many simultaneous simulation instances. (3) We could not make use of the learnt model at inference time in the real world. Due to computation time limitations, we could only afford a single pass of the policy network and could not take advantage of sampling the learnt model for multiple time steps to select an action. As a side note, coming from a neuroscience and cognitive science background, I find it interesting that model-free RL seemed better suited to this problem. The finding appears to mimic how humans become professional players. While general knowledge of physics and planning can help beginner players to play table tennis, to become a world champion requires many many years of repeated training in a fixed environment. This requirement of humans to learn in an incremental, trial and error-based manner to perfect motor control has many similarities to model-free RL. Whether future robotic systems can achieve levels of learnt motor accuracy beyond that of human professionals without the need for such learning is an open question.
Thoughts in Hindsight / What Surprised Me
There were several insights from working on this project that surprised me and challenged my preconceptions. Firstly, when I first started learning about RL in robotics the notion of a sim2real gap was often quoted as the greatest limitation. Many people were working on various algorithms and approaches to address it from meta-learning in a variety of simulation environments to fine-tuning polices in the real-world. It has therefore genuinely shocked me that we were able to achieve the results we did with zero-shot transfer i.e. only training in a single simulation environment. Indeed, most of the data we collected in the real-world went towards improving the accuracy of our physics models, sensor models and initial state distributions in simulation. This simple approach allowed us to quantify the accuracy of different parts of the system and de-couple policy training from improving transfer to the real world.
With this being said, one obvious omission from our simulation training environment was an accurate model of a professional human opponent. With such a model we could have access to the true reward signal (whether the point was won or not) as opposed to the surrogate one we used (rewarding high velocity and spin). Efforts were made in this direction but it proved a difficult technical challenge to truly model human behaviour and how it changes over time. In general this raises a fundamental question; as robots move into our everyday lives and interact with us, how can we train safe and reliable policies in simulation that accurately model these interactions? At some point there are phenomena that we cannot model in simulation and it’s an open question how we will tackle such problems. Perhaps training under approximations of human behaviour in simulation will be enough for robots to live among us and then collect data on real-world human interactions to continuously improve.
On this subject, we probably could have pushed harder on self-play to investigate whether it could achieve super-human performance without the need for a model of a human opponent. However the morphology of the robot is very different to that of a human and so the learnt strategies and training distribution of robot-robot play may be significantly different to robot-human play. This also raises another obvious but important point that I had not truly considered; the development of AI is currently heavily biased to companies with vast computational resources. This might seem obvious now but during the initial years of this project the true scaling of AI had not been uncovered and I did not appreciate how much of an impact this has. On this project, we would have loved to explore self-play leagues (e.g. Vinyals et al. 2019) but we simply did not have the computational resources to train many different policies and continuously have them compete against each other for days on end. At most we had the resources to explore self-play of a single policy playing against previous versions of itself.
While we could use a surrogate reward for training in simulation to great effect, it did not help us when it came to evaluating our policies in the real-world. Indeed rigorous evaluation of policies was one of the biggest obstacles on the whole project. As Ace improved in ability we needed better and better players to both assess how competitive it was and quantify the severity of the sim2real gap. Towards the end of the project it got to the point where we could no longer assess our policies against players of the same level as those we were going to play in official tournaments. Furthermore, even when we could arrange for professional players to test our system we could generally only play a handful of matches against them due to time limitations. This created one of our biggest dilemmas; when developing a feature, how can we know whether it produces an observable improvement or not when the amount of real-world data we collect is too low to provide any statistical significance. This was a real problem on the project and led to many issues within the team where it felt like feature selection for the final tournaments was more based on intuition and human bias than quantitative results. One piece of work that helped us to alleviate this problem was a ‘hybrid’ benchmark that we could use to pick between features/policies. This hybrid benchmark took shots from real-world matches and replayed them in simulation (at the sensor level) so we could assess how well new policies would perform against the same distribution of shots. Importantly, as soon as the robot would hit the ball in simulation we would switch to our simulated physics to get the result of the shot. This hybrid benchmark really helped us to make more informed decisions because it allowed us to quantify how well policies performed under the same competitive shot distribution while also using real-world sensor characteristics. We could also easily extend the benchmark over time with any new data that we collected. With this being said, it was still not a full substitute for real-world games because it was still on a single-shot basis and did not encompass the whole sim2real gap.
On a different note, one area I wish we had explored was a bi-directional relationship between the team I was on (AI for control) and the hardware team responsible for designing our custom robot. Due to scheduling and planning requirements, we worked on an industrial robot for the first few years of the project while the hardware team designed and built our own custom robot. Then around half-way through the project we were given the new custom robot and we had to apply all our approaches to this new morphology. This quickly made me appreciate how much of an impact the robot morphology has on the learning problem. In hindsight, it would have been great to have had some back and forth such that we explored many different robot designs. Using our general learning infrastructure we could have searched over different morphologies to find the one most suited to playing professional-level table tennis. I believe for those who are working on the full stack (from robot hardware to AI-control) this will be a key avenue of work for realising the true potential of autonomous robotics.
A final insight that really struck me on the project is one related to our own human biases. This project attracted some of the most intelligent people I have had the privilege to work with. However, I noticed as intellectuals we tend to favour complex, over-engineered solutions that serve as a demonstration of our intellect. This may manifest itself as a bias towards the latest and fanciest algorithm, or a bias towards imbuing the solution with our own domain knowledge. However at almost all points in the project this led to worse results; the best results always came from simple, scalable solutions that avoided as much domain engineering as possible (i.e. the bitter lesson). Nowhere was this felt more than in the task of reward engineering. As mentioned before we did not have access to the true reward in training (win/lose) and so we had to manually design surrogate rewards that led to competitive and varied shot types. Combine this with the fact that defining rewards is a natural way of incorporating domain knowledge, launching training runs is easy, and generating results appeals to our own human dopamine-driven reward loop, and you have a recipe for a substantial rabbit-hole. I believe we lost a lot of time and resources on this topic as we explored many complex reward formulations that probably distracted us from fundamental solutions to reward modelling that could have had a larger impact.
Relation to Robotics Research and Current AI Trends
After spending nearly 6 years on this project it has been amazing to see how the whole AI landscape has shifted while I focused on a singular problem. Large Language Models (LLMs), generative AI, humanoid robots, world models, Vision Language Action models (VLAs) and AI investment have all exploded during this time. In many ways our use of model-free RL from scratch in simulation can seem quite out-dated in light of these recent advancements, and yet we managed to beat a historical challenge in robotics and compete for the first time directly with human professionals in the real world, what does this say then about the current state of AI?
For me I think it really reflects the nature of the problem that we were working on. While many people in the field of AI and robotics are currently focused on generalist approaches, which are typically computationally expensive and involve slow deliberate planning, we focused on a single problem that required fast, accurate control at the physical limits of a non-humanoid system. While topics such as world models and VLAs can give you a general understanding of the world they are not necessarily the best tool for learning precise world-class sporting movements. As mentioned previously, such learning likely has to reflect how humans solve such problems; repetitive trial and error learning through repeated practice to perfect motions given the current state of the environment - a key strength of model-free RL. I also believe that our approach had the advantage that learning was completely tabula rasa with minimal human bias towards a specific solution. Through the combination of model-free RL and simulation, the policy could learn how best to control the robotic hardware with no prior perceptions of how the problem should be solved. This would not be possible through commonly used techniques in the robotics field, which often rely on demonstration data or are heavily biased by the offline data that they are trained on.
I believe the current state of AI actually has much bigger implications for the kind of work I do in the future. It is quite shocking to see in real time how AI is able to automate more and more of my work responsibilities. As I slowly see my work being automated, it has got me thinking about what it means to be human and how we feel a sense of value in the work we do. As someone who works on improving the capabilities of AI, is iterative AI self-improvement inevitable? If we take the bitter lesson to the extreme (i.e. our own knowledge is a hindrance to developing truly intelligent systems) then am I actually the bottleneck? How much does this line of thinking apply to other fields I wonder, are knowledge-based workers going to be a thing of the past? Alas this is probably a discussion for another day, but the trope of AI researchers fuelling their own redundancy has never felt so real. I would hope that the next wave of AI and also robotics focuses on problems not of a human nature, where a fundamental solution does not currently exist and where a solution would represent a positive societal impact (as opposed to simply an efficiency gain). Now more than ever it seems there is a need for connecting with others, understanding their problems and using our empathy to ensure technology enhances all our lives.
Closing Remarks
I am deeply grateful for my time working on Project Ace at Sony AI. It gave me many things including an introduction to robotics, a life in Switzerland and great colleagues who challenged my preconceptions and made me a better researcher and human being. We achieved something I genuinely did not think was possible and made some great memories in the process. I have learnt how powerful a collective vision can be and that many diverse opinions are required to find the best solution. I am excited to see how I can apply my learnings to future endeavours and make a genuine positive societal impact.
If you wish to cite this blog post then you can use the following:
Blakeman, S. (2026, June 15). Project Ace. Tech & Robotics Blog. https://www.srablakeman.com/post/project-ace
@misc{blakeman2026pjace,
author={Blakeman, Sam},
title={Project Ace},
howpublished={Tech \& Blog},
year={2026},
month={June},
url={https://www.srablakeman.com/post/project-ace},
}Disclaimer: The views and opinions expressed on this blog are solely my own and do not reflect the official policy or position of Sony or any other organization, committee, or individual I am associated with.
References
Tebbe et al. 2021 entitled ‘Sample-Efficient Reinforcement Learning in Robotic Table Tennis ‘LASER: Learning a Latent Action Space for Efficient Reinforcement Learning’ by Allshire et al. (2021)The Bitter Lesson - Sutton, Richard (March 13, 2019) http://www.incompleteideas.net/IncIdeas/BitterLesson.html.Zhang, Z., Lu, H., Lian, Y., Chen, Z., Liu, Y., Lin, C., ... & Yi, L. (2026). Learning athletic humanoid tennis skills from imperfect human motion data. arXiv preprint arXiv:2603.12686.Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015, June). Universal value function approximators. In International conference on machine learning (pp. 1312-1320). PMLR.Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., ... & Levine, S. (2018). Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905.Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., ... & Zaremba, W. (2017). Hindsight experience replay. Advances in neural information processing systems, 30.Kompella, V., Walsh, T. J., Barrett, S., Wurman, P., & Stone, P. (2022). Event tables for efficient experience replay. arXiv preprint arXiv:2211.00576.Lee, H., Lee, Y., Seno, T., Kim, D., Stone, P., & Choo, J. (2025). Hyperspherical normalization for scalable deep reinforcement learning. arXiv preprint arXiv:2502.15280.Bellemare, M. G., Dabney, W., & Munos, R. (2017, July). A distributional perspective on reinforcement learning. In International conference on machine learning (pp. 449-458). Pmlr.Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2020). Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), eabc5986.Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., ... & Hassabis, D. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 1140-1144.Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023). Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104.Hansen, N., Wang, X., & Su, H. (2022). Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04955.Eschmann, J., Albani, D. & Loianno, G. Learning to fly in seconds. IEEE Robot. Autom. Lett. 9, 6336–6343 (2024)Geles, I., Bauersfeld, L., Romero, A., Xing, J. & Scaramuzza, D. Demonstrating agile flight from pixels without state estimation. In 2024 Proceedings of Robotics: Science and Systems (RSS) (2024).Vasco, M. et al. A super-human vision-based reinforcement learning agent for autonomous racing in Gran Turismo. In 2024 Reinforcement Learning Conference (RLC) (2024)Abeyruwan, S.W., Graesser, L., D’Ambrosio, D.B., Singh, A., Shankar, A., Bewley, A., Jain, D., Choromanski, K.M. & Sanketi, P.R.. (2023). i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loops. <i>Proceedings of The 6th Conference on Robot Learning</i>, in <i>Proceedings of Machine Learning Research</i> 205:212-224 Available from https://proceedings.mlr.press/v205/abeyruwan23a.html.Hiraoka, T., Imagawa, T., Hashimoto, T., Onishi, T., & Tsuruoka, Y. (2021). Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint arXiv:2110.02034.Zhou, Z., Peng, A., Li, Q., Levine, S., & Kumar, A. (2025, May). Efficient online reinforcement learning fine-tuning need not retain offline data. In International Conference on Learning Representations (Vol. 2025, pp. 32343-32368).Kumar, A., Singh, A., Ebert, F., Nakamoto, M., Yang, Y., Finn, C., & Levine, S. (2022). Pre-training for robots: Offline rl enables learning new tasks from a handful of trials. arXiv preprint arXiv:2210.05178Julian, R., Swanson, B., Sukhatme, G. S., Levine, S., Finn, C., & Hausman, K. (2020). Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. arXiv preprint arXiv:2004.10190.Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., ... & Hadsell, R. (2016). Progressive neural networks. arXiv preprint arXiv:1606.04671.Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., ... & Silver, D. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. nature, 575(7782), 350-354.
Comments