Design for two interacting, extensible classes

Question

Hi,
I have a design problem where I have two interacting classes that both need to be highly extensible. I'm pretty sure this not an uncommon problem but I can't really seem to find an appropriate pattern to implement it.
The problem comes from the field of reinforcement learning, where an Agent learns some task in an Environment by executing Actions based on the current State of the environment. The State has to be provided by the Environment, and the Action chosen by the Agent based on that State has to be executed on the Environment. I thus have an Environment interface (for multi-agent problems) like so:
public interface Environment {
     //execute given action for given agent (e.g., move the agent, let the agent pick up smth
        //in the env, etc), and return a reward
     public double executeAction( Action act, Agent a );
     //compute and return state of given agent (e.g., what the agent can currently see in the
       //environment)
     public State getState( Agent a );
}There can be many different types of environments, and many different types of actions. So I thought of having a callback method on the Action that the Environment calls in method executeAction, like:
public interface Action {
     //execute on the given environment for the given agent
     public void execute(Environment env, Agent a);
}In that case you could have the same environment (e.g., a world with blocks), and different actions (say move, push a block away, pick up a block, etc). Does this make any sense? It's a little like the Command pattern perhaps?
There are two things I'm worried about. One is that the Action would need to cast the received Environment to the right implementation, but I guess that's unavoidable. The other (actually two) is that the Environment implementation would need to expose the right methods for the action to be able to execute, plus I don't always want an Action to be able to do anything it wants with the environment (e.g. teleport the agent, remove all obstacles). On the other hand maybe I shouldn't worry about the latter since it's just meant to be a toolkit, so if people using it want to program actions to teleport the agent, that's their business.
Any suggestions/comments greatly appreciated.
Thanks
Matt

Answer 1

I am not a native english speaker, and I'm not sure I understand your description. Is the following pseudo-code a correct description of your design?No problem, me neither. Maybe that's why it was hard to understand :) Actually, an agent should by no means have direct access to the environment: it is only allowed to perceive things through the State object it receives. So the design is something like:
public interface Agent {
        //do a step: choose an action based on the current state, and learn from
        //the reward received after the last action.
     public Action doStep( State s, double reward );
public class Simulator {
        //do one simulation step
     public void doStep() {
          State s;
          Action act;
          for ( Agent a : agents ) {
                        //get the agent's state from the environment
               s = env.getState(a);
                        //let the agent choose an action based on that state (lastReward is stored in a Map, omitted)
               act = a.doStep(s, lastReward);
                        //execute action on the environment and store the reward for this agent
               lastReward = env.executeAction(act, a));
There are two things I'm worried about. One is that the Action would need to cast the received Environment to the right implementation, but I guess that's unavoidable.Why? Cant he environment interface define abstract "modifier" methods that make sense to all implementations?Not really. An environment can be anything: a world with objects that the agent has to collect/avoid (I chose that example because I thought it might be easy to visualise and understand). A Blackjack game in which the agent has to choose to get another card or stop. The stock market in which the agent has to make trading decisions. And so on.
And iff the actions are specific to some environment implementations, then each environment can implement executeAction his way, for example calling different Action methods. In this case the Action become merely a Visitor.Yes, I indeed looked at the Visitor pattern, but I think it's not flexible enough. The types of the environments / actions need to be known beforehand, and in my case they can be anything.
The other (actually two) is that the Environment implementation would need to expose the right methods for the action to be able to execute, plus I don't always want an Action to be able to do anything it wants with the environment (e.g. teleport the agent, remove all obstacles).Method executeAction should use only those methods exposed in interface Environment (which is the type of the env argument).
If the Action and Agent implementations need some different views of the environments, just define two different interfaces.Hmm, yes. The problem is that the Environment interface should pretty much only specify those methods that I listed. Those are the methods necessary for the reinforcement learning paradigm (basically as implemented in the Simulator). Adding additional, problem-specific methods to that interface would not make sense, because environments can be anything. But, perhaps I could do something like:
public interface BlockWorld {
     public boolean move(Block b);
     public Block getBlockAt(int x, int y);
public class SomeWorld implements Environment, BlockWorld {}
public class PushAction implements Action {
     @Override
     public void execute(Environment env, Agent a) {
                //unavoidable (?) cast
          BlockWorld world = (BlockWorld) env;
          //get the block that the agent is facing, and move it
          //as specified in the action
}So define an interface according to the methods the world wants to expose. The only concern is that the same world (with obstacles and what not) could be used in a lot of different contexts, and I guess it's not desirable to subclass it every time just to make it implement an additional interface because one wants to expose different methods. But maybe I can use the Adapter pattern for this?