Je ha, auto add/remove machine?

Question

Cloud servers such as Amazon ec2 have the ability to start/stop machines easily.
Can I add/remove machines for ha groups automatically? And how to decide when to do that?

Answer 1

I haven't asked this yet so, do you already have an application built on top of JE HA? How many nodes are there in the replication group? Are you observing a good throughput in the application (for read and write operations) and implicitly a reduced latency in response times?
There's no guide or documentation material covering what exact information you should be looking for when deciding whether there's a need to add or remove a replication node.
JE HA is a single master/writer - multiple clients/readers HA system, so adding a node to a replication group increases the ability of the group to do reads, and makes additional copies of the data, which makes it more secure. Though, it does nothing for writes.
The decision to add a node to a replication group (generically, not necessarily for a JE HA rep group), aside the infrastructure, network partitions, bandwidth, hardware availability, replication factor etc criteria, should be governed by the expected response time for a read operation (the time it takes a node, replica nodes or master node, to service a query). This is something the application should monitor and when the read request's response time increases above an allowed limit, you could add a new (replica) node. Of course, all of this revolves around load balancing the read requests, which is a good use case for using a Monitor node in the rep group that routes requests (both write and read requests) and balances the read requests between the replicas.
Another criteria for deciding to add a new node can be the availability of enough replicas to satisfy the durability and consistency policies.
CPU/mem/disk info are outside of JE's ability to obtain -- there are system tools that you can use to collect such info. Replication related information can be obtained in JE through the ReplicatedEnvironmentStats and through the monitoring and diagnostic JMX MBeans RepJEMonitor and RepJEDiagnostics (or the RepJEJConsole JConsole Plugin).
Though, what you really need to do at this stage is to see what the latency and throughput is, at the application level. Plus there's the question of load balancing, so you need to be aware that you have to spread the reads among different nodes.
You should also your system stats. Are the replicas loaded to the max or not? If they're very loaded in terms of CPU and I/O, maybe you need more rep nodes.
Whereas for removing, you could remove a node from a rep group if for example the master spends too much time to achieve a durability ack policy (ReplicatedEnvironmentStats.getAckWaitMs() might be useful in this case) or if the replica node is lagging the master too much (compared to the other replica nodes -- ReplicatedEnvironmentStats.getTrackerLagConsistencyWaitMs() is relevant).
Regards,
Andrei