Can Cloud Control 12c be on the Exadata client network?

Question

Hi, there,
We’re trying to configure our brand new X3 machine but we’re having a problem with connecting our Cloud Control OMS to it (the install of the agents fails because it can't connect to the hostname we provided on the management interface). We’re running the latest and greatest Exadata stack (11.2.3.2.1, 11.2.0.3.18) and we’re running Cloud Control 12.1.0.2
The OMS server – oem12oms – is on the same subnet as the client network on the Exadata comp nodes. However, when we try to ping the management hostname of the comp nodes (the docs say using the client hostname is not supported until 12.1.0.3) from the OMS server, we have no success:
For instance:
oem12oms – 10.50.50.122 (running on eth0)
exadbclient1 – 10.50.50.123 (running on eth1 on the first comp node)
exadbclient2 – 10.50.50.124 (running on eth1 on the second comp node)
exadbadm1 – 10.75.100.23 (running on eth0 on the first comp node)
exadbadm2 – 10.75.100.24 (running on eth0 on the second comp node)
From oem12oms, when both eth0 and eth1 are up on the comp nodes, it fails:
traceroute exadbadm1 (10.75.100.23)
10.50.50.1 (the default gateway for both the client network and the Exadata comp node itself)
Then goes nowhere
When we shut down the eth1 interface, I am able to access the management interface from my OMS server:
traceroute exadbadm1 (10.75.100.23)
                             10.50.50.1 (the default gateway for both the client network and the Exadata comp node itself)
exadbadm1 (10.75.100.23)
We have rules/routes which set on the comp node (by default) that dictate all traffic from:
the management interface (eth0) uses routing table 220 and comes from 10.75.100.xxx and goes out the same interface on 10.75.100.xxx.
the client (eth1) uses routing table 221 and comes from 10.50.50.xxx and goes out the same interface on 10.50.50.xxx
Essentially, while we want to tell incoming traffic NOT to go out a different interface to which it came in, I think we want to workaround this for one exception: the OMS server’s IP address (10.50.50.122) should come in from the MANAGEMENT interface (not the client) and go out the MANAGEMENT interface.
There are a lot of references to advanced Linux network policies, etc. And we’ve read 1306154.1. However, it doesn’t seem to solve our problem and we’ve tried all the ways that we can think of: essentially, it looks like Exadata cannot connect to clients over the management interface which live on its client network (obviously, it will still be able to be accessed through the SCAN addresses), so your Cloud Control cannot be on the client network.
I haven’t read anything which states this, but that’s how it looks to us.
Anyone had this issue before? Our ACS engineer is stumped and has asked around internally to no avail.
Mark

Answer 1

Mark,
I read your article about this - it's the best article out there by a LONG way, but it still confuses me because of my eye-popping ignorance. Well done, though, because I almost understood it so you're clearly skilled in explaining complex stuff to untrained monkeys
Oracle don't appear to know about this issue, which astounds me. We can't be the only customer who has their client network on their 'main' network. We've had two ACS engineers who have been flummoxed by this and have tried to reach out to the 'internal guru' network in Oracle with joy.
I tried to type out what I should do (the configuration is in a mess right now, so I haven't tried this out yet). This is to get things clear in my mind.
OMS server: 10.50.50.122 (client interface - uses the same subnet as the 'general Production' network)
DB Comp Node 1: 10.75.100.23 (management interface - uses an exclusive subnet for Exadata management hostnames)
First, we add a custom routing table 230 to the /etc/iproute2/rt_tables
     echo 230 >> /etc/iproute2/rt_tables
Then we need to inform the magical, mysterious routing database of this new table and what we want to do with it:
Add a route to direct traffic from the OMS server (client network) to the comp node (management network) using table 230
Add a rule which tells traffic from the OMS server to use table 230 (with higher priority than other rules)
Add a rule which tells traffic from the comp node to use table 230 (with even higher priority)
          ip route add 10.50.50.122/32 via 10.75.100.23 dev eth0 table 230
          ip rule add from 10.50.50.122 prio 32750 table 230
          ip rule add to 10.75.100.23 prio 32751 table 230
The ifcfg files look OK so I didn't see the need to change them.
I thought I'd add a route associated with routing table 230 to the rule-eth0 file so that it has two entries (added the last two lines)
The first entry uses table 220 to route traffic coming in on the management interface to the management interface
The second (new) entry uses table 230 to route traffic coming from the OMS server (client network) to the management interface
          from 10.75.100.23 table 220
          to 10.75.100.23 table 220
          from 10.50.50.122 table 230
          to 10.75.100.23 table 230
Then I'd change the route-eth01 to add the new rule (added the last line)
The first entry tells anything coming in from the management interface to use the eth0 NIC and table 220 (which instructs anything coming from the management network to stay on the management network)
The second entry tells anything coming in from the management interface gateway to do the same (eth0 NIC and table 220)
The third entry (new) tells anything coming from that specific IP address to route to the management IP of the comp node via eth0 but using our new table 230
          10.75.100.0/24 dev eth0 table 220
          default via 10.75.100.1 dev eth0 table 220
          10.50.50.122/24 via 10.75.100.23 eth0 table 230
This should mean that the rules/routes will be statically configured so when we restart the network services or the entire box, it should save them?
Does that look as it would make sense? The whole thing SEEMS to make perfect sense and seems easy when reading about it, but doesn't seem to play nicely when you come to actually implement it.
Mark