Routing conundrum with VM and bridges on Linux
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I have the following setup:
Hypervisor host 1 | Hypervisor host 2
|
------------------ |
| VM | |
| 10.244.140.10 | |
| GW 10.244.140.1| | Route to 10.244.8.0/24
------------------ | goes through br1
| |
/-------------- /------------- | /------------- /--------------
| br0 | | br1 | | | br1 | | br0 |
|10.244.140.254| | 10.244.8.21 | | | 10.244.8.1 | | 10.244.140.1 |
| | | .... | | | 10.244.8.33 | | |
--------------/ -------------/ | -------------/ --------------/
| | | | |
eth0 eth1------------------eth1 eth0
| | |
--------------------------------------------------------/
Two hosts with virtual machines. Several subnets are defined: 10.244.140.0/24 on the br0 bridges, 10.244.8.0/24 on the br1 bridges. The br1 bridges have several IP addresses. No netfilter rules. Netfilter policies are ACCEPT. ip_forward is set.
A VM is connected to br0 on host 1. It pings 10.244.8.21, an address that is in a different subnet but resides on the same host 1. The VM's default gateway is 10.244.140.1 and resides on host 2.
My naive expectation: The echo request packet is sent to br0 on host2, since br0 has the gateway address. The packet is then routed to br1 and back to host1.
Reality: The packet arrives as expected. However, it is not routed. On the other hand, packets sent to 10.244.8.33 arrive at br1 on host 2, and echo replies go back.
Where am I wrong? And more importantly: How can I fix this?
The hypervisor hosts run OpenSuse Leaf 15 and KVM.
EDIT: Since I don't have enough reputation to add comments, I put answers in the question. Thanks for your concern.
Fang: your packet arrives "as expected", it has passed subnet borders and thus, was routed. Sorry, my description was incomplete. It arrives at br0 on host 2, in the same subnet, and is not routed further.
Grawity: I guess your first question is answered now.
Which destination MAC address does the packet have when it traverses br0 on host1?
The router's MAC, i.e the MAC of br0 on host 2. It knows it because of an ARP for which it does get a response.
networking virtual-machine routing bridge
asked Feb 7 at 11:39
berndbauschberndbausch
13
add a comment |
I have the following setup:
Hypervisor host 1 | Hypervisor host 2
|
------------------ |
| VM | |
| 10.244.140.10 | |
| GW 10.244.140.1| | Route to 10.244.8.0/24
------------------ | goes through br1
| |
/-------------- /------------- | /------------- /--------------
| br0 | | br1 | | | br1 | | br0 |
|10.244.140.254| | 10.244.8.21 | | | 10.244.8.1 | | 10.244.140.1 |
| | | .... | | | 10.244.8.33 | | |
--------------/ -------------/ | -------------/ --------------/
| | | | |
eth0 eth1------------------eth1 eth0
| | |
--------------------------------------------------------/
Two hosts with virtual machines. Several subnets are defined: 10.244.140.0/24 on the br0 bridges, 10.244.8.0/24 on the br1 bridges. The br1 bridges have several IP addresses. No netfilter rules. Netfilter policies are ACCEPT. ip_forward is set.
A VM is connected to br0 on host 1. It pings 10.244.8.21, an address that is in a different subnet but resides on the same host 1. The VM's default gateway is 10.244.140.1 and resides on host 2.
My naive expectation: The echo request packet is sent to br0 on host2, since br0 has the gateway address. The packet is then routed to br1 and back to host1.
Reality: The packet arrives as expected. However, it is not routed. On the other hand, packets sent to 10.244.8.33 arrive at br1 on host 2, and echo replies go back.
Where am I wrong? And more importantly: How can I fix this?
The hypervisor hosts run OpenSuse Leaf 15 and KVM.
EDIT: Since I don't have enough reputation to add comments, I put answers in the question. Thanks for your concern.
Fang: your packet arrives "as expected", it has passed subnet borders and thus, was routed. Sorry, my description was incomplete. It arrives at br0 on host 2, in the same subnet, and is not routed further.
Grawity: I guess your first question is answered now.
Which destination MAC address does the packet have when it traverses br0 on host1?
The router's MAC, i.e the MAC of br0 on host 2. It knows it because of an ARP for which it does get a response.
networking virtual-machine routing bridge
asked Feb 7 at 11:39
berndbauschberndbausch
13
You're pinging 10.244.8.21 in 10.244.8.0/24 from 10.244.140.10 in 10.244.140.0/24. If, as you say, your packet arrives "as expected", it has passed subnet borders and thus, was routed.
– Fang
Feb 7 at 12:43
When you say "the packet arrives as expected", which host and which interface do you mean at that point? Also, which destination MAC address does the packet have the moment it shows up on host1's br0?
– grawity
Feb 7 at 12:53
add a comment |
I have the following setup:
Hypervisor host 1 | Hypervisor host 2
|
------------------ |
| VM | |
| 10.244.140.10 | |
| GW 10.244.140.1| | Route to 10.244.8.0/24
------------------ | goes through br1
| |
/-------------- /------------- | /------------- /--------------
| br0 | | br1 | | | br1 | | br0 |
|10.244.140.254| | 10.244.8.21 | | | 10.244.8.1 | | 10.244.140.1 |
| | | .... | | | 10.244.8.33 | | |
--------------/ -------------/ | -------------/ --------------/
| | | | |
eth0 eth1------------------eth1 eth0
| | |
--------------------------------------------------------/
Two hosts with virtual machines. Several subnets are defined: 10.244.140.0/24 on the br0 bridges, 10.244.8.0/24 on the br1 bridges. The br1 bridges have several IP addresses. No netfilter rules. Netfilter policies are ACCEPT. ip_forward is set.
A VM is connected to br0 on host 1. It pings 10.244.8.21, an address that is in a different subnet but resides on the same host 1. The VM's default gateway is 10.244.140.1 and resides on host 2.
My naive expectation: The echo request packet is sent to br0 on host2, since br0 has the gateway address. The packet is then routed to br1 and back to host1.
Reality: The packet arrives as expected. However, it is not routed. On the other hand, packets sent to 10.244.8.33 arrive at br1 on host 2, and echo replies go back.
Where am I wrong? And more importantly: How can I fix this?
The hypervisor hosts run OpenSuse Leaf 15 and KVM.
EDIT: Since I don't have enough reputation to add comments, I put answers in the question. Thanks for your concern.
Fang: your packet arrives "as expected", it has passed subnet borders and thus, was routed. Sorry, my description was incomplete. It arrives at br0 on host 2, in the same subnet, and is not routed further.
Grawity: I guess your first question is answered now.
Which destination MAC address does the packet have when it traverses br0 on host1?
The router's MAC, i.e the MAC of br0 on host 2. It knows it because of an ARP for which it does get a response.
networking virtual-machine routing bridge
asked Feb 7 at 11:39
berndbauschberndbausch
13
I have the following setup:
Hypervisor host 1 | Hypervisor host 2
|
------------------ |
| VM | |
| 10.244.140.10 | |
| GW 10.244.140.1| | Route to 10.244.8.0/24
------------------ | goes through br1
| |
/-------------- /------------- | /------------- /--------------
| br0 | | br1 | | | br1 | | br0 |
|10.244.140.254| | 10.244.8.21 | | | 10.244.8.1 | | 10.244.140.1 |
| | | .... | | | 10.244.8.33 | | |
--------------/ -------------/ | -------------/ --------------/
| | | | |
eth0 eth1------------------eth1 eth0
| | |
--------------------------------------------------------/
Two hosts with virtual machines. Several subnets are defined: 10.244.140.0/24 on the br0 bridges, 10.244.8.0/24 on the br1 bridges. The br1 bridges have several IP addresses. No netfilter rules. Netfilter policies are ACCEPT. ip_forward is set.
A VM is connected to br0 on host 1. It pings 10.244.8.21, an address that is in a different subnet but resides on the same host 1. The VM's default gateway is 10.244.140.1 and resides on host 2.
My naive expectation: The echo request packet is sent to br0 on host2, since br0 has the gateway address. The packet is then routed to br1 and back to host1.
Reality: The packet arrives as expected. However, it is not routed. On the other hand, packets sent to 10.244.8.33 arrive at br1 on host 2, and echo replies go back.
Where am I wrong? And more importantly: How can I fix this?
The hypervisor hosts run OpenSuse Leaf 15 and KVM.
EDIT: Since I don't have enough reputation to add comments, I put answers in the question. Thanks for your concern.
Fang: your packet arrives "as expected", it has passed subnet borders and thus, was routed. Sorry, my description was incomplete. It arrives at br0 on host 2, in the same subnet, and is not routed further.
Grawity: I guess your first question is answered now.
Which destination MAC address does the packet have when it traverses br0 on host1?
The router's MAC, i.e the MAC of br0 on host 2. It knows it because of an ARP for which it does get a response.
networking virtual-machine routing bridge
networking virtual-machine routing bridge
asked Feb 7 at 11:39
berndbauschberndbausch
13
asked Feb 7 at 11:39
berndbauschberndbausch
13
edited Feb 7 at 13:47
berndbausch
asked Feb 7 at 11:39
berndbauschberndbausch
13
asked Feb 7 at 11:39
berndbauschberndbausch
13
asked Feb 7 at 11:39
berndbauschberndbausch
13
13
You're pinging 10.244.8.21 in 10.244.8.0/24 from 10.244.140.10 in 10.244.140.0/24. If, as you say, your packet arrives "as expected", it has passed subnet borders and thus, was routed.
– Fang
Feb 7 at 12:43
When you say "the packet arrives as expected", which host and which interface do you mean at that point? Also, which destination MAC address does the packet have the moment it shows up on host1's br0?
– grawity
Feb 7 at 12:53
add a comment |
You're pinging 10.244.8.21 in 10.244.8.0/24 from 10.244.140.10 in 10.244.140.0/24. If, as you say, your packet arrives "as expected", it has passed subnet borders and thus, was routed.
– Fang
Feb 7 at 12:43
When you say "the packet arrives as expected", which host and which interface do you mean at that point? Also, which destination MAC address does the packet have the moment it shows up on host1's br0?
– grawity
Feb 7 at 12:53
You're pinging 10.244.8.21 in 10.244.8.0/24 from 10.244.140.10 in 10.244.140.0/24. If, as you say, your packet arrives "as expected", it has passed subnet borders and thus, was routed.
– Fang
Feb 7 at 12:43
You're pinging 10.244.8.21 in 10.244.8.0/24 from 10.244.140.10 in 10.244.140.0/24. If, as you say, your packet arrives "as expected", it has passed subnet borders and thus, was routed.
– Fang
Feb 7 at 12:43
When you say "the packet arrives as expected", which host and which interface do you mean at that point? Also, which destination MAC address does the packet have the moment it shows up on host1's br0?
– grawity
Feb 7 at 12:53
When you say "the packet arrives as expected", which host and which interface do you mean at that point? Also, which destination MAC address does the packet have the moment it shows up on host1's br0?
– grawity
Feb 7 at 12:53
add a comment |
1 Answer
1
active
oldest
votes
Reverse path filtering.
I had known that it existed, but I didn't make the connection to my problem. RPF limits reply packets to be sent out through the receiving interface. in my case, replies would go through a different interface and are therefore blocked.
The solution is to disable reverse path filtering on both hypervisor hosts, for example
# sysctl net.ipv4.conf.all.rp_filter=0
This makes hosts vulnerable to certain network attacks, but my environment is isolated from the big bad internet and I don't care.
For more information, see the rp_filter paragraph on the kernel documentation page for network parmeters.
answered Feb 8 at 2:27
berndbauschberndbausch
13
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1403092%2frouting-conundrum-with-vm-and-bridges-on-linux%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Reverse path filtering.
I had known that it existed, but I didn't make the connection to my problem. RPF limits reply packets to be sent out through the receiving interface. in my case, replies would go through a different interface and are therefore blocked.
The solution is to disable reverse path filtering on both hypervisor hosts, for example
# sysctl net.ipv4.conf.all.rp_filter=0
This makes hosts vulnerable to certain network attacks, but my environment is isolated from the big bad internet and I don't care.
For more information, see the rp_filter paragraph on the kernel documentation page for network parmeters.
answered Feb 8 at 2:27
berndbauschberndbausch
13
add a comment |
Reverse path filtering.
I had known that it existed, but I didn't make the connection to my problem. RPF limits reply packets to be sent out through the receiving interface. in my case, replies would go through a different interface and are therefore blocked.
The solution is to disable reverse path filtering on both hypervisor hosts, for example
# sysctl net.ipv4.conf.all.rp_filter=0
This makes hosts vulnerable to certain network attacks, but my environment is isolated from the big bad internet and I don't care.
For more information, see the rp_filter paragraph on the kernel documentation page for network parmeters.
answered Feb 8 at 2:27
berndbauschberndbausch
13
add a comment |
Reverse path filtering.
I had known that it existed, but I didn't make the connection to my problem. RPF limits reply packets to be sent out through the receiving interface. in my case, replies would go through a different interface and are therefore blocked.
The solution is to disable reverse path filtering on both hypervisor hosts, for example
# sysctl net.ipv4.conf.all.rp_filter=0
This makes hosts vulnerable to certain network attacks, but my environment is isolated from the big bad internet and I don't care.
For more information, see the rp_filter paragraph on the kernel documentation page for network parmeters.
answered Feb 8 at 2:27
berndbauschberndbausch
13
Reverse path filtering.
I had known that it existed, but I didn't make the connection to my problem. RPF limits reply packets to be sent out through the receiving interface. in my case, replies would go through a different interface and are therefore blocked.
The solution is to disable reverse path filtering on both hypervisor hosts, for example
# sysctl net.ipv4.conf.all.rp_filter=0
This makes hosts vulnerable to certain network attacks, but my environment is isolated from the big bad internet and I don't care.
For more information, see the rp_filter paragraph on the kernel documentation page for network parmeters.
answered Feb 8 at 2:27
berndbauschberndbausch
13
answered Feb 8 at 2:27
berndbauschberndbausch
13
answered Feb 8 at 2:27
berndbauschberndbausch
13
answered Feb 8 at 2:27
berndbauschberndbausch
13
13
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1403092%2frouting-conundrum-with-vm-and-bridges-on-linux%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You're pinging 10.244.8.21 in 10.244.8.0/24 from 10.244.140.10 in 10.244.140.0/24. If, as you say, your packet arrives "as expected", it has passed subnet borders and thus, was routed.
– Fang
Feb 7 at 12:43
When you say "the packet arrives as expected", which host and which interface do you mean at that point? Also, which destination MAC address does the packet have the moment it shows up on host1's br0?
– grawity
Feb 7 at 12:53