Overview

Supporting MPTCP on the server side is easy when services are directly exposed to the Internet: it’s generally just a matter of enabling MPTCP support in the applications, or forcing them to use it, that’s it.

When services are exposed behind a L4 load balancer, it is important to make sure the additional subflows will reach the same end-server, and not another one sharing the same public IP (ECMP or Anycast servers).

First path: no change

Creating the first subflow (or path) is easy: on the wire, this MPTCP subflow is seen as a TCP connection with extra TCP options. It means nothing needs to be modified.

flowchart LR
    C("fa:fa-mobile<br />Client") == Initial subflow ==> LB{"fa:fa-cloud<br />Load Balancer"}
    LB -.-> S1["fa:fa-server<br />Server 1"]
    LB == Initial subflow ==> S2["fa:fa-server<br />Server 2"]
    LB -.-> S3["fa:fa-server<br />Server 3"]

    linkStyle 0 stroke:green,fill:none
    linkStyle 2 stroke:green,fill:none

Extra paths: static redirection

The extra subflows need to reach the same end-server. Such subflows will have different source IP addresses and/or ports. A stateless L4 load-balancer needs extra information to pick the same end-server as the one which accepted the initial subflow.

flowchart LR
    C("fa:fa-mobile<br />Client") -- Initial subflow --> LB{"fa:fa-cloud<br />Load Balancer"}
    C == Second subflow ==> LB
    LB -.-> S1["fa:fa-server<br />Server 1"]
    LB -- Initial subflow --> S2["fa:fa-server<br />Server 2"]
    LB == Second subflow ==> S2
    LB -.-> S3["fa:fa-server<br />Server 3"]

    linkStyle 0 stroke:green,fill:none
    linkStyle 1 stroke:orange,fill:none
    linkStyle 3 stroke:green,fill:none
    linkStyle 4 stroke:orange,fill:none

If the extra subflows try to connect to the same destination IP address and port, a stateless L4 load-balancer will not be able to pick the right server.

Solution

The MPTCP protocol suggests handling this case like this:

  • A server behind a L4 load-balancer should mention in its replies to MPTCP connection requests (MP_CAPABLE) that it will not accept additional MPTCP subflows to the same IP address and port (via the C-flag).
  • Additionally, such server should announce an extra address (ADD_ADDR) with a v4/v6 IP address and/or port that are specific to this server.
  • A L4 load-balancer should route traffic to this specific IP and/or port to the right server.

In other words, on Linux, it means that each server should:

  • set the net.mptcp.allow_join_initial_addr_port sysctl knob to 0
  • add a signal MPTCP endpoint with a dedicated IP address and/or port: 
    ip mptcp endpoint add <public IP address> dev <interface> [ port NR  ] signal

A stateful load-balancer could compute the MPTCP receiver’s token from its key exchanged in the connection request (MP_CAPABLE), and route additional subflows to the same server by identifying the receiver’s token from the join request (MP_JOIN). Be careful that there is a risk of token collision, and such load-balancer should handle the case where multiple end-servers are using the same token for active MPTCP connections.

CDNs

Supporting MPTCP would be beneficial for the users, to be able to easily benefit from MPTCP: seamless handovers, best network selection, and network aggregation.

Here is a checklist for CDN owners implementing MPTCP support: