CS3 comms talk

George Ross March 1998

\overline{History 1}

• Department started out with a small collection of minicomputers (DEC PDP-8, PDP-9, PDP-15, Interdata 70 and 74, and the like), connected together by point-to-point 1Mb links. Central provision by ERCC on EMAS (ICL 4/75).

• Some multi-user machines came along, again connected up by point-to-point links.

• One (later three) “filestores” held user files.

• External connectivity initially provided by a PDP-11, later ECSVAX.

• First forays into ethernet ca. 1978.

• By mid 1980s had fairly extensive home-brew “slow” 2Mb and standard 10Mb ether.

• One main wire and several secondary wires

• Suns as routers

– cheaper than “real” dedicated router hardware – several in parallel for resilience

– RIP (RFC1058, RFC1723) keeps track of how to get around

• When more bandwidth was needed we just added another wire But 10base2 has problems:

• bus strucure makes optimal machine placement difficult — at one time we had half-a dozen wires through most of the Department but some people were still constrained to occupy only certain offices.

• it’s rather fragile — there are too many ways to break it, and it’s too hard to diagnose and fix problems.

Solution: install 10baseT — star topology, individual outlets can be patched as required. We’ve adopted a “spine and ribs” network.

(But we now have a lot of obsolescent Cat-3 wiring …)

The 100baseT backbone has enough bandwidth for now (though not all machines are on it that we might want). But new machines can saturate even 100baseT. The “usual” solution (“switches”) doesn’t work well for us because of traffic patterns — NFS/UDP doesn’t like dropped fragments!

RIP doesn’t distinguish between “fast” and “slow” links. We might have to switch to OSPF (RFC1583) instead.

It can be quite difficult to provide good connectivity for central services without using an unreasonable number of interfaces.

The network is much too complicated to visualise as a whole.

The new EdLAN may give us some useful connection options which we might not be able to support with our current structures.

Many of our servers are now “multi-homed,” either because they’re also routers, because they’re on one or more of the parallel spines, or because we put them on several wires to avoid unnecessary hops. How do clients find the “best” way to speak to them?

• We could give each interface its own distinguished name. Users would then have to know the relevant network topology in order to choose the best one (almost invariably they get it wrong!). We did have our automount maps constructed like this for a long time.

• We could name only one of the interfaces. Requests would then often go to the “wrong” one, but replies would go back more or less optimally. We do this for our web server, because it’s the easiest way to keep external traffic under control. (We do external mail and telnet and ssh this way too.)

• We could give one single name to all the interfaces and configure each system so that it can pick the “best” one. This is our general solution. Each machine generates a sortlist when it boots up, based on its own local connection(s) and hints provided by the system managers. It’s limited, but given our current traffic patterns it’s usually good enough.

The Internet is a hostile place. We see some “unexpected” traffic just about every day, and serious probing roughly once a month. How do we handle it?

• Where possible services are “wrapped” so that we can control who can connect to them and from where. Unfortunately many standard daemons and services can’t be. So …

• All our routers are running IP filters — passing most local traffic, passing specific external traffic, and blocking everything else. The default ruleset has nearly 70 individual rules (including generated ones).

It would be much more efficient if this traffic could be filtered at the perimeter instead, of course.

Specific protocols give rise to security problems too:

• NFS is currently implemented such that the server trusts the client to identify the user. We could work around this to some extent by only exporting filesystems to machines under our own central management, but this is beginning to prove restrictive. The medium-term answer will involve Kerberos or something similar, but deployment of this is being held up by the US government which regards mathematics as being like tanks and atom-bombs …

• Although our own networks are “reasonably” secure, anyone coming in from outside (at a conference or collaborating site, from an ISP at home) has to expect that their traffic (including passwords) can be snooped on. Our preferred solution is to use ssh, which provides an end-to-end encrypted tunnel for terminal emulator, X, pop, and indeed just about every useful protocol that’ll run over TCP except for FTP