COMMAND
NIS/NIS+
SYSTEMS AFFECTED
Systems runnins NIS/NIS+
PROBLEM
Following is based on ISS advisory. For purposes of this report,
NIS refers to both NIS and NIS+ as this problem has been observed
and reproduced on both services. It is possible, through a well
orchestrated attack using the finger service against multiple NIS
clients, to disrupt an entire NIS based network and/or starve the
NIS servers for resources. The problem is in the finger service
but the attack causes long duration, network-wide, congestion and
resource exhaustion on NIS servers.
A finger request results in multiple NIS requests as the
responding daemon attempts to locate all account records matching
the finger request. A request for finger foo@bar.com will result
in one finger daemon searching incrementally through all of the
entries in the passwd map to locate any accounts with foo in the
name. As a consequence, a single finger request can result in a
significantly larger amount of traffic between the NIS client and
the NIS server than the originating traffic to and from the finger
service. The amount of NIS traffic is dependent on the size of
the NIS passwd map. With a passwd map of 10,000 entries, a single
finger request would result in roughly 10,000 NIS requests and
10,000 NIS responses. This does NOT count retries from packet
loss or other failures (a highly significant factor in this
attack). By sending a large number of overlapping finger requests
to a single host, it is possible to load that host down with a
very significant amount of traffic just processing the NIS
requests. If this is done to multiple hosts, the network traffic
rises dramatically.
Eventually, a condition arises in which congestion and/or resource
exhaustion on the NIS server begin to cause a significant rise in
lost packets and failed requests. This results in retry attempts
from the NIS clients, adding to the already overloaded network
traffic. The failure / retry / failure cycle becomes an NIS
traffic "storm" in which the retry traffic dominates and little
other traffic can squeeze through.
Network congestion combined with NIS server resource exhaustion
work together to not only deny service to the requesting clients
but also to rapidly clog the network bandwidth and render the
network unusable by anything on the network.
In analyzing this attack, a perl script was used to generate
finger traffic attacking a dozen hosts with four finger requests
for each of approximately 100 names (~400 finger requests per
host). A demonstration NIS map of approximately 1000 accounts was
used. At an issue rate of approximately 4 finger requests every
two seconds against a given host, 10's to 100's of lingering
finger requests would build up even as some finger requests would
be fufilled. These lingering finger processes would be attempting
to paw their way through the entire NIS password map. A typical
test run attack lasted approximately 30-50 seconds in duration.
During analysis of this attack, network traffic from even a short
~30 seconds blast from the perl test script resulted in traffic
levels that caused network disruptions extending for as much as
45 minutes to an hour after cessation of the attack. During this
time, some systems were impacted to the extent that screen savers
froze and systems were unresponsive to the keyboards. Many
systems were left with seemingly hung finger processes. These
stayed on the system for a half an hour or more while the network
congestion cleared. Some systems ran out of swap space because of
the resource demands of the finger processes. On a few of the
test runs the network traffic was observed to have risen to a
level which caused a switched ethernet hub to disable ports due to
excesive collisions. Finger requests to perform this action have
to be distributed and timed properly. Too many requests, too
quickly, seem to result in inetd disabling the finger service.
Too slowly, and the network traffic rises too slowly and fails to
reach the catastrophic level where packet loss and retries become
the dominant traffic input to the network.
Because the finger requests are TCP based and not dependent on
preauthentication, finger requests can still be delivered by the
attacker to the systems under attack even in the face of
increasing network congestion. By the time the attacking
connections are significantly impacted by the network congestion,
the network has been rendered unusable by systems requiring NIS
or other services. Timed correctly, an attack of only a few
seconds, targeting as few as a dozen NIS clients on a network with
a moderate NIS passwd map can render even a small network unusable
for as long as a half an hour to an hour or more. Increasing the
size of the NIS passwd map, the number of attacked clients, or the
number of requests sent to any given client causes the recovery
time to extend out dramatically and disproportionately to any
particular increases in any particular factor. If the NIS server
is also one of the attacked systems, it can rapidly run out of
system resources, causing NIS request failures and accelerating
the resulting NIS traffic "storm". When the NIS server was one of
the systems attacked by finger requests, it was not unusual to see
warnings about unable to grow stack, exhausted virtual memory, or
other resource related errors.
MOST client systems seem to clean themselves up EVENTUALLY. This
can take anywhere from a few moments for some Linux boxes, to a
significant fraction of an hour for some SUN boxes. It was
observed that some IRIX boxes and AIX boxes would become
unreachable from the network and unresponsive to the keyboard,
requiring a power cycle to recover. These last systems may have
recovered on their own eventually, but that time frame appears to
be geological. Recovery time seems to also be dependent on the
recovery time of the NIS server for those clients which were
observed to recover. Resetting the targeted systems permits the
network to recover. All tested systems were affected to some
extent. Because the resulting traffic and congestion is
proportional to the size of the NIS passwd map times the number of
attacked hosts times the number of requests in the attack, large
networks are disproportionately vulnerable to this attack. Even
small networks of a few dozen systems can be disabled by a
determined attacker if they have a sufficiently large NIS passwd
map.
The finger service permits a condition where a limited number of
requests can result in a vastly larger number of internal requests
against a central naming service such as NIS. This permits an
attacker to mount a distributed attack by launching smaller
attacks against numerous hosts. These combine to form a
disasterous level of congestion on the internal systems,
disrupting an internal network for an extended period of time. It
is unknown, at this time, if any other services exhibit similar
characteristics with regard to NIS traffic as does finger.
Michael H. Warfield added more tips how to reproduce this. To
reproduce this problem, you are going to have to do some tricky
juggling.
1) Make sure you have a thousand or more account names in your NIS
password map (this is very IMPORTANT) Obviously this affects
only large networks in practice. You need to add that many
fake names if you are attempting to reproduce this in a test
environment on a small network.
2) Start a few process, for each address to attack, to attack each
of a large number of hosts.
- In each process -
3) Open up a finger connection and feed down the request.
4) If the request returns data within a certain timeout, sleep for
a period of time before launching the next request.
5) If the request does not return data within the timeout, close
the connection and start another connection immediately (this
is the first indication that you are suceeding in congesting
the network).
6) Cycle through this process, steps 3-5, for the specified number
of names in your attack list or for the attack period of time.
Now... This is something like "tickling the dragon's tail", an
old, very dangerous, no longer practiced, experiment in nuclear
physics where one would determine the critical mass of a
radioactive element by piling it up and shaping the pile until it
went critical (several scientists died doing this). Everything
behaves resonably quiet until you hit that right combination and
then it goes through the roof. When you find that right
combination, the results are pretty impressive.
SOLUTION
Disable finger service on any systems connected to any NIS based
network. Disable access to internal finger services at all
perimeter defenses (firewalls, gateways, filtering routers, etc.).
If the finger service is required for some specific purpose, limit
it to the minimum number of restricted hosts or to hosts which are
NOT participating in NIS.
For those who wish to continue to run the finger service on their
systems, there are other possible actions that could be taken,
like using a public domain fingerd that doesn't do ambiguous
lookups.
Suggestion is at the very least deny access to finger requests
from the outside.