I ran into a really weird problem the other day. I did some maintinence the night before which involved implementing voice VLAN's for our ip phones and updating the PIX os on our, what else Cisco PIX.
The next day we had users having many differnet but related problems. First users were having issues opening Outlook and running some network applications, some over our internal web apps would not work properly, and we could not terminal service to our window servers.
We started troubleshooting our network and server and could not fins any problems. We isolated servers from our network and they would work fine but when the switch we isolated them to was reconnected to the network the problem would re-occur. We did many packets sniffs and did not see anything. We rolled back all of our switching configurations to no effect. We went as far as blanking our switch config to no effect. Eventually we rolled back our PIX os to the previous version and that resolved the issue. A very weird issue that we still have yet to find why the PIX was causing this issue.
Update: After doing some research on pix and proxy arp at the recomendation of a co-worker i found the following.
sysopt noproxyarp
ARP (Address Resolution Protocol) is a layer two protocol that resolves an IP address to a physical address, also called a Media Access Controller (MAC) address. A host sends an ARP request asking "Who is this IP?" The device owning the IP should reply with "Hey, I am the one, here's my MAC address."
Proxy ARP refers to a gateway device, in this case, the firewall, "impersonating" an IP address and returning its own MAC address to answer an ARP request for another device.
The firewall builds a table from responses to ARP requests to map physical addresses to IP addresses. A periodic ARP function is enabled in the default configuration. The presence of entries in the ARP cache indicates that the firewall has network connectivity. The show arp command lists the entries in the ARP table. Usually, administrators do not need to manually manipulate ARP entries on the firewall. This is done only when troubleshooting or solving network connectivity problems.
The arp command is used to add a permanent entry for host on a network. If one host is exchanged for another host with the same IP address then the "clear arp" command can be used to clear the ARP cache on the PIX. Alternatively, you can wait for the duration specified with the arp timeout command to expire and the ARP table rebuilds itself automatically with the new host information.
The sysopt noproxyarp command is used to disable Proxy ARPs on an interface from the command-line interface. By default, the PIX Firewall responds to ARP requests directed at the PIX Firewall's interface IP addresses as well as to ARP requests for any static or global address defined on the PIX Firewall interface (which are proxy ARP requests).
The sysopt noproxyarp if_name command lets you disable proxy ARP request responses on a PIX Firewall interface. However, this command does not disable (non-proxy) ARP requests on the PIX Firewall interface itself. Consequently, if you use the sysopt noproxyarp if_name command, the PIX Firewall no longer responds to ARP requests for the addresses in the static, global, and nat 0 commands for that interface but does respond to ARP requests for its interface IP addresses.
To disable Proxy ARPs on the inside interface:
sysopt noproxyarp inside
To enable Proxy ARPs on the inside interface:
no sysopt noproxyarp inside
I then looked back at my packet sniffs that i ran and saw some weird things. interfaces on my PIX responding for other networks that it was not part of (this i believe was due to a VLAN misconfiguration). We are going to do some more testing and some sniffs on our current version of PIX os.
Internap suffered a power outage at Fisher Plaza in Seattle, Washington on Friday January 14th.
From eWeek:
A sudden power outage has knocked millions of Six Apart Ltd.'s LiveJournal blogs offline.
The power failure occurred on Friday evening at the Internap data center affected more than 100 servers that keep LiveJournal's blogging network up and running.
"LiveJournal is currently completely inaccessible, and we're waiting on Internap for an estimate when power will be restored. Once power is restored, the service will be brought back up slowly so that we can ensure data integrity," Six Apart said in a notice. "We'll [provide an] update with an estimate for when the service will be brought back up once we hear back from Internap."
On the LiveJournal site, visitors are being directed to an powerloss update page that provides more information on the battle to get the 5.6 million blogs back online.
"The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape," the update notice read. "For now, please be patient. We'll be working all weekend on this if we have to."
"We're going to be buying a bunch of rack-mount UPS units on Monday so this doesn't happen again," the company added.