I borrowed this image from here
]]>This appeared to be a global outage on the Level(3) Network. The RFO will be posted here once recieved.
]]>The next day we had users having many differnet but related problems. First users were having issues opening Outlook and running some network applications, some over our internal web apps would not work properly, and we could not terminal service to our window servers.
We started troubleshooting our network and server and could not fins any problems. We isolated servers from our network and they would work fine but when the switch we isolated them to was reconnected to the network the problem would re-occur. We did many packets sniffs and did not see anything. We rolled back all of our switching configurations to no effect. We went as far as blanking our switch config to no effect. Eventually we rolled back our PIX os to the previous version and that resolved the issue. A very weird issue that we still have yet to find why the PIX was causing this issue.
Update: After doing some research on pix and proxy arp at the recomendation of a co-worker i found the following.
sysopt noproxyarp
ARP (Address Resolution Protocol) is a layer two protocol that resolves an IP address to a physical address, also called a Media Access Controller (MAC) address. A host sends an ARP request asking "Who is this IP?" The device owning the IP should reply with "Hey, I am the one, here's my MAC address."
Proxy ARP refers to a gateway device, in this case, the firewall, "impersonating" an IP address and returning its own MAC address to answer an ARP request for another device.
The firewall builds a table from responses to ARP requests to map physical addresses to IP addresses. A periodic ARP function is enabled in the default configuration. The presence of entries in the ARP cache indicates that the firewall has network connectivity. The show arp command lists the entries in the ARP table. Usually, administrators do not need to manually manipulate ARP entries on the firewall. This is done only when troubleshooting or solving network connectivity problems.
The arp command is used to add a permanent entry for host on a network. If one host is exchanged for another host with the same IP address then the "clear arp" command can be used to clear the ARP cache on the PIX. Alternatively, you can wait for the duration specified with the arp timeout command to expire and the ARP table rebuilds itself automatically with the new host information.
The sysopt noproxyarp command is used to disable Proxy ARPs on an interface from the command-line interface. By default, the PIX Firewall responds to ARP requests directed at the PIX Firewall's interface IP addresses as well as to ARP requests for any static or global address defined on the PIX Firewall interface (which are proxy ARP requests).
The sysopt noproxyarp if_name command lets you disable proxy ARP request responses on a PIX Firewall interface. However, this command does not disable (non-proxy) ARP requests on the PIX Firewall interface itself. Consequently, if you use the sysopt noproxyarp if_name command, the PIX Firewall no longer responds to ARP requests for the addresses in the static, global, and nat 0 commands for that interface but does respond to ARP requests for its interface IP addresses.
To disable Proxy ARPs on the inside interface:
sysopt noproxyarp inside
To enable Proxy ARPs on the inside interface:
no sysopt noproxyarp inside
I then looked back at my packet sniffs that i ran and saw some weird things. interfaces on my PIX responding for other networks that it was not part of (this i believe was due to a VLAN misconfiguration). We are going to do some more testing and some sniffs on our current version of PIX os.
]]>From eWeek:
A sudden power outage has knocked millions of Six Apart Ltd.'s LiveJournal blogs offline.
The power failure occurred on Friday evening at the Internap data center affected more than 100 servers that keep LiveJournal's blogging network up and running.
"LiveJournal is currently completely inaccessible, and we're waiting on Internap for an estimate when power will be restored. Once power is restored, the service will be brought back up slowly so that we can ensure data integrity," Six Apart said in a notice. "We'll [provide an] update with an estimate for when the service will be brought back up once we hear back from Internap."
On the LiveJournal site, visitors are being directed to an powerloss update page that provides more information on the battle to get the 5.6 million blogs back online.
"The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape," the update notice read. "For now, please be patient. We'll be working all weekend on this if we have to."
"We're going to be buying a bunch of rack-mount UPS units on Monday so this doesn't happen again," the company added.
]]>At work we have been evaluating the RazorGate project. This product is a Anti-Spam and Anti-Virus mail filter appliance product.
Administration: The web interface is quite bleak. Very light weight and not incredibly user friendly. There isn't a whole lot of configuration to the product, you enable the various features and not much else. It does keep extensive logs on the messages that pass through the device. The server supports SSL. Also it has a telnet interface which I have not used extensivly but can do real-time displays if mail traffic.
Anti-Spam: One if the first things I noticed is right on the anti-spam it says that it is based on SpamAssassin. The rules it uses are based off a service that Mirapoint provides. They are very strict rules. From what i've found if the message has HTML is will get makes pretty highly. They use a 0-300 score level It does White and Black listing. and use RBL lists.
Anti-Virus: It is based off of Sophos for antivirus. This seems to work pretty well. her have gotten out fair share of the common virii out there and they have been caught. I have not seen any on the antivirus on the internal mail server.
Untested Features: This box will also act as a basic mail server handling POP, IMAP, and Webmail. Mailhurdle, they advertise this anti-spam feature very highly but i have not tested it out.
Over all this is a pretty effective appliance. i would recomend it but you will as with any anti-spam solution have to play with the filter to get the right ballance of score for your mail traffic. We have had to make heavy use of the White list.
]]>Update: as of 2:30 our circuit appears to be back live
Update 2: Sprint had a order in to place a shelf, and a SBC technicion replaced the wrong shelf. Classic SBC.
]]>I found that there is not much documentation around for this card. to set it up you fisrt need to use the card type command. the syntax is card type *T3 or E3* *module number*. You will then have a controller interface and a serial interface to work with. From there is a pretty standard setup.
If you need a T3 on a lowend platform this is the card for you.
]]>I have just gotten this information that was sent to a client of ours tha tis having a problem. This Message is from AT&T by the way not SBC.
SBC had an OC3 failure in Pontiac today at about noon. This failure has caused many disruptions in service for over 1300 SBC customers in Michigan.
How this failure has been affecting us, is that the failure is affecting
SS7 (signalling service 7) issue with AT&T toll connect service to our area. The AT&T NOC and SBC Technicians are working to fix the problem.
It's not geeks vs. geeks. It's geeks vs. industrial and regulatory morass.
Picture a city with two transportation systems, one network of roads and one network of canals. Traditionally, the milkman has made his rounds in a cart, leaving bottles of milk on doorsteps as you might imagine. But the company who runs the canals sees an opportunity to compete in the milk delivery business. It's very simple, they just dump a million gallons of milk into the canal system, and install special pumps at each building to suck up the milk, filter it, and bottle it. It's completely the wrong way to do things, but it's the only way the canal company can get into the milk industry.
Any geek with a modicum of physical-layer understanding will tell you why BPL is a conceptual abortion. Power lines carry a lot of current at very high voltage, but luckily 60Hz is a low frequency. They still blast a lot of noise out, because it's not practical to shape them in such a way as to keep the interference contained. We deal with it. Any audio engineer will tell you the extreme measures it takes to keep 60Hz noise out of a recording. Certain types of measurement gear include special sampling strategies to ignore 60Hz noise which would otherwise swamp the subject signal. ULF radio transmissions use other frequencies, because 60Hz is just useless. And of course, property values are lower near large powerlines because of the potential health effects of the magnetic field. (I'm not going to get into the verifiability of those effects, simply saying that the effect on property value is real.)
All this is because powerlines are spaced several feet apart, to keep the voltage from arcing from one conductor to the other. This spacing turns them into effective antennae, happily radiating the signal that's fed into them. If aliens are going to notice Earth, it'll be from the low-frequency emanations from our power grids.
Now consider Ethernet for a moment, specifically its original incarnation, known as 10base2. The name tells us a lot, 10 is the signalling rate, in megabits per second. "base" means it's a baseband medium, that the signalling is applied directly to the wire, not used as a modulation for some sort of a carrier signal. The 2 tells us the kind of wire, 2-conductor coaxial cable. The inner conductor is used for the signal, and the outer shield keeps the noise from leaking out. This is important, because in order to achieve a useful data rate in the megabits, the spectral width of the signal is pretty huge. If not carefully contained in cabling designed for it, your LAN would obliterate radio transmissions for quite a radius. And that's just the energy produced by a little network card, intended to go a few hundred feet through cable.
Now consider DSL and cable modems. Each of these technologies uses modulation to shift the baseband data up into another part of the radio spectrum. DSL uses the same twisted pair that carries POTS, but since voice only uses about 4KHz of bandwidth, they put the data starting just above 4KHz, and let it run up to about 2.2MHz. (Different DSL flavors split the spectrum differently between upstream and downstream directions) Sending a balanced signal down twisted pair also keeps the noise in check, so you can run several DSL circuits in the same binder and they don't interfere with each other very much. Cable modems do a similar technique, but since their cable is already carrying TV signals, they use different bits of the spectrum for it. The trick here is that the cable network is a bus architecture, and each modem's upstream transmissions have to be strong enough to be heard at the head end. This means that careful shielding is critical. A noisy cable amplifier can splatter interference all over, causing problems with any number of radio systems.
Providing a useful data rate to customers means generating a lot of RF energy, and then keeping it contained to minimize interference. Cable and DSL have their work cut out for them, simply maintain the physical network and the signal stays in the wires where it belongs. Wireless ISPs do things differently, using antennae to carefully focus their transmissions, and occasionally using filters to clean up the edges of their spectrum where the equipment might be a bit too noisy.
Broadband over powerlines does just the opposite. BPL uses a modulated signal like you might find in DSL or cable, but it blasts the signal onto a wiring system that can't possibly contain the noise. Because of the extremely poor match between the signal and the wiring, the power levels are very high just to get a useful signal at the receiver. The power lost in the middle is all noise, and it goes everywhere. The power grid acts as an antenna. There's no way to make it work cleanly and efficiently, short of replacing all the powerlines with coax or twisted pair, and at that point it won't be useful as a power grid anymore.
BPL is the wrong thing to do with data, and it's the wrong thing to do with powerlines. It happens to be the only way the electric utility can compete in the data arena, and I don't really blame them for trying. If competition for cable and DSL service were meaningful, the power grid might not be the avenue of last resort. That's a problem with regulation and monopolies, broader than I want to get into right now. The fact remains that BPL is destructive to radio spectrum and a bad idea in general. If you understand this mess, it's your responsibility to oppose BPL with every resource available to you.
-Myself-
]]>
I borrowed this image from here
]]>It has come to my attention that there is some confusion regarding what our CSA for Voice Applications protects against regarding the recent Sasser worm.
Here is what we know so far:
- our policies do prevent sasser from copying itself.
- our policies do prevent sasser from executing.
- our policies do not prevent sasser from crashing LSASS.EXE with a buffer overrun, therefore creating a denial of service attack risk since this crash causes the machine to reboot.
The CSA for Voice Applications policies do include the default bufffer overrun rule from the CSA product, but apparently this was not effective in this case. We are still working with VsecBU on this and will keep you posted once we know more.
In the meantime, even though our custumers that have CSA properly running on their voice application servers will not get infected with Sasser, they need to patch their systems to prevent them from rebooting once/if the worm makes it into the network those servers are directly connected to (firewalls are effective at stoping sasser). We don't know exactly at what rate they are bound to see the DOS attack occur, but we do know it can happen.
Included below you will find instructions on patching customer's servers as per the the announcement posted on customer-ccm-announce@cisco.com on Monday titled:
"Fixes for Sasser Virus (active exploit of MS04-011) for Cisco CallManager and Unity":
Callmanager, CER, PA, etc (non-Unity):
MS04-011 is in 2000.2.5sr7 and sr8 or 2000.2.6 posted on Cisco.com:
http://www.cisco.com/cgi-bin/tablebuild.pl/cmva-3des
For CallManager, please install the win-OS update and do not apply the patch from Microsoft.
Unity:
Since this is not a service pack, you can install the MS04-011 patch directly to a Unity server.
http://www.microsoft.com/technet/security/bulletin/MS04-011.mspx
Microsoft's Official Page:
http://www.microsoft.com/security/incident/sasser.asp
This page also includes a link to run a program to remove the virus AFTER the patch has been installed.
More information on the virus:
http://us.mcafee.com/virusInfo/default.asp?id=description&virus_k=125008
http://securityresponse.symantec.com/avcenter/venc/data/w32.sasser.worm.html
http://www.trendmicro.com/vinfo/virusencyclo/default5.asp?VName=WORM_SASSER.A
There are at least 3 variants A, AB, and B.
Please let me know if you have further concerns regarding the effect of sasser on our voice application servers.
]]>Why another blog? I have yet to find something out there the is vendor neutral and talks in a real world way about networking.
]]>