Networking

SSH key error on Cisco devices

2011-04-19T11:46:00.004+03:00

I'm using Rancid's clogin in a script to periodically retrieve the configs from Cisco switches/firewalls. On one of the switches I have a problem connecting with clogin via ssh. Even though I generated a key larger than 768 bits when I try to connect it gives this error:

ssh -c 3des -x -l user 192.168.1.1
ssh_rsa_verify: RSA modulus too small: 512 < minimum 768 bits
key_verify failed for server_host_key

Error: Couldn't login: 192.168.1.1

Normally this would mean that the key on the device is smaller that 768 bits. To generate a key on an IOS (in this case a Catalyst switch) you would use:

crypto key generate rsa general-keys modulus 1024

Like I said, even though I have a 1024-bit key, openssh was unable to connect, giving the error above. One solution which seems to solve this problem would be to force SSH globally on the system to use version 1 for the specific host with the problem. To do this, create a ~/.ssh/config file:

Host 192.168.1.1
Protocol 1

Also, clogin gives you the possibility to replace the ssh command using a statement in ~/.cloginrc:

add sshcmd * /path/ssh_replacement

You could put in this ssh_replacement something like (-1 to use SSH v1):

ssh -1 $1 $2 $3 $4 $5 $6

ESXi 4.1 and PS/2 keyboards

2010-11-30T11:48:00.007+02:00

I've recently upgraded/migrated our ESX 4.0 servers to ESXi 4.1, as VMware stated that 4.1 will be the last ESX release and recommended the migration to ESXi.
The migration itself didn't pose any problem as there isn't a migration process per se. You just have to migrate your VMs to another host and install the ESXi hypervisor. All went well until I had to press 'Enter' to install. It just didn't work.
It seems that ESXi 4.1 has a problem with (some?) PS/2 keyboards on (some?) servers. I tried the install CD on 3 different models of Fujitsu-Siemens servers and it didn't work on eighter one of them.
The solution was to use an USB keyboard for the install. It still remains an inconvenience as we have all the 30-40 servers (still, only 3 ESX :) connected through KVMs with PS/2 keyboard connectors.

LATER EDIT:
Thanks to Valentin Hristev (see comment below), there is solution to use the PS/2 keyboard:

When you boot your ESXi 4.1 and see blue screen fast press TAB to edit boot options
then edit with "acpi=off" it should look like this:
mboot.c32 vmkboot.gz acpi=off --- vmkernel.gz --- sys.vgz --- cim.vgz --- ienviron.vgz --- install.vgz

Dell MD3000i slow write speed - cache deactivated

2010-11-24T09:13:00.008+02:00

(credits for this article to vmtoday.com, for an article about a similar IBM array, with slightly different commands)
I stumbled across a problem earlier this year with PowerVault MD3000i SAN array and didn't find a specific resolution for this storage, so I though I write about it here so you can solve it easier.
When you buy the MD3000i with only one controller, it deactivates the write cache, becouse the write mirroring feature is enabled (which requires two controllers). The result is that the writing speed is very low and the latency is high.
To activate the cache you need to use the CLI tool provided with the Modular Disk Storage Manager software, which you can find here: C:\Program Files\Dell\MD Storage Manager\client\SMcli.exe. You can first locate the virtual disks that have the cache suspended (Array0 is the name of my array, you should change it to whatever yours is named)

smcli.exe -n Array0 -c "show allVirtualDisks;"
... Write cache: Enabled (currently suspended)
... Write cache with mirroring: Enabled

To disable the mirroring feature and enable caching:

smcli.exe -n Array0 -c "set allVirtualDisks mirrorEnabled=false;"
smcli.exe -n Array0 -c "set allVirtualDisks writeCacheEnabled=true;"

Later edit:

smcli.exe -n Array0 -c "set allVirtualDisks cacheWithoutBatteryEnabled=true;"

vSphere - Sluggish mouse on Windows 7 and 2008 Server

2010-03-12T12:32:00.006+02:00

I had this issue after installing Windows 2008 R2 on VMWare ESX 4.0 (208111). Even though I installed VMWare tools, the mouse was still acting as if the driver wasn't installed - the mouse is jumpy and is not leaving the console after the "real" mouse is exiting the screen.
It seems that VMWare is aware of this issue and will solve it eventually.
Anyway, this is the quick fix:

install VMWare Tools in your machine, reboot;
VMWare tools has created the folder: C:\Program Files\Common Files\VMware\Drivers\wddm_video;
go to Device Manager, to your video adapter and select "Update driver";
when asked for the files, browse to the folder above;
reboot;
problem fixed!

LE: after every VMWare tools update you need to redo these steps.

Integrating VMWare ESX into your enterprise network with VLANs, trunking, load balancing

2010-03-09T03:12:00.010+02:00

VMWare ESX 3 and VSphere can fit nicely into your enterprise network, adding unprecedented flexibility and functionality; after you've got the taste you'll be hooked on it. Virtual machines can be moved from one VLAN to another with a few cliks.
First of all, about load balancing... If you have VSphere installed on an enterprise server then you most likely have 2 gigabit network interfaces at your disposal. These NICs can be "teamed" for load balancing and failover purposes. As you will see further in the article, load balancing will work even if the swith doesn't support load balancing, but it will only work for outgoing traffic. To make the most of load balancing you need to use a switch that supports etherchannel; in this way the switch will send traffic to the server on both ports, based on the load balancing scheme we configure. In this example we'll be using the Cisco Catalyst 2960G switch.
You can find network configuration of the ESX host on tab Configuration - section Hardware - Networking. You'll probably use vSwitch0 to configure your network interfaces and VLANs. At first you'll have only one NIC added to the vSwitch. You need to add the second interface. Go to the properties of vSwitch0, go to Network Adapters and click on Add to add the other interface; this is how vSwtich0 will look like aftewards:

Go back to Ports on the tab above to configure load balancing. Edit the vSwitch device and go to NIC teaming. In this example we'll be using load balancing based on IP hash. So configure "Route based on ip hash" in the Load balancing section:

Next step is to configure the switch. In this first phase we assume that you're running the ESX server in access mode (no VLAN trunking), in VLAN10. Identify the ports in the switch that belong to the ESX server and configure the port channel (we'll assume that you don't have any other port channels configured):

sw(config)#interface range g0/1 - 2
sw(config-if-range)#switchport mode access
sw(config-if-range)#switchport access vlan 10
sw(config-if-range)#channel-group 1 mode on

Now the communication with the ESX should work; if not, take the ports out of the port channel and fix your configuration.
If everything is ok now, we can prepare the ESX server for VLAN trunking. We need to put the Service Console interface on the desired VLAN - for example, VLAN 10: go to the properties of vSwitch0 and edit the Service Console; click on "Continue modifying the connection" to pass that warning; type your VLAN ID in the next window:

At this point you will lose connectivity to the ESX server. To restore it, configure VLAN trunking on the switch:

sw(config)#interface port-channel 1
sw(config-if)#switchport mode trunk
sw(config)#interface range g0/1 - 2
sw(config-if-range)#switchport mode trunk

If you have VMotion or iSCSI or NFS volumes configured, then you need to configure also the VMkernel VLAN, in the same way you configured the Service Console.
In case something goes wrong and you need to troubleshoot you configuration:

you can check the etherchannel load balancing on the switch:

sw#show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC addressIPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address

you can log in to the ESX server console and, for example, configure the VLAN ID of the service console:

[root@server root]# esxcfg-vswitch -v -p “Service Console” vSwitch0

To add a VLAN for your virtual machines go to vSwtich0 properties and "Add...", choose connection type "Virtual machine", type your VLAN name for easy identification and the VLAN ID:

Then, moving the virtual machines from one VLAN to another is simple: edit settings on the desired virtual machine and select the network adapter; on the network label drop-down list you will get the VLAN you configured.

Adding space to ext3/LVM partitions without losing data

2009-03-26T12:40:00.001+02:00

I've recently ran out of space on a server and decided to write a tutorial about this. There are some resources out there but this is a more complete guide of doing this. Mostly I wrote it for me to have a procedure in the future.
Let's assume we add a new disk on the system and we want to increase the space on the existing partitions without having to delete and recreate them. This would be a nighmare when you have partitions streching on 2-3 1TB disks - lost time and also you need to have the space on some other disk to copy the files.
First of all, add the new disk on the machine. Best would be to boot a live CD (e.g. Knoppix) so that we can make the lvm operations "offline". Create a new partition on the disk using fdisk of type "8e" - Linux LVM.
Next, create the physical volume within the new partition. We'll take as example partition /dev/sde1:

# pvcreate /dev/sde1

Extend the existing volume with the newly created physical volume:

# vgextend VolGroup00 /dev/sde1

Next, extend the logical volume to use the space made available by extending the volume group:

# lvextend -L 1000G /dev/VolGroup00/LogVol00

The final step is to enlarge the ext3 partition to fill up the free space. First you'll have to run a check on the filesyste. These operations will take some time, proportional with the size of the partition:

# e2fsck -f /dev/VolGroup00/LogVol00
# resize2fs /dev/VolGroup00/LogVol00

Routing subnets on PIX/ASA interface

2008-11-27T10:29:00.006+02:00

Generally is not a good ideea to have your firewall do the job of other devices like routers. However there are situations where is not feasible to invest in a router only to do a small task.
The PIX/ASA does not support the secondary ip address on their interfaces. There is a workaround:

add a static ARP entry so that your firewall replies to ARP requests; use the MAC of the respective interface:
arp interface 192.168.2.1 1234.5678.90ab alias
add a route to your network (in this example 192.168.1.1 is the IP of the interface):
route interface 192.168.2.0 255.255.255.0 192.168.1.1

Securing Cisco LANs

2008-11-05T08:39:00.030+02:00

Security at Layer 2 is often neglected. Yet, having an open data link layer is like sending an invitation to everybody to hack into your network. Here are some simple "best practice" measures (also some not related to Layer 2) you can take so that attacks don't do damage to your network.

Disable Telnet and enable SSH; secure the management protocol
Telnet is a management protocol that should never be used on production environments. It's main weakness is that it's not encrypted. Someone sniffing your traffic could gain knowledge about your credentials and network configuration.
Example for doing this on a Catalyst switch running IOS:

router(config)# hostname router
router(config)# ip domain-name domain
router(config)# aaa new-model
router(config)# username cisco password cisco
router(config)# crypto key generate rsa
router(config)# access-list 10 permit 192.168.23.45 # the IP of your management console
router(config)# line vty 0 4
router(config-line)# transport input ssh
router(config-line)# access-class 10 in

This also permits only your management station to login to the switch (access-list 10).
Don't use other unencrypted protocols eighter, like SNMP v1 or v2, FTP of TFTP.

Software updates
Keep your switch/router/firewall software updated. In every update security holes are resolved. IOS images or ASA/PIX software can be found in http://www.cisco.com/kobayashi/sw-center/index.shtml. You need a CCO account though; this isn't a problem if your organisation is a Cisco customer.

Don't use VLAN 1 for any purpose
VLAN 1 is the VLAN where switch ports are asigned by default, protocols like CDP, VTP and PAgP where designed to communicate. That's why sometimes VLAN 1 spans across the whole network if not carefully pruned. A few recommendations:

don't use VLAN for regular traffic. Prune VLAN 1 (including shutdown and not connected ports):
switchport trunk pruning vlan {add | except | none | remove} vlan-list [,vlan[,vlan[,,,]]
don't use VLAN for inband management; dedicate another VLAN for management. Apply ACLs to this VLAN to allow SSH traffic only from the administrator's management systems;
don't configure the management VLAN on ports that don't require it;
the most secure, yet expensive, is out-of-band management. Deploy it where it's worth;
use a VLAN other than VLAN 1 as native vlan. Set all your trunk ports to this native VLAN. This also mitigates VLAN hopping attacks:

switchport trunk native vlan vlan-id

Disable unused services that can favour attacks
Disable Dynamic Trunking Protocol (DTP). If enabled can permit VLAN hopping attacks:

switch(config-if)# switchport nonegotiate

Disable Cisco Discovery Protocol (CDP). Besides the information this could give to an attacker there were some IOS versions that had a vulnerability that allowed Cisco devices to run out of memory or even crash. Here's how to disable it:

switch(config-if)# no cdp enable

Turn off VLAN Trunking Protocol (VTP) if not used. If you must use it use MD5 authentication to protect the message exchange between switches. To turn it off put it in transparent mode on all switches:

switch(config)# vtp mode transparent

To set the VTP password:

switch(config)# vtp password your_password

Use port security
In ARP attacks the intruder obtains IP addresses and other information and then uses it to issue the attack. Using port security you are able to specify the number of MAC addresses or the specific MAC address allowed to connect through the port. Enabling port security and example:

switch(config)# set port security port enable
switch(config)# switchport port-security maximum 5
switch(config)# switchport port-security aging time 2
switch(config)# switchport port-security aging type inactivity

This prenvents also CAM table attacks.
As the functionality/security ratio drops significantly you may want to use this only on server ports.

Use DHCP snooping
The easiest way of doing a Denial Of Service attack on a LAN is to activate a DHCP server that intercepts legitimate user's DHCP requests. In most cases though this happens when an inocent user mistakingly activates the DHCP server on his host. DHCP snooping is a feature that filters untrusted DHCP messages and builds a DHCP snooping binding table.
You can find out how to configure it here: http://crazyvlan.blogspot.com/2008/03/protecting-your-network-against-rogue.html

Dynamic ARP inspection
A client is allowed to send an unsolicited ARP reply thus poisoning the ARP tables of the routers. The information is stored by other hosts in the subnet. An attacker could use this behaviour to "hijack" the traffic. The countermeasure to this attack is to use dynamic ARP inspection.
Dynamic ARP inspection uses the DHCP snooping binding table to allow only those IP-MAC pairs found in the table. Of course you must configure first DHCP snooping otherwise there would be no binding table to use.
To enable ARP inspection:

switch(config)# ip arp inspection vlan vlan-range

Trust interfaces connected to other switches:

switch(config)# ip arp inspection trust

Mitigate spoofing attacks - IP source guard
If IP source guard is activated the switch doesn't learn the MAC address from the device, but from the DHCP snooping binding table. It operates like dynamic ARP inspection but it looks at every packet not just ARP packets.
To activate IP source guard with source IP address filtering:

switch(config-if)# ip verify source

To activate IP source guard with source IP and MAC filtering:

switch(config-if)# ip verify source port-security

STP attack mitigation - BPDU Guard, Root Guard
BPDU guard - disables ports using portfast upon detection of a BPDU message on the port:

switch(config-if)# spanning-tree bpduguard enable

Root guard - disables ports who would become the root bridge due to their BPDU
advertisement

switch(config-if)# spanning-tree guard root

BPDU filter - ports configured with BPDU filter will not send BPDUs and drop all received BDPUs:

switch(config-if)# spanning-tree bpdufilter enable

Preventing broadcast storms
A broadcast storm occurs when broadcast or multicast packets flood a subnet degrading network performance. Enabling broadcast supression:

switch(config-if)# broadcast suppression threshold

Implement 802.1x
802.1x provides an authentication system for devices wishing to attach to a LAN port. Basically only authenticated systems can access the network; the authentication could be even Windows domain credentials. I won't detail it here; this would be the subject of a later article.

Filtering routing protocol updates
Routing protocol updates towards the LAN make no sense as you don't have any routers requiring them. Only hackers could make use of them gaining knowledge of your network. This can be activated on all interfaces by default and disabled only on those interfaces that need it. Here's an example of OSPF:

router(config-router)#passive-interface default
router(config-router)#no passive-interface fastEthernet 0/2

With this only interface f0/2 will exchange routing updates.

If you think of other attacks and measures to mitigate them please post it and I can update the article with new stuff.

Multiple-IP static NAT/port mapping with PIX/ASA

2008-11-01T16:29:00.015+02:00

This is the scenario:

When there is a need of mapping an inside IP address to an outside IP we can do static NAT; you can also do only a port redirection. Here's how it's done in ASDM:

But what there is to be done when you have multiple inside IP's that need to be mapped to multiple addresses on the outside? Sure, you can choose a port forward for every inside host, but sometimes this is not enough - the hosts need to have outside "correspondents". ASA/PIX doesn't support adding multiple IPs on the interfaces ("secondary", like you would do on a router). A solution to this is to add a static ARP entry:

Now you can add your new IP for static NAT:

Speedy sh run

2008-10-31T10:36:00.006+02:00

Did you ever had to wait 10 seconds until your router config shows up after "show running-config"? Cisco enhaced your routing configuration experience by introducing a command that caches the configuration in memory:

Router(config)# parser config cache interface

After issuing this you have to do a "sh run" to cache the config.
This works on devices with the 12.2 and 12.3 release.

More info:

http://www.cisco.com/en/US/docs/ios/12_3t/12_3t7/feature/guide/gtinvgen.html

QoS on PIX/ASA - policing, traffic shaping, priority queuing

2008-10-30T14:45:00.005+02:00

I'll try not to make a habit by posting someone else's articles but this is really intreresting and useful:

QoS on the PIX/ASA - Part 1:What Tools are Available?

QoS on the PIX/ASA – Part 2:The Modular Policy Framework

QoS on the PIX/ASA – Part 3:Priority Queuing

QoS on the PIX/ASA – Part 4:Traffic Shaping and Traffic Policing

DHCP relay on ASA and DHCP snooping on Catalyst problem

2008-10-18T13:18:00.008+03:00

I ran into a problem that took me 2 hours to figure out and thought I'd share it :)
I'm running DHCP snooping on my network with Catalyst 2950/2960 switches and routed all VLANs through a 3550. My DHCP server is a Windows 2003.
For objective reasons I decided to route a VLAN through the firewall (ASA 5510). The problem occured while trying to obtain an IP from the DHCP for a device in that VLAN. All I got debugging ASA was (123.456.789.123 is my DHCP server):

dhcpd_forward_request: request from abcd.acbd.dabc forwarded to 123.456.789.123.
DHCPD: setting giaddr to 192.168.0.1.

When debugging DHCP snooping on one of the switches my eye cought this:

31w1d: DHCP_SNOOPING_SW: Encoding opt82 CID in vlan-mod-port format
31w1d: DHCP_SNOOPING_SW: Encoding opt82 RID in MAC address format

I disabled option 82 on the switch as I didn't need it anyway...

no ip dhcp snooping information option

...and bingo:

dhcpd_forward_request: request from abcd.acbd.dabc forwarded to 123.456.789.123.
dhcp_l3_punt_cb: pkt src 123.456.789.123/17152, dest 192.168.0.1/17152
DHCPD/RA: Punt 123.456.789.123/17152 --> 192.168.0.1/17152 to CP
DHCPRA: Received a BOOTREPLY from interface 1
DHCPRA: relay binding found for client abcd.acbd.dabc.
DHCPRA: Adding rule to allow client to respond using offered address 192.168.0.53
DHCPRA: forwarding reply to client abcd.acbd.dabc.
DHCPRA: relay binding found for client abcd.acbd.dabc.

Maybe I'll edit later the article including the explanation but I can't do it now as I have a lot of work ahead of me today.
Cheers!

Rate My Network Diagram

2008-05-05T17:16:00.003+03:00

This is a very interesting site, not so much for admiring other people's network diagram for their "beauty" but for learning how other people think, design and document their networks:
http://www.ratemynetworkdiagram.com

Obtaining list of IP addresses from a subnet

2008-03-19T16:26:00.006+02:00

I made a script today and I needed to get a list of IP addresses from some subnets. The solution to do this is nmap:

nmap -sL -n $subnet

gives you a lists all the IPs contained by the $subnet subnet.
To actually get a list you can parse the output:

nmap -sL -n 10.1.1.0/29 | grep "not scanned" | column -t | cut -d ' ' -f 3

10.1.1.0
10.1.1.1
10.1.1.2
10.1.1.3
10.1.1.4
10.1.1.5
10.1.1.6
10.1.1.7

GNS3 - Graphical frontend for Cisco emulation

2008-03-18T09:40:00.005+02:00

http://www.gns3.net
This is a frontend for dynamips and dynagen. It makes life much easier as it allows graphical design and simulation of your network. It's a very good tool when you study for CCNA or CCNP - routing mainly. The developers say they will incorporate PIX support in future releases.
Watch out for high CPU loads. There's a tutorial about how to configure it so that the CPU levels stay at a reasonable level - in the documentation PDF.
Although current release is pretty buggy it's much easier than working with dynagen's config files.

Live monitoring of your network with syslog and email

2008-03-14T14:10:00.007+02:00

If you're a control freak (like I am) here's a little tip for having live monitoring of what's happening on your network: have your switches/routers send syslog messages to a Linux box, filter those messages that don't interest you then send it to yourself via email. Here's how it's done...
First, point the device to the desired syslog server and make sure that you also use a ntp server to provide the correct time:

logging trap debugging
logging 10.0.0.1

Next, on the Linux box enable syslogd to accept remote logging: open the file /etc/init.d/sysklogd and make the change:

SYSLOGD="-r"

Restart sysklogd:

/etc/init.d/sysklogd restart

Add the following to /etc/syslog.conf:

local7.debug /var/log/network.log

If you're you're using iptables-based firewall add the following line to you firewall script to open port 514/UDP:

$ipt -A INPUT -p udp --dport 514 -j ACCEPT

Next test the config so far by watching the /var/log/network.log file for incoming messages (make sure there are any - log on to the device, make some config changes, save etc.):

tail -f /var/log/network.log

Next we need to do a script that parses this config. For example:

#!/bin/sh

MAILPL=/usr/local/sysm/mail_syslog.sh
TEMPF=/usr/local/sysm/out_syslog
EMAILS_SYSLOG=/usr/local/sysm/emails_syslog

pastminute=`date --date "1 minute ago" +%b' '%e' '%H:%M`
cat /var/log/network.log | grep "$pastminute" > $TEMPF
while read line
do
if echo $line | grep LINK-3-UPDOWN || echo $line | grep LINEPROTO-5-UPDOWN
then
else
for email in $( cat $EMAILS_SYSLOG )
do
$MAILPL $email "$line"
done
fi
done < $TEMPF

File TEMPF file contains the syslog lines from previous minute and it's generated at each execution. You should have a file, emails_syslog, containing the email addresses to which you want to send the emails.
The above script executes another script, mail_syslog.sh which sends the emails using msmtp (available on Debian: apt-get install msmtp):

#!/bin/sh

email=$1
text=$2
textfile="/usr/local/sysm/text"
date=`date`

echo "From: me@mycompany.com" > $textfile
echo "To: $email" >> $textfile
echo "Subject: syslog is saying" >> $textfile
echo "Priority: Urgent" >> $textfile
echo "Importance: high" >> $textfile
echo "" >> $textfile
echo $text >> $textfile
echo "" >> $textfile
echo "" >> $textfile
echo "--" >> $textfile
echo "Report generated at $date" >> $textfile

cat $textfile | msmtp --host smtp.mycompany.com $email -f me@mycompany.com

I know, the scripts aren't exactly top of the line but they do the job :)
Needless to day that you need to have your time correct on the syslog server if want to use the syslog data later.

Protecting your network against rogue DHCP servers

2008-03-13T11:34:00.006+02:00

If you haven't done it yet you have a ticking bomb on your network. Possible "attackers" are: an unknowing user playing with "test" installation of some freeware software that bundles generously a DHCP server, a bug in a program (no offence to the developers, but I had this problem several times with VMWare which gave IP addresses in the "real" LAN in certain configurations) or even an attacker trying to bring down your network at least for a while or, worse, do a man-in-the-middle attack substituting the real default gateway with its own, eavesdropping on your traffic.
The Cisco solution to this is called DHCP snooping. This feature filters untrusted DHCP traffic and also builds a DHCP snooping binding table.
To use DHCP snooping on the Catalyst switch you must enable it globally:

ip dhcp snooping

To enable DHCP snooping on your VLANs:

ip dhcp snooping vlan number

From the DHCP snooping point of view there are 2 kinds of ports: trusted and untrusted. Trusted ports are the ones that should permit DHCP messages to pass. These are ports that are connected to legitimate DHCP servers and -very important - trunk ports, becouse they should carry DHCP traffic to DHCPD servers and clients. Untrusted ports are the rest of the ports, normal clients that act as DHCP clients.
Trusted ports should have:

ip dhcp snooping trust

Untrusted ports don't need any config option.
Additional, you can configure a limit on the DHCP messages rate (per second) that can be sent on the port - usually untrusted ports. It's not recommended to configure a limit higher that 100 on untrusted ports:

ip dhcp snooping limit rate 100

To enable insertion of DHCP Option 82:

ip dhcp snooping information option

A common caveat in configuring DHCP snooping is the situation when you have multiple VLANs and one DHCP server and you use ip helper-address. To pass trusted DHCP messages you must configure on the routed interface (e.g. VLAN interface):

ip dhcp relay information trusted

To see the DHCP snooping binding table (the connection between MAC addresses, IPs, lease time and port) you can issue:

sh ip dhcp snooping binding

Another security use for DHCP snooping (if it's activated) is to do Dynamic ARP Inspection which is using the DHCP snooping binding table to discard ARP packets with invalid IP to MAC bindings in an attempt to stop man-in-the-middle attacks.

NEWS - Adventnet Firewall Analyzer update - SP5

2008-03-10T16:17:00.009+02:00

I would like to take this opportunity to introduce to you (if you didn't knew it already) Firewall Analyzer from Adventnet. It's log analysis software that collects syslog messages and makes reports from firewalls, VPN concentrators IDS and proxy servers.
I use it monitoring ASA firewalls. It can create detailed reports about bandwidth usage, protocol or IP usage, it can send configurable alerts about events on the firewall, reports about network security incidents.
One of the nicest features that comes with this update is the possibility to search in raw logs. Here are the rest of the new features:

New devices supported
3Com
Juniper IDP
Cisco IOS
New log formats supported
i-Filter
Free BSD
Bluecoat Proxy
New reports supported for the following devices
Unused rules report for Cisco, Fortigate, and Netscreen devices.
Remote Client VPN support for Netscreen device.
Admin Reports support for Checkpoint device.
Live Reports support for Cisco IOS device.
VPN Live Reports for all VPN devices.
VPN Trend Reports for all VPN devices.
OpManager integration with Firewall Analyzer.
Configuring Data retention policies for database and archived logs.
Alert administration
Assign owner
Add/View Note options for Alerts
Alert History view
Delete alert
Script/Batch file or Command execution on triggering of Alert.
Triggering alert for Firewall non-availability.
Imported Log Files enhancements
Bulk deletion
Sorting log files
Bulk deletion of archieved files.
Raw log search in Advanced Search option.
Searching Archieve log data.
SysLog Viewer enhancements
Search facility
Configuring display columns
Security reports drill down to the depth of raw SysLog view.
View SysLog available in Security tab.
Provision to drop only the log data and retain the cinfiguration data in the database.
Dynamic file name support for log file import through FTP.
Provision for automatic DNS resolution.
Localization supported for reports exported to PDF format.
Bug Fixes
Delete and readd Report Profile with the same name.
Disabled Simulate feature for Operator and Guest users.
Notifying last data availability date in the home page.
Exporting only the selected report to PDF format instead of all the reports of a resource.
Creation of archive directory with DNS name instead of IP address.
Archieving structure divided into Hot, Warm, and Cold, to keep with the industry requirements.
Resolve DNS is slow and not resolving all IP Addresses issue has been fixed.
Disabled scheduled FTP import tasks are started after restarting the FWA issue has been fixed.
PDF Generation issues for Custom Reports with 3, 4, and 5 columns.
A number of bug fixes for perfomance enhancement.

http://manageengine.adventnet.com/products/firewall

Using ASA/PIX with policy NAT to map an inside host

2008-03-10T14:10:00.007+02:00

Possible scenarios when using policy NAT would be:

a LAN has private IP addresses that connect to the Internet using NAT overload; one host from this LAN needs to be accessed from outside with a public IP address only from certain hosts
a branch of an organization has a testbed with IP addresses that are routed only in the local network. In order to give access from other hosts in the intranet to a host in the testbed a PIX/ASA firewall is used to map the private IP address of the testbed host to a enterprise-wide routed IP that will be accessed only from authorised IPs.

We'll be tackling the latter situation. Here's the network diagram of our example:

Networks behind R are: a enterprise-routed VLAN (10.100.1.0/24) and a testbed VLAN (192.168.0.0/24) which only has static routes on R and FW. The desire is that host H2 could be accessed from the outside of the FW, only from 10.200.1.2. The IP address that will be mapped on the FW should be from the 10.100.1.0/24 subnet.
This can be accomplished using policy NATting on the FW firewall. In order to do this we need to create an ACL that will filter only specified IP pairs that will trigger NAT. The translation policy should look like this:

access-list acl_name permit ip real_ip real_mask foreign_ip foreign_mask

To keep the ACL manageable we'll use on object-group that contains the foreign_ip:

object-group network mapping
description List of IPs that can do policy NAT
network-object host 10.200.1.2

The ACL in our case is:

access-list policynat extended permit ip host 192.168.0.2 object-group mapping

Finally, to do the mapping we'll use a static translation:

static (inside,outside) 10.100.1.3 access-list policynat

10.100.1.3 is the mapped IP address. To test the configuration ping 10.100.1.3 from 10.200.1.2. The result is that FW will translate 10.100.1.3 to 192.168.0.2 which takes the route through R and then H2.
To troubleshoot:

ciscoasa# sh nat

NAT policies on Interface inside:
match ip inside host 192.168.0.2 outside host 10.200.1.2
static translation to 10.100.1.3
translate_hits = 10, untranslate_hits = 3

untranslate_hits are traffic from IPs other than 10.200.1.2.

Debian mirror through HTTP proxy

2008-03-05T16:03:00.011+02:00

Ever wanted to put together a Linux Debian mirror in an intranet with no direct access to Internet? The classical solution is, of course, rsync but what do you do when you can't access the source directly and your only way out is a HTTP proxy? The solution is debmirror. To install it: apt-get install debmirror.
First make sure that you exported your proxy server:

export http_proxy=http://proxy.localnet:port

Then set up a script that runs periodically (daily, for example):

debmirror -a i386 -m -h ftp.de.debian.org -e http -d stable -d testing -d unstable --root /debian --getcontents --progress -v /mirror/debian

This is pulling the stable, testing and unstable from ftp.de.debian.org - of course you should choose whatever mirror is closer to you. If you're having problems with the PGP signatures you can add --ignore-release-gpg.
You can log the operation by adding:

1> /log/mirror-debian-repository.out.log 2> /log/mirror-debian-repository.err.log

Cisco-centric Open Source Exchange Community - COSI

2008-03-03T21:18:00.007+02:00

http://cosi-nms.sourceforge.net/
I stumbled across this site many times but since I'm posting on this blog since last week I thought I should mention it now.
The site is a community that develops open source software for managing Cisco equipment. It collects links to many Cisco-related open source projects. This is a very useful resource as "real" tools come with a high price and actually you can't buy many of them.
I use myself VMPS-SRV, a web interface for managing VMPS. This tool (well, slightly modified) can be used in conjunction with an open-source VMPS server (which actually isn't listed in this site :). Maybe this will make the subject of another article, in spite of the fact that the current trend is to use 802.1x. However, if anyone is still interested in using VMPS please let me know - add a comment. In some environments maybe it's still the good choice.

Time-based ACL on ASA firewall

2008-03-03T16:50:00.007+02:00

I had a request to limit the bandwidth of some hosts in a LAN only on the business hours. Not knowing that time-based ACLs are supported I decided to do that without the time component on the local ASA 5510 firewall located at the border of the network. After doing that something caught my eye browsing trough the ASDM:

The "Time ranges" object. This has a very flexible definition: it has 2 layers; the first layer defines the start time and the end time (for example: the time range of the second layer will begin on 03.03.2008, 2100 hours and end on 05.03.2008, 0800 hours); the second second layer is weekly-based - eighter select the days of the week with a hourly interval or select a weekly interval when this range will be active.
For example, if you want to define the range of working hours (9:00 to 17:00) you can do it like this:

time-range working_hours
periodic weekdays 9:00 to 17:00

Now back to the starting case, limiting some hosts during business hours:

define the time range - done above;
define a ACL with the IPs of the hosts, for example:

access-list limit extended permit ip host 10.0.0.1 any time-range working_hours
access-list limit extended permit ip any host 10.0.0.1 time-range working_hours

make a class-map:

class-map class
match access-list limit

write a policy-map for policing at 2Mbps (in this example):

policy-map outside-policy
class class
police input 2000000 16000
police output 2000000 16000
service-policy outside-policy interface outside

Or you can do it with ASDM :D

Monitoring IP segment usage with MRTG

2008-02-29T17:14:00.007+02:00

Here's a little ready-made script for graphing the usage of a subnet or a VLAN with multiple subnets.
We'll be using nmap for quickly surveying the subnets and MRTG for creating graphs like this:

First, the script:

nets="192.168.200.128/25 192.168.200.0/26"
nrips=188

for net in $nets
do
nrup=`nmap -sP $net | grep "Nmap finished" | column -t | cut -d ' ' -f 11 | cut -d '(' -f 2`
done

nrtot=$nrup

proc=`echo "scale=2; $nrtot/$nrips*100" | bc -l | cut -d '.' -f 1`
echo $proc
echo 0

The script doesn't calculate the number of total IPs, so you must do it yourself, in $nrips.
It returns the usage percent and 0 (to suit MRTG).

The MRTG config:

Target[xyz]: `/etc/mrtg/the_script`
MaxBytes[xyz]: 100
Options[xyz]: growright,gauge,noinfo,nopercent
ShortLegend[xyz]: %
Title[xyz]: % IP usage
YLegend[xyz]: Percent
...

And don't forget to schedule the script every 5 minutes in /etc/crontab:

0-59/5 * * * * root /usr/bin/mrtg /etc/mrtg/ip_usage >/dev/null 2>&1

Squid RADIUS authentication

2008-02-29T13:57:00.010+02:00

If you use a RADIUS server for other authentication needs in your organisation why not use it for proxy access? One possible scenario is giving access to web services only to users in specific Active Directory groups.
As a RADIUS server you can use freeRADIUS or Microsoft's IAS server. If you're using the latter you can find info and basic configuration for the IAS server in the first part of this article. Basically you have to declare the client and set up a remote access policy (set Service-Type = Login) and a connection request policy. You also need a security group that will be allowed to validate the remote access policy (you can't specify single users).
I won't describe the installation of Squid, it's not this article's topic.
The authenticating module I used is squid_radius_ath. You can find it here. Download it, unpack it and compile it. You will get a squid_radius_auth executable that you can move to a safe place. It needs a config file, squid_radius_auth that should contain the name of the RADIUS server and the secret:

server radius_server
secret secret_phrase

You need to point Squid to the authenticator so place something like this in squid.conf:

auth_param basic program /path_to_auth/squid_radius_auth
auth_param basic children 5
auth_param basic realm Please enter your domain credentials
auth_param basic credentialsttl 8 hours

Next you have to condition Squid to allow only authenticated users. In the following example users that are in the local LAN are allowed without logging in but users that don't show up in the local users file (localusers) are asked to login:

acl passwd proxy_auth
acl localusers src "/etc/squid/localusers"

http_access allow localusers
http_access allow all passwd
http_access allow all

You'll also have a log of who and when logged on to use the web services on the RADIUS server's logs.

PuTTY Connection Manager

2008-02-29T11:24:00.006+02:00

I discovered a very useful new toy: PuTTY Connection Manager.
It simply provides a tabbing interface with PuTTY consoles.

You still have to use the classic PuTTY for this to work. They recommend the latest - 0.60.

PuTTY Connection Manager - http://puttycm.free.fr/
PuTTY - google it!

Automating SSH and telnet tasks

2008-02-27T14:50:00.008+02:00

This is a short tutorial about automating those annoying tasks that need to be done on a (large) number of routers of switches with the same commands.
I used rancid (http://www.shrubbery.net/rancid) for that purpose. It supports Linux among other operating systems. Those of you that use Debian will have it installed with a simple apt-get install rancid-core.
The tool we'll be using, clogin, is located in /usr/lib/rancid/bin/. It needs a bit of configuration that can be stored in a ~/.cloginrc file. The file should look like this:

add user * {user}
add password * {password} {enable_password}
add method * {ssh}

The method can be ssh or telnet. After this, clogin can be called like: clogin -c "commands" IP.
Here is a simple example script for sending the same commands to a list of Cisco routers/switches:

clogin=/usr/lib/rancid/bin/clogin
switch="192.168.0.1 192.168.0.2 192.168.0.3 192.168.0.4"

$clogin -c "conf t\ncommand1\ncommand2\nend\nwr mem\n\n\n" $switch > logs/log_`date`

That's all there is to it.

VPN and Radius with Cisco ASA and Windows 2003 Server

2008-02-26T10:02:00.023+02:00

Here's an article about integrating the Cisco ASA firewall/VPN concentrator 5500 family with Windows 2003 Server's RADIUS server, called Internet Authentication Server - AIS. What I will describe is setting up a ASA 5500 appliance to do Remote Access VPNs authenticating against a MS IAS RADIUS server; the twist to this setup is differentiating between multiple Windows user groups - linking a Windows AD group to a specific tunnel-group.
I decided to write this article becouse I had to search for too much how to do some of the things described below.

In the first part we'll take care of setting up the 2003 Server.

First of all make sure that if you're deploying RADIUS for a large organization you're using the Enterprise flavor of Windows 2003 Server. It has more extensive capabilities than the Standard edition - see http://www.winsupersite.com/showcase/winserver2003_editions.asp for example.

Second, install IAS on the 2003 Server - it doesn't come installed by default:

In order to use the IAS with a client (in our case, the ASA device) you have to declare the client to IAS, otherwise the server will not answer the queries: enter the IAS management console, right click on the "RADIUS Clients" on the left > New RADIUS Client. Here choose a name for the ASA device; this will be unique and you'll be using it later. Next, as Client-Vendor choose RADIUS standard, and as secret - a phrase that you'll use later in your ASA config to pair with the IAS server.
Next, the server needs a Connection request policy to allow the client to connect: Connection Request Processing > Connection Request Policies > New... Here make a custom policy and as Policy Condition you can use for example "Client-Friendly-Name" and specify the name you chose in the previous step when you declared the client:

The next thing to do is to create a Remote Access Policy, again a custom one.
For this you have to have prepared one or more Windows groups (local or better - AD groups) in which you include the users that can access the VPN.
So, in policy conditions add the Client Friendly name (for example) and the Windows-Groups attribute; here you'll be asked for the group. Next, "Grant remote access permision" and edit the profile. In the Authentication tab choose only "Unencrypted authentication" and in the Encryption tab choose only "No encryption". We'll handle the Advanced tab later.

Now the ASA appliance.
I won't describe in detail how to set up a Remote Access VPN, there are plenty of tutorials and guides for doing this. One would be http://www.cisco.com/warp/public/110/asa-remotevpn-asdm.pdf.
Here's a sample setup of doing this:

aaa-server group1 protocol radius aaa-server group1 host 192.168.1.2 key secret

group1 will be used to authenticate the VPN users against the 192.168.1.2 server (this is the IP of the 2003 Server box with IAS). Before continuing with the setup let's test the RADIUS communication:

ASA# test aaa-server authentication group1 username user password passwd
Server IP Address or name: 192.168.1.2
INFO: Attempting Authentication test to IP address <192.168.1.2> (timeout: 12 seconds)
INFO: Authentication Successful

If the authentication was successful you will get the above message. If not, you need to debug it. On the IAS server side you can check the Event Log in the System category. Every attempt of authentication is logged there is successful or not. Typical pitfalls are misconfigured Remote Access Policies of Connection Request Policies. You will get a "Reason" for the failure.
If you don't even get a Event Log message then you need to check your security device configuration or IP connectivity.
The rest of the configuration:

ip local pool vpn-pool 10.0.1.2 - 10.0.1.255
group-policy testvpn internal
group-policy testvpn attributes
dns-server value 192.168.1.3
vpn-tunnel-protocol IPSec
default-domain value test.local
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto dynamic-map outside_dyn_map 10 set transform-set ESP-3DES-SHA
crypto dynamic-map outside_dyn_map 10 set security-association lifetime seconds 288000
crypto map Outside_map 10 ipsec-isakmp dynamic outside_dyn_map
crypto map Outside_map interface outside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
crypto isakmp nat-traversal 20
tunnel-group testvpn-group type ipsec-ra
tunnel-group testvpn-group general-attributes
address-pool vpn-pool
authentication-server-group group1
default-group-policy testvpn
tunnel-group testvpn-group ipsec-attributes
pre-shared-key *

Please note that is only an example config and your needs may require some other config options.
To complicate matters, if you have multiple VPN tunnel groups then you need to differentiate between users that are allowed to access each tunnel then you need to add a special attribute in the Remote Access Policy: go to the Remote Access Policy you're editing and in the Advanced tab add a Class attribute with a string value of OU=value. The value must match the name of the tunnel group:

On the ASA:

group-policy testvpn attributes
group-lock value testvpn-group

What I couldn't figure out is how to differentiate the VPN users from the management users (console, ASDM etc). If anybody knows please let me know.