Troubleshooting of Linux Issues Part - 2

You can also just use the plain old more command to see one screen at a time of the entire log file without filtering with grep. Here is an example:

# more /var/log/messages

Similar commands can be applied to all log files. This is probably one of the best troubleshooting tools available in Linux. Another good command to use apart from tail is grep. grep will help you search for all occurrences of a string in a log file; you can pipe it through the more command so that you only get one screen at a time. Here is an example:

#grep eth /var/log/messages | more

#grep vga /var/log/messages | more

Certain applications will additionally log to their own application specific log files and directories independent of the syslog.conf file. Here are some common examples:

Files:

/var/log/maillog             : Mail

/var/log/httpd/access_log    : Apache web server page access logs

Directories:

/var/log

/var/log/samba                      : Samba messages

/var/log/mrtg                       : MRTG messages

/var/log/httpd                      : Apache webserver messages

Note: In some older versions of Linux the /etc/rsyslog.conf file was very sensitive to spaces and would recognize only tabs. The use of spaces in the file would cause unpredictable results. Check the formatting of your /etc/rsyslog.conf file to be safe.

Activating Changes to the syslog Configuration File

Changes to /etc/rsyslog.conf will not take effect until you restart syslog.

Managing the syslog daemon is easy to do, but the procedure differs between Linux distributions. Here are some things to keep in mind.

Firstly, different Linux distributions use different daemon management systems. Each system has its own set of commands to do similar operations. The most commonly used daemon management systems are SysV and Systemd.

Secondly, the daemon name needs to be known. In this case the name of the daemon is rsyslog.

Armed with this information you can know how to:

Start your daemons automatically on booting

Stop, start and restart them later on during troubleshooting or when a configuration file change needs to be applied.

For more details on this, please take a look at the "Managing Daemons" section of Chapter 6 "Installing Linux Software"

How to View New Log Entries as They Happen

If you want to get new log entries to scroll on the screen as they occur, then you can use this command:

[root@bigboy tmp]# tail -f /var/log/messages

Logging syslog Messages to a Remote Linux Server

Logging your system messages to a remote server is a good security practice. With all servers logging to a central syslog server, it becomes easier to correlate events across your company. It also makes covering up mistakes or malicious activities harder because the purposeful deletion of log files on a server cannot simultaneously occur on your logging server, especially if you restrict the user access to the logging server.

Configuring the Linux Syslog Server

By default syslog doesn't expect to receive messages from remote clients. Here's how to configure your Linux server to start listening for these messages.

As we saw previously, syslog checks its /etc/rsyslog.conf file to determine the expected names and locations of the log files it should create. It also checks the file /etc/sysconfig/syslog to determine the various modes in which it should operate. Syslog will not listen for remote messages unless the SYSLOGD_OPTIONS variable in this file has a -r included in it as shown below.

# Options to syslogd

# -m 0 disables 'MARK' messages.

# -r enables logging from remote machines

# -x disables DNS lookups on messages received with -r

# See syslogd(8) for more details

 SYSLOGD_OPTIONS="-m 0 -r"

# Options to klogd

# -2 prints all kernel oops messages twice; once for klogd to decode, and

#    once for processing with 'ksymoops'

# -x disables all klogd processing of oops messages entirely

# See klogd(8) for more details

KLOGD_OPTIONS="-2"

Note: In Debian / Ubuntu systems you have to edit the syslog startup script /etc/init.d/sysklogd directly and make the SYSLOGD variable definition become "-r".

# Options for start/restart the daemons

#   For remote UDP logging use SYSLOGD="-r"

#

#SYSLOGD="-u syslog"

SYSLOGD="-r"

You will have to restart syslog on the server for the changes to take effect. The server will now start to listen on UDP port 514, which you can verify using either one of the following netstat command variations.

[root@bigboy tmp]# netstat -a | grep syslog

udp        0      0 *:syslog                *:*

[root@bigboy tmp]# netstat -an | grep 514

udp        0      0 0.0.0.0:514             0.0.0.0:*

[root@bigboy tmp]#

Configuring the Linux Client

The syslog server is now expecting to receive syslog messages. You have to configure your remote Linux client to send messages to it. This is done by editing the /etc/hosts file on the Linux client named smallfry. Here are the steps:

1) Determine the IP address and fully qualified hostname of your remote logging host.

2) Add an entry in the /etc/hosts file in the format:

IP-address    fully-qualified-domain-name    hostname    "loghost"

E

xample:

192.168.1.100    bigboy.my-site.com    bigboy     loghost

Now your /etc/hosts file has a nickname of "loghost" for server bigboy.

3) The next thing you need to do is edit your /etc/rsyslog.conf file to make the syslog messages get sent to your new loghost nickname.

*.debug                                       @loghost

*.debug                                       /var/log/messages

You have now configured all debug messages and higher to be logged to both server bigboy ("loghost") and the local file /var/log/messages. Remember to restart syslog to get the remote logging started.

You can now test to make sure that the syslog server is receiving the messages with a simple test such as restarting the lpd printer daemon and making sure the remote server sees the messages.

Linux Client

[root@smallfry tmp]# systemctl restart lpd.service

Linux Server

[root@bigboy tmp]# tail /var/log/messages

...

...

Apr 11 22:09:35 smallfry lpd: lpd shutdown succeeded

Apr 11 22:09:39 smallfry lpd: lpd startup succeeded

...

...

[root@bigboy tmp]#

Syslog Configuration and Cisco Network Devices

syslog reserves facilities "local0" through "local7" for log messages received from remote servers and network devices. Routers, switches, firewalls and load balancers each logging with a different facility can each have their own log files for easy troubleshooting. Appendix 4 has examples of how to configure syslog to do this with Cisco devices using separate log files for the routers, switches, PIX firewalls, CSS load balancers and LocalDirectors.

Activating logrotate

The above logrotate settings in the previous section will not take effect until you issue the following command:

[root@bigboy tmp]# logrotate -f

If you want logrotate to reload only a specific configuration file, and not all of them, then issue the logrotate command with just that filename as the argument like this:

[root@bigboy tmp]# logrotate -f /etc/logrotate.d/syslog

Installing and Starting syslog-ng

You can install syslog-ng using standard Linux procedures.The syslog-ng and rsyslog packages cannot be installed at the same time. You have to uninstall one in order for the other to work. Here’s how you can install syslog-ng using RPM package files.

1.         Uninstall rsyslog using the rpm command. There are some other RPMs that rely on rsyslog so you will have to do this while ignoring any dependencies with the –nodeps flag.

[root@bigboy tmp]# rpm -e --nodeps rsyslog

2.         Install syslog-ng using yum.

[root@bigboy tmp]# yum -y install syslog-ng

3.         Start the new syslog-ng daemon immediately and make sure it will start on the next reboot.

Systems using sysvinit:

[root@bigboy tmp]# chkconfig syslog-ng on

[root@bigboy tmp]# service syslog-ng start

Starting syslog-ng: [  OK  ]

[root@bigboy tmp]#

Systems using systemd:

[root@bigboy tmp]# systemctl enable syslog-ng.service

[root@bigboy tmp]# systemctl start syslog-ng.service

Starting syslog-ng: [  OK  ]

[root@bigboy tmp]#

Your new syslog-ng package is now up and running and ready to go!

[root@bigboy tmp]# ifconfig -a

[root@bigboy tmp]# cat /proc/interrupts

Changing Your IP Address

[root@bigboy tmp]# ifconfig eth0 10.0.0.1 netmask 255.255.255.0 up

[root@bigboy network-scripts]# ifdown eth0

[root@bigboy network-scripts]# ifup eth0

Multiple IP Addresses on a Single NIC

[root@bigboy tmp]# ifconfig wlan0:0 192.168.1.99 netmask 255.255.255.0 up

he commands to activate and deactivate the alias interface would therefore be:

[root@bigboy tmp]# ifup wlan0:0

[root@bigboy tmp]# ifdown wlan0:0

How to Activate/Shut Down Your NIC

[root@bigboy tmp]# ifdown eth0

[root@bigboy tmp]# ifup eth0

How to View Your Current Routing Table

[root@bigboy tmp]# netstat -nr

[root@bigboy tmp]# netstat -nr

How to Change Your Default Gateway   ----Temporary Default Gateway Assignment

[root@bigboy tmp]# route add default gw 192.168.1.1 wlan0

Permanent Default Gateway Assignment

Remember, the previous command is temporary and its effects will be lost after the next reboot. There are many ways to do this.

Note: In Debian based systems the default gateway is permanently defined in the /etc/network/interfaces file. See the section "Debian / Ubuntu Network Configuration" later in this chapter for more details.

The /etc/sysconfig/network file

Once done, you'll need to update your /etc/sysconfig/network file to reflect the change. This file is used to configure your default gateway each time Linux boots.

# /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=bigboy

GATEWAY=192.168.1.1

The /etc/sysconfig/network-scripts/ifcfg-<interface> file

Adding Temporary Static Routes

[root@bigboy tmp]# route add -net 10.0.0.0 netmask 255.0.0.0 gw 192.168.1.254 wlan0

[root@bigboy tmp]# route add -host 10.0.0.1 gw 192.168.1.254 wlan0

Adding Permanent Static Routes

In Fedora Linux, permanent static routes are added on a per interface basis in files located in the /etc/sysconfig/network-scripts directory. The filename format is route-interface-name so the filename for interface wlan0 would be route-wlan0.

The format of the file is quite intuitive with the target network coming in the first column followed by the word via and then the gateway's IP address. In our routing example, to set up a route to network 10.0.0.0 with a subnet mask of 255.0.0.0 (a mask with the first 8 bits set to 1) via the 192.168.1.254 gateway, we would have to configure file /etc/sysconfig/network-scripts/route-wlan0 to look like this:

# File /etc/sysconfig/network-scripts/route-wlan0

#

10.0.0.0/8 via 192.168.1.254

How to Delete a Route

Here's how to delete the routes added in the previous section.

root@bigboy tmp]# route del -net 10.0.0.0 netmask 255.0.0.0 gw 192.168.1.254 wlan0

The file /etc/sysconfig/network-scripts/route-wlan0 will also have to be updated so that when you reboot the server will not reinsert the route. Delete the line that reads:

10.0.0.0/8 via 192.168.1.254