引用

• Linux Logging Complete Guide – devconnected

• 内核printk原理介绍 - 知乎 (zhihu.com)

• Message logging with printk — The Linux Kernel documentation

• How to get printk format specifiers right — The Linux Kernel documentation

• (24条消息) Linux pstore 实现自动“抓捕”内核崩溃日志_宋宝华的博客-CSDN博客

• linux内核调试（三）内核崩溃日志抓取pstore - 知乎 (zhihu.com)

• The GNU C Library

• util-linux/dmesg.c at master · util-linux/util-linux (github.com)

• BusyBox

• klogd.c « sysklogd - busybox - BusyBox: The Swiss Army Knife of Embedded Linux

• The rocket-fast Syslog Server - rsyslog

• rsyslog/rsyslog: a Rocket-fast SYStem for LOG processing (github.com)

• kernel - What is the difference between /proc/kmsg and /dev/kmsg? - Unix & Linux Stack Exchange

• syslog-ng/syslog-ng: syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. (github.com)

• Kernel logging: APIs and implementation - IBM Developer

• syslog(2) - Linux manual page (man7.org)

• syslog(3) - Linux manual page (man7.org)

• dmesg(1) - Linux manual page (man7.org)

• logrotate(8) - Linux manual page (man7.org)

• Linux内核基础篇——printk调试 (betheme.net)

一. overall

二. 接口差异

2.1 /proc/kmsg 和 /dev/kmsg 的差异？

• /proc/kmsg provides a root-only, read-only, consuming view of the kernel log buffer. It’s equivalent to calling syslog(2) with the SYSLOG_ACTION_READ action. As mentioned in the proc manpage,

A process must have superuser privileges to read this file, and only one process should read this file. This file should not be read if a syslog process is running which uses the syslog(2) system call facility to log kernel messages.

• /dev/kmsg provides access to the same kernel log buffer, but in an easier-to-use fashion. Reads are tracked per open, so multiple processes can read in parallel, and entries aren’t removed from the buffer as they are read. /dev/kmsg also provides write access to the log buffer, so it can be used to add entries to the log buffer. See the /dev/kmsg documentation for details.

• As for why both are present, and why one is in /proc (albeit not process-related) and one in dev, /proc/kmsg is an old convenience “export” of kernel internals, and /dev/kmsg is a more recent addition, designed as a usable interface to the log buffer.

2.2 系统调用 syslog()?

The syslog call serves as the input/output (I/O) and control interface to the kernel's log message ring buffer. From the syslog call, an application can read log messages (partial, in their entirety, or only new messages) as well as control the behavior of the ring buffer (clear contents, set the level of messages to be logged, enable or disable console, and so on).

• syslog(2) vs. syslog(3)

Note that the syslog defined here ( syslog(2)) is different from the API for sending messages to the system logger ( syslog(3)). The latter allows messages to be sent to the syslog (through functions to open, close, and write to the log using a particular priority).

• syslog(2) - 调用系统调用 syslog().

/** syslog, klogctl - read and/or clear kernel message ring buffer;* set console_loglevel*/#include         /* Definition of SYSLOG_* constants */
#include      /* Definition of SYS_* constants */
#include int syscall(SYS_syslog, int type, char *bufp, int len);/* The glibc interface */
#include int klogctl(int type, char *bufp, int len);

• syslog(3) - 向系统logger发送日志，例如，klogd使用该组接口与syslogd对接，写入log到syslogd中。

/** closelog, openlog, syslog, vsyslog - send messages to the system logger*/#include void openlog(const char *ident, int option, int facility);
void syslog(int priority, const char *format, ...);
void closelog(void);void vsyslog(int priority, const char *format, va_list ap);

2.3 glibc提供的 klogctl()?

The syslog call (called do_syslog within the kernel in ./linux/kernel/printk.c) is a relatively small function that provides the ability to read and control the kernel ring buffer. Note that in glibc 2.0, this function is called klogctl because of overuse of the term syslog, which refers to a variety of calls and applications. The prototype function (in user space) for syslog and klogctl is defined as:

int syslog( int type, char ∗bufp, int len );
int klogctl( int type, char ∗bufp, int len );

2.4 klogd和syslogd的关系与作用？

syslogd和klogd是很有意思的守护进程，syslogd是一个分发器，它将接收到的所有日志按照/etc/syslog.conf的配置策略发送到这些日志应该去的地方，当然也包括从klogd接收到的日志。klogd首先接收内核的日志，然后将之发送给syslogd。

syslogd日志记录器由两个守护进程（klogd，syslogd）和一个配置文件（syslog.conf）组成。klogd不使用配置文件，它负责截获内核消息，它既可以独立使用也可以作为syslogd的客户端运行。syslogd默认使用/etc/syslog.conf作为配置文件，负责截获应用程序消息，还可以截获klogd向其转发的内核消息。支持internet/unix domain sockets的特性使得这两个工具可以用于记录本地和远程的日志。

klogd会调用 glibc中的api，openlog 、syslog 、closelog 接口，将kernel log发送到 syslogd。

2.5 console log level?

printk() is typically used like this:

printk(KERN_INFO "Message: %s\n", arg);

where KERN_INFO is the log level (note that it’s concatenated to the format string, the log level is not a separate argument). The available log levels are:

The log level specifies the importance of a message. The kernel decides whether to show the message immediately (printing it to the current console) depending on its log level and the current console_loglevel (a kernel variable). If the message priority is higher (lower log level value) than the console_loglevel the message will be printed to the console.

If the log level is omitted, the message is printed with KERN_DEFAULT level.

You can check the current console_loglevel with:

    $ cat /proc/sys/kernel/printk4        4        1        7

The result shows the current, default, minimum and boot-time-default log levels.

即默认只有0-3等级的log才能输出到串口或者console。

To change the current console_loglevel simply write the desired level to /proc/sys/kernel/printk. For example, to print all messages to the console:

# echo 8 > /proc/sys/kernel/printk

Another way, using dmesg:

# dmesg -n 5

sets the console_loglevel to print KERN_WARNING (4) or more severe messages to console. See dmesg(1) for more information.

或者在启动命令行中添加配置：

通过在启动内核时传递commandline给内核的方法来修改系统默认的输出等级。例如，使用uboot引导内核时，可以在uboot传参的bootargs参数上，加上“loglevel=8”，这样在系统启动时，就打开了所有内核输出。

2.6 kmsg dump接口

kmsg dump接口主要提供给pstore和mtdoops使用，mtdoops用于在系统oops时将log buffer中的信息保存到mtd设备中，在当前内核版本中pstore也已支持将log保存到mtd中，因此pstore可以代替mtdoops的功能了，因此本文主要介绍pstore的log dump功能。

pstore的主要作用是在系统panic或oops时，将系统崩溃日志保存到backend设备中，以供调试分析。backend设备可以是重启不掉电的ram，mtd或者块设备，根据不同的配置可以选择使用不同的后端设备。同时pstore前端除了panic和oops之外，还支持通过ftrace，pmsg和console，如ftrace前端用于将ftrace信息保存到pstore中，pmsg前端用于将用户态信息保存到pstore中。