pcap和pcapng格式的一点研究

前言

这次L3HCTF跟着队友混了个冠军。

在看官方a-sol的wp的时候看到一个脚本中有如下代码，直接运行会报错。ValueError: invalid tcpdump header

1

dpkt.pcap.Reader(open("a-sol.pcap",'rb'))

事实上dpkt有pcang的解析模块，将代码修改如下即可正确运行。

1

dpkt.pcapng.Reader(open("a-sol.pcapng",'rb'))

这个问题之前在写恶意流量识别项目的时候也遇到过，当时对pcap和pcapng差别不很清楚，只是知道格式不一样，并没有深入的进行研究，而是选择使用WinDump替代tshark进行流量捕获。这次又遇到了，就来仔细看一下，本来是准备写就看一下格式转换方法，发现越看越多，那就看到哪写到哪吧。

一点研究

现象及验证

有两个以前遇到的流量包，一个可以用dpkt.pcap包解析，一个不能用dpkt.pcap

1
2
3
4


 ⚡ /mnt/c/project/pcap > hexdump 03.pcap | head -n 1
0000000 c3d4 a1b2 0002 0004 0000 0000 0000 0000
 ⚡ /mnt/c/project/pcap > hexdump a-sol.pcapng | head -n 1
0000000 0d0a 0a0d 00c8 0000 3c4d 1a2b 0001 0000

虚拟机使用tcpdump捕获是pcap格式

1
2
3
4
5
6
7


 ~ > sudo tcpdump -i ens32 -s 10 -w vmcap.pcap
tcpdump: listening on ens32, link-type EN10MB (Ethernet), capture size 10 bytes
^C2698 packets captured
2728 packets received by filter
0 packets dropped by kernel
 ~ > hexdump vmcap.pcap | head -n 1
0000000 c3d4 a1b2 0002 0004 0000 0000 0000 0000

win10wireshark捕获默认为pcapng格式，可以选择保存为pcap格式

1
2
3
4


 ⚡ > /mnt/c/project/pcap > hexdump out.pcap | head -n 1
0000000 c3d4 a1b2 0002 0004 0000 0000 0000 0000
 ⚡ > /mnt/c/project/pcap > hexdump out.pcapng | head -n 1
0000000 0d0a 0a0d 00c0 0000 3c4d 1a2b 0001 0000

WinDump捕获的是pcap格式

1
2
3
4
5
6
7
8
9


PS C:\Users\ruokeqx\Downloads> WinDump.exe -s 10000 -w out.pcap
C:\Windows\system32\WinDump.exe: listening on {1BEDAF4F-8ABD-46F4-B122-453989CD2B2F}

222 packets captured
222 packets received by filter
0 packets dropped by kernel
PS C:\Users\ruokeqx\Downloads> wsl
 ⚡ > /mnt/c/Users/ruokeqx/Downloads > hexdump out.pcap | head -n 1
0000000 c3d4 a1b2 0002 0004 0000 0000 0000 0000

引入

对于两种格式的研究更像是对历史的研究（

在Winpcap官方的项目NTAR首页的介绍可以看到下面一段话。

NTAR implements a new "wanna-be" dump standard that overcomes the limits of the current libpcap/WinPcap dump format, by allowing the user to save packets coming from multiple interfaces and different data-links in a single trace file.

这里可以获得很多信息，新的标准(即pcapng)通过允许用户保存packets coming from multiple interfaces and different data-links到一个流量包里来克服现存libpcap/WinPcap捕获格式的缺陷。

下面链接中对PCAP Next Generation Dump File Format进行了详细阐述。

https://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html

至于最显示的特征差别就在于文件头了，可以通过文件头快速确定格式。

The block type of the Section Header Block is the integer corresponding to the 4-char string "\r\n\n\r" (0x0A0D0D0A).

选用这个文件头的原因文中有描述，简单说一是检测inappropriate ASCII conversion判断文件是否损坏，二是回文任意字节序都能识别出来。

而pcap的文件头是0xA1B2C3D4，至此文件格式和对于的捕获文件头都符合了。

The PCAP format has become the standard format for dumping captured packets in the free and open-source software community. Hone strives to adhere to this format as much as possible to help achieve acceptance within the networking community and to allow interoperability with other software. The original PCAP format, however, is deficient in describing anything except packets. Luckily, there is a new PCAP format, PCAP-NG (PCAP Next Generation), on the horizon with initial support in libpcap, wireshark, and other analysis software.

pcapng 就是 PCAP next generation 的意思，他的文件格式规范在下面链接中。

https://github.com/pcapng/pcapng

https://github.com/HoneProject/Linux-Sensor/wiki/Augmented-PCAP-Next-Generation-Dump-File-Format

pcap格式草案

https://datatracker.ietf.org/doc/html/draft-gharris-opsawg-pcap-02

pcapng格式草案

https://datatracker.ietf.org/doc/html/draft-tuexen-opsawg-pcapng

NTAR可以读写pcapng文件但是项目已经frozen并且不支持一些块。

The Network Trace Archival and Retrieval library is able to read and write pcapng files. Caveats: i) frozen since 2007, ii) no plugins for EPB, ISB, and NRB blocks

wireshark的wiki中有一个NTAR的扩展程序ntartest（个人感觉没啥用）

https://gitlab.com/wireshark/wireshark/-/wikis/Development/PcapNg#ntartest-a-simplistic-standalone-pcapng-ntar-file-reader

Tcpdump\libpcap

Tcpdump通过libpcap抓取报文，在linux平台，libpcap通过AF_PACKET套接字抓取。Tcpdump官方推荐windows用户使用winpcap和windump(现在看来当时似乎没选错)。保存格式为pcap

Tcpdump uses libpcap, a system-independent interface for user-level packet capture.

People with Windows distributions are best to check the Windows PCAP page for references to WinDUMP.

WinDump\WinPcap

WinDump是windows平台的tcpdump，他基于WinPcap的。保存格式也是pcap

WinDump is the Windows version of tcpdump, the command line network analyzer for UNIX. WinDump is fully compatible with tcpdump and can be used to watch, diagnose and save to disk network traffic according to various complex rules. It can run under Windows 95, 98, ME, NT, 2000, XP, 2003 and Vista.

WinDump captures using the WinPcap library and drivers, which are freely downloadable from the WinPcap.org website. WinDump supports 802.11b/g wireless capture and troubleshooting through the Riverbed AirPcap adapter.

而WinDump官方的changelog显示他们在06年以后就没有更新了。

而WinPcap也在13年发布最后一次更新或后再也没有更新，他们在18年发布了如下通知，就是说不会再有更新，社区支持还会存在，替代物建议使用nmap作者创建的npcap。

WinPcap, though still available for download (v4.1.3), has not seen an upgrade in many years and there are no road map/future plans to update the technology. While community support may persist, technical oversight by Riverbed staff, responses to questions posed by Riverbed resources, and bug reporting are no longer available.
Gordon Lyon, Nmap project founder, has created Npcap, a packet capture library for Windows, that includes WinPcap compatibility and may be a suitable replacement for WinPcap and WinPcap Pro. Information can be found at https://nmap.org/npcap/.

Wireshark\Npcap

Wireshark is the world’s foremost and widely-used network protocol analyzer.

The Wireshark installer includes Npcap which is required for packet capture.

windows安装wireshark的时候会让安装npcap，也就是说wireshark作为最先进的抓包和分析工具之一已经选用了npcap。

wireshark安装附带多个程序

1
2
3
4
5
6
7


Tools - Additional command line tools to work with capture files
    Editcap - Reads a capture file and writes some or all of the packets into another capture file.
    Text2Pcap - Reads in an ASCII hex dump and writes the data into a pcap capture file.
    Reordercap - Reorders a capture file by timestamp.
    Mergecap - Combines multiple saved capture files into a single output file.
    Capinfos - Provides information on capture files.
    Rawshark - Raw packet filter.

下面是参数中有对格式进行说明的，他们都可以指定文件格式，而且全部都是默认使用pcapng格式。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


dumpcap.exe --help
	-n                       use pcapng format instead of pcap (default)
	-P                       use libpcap format instead of pcapng
tshark.exe --help
	-F <output file type>    set the output file type, default is pcapng
                           an empty "-F" option will list the file types
editcap.exe --help
	-F <capture type>      set the output file type; default is pcapng.
						An empty "-F" option will list the file types.
mergecap.exe --help
	-F <capture type>      set the output file type; default is pcapng.
						An empty "-F" option will list the file types.

其中只指定-F输出的提示性可用格式如下。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


mergecap: The available capture file types for the "-F" flag are:
    5views - InfoVista 5View capture
    btsnoop - Symbian OS btsnoop
    commview - TamoSoft CommView
    dct2000 - Catapult DCT2000 trace (.out format)
    erf - Endace ERF capture
    eyesdn - EyeSDN USB S0/E1 ISDN trace format
    k12text - K12 text file
    lanalyzer - Novell LANalyzer
    logcat - Android Logcat Binary format
    logcat-brief - Android Logcat Brief text format
    logcat-long - Android Logcat Long text format
    logcat-process - Android Logcat Process text format
    logcat-tag - Android Logcat Tag text format
    logcat-thread - Android Logcat Thread text format
    logcat-threadtime - Android Logcat Threadtime text format
    logcat-time - Android Logcat Time text format
    modpcap - Modified tcpdump - pcap
    netmon1 - Microsoft NetMon 1.x
    netmon2 - Microsoft NetMon 2.x
    nettl - HP-UX nettl trace
    ngsniffer - Sniffer (DOS)
    ngwsniffer_1_1 - NetXray, Sniffer (Windows) 1.1
    ngwsniffer_2_0 - Sniffer (Windows) 2.00x
    niobserver - Network Instruments Observer
    nokiapcap - Nokia tcpdump - pcap
    nsecpcap - Wireshark/tcpdump/... - nanosecond pcap
    nstrace10 - NetScaler Trace (Version 1.0)
    nstrace20 - NetScaler Trace (Version 2.0)
    nstrace30 - NetScaler Trace (Version 3.0)
    nstrace35 - NetScaler Trace (Version 3.5)
    pcap - Wireshark/tcpdump/... - pcap
    pcapng - Wireshark/... - pcapng
    rf5 - Tektronix K12xx 32-bit .rf5 format
    rh6_1pcap - RedHat 6.1 tcpdump - pcap
    snoop - Sun snoop
    suse6_3pcap - SuSE 6.3 tcpdump - pcap
    visual - Visual Networks traffic capture

Npcap是nmap在win平台的流量捕捉库，linux和mac也支持了npcap，其是winpcap的改善和扩展。

Npcap is the Nmap Project's packet capture (and sending) library for Microsoft Windows. It implements the open Pcap API using a custom Windows kernel driver alongside our Windows build of the excellent libpcap library.

Mac and Linux systems already include the Pcap API, so Npcap allows popular software such as Nmap and Wireshark to run on all these platforms (and more) with a single codebase.

Npcap began in 2013 as some improvements to the (now discontinued) WinPcap library, but has been largely rewritten since then with hundreds of releases improving Npcap's speed, portability, security, and efficiency.

转换方法

pcapng to pcap

方法一：直接用wireshark打开pcapng文件另存为pcap文件

方法二：editcap指定格式输出

1

editcap.exe a-sol.pcapng a-sol.pcap -F pcap

方法三：用mergecap将pcapng转化为指定的pcap格式

1

mergecap.exe a-sol.pcapng -w a-sol.pcap -F pcap

此外mergecap功能如其名，还能将多个包合到一起。

方法四：tcpdump/libpcap (4.1.0+/1.1.0+)

1

tcpdump -r a-sol.pcapng -w a-sol.pcap

pcap to pcapng

1
2


editcap -F pcapng file.pcap file.pcapng
tshark -F pcapng -r file.pcap -w file.pcapng