TCP STatistic and Analysis Tool 
Locations of visitors to this page


Tstat home | TNG home | workgroup | people | software | papers

Tstat generates three different types of measurement collections:
  • Log files, storing flow-level measurements.
  • Histograms, storing the distribution of a given quantity during a time interval.
  • RRD, storing histograms in a RRD database.

LOG Files

Tstat creates a set of TXT files where each row corresponds to a different flow and each column is associated to a specific measure. When it is useful, the columns are grouped according to C2S - Client-to-Server and S2C - Server-to-Client traffic directions.
The generated logs are:
log_tcp_complete, log_tcp_nocomplete:
report every TCP connection that has been tracked by Tstat. A TCP connection is identified when the first SYN segment is observed, and is ended when either:
  • the FIN/ACK or RST segments are observer;
  • no data packet has been observed (from both sides) for a default timeout of 10s after the thress-way handshake or 5min after the last data packet (see TCP_SINGLETON_TIME and TCP_IDLE_TIME in param.h);
Tstat discards all the connections for which the three way handshake is not properly seen. Then, in case a connection is correctly closed it is stored in log_tcp_complete, otherwise in log_tcp_nocomplete.

log_udp_complete:
reports every tracked UDP flow pair. An UDP flow pair is identified when the first UDP segment is observed for a UDP socket pair, and is ended when no packet has been observed (from both sides) for 10s after the first packet or 3min after the last data packet (see UDP_SINGLETON_TIME and UDP_IDLE_TIME in param.h).
By default, Skype and chat protocols running over UDP are reported only in a separate file (see LOG_ALL_UDP in param.h).

log_mm_complete:
reports statistics for the RTP and RTCP flows. The classification process exploits a finite state machine that perform checks of version field, sequence numbers and payload types. In particupar, if in the first UDP packet
  • the version field is set to 2
  • the payload type field has an admissible values (for RTP or for RTCP)
  • the UDP ports are larger than 1024 and are even/odd for RTP/RTCP
the flow is marked as possible RTP/RTCP flow (FIRST_RTP/FIRST_RTCP).
When the second UDP segment of this UDP flow (same IP/ports) is observed, then Tstat double checks if it still be interpreted as RTP/RTCP payload controlling if
  • the version is equal to 2
  • the same ssrc is present
  • the seqno is the expected one
  • the payload type is the same as before.
If checks succedes, then the flows is marked as RTP and its analysis may start.


For RTCP flows, a simpler heuristic is used:
  • the version must be equal to 2
  • the payload type must be a correct one
  • the UDP ports are larger than 1024 and are even/odd for RTP/RTCP.
If so, the flow is considered a RTCP flow and its analysis may start.

log_skype_complete:
reports statistics for each SKYPE flow identified using the methodology described into "Revealing skype traffic: when randomness plays with you". Note that records change according to the trasport layer (UDP or TCP) used by Skype.

log_chat_complete, log_chat_messages:
Tstat is able to classify MSN Messenger, Yahoo! Messenger and Chat based on XMPP Protocol such as Jabber or Google Talk. In log_chat_complete are reported statistics for each chat flow while in log_chat_messages for each chat message.

log_video_complete:
If VIDEO_DETAILS is defined (see tstat/Makefile.conf), Tstat tracks statistics related to TCP Video connections. Currently are supported both RTMP video download and HTTP video downloads (YouTube, Vimeo, generic FLV/MP4). This log contains a subset of the data already reported in log_tcp_complete, with additional columns that are mostly relevant for YouTube connections and other video streams. For the semantics of the TCP connections and the meaning of the specific fields, you can refer to the description of log_tcp_complete.

log_streaming_complete:
If STREAMING_CLASSIFIER is defined (see tstat/Makefile.conf), Tstat produces a log_streaming_complete file which logs every HTTP Video connection that has been tracked. Currently are classified as Video, HTTP connections based on 2 distinguished approaches:
  • Value of Content-Type information in the HTTP's header
  • Signature matching in the video payload, to identify the video container
This log contains a subset of the data already reported in log_tcp_complete, with additional columns that are mostly relevant for the video characterization (duration,bitrate,size..). For the semantics of the TCP connections and the meaning of the specific fields, you can refer to the description of log_tcp_complete.
There is an overlapping between log_video_complete and log_streaming_complete, where the former contains mostly YouTube related information, including connections to the YouTube sites not carrying video content, with little other formats, while the latter contains only flows carrying video content, as identified by the STREAMING_CLASSIFIER engine.

[Logs]  
log_tcp_complete - log_tcp_nocomplete
C2S S2C Short description Unit Long description
1 45 Client/Server IP addr - IP addresses of the client/server
2 46 Client/Server TCP port - TCP port addresses for the client/server
3 47 packets - total number of packets observed form the client/server
4 48 RST sent 0/1 0 = no RST segment has been sent by the client/server
5 49 ACK sent - number of segments with the ACK field set to 1
6 50 PURE ACK sent - number of segments with ACK field set to 1 and no data
7 51 unique bytes bytes number of bytes sent in the payload
8 52 data pkts - number of segments with payload
9 53 data bytes bytes number of bytes transmitted in the payload, including retransmissions
10 54 rexmit pkts - number of retransmitted segments
11 55 rexmit bytes bytes number of retransmitted bytes
12 56 out seq pkts - number of segments observed out of sequence
13 57 SYN count - number of SYN segments observed (including rtx)
14 58 FIN count - number of FIN segments observed (including rtx)
15 59 RFC1323 ws 0/1 Window scale option sent
16 60 RFC1323 ts 0/1 Timestamp option sent
17 61 window scale - Scaling values negotiated [scale factor]
18 62 SACK req 0/1 SACK option set
19 63 SACK sent - number of SACK messages sent
20 64 MSS bytes MSS declared
21 65 max seg size bytes Maximum segment size observed
22 66 min seg size bytes Minimum segment size observed
23 67 win max bytes Maximum receiver window announced (already scale by the window scale factor)
24 68 win min bytes Maximum receiver windows announced (already scale by the window scale factor)
25 69 win zero - Total number of segments declaring zero as receiver window
26 70 cwin max bytes Maximum in-flight-size computed as the difference between the largest sequence number so far, and the corresponding last ACK message on the reverse path. It is an estimate of the congestion window
27 71 cwin min bytes Minimum in-flight-size
28 72 initial cwin bytes First in-flight size, or total number of unack-ed bytes sent before receiving the first ACK segment
29 73 Average rtt ms Average RTT computed measuring the time elapsed between the data segment and the corresponding ACK
30 74 rtt min ms Minimum RTT observed during connection lifetime
31 75 rtt max ms Maximum RTT observed during connection lifetime
32 76 Stdev rtt ms Standard deviation of the RTT
33 77 rtt count - Number of valid RTT observation
34 78 ttl_min - Minimum Time To Live
35 79 ttl_max - Maximum Time To Live
36 80 rtx RTO - Number of retransmitted segments due to timeout expiration
37 81 rtx FR - Number of retransmitted segments due to Fast Retransmit (three dup-ack)
38 82 reordering - Number of packet reordering observed
39 83 net dup - Number of network duplicates observed
40 84 unknown - Number of segments not in sequence or duplicate which are not classified as specific events
41 85 flow control - Number of retransmitted segments to probe the receiver window
42 86 unnece rtx RTO - Number of unnecessary transmissions following a timeout expiration
43 87 unnece rtx FR - Number of unnecessary transmissions following a fast retransmit
44 88 != SYN seqno 0/1 1 = retransmitted SYN segments have different initial seqno
89 Completion time ms Flow duration since first packet to last packet
90 First time ms Flow first packet since first segment ever
91 Last time ms Flow last segment since first segment ever
92 C first payload ms Client first segment with payload since the first flow segment
93 S first payload ms Server first segment with payload since the first flow segment
94 C last payload ms Client last segment with payload since the first flow segment
95 S last payload ms Server last segment with payload since the first flow segment
96 C first ack ms Client first ACK segment (without SYN) since the first flow segment
97 S first ack ms Server first ACK segment (without SYN) since the first flow segment
98 First time abs ms Flow first packet absolute time (epoch)
99 C Internal 0/1 1 = client has internal IP, 0 = client has external IP
100 S Internal 0/1 1 = server has internal IP, 0 = server has external IP
101 Connection type - Bitmask stating the connection type as identified by TCPL7 inspection engine (see protocol.h)
102 P2P type - Type of P2P protocol, as identified by the IPP2P engine (see ipp2p_tstat.h)
103 P2P subtype - P2P protocol message type, as identified by the IPP2P engine (see ipp2p_tstat.c)
104 ED2K Data - For P2P ED2K flows, the number of data messages
105 ED2K Signaling - For P2P ED2K flows, the number of signaling (not data) messages
106 ED2K C2S - For P2P ED2K flows, the number of client<->server messages
107 ED2K C2C - For P2P ED2K flows, the number of client<->client messages
108 ED2K Chat - For P2P ED2K flows, the number of chat messages
109 HTTP type - For HTTP flows, the identified Web2.0 content (see the http_content enum in struct.h)
110 SSL Client Hello - For SSL flows, the server name indicated by the client in the Hello message extensions
111 SSL Server Hello - For SSL flows, the subject CN name indicated by the server in its certificate
Connection type - col.101 (see protocol.h)
Bitmask Value Protocol
0 Unknown protocol
1 HTTP protocol
2 RTSP protocol
4 RTP protocol
8 ICY protocol
16 RTCP protocol
32 MSN protocol
64 YMSG protocol
128 XMPP protocol
256 P2P protocol
512 SKYPE protocol
1024 SMTP protocol
2048 POP3 protocol
4096 IMAP4 protocol
8192 SSL/TLS protocol
16384 ED2K protocol (obfuscated)
32768 SSH 2.0/1.99 protocol
65536 RTMP protocol
131072 Bittorrent MSE/PE protocol
P2P type - col. 102 (see ipp2p_tstat.h)
Bitmask n-th bit Internal Protocol
1 IPP2P_ED2K eMule
2 IPP2P_DATA_KAZAA Kazaa Data
3 IPP2P_DATA_ED2K Ed2k Data
4 IPP2P_DATA_DC DirectConnect++ Data
5 IPP2P_DC DirectConnect++
6 IPP2P_DATA_GNU Gnutella Data
7 IPP2P_GNU Gnutella
8 IPP2P_KAZAA Kazaa
9 IPP2P_BIT BitTorrent
10 IPP2P_APPLE Apple
11 IPP2P_SOUL SoulSeek
12 IPP2P_WINMX WinMX
13 IPP2P_ARES Ares
14 IPP2P_MUTE Mute
15 IPP2P_WASTE Waste
16 IPP2P_XDCC XDCC
17 IPP2P_KAD eMule KAD
18 IPP2P_KADU Adunanza (eMule mod)
HTTP Web2.0 type - Col 109 (see struct.h)
Type Internal Description
1 HTTP_GET Unclassified GET command
2 HTTP_POST Unclassified POST command
3 HTTP_MSN MSN Chat command tunneled over HTTP (POST)
4 HTTP_RTMPT RTMPT - RTMP over HTTP Tunnel (POST)
5 HTTP_YOUTUBE_VIDEO YouTube video content download (GET)
6 HTTP_VIDEO_CONTENT Generic FLV or MP4 video download (GET)
7 HTTP_VIMEO Vimeo video content download (GET)
8 HTTP_WIKI Wikipedia (GET)
9 HTTP_RAPIDSHARE RapidShare file download (GET)
10 HTTP_MEGAUPLOAD MegaUpload file download (GET)
11 HTTP_FACEBOOK Facebook-related connections (GET/POST)
12 HTTP_ADV Site advertisement (GET)
13 HTTP_FLICKR Flickr photo download (GET)
14 HTTP_GMAPS GoogleMaps images (GET)
15 HTTP_VOD Video-on-Demand download (GET) 1
16 HTTP_YOUTUBE_SITE YouTube site content download (GET)
17 HTTP_SOCIAL Localized social-networking (GET/POST) 2
18 HTTP_FLASHVIDEO Generic FLV video download (GET) 3
19 HTTP_MEDIAFIRE MediaFire file download (GET)
20 HTTP_HOTFILE Hotfile.com file download (GET)
21 HTTP_STORAGE Storage.to file download (GET)
22 HTTP_YOUTUBE_204 YouTube "pre-loading" (GET) 4
23 HTTP_YOUTUBE_VIDEO204 YouTube "pre-loading" and video (GET) 4
24 HTTP_YOUTUBE_SITE_DIRECT YouTube: video request on YouTube site (GET) 5
25 HTTP_YOUTUBE_SITE_EMBED YouTube: embedded video request (GET) 5
26 HTTP_TWITTER Twitter unencrypted traffic (GET/POST) 6
27 HTTP_DROPBOX Dropbox presence traffic (GET) 7
These values are different from 0 only for identified HTTP connections (column no. 97). There constants are also used in the RRD data and in histograms (decreased by one so that HTTP_GET is 0 and HTTP_GMAPS is 13).

1) HTTP_VOD connection identification is experimental and not valid for usage outside Politecnico di Torino.

2) HTTP_SOCIAL is a set of matchings tailored for Nasza-Klasa (PL) and IWIW (HU). Since IWIW seems to be based on OpenSocial, it should match also generic OpenSocial traffic. Probably not useful outside Poland or Hungary.

3) HTTP_FLASHVIDEO identify traffic from a few popular flash-based video distribution sites.

4) HTTP_YOUTUBE_204 and HTTP_YOUTUBE_VIDEO204 are counted as HTTP_YOUTUBE_VIDEO in RRDs and histograms (i.e. they are classified in idx4 ).

5) HTTP_YOUTUBE_SITE_DIRECT and HTTP_YOUTUBE_SITE_EMBED are counted as HTTP_YOUTUBE_SITE in RRDs and histograms (i.e. they are classified in idx15 ).

6) HTTP_TWITTER refers just to Twitter unencrypted connections, mostly related to the Twitter widgets in web pages. HTTP_TWITTER is counted as HTTP_SOCIAL and WEB_SOCIAL in RRDs and histograms (i.e. it is classified in idx16 ).

7) HTTP_DROPBOX refers to the presence/keep-alive connections maintained by the Dropbox client. Experimental. It is counted as HTTP_GET and WEB_OTHER in RRDs and histograms.
[Logs]  
log_udp_complete
C2S S2C Short description Unit Long description
1 9 Client/Server IP addr - IP addresses of client/server
2 10 Client/Server UDP port - UDP port addresses of client/server
3 11 First time s client/server first packet in absolute time (epoch)
4 12 Completion time s Time between the first and the last packet from the 'client'
5 13 Data bytes bytes Number of bytes transmitted in the payload
6 14 Packets - Total number of packets observed from the client/server
7 15 Internal 0/1 1 = IP address is internal
8 16 UDP Type - Protocol type (see also the udp_type enum in struct.h)
UDP type - col. 8, 16 (see struct.h)
Value Internal Description
0 UDP UNKNOWN Unknown (unclassified)
1 FIRST_RTP Unknown (possible unclassified RTP flow)
2 FIRST_RTCP Unknown (possible unclassified RTCP flow)
3 RTP RTP protocol
4 RTCP RTCP protocol
5 SKYPE_E2E Skype End-to-End
6 SKYPE_E2O SkypeOut
7 SKYPE_SIG Skype signalling
8 P2P_ED2K eMule ED2K protocol
9 P2P_KAD eMule KAD (Kamdelia) protocol
10 P2P_KADU Adunanza (eMule mod) KAD (Kamdelia) protocol
11 P2P_GNU Gnutella protocol
12 P2P_BT BitTorrent DHT protocol (only)
13 P2P_DC DirectConnect protocol
14 P2P_KAZAA KaZaa protocol
15 P2P_PPLIVE PPLive IP-TV protocol
16 P2P_SOPCAST SopCast IP-TV protocol
17 P2P_TVANTS TV-Ants IPTV protocol
18 P2P_OKAD eMule obfuscated KAD protocol
19 DNS DNS protocol
20 P2P_UTP BitTorrent uTP protocol (only)
21 P2P_UTPBT BitTorrent DHT and uTP protocols (mixed)
22 UDP_VOD MPEG2 PES Streaming over UDP
23 P2P_PPSTREAM PPStream IP-TV protocol
24 TEREDO Teredo IPv6 tunneling over UDP (mostly BitTorrent)
[Logs]  
log_mm_complete
C2S S2C Short Description Unit Long Description Protocol
1 L4 Proto 1/2 1 = TCP, 2 = UDP All
2 38 Protocol 3/4 3 = RTP, 4 = RTCP All
3 39 IP address - Client/Server IP addresses All
4 40 L4 port - TCP/UDP port addresses for the Client/Server All
5 41 Internal 0/1 1 = internal ip All
6 42 Packets - Number of packets Tstat has seen belonging to the flow All
7 43 IPG ms Inter Packet Gap (IPG) All
8 44 Jitter AVG ms/ts Jitter (average):
- if RTP, computed by Tstat as in RFC3550 [ms]
- if RTCP, extracted from the RTCP header [codec timestamps units];
- if TCP, computed using only data packets [ms] 		All
9 45 Jitter Max ms/ts Jitter (max)
- if RTP, computed by Tstat as in RFC3550 [ms]
- if RTCP, extracted from the RTCP header [codec timestamps units]
- if TCP, computed using only data packets [ms]
 All
10 46 Jitter Min ms/ts Jitter (min)
- if RTP, computed by Tstat as in RFC3550 [ms]
- if RTCP, extracted from the RTCP header [codec timestamps units]
- if TCP, computed using only data packets [ms] 		All
11 47 TTL AVG - Time to live (TTL) (average) All
12 48 TTL Max - Time to live (TTL) (max) All
13 49 TTL Min - Time to live (TTL) (min) All
14 50 Start s Start time All
15 51 Duration s Duration All
16 52 Data bytes Data transfered All
17 53 Bitrate bit/s Average speed [bit/s] All
18 54 SSRC - SSRC RTP, RTCP
19 55 Lost pkts - Lost packets, computed by Tstat using a window based algorithm RTP
20 56 Out of seq. pkts - Out of sequence packets computed by Tstat computed by Tstat using a window based algorithm TCP,RTP
21 57 Dup pkts - Duplicate packets computed by Tstat
- if RTP, computed by Tstat using a window based algorithm
- if TCP, computed as retrasmissions 		TCP,RTP
22 58 Late pkts - Late packets computed by Tstat computed by Tstat using a window based algorithm RTP
23 59 RTP type - RTP payload type RTP
24 60 Reset - Bogus reset RTP
25 61 Cum lost pkts - Cumulative packet loss:
- each lost packets increments this counter,
- each duplicated packets decremnets it from RTCP 		RTCP
26 62 Frac lost pkts - Extracted from the RTCP header [%] RTCP
27 63 Flow length - Associated RTP flow length RTCP
28 64 Flow length bytes Associated RTP flow length RTCP
29 65 RTT AVG ms Round Trip Time (RTT) (average) TCP, RTCP
30 66 RTT Max ms Round Trip Time (RTT) (max) TCP, RTCP
31 67 RTT Min ms Round Trip Time (RTT) (min) TCP, RTCP
32 68 RTT ms Round Trip Time (RTT) (samples) TCP, RTCP
33 69 Truncated RTCP header - Truncated RTCP header RTCP
34 70 First HTTP s First HTTP packet TCP
35 71 First RTSP s First RTSP packet TCP
36 72 FIRST RTP s First RTP packet TCP
37 73 FIRST ICY s First ICY packet TCP
[Logs]  
log_skype_complete (TCP)
C2S S2C Short Description Unit Long Description
1 17 Client/Server IP address - Client IP address
2 18 Client/Server TCP Port - Client TCP port
3 19 Internal 0/1 1 = internal IP address
4 20 Flow Size bytes Flow Size
5 21 Total packets - No. of Total flow packets
6 22 Audio/video pkts - No. of audio or audio+video packets
7 23 Video only pkts - No. of video only packets
8 24 Avg Pktsize - Average Packet size
9 25 Avg Pktsize: MMB - Average Packet Size: Max Mean Belief
10 26 Avg IPG - Average Inter-packet Gap
11 27 Avg IPG: MMB - Average IPG: Max Mean Belief
12 28 CHI HDR max - Chi-square on Header: max value
13 29 CHI PAY max - Chi-square on Payload: max value
14 30 BFT - Bayesian Flow Type
15 31 CSFT - Chi-square Flow Type
16 32 Video present 0/1 1 = Video is present
33 Start Time s Flow Start Time
34 Elapsed Time s Flow Elapsed Time
35 L4 proto 'U' Label to state a UDP flow
log_skype_complete (UDP)
C2S S2C Short description Unit Long description
1 24 Client/Server IP addr - IP address of the 'client'
2 25 Client/Server port - TCP/UDP port address for the 'client'
3 26 Internal 0/1 1 = internal IP address
4 27 Flow Size bytes Flow Size
5 28 Total packets - No. of Total flow packets
6 29 E2E packets - No. of End-to-End packets
7 30 E2O packets - No. of SkypeOut packets
8 31 SIG packets - No. of Signaling packets
9 32 UNK packets - No. of Unknown packets
10 33 Audio/Video pkts - No. of audio or audio+video packets
11 34 Video only pkts - No. of video only packets
12 35 Avg Pktsize - Average Packet size
13 36 Avg Pktsize: MMB - Average Packet Size: Max Mean Belief
14 37 Avg IPG ms Average Inter-packet Gap
15 38 Avg IPG: MMB - Average IPG: Max Mean Belief
16 39 CHI HDR min - Chi-square on Header: min value
17 40 CHI HDR max - Chi-square on Header: max value of {1-4} & {7,8} blocks
18 41 CHI HDR min 5,6 - Chi-square on Header: min value of {5,6} blocks
19 42 CHI PAY max - Chi-square on Payload: max value
20 43 DFT - Deterministic Flow Type
21 44 BFT - Bayesian Flow Type
22 45 CSFT - Chi-square Flow Type
23 46 Video present 0/1 1 = Video is present
47 Start Time s Flow Start Time (epoch)
48 Elapsed Time s Flow Elapsed Time
49 L4 proto 'T' Label to state a TCP flow
[Logs]  
log_chat_complete
C2S S2C Short description Unit Long description
1 11 Client/Server IP addr - IP address of client/server
2 12 Client/Server port - TCP port address of client/server
3 13 Flow Size bytes Flow Size [Bytes]
4 14 Total packets - No. of Total flow packets
5 15 Total messages - No. of Total messages sent by client
6 16 MSG_A - No. of MSG_A sent by client
[for MSN only, 0 for the others]
7 17 MSG_D - No. of MSG_D sent by client
[for MSN only, 0 for the others]
8 18 MSG_N - No. of MSG_N sent by client
[for MSN only, 0 for the others]
9 19 MSG_U - No. of MSG_U sent by client
[for MSN only, 0 for the others]
10 20 MSG_Y - No. of MSG_Y sent by client
[for MSN only, 0 for the others]
21 Start Time s Flow Start Time
22 End Time s Flow End Time
23 Chat Flow Type - Chat Flow Type
24 Chat Version - Version of the protocol used by the Instant Messaging application
25 Internal 0/1 1 = internal IP address
26 TCP Flow No. - TCP Flow ID Number
27 'T' - Label to state a TCP Flow
28 Chat Protocol 32=MSN
64=Yahoo
128=Jabber/GTalk		 Type of Upper Level Protocol
Chat Flow Type - col. 23
Value Description IM Protocols
0 Unknown All
1 Login All
2 Presence All
3 Chat All
4 Presence+Chat Yahoo only
5 Http Tunneling MSN only
6 Peer-to-Peer Chat (i.e. direct connection between clients) Yahoo only
7 Unclassified Yahoo Messenger flow Yahoo only


log_chat_messages
Col.no. Short descr Long description
1 TCP Flow No. TCP Flow ID Number
2 Message type Type of Message (? if not available)
3 Dir TCP Flow Direction (1=C2S, -1=S2C)
4 Message size Message Payload Size [Bytes] (? if not available)
5 Payload size TCP Payload Size [Bytes]
6 Start Time Flow Start Time [in Unix Epoch Time]
7 Arrival Time Message Arrival Time [s]
[Logs]  
log_video_complete
C2S S2C Short desc. Unit Long description
1 30 Client/Server IP addr - IP addresses of the client/server
2 31 Client/Server TCP port - TCP port addresses for the client/server
3 32 packets - total number of packets observed form the client/server
4 33 RST sent 0/1 0 = no RST segment has been sent by the client/server
5 34 unique bytes bytes number of bytes sent in the payload
6 35 data pkts - number of segments with payload
7 36 data bytes bytes number of bytes transmitted in the payload, including retransmissions
8 37 rexmit pkts - number of retransmitted segments
9 38 rexmit bytes bytes number of retransmitted bytes
10 39 out seq pkts - number of segments observed out of sequence
11 40 FIN count - number of FIN segments observed (including rtx)
12 41 max seg size bytes Maximum segment size observed
13 42 cwin max bytes Maximum in-flight-size computed as the difference between the largest sequence number so far, and the corresponding last ACK message on the reverse path. It is an estimate of the congestion window.
14 43 cwin min bytes Minimum in-flight-size [bytes]
15 44 Average rtt ms Average RTT computed measuring the time elapsed between the data segment and the corresponding ACK
16 45 rtt min ms Minimum RTT observed during connection lifetime
17 46 rtt max ms Maximum RTT observed during connection lifetime
18 47 Stdev rtt ms Standard deviation of the RTT
19 48 rtt count - Number of valid RTT observation
20 49 ttl_min - Minimum Time To Live
21 50 ttl_max - Maximum Time To Live
22 51 Rate Samples - Number of samples C2S/S2C in the rate measurement
23 52 Zero Samples - Number of empty samples C2S/S2C in the rate measurement
24 53 Zero Streak - Maximum number of consecutive C2S/S2C empty samples
25 54 Average rate kbps Average rate in the C2S/S2C direction
26 55 Stdev rate kbps Standard deviation rate in the C2S/S2C direction
27 56 min rate - Minimum (non zero) rate sample
28 57 max rate - Maximum rate sample
29 58 Src Internal 0/1 0 = external ip address, 1 = internal ip address
59 Completion time ms Flow duration since first packet to last packet
60 First time ms Flow first packet since first segment ever
61 Last time ms Flow last segment since first segment ever
62 C first payload ms Client first segment with payload since the first flow segment
63 S first payload ms Server first segment with payload since the first flow segment
64 C last payload ms Client last segment with payload since the first flow segment
65 S last payload ms Server last segment with payload since the first flow segment
66 C first ack ms Client first ACK segment (without SYN) since the first flow segment
67 S first ack ms Server first ACK segment (without SYN) since the first flow segment
68 First time abs ms Flow first packet absolute time (epoch)
69 Connection type - Bitmask stating the connection type (by TCPL7 payload inspection engine). See protocol.h
70 P2P type - Type of P2P protocol, as identified by the IPP2P engine. See ipp2p_tstat.h
71 HTTP type - For HTTP flows, the identified Web2.0 content. See the http_content enum in struct.h
72 HTTP Response - First HTTP Response code seen in the server->client communication
73 Video ID16 - 16-char YouTube video identifier, '--' otherwise
74 Video ID11 - 11-char YouTube video request ID if YOUTUBE_REQUEST_ID is defined, '--' otherwise
75 Video Format - YouTube Video Format code [*], '--' otherwise.
76 Begin Offset ms Playback offset for the Youtube video, 0 otherwise
77 FLV duration s Video duration as indicated in the FLV file metadata [+]
78 FLV start time s Video start time as indicated in the FLV file metadata [+]
79 FLV total dur. s Total Video duration as indicated in the FLV file metadata [+]
80 FLV width pixel Video width as indicated in the FLV file metadata [+]
81 FLV height pixel Video heigth as indicated in the FLV file metadata [+]
82 FLV video datarate kbps Video data rate as indicated in the FLV file metadata [+]
83 FLV audio datarate kbps Audio data rate as indicated in the FLV file metadata [+]
84 FLV total datarate kbps Total data rate as indicated in the FLV file metadata [+]
85 FLV framerate fps Video framerate as indicated in the FLV file metadata [+]
86 FLV size bytes Video size as indicated in the FLV file metadata [+]
87 Redir Mode - Server Redirection Type [=]
88 Redir Count - Redirection counter [=]
89 Mobile Media 0/1 1 = video is probably accessed using a mobile term (smartphone) or a Media Center, 0 otherwise
90 Mobile Device - Type of mobile device 0=None/Undefined 1=Apple iOS 2=Android 3=Other

[*] The YouTube video format is the 'fmt/itag' value indicated on Wikipedia. Common values are 34 (360p FLV), 35 (480p FLV), and 22 (720p MP4).
[+] Duration and size are not reported for MP4 videos.

[=] Redir Mode and Redir Count are based on the parameters redirect_count and st provided in the videodownload URL.
Redir
Mode		 Redir
Count		 Comment
0 0 Missing 'redirect_counter=' and 'st='
1 X 'redirect_counter=X', missing 'st='
2 X+1 'redirect_counter=X, 'st=tcts'
3 X+1 'redirect_counter=X, 'st=nx'
4 1 Missing 'redirect_counter=', 'st=lc'
5 1 Missing 'redirect_counter=', 'st=nx'
6 X+1 Any other combination
Redir Count is set when the video is redirected (using "Location" HTTP message) from v<X>.lscache<Y>.c.youtube.com address to the corresponding
- tc.v<X>.lscache<Y>.c.youtube.com or
- v<X>.nonxt<Y>.c.youtube.com or
- v<N>.cache<M>.c.youtube.com

st=tcts is set with Redir Count when the (already redirected) request is redirected to a location-identified cache r<N>.<city><X>[gst]<Y>.c.youtube.com

st=lc is set (with no Redir Count) when the lscache request is redirected to a location-identified cache r<N>.<city><X>[gst]<Y>.c.youtube.com

st=nx is set (with or without redirect_counter parameter) when the nonxt request is redirected to a location-identified cache r<N>.<city><X>[gst]<Y>.c.youtube.com. nonxt<N> addresses are used for unlisted and private videos.
[Logs]  
log_streaming_complete
C2S S2C Short desc. Unit Long description
1 30 Client/Server IP addr - IP addresses of the client/server
2 31 Client/Server TCP port - TCP port addresses for the client/server
3 32 packets - total number of packets observed form the client/server
4 33 RST sent 0/1 0 = no RST segment has been sent by the client/server
5 34 unique bytes bytes number of bytes sent in the payload
6 35 data pkts - number of segments with payload
7 36 data bytes bytes number of bytes transmitted in the payload, including retransmissions
8 37 rexmit pkts - number of retransmitted segments
9 38 rexmit bytes bytes number of retransmitted bytes
10 39 out seq pkts - number of segments observed out of sequence
11 40 FIN count - number of FIN segments observed (including rtx)
12 41 max seg size bytes Maximum segment size observed
13 42 cwin max bytes Maximum in-flight-size computed as the difference between the largest sequence number so far, and the corresponding last ACK message on the reverse path. It is an estimate of the congestion window.
14 43 cwin min bytes Minimum in-flight-size [bytes]
15 44 Average rtt ms Average RTT computed measuring the time elapsed between the data segment and the corresponding ACK
16 45 rtt min ms Minimum RTT observed during connection lifetime
17 46 rtt max ms Maximum RTT observed during connection lifetime
18 47 Stdev rtt ms Standard deviation of the RTT
19 48 rtt count - Number of valid RTT observation
20 49 ttl_min - Minimum Time To Live
21 50 ttl_max - Maximum Time To Live
22 51 Rate Samples - Number of samples C2S/S2C in the rate measurement
23 52 Zero Samples - Number of empty samples C2S/S2C in the rate measurement
24 53 Zero Streak - Maximum number of consecutive C2S/S2C empty samples
25 54 Average rate kbps Average rate in the C2S/S2C direction
26 55 Stdev rate kbps Standard deviation rate in the C2S/S2C direction
27 56 min rate - Minimum (non zero) rate sample
28 57 max rate - Maximum rate sample
29 58 Src Internal 0/1 0 = external ip address, 1 = internal ip address
59 Completion time ms Flow duration since first packet to last packet
60 First time ms Flow first packet since first segment ever
61 Last time ms Flow last segment since first segment ever
62 C first payload ms Client first segment with payload since the first flow segment
63 S first payload ms Server first segment with payload since the first flow segment
64 C last payload ms Client last segment with payload since the first flow segment
65 S last payload ms Server last segment with payload since the first flow segment
66 C first ack ms Client first ACK segment (without SYN) since the first flow segment
67 S first ack ms Server first ACK segment (without SYN) since the first flow segment
68 First time abs ms Flow first packet absolute time (epoch)
69 Connection type - Bitmask stating the connection type (by TCPL7 payload inspection engine). See protocol.h
70 P2P type - Type of P2P protocol, as identified by the IPP2P engine. See ipp2p_tstat.h
71 HTTP type - For HTTP flows, the identified Web2.0 content. See the http_content enum in struct.h
72 HTTP Response - First HTTP Response code seen in the server->client communication
73 Video ID16 - 16-char YouTube video identifier, '--' otherwise
74 Video ID11 - 11-char YouTube video request ID if YOUTUBE_REQUEST_ID is defined, '--' otherwise
75 Video Format - YouTube Video Format code1, '--' otherwise.
76 Begin Offset ms Playback offset for the Youtube video, 0 otherwise
77 Video Content-Type - The identified video format, based on the HTTP Content-Type information. See below for the description
78 Video Payload - The identified video format, based on the video payload information. See below for the description
79 Video duration s Video duration as indicated in the payload2
80 Video total datarate kbps Total data rate as indicated in payload3
81 Video width pixel Video width as indicated in the payload2
82 Video height pixel Video heigth as indicated in the payload2
HVIDEO format - col. 77/78
Value VIDEO FORMAT Description
0 NOT_DEFINED Unclassified or not video
1 FLV Adobe Flash Video container
2 MP4 MPEG-4 video, including F4V format and fragmented MP44
3 AVI AVI video format and DivX media format
4 WMV Microsoft Media Video File (WMV) and ASF content
5 MPEG MPEG-1, MPEG-2 and VOB video5
6 WEBM Video format based on VP8 codec
7 3GPP 3rd Generation Partnership Project (3GPP). The releases 5 and 6 are classified as MP4
8 OGG Ogg Vorbis Codec compressed Multimedia file
9 QUICKTIME Video exported with QuickTime Apple Inc software6
10 ASF ASF control packets (ASF video are generally classified as WMV)
11 UNKNOWN Other videos formats or Content-Type values like 'video/*'
These values are different from 0 only for identified HTTP connections (column no. 97). There constants are also used in the RRD data and in histograms (decreased by one so that HTTP_GET is 0 and HTTP_GMAPS is 13).

1) The YouTube video format is the 'fmt/itag' value indicated in Wikipedia Common values are 34 (360p FLV), 35 (480p FLV), and 22 (720p MP4).

2) Values reported only for FLV, MP4.

3) Value not reported for AVI format.

4) F4V and FLV differences are summarized here.

5) The signatures for MPEG encoded videos are based on the rules described here.

6) The classification relays only on the Content-Type value announce by the server. Currently the payload matching is not supported for this video format.


Histogram description

An Histogram represents the empirical distribution of a specific index considering a fixed measurement period. For each measured index, Tstat creates and updates an histogram that collects the hit number of that quantity. For examples, considering the IP packet length, Tstat updates, for each observed IP packet, the counter of the number of observed packets with a particular length. At the end of the measurement period, Tstat saves each histogram in a separate TXT file, reset all the values and then restarts to collect samples. The duration of a measurement period is defined by the MAX_TIME_STEP parameter, which is defined in the file param.h, and by default, it is set to 5 minutes.

Recalling that (see HOWTO) Tstat is able to distinguish between IN-coming, OUT-going and LOC-al traffic and among C2S - Client-to-Server and S2C - Server-to-Client, it follows that, when applicable, it generates histograms according to traffic directions. Histograms names are strictly related both to the direction and the type of measure and as to have a quick remainder of the supported indexes it can be used: 

    bash> tstat -H ?
    #name             min  bin_size     max  description
    profile_flows     0    1            5    flows handled
    profile_cpu       0    1            4    cpu load [clock/time]
    chat_flow_num     0    1            7    Number of tracked IM flow
    web_bitrate_loc   0    1            7    Web 2.0 content bitrate [bit/s] - local segments
    web_bitrate_out   0    1            7    Web 2.0 content bitrate [bit/s] - outgoing segments
    web_bitrate_in    0    1            7    Web 2.0 content bitrate [bit/s] - incoming segments
    L7_WEB_num_loc    0    1            7    Number of tracked Web 2.0 flows - local flows
    ...

The following tables report a verbose description of all the supported histograms grouped as:

  • IP Layer: statistics related to ip addresses and IP protocol;
  • TCP Segments: statistics related to individual TCP segments;
  • TCP Flows: statistics related to TCP flows;
  • UDP Layer: statistics related to UDP flows;
  • Streaming Flows: statistics related to streaming flows;
  • RTCP Flows: statistics related to RTCP protocol;
  • HTTP Flows: statistics related to HTTP protocol;
  • Profile: profiling of the machine running Tstat;


[Histograms] 
IP Layer
Name Direction Min Bin Size Max Unit Description
ip_tos loc,out,in 0 1 255 - IP TOS field
ip_ttl loc,out,in 0 1 255 - IP TTL field
ip_len loc,out,in 0 4 1500 byte IP packet length
ip_bitrate loc,out,in 0 1 4 kb/s IP bitrate
ip_protocol loc,out,in 0 1 255 - IP protocol
addresses - - - - - This file collects the number of packets originated/destined to a particular IP subnet. By default, Tstat considers /24 subnets, and counts how many packets have been sent/received having a particular IP subnet source/destination address. The format of this histogram is different from the others, has it stores in the first column the subnet address, in the second column the number of packets whose IP source in the subnet, and in the third column the number of packets whose IP destination is in the subnet. No particular order is applied when saving the histogram, so that sorting is left to the user.


[Histograms] 
TCP Segments
Name Direction Min Bin Size Max Unit Description
tcp_mss_used - 0 4 1600 - Negotiated TCP MSS: minimum between MSS declared by the server and the client
tcp_mss_b - 0 4 1600 - Server TCP MSS declared
tcp_mss_a - 0 4 1600 - Client TCP MSS declared
tcp_opts_TS - 1 1 4 - TCP option: Timestamp. 1 = ok, 2 = only client offered, 3 = only server offered, 4 = none offered
tcp_opts_WS - 1 1 4 - TCP option: WindowScale. 1 = ok, 2 = only client offered, 3 = only server offered, 4 = none offered
tcp_opts_SACK - 1 1 4 - TCP option: SACK. 1 = ok, 2 = only client offered, 3 = only server offered, 4 = none offered
tcp_bitrate loc,out,in 0 1 29 bit/s TCP application bitrate
tcp_port_syndst loc,out,in 0 1 65536 - TCP destination port of SYN segments only
tcp_port_synsrc loc,out,in 0 1 65536 - TCP source port of SYN segments only
tcp_port_dst loc,out,in 0 1 65536 - TCP destination port
tcp_port_src loc,out,in 0 1 65536 - TCP source port


[Histograms] 
TCP Flows
Name Direction Min Bin Size Max Unit Description
tcp_interrupted - 0 1 1 - TCP Early interrupted flows. A flow is considered early interrupted according to the rules identified in:
Rossi D., Casetti C. and Mellia M.,
“User Patience and the Web: a hands-on investigation” ,
IEEE Globecom 2003, San Francisco, CA, USA, December 1-5, 2003.
tcp_thru c2s,s2c 0 1 1000 kb/s TCP application throughput. The throughput is defined as the ratio between the data sent by the server/client over the time since the first SYN segment up to the last segment carrying data from the server/client, i.e., no TCP tear-down latency is included.
tcp_tot_time - 0 50 720000 ms TCP flow lifetime, i.e., the time since the first ever seen SYN segment up to the very last segment of this flow.
tcp_anomalies s2c,c2s,loc,out,in 0 1 64 - TCP total number of anomalies per each flow. TCP anomalies are identified according to the algorithm described in
Mellia M., Meo M. and Muscariello L.,
“TCP Anomalies: identification and analysis”,
Tyrrhenian International Workshop on Digital Communications Sorrento, July 4-6.
tcp_rtx_RTO s2c,c2s,loc,out,in 0 1 100 - TCP anomaly: Number of RTO Retransmission
tcp_rtx_FR s2c,c2s,loc,out,in 0 1 100 - TCP anomaly: number of FR Retransmission
tcp_flow_ctrl s2c,c2s,loc,out] 0 1 100 - TCP anomaly: number of Flow Control
tcp_flow_control_in - 0 1 100 - TCP anomaly: number of Flow Control - incoming flows
tcp_net_dup s2c,c2s,loc,out,in 0 1 100 - TCP anomaly: number of Network duplicates
tcp_reordering s2c,c2s,loc,out,in 0 1 100 - TCP anomaly: number of packet reordering
tcp_unnrtx_FR s2c,c2s,loc,out 0 1 100 - TCP anomaly: number of Unneeded FR retransmission
tcp_unnecessary_rtx_FR_in - 0 1 100 - TCP anomaly: number of Unneeded FR retransmission - incoming flows
tcp_unnrtx_RTO s2c,c2s,loc,out 0 1 100 - TCP anomaly: number of Unneeded RTO retransmission
tcp_unnecessary_rtx_RTO_in - 0 1 100 - TCP anomaly: number of Unneeded RTO retransmission - incoming flows
tcp_unknown s2c,c2s,loc,out,in 0 1 100 - TCP anomaly: number of unknown anomalies
tcp_rtt_cnt s2c,c2s,loc,out,in 0 1 200 - TCP flow RTT: number of valid valid samples
tcp_rtt_stdev s2c,c2s,loc,out,in 0 10 3500 ms TCP flow RTT: standard deviation
tcp_rtt_max s2c,c2s,loc,out,in 0 10 3500 ms TCP flow RTT: maximum RTT
tcp_rtt_avg s2c,c2s,loc,out,in 0 10 3500 ms TCP flow RTT: average RTT
tcp_rtt_min s2c,c2s,loc,out,in 0 10 3500 ms TCP flow RTT: minimum RTT
tcp_cl_b_l s2c,c2s,loc,out,in 0 50000 50000000 byte TCP flow length - coarse granularity histogram
tcp_cl_b_s s2c,c2s,loc,out,in 0 50 50000 byte TCP flow length - fine granularity histogram
tcp_cl_p s2c,c2s,loc,out,in 0 1 1000 packet TCP flow length
tcp_cwnd - 0 256 65536 byte TCP in-flight-size: the difference among the highest sequence number and the highest acknowledgment number on the reverse path seen when a new ACK is received.
tcp_win_max - 0 256 65536 byte TCP max RWND: the maximum RWND (eventually scaled by the WS option observed during flow lifetime. Only RWND values sent by the client are considered.
tcp_win_avg - 0 256 65536 byte TCP average RWND: the average RWND (eventually scaled by the WS option observed during flow lifetime. Only RWND values sent by the client are considered.
tcp_win_ini - 0 256 65536 byte TCP initial RWND: the first ever observed value of the RWND (eventually scaled by the WS option)


[Histograms] 
UDP Layer
Name Direction Min Bin Size Max Unit Description
udp_port_flow_dst - 0 1 65536 - UDP destination port per flow
udp_port_dst loc,in,out 0 1 65536 - UDP destination port per segment
udp_tot_time - 0 50 720000 ms UDP flow lifetime: time since the first segment ever observed to the last observed segment
udp_cl_b_l loc,in,out 0 50000 50000000 byte UDP flow length - coarse granularity histogram
udp_cl_b_s loc,in,out 0 50 50000 byte UDP flow length - fine granularity histogram
udp_cl_p loc,in,out 0 1 1000 packet UDP flow length
udp_bitrate loc,in,out 0 1 50 bit/s UDP application bitrate


[Histograms] 
Streaming Flows
Name Direction Min Bin Size Max Unit Description
mm_burst_loss loc,out,in 0 1 20 packet Stream burst length of lost packets: number of missing packets with continuous sequence number
mm_p_late loc,out,in 0 1 1000 - Stream prob of late packets per flow: ratio between the number of packet arrived with a delay larger than 20 sequence number (i.e., packet 32 arrived when expecting packet 55) and the total number of flow packets.
mm_p_lost loc,out,in 0 1 1000 - Stream prob of lost packets per flow: ratio between the number of missing segments over the flow total number of segments
mm_p_dup loc,out,in 0 1 1000 - Stream prob of duplicate packets per flow: ratio between the number of duplicated segments over the total flow number of segments
mm_p_oos loc,out,in 0 1 1000 - Stream prob of out-of-sequence packets per flow: ratio between the number of out-of-sequence segments over the flow total number of segments
mm_n_oos loc,out,in 0 1 100 - Stream number of out-of-sequence packets per flow: total number of out-of-sequence segments (any segment whose seqno is not the largest ever seen plus 1) observed in the whole flow life
mm_oos_p loc,out,in 0 1 0 - Total stream number of out of sequence packets
mm_reord_p_n loc,out,in 0 1 0 - Total stream number of reordered packets observed in during the time intervals
mm_reord_delay loc,out,in 0 1 100 - Stream delay of reordered packets: time elapsed since the reception of the out-of-sequence packet and its immediate predecessor
mm_avg_jitter loc,out,in 0 1 5000 0.1m Stream average jitter per flow
mm_avg_ipg loc,out,in 0 1 5000 0.1m Stream average IPG per flow
mm_avg_bitrate loc,out,in 0 10 10000 kb/s Stream bitrate
mm_cl_b loc,out,in 0 50000 100000000 byte Long stream flow length
mm_cl_p loc,out,in 0 10 50000 packet Long stream flow length
mm_cl_b_s loc,out,in 0 100 100000 byte Short stream flow length
mm_cl_p_s loc,out,in 0 1 1000 packet Short stream flow length
mm_tot_time_s loc,out,in 0 1 5000 ms Short stream flow lifetime
mm_tot_time loc,out,in 0 1 5400 s Stream flow lifetime
mm_rtp_pt loc,out,in 0 1 128 - RTP payload type
mm_uni_multi loc,out,in 0 1 1 - Unicast/multicast flows
mm_type loc,out,in 0 1 8 - Stream type


[Histograms] 
RTCP Flows
Name Direction Min Bin Size Max Unit Description
rtcp_bt loc,out,in 0 10 10000 bit/s RTCP average bitrate
rtcp_mm_bt loc,out,in 0 1 5000 kb/s RTCP associated MM flow average bitrate during interval
rtcp_mm_cl_b loc,out,in 0 50000 100000000 byte RTCP associated MM flow length
rtcp_mm_cl_p loc,out,in 0 10 50000 packets RTCP associated MM flow length
rtcp_t_lost loc,out,in 0 10 10000 - RTCP lost packets per flow
rtcp_f_lost loc,out,in 0 1 1000 - RTCP fraction of lost packets during interval
rtcp_dup loc,out,in 0 1 1000 - RTCP duplicated packets during interval
rtcp_lost loc,out,in 0 1 1000 - RTCP lost packets during interval
rtcp_jitter loc,out,in 0 1 1000 - RTCP jitter during interval
rtcp_rtt loc,out,in 0 1 3000 ms RTCP round trip time
rtcp_avg_inter loc,out,in 0 1 5000 - RTCP interarrival delay
rtcp_cl_b loc,out,in 0 1 3000 byte RTCP flow length
rtcp_cl_p loc,out,in 0 1 3000 packet RTCP flow length


[Histograms] 
HTTP Flows
Name Direction Min Bin Size Max Unit Description
http_bitrate loc,in,out 0 1 21 bit/s HTTP content bitrate
web_bitrate loc,in,out 0 1 7 bit/s Web2.0 content bitrate
L7_HTTP_num loc,in,out 0 1 21 - Number of tracked HTTP flows
L7_WEB_num loc,in,out 0 1 7 - Number of tracked Web2.0 flows


[Histograms] 
Profile
Name Direction Min Bin Size Max Unit Description
profile_flow - 0 1 5 - Flows handled
profile_cpu - 0 1 4 - CPU load (clock/time)
-->
  Navigation Shortcuts
  Main
  Overview
  Web Interface
        Gallery
  Download
        Archives
        SVN
  Available Traces
        Skype
        Instant Messaging
        Multicast IP-TV
  Documentation
        Measurement
              Histograms
              Logs
              RRD interface
        Publications
        HOWTO
  Useful Links
  Contacts
  Tstat Mailing List
  M. Mellia
  M. Munafò


Tstat home | TNG home | workgroup | people | software | papers

©2008 Telecommunication Networks Group - Politecnico di Torino