Credit: Data Science Central
SIP application server (AS) text logs analysis may help in detection and, in some specific situations, prediction of different types of issues within a VoIP network. SIP server text logs contain the information which is difficult to obtain or even cannot be obtained from other sources, such as CDRs or signaling traffic captures.
The following parameters, among others, can help in estimating VoIP signaling network status:
- SIP dialog length. SIP dialog length of hundreds or even thousands messages points to a possible IP network problem, VoIP equipment malfunction, SIP signaling fraud, abnormal subscriber behavior
- Number and type of SIP messages retransmissions. A relatively high number of the retransmissions may be caused by AS hardware (HW) and/or software (SW) issues, IP network issues, peer SIP entities issues
- Request-response times for different SIP transactions in different SIP dialogs. Request-response time trends can help to predict overloads, IP network issues etc.
Depending on signaling load, a SIP AS can generate up to several tens of gigabytes of logs in text format per day, that’s why analysis of the SIP AS text logs is time- and resource-consuming task. Pandas data frames (DF) can help in such analysis. Pandas provides powerful tools for working with large DFs. Maximization of large DF processing speed may be achieved, in particular, by vectorizing all operations applied to the DF. All of the code for this post is available on GitHub. See also
SIP text log file processing steps:
- Open a SIP text log file for reading. I recommend opening SIP text log files in the same order as they were created by AS SW
- Read one line at a time from the opened SIP log file
- Extract SIP messages. Usually, these messages are located between specific delimiters
- Create a list consisting of dictionaries. Each of the dictionaries consists of SIP message timestamp (key) and SIP messages (value, as a list)
- Save the list to a pickle file on HDD or network storage. The file will be used for creating different DFs
- Create a DF containing specific information extracted from SIP messages stored in the pickle file
In this concrete case, the SIP DF contains the following columns:
- ‘Timestamp’ – added by SIP AS
- ‘Call-ID’ – extracted from SIP ‘Call-ID’ header
- ‘CSeq_num’, ‘CSeq_meth’ – derived from CSeq header of a SIP message
- ‘Direction’ – transmitted (Tx->) or received (Rx<-) SIP message, added by SIP AS
- ‘SIP method’ – SIP method name
Fig. 1. SIP DF example
Having such SIP DF, we can extract some amount of helpful information.
1. SIP dialog length
Fig. 2. The number of long SIP dialogs is very low, each dialog of length > 100 messages may be analyzed for clarification the particular call scenario.
2. Request-response times (in ms) for transmitted INFO or INVITE requests
Fig. 3. Resp_Req_Time plots show approximately the same distribution of request-response times for INFO- and INVITE-transactions for the same groups of SIP peers. Request-response times > 500 ms point to retransmits. 500 ms is the default value for SIP T1 timer.
3. The number of retransmissions of INVITE or INFO requests.
Retransmit of a SIP request may be detected as a sequence of the transmitted SIP requests with the same Call-ID and SIP method and CSeq sequence number.
4. Request-response times (in ms) for received INFO-requests.
We cannot use Pandas
groupby operation in this case because of the following reasons:
- Different INFO-200 OK transactions in a SIP dialog may share the same Call-ID and CSeq_num values
- INFO-requests and 200 OK-responses belonging to different dialogs may arrive in arbitrary moments of time and, consequently, will be stored in SIP DF in arbitrary order
- Retransmits of INFO-requests are possible, i.e. ‘SIP method’ column may contain sequences of retransmitted INFO-requests and 200 OK-responses
One of possible solutions is splitting DF into two separate data frames df_req and df_resp. ‘Timestamp’, ‘Call ID’, ‘CSeq_num’, ‘SIP method’ columns are the same for both DFs, ‘TS_req’ and ‘TS_resp’ are unique for df_req and df_resp. ‘Call ID’ and ‘CSeq_num’ columns are necessary for further analysis of particular INFO-200 OK transactions.
Fig. 4. Request-response time count plot for received INFO messages
Pandas DFs may be used as an additional tool for obtaining helpful information from SIP logs. Some of the methods described in this post may be used to analyze text log files of other protocols based on the request-response model.