What do you need
- Installed Ubuntu server (personally in my implementation it is version 13.04)
- Installed Apache2
- Installed configured and already collecting squid statistics (in my case it is 2.7)
We will preliminarily stipulate the conditions:
- The user is identified by his IP address
- Each user can only watch his own statistics
- Statistics are displayed via the WEB interface
Begin
Need lighsquid himself
The first step is to download LightSquid
wget http://downloads.sourceforge.net/project/lightsquid/lightsquid/1.8/lightsquid-1.8.tgz
Unzip it. I preferred to unzip to the root / var, in fact, you can also in / www (if you need charts, etc.)
tar -zxf lightsquid-1.8.tgz / var / lightsquid
We set recursively 755 rights to the user and group www-data on the directory and its contents
chmod -R 755 / var / lightsquid
chown -R www-data: www-data / var / lightsquid
We make the program scripts executable:
chmod + x * .cgi
chmod + x * .pl
If you really need graphics then you need to install the libgd-gd2-perl package
sudo apt-get install libgd-gd2-perl
Now you need to tweak the LightSquid config itself
nano /var/lightsquid/lightsquid.cfg
Editing paths in the GLOBAL VARIABLES section
#path to additional `cfg` files
$ cfgpath =" / var / lightsquid ";
#path to `tpl` folder
$ tplpath =" / var / lightsquid / tpl ";
#path to `lang` folder
$ langpath =" / var / lightsquid / lang ";
#path to `report` folder
$ reportpath =" / var / lightsquid / report ";
#path to access.log
$ logpath = "/ var / log / squid";
#path to `ip2name` folder
$ ip2namepath =" / var / lightsquid / ip2name ";
In the WEB VARIABLES section, set the desired interface language. In this case, Russian.
$ lang = "ru";
Well all the lightsquid should be ready to go. If you have squid logs, you can parse them so that you can then see them in the report:
/ var / lightsquid # ./lightparser.pl access.log.59 && ./lightparser.pl access.log.58 && ./lightparser.pl access.log.57 && ./lightparser.pl access.log.56 &&. /lightparser.pl access.log.55 && ./lightparser.pl access.log.54 && ./lightparser.pl access.log.53 ... etc.
The process itself
Because LightSquid with the help of cgi scripts generates html tagged files, then gives them to the server, the server to the user, and the user, in turn, sees ALL statistics on ALL users - this is not good. Therefore, there is a task, "intercept" the output, make your own html files with preference and courtesans and give the processed file to the user.
First PHP script
Create a file:
nano /var/www/index.php
We will have the first script on PHP: this will be the start page that will show the user the "processed" first page of ligsquid. We enter the data:
<?php
$year = " {$_GET['year']}";
$month = " {$_GET['month']}";
$day = " {$_GET['day']}";
$ip_host = "{$_SERVER['REMOTE_ADDR']}";
$str = "sudo /var/lightsquid/my_scr/index.sh ".$ip_host.$year.$month.$day;
echo exec($str);
include "/var/lightsquid/my_tmp/".$ip_host."log_file.html"
?>
The principle is as follows. We receive any parameters through PHP, in this case the year, month, day, on the start page these parameters are empty, but if you switch to another month or year, they will be filled in immediately.
Next, we ourselves will find out the IP of the host that requested this page. We need this ip to form data ONLY on it, and in general to somehow navigate among the files we create, because we will bind the file name to its IP address.
Next, we form a string. Which launches the next script (described below) and transfers IP parameters, year, month, day to it.
We call this script for execution.
We show the html page. Note that the name of the page is formed from the IP address of the host and the prefix "log_file.html"
Second shell script
This script will implement the lightsquid cgi script launch and write the execution result (cgi script) to a separate file. And then it initializes the launch of another script with the transfer of parameters to it (about which below).
Create a file
nano /var/lightsquid/my_scr/index.sh
Making it executable
chmod + x index.sh
An important note, as the script must be executed under the rights of the www-data user. Then you should add the following line to the / etc / sudoers file in the # User privilege specification block:
www-data ALL = (ALL) NOPASSWD: /var/lightsquid/my_scr/index.sh
Next, we write the data itself to the file
#!/bin/sh
$1 #ip
$2 #
$3 #
$4 #
str="log_file"
param="year="$2"&month="$3"&day="$4
LOG_FILE=/var/lightsquid/my_tmp/$1$str.html
{
export REQUEST_METHOD=GET
export QUERY_STRING=$param
/var/lightsquid/index.cgi
}> $LOG_FILE 2>&1
/var/lightsquid/my_scr/index.py $1
Let me explain:
We get the IP parameters, year, day, month
Into the variable we write the string "log_file" We
form the request and write it to the variable
Run the cgi script into which we pass the parameters, save the result of the script to a file with the name of the host IP + "log_file.html"
Call another script, we pass parameters to it.
Third Python script
Because the generated page contains a lot of unnecessary and incorrect links (for example, it refers to cgi which we cannot intercept), and so on, this file needs to be corrected. replace all links with ours and delete a couple of lines.
Create a file:
nano /var/lightsquid/my_scr/index.py
We make the file executable.
chmod + x index.sh
We write the following data into it:
#!/usr/bin/env python
#coding: utf-8
import sys # sys
ip_host = sys.argv[1] # IP
# lightsquid
seach_ip = ip_host
path_file = '/var/lightsquid/my_tmp/'+ ip_host + 'log_file.html' #
count_line = sum(1 for l in open(path_file, 'r')) #
#len(open(path_file, 'r').readlines()) #
file = open(path_file, 'r') #
lines = file.readlines() #
file.close() #
i = 0
for i in range(count_line): # 0
if "day_detail.cgi" in lines[i]: # cgi
lines[i] = lines[i].replace('day_detail.cgi','day_detail.php') # ,
if "group_detail.cgi" in lines[i]: #
lines[i] = " <code><TD> </TD></code>" #
if "index.cgi" in lines[i]:
lines[i] = lines[i].replace('index.cgi','index.php')
if "month_detail.cgi" in lines[i]:
lines[i] = lines[i].replace('month_detail.cgi','month_detail.php')
if "graph.cgi" in lines[i]:
lines[i] = " "
if "topsites.cgi" in lines[i]:
lines[i] = lines[i].replace('topsites.cgi','topsites.php')
file = open('/var/lightsquid/my_tmp/' + ip_host + 'log_file.html','w')
file.writelines(lines)
file.close
In general, indentation (four spaces) is very important in Python for those who do not know :)
Let me explain the script a little.
We get the IP address of the user who requested the statistics (this IP address is transmitted here from the previous script).
Next, open the file generated by the lightsquid cgi script. Find out
how many lines it contains.
We make a loop, in this loop we check each line for compliance with our requirements. Namely, it changes the paths of links from cgi to php, also I did not need users not to see the groups, so I make this column generally empty ().
Well, that's it, we write back the already processed file.
And so, I figured out one way how everything should be done. But 27 scripts are needed for full functioning. Of course, it was possible to cram the whole thing into one script and in general it would have turned out only 3, but I did not complicate my life. Below I would like to give the rest of the scripts, but with a short analysis of them and without unnecessary steps to create them (since similar actions were described above).
The rest of the skips
Everything related to bigfiles
bigfiles.php
<?php
$year = " {$_GET['year']}";
$month = " {$_GET['month']}";
$day = " {$_GET['day']}";
$ip_host = "{$_SERVER['REMOTE_ADDR']}";
$str = "sudo /var/lightsquid/my_scr/my_bigfiles.sh ".$ip_host.$year.$month.$day;
echo exec($str);
include "/var/lightsquid/my_tmp/".$ip_host."log_file.html"
?>
my_bigfiles.sh
#!/bin/sh
$1 #ip
$2 #
$3 #
$4 #
str="log_file"
param="year="$2"&month="$3"&day="$4"&user="$1
LOG_FILE=/var/lightsquid/my_tmp/$1$str.html
{
export REQUEST_METHOD=GET
export QUERY_STRING=$param
/var/lightsquid/bigfiles.cgi
}> $LOG_FILE 2>&1
/var/lightsquid/my_scr/my_bigfiles.py $1
my_bigfiles.py
#!/usr/bin/env python
#coding: utf-8
import sys # sys
ip_host = sys.argv[1] # IP
seach_ip = ip_host
path_file = '/var/lightsquid/my_tmp/'+ ip_host + 'log_file.html' #
count_line = sum(1 for l in open(path_file, 'r')) #
#len(open(path_file, 'r').readlines()) #
file = open(path_file, 'r') #
lines = file.readlines() #
file.close() #
i = 0
for i in range(count_line): # 0
if "index.cgi" in lines[i]:
lines[i] = lines[i].replace('index.cgi','index.php')
if "user_detail.cgi" in lines[i]:
lines[i] = lines[i].replace('user_detail.cgi','user_detail.php')
file = open('/var/lightsquid/my_tmp/' + ip_host + 'log_file.html','w')
file.writelines(lines)
file.close
Everything related to day_detail
bigfiles.php
<?php
$year = " {$_GET['year']}";
$month = " {$_GET['month']}";
$day = " {$_GET['day']}";
$ip_host = "{$_SERVER['REMOTE_ADDR']}";
$str = "sudo /var/lightsquid/my_scr/my_day_detail.sh ".$ip_host.$year.$month.$day;
echo exec($str);
include "/var/lightsquid/my_tmp/".$ip_host."log_file.html"
?>
!/bin/sh
$1 #ip
$2 #
$3 #
$4 #
str="log_file" #
param="year="$2"&month="$3"&day="$4 #
LOG_FILE=/var/lightsquid/my_tmp/$1$str.html #
{
export REQUEST_METHOD=GET
export QUERY_STRING=$param
/var/lightsquid/day_detail.cgi
}> $LOG_FILE 2>&1
/var/lightsquid/my_scr/my_day_detail.py $1 # , ;)
#!/usr/bin/env python
#coding: utf-8
import sys # sys
ip_host = sys.argv[1] # IP
# lightsquid
# ip
seach_ip = ip_host
path_file = '/var/lightsquid/my_tmp/'+ ip_host + 'log_file.html' #
count_line = sum(1 for l in open(path_file, 'r')) #
#len(open(path_file, 'r').readlines()) #
file = open(path_file, 'r') #
lines = file.readlines() #
file.close() #
for i in range(count_line): # 0
if "tr bgcolor=" + "\42" + "cornsilk" + "\42" in lines[i] or "tr bgcolor=" + "\42" + "beige" + "\42" in lines[i]: # 1 2 ,
if seach_ip in lines[i+3]: continue # ip ,
else:
lines[i] = " "
lines[i+1] = " "
lines[i+2] = " "
lines[i+3] = " "
lines[i+4] = " "
lines[i+5] = " "
lines[i+6] = " "
lines[i+7] = " "
lines[i+8] = " "
lines[i+9] = " "
lines[i+10] = " "
lines[i+11] = " "
lines[i+12] = " "
lines[i+13] = " "
lines[i+14] = " "
lines[i+15] = " "
lines[i+16] = " "
if "user_time.cgi" in lines[i]:
lines[i] = lines[i].replace('user_time.cgi','user_time.php')
if "index.cgi" in lines[i]:
lines[i] = lines[i].replace('index.cgi','index.php')
if "user_detail.cgi" in lines[i]:
lines[i] = lines[i].replace('user_detail.cgi','user_detail.php')
if "topsites.cgi" in lines[i]:
lines[i] = " "
if "bigfiles.cgi" in lines[i]:
lines[i] = " "
if "group_detail.cgi" in lines[i]: #
lines[i] = " <TD> </TD>" #
if "get.cgi" in lines[i]:
lines[i] = lines[i].replace('get.cgi?png=datetime','datetime.png')
file = open('/var/lightsquid/my_tmp/' + ip_host + 'log_file.html','w')
file.writelines(lines)
file.close
I consider it necessary to clarify this script.
The script gets the IP of the user who requested the data.
Because the file that generates cgi is of the same type, I decided to bind the search for the line I needed to the color of the line, there are two of them
"cornsilk"
and
"beige"
. If there is such a line where there is this color, then we move three lines down, there is an IP address. We compare one or the wrong one, and if it is not the one, then we simply delete all 16 lines. (This is exactly how much the description of one line in the report takes).
Everything related to month_detail
month_detail.php
<?php
$year = " {$_GET['year']}";
$month = " {$_GET['month']}";
$mode = " {$_GET['mode']}";
$ip_host = "{$_SERVER['REMOTE_ADDR']}";
$str = "sudo /var/lightsquid/my_scr/my_month_detail.sh ".$ip_host.$year.$month.$mode;
echo exec($str);
include "/var/lightsquid/my_tmp/".$ip_host."log_file.html"
?>
my_month_detail.sh
!/bin/sh
$1 #ip
$2 #
$3 #
$4 #
str="log_file" #
param="year="$2"&month="$3"&mode="$4 #
LOG_FILE=/var/lightsquid/my_tmp/$1$str.html #
{
export REQUEST_METHOD=GET
export QUERY_STRING=$param
/var/lightsquid/month_detail.cgi
}> $LOG_FILE 2>&1
/var/lightsquid/my_scr/my_month_detail.py $1 $3 # , ;)
my_month_detail.py
#!/usr/bin/env python
#coding: utf-8
import sys # sys
ip_host = sys.argv[1] # IP
month = sys.argv[2]
# lightsquid
# ip
seach_ip = ip_host
path_file = '/var/lightsquid/my_tmp/'+ ip_host + 'log_file.html' #
count_line = sum(1 for l in open(path_file, 'r')) #
#len(open(path_file, 'r').readlines()) #
file = open(path_file, 'r') #
lines = file.readlines() #
file.close() #
for i in range(count_line): # 0
if "tr bgcolor=" + "\42" + "cornsilk" + "\42" in lines[i] or "tr bgcolor=" + "\42" + "beige" + "\42" in lines[i]: # 1 2 ,
if seach_ip in lines[i+3]: continue # ip ,
else:
if month == "all": # . , , ,
lines[i] = " "
lines[i+1] = " "
lines[i+2] = " "
lines[i+3] = " "
lines[i+4] = " "
lines[i+5] = " "
lines[i+6] = " "
lines[i+7] = " "
lines[i+8] = " "
lines[i+9] = " "
lines[i+10] = " "
lines[i+11] = " "
lines[i+12] = " "
lines[i+13] = " "
lines[i+14] = " "
else:
lines[i] = " "
lines[i+1] = " "
lines[i+2] = " "
lines[i+3] = " "
lines[i+4] = " "
lines[i+5] = " "
lines[i+6] = " "
lines[i+7] = " "
lines[i+8] = " "
lines[i+9] = " "
lines[i+10] = " "
lines[i+11] = " "
lines[i+12] = " "
lines[i+13] = " "
lines[i+14] = " "
lines[i+15] = " "
for i in range(count_line): # 0
if lines[i]<> " ":
if "index.cgi" in lines[i]:
lines[i] = lines[i].replace('index.cgi','index.php')
if "user_time.cgi" in lines[i]:
lines[i] = lines[i].replace('user_time.cgi','user_time.php')
if "graph.cgi" in lines[i]:
lines[i] = '<TD></TD>'
if "user_month.cgi" in lines[i]:
lines[i] = lines[i].replace('user_month.cgi','user_month.php')
if "user_detail.cgi" in lines[i]:
lines[i] = lines[i].replace('user_detail.cgi','user_detail.php')
if "get.cgi" in lines[i]:
lines[i] = lines[i].replace('get.cgi?png=datetime','datetime.png')
if "get.cgi" in lines[i]:
lines[i] = lines[i].replace('get.cgi?png=graph','graph.png')
file = open('/var/lightsquid/my_tmp/' + ip_host + 'log_file.html','w')
file.writelines(lines)
file.close
In this script, almost the same thing as described above, only the difference is this, different parameters are received (year and month), therefore the number of lines forming the data is different. depending on the received parameter,
month = sys.argv[2]
we choose the action.
Everything related to topsites
topsites.php
<?php
$year = " {$_GET['year']}";
$month = " {$_GET['month']}";
$day = " {$_GET['day']}";
$mode = " {$_GET['mode']}";
$order = " {$_GET['order']}";
$ip_host = "{$_SERVER['REMOTE_ADDR']}";
$str = "sudo /var/lightsquid/my_scr/my_topsites.sh ".$ip_host.$year.$month.$mode.$order;
echo exec($str);
include "/var/lightsquid/my_tmp/".$ip_host."log_file.html"
?>
my_topsites.sh
#!/bin/sh
$1 #ip
$2 #
$3 #
$4 #\
$5 #\
$6 #
str="log_file" #
if [ $5 = "hits" -o $4 = "size" ]
then
param="year="$2"&month="$3"&day=&mode="$4"&order="$5
else
param="year="$2"&month="$3"&mode="$4 #
fi
LOG_FILE=/var/lightsquid/my_tmp/$1$str.html #
{
export REQUEST_METHOD=GET
export QUERY_STRING=$param
/var/lightsquid/topsites.cgi
}> $LOG_FILE 2>&1
/var/lightsquid/my_scr/my_topsites.py $1 # , ;)
Here it is a little more complicated, also different parameters are passed to the script, and depending on the parameters accepted by the script, it forms a request to the cgi script.
my_topsites.py
!/usr/bin/env python
#coding: utf-8
import sys # sys
ip_host = sys.argv[1] # IP
seach_ip = ip_host
path_file = '/var/lightsquid/my_tmp/'+ ip_host + 'log_file.html' #
count_line = sum(1 for l in open(path_file, 'r')) #
#len(open(path_file, 'r').readlines()) #
file = open(path_file, 'r') #
lines = file.readlines() #
file.close() #
i = 0
for i in range(count_line): # 0
if "whousesite.cgi" in lines[i]: #
lines[i] = " <TD> </TD>" #
if "index.cgi" in lines[i]:
lines[i] = lines[i].replace('index.cgi','index.php')
if "topsites.cgi" in lines[i]:
lines[i] = lines[i].replace('topsites.cgi','topsites.php')
file = open('/var/lightsquid/my_tmp/' + ip_host + 'log_file.html','w')
file.writelines(lines)
file.close
Everything related to user_detail
user_detail.php
<?php
$year = " {$_GET['year']}";
$month = " {$_GET['month']}";
$day = " {$_GET['day']}";
$mode = " {$_GET['mode']}";
$ip_host = "{$_SERVER['REMOTE_ADDR']}";
$str = "sudo /var/lightsquid/my_scr/my_user_detail.sh ".$ip_host.$year.$month.$day.$mode;
echo exec($str);
include "/var/lightsquid/my_tmp/".$ip_host."log_file.html"
?>
user_detail.sh
#!/bin/sh
$1 #ip
$2 #
$3 #
$4 #\
str="log_file" #
if [ $4 = "month" -o $4 = "year" ]
then
param="year="$2"&month="$3"&user="$1"&mode="$4 #
else
param="year="$2"&month="$3"&day="$4"&user="$1
fi
LOG_FILE=/var/lightsquid/my_tmp/$1$str.html #
{
export REQUEST_METHOD=GET
export QUERY_STRING=$param
/var/lightsquid/user_detail.cgi
}> $LOG_FILE 2>&1
/var/lightsquid/my_scr/my_user_detail.py $1 # , ;)
user_detail.py
#!/usr/bin/env python
#coding: utf-8
import sys # sys
ip_host = sys.argv[1] # IP
seach_ip = ip_host
path_file = '/var/lightsquid/my_tmp/'+ ip_host + 'log_file.html' #
count_line = sum(1 for l in open(path_file, 'r')) #
#len(open(path_file, 'r').readlines()) #
file = open(path_file, 'r') #
lines = file.readlines() #
file.close() #
i = 0
for i in range(count_line): # 0
if "index.cgi" in lines[i]:
lines[i] = lines[i].replace('index.cgi','index.php')
if "user_time.cgi" in lines[i]:
lines[i] = lines[i].replace('user_time.cgi','user_time.php')
if "bigfiles.cgi" in lines[i]:
lines[i] = lines[i].replace('bigfiles.cgi','bigfiles.php')
if "get.cgi" in lines[i]:
lines[i] = lines[i].replace('get.cgi?png=datetime','datetime.png')
file = open('/var/lightsquid/my_tmp/' + ip_host + 'log_file.html','w')
file.writelines(lines)
file.close
Everything related to user_month
user_month.php
<?php
$year = " {$_GET['year']}";
$month = " {$_GET['month']}";
$ip_host = "{$_SERVER['REMOTE_ADDR']}";
$str = "sudo /var/lightsquid/my_scr/my_user_month.sh ".$ip_host.$year.$month;
echo exec($str);
include "/var/lightsquid/my_tmp/".$ip_host."log_file.html"
?>
user_month.sh
#!/bin/sh
$1 #ip
$2 #
$3 #
str="log_file"
param="year="$2"&month="$3"&user="$1
LOG_FILE=/var/lightsquid/my_tmp/$1$str.html
{
export REQUEST_METHOD=GET
export QUERY_STRING=$param
/var/lightsquid/user_month.cgi
}> $LOG_FILE 2>&1
/var/lightsquid/my_scr/my_user_month.py $1
user_month.py
#!/usr/bin/env python
#coding: utf-8
import sys # sys
ip_host = sys.argv[1] # IP
# lightsquid
# ip
seach_ip = ip_host
path_file = '/var/lightsquid/my_tmp/'+ ip_host + 'log_file.html' #
count_line = sum(1 for l in open(path_file, 'r')) #
#len(open(path_file, 'r').readlines()) #
file = open(path_file, 'r') #
lines = file.readlines() #
file.close() #
i = 0
for i in range(count_line): # 0
if "user_detail.cgi" in lines[i]: # cgi
lines[i] = lines[i].replace('user_detail.cgi','user_detail.php') # ,
if "index.cgi" in lines[i]:
lines[i] = lines[i].replace('index.cgi','index.php')
if "graph.cgi" in lines[i]:
lines[i] = lines[i].replace('graph.cgi','graph.php')
file = open('/var/lightsquid/my_tmp/' + ip_host + 'log_file.html','w')
file.writelines(lines)
file.close
Everything related to user_time
user_time.php
<?php
$year = " {$_GET['year']}";
$month = " {$_GET['month']}";
$day = " {$_GET['day']}";
$mode = " {$_GET['mode']}";
$ip_host = "{$_SERVER['REMOTE_ADDR']}";
$str = "sudo /var/lightsquid/my_scr/my_user_time.sh ".$ip_host.$year.$month.$mode;
echo exec($str);
include "/var/lightsquid/my_tmp/".$ip_host."log_file.html"
?>
user_time.sh
#!/bin/sh
$1 #ip
$2 #
$3 #
$4 #\
str="log_file" #
if [ $4 = "month" -o $4 = "year" ]
then
param="year="$2"&month="$3"&user="$1"&mode="$4 #
else
param="year="$2"&month="$3"&day="$4"&user="$1
fi
LOG_FILE=/var/lightsquid/my_tmp/$1$str.html #
{
export REQUEST_METHOD=GET
export QUERY_STRING=$param
/var/lightsquid/user_time.cgi
}> $LOG_FILE 2>&1
/var/lightsquid/my_scr/my_user_time.py $1 # , ;)
user_time.py
#!/usr/bin/env python
#coding: utf-8
import sys # sys
ip_host = sys.argv[1] # IP
# lightsquid
# ip
seach_ip = ip_host
path_file = '/var/lightsquid/my_tmp/'+ ip_host + 'log_file.html' #
count_line = sum(1 for l in open(path_file, 'r')) #
#len(open(path_file, 'r').readlines()) #
file = open(path_file, 'r') #
lines = file.readlines() #
file.close() #
i = 0
for i in range(count_line): # 0
if "index.cgi" in lines[i]:
lines[i] = lines[i].replace('index.cgi','index.php')
file = open('/var/lightsquid/my_tmp/' + ip_host + 'log_file.html','w')
file.writelines(lines)
file.close
Outcome
Well, that's all, I hope no one is tired of the number of scripts. I hope this article will help someone solve the same issue, and maybe he will improve it.