I work as a teacher at the Quantorium children's technology park. During the period of self-isolation, we, like everyone else, switched to distance learning. And due to the fact that children began to spend even more time at the computer, the administration decided to shorten the academic hour and take breaks between work - to preserve their eyesight. We have written an application that calculates the time spent at the computer, keeps statistics in excel - useful for parents, and gives a sound alert when it’s time to take a break.
The application will be useful for those who are lost in time working at the PC and want to drive themselves into the time frame or keep track of what part of life is lost in the digital space.
Link to the repository
Under the cut a detailed analysis.
To create a program, you need to solve the following tasks:
- web ( mtcnn);
- ( time)
- excel ( openpyxl);
- ;
- .
web
We considered the option to track the presence at the computer by the movement of the computer mouse, perhaps it is easier and the web camera will remain free, but then we will not be able to take into account the viewing of TV shows or youtube videos, and besides, using face recognition is much more interesting.
For face detection, we use the mtcnn architecture. A head positioned at any angle, wearing glasses, in poor lighting conditions, even if the face does not completely fall into the frame or is covered by a hand, the neural network is working properly. We will not build it on our own, we will connect the library of the same name, which will solve our problem using a few lines of code.
Initially, the Viola-Jones method was used. The accuracy is much worse than that of mtcnn, the processor loads the same.
We will move the output of the result into a separate function, it is only needed to see how recognition works.
import cv2
import os
from mtcnn.mtcnn import MTCNN
def show_face_frame():
if faces:
bounding_box = faces[0]['box']
keypoints = faces[0]['keypoints']
cv2.rectangle(img,
(bounding_box[0], bounding_box[1]),
(bounding_box[0] + bounding_box[2],
bounding_box[1] + bounding_box[3]),
(0, 0, 255),
2)
cv2.circle(img, (keypoints['left_eye']), 3, (0, 0, 255), 2)
cv2.circle(img, (keypoints['right_eye']), 2, (0, 0, 255), 2)
cv2.circle(img, (keypoints['nose']), 3, (0, 0, 255), 2)
cv2.circle(img, (keypoints['mouth_left']), 3, (0, 0, 255), 2)
cv2.circle(img, (keypoints['mouth_right']), 3, (0, 0, 255), 2)
cv2.imshow('frame', img)
cap = cv2.VideoCapture(0)
detector = MTCNN()
while cap.isOpened():
_, img = cap.read()
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
faces = detector.detect_faces(img)
show_face_frame()
cv2.waitKey(300)
cap.release()
cv2.destroyAllWindows()
Big delay cv2.waitKey (300) so as not to load the processor. Processing 3-4 frames per second loads the i3-8145U by an average of 15%.
Counting time
The program should not write to the log in the following cases:
- sat down at the computer for 5 seconds to do something byrik;
- went out to pour coffee or stretch my legs.
To solve these problems, the program uses two stopwatches time_here (counts down the time when the face is in the frame) and time_not_here (when there is no face). Both stopwatches are reset when a log entry is made. In addition, time_not_here is reset every time a face appears in the frame.
The min_time_here variable indicates the minimum time spent at the computer, after which it is worth writing to the log. The recording will be made after the person is distracted from the computer for a time longer than specified in min_time_not_here .
In the code below, the recording is carried out if you have spent at least 5 minutes at the computer. The program will not take into account the fact that I am distracted for less than 2 minutes.
import cv2
import time
import os
from mtcnn.mtcnn import MTCNN
cap = cv2.VideoCapture(0)
path = os.path.abspath(__file__)[:-11] #
here, not_here = 0, 0 #
time_here, time_not_here = 0, 0 #
switch = True #
min_time_here = 300 # , ( )
min_time_not_here = 120 # ( )
detector = MTCNN()
try:
while cap.isOpened():
_, img = cap.read()
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
faces = detector.detect_faces(img)
show_face_frame()
audio_message(path, here)
if faces and switch:
not_here = 0
switch = False
if here == 0:
here = time.time()
print(' ' + time.strftime("%H:%M:%S", time.localtime(here)))
elif not faces and not switch:
not_here = time.time()
switch = True
if not_here != 0:
time_not_here = time.time() - not_here
if here != 0 and not switch:
time_here = time.time() - here
if time_here > min_time_here and time_not_here > min_time_not_here: # min_time_here
write_excel(path, here)
print(' ' + time.strftime('%H:%M:%S', time.gmtime(time.time() - here)))
time_here, time_not_here = 0, 0
here, not_here = 0, 0
elif time_not_here > min_time_not_here and time_here < min_time_here: # min_time_here,
print(' ')
time_here, time_not_here = 0, 0
here, not_here = 0, 0
cv2.waitKey(300)
except KeyboardInterrupt:
if time_here > min_time_here: # min_time_here
write_excel(path, here)
cap.release()
cv2.destroyAllWindows()
If the user wants to turn off the computer, first you need to interrupt the program (ctrl + c) and the time of the current work will be logged. An exception handler is used for this.
Writing data to excel document
For working with excel, the wonderful openpyxl library will help, which simultaneously read and write data.
Every day the program creates a new sheet with the current date.
On 1 sheet, the daily information is recorded, on which day how much time was spent at the computer.
The write_excel function opens the document, writes data, and closes it immediately.
You cannot modify a document through openpyxl if it is already open by the user. An exception has been added for this case.
def write_excel(path_excel, time_start):
wb = openpyxl.load_workbook(path_excel + r'\dnevnik.xlsx')
today = time.strftime("%d.%m.%Y", time.localtime(time.time()))
if today not in wb.sheetnames: # ,
wb.create_sheet(title=today)
sheet = wb[today]
sheet.column_dimensions['A'].width = 20
sheet.column_dimensions['B'].width = 20
sheet.column_dimensions['C'].width = 20
sheet.column_dimensions['D'].width = 31
sheet['A1'] = ' '
sheet['B1'] = ' '
sheet['C1'] = ' '
sheet['D1'] = ' :'
sheet['D2'] = ' :'
sheet = wb[today]
row = 2
all_time = 0
else: #
sheet = wb[today]
row = sheet['E1'].value # excel
all_time = sheet['E2'].value #
all_time = all_time.split(':')
all_time = int(all_time[0]) * 3600 + int(all_time[1]) * 60 + int(all_time[2]) #
row = row + 2 #
sheet['A' + str(row)] = time.strftime("%H:%M:%S", time.localtime(time_start)) #
sheet['C' + str(row)] = time.strftime("%H:%M:%S", time.localtime(time.time())) # -
seconds = time.time() - time_start #
sheet['B' + str(row)] = time.strftime('%H:%M:%S', time.gmtime(seconds))
sheet['E1'] = row - 1 #
all_time = all_time + seconds
all_time = time.strftime('%H:%M:%S', time.gmtime(all_time))
sheet['E2'] = all_time
#
sheet_0 = wb.worksheets[0]
sheet_0['A' + str(len(wb.sheetnames))] = today
sheet_0['B' + str(len(wb.sheetnames))] = all_time
while True: #
try:
wb.save(path_excel + r'\dnevnik.xlsx')
except PermissionError:
input(' excel enter')
else:
break
Sound alert
For a change, we generated 8 audio files using a speech synthesizer , which run randomly after an hour of work. Until you take a break, there will be no repeated audio alerts.
def audio_message(path, here):
if here == 0:
pass
elif time.strftime('%H:%M:%S', time.gmtime(time.time()-here)) == "01:00:00":
wav = random.choice(os.listdir(path + r'\audio'))
winsound.PlaySound(path + r'\audio\\' + wav, winsound.SND_FILENAME)
Startup script when turning on the PC
In order for the program to load together with Windows, you can create a batch file that will run the python script. The batch file should be located in the directory:
C: \ Users \% username% \ AppData \ Roaming \ Microsoft \ Windows \ Start Menu \ Programs \ Startup
Since the batch file and the source file are stored in different directories, you cannot use the os.getcwd ( ) since it will return the path to the batch file. We are using os.path.abspath (__ file __) [: - 11]
UPD: ak545suggested a better option os.path.dirname (os.path.abspath (__ file__))
In future
I wanted the program to follow my face, and not everyone who sits at the computer. For this, FaceNet will be used, which is capable of facial recognition.
There are also plans to develop a beautiful widget instead of an ugly black console, and so that when the computer is turned off, the recording is done automatically, and not using an interrupt.
Thank you for attention. If you have questions, write in the comments or at https://www.linkedin.com/in/evg-voronov/