World & AI Data Challenge: Getting Started with Braille Recognition

Artificial intelligence and data analysis technologies are entering our lives more and more rapidly, they can give another chance to solve social problems that are really important for people that have not been previously implemented. To this end, the ASI digital development center organized the World AI & Data Challenge competition , the purpose of which is to structure the process of searching for social problems and their solutions. In February 2020, the team of the ASI Digital Development Center invited me to join the experts of this competition. In this post I will tell you a little about the competition itself, as well as how you can start solving one of the interesting tasks of this competition - Braille recognition . You can participate in solving this and other tasks of the competition until August 31, 2020.

About the competition

, . .

, , . , , . 147 43 , , , . 30 30 , 8 — .

, . 2 . : - , - - , . . — http://git.asi.ru 31 2020 ..

( — 2021 .), . . .

, 1,5 . , 1 Data Science -.

— . , , , .

, :

• ,
• ,

— , .

, , . , 2017 , GitHub, , — Python ( , , - ). , , . !

, . : Smart Braille System Recognizer (2013), Optical Braille Recognition Based on Semantic Segmentation Network.

, . - , tutorial

, , . , :

, .

, , - :

• [ Kaggle][KaggleData], , , . .
• GitHub , , data augmentation.
• , . , , .. , , ..

, :

1. /
2. , ( )
4. ,
5. . , , spell checker, , Microsoft.

. , , GitHub https://github.com/shwars/braillehack. — fork! Visual Studio Codespaces

OpenCV. , - :

im = cv2.imread('../data/Photo_Turlom_C1_2.jpeg')
im = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)

, . , , . : — .

, :

im = cv2.blur(im,(3,3))
im = cv2.adaptiveThreshold(im, 255, cv2.ADAPTIVE_THRESH_MEAN_C,
                           cv2.THRESH_BINARY_INV, 5, 4)
im = cv2.medianBlur(im, 3)
_,im = cv2.threshold(im, 0, 255, cv2.THRESH_OTSU)
im = cv2.GaussianBlur(im, (3,3), 0)
_,im = cv2.threshold(im, 0, 255, cv2.THRESH_OTSU)
plt.imshow(im)

:

, , .



.

. feature detectors, OpenCV ORB, :

orb = cv2.ORB_create(5000)
f,d = orb.detectAndCompute(im,None)

5000 — , . , :

def plot_dots(im,dots):
    img = np.zeros_like(im)
    for x in dots:
        cv2.circle(img,(int(x[0]),int(x[1])),1,(255,0,0))
    plt.imshow(img)

pts = [x.pt for x in f]
plot_dots(cim,pts)  

, :

• . , . , , .
• .

feature detector , .

: OR- , . , — , .

, ! , - !

, , , ( ) . , . , , . "" , , :

min_x, min_y, max_x, max_y = \
   [int(f([z[i] for z in pts])) 
       for f in (min,max) for i in (0,1)]

(, 500):

off = 5
src_pts = np.array([(min_x-off,min_y-off),(min_x-off,max_y+off),
                    (max_x+off,min_y-off),(max_x+off,max_y+off)])
dim = 500
dst_pts = np.array([(0,0),(0,dim),(dim,0),(dim,dim)])
h,m = cv2.findHomography(src_pts,dst_pts)
trim = cv2.warpPerspective(cim,h,(dim,dim))
plt.imshow(trim)

.

, , .

char_h = 32
char_w = 22
def slice(img):
    dy,dx = img.shape
    y = 0
    while y+char_h<dy:
        x=0
        while x+char_w<dx:
            #    x
            while np.max(img[y:y+char_h,x])!=0:
                x+=1
            while np.max(img[y:y+char_h,x+char_w])!=0:
                x-=1
            #    
            if np.max(img[y:y+char_h,x:x+char_w])>0:
                yield img[y:y+char_h,x:x+char_w]
            x+=char_w
        y+=char_h

sliced = list(slice(trim))

. - , , "" . , - , x. .

, :

, , MNIST, , :

• Keras/Tensorflow —
• , PyTorch — , .

, . - , . , - , .

, . , , , . ,

Microsoft, REST-. MVP .

( Python), . , , . (MVP), Microsoft Azure .

:

• - Telegram Skype, . - Microsoft Bot Framework.
• , -, Xamarin React Native
• -

Python , REST-. -API, . API Azure Function HTTP-.

Azure Function — - ( Python), - ( — REST-). , Python — .

- . !

, , , MVP-. , , .

, , http://github.com/shwars/braillehack. , , " " — , . , - — , http://soshnikov.com.

! , - , , , - .