Clever Geek Handbook

Aspects of accounting and search for geographic information objects using MongoDB

Online course :  "OTUS.NoSQL" .





Project : https://github.com/BorisPlus/mongodb_geo





Introduction

Geoinformation  is any information that reflects the location, shape and size of an object (hereinafter referred to as a geo object). It is recorded in cartography, geology, meteorology, land management, ecology, municipal administration, transport, economics, defense and many other areas. Geoinformation is an integral part of the so-called Big Data, which leads to the need to develop tools for its analysis and visualization.





The academic approach of writing articles meant "information reflecting the properties ... of objects of the material world." However, in practice, there was the fact that the enthusiast overlayed Google maps through the standard API of drawings with the topography of Middle-earth and the construction of routes for the heroes of J. Tolkien, which is not entirely "material". Another junction with the intangible is an example of datasets like  GeoIP ,  E.164 ,  ABC .





The research result is a tool for displaying information about geo objects stored in MongoDB on a map using web access. The front end is implemented using  Leaflet  (an open source JavaScript library for mobile interactive maps) and a set of related routines to asynchronously retrieve information from the back end. The service is developed on the basis of the "Dummy WSGI Framework"  software constructor   ( repository ) created earlier in the course  "OTUS.Web-python" in the Python3 programming language with the use of WSGI.





MongoDB. .





,  MongoDB  .    . , MongoDB    , .





- MongoDB

MongoDB   : , , .





, :  ,  ,       .





. . , , , - ident



. MongoDB    :





<field>: { type: <GeoJSON type> , coordinates: <coordinates> }

- .





, , , , , .    . - ( . -).





- , , . "", ( ) ().





, MongoDB . 
mongo  192.168.102.99  --port 49326 
---
> use otus 
switched to db otus 
> db.dropDatabase() 
{ "dropped" : "otus", "ok" : 1 } 
> use otus 
switched to db otus 
> db otus 
> show collections

()

   . , :





> db.meteorites.createIndex( { "ident": 1 }, { unique: true } )
> db.meteorites.createIndex( { "location" : "2dsphere" } )

 geolocation



. "". , ,  location



, :





location: { type: 'Point' , coordinates: [ LON, LAT ] }

:





mongoimport --host 192.168.102.99  --port 49326 \
--db otus --collection meteorites --jsonArray \
--file ./foreign/meteorites/data.json

2021-03-28T10:28:09.443+0300    connected to: mongodb://192.168.102.99:49326/
2021-03-28T10:28:12.443+0300    [<span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);"><span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);">#</span>##.....................] otus.meteorites      1.62MB/10.1MB (16.0%)</span>
2021-03-28T10:28:15.443+0300    [<span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);"><span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);">#</span>########...............] otus.meteorites      3.97MB/10.1MB (39.4%)</span>
2021-03-28T10:28:18.443+0300    [<span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);"><span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);">#</span>###########............] otus.meteorites      5.39MB/10.1MB (53.4%)</span>
2021-03-28T10:28:21.443+0300    [<span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);"><span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);">#</span>################.......] otus.meteorites      7.23MB/10.1MB (71.6%)</span>
2021-03-28T10:28:24.443+0300    [<span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);"><span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);">#</span>####################...] otus.meteorites      8.83MB/10.1MB (87.5%)</span>
2021-03-28T10:28:27.443+0300    [<span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);"><span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);">#</span>######################.] otus.meteorites      9.71MB/10.1MB (96.3%)</span>
2021-03-28T10:28:28.453+0300    [<span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);"><span class="pl-c" style="box-sizing: border-box; color: var(--color-prettylights-syntax-comment);">#</span>#######################] otus.meteorites      10.1MB/10.1MB (100.0%)</span>
2021-03-28T10:28:28.454+0300    45716 document(s) imported successfully. 0 document(s) failed to import.

, 45716 , (  Meridiani Planum), (  Can't extract geo keys: ... longitude/latitude is out of bounds, ...



, ).





db.meteorites.remove({"ident" : "32789"});

7315 , . - . , .





db.meteorites.updateMany( 
    {"geolocation":{$exists:true}},
    [{
        $set: {
            "location" : {
                "type": "Point",
                "coordinates" : [ 
                    { $toDouble: "$geolocation.longitude" } , 
                    { $toDouble: "$geolocation.latitude" } 
                ]
            }
        }
    }]
);

MongoDB  meteorites



   location



  38400 45716.





:     { , }



  MongoDB (If specifying latitude and longitude coordinates, list the longitude first and then latitude



). , Leaflet    - { , }



. , MongoDB "" { lon: , lat: }



. , . , MongoDB  { , }



.





()

   , . - API  WikiMapia.





WikiMapia , ( , ). ( ) API- .





, :





db.geo_wikimapia_polygons.createIndex( { "ident": 1 }, { unique: true } )
db.geo_wikimapia_polygons.createIndex( { "area" : "2dsphere" } )

  Python3  pymongo



.





Python-: - (UPSERT



) (bulk_write(instructions)



), ( ) . - .





, MongoDB "".  area



, (    -).  



  :





area: { type: 'Polygon' , coordinates: [[ [LON_1, LAT_1], [LON_2, LAT_2], ..., [LON_1, LAT_1] ]] }

WikiMapia:





python3 ./foreign/onetime_static_load_polygons_wikimapia.py 
Page 1 has docs count 50
Page 2 has docs count 50
...
Page 37 has docs count 35
Max page 37 with some data

MongoDB :





> db.geo_wikimapia_polygons.count()

1832 .





: MongoDB ( 3.1.6 RFC 7946 "GeoJSON" August 2016). , , , ( MongoDB  Edges <number K> and <number M> cross. Edge locations in degrees: [Kx1, Ky1]-[Kx2, Ky2] and [Mx1, My1]-[Mx2,My2]



). , , "", ( MongoDB  Loop is not closed



). WikiMapia . 1835 , (36 * 50 + 35 = 1835 ), 1832 .





()

   , "" .





: ( 3.1.4 RFC 7946 "GeoJSON" August 2016), ( ).





   Python3  requests



, pymongo



   multiprocessing



. ( ). . -. "" .   "", , .





Python-: , . "" .





MongoDB "".  path



, .  



  :





path: { type: 'LineString' , coordinates: [ [LON_1, LAT_1], [LON_2, LAT_2], ..., [LON_N, LAT_N] ] }

  : $geoIntersects



  $nearSphere



.





 $geoIntersects  , . , ( , ) () (), . , , .





: , " ". , , MongoDB.





 $nearSphere  MongoDb "" ( - ).





 $geoWithin ( , )  $near ( ) .





 $near



  $nearSphere



  , " ", ($maxDistance



) , ($minDistance



). , "" : ".", "" , .





Web- "Dummy WSGI Framework" (), ( 45 ).





pip3 install -r ./service/requirements.txt
uwsgi --http 127.0.0.1:8080 --wsgi-file ./service/application.py

 http://127.0.0.1:8080  . -, . .





: GIF-  Peek.





( . - "7") .





, . . , "" , Leaflet .





NASA 64 , . ( ).





.





> db.meteorites.find({"location.coordinates": [13.43333,58.58333] }).count()
    64
> db.meteorites.find({"location.coordinates": [13.43333,58.58333] }, {name: 1, _id: 0})
    { "name" : "Osterplana" }
    { "name" : "ร–sterplana 002" }
    { "name" : "ร–sterplana 003" }
    ...
    { "name" : "ร–sterplana 064" }

"" "ร–sterplana",   (.).





:





db.geo_yandex_taxi.deleteMany({})
db.geo_yandex_taxi.createIndex( { "ident": 1 }, { unique: true } )
db.geo_yandex_taxi.createIndex( { "last_point" : "2dsphere" } )
db.geo_yandex_taxi.createIndex( { "path" : "2dsphere" } )

:





python3 ./foreign/upsert_yandex_taxi_loop.py 
9       2.6140940189361572
9       2.481816291809082
9       2.528238296508789
9       2.374605894088745
9       2.5337154865264893
9       2.7297616004943848
9       2.60577392578125
9       2.586944818496704
9       2.5660433769226074

"" :





{'id': 'bcc095db8e3b56e057caebdb97af5694', 'display_tariff': 'business', 'free': True, 'static_icon': False, 
'positions': [{'lon': 30.326291, 'lat': 59.974395, 'direction': 50.0, 'timestamp': '2021-03-24T23:49:01.000000+0000'}, 
{'lon': 30.326291, 'lat': 59.974395, 'direction': 50.0, 'timestamp': '2021-03-24T23:48:52.000000+0000'}, 
{'lon': 30.326291, 'lat': 59.974395, 'direction': 50.0, 'timestamp': '2021-03-24T23:48:43.000000+0000'}, 
{'lon': 30.326291, 'lat': 59.974395, 'direction': 50.0, 'timestamp': '2021-03-24T23:48:34.000000+0000'}]}

, "" .





.





. ( "" ).





 http://localhost:8080/circle/  , , .





:  near



  nearSphere



   2d



  2dsphere



, :





error processing query: ns=otus.geo_wikimapia_polygonsTree: GEONEAR  field=area maxdist=500 isNearSphere=0
Sort: {}
Proj: { _id: 0 }
 planner returned error :: caused by :: unable to find index for $geoNear query, full error: {'ok': 0.0, 'errmsg': 'error processing query: ns=otus.geo_wikimapia_polygonsTree: GEONEAR  field=area maxdist=500 isNearSphere=0\nSort: {}\nProj: { _id: 0 }\n planner returned error :: caused by :: unable to find index for $geoNear query', 'code': 291, 'codeName': 'NoQueryExecutionPlans'}

.





 base_config.py



  ("", ""), MongoDB ("meteorites", "geo_wikimapia_polygons", "geo_yandex_taxi") ("location", "area", "last_point", "path"), GeoJSON- ("Point", "LineString", "Polygon"), :





...
MONGODB_DB_COLLECTIONS = dict(
    static={
        "meteorites": {
            "location": POINT_OBJECT,
        },
        "geo_wikimapia_polygons": {
            "area": POLYGON_OBJECT,
        },
    },
    dynamic={
        "geo_yandex_taxi": {
            "last_point": POINT_OBJECT,
            "path": LINE_STRING_OBJECT,
        },
    },
)
...

, ("", ""), ("", "", "") , MongoDB .





, HTML- JavaScript, . .





, HTML- .





"" AJAX

JavaScript (AJAX). , "", , . . AJAX-, .





 function get_data(...){
    ...
    if (xhr && !(xhr.readyState === 4)) {
        xhr.abort();
        console.log('Previous AJAX #' + xhr.__dt + ' was aborted');
    }
    clearTimeout(timer);
    xhr = new XMLHttpRequest();
    xhr.responseType = 'json';
    xhr.__dt = Date.now();
    console.log('Start AJAX #' + xhr.__dt);
    timer = setTimeout(function() {
        // find objects in area.
    }
}

AJAX- web- "" (F12).





MongoDB WGS84 (MongoDB geospatial queries on GeoJSON objects calculate on a sphere; MongoDB uses the WGS84 reference system for geospatial queries on GeoJSON objects



) (    "wgs84").





Leaflet - EPSG 3857.





, EPSG 3857   85.06ยฐS



  85.06ยฐN



.





Leaflet CRS  "L.CRS.EPSG4326",    MongoDB.





, MongoDB . , , Leaflet , :





pymongo.errors.OperationFailure: longitude/latitude is out of bounds, lng: 561.213 lat: 89.9823 ... 
Valid longitude values are between -180 and 180, both inclusive.
Valid latitude values are between -90 and 90, both inclusive.

, "" ( , "" ""), - "" ( ).





ฮฒ-

(   MongoDB  4.0) .





, , MongoDB. .





, Leaflet . , - (, , , , ) ( , - , (, ) ).





, .





, "".





Thanks to the courses  "OTUS" ( "OTUS.Web-python" 2018 and "OTUS.NoSQL" 2020) for the acquired experience in Fullstack development (in particular, the integration of Python, Javascript and MongoDB).







More articles:


All Articles