Exceed the speed limit

-

Speed Limit
Last week I worked successfully on tuning and improving the performance of the reverse geocoding engine. We have used a tool, named K6, to do some load tests. The test consists of doing massive requests to the service using a customized script to generate random geocoding queries all around the world. The first results showed a speed of about 200 queries/second as about 800K in an Hour.
Working on tuning SQLite compile and configuration parameters we are able to surpass 1 Million queries in an Hour, about 300 geocoding queries/seconds.
The decision to use SQLite than other DBs, like PostgreSQL, was deliberately done by design. There is a small presentation that explain this decision. We have also fixed a problem of instability of the service (crash), that emerged during the tests. We apologize to our users or customers (apparently no one 😇) for the abrupt interruption of the service caused by our stress tests.
As I'm writing this text, a big load test of 24 hr is undergone...
 

A snapshot of the ongoing load test of the Feroeg - reverse geocoding - service.

There are passed about 4hrs and 4.5M queries are done. Apparently no problem regarding memory leaks of the georef_server process. The one 'problem' I noticed is in the testing tool side: the memory grows as time passes. Tomorrow we will update this post reporting the definitive speed results of this test. 

After 10 hours of run ... 10.7M queries were done ... same memory pressure ...


Final report of the 24 Hrs stress test. 

After one day of running the backend has served 25.1M of reverse geocoding query. That is 1.04 millions of queries/hour. The memory usage by the main process 'georef_server' is about 250 MB. The only memory unresolved issue was 5 GB less of free space on SSD. I had to investigate it (*).
* The issue of less space on the HDD was related to logging files, generated from Nginx. The retention policy was 1 week. After few days, the logging rotation policy automatically freed up the space.  







Comments

Post a Comment

Popular posts from this blog

FEROEG a free service of Reverse geocoding

-
Image
Good morning. In this first post, I want to introduce Feroeg, a free service of reverse geocoding. Reverse geocoding is the process of converting a location as described by  geographic coordinates  (latitude, longitude) to a human-readable address or place name. So, if you think that your company, or yourself, can benefit from this kind of service, please give it a try visiting my web site, www.feroeg.com . The data are taken from  OpenStreetMap . At the moment the service is at the starting phase. Other features and improvements will be added, as the posts in the blog. 
Read more »

The open-source GeocodingLite project is powered by Feroeg.com

-
Image
"GeocodingLite is the open-source geocoding server engine for getting the GPS coordinates of a city/location all around the world." With these words, we present our first release of an open-source geocoding engine on GitHub at https://github.com/Enoch72/GeocodingLite . You can launch a textual search to get the coordinates and/or the postal codes and other info about the wanted location. As the name suggests the project is based on the SQLite database and is a very successful attempt to explore the potential of the SQLite FTS5 search text engine. The data comes from GeoNames. They contain 'only' the centroids (the geographic centre point) of the cities of about 100 nations of the world and other useful pieces of information, such as postal codes. The search engine is 'lite' also because the search can't resolve address coordinates at street level. The GitHub repository contains the source code (C++) of the search server and all the data needed to run the ...
Read more »

Feroeg API - Available in RapidAPI Hub

-
Image
From now the Feroeg reverse geocoding service is available from the RapidAPI Hub located at https://rapidapi.com/hub . RapidAPI is a web hub allowing businesses to connect disparate API gateways, publish APIs deployed across multiple environments, and more. The service is offered free of charge. It's not requested to give credits to Feroeg.com. The user can query at maximum 50000 requests per day. The rate limit is 60 queries/minute. The advanced batch geocode feature will not be available using the API from RapidAPI but is accessible querying directly at our website.    We think that the web visibility of the www.feroeg.com site will be improved in the next months.
Read more »