Project Proposal: Map Tile Distribution and Master Directory of Map Tile Distributors

By agron on February 12, 2012

alt During a recent mapping project implementation at work I realized the open source map content is limited and that the resources to share that content are even more limited. There a few map tile servers out there all of them using resources of open source volunteers, and those servers are being hammered by millions of users. The map content is not usualy a light content, it’s all bitmap images the are already compressed.  Many servers go offline because they can’t take the load or that they have reached their maximum bandwidth for the day.

The number of people volunteering to host the tiles is small because the tiles ocuppy about 1.2 TB of disk space . But that’s for the whole world.

Here’s the solution.

  • Let’s create a subdomain mdmtd.flossk.org  (for Master Directory of Map Tile Distributors)
  • Let’s create a little PHP script that allows volunteers fetch a protion of tiles from the main servers directly into their servers. Everyone needs to run this script on their own servers, we will share it when a new volunteer is registered.
  • Let’s also create a database to tell the world that this new regional tile distributor is mirroring tiles for x,y region with radius of m kilometers.
  • Let’s create and submit  patches for a few major javascript layers, like OpenLayers, Mapquest for OSM etc. so that these scripts lookup our database at mdmtd.flossk.org.

These volunteers most likely have created a program or a web application that may be specific to a region. Let’s take http://prishtinabuses.info/ for example. Why would thousands of Prishtina citizens download Prishtina maps from California? Well, first of all there is no map tile server in Prishtina. Even if there was one, OpenLayers doesn’t know about it.

Also, there seem to be a lot business applications that are used in large corporations inside their intranets. We can offload these corporations easily by redirecting them to their internal map server.

And just before I finish for tonight let me just say what we need:

  • Issue tracker
  • Bugzilla
  • GIT or SVN server
  • Subdomain (to be under FLOSSK for next thousand years)
  • Mysql server
  • Apache server
  • 2 PHP programers
  • 2 Javascript programers
  • Web artist
  • Technical writer
  • Blogger
  • Mailing list
  • Support forum

This of course is a summary of an entire mapping ecosystem. I will continue with detailed design description for each item in the next blog/s. 

ClientSide Javascript - Getting the list of tile servers

This script is most likely going to be a patch for OpenLayers. This script is responsible of making an http request to the Master Tile Directory mdmtd.flossk.org with the following information: mdmtd.flossk.org/getServerList.php

Longitude, Latitude, Radius, Zoom level, Tyle Style getServerList.php will respond with a list of servers in json format in the following structure:

{
     "query": "Longitude, Latitude, Radius, Zoom level, Tyle Style",
     "serverList":
     [{
     "ID" : uuid,
     "Name": "Agrons New York Tile Server",
     "URL" : "http://url.toServer.com/TileStyle/GreenTileStyle",
     "Bandwidth"      : 150,
     "Speed": -1},
     {
     "ID" : uuid,
     "Name": "Obamas New York Tile Server",
     "URL" : "http://url.toAnotherServer.com/TileStyle/GreenTileStyle",
     "Bandwidth"      : 45,
     "Speed": -1}
     ],
     "mainServerList":

	[{
"ID" : uuid, "Name": "University of Linux Programmers", "URL" : "http://url.toMainServerClone.com/TileStyle/GreenTileStyle", "Bandwidth" : 50, "Speed": -1}, {  "ID" : uuid, "Name": "Main OSM Tile Server", "URL" : "http://tah.openstreetmap.org/Tiles/tile/",
     "Bandwidth"      : 10,
     "Speed": -1}
     ]

	 
}

This script will, if local storage is enabled, test the servers on the list for speed and update the serverList[n].Speed value and then save the array to the local storage. Current time and date will be stored as well as the group Longitude, Latitude, Radius, Zoom level, Tyle Style as a reference key.

When local storage is enabled, it will be used first.

An example of localStorage:

var testObject = { 'one': 1, 'two': 2, 'three': 3 };
// Put the object into storage
localStorage.setItem('testObject', JSON.stringify(testObject));
// Retrieve the object from storage
var retrievedObject = localStorage.getItem('testObject');
console.log('retrievedObject: ', JSON.parse(retrievedObject));

The way getServerList.php generates the server list array is with most preferred first. If this client side Javascript uses the first list on the server it should be fine, but it should also test the speed from it’s own location.

ClientSide Javascript - Choosing the fastest tile server

This script will send asinchronus http requests to every server on the list and it will donwload tile 0.png. It wil then compare and save the choice in a local storage file for at least a week. After a week this script should run again, but we need to let the user know that this script is going to take a minute.

PHP Script for managing the Volunteer Server

This script is something like webmin or myPhpAdmin, but it’s duty is to provide simple tile managing tools over the web. We will call it myTileAdmin.php (all in lowercase). This script will be distributed from mdmtd.flossk.org and it will provide the following functions:

  • Link to volunteers account at mdmtd
  • configure the bandwidth and let mdmtd know about it
  • choose regions and styles volunteer wishes to copy and share -create the directory structure of the tiles
  • do the actual copy/mirroring of tiles of and keeping them organized in styles
  • notify mdmtd that a new tile update is available
  • listen to notifications from mdmtd

All information needed will be saved on the volunteer server in a .conf file.

It is important to stress that each of the volunteer servers will be serving multiple regions.

Linking to account at mdmtd

A volunteer will initially register at mdmtd.flossk.org by creating a user account and choosing a password. Ideally this would an OpenID solutions but a simple username or email and a passord will do. With myTileAdmin.php user will enter the same credentials as in mdmtd. The script will encode the password and save it in the .conf file.

Setup

myTileAdmin.php will ask the user for the ammount of the bandwidth he/she wishes to share, styles and regions. Also, the desired location on the local storage. On submit / save the script will generate a new .conf file for apache so that the new folder and files (soon to be copied) are accesable by the world.

Choosing the regions

This page show an OpenStreet Map that will allow the user to zoom, pan and put a circle and set its radius on the map. The user will be able to put any number circles on the map. He can also delete them. When the page is saved the regions are saved locally in .conf file as well as at mdmtd.

Back to updates