Using geospatial data can quickly become overwhelming as the volume and variety of data increases. Maxar’s software engineers and geospatial data scientists are creating innovative visualization techniques by leveraging and contributing to the open-source community to turn geospatial and geotemporal vector data into visual narratives. These narratives make it easy for analysts to understand the overall situation and derive actionable insights.
Traditionally, Maxar has relied heavily on GeoServer, an open-source server for sharing geospatial data that integrates easily with a variety of databases, allowing our teams to provide tiles of data using the Open Geospatial Consortium’s Web Map Service standard. While GeoServer still plays a role, my team increasingly relies on Datashader, an open-source Python library that enables creation of meaningful representations of large datasets quickly and flexibly.
Many developers and data scientists are familiar with Datashader and its ability to render large datasets dynamically in a Jupyter notebook—but that capability requires end users to be comfortable working with Python. In many cases, while these professionals have the technical acumen, they may not have the time to incorporate a new tool into their day-to-day workflow. How do we bring that visual narrative to our customers without increasing their workloads?
Answer: Tile Map Service (TMS) tilesets!
Datashader can export Mercator Tiles to disk, which allows the tiles to be easily incorporated into existing mapping applications that our customers already use, like web clients built using OpenLayers and Leaflet. As part of Maxar’s continued support of the open-source community, I contributed an update to Datashader’s 0.11.0 release to improve rendering performance. The rendering times improved significantly: roughly 10 times faster at level 2, where the globe is composed of 16 tiles, and three times faster at level 8, where the globe is composed of 65,536 tiles.
Visualizing geospatial data with Datashader
Let’s see this process in action using a simulated traffic dataset near Maxar’s office in Herndon, Virginia. The code below reads the traffic dataset from a file and supplies it to Datashader, specifying which zoom levels of the TMS tileset to export and how to style those individual tiles.
import dask.dataframe
import datashader
from datashader.tiles import render_tiles
from holoviews.element.tiles import lon_lat_to_easting_northing
import colorcet
def _get_extents():
return df.x.min().compute(), df.y.min().compute(), df.x.max().compute(), df.y.max().compute()
def _load_data_func(x_range, y_range):
return df.loc[df.x.between(*x_range) & df.y.between(*y_range)]
def _rasterize_func(df, x_range, y_range, height, width):
cvs = datashader.Canvas(x_range=x_range, y_range=y_range,
plot_height=height, plot_width=width)
agg = cvs.points(df, 'x', 'y')
return agg
def _shader_func(agg, span=None):
img = datashader.tf.shade(agg, cmap=colorcet.fire)
return img
# Can be utilized to customize image with watermark, etc.
def _post_render_func(img, **kwargs):
return img
if __name__ == '__main__':
df = dask.dataframe.read_csv('Herndon.csv', usecols=['lon', 'lat'])
# Coordinates should be in Mercator format.
df['x'], df['y'] = lon_lat_to_easting_northing(df['lon'], df['lat'])
min_zoom = 5
max_zoom = 14
output_path = 'tiles'
render_tiles(_get_extents(),
range(min_zoom, max_zoom + 1),
load_data_func=_load_data_func,
rasterize_func=_rasterize_func,
shader_func=_shader_func,
post_render_func=_post_render_func,
output_path=output_path)
Voila! Below is the visualization of the traffic dataset created by the above code: the more cars on the roadway, the darker the shade of red on the map, enabling the end user to quickly identify what parts of Herndon are busiest. Although this script is simple, it allows a lot of flexibility to customize input and output.
The resulting graphic brings geospatial insight (in this case, which roads in our simulated dataset have the densest traffic) to our customers. But data isn’t static. At least, it usually isn’t. A lot of geospatial data changes over time. Whether that’s traffic in motion or construction activity creating new buildings, it’s important to highlight the geotemporal nature of the data—how the data changes across area and time.
In the case of our simulated dataset, for a city planner, it may be important to understand which roads are the most congested overall. As a commuter, I’d like to understand how that congestion changes over time so I can plan departures and routes accordingly.
Generating geotemporal visuals
What if we used Datashader to render a stack of images into a GIF or MP4? This is the concept that pushed me to develop a new open-source Python library, Motionshader. Let’s see a simple example that illustrates the core concepts of the library:
import multiprocessing as mp
from datetime import datetime, timedelta
import dask.dataframe
import requests
from PIL import ImageFont
import colorcet
import motionshader
if __name__ == "__main__":
# Motionshader expects a time indexed, sorted, Dask DataFrame with columns containing EPSG:4326 coordinates.
# Motionshader is opinionated about using Dask DataFrames, for scaling this process to big data.
df = dask.dataframe.read_csv("Herndon.csv", usecols=["time", "lon", "lat"])
df["timestamp"] = dask.dataframe.to_datetime(df["time"], unit="ms")
df = df.set_index("timestamp", npartitions=mp.cpu_count(), compute=True).persist()
# Define a Basemap using a WMS Service and associated layer. Assumes EPSG:4326.
# Provide a requests.Session() that can be customized for authentication or 2-way SSL verification.
basemap = motionshader.Basemap(
requests.Session(), "https://ows.terrestris.de/osm/service", "OSM-WMS"
)
# Define the Dataset, providing a Dask DataFrame and the names of the longitude and latitude columns.
dataset = motionshader.Dataset(df, "lon", "lat")
# Define the MotionVideo, providing the dataset to be rendered and the basemap service.
motion_video = motionshader.MotionVideo(dataset, basemap)
start_datetime, end_datetime = df.index.min().compute(), df.index.max().compute()
min_longitude, max_longitude = df.lon.min().compute(), df.lon.max().compute()
min_latitude, max_latitude = df.lat.min().compute(), df.lat.max().compute()
# GeospatialViewport allows defining where the 'camera' looks on the globe, and the resolution of the output.
# TemporalPlayback defines the temporal bounds, temporal size of a frame, and the frames per second for the output.
# In this example, a single frame contains 30 minutes of data, and steps forward 15 minutes between frames.
viewport = motionshader.GeospatialViewport(
min_longitude, max_longitude, min_latitude, max_latitude, 1920, 1080
)
playback = motionshader.TemporalPlayback(
start_datetime, end_datetime, timedelta(seconds=300), timedelta(seconds=300), 1
)
# If a FrameAnnotation is provided to the rendering function, the time range and center coordinate will be
# added onto each frame. The FrameAnnotation allows customizing the position of the label in pixel coordinates,
# the font, the font color, the coordinate order, and the date format.
annotation = motionshader.FrameAnnotation(
10, 10, ImageFont.truetype("arial", 14), "#000000", True, "%Y-%m-%dT%H:%M:%S%z"
)
# If a Watermark is provided to the rendering function, the watermark text provided will be added
# onto each frame. The Watermark allows customizing the position of the watermark in pixel coordinates,
# the font, and the font color.
watermark = motionshader.FrameWatermark(
"Rendered using Motionshader",
10,
viewport.height_pixels - 40,
ImageFont.truetype("arial", 14),
"#000000",
)
# MotionVideo can be output as either a GIF or an MP4.
motion_video.to_gif(
viewport, playback, "Herndon", annotation, watermark, 1, colorcet.fire
)
motion_video.to_video(
viewport, playback, "Herndon", annotation, watermark, 1, colorcet.fire
)
This code creates the following GIF, allowing an end user to see how traffic patterns change in the region over the course of a single day.
By showing our customer this visualization, we can tell them a story and highlight activity of interest without overburdening them with millions of data points that they must consume using highly specialized, resource-intensive tools to get to the same conclusions. Visualizations like this enable us to help our customers gain insights faster from geospatial and geotemporal data for better decision-making.
If you’re a developer or an aspiring college graduate with a passion for geospatial visualization and open-source development, learn how you can start your career at Maxar.