Introduction
Data is everywhere, all the time. The overabundance of data streams can be not only personally exhausting but can also undermine otherwise successful sensor-based projects. You can have the best sensors and the most intelligent sensing design, but if the data coming from those sensors isn’t used intelligently, the project will not live up to its potential.
Dashboards are an effective way to visualize data. The best dashboards aggregate live streams into cohesive data insights that deliver the most important information upfront, with compelling associated visuals.
In this tutorial, we’ll develop a “live” dashboard — that is, one that is constantly refreshing with new data. We’ll do this using a programming language called Processing . It’s built on Java, but if you have experience with Arduino it’s pretty straightforward (with only several weeks of Arduino experience, I was able to pick up Processing within a couple of days). In this tutorial we’ll be pulling data from Coindesk’s API to build a dashboard that gives us the latest price of Bitcoin in US dollars. I chose this API because it is free and open (no authentication required), but most importantly, this data is considered “high velocity,” which means it’s constantly changing.
What does this mean for the Arduino community and sensor data? Well many of the highest velocity data streams are those built on sensors. High-velocity data goes hand-in-hand with Arduino — any sensor data that populates in the Serial Monitor frequently could be effectively handled and displayed using the tools here. The data format will likely be different for Arduino’s as well. Rather than JSONs from an API, you might be handling CSVs processed from the serial monitor. The tools here are flexible and should be useful for any data format.
So using these tools, we can build dashboards for any sensed data. We can then process and visualize that data in real time. That is extremely significant. Let’s dive in.
Parts List
This is a software tutorial so your parts list is digital and short. Really short.
- Processing 3
- Data stream (the idea here is the stream updates every few seconds)
The data
As noted above, the data in this tutorial comes from an API. It’s in a JSON format, a very common data structure. You can read more about handling JSONs in Processing here. Several of the functions described in the link are used in this tutorial. They are all part of Processing’s base code, so no need to worry about loading additional libraries.
Code
Let’s jump into the code. Beyond the standard format of a Processing sketch (which is similar to Arduino), it’s important to note that this sketch makes use of the thread structure. Read through the linked site, but essentially threads are an important tool here in allowing us to refresh the data stream asynchronously from our dashboard. Essentially, we don’t want the data pull (and any problems that might occur there) to slow down our dashboard. In the event that our data stalls out for whatever reason, our dashboard will still function.
To support this threading structure, the sketch has a custom function, retrieveData, where all the data retrieval and calculation is performed. This is important, because it allows the draw function in the sketch to only visualize the dashboard. This is a strategic decision to enhance sketch & dashboard performance.
Finally, the code includes some aggregate statistics that are computed “on the fly”. In short, an array is created to store the 10 most recent values (this number can be modified in the global variable section), and then the mean and standard deviation of those values is calculated. We can imagine many situations where this could be useful: perhaps the most recent data point isn’t the best metric to display, but rather a smoothed average of recent data points. Or perhaps you want to compute the standard deviation as a way of checking incoming data points for outliers. Depending on the types of sensors being used in a project (and features such as the sensor sensitivity) and the responses, these types of aggregations can be useful in contextualizing real-time data streams.
Okay, enough preamble. Here’s the code.
// Sketch to build a real-time dashboard in Processing.
// Author: Tyler Bradford
// Tutorial to build a local responsive dashboard based on real time data. Sample data is pulled from an API feed, but this example could be used to build visuals on any type of live data feed, including sensed data from an Arduino. The dashboard populates new data every second and the display adjusts based on preset thresholds (i.e., colors indicate when the data is in a certain range). Aggregate metrics (mean and standard deviation) are calculated to support live data cleaning, in anticipation of use cases where outlier data may need to be handled separately.
// This tutorial pulls data in real time from Coindesk's API which can be found here: https://api.coindesk.com/v1/bpi/currentprice/USD.json
// Per Coindesk's terms of use, I am obliged to say this is "Powered by Coindesk" and more info can be found here: https://www.coindesk.com/price/bitcoin.
// This tutorial also uses some sample code from tutorials found here: https://www.processing.org/tutorials/data/
// Declare some global variables
JSONObject response; // To store data from the API
// Variables that will be incorporated into the dashboard
String updatetime;
String description;
String rate;
String today;
String now;
Float rate_float1;
double std_dev;
PFont f; // A font variable for the dashboard
// Aggregate statistics metrics to real-time data cleaning
int moving_window = 10; // Set the length of a moving window that will define an array of recent prices. This array will be used to calculate a rolling recent average price.
float prices[] = new float[moving_window]; // Initiate the array. The length is defined by the variable set in the line above.
float price_mean = 0; // Intiate the average that will be defined later
void setup() {
size(1600,950); // Initialize the size of the dashboard
f = createFont("Arial",76,true); // Set font type and size
thread("retrieveData"); // Run the data pull in a thread. Important for asynchronous updating to avoid "stalling out" the dashboard.
}
// This function is where the data pull occurs. As noted above, it happens via a thread.
// All calculations will occur within this function as well, allowing the draw function to only display which will enhance performance.
void retrieveData () {
response = loadJSONObject("https://api.coindesk.com/v1/bpi/currentprice/USD.json"); // Initial pull
// Extract date and time from local machine to populate an update time in dashboard.
today = str(month()) + "/" + str(day()) + "/" + str(year());
now = str(hour()) + ":" + str(minute()) + ":" + str(second());
// Extract price information from the API. The API is structured with a series of nested JSON objects.
JSONObject bpi = response.getJSONObject("bpi");
JSONObject usd = bpi.getJSONObject("USD");
// Extract three strings from the USD JSON object that will go into the dashboard.
description = usd.getString("description");
rate = usd.getString("rate");
rate_float1 = usd.getFloat("rate_float"); // This float will be used for calculating aggregate metrics in real time.
// Define some aggregate metrics.
float total = 0;
double sd = 0;
prices[moving_window-1] = rate_float1; // Set the last value in the array to the most recent price recording
// Sum up all the values in the array to generate a total
for (int i = 0; i < moving_window; i++) {
total = total + prices[i];
}
price_mean = total / moving_window; // Calculate the average value by dividing the total by the number of values
// Calculate the variance of the recent values. This will be used to check for outliers
for (int i = 0; i < moving_window-1; i++) {
//std_dev += square(prices[i] - price_mean) / moving_window;
sd += ((prices[i] - price_mean)*(prices[i] - price_mean)) / (moving_window); // Variance
prices[i] = prices[i+1]; // Also move the values down here. This prepares the array for the next price recording.
}
std_dev = Math.sqrt(sd); // Calculate standard deviation
}
void draw() {
retrieveData();
// The following if statements create conditional formatting for the dashboard based on preset thresholds. In this case, the background color is based on the bitcoin price. Three color levels have been set: green for good, orange for caution, and red for danger. The idea here is to set an intermediate step that warns the user when the key metric is rising or falling to a certain level before it actually hits a danger area.
if (rate_float1 < 6000) {
background(220,20,60); // red
}
else if (rate_float1 < 6955) {
background(255,140,0); // orange
}
else {
background(0,150,0); // green
}
textFont(f,76); // Set the font size
fill(0); // Set the font color
// The following are all the items that are presented on the dashboard. Item titles are positioned on the left and values are on the right.
// The dashboard includes when it was updated, the current price, the displayed price's currency, and the aggregate metrics.
text("Last Update Date:",10,100); // Title on the left (the second parameter indicated the x position)
text(today,810,100); // Value on the right
text("Last Update Time:",10,250);
text(now,810,250);
text("Current Price:",10,400);
text(rate,810,400);
text("Average Price:",10,550);
text(price_mean,810,550);
text("Standard deviation:",10,700);
text(String.format("%.3f", std_dev),810,700);
text("Currency:",10,850);
text(description,810,850);
delay(1000); // Refresh the data (i.e., pull and redraw dashboard) every second.
}Final product
The GIF below shows a sample of the data stream. Of note here is the change in background color when the bitcoin price increases above the preset threshold. Average price and standard deviation are being calculated in real time. The update time (which is pulling from the local machine) indicates when the last pull was, which is every second. The reason the price doesn’t change with every update is due to the API refresh rate. An important consideration to be had with any project — perhaps you don’t need to refresh your dashboard more frequently than the underlying data sources refreshes. I set it to a high frequency here as proof of concept.
Happy dashboarding!