NPM module array-correl

NPM package Build Size

This npm module generates an array (of arbitrary size) of pairs with numbers with a specified correlation. It also allows inspection of such array to determine the pearson correlation coefficient of the array’s value pairs. The generate method takes the input count (array length), desired correlation, mean and deviation, and from this it generates an array of numeric pairs that satisfies the input parameters. The inspect method inspects such array and provides several statistics, such as the actual pearson correlation coefficient, deviation, mean and extent. The only dependency is d3-array which could easily be factored away in a future version.

The module on npm here

The Repo is here

NPM module array-unsort

NPM package Build Size

This npm module unsorts (shuffles) an arbitrary array. The unsort method scrambles the input array and returns a new array. The unsortInplace method scrambles the input array in-place. Two algorithms are available: Fisher-Yates and Modified Fisher-Yates. The latter guarantees that no array element remain at the same array index after shuffling.

The module on npm here

The Repo is here

zero-config-react-webapp / zero-config-modern-webapp

Zero-config modern React and Js front-end web app templates, with Webpack (bundler), Babel (compiler), Eslint/Airbnb (strict code style linting), Jest (unit tests), and D3 (data visualization) – all pre-configured and ready to go. You just add your code!

In development mode, the webapp (served on localhost:8080) will reload after each file save.

Also, these cool new/newish javascript features are enabled: async/await, spread, generators, Map/Set, WeakMap/WeakSet, getters/setters and optional chaining (?.) – and work in legacy browsers!

The React Repo is here

TODO: Provide live demo using the repo’s production server component

Zero Config React Template

The Basic Javascript Repo is here

license-suspensions

The visualization demonstrates how to syncronize the state of two side-by-side MapboxGL-based maps. As the user interacts with one of the two maps, the state of the map (center position, zoom level, pitch and bearing) is dynamically copied to the second map (and vice versa). The code also demonstrates how to prevent call stack overflow due to recursive event handler triggering when the map state is updated.

The project was made for the East Bay Community Law Center’s “Back on the Road” initiative. The initiative’s purpose was to highlight injustices in how driver licenses were suspended for failure to appear in court or failure to pay traffic tickets. As the visualization shows, there is significant correlation between driver license suspensions and poverty level

The dataset is based on driver license suspensions from California DMV and East Bay Community Law Center. See the initiative here

See the project live here

The repo is here

dataset-insights

The objective with this project is to provide near-instanteous insights of the contents of numeric columnar datasets. The following information is calculted for each numeric column in the dataset: null count, unique values (% of total values), extent (min/max), histogram and quantiles.

When a column is clicked, a red banner is shown on top of the column (please see screenshot below). Scatter plots of the other numeric columns are then generated (with x values from the column and y values from the selected column). The pearson correlation coefficent is calculated and displayed and a red line is drawn on top of the scatterplot with a slope equal to the correlation coefficient. The line’s opacity is driven by the absolute value of the correlation coefficient.

Open the screenshot below in another browser window to see image details.

See the project live here

TODO: Publish the repo

geo-buffer

Creates “buffers” of arbitrary size around geojson polygons and multi-polygons, in this case parks in San Francisco. It is using MapboxGL and Turf. Each GeoJson object (SF park) is “enlarged” with the value (in meters) of the input range control. The visualization demonstrates how overlapping GeoJson objects can be merged.

Please note: the polygon merge process is currently unoptimized (meaning, for example, that each polygon is unnecessarily tested for overlap against all others), and will be improved by quad trees.

See the project live here

The Repo is here

canvas-in-svg

The visualization demonstrates the use of an embedded Canvas in an SVG element. While this example includes only a single SVG element, an embedded canvas may make sense when a number of SVG elements needs to be generated, with each requiring to render large datasets that would othwewise overwhelm the DOM.

In this implementation, up to 50,000 items can be generated and visualized in the embedded canvas with no impact to the DOM (other than creating the canvas). Mouse hit detection of data elements is performed with a 4x4 pixel zone under the current mouse position.

The app is using the array-correl npm module referenced at the top

See the project live here

The repo is here

Smaller D3 data viz demos

A variety of smaller projects, most of them using D3

See the Blocks here

infinite-scroll

React app demo of inifite scroll of images from the cat.api where only a finite number of images are kept in the browser. The project demonstrates loading resources on-demand during scrolling actions while keeping the browser memory requirement low. It also demonstrates how to keep the images in a stable position as the user is scrolling up/down using an API that provides random images.

The overlay window shows the currently visible images in yellow and the images kept in the browser in gray.

TODO: Provide live demo using the production server

The Repo is here

traffic-fines

Interactive map that shows traffic fines across the US. The visualization was created for the Laywers’ Committee for Civil Rights of the San Francisco Bay Area, to illustrate the high traffic fines in the state of California (please see screenshot) relative to other states. The user can select one of three data series (Fines for Redlight, Stop sign and Speeding violations). By hovering over a state, an overlay shows the specific data to that state. As the user is moving the mouse over the map, the legend at the bottom shows each state’s position.

TODO: Provide live demo

The Repo is here

network-viz

Network visualization of a simulated compute infrastructure with some 400 nodes/links in 3d using the 3d-force-graph npm module. As can be seen from the viz, some segments are disconnected from core infrastructure. The viz provides insights into the topology of a network that would be difficult to obtain through any other means.

See the project live here

How to use:

TODO: Publish repo

point-cloud-deck-gl

Visualization of a “point cloud” using DeckGL and MapBoxGL. Close to 300k vertices are rendered above the map. The user can zoom in/out, rotate the map and change the pitch angle with the keyboard.

See the project live here

TODO: Publish repo

event-hub

The event-hub project demonstrates how to create a real-time front-end event dispatcher and event aggregator server-side using websockets. The system is comprised of three main components:

The server is a combined HTTP and Websockets server. It delivers the required HTML files and related dependencies (assets and scripts). It also establishes a Websockets listener and dispatcher of Websockets messages. The server maintains a default configuration for event handling for each client. The server forwards to the analytics app the event summary messages produced by the client app. It also forwards configuration messages from the analytics app to the client app.

Please note that there may be multiple instances of client and analytics web apps, doesn’t matter – all information is distributed to the full ecosystem of attached apps using Websockets.

The Client, more specifically the eventhub is the most interesting piece of the project. The client loads an image with some buttons. Two scripts are used, the eventhub and events. The former is an event tracker/consolidator/forwarder, and the latter is an event generator.

The event manager eventhub.js upon init, first checks for Websockets support (and terminates if no such support is available). It then attempts to establish communication with the server over Websockets and sets an event handler for incoming Websockets messages. It then adds the eventhub function to the window object, making it available to all scripts running in the browser.

Run the client web app in a separate window. Then run the analytics web app in another separate window. Then trigger events in the client (by moving the mouse, clicking buttons, etc) and watch the events being consolidated and forwarded to the analytics app (via Websockets)

The Repo is here

sensor-vue

The sensor-vue project is a full stack implementation of:

  1. Sensor software running on a dozen remote Raspberry PIs
  2. An HTTP server implementation, using Node and Express
  3. A React webapp to view the collected sensor data

The Raspberry Pies are programmed to sample the various attached sensors (temperature, water, humidity and air pressure) every minute and to forward the collected data to the Server component (also every minute) via HTTP.

A flow control protocol is implemented between the pies and the server to eliminate any chance that the server becomes overloaded after a long server outage. In essence, the following occurs at every upload moment:

  1. The pie tells the server how many data packets are contained in the pie’s data buffer. Normally that would be one packet, but after a server outage, the pie’s buffer will contain a number of packets equal to the server’s (or communication link) outage in minutes
  2. The server directs the pie to send either all packets or a subset of the packets that may have stacked up
  3. The pie sends the number of packets it is allowed to send, in FIFO order
  4. The server stores the packets in a DB (currently file system)
  5. At each upload moment, the server directs the pie to update its internal clock to the time provided by the server, which is set to UTC
  6. While each pie is sampling its sensors and sends the data every minute to the server (as outlined above), to avoid that the server is hit simultaneously by all sensors attempting to push the latest data, there is a time offset in seconds from the beginning of the minute determined by the last nibble in the pie’s MAC address (time dithering).

In addition to the sensor data embedded in the data payload, a number of other values are provided in the payload, such the sequence number (which increments every minute), the current wifi link quality and bitrate, the uptime, the current temperature of the CPU, and sweep time (the time from probing the first sensor attached to the pie to obtaining the result from the last sensor)

The server also delivers the React web application on demand. The server and web app use Websockets for real time transmission of received sensor data to the web app. The server is also providing an API which the web app uses to retrieve non-real-time data

The system has been operational since 2017 and no data packets have been lost other than during power outage at the sensor locations. As of June 2020, the sensors have each delivered over 1.5 million packets to the server. Some pies died however, probably due to failure of the flash memory cards.

Open the screenshot below in another browser window to see image details.

Please note: Login is required to see sensor status. When logged in, in the debugger you can view the realtime sensor data, streaming over websockets.

TODO: Publish repo, open server for public access

real-time-chart-multi

The real-time-chart-multi project is a resuable Javascript component that accepts real time data from multiple data streams. The purpose is to show the arrival time of real time events (or lack thereof), piped to the browser (for example via Websockets). Various attributes of each event can be articulated using size, color and opacity of the object generated for the event.

See the project live here. Please wait for 30 sec to see events from the 2nd data stream

The Repo is here

vertex-generator

The vertex generator allows rapid capture of geo verticies of a simulated path of a vechicle or person. Each captured vertex is a geo location where the path is changing direction. A syntactically valid GeoJSON structure is produced when the button is clicked. The GeoJSON structure contains both the Points and LineString elements.

The geo locations and speed information can be used by a post processing tool to “fill in” the geo location between the verticies.

To use, create vertexes in the map by clicking. Then click the Get GEOJson button, and copy the just-created GEOJson structure to the clipboard. Then drop it into for example GeoJSONLint, to verify the integrity of the structure.

See the project live here

The Repo is here