coronavirus-charts

0.1.0-SNAPSHOT


Cite data associated with the novel coronavirus (covid-19)

dependencies

cheshire
5.10.0
metosin/jsonista
0.2.5
clojure.java-time
0.3.2
cprop
0.1.16
expound
0.8.4
funcool/struct
1.4.0
luminus-immutant
0.2.5
luminus-transit
0.1.2
markdown-clj
1.10.2
metosin/muuntaja
0.6.6
metosin/reitit
0.4.2
metosin/ring-http-response
0.9.1
mount
0.1.16
nrepl
0.6.0
org.clojure/clojure
1.10.1
org.clojure/tools.cli
1.0.194
org.clojure/tools.logging
1.0.0
org.webjars.npm/bulma
0.8.0
org.webjars.npm/material-icons
0.3.1
org.webjars/webjars-locator
0.39
org.webjars/webjars-locator-jboss-vfs
0.1.0
ring-webjars
0.2.0
ring/ring-core
1.8.0
ring/ring-defaults
0.3.2
tick
0.4.23-alpha
org.clojure/math.numeric-tower
0.0.4
hiccup
1.0.5
herb
0.10.0
com.cerner/clara-rules
0.20.0
metasoarous/oz
1.6.0-alpha6
enlive
1.1.6
com.rpl/specter
1.1.3
com.brunobonacci/where
0.5.5
clj-http
3.10.0
selmer
1.12.18



(this space intentionally left almost blank)
 
(ns coronavirus-charts.handler
  (:require
    [coronavirus-charts.middleware :as middleware]
    [coronavirus-charts.layout :refer [error-page]]
    [coronavirus-charts.routes.home :refer [home-routes]]
    [reitit.ring :as ring]
    [ring.middleware.content-type :refer [wrap-content-type]]
    [ring.middleware.webjars :refer [wrap-webjars]]
    [coronavirus-charts.env :refer [defaults]]
    [mount.core :as mount]))
(mount/defstate init-app
  :start ((or (:init defaults) (fn [])))
  :stop  ((or (:stop defaults) (fn []))))
(mount/defstate app-routes
  :start
  (ring/ring-handler
    (ring/router
      [(home-routes)])
    (ring/routes
      (ring/create-resource-handler
        {:path "/"})
      (wrap-content-type
        (wrap-webjars (constantly nil)))
      (ring/create-default-handler
        {:not-found
         (constantly (error-page {:status 404, :title "404 - Page not found"}))
         :method-not-allowed
         (constantly (error-page {:status 405, :title "405 - Not allowed"}))
         :not-acceptable
         (constantly (error-page {:status 406, :title "406 - Not acceptable"}))}))))
(defn app []
  (middleware/wrap-base #'app-routes))
 
(ns coronavirus-charts.sessions
  (:require
   [tick.alpha.api :as t]
   [clojure.pprint :refer [pprint]]
   [clj-http.client :as client]
   [clara.rules :refer :all]
   [clara.rules.accumulators :as acc]
   [coronavirus-charts.chartwell :as cw]
   [coronavirus-charts.charts :as charts]
   [coronavirus-charts.data :as external-data]
   [cheshire.core :as cheshire]
   [clojure.string :as string]))

Given an tick/inst, returns true if that time is within a tolerance (mins)

Helpers

(defn is-within-time?
  [time tolerance]
  (< (t/hours (t/between time (t/inst))) tolerance))

Given two tick/inst, returns true if they are on the same day

(defn is-same-day?
  [t1 t2]
  (= (t/date t1) (t/date t2)))
(defn is-date? [date-string]
  (let [date (try (t/date date-string) (catch Exception e false))]
  (if date
    true
    false)))

Given a url path, eg, /tt/us or /bar/en,us,tt/2020-03-28 return a flattened list of all arguments eg. ['tt' 'us']

This is an important function that needs work. The intention is that it reads the path, and then splits it into each discrete parameter to be inserted as individual facts in the session. Commas are valid seperators because it takes two actions to get to a slash when you're typing in Whatsapp, but commas are usually one press away. 📊📊.to/cn,us,es,tt would be a valid url

(defn parse-path
  [path]
  (flatten
   (map (fn [v] (string/split v #","))
        (string/split path #"/"))))

END Helpers

FactTypes TODO: Implement core.spec or schema on records.

(defrecord WebRequest [url])
(defrecord ParsedRequest [path argument])
(defrecord LocationRequest [path
                            location-name
                            country-code
                            location-report])
(defrecord DateRequest [path date])
(defrecord NavFragment [path nav-type body])
(defrecord ChartFragment [path chart-type body])
(defrecord RenderedPage [path html])
(defrecord GlobalReport [source-name
                         source-url
                         last-queried
                         latest])
(defrecord LocationReport [source-name
                           source-url
                           country
                           country-code
                           country-population
                           province
                           last-updated
                           coordinates
                           timelines
                           latest])

END FactTypes

Report Manipulation

Given the parsed json from the api call, return a LocationReport record

(defn create-location-report
  [r]
  (let [latest (:latest r)]
    (LocationReport.
     "Johns Hopkins University"
     "https://github.com/CSSEGISandExternal-Data/COVID-19"
     (:country r)
     (:country_code r)
     (:country_population r)
     (:province r)
     (t/inst (:last_updated r))
     (:coordinates r)
     (:timelines r)
     {:confirmed (:confirmed latest)
      :deaths (:deaths latest)})))

(def location-reports (map create-location-report (external-data/get-all-locations-jhu)))

(defn create-jhu-global-report
  [stats]
  (GlobalReport.
   "Johns Hopkins University"
   "https://github.com/CSSEGISandExternal-Data/COVID-19"
   (t/inst)
   stats))
(defn create-nav [nav-data]
  (let [locations (get nav-data "Location")
        dates  (get nav-data "Date")]
    (list
     (charts/make-nav (map :body locations) "locations" "Locations")
     (charts/make-nav (map :body dates) "dates" "Dates"))))

END Report Manipulation

RULES

Whenever there's a WebRequest, create a corresponding ParsedRequest

This only has to check for blanks because my parse function is bad.

(defrule parse-request
  [WebRequest
   (= ?url url)
   (= ?arguments (parse-path ?url))]
  =>
  (insert-all! (map (fn [arg]
                      (if (not (string/blank? arg))
                        (ParsedRequest. ?url arg))) ?arguments)))
(defrule print-args
  [ParsedRequest (= ?arg argument)]
  =>
  (println ?arg))
(defrule create-latest-chart-by-country-code
  [LocationRequest
   (= ?path path)
   (= ?report location-report)]
  =>
  (insert-all! (list
                (ChartFragment. ?path "bar" (cw/latest-bar ?report))
                (ChartFragment. ?path "table" (cw/latest-table ?report))
                (ChartFragment. ?path "source" (cw/source-box ?report)))))
(defrule create-home-page-charts
  [WebRequest
   (= ?url url)
   (or
    (= "/" ?url)
    (string/blank? ?url))]
  [?global-report <- (acc/max :last-queried :returns-fact true) :from [GlobalReport (= ?latest latest)]]
  =>
  (insert-all! (list
                (ChartFragment. ?url "global" (cw/global-bar ?global-report))
                (ChartFragment. ?url "table" (cw/latest-table ?global-report)))))
(defrule create-chart-page
  [?chart-fragments <- (acc/all) :from [ChartFragment (= ?path path)]]
  [?nav-fragments <- (acc/grouping-by :nav-type) :from [NavFragment (= ?path path)]]
  =>
  (insert! (RenderedPage. ?path
                          (charts/base-chart "Recorded Coronavirus (COVID-19) Cases"
                                             (create-nav ?nav-fragments)
                                             (map :body ?chart-fragments)))))

The intention is that if there is a single LocationReport that's within a tolerance of 48 hours, then we can assume that the dataset has been updated within 48 hours and then do nothing. else, in the case that we there are no valid reports, we query xapix and insert an entire list of records. I'm not sure if this implementation is correct.

(defrule update-location-reports
  [:not [LocationReport (is-within-time? last-updated 48)]]
  =>
  (println "doing an update")
  (insert-all! (map create-location-report (external-data/get-all-timelines-jhu))))

Checks if the argument parsed matches any country-codes

(defrule parse-locations
  [?parsed <- ParsedRequest (= ?arg argument) (= ?path path)]
  [?report <- LocationReport (= ?country-code country-code) (= ?country country)]
  [:test (or
          (= ?country-code (string/upper-case ?arg))
          (= (string/lower-case ?country) (string/lower-case ?arg)))]
  =>
  (insert! (LocationRequest. ?path (:country ?report) (:country-code ?report) ?report))
  (pprint (:timelines ?report))
  (insert! (NavFragment. ?path "Location" (:country ?report))))

How do we account for States?

Checks if the argument parsed matches a date int he form YYYY-MM-DD

(defrule parse-dates
  [ParsedRequest (= ?arg argument) (= ?path path)]
  [:test (is-date? ?arg)]
  =>
  (println "parsing dates")
  (insert! (DateRequest. ?path (t/date ?arg)))
  (insert! (NavFragment. ?path "Date" (t/date ?arg))))

END RULES

Queries

(defquery query-country
  [:?country-code]
  [?report <- LocationReport (= ?country-code country-code)])
(defquery query-chart-request
  [:?path]
  [ChartFragment (= ?path path) (= ?chart-type chart-type) (= ?body body)])
(defquery query-location-request
  [:?path]
  [LocationRequest (= ?path path) (= ?location-name location-name)])

Right now there's only ever one WebRequest in a session, and therefore only one RenderedPage, but it's easy to imagine caching where we keep every RenderedPage, and only fire rules if the most recent RenderedPage fact is beyond some time/freshness tolerance.

(defquery query-rendered-page
  [:?path]
  [RenderedPage (= ?path path) (= ?html html)])
(defquery query-parsed-request
  [:?path]
  [ParsedRequest (= ?path path) (= ?argument argument)])
(defquery all-parsed [] [ParsedRequest (= ?path path)])
(defquery all-locations [] [LocationRequest (= ?path path) (= ?location-name location-name)])

END QUERIES

Exploratory Functions

(defn search-location-reports [country-code]
  (let [facts  (map create-location-report (external-data/get-all-timelines-jhu))
        session (-> (mk-session [query-country])
                    (insert-all facts)
                    (fire-rules))]
    (query session query-country :?country-code country-code)))
(def jhu-session (atom (-> (mk-session [parse-request
                                        query-country
                                        update-location-reports
                                        create-home-page-charts
                                        create-latest-chart-by-country-code
                                        create-chart-page
                                        all-locations
                                        parse-locations
                                        query-parsed-request
                                        query-chart-request
                                        query-location-request
                                        query-rendered-page
                                        parse-dates
                                        ])
                           (insert-all (map create-location-report (external-data/get-all-timelines-jhu)))
                           (insert (create-jhu-global-report (external-data/get-latest-global-jhu)))
                           (fire-rules))))

Look how we dereference the atom to get access to the current state of the session then we insert a new fact into that state and then we fire the rules (-> @jhu-session (insert (->WebRequest "/hello/world")) (fire-rules) (query query-parsed-request))

Given a url path, eg. '/us/2020-03-28/tt' insert a new WebReqest fact into a rules engine session, fire all the rules, and then query for any RenderedPage facts. Finally, we return the raw html content of that fact

I'm not sure if 'render' is the right name will revisit soon. The intention is that a web request comes in

(defn render-web-request
  [path]
  (-> @jhu-session
      (insert (->WebRequest path))
      (fire-rules)
      (query query-rendered-page :?path path)
      (first)
      (:?html)))

(render-web-request "tt/us")

(defn search-reports-by-country [session country-code]
  (fire-rules session)
  (:?report
   (first (query session query-country  "?country-code" country-code))))

(cw/latest-bar (search-reports-by-country @jhu-session "ES"))

END Exploratory

Example location result (without timelines) {:id 0, :country "Afghanistan", :country_code "AF", :country_population 29121286, :province "", :last_updated "2020-03-29T13:14:06.151058Z", :coordinates {:latitude "33.0", :longitude "65.0"}, :latest {:confirmed 110, :deaths 4, :recovered 0}}

 
(ns coronavirus-charts.nrepl
  (:require
    [nrepl.server :as nrepl]
    [clojure.tools.logging :as log]))

Start a network repl for debugging on specified port followed by an optional parameters map. The :bind, :transport-fn, :handler, :ack-port and :greeting-fn will be forwarded to clojure.tools.nrepl.server/start-server as they are.

(defn start
  [{:keys [port bind transport-fn handler ack-port greeting-fn]}]
  (try
    (log/info "starting nREPL server on port" port)
    (nrepl/start-server :port port
                        :bind bind
                        :transport-fn transport-fn
                        :handler handler
                        :ack-port ack-port
                        :greeting-fn greeting-fn)
    (catch Throwable t
      (log/error t "failed to start nREPL")
      (throw t))))
(defn stop [server]
  (nrepl/stop-server server)
  (log/info "nREPL server stopped"))
 
(ns coronavirus-charts.core
  (:require
    [coronavirus-charts.handler :as handler]
    [coronavirus-charts.nrepl :as nrepl]
    [luminus.http-server :as http]
    [coronavirus-charts.config :refer [env]]
    [clojure.tools.cli :refer [parse-opts]]
    [clojure.tools.logging :as log]
    [mount.core :as mount])
  (:gen-class))

log uncaught exceptions in threads

(Thread/setDefaultUncaughtExceptionHandler
  (reify Thread$UncaughtExceptionHandler
    (uncaughtException [_ thread ex]
      (log/error {:what :uncaught-exception
                  :exception ex
                  :where (str "Uncaught exception on" (.getName thread))}))))
(def cli-options
  [["-p" "--port PORT" "Port number"
    :parse-fn #(Integer/parseInt %)]])
(mount/defstate ^{:on-reload :noop} http-server
  :start
  (http/start
    (-> env
        (assoc  :handler (handler/app))
        (update :port #(or (-> env :options :port) %))))
  :stop
  (http/stop http-server))
(mount/defstate ^{:on-reload :noop} repl-server
  :start
  (when (env :nrepl-port)
    (nrepl/start {:bind (env :nrepl-bind)
                  :port (env :nrepl-port)}))
  :stop
  (when repl-server
    (nrepl/stop repl-server)))
(defn stop-app []
  (doseq [component (:stopped (mount/stop))]
    (log/info component "stopped"))
  (shutdown-agents))
(defn start-app [args]
  (doseq [component (-> args
                        (parse-opts cli-options)
                        mount/start-with-args
                        :started)]
    (log/info component "started"))
  (.addShutdownHook (Runtime/getRuntime) (Thread. stop-app)))
(defn -main [& args]
  (start-app args))
 
(ns coronavirus-charts.data
    (:require
     [clojure.pprint :refer [pprint]]
     [clj-http.client :as client]
     [cheshire.core :as cheshire]
     [clojure.string :as string]
     [tick.alpha.api :as t]
     [clojure.walk :refer [stringify-keys]]))

Base function to query a url and parse

External Data jsonista is a faster alternative to cheshire and already in project.clj not sure how to use it yet.

(defn fetch-data
  [url]
  (cheshire/parse-string (:body (client/get url {:accept :json}))
                         true))

Base function to extract keys from a parsed location.

This will be handy when creating timelines later

(defn extract-location
  [loc]
  {:id (:id loc)
   :latest (:latest loc)})

Queries the xapix covid-19 api to return the latest confrimed and deaths I expect to update these end points when xapix settles on a spec. [source: Johns Hopkins University]

(defn get-latest-global-jhu
  []
  (let [data (:latest (fetch-data "http://covid19api.xapix.io/v2/latest"))]
    {:confirmed (:confirmed data) :deaths (:deaths data)}))

Queries the xapix covid-19 api to return imformation about all locations [source: Johns Hopkins University]

(defn get-all-locations-jhu
  []
  (let [data (fetch-data "http://covid19api.xapix.io/v2/locations")]
    (:locations data)))

Queries the xapix covid-19 api to return imformation about all locations, including timelines [source: Johns Hopkins University]

(defn get-all-timelines-jhu
  []
  (let [data (fetch-data "http://covid19api.xapix.io/v2/locations?timelines=true")]
    (:locations data)))

END EXTERNAL DATA

(def jhu-timelines (get-all-timelines-jhu))
(defn extract-timeline-by-key [timeline key]
  (map (fn [[d v]]
       [(t/date (t/inst d)) v])
       (stringify-keys (get-in timeline [key :timeline]))))
(defn extract-timeline-metadata [report]
  {:coordinates (:coordinates report)
   :country-code (:country_code report)
   :latest {:confirmed (:confirmed (:latest report))
            :deaths (:deaths (:latest report))}
   :country-population (:country_population report)
   :last-updated (t/inst (:last_updated report))
   :province (:province report)
   :country (:country report)})
(defn extract-dated-cases [timeline]
  (let [confirmed (extract-timeline-by-key timeline :confirmed)
        deaths (extract-timeline-by-key timeline :deaths)]
    (zipmap confirmed deaths)))

(second (extract-dated-cases (:timelines (first jhu-timelines)))) Confirmed Cases Deaths => [[#time/date "2020-03-26" 110] [#time/date "2020-03-26" 4]]

(defn date-readings [report]
  "Given the a parsed jhu-report, return list of maps
   representing the # of confirmed cases for each individual day at
   that location. ({:date X :deaths Y :confirmed Z}, ...)"
  (let [time-reports (extract-dated-cases (:timelines report))]
    (map (fn [[deaths confirmed]]
           {:date (first deaths)
            :deaths (second deaths)
            :confirmed (second confirmed)}) time-reports)))
(pprint (date-readings  (first jhu-timelines)))
(pprint (take 2 (map date-readings jhu-timelines)))

Each of these needs to be transformed into a Fact

(defn compose-location-time-reports [report]
  (let [location-data (extract-timeline-metadata report)
        case-data (date-readings report)]
    (map (fn [c]
           (merge c location-data))
         case-data)))
(pprint (take 2 (compose-location-time-reports (first jhu-timelines))))

({:last-updated #inst "2020-04-10T13:49:21.642-00:00", :date #time/date "2020-03-25", :coordinates {:latitude "33.0", :longitude "65.0"}, :deaths 94, :country-code "AF", :confirmed 4, :country-population 29121286, :latest {:confirmed 484, :deaths 15}, :province "", :country "Afghanistan"} {:last-updated #inst "2020-04-10T13:49:21.642-00:00", :date #time/date "2020-03-26", :coordinates {:latitude "33.0", :longitude "65.0"}, :deaths 110, :country-code "AF", :confirmed 4, :country-population 29121286, :latest {:confirmed 484, :deaths 15}, :province "", :country "Afghanistan"})

(def location-time-reports (map compose-location-time-reports jhu-timelines))
(pprint (first location-time-reports))
(defrecord LocationReport2 [source-name
                           source-url
                           country
                           country-code
                           country-population
                           province
                           last-updated
                           coordinates
                           date
                           deaths
                           confirmed
                           latest])

Given a location-time-report map, return a LocationReport record

(defn create-location-report
  [{:keys [country
           country-code
           country-population
           province
           last-updated
           coordinates
           date
           deaths
           confirmed
           latest]}
   r]
  (LocationReport2.
   "Johns Hopkins University"
   "https://github.com/CSSEGISandExternal-Data/COVID-19"
   country
   country-code
   country-population
   province
   last-updated
   coordinates
   date
   deaths
   confirmed
   latest))

=> {:date #time/date "2020-03-26", :deaths 26, :confirmed 409}

These vectors represent a date and a confirmed case count for a region. Next is to get all deaths Finally combine into all to create one record per time/date that include stats and metadata

 
(ns coronavirus-charts.layout
  (:require
    [clojure.java.io]
    [selmer.parser :as parser]
    [selmer.filters :as filters]
    [markdown.core :refer [md-to-html-string]]
    [ring.util.http-response :refer [content-type ok]]
    [ring.util.anti-forgery :refer [anti-forgery-field]]
    [ring.middleware.anti-forgery :refer [*anti-forgery-token*]]
    [ring.util.response]))
(parser/set-resource-path!  (clojure.java.io/resource "html"))
(parser/add-tag! :csrf-field (fn [_ _] (anti-forgery-field)))
(filters/add-filter! :markdown (fn [content] [:safe (md-to-html-string content)]))

renders the HTML template located relative to resources/html

(defn render
  [request template & [params]]
  (content-type
    (ok
      (parser/render-file
        template
        (assoc params
          :page template
          :csrf-token *anti-forgery-token*)))
    "text/html; charset=utf-8"))

error-details should be a map containing the following keys: :status - error status :title - error title (optional) :message - detailed error message (optional)

returns a response map with the error page as the body and the status specified by the status key

(defn error-page
  [error-details]
  {:status  (:status error-details)
   :headers {"Content-Type" "text/html; charset=utf-8"}
   :body    (parser/render-file "error.html" error-details)})
 
(ns coronavirus-charts.middleware.formats
  (:require
    [cognitect.transit :as transit]
    [luminus-transit.time :as time]
    [muuntaja.core :as m]))
(def instance
  (m/create
    (-> m/default-options
        (update-in
          [:formats "application/transit+json" :decoder-opts]
          (partial merge time/time-deserialization-handlers))
        (update-in
          [:formats "application/transit+json" :encoder-opts]
          (partial merge time/time-serialization-handlers)))))
 
(ns coronavirus-charts.charts
    (:require [cheshire.core :as cheshire]
            [hiccup.core :refer [h]]
            [coronavirus-charts.chartwell :as cw])
    (:use [hiccup.page :only (html5 include-css include-js)]))

PARTIALS

This is required for our reel layout to prevent odd sizing of components

(defn reel-script []
  [:script "(function() {
  const className = 'chart-boards';
  const reels = Array.from(document.querySelectorAll(`.${className}`));
  const toggleOverflowClass = elem => {
    elem.classList.toggle('overflowing', elem.scrollWidth > elem.clientWidth);
  };
  for (let reel of reels) {
    if ('ResizeObserver' in window) {
      new ResizeObserver(entries => {
        toggleOverflowClass(entries[0].target);
      }).observe(reel);
    }
    if ('MutationObserver' in window) {
      new MutationObserver(entries => {
        toggleOverflowClass(entries[0].target);
      }).observe(reel, {childList: true});
    }
  }
})();"])
(defn make-nav [values class title]
  [:div.nav {:class class}
   [:h2 title]
   [:ul
    (map (fn [val]
           [:li val]) values)]])

CHARTS These functions all take a Report record, and then return a full html page

(defn base-chart [heading nav content]
  (html5 [:head
          [:title "coronavirus-charts.org"]
          (include-css "/css/screen.css")]
         [:body
          [:div.citation-strip nav [:h1 heading]]
          [:div.center
           content]
          (reel-script)]))
 
(ns coronavirus-charts.config
  (:require
    [cprop.core :refer [load-config]]
    [cprop.source :as source]
    [mount.core :refer [args defstate]]))
(defstate env
  :start
  (load-config
    :merge
    [(args)
     (source/from-system-props)
     (source/from-env)]))
 
(ns coronavirus-charts.oz
  (:require [oz.core :as oz]))
(oz/start-server!)
(defn play-data [& names]
  (for [n names
        i (range 20)]
    {:time i :item n :quantity (+ (Math/pow (* i (count n)) 0.8) (rand-int (count n)))}))
(def line-plot
  {:data {:values (play-data "monkey" "slipper" "broom")}
   :encoding {:x {:field "time" :type "quantitative"}
              :y {:field "quantity" :type "quantitative"}
              :color {:field "item" :type "nominal"}}
   :mark "line"})

Render the plot

(oz/view! line-plot)
(def viz
  [:div
    [:h1 "Look Up Chicken Little"]
    [:p "A couple of small charts"]
    [:div {:style {:display "flex" :flex-direction "row"}}
     [:vega-lite line-plot]]
    [:p "A wider, more expansive chart"]
    [:h2 "If ever, oh ever a viz there was, the vizard of oz is one because, because, because..."]
   [:p "Because of the wonderful things it does"]])

The default oz css stylying really leans into the whole storybook thing and I kinda love it.

(oz/export! viz "export-test.html")
(oz/build! line-plot)
 
(ns coronavirus-charts.facts)
(defrecord JHUReport [source-name
                           source-url
                           country
                           country-code
                           country-population
                           province
                           last-updated
                           coordinates
                           date
                           deaths
                           confirmed
                      latest])
 
(ns coronavirus-charts.middleware
  (:require
    [coronavirus-charts.env :refer [defaults]]
    [cheshire.generate :as cheshire]
    [cognitect.transit :as transit]
    [clojure.tools.logging :as log]
    [coronavirus-charts.layout :refer [error-page]]
    [ring.middleware.anti-forgery :refer [wrap-anti-forgery]]
    [coronavirus-charts.middleware.formats :as formats]
    [muuntaja.middleware :refer [wrap-format wrap-params]]
    [coronavirus-charts.config :refer [env]]
    [ring.middleware.flash :refer [wrap-flash]]
    [immutant.web.middleware :refer [wrap-session]]
    [ring.middleware.defaults :refer [site-defaults wrap-defaults]]))
(defn wrap-internal-error [handler]
  (fn [req]
    (try
      (handler req)
      (catch Throwable t
        (log/error t (.getMessage t))
        (error-page {:status 500
                     :title "Something very bad has happened!"
                     :message "We've dispatched a team of highly trained gnomes to take care of the problem."})))))
(defn wrap-csrf [handler]
  (wrap-anti-forgery
    handler
    {:error-response
     (error-page
       {:status 403
        :title "Invalid anti-forgery token"})}))
(defn wrap-formats [handler]
  (let [wrapped (-> handler wrap-params (wrap-format formats/instance))]
    (fn [request]
      ;; disable wrap-formats for websockets
      ;; since they're not compatible with this middleware
      ((if (:websocket? request) handler wrapped) request))))
(defn wrap-base [handler]
  (-> ((:middleware defaults) handler)
      wrap-flash
      (wrap-session {:cookie-attrs {:http-only true}})
      (wrap-defaults
        (-> site-defaults
            (assoc-in [:security :anti-forgery] false)
            (dissoc :session)))
      wrap-internal-error))
 
(ns coronavirus-charts.routes.home
  (:require
   [coronavirus-charts.layout :as layout]
   [clojure.java.io :as io]
   [clojure.string :as string]
   [coronavirus-charts.middleware :as middleware]
   [coronavirus-charts.sessions :refer :all]
   [ring.util.response]
   [ring.util.http-response :as response]
   [clara.rules :refer :all]))
(defn home-page [request]
  (layout/render request "home.html" {:docs (-> "docs/docs.md" io/resource slurp)}))
(defn about-page [request]
  (println  (string/split (:path-info request) #"/"))
  (layout/render request "about.html"))
(defn chart-page [request]
  (let [path (:path-info request)
        params  (string/split path #"/")]
    (println params)
    (layout/render request "home.html")))
(defn home-routes []
  [
   {:middleware [middleware/wrap-csrf
                 middleware/wrap-formats]}
   ["/*" {:get (fn [resp]
                 (-> (response/ok (render-web-request (:path-info resp)))
                     (response/header "content-type" "text/html")))}]])

(-> (mk-session) (insert (->WebRequest "hh.to/tt")) (fire-rules) (get-parsed-requests))

 
(ns coronavirus-charts.chartwell
  (:require [clojure.math.numeric-tower :as math]))

FF-Chartwell is a webfont that creates simple charts out of integers.

For example, this span: 10+50+100

with this css: .vertical-bars{ font-family: "FFChartwellBarsVerticalRegular"; font-variant-ligatures: discretionary-ligatures; }

will return a bar-graph with 3 bars: 1: 10% of font-height 2: 50% 3: 100%

It also has support for horizontal-bars, pies, radars, rings, areas, roses, bubbles, scatters and lines... as well as a couple "floating" variants.

The font technology used (discretionary liglatures) is supported in everything except IE, and I assume that most of the bugs in FF Chartwell have been ironed out because it's been active since 2012.

Here is a sample reagent component (defn simple-component [] [:div [:span {:style {:color "blue"}} "I am blue text"] [:span {:style {:color "red"}} "I am red text"]]])

I'm imagining that the api will look like: ( )

With the function for vertical-bars resembling: (vertical-bars [10 20] ["#bee" "#fab] "v-bar")

=> [:div.vertical-bar [:span "10" {:style {:color "#bee"} :class "chart-segment"}] => [:span "20" {:style {:color "#fab"} :class "chart-segment"}]]

We can abstract the "span" structure into a chart-segment component since it'll represent the smallest building block of a chart.

Since each chart segment is a part of a whole chart, react/reagent requires us to provide a unique key for the virtual dom

Given the integer size, a hexcode color, and a class string, return a reagent span component.

(defn chart-segment
  [content color class]
;;  ^{:key (random-uuid)} Don't think that this is necessary in clj
  [:span {:style (str "color:" color)
          :class (str "chart-segment " class)}
   (reduce str content)])

It's meant to work with the (herb)[http://herb.roosta.sh/] library for more complex styling like: (chart-segment 10 "#bee" (

Note how the key prop is stored as meta (meta (chart-segment "lol" "red" (sample-id-func "101010"))) => {:key "1010101584214541804"}

Our dream is to return [:div.vertical-bar [chart-segment _ _ _ ] [chart-segment _ _ _ ] ]

We're handing the anonymous fn, [size color] pairs and then we're destructuring it through the [[x y]] syntax in order to generate list of chart-segment

I'm sure that there's a more elegant way of doing this in one function. Note that the class-func must be a function that accepts [size color] [ Is this something we can express in clojure.spec? ]

(defn herb-vertical-bars [sizes colors grid class-func]
  [:div.vertical-bars
   (cons [:span {:class "vertical-bars-grid"} (str grid "+")]
         (map
          (fn [[size color]] (chart-segment (str size "+")
                                               color
                                               (class-func size color)))
          (map vector sizes colors)))])

(def sample-sizes [10 50 100]) (def sample-colors ["#bee" "#fab" "#ada"])

This is effectively a convert to percentage function except that it rounds up

(defn v-bar-scale [v target]
  (int (math/ceil (* 100 (/ v target)))))

the target scale is calculated through finding the largest value in the dataset. I'd like to make it round up to the nearest hundred.

(defn target-scale [dataset]
  (apply max dataset))
(defn vertical-bars [sizes colors grid class]
  (let [target (target-scale sizes)]
    [:div.vertical-bars
     (cons
      [:span {:class "vertical-bars-grid"} (str grid "+")]
      (map
       (fn [[size color]] (chart-segment (str (v-bar-scale size target) "+")
                                            color
                                            class))
       (map vector sizes colors)))]))
(defn source-box [r]
  (let [source-name (:source-name r)
        source-url (:source-url r)]
    [:p.sources "Source: "
     [:a {:href source-url} source-name]
     " via "
     [:a {:href "https://xapix-io.github.io/covid-data/"} "xapix" ]]))
(defn latest-bar [r]
  (let [latest (:latest r)
        c (:confirmed latest)
        d (:deaths latest)]
    [:div.bar-chart
     [:h2 (:country r)]
     (vertical-bars [c d] ["#dab101" "#110809"] "d" "latest-bar")]))
(defn global-bar [g]
  (let [latest (:latest g)
        c (:confirmed latest)
        d (:deaths latest)]
    [:div.bar-chart
     [:h2 "Global Cases"]
     (vertical-bars [c d] ["#dab101" "#110809"] "d" "latest-bar")]))

Given a C19Report, create a table from the :latest key

(defn latest-table
  [r]
  (let [latest (:latest r)
        c (:confirmed latest)
        d (:deaths latest)]
    [:table.blocky-table
     [:thead
      [:tr
       [:th "Confirmed"]
       [:th "Deaths"]]]
     [:tbody
      [:tr
       [:td c]
       [:td d]]]]))
 
(ns coronavirus-charts.sources.jhu
  (:require
   [coronavirus-charts.data :as d]
   [coronavirus-charts.facts :refer [hello]]
   [clojure.pprint :refer [pprint]]
   [clojure.walk :refer [stringify-keys]]
   [tick.alpha.api :as t]
   [where.core :refer [where]]
   [com.rpl.specter :refer :all]))

Queries the xapix covid-19 api to return the latest confrimed and deaths I expect to update these end points when xapix settles on a spec. [source: Johns Hopkins University]

(defn get-latest-global-jhu
  []
  (let [data (:latest (d/fetch-data "http://covid19api.xapix.io/v2/latest"))]
    {:confirmed (:confirmed data) :deaths (:deaths data)}))

Queries the xapix covid-19 api to return imformation about all locations [source: Johns Hopkins University]

(defn get-all-locations-jhu
  []
  (let [data (d/fetch-data "http://covid19api.xapix.io/v2/locations")]
    (:locations data)))

Queries the xapix covid-19 api to return imformation about all locations, including timelines [source: Johns Hopkins University]

(defn get-all-timelines-jhu
  []
  (let [data (d/fetch-data "http://covid19api.xapix.io/v2/locations?timelines=true")]
    (:locations data)))

END EXTERNAL DATA

(def jhu-timelines (get-all-timelines-jhu))
(defn extract-timeline-by-key [timeline key]
  (map (fn [[d v]]
       [(t/date (t/inst d)) v])
       (stringify-keys (get-in timeline [key :timeline]))))
(defn extract-timeline-metadata [report]
  {:coordinates (:coordinates report)
   :country-code (:country_code report)
   :latest {:confirmed (:confirmed (:latest report))
            :deaths (:deaths (:latest report))}
   :country-population (:country_population report)
   :last-updated (t/inst (:last_updated report))
   :province (:province report)
   :country (:country report)})
(defn extract-dated-cases [timeline]
  (let [confirmed (extract-timeline-by-key timeline :confirmed)
        deaths (extract-timeline-by-key timeline :deaths)]
    (zipmap confirmed deaths)))

(second (extract-dated-cases (:timelines (first jhu-timelines)))) Confirmed Cases Deaths => [[#time/date "2020-03-26" 110] [#time/date "2020-03-26" 4]]

(defn date-readings [report]
  "Given the a parsed jhu-report, return list of maps
   representing the # of confirmed cases for each individual day at
   that location. ({:date X :deaths Y :confirmed Z}, ...)"
  (let [time-reports (extract-dated-cases (:timelines report))]
    (map (fn [[deaths confirmed]]
           {:date (first deaths)
            :deaths (second deaths)
            :confirmed (second confirmed)}) time-reports)))

(pprint (date-readings (first jhu-timelines))) (pprint (take 2 (map date-readings jhu-timelines)))

Each of these needs to be transformed into a Fact

(defn compose-location-time-reports [report]
  (let [location-data (extract-timeline-metadata report)
        case-data (date-readings report)]
    (map (fn [c]
           (merge c location-data))
         case-data)))

(pprint (take 2 (compose-location-time-reports (first jhu-timelines)))) ({:last-updated #inst "2020-04-10T13:49:21.642-00:00", :date #time/date "2020-03-25", :coordinates {:latitude "33.0", :longitude "65.0"}, :deaths 94, :country-code "AF", :confirmed 4, :country-population 29121286, :latest {:confirmed 484, :deaths 15}, :province "", :country "Afghanistan"} {:last-updated #inst "2020-04-10T13:49:21.642-00:00", :date #time/date "2020-03-26", :coordinates {:latitude "33.0", :longitude "65.0"}, :deaths 110, :country-code "AF", :confirmed 4, :country-population 29121286, :latest {:confirmed 484, :deaths 15}, :province "", :country "Afghanistan"})

(def location-time-reports
  (map compose-location-time-reports jhu-timelines))

(pprint (first location-time-reports))

Given a location-time-report map, return a LocationReport record

(defn create-location-report
  [r]
  (let [{:keys [country
           country-code
           country-population
           province
           last-updated
           coordinates
           date
           deaths
           confirmed
           latest]}
        r]
    (JHUReport.
     "Johns Hopkins University"
     "https://github.com/CSSEGISandExternal-Data/COVID-19"
     country
     country-code
     country-population
     province
     last-updated
     coordinates
     date
     deaths
     confirmed
     latest)))
(def location-report-records
  (transform [ALL ALL] create-location-report location-time-reports))
(count (last location-report-records))

SPECTER EXPERIMENTS ;; (def jhu-timelines (get-all-timelines-jhu))

(def stringified-timelines "When we first parse the json, we end up storing time strings as keywords. We first go in and turn them to strings." (transform [ALL :timelines MAP-VALS :timeline] stringify-keys jhu-timelines))

(def parsed-timelines "Next, read them as tick dates." (transform [ALL :timelines MAP-VALS :timeline MAP-KEYS] #(t/date (t/inst %)) stringified-timelines))

(def pruned-timelines "Finally, we remove the extra :recovered key which is no longer reported by jhu" (setval [ALL :timelines :recovered] NONE parsed-timelines))

Now that we have our timelines in a more consistent form, we can begin constructing our Facts. First let's try to extract the report metadata... basically anything that's not the core timelines.

(defn extract-timeline-metadata [report] {:coordinates (:coordinates report) :country-code (:country_code report) :latest {:confirmed (:confirmed (:latest report)) :deaths (:deaths (:latest report))} :country-population (:country_population report) :last-updated (t/inst (:last_updated report)) :province (:province report) :country (:country report)})

(defn extract-dated-cases [report] (let [timelines (:timelines report) confirmed (select [:confirmed :timeline] timelines) deaths (select [:deaths :timeline] timelines) dates (select [ALL MAP-KEYS] confirmed)] dates))

(take 6 (select [ALL :timelines MAP-KEYS] pruned-timelines)) => (:confirmed :deaths :confirmed :deaths :confirmed :deaths)

This is going in and getting a reading on a date. (get (first (select [ALL :timelines MAP-VALS :timeline] pruned-timelines)) (t/date "2020-03-21"))

This gets the latest based on a country code (select [ALL (where :country_code :IS? "us") :latest] pruned-timelines)

 
(ns coronavirus-charts.sources.wikipedia)

This file will scrape the wikipedia index in order to find entries that reference the coronavirus in a particular location https://en.wikipedia.org/wiki/Special:AllPages?from=Coronavirus+in&to=&namespace=0

Then we will parse it and figure out what are the reported cases for that location as stated on wikipedia

We then enter that into our rules engine as a WikiPage (or something), after which we can compare what is published on wikipedia, to what we have in our JHU dataset. If a figure reported is smaller than the figure in the dataset, then we flag it as needing to be updated using something like

RULE IF [WikiURL ?url] THEN [WikiScrape ?url ?body ?type(country/province) ?location ?latest]

RULE IF [WikiScrape ?url (= ?type "country") ?location ?latest-wiki] [JHUReport (= ?country-name ?location ?latest-jhu)] (not ?latest-wiki ?latest-jhu)

THEN [FlaggedWiki ?url ?latest-wiki ?latest-jhu]

RULE IF [?all-flagged-wikis <- FlaggedWiki] THEN [(make-fragments ?all-flagged-wikis)]

REFERENCES https://www.youtube.com/watch?v=L9b8EGyiVXU (practicalli youtube) https://practicalli.github.io/blog/posts/web-scraping-with-clojure-hacking-hacker-news/ Old Clojure mediawiki library;; (10 years old... might be worth it to either rewrite, or just export to html files and then call a python script on the folder. MediaWiki API python library