Common Support Tasks

Owned by Jim Van Fleet
Last updated: May 10, 2017
4 min read

Bringing a new dataset online

Though the process for cities and their representatives is documented on the GitHub wiki but this document is intended to be the companion piece for operators.

Prerequisites:

Your terminal program must respond to these commands:

  • heroku
  • ruby
  • gem
  • bundle

In general, homebrew with rbenv and heroku packages are your friend here.

Coaching through a submission

Whether from government representatives or from interested citizens, inbound interest in coming online with Citygram is relatively healthy.  Sometimes there will be a pull request, sometimes there won't be.  Sometimes they will have read the documentation, sometimes they won't have read it.  Our goal so far has been to coach in good faith through a submission, even with those for whom Ruby is not a first language.

It's a reasonable time to note that there is no requirement for the middleware layer to be delivered via citygram-services.  Any URL that returns GeoJSON (including those straight from an Open Data Portal) is fine.

Verifying submission quality 

While verifying submission quality, you'll be using the citygram-services repo, almost certainly.

Once there is at least some kind of pull request available on citygram-services (or a URL submission of another variety), it's time to verify.  Lets consider the most common: a PR to citygram-services.  That verification process is roughly:

Add their fork as a remote in order to acquire their branch

This is basically "acquire their code to run it locally"

Run citygram-services locally with their code available

At this point, you can reference the name of the file that they've added to the repo in order to divine the URL you should be accessing locally.  (It's /name-of-file-without-the-dot-rb usually at localhost:5000)

Access to the specific errors that the endpoint generates are helpful.

Ensure that the endpoint loads (it often does not)

Common problems include the following:

  • The URL isn't a JSON endpoint but a human-oriented one.  (Author just guesses based on column names, I'm not even upset that's impressive!)
  • A row with bad data can malfunction during transform and crash the whole response.  (These rows can be filtered by query, add a short Ruby clause to correct the field in the degenerate cases or otherwise.)
  • Reasonable row processing, but needs help to formulate as valid GeoJSON (e.g. they left out that it's supposed to be a FeatureCollection)
Perform reasonable debugging efforts therein

(e.g. for Edmonton)

Verify with lint locally

GeoJSONLint is a heckuva website.  You can copy paste the results from your web browser or curl | pbcopy incantation

Occasionally, you will get a submission here with no event data.   GROAN!!!  Citygram prefers to have access to some backlog, it's a pretty sophisticated deduper.

Merge to master

The data source verifying with GeoJSONLint means you are ready to publish.  I usually merge the fixed branch to master locally and then push directly to GitHub (this is an old fashioned PR merge-- any PR filed will mark as merged.)

Deploy

git push heroku master from the master branch will make the endpoint available

Verify citygram-services with GeoJSONLint

This is the equivalent of production.

Celebrate!

This is the end of phase 1!  Aren't you excited that you have usually two more phases to go?  Don't answer that.

Assembling and verifying publisher stanza

With the data source ready for processing, you'll transfer your focus to the main Citygram repo.  Broadly, the goal of this phase is to verify that citygram has the capability to ingest the data source as intended, then ensuring that data source can be accessed by end users to citygram.org

In the verbiage of codeforamerica/citygram, we are creating a new Publisher, so that's the language I'll use here.

Odds are less than 50% that you'll receive a stanza like this to paste into your console.  You can likely figure out most of the fields based on the above (hopefully they will describe the feed for you!).  You can pick the appropriate icon here.  If you generate this stanza yourself, it's up to you whether you verify this with your submitter or just go straight in to go-live.

Final go-live sequencing

Persist the publisher online (in prod or locally)

JVF always does this in prod, not wanting to negotiate running publisherpoll process locally.  But it's up to you.

Add invisibly Active

Regardless of the stanza that you begin with, you will want to make sure that pub.visible starts as being set to false and pub.active set to true.

This will allow the publisher to be polled for events, but will not yet expose the publisher to the user interface (since you still aren't 100% sure it's going to work yet).

Copy and pasting the stanza into your heroku run rake console will advance you to the next step

Watch PublisherPoll Logs

heroku logs -t

In my terminal, I have a search for the URL fragment of the citygram-services endpoint I'm monitoring.  I am looking to see that the endpoint is processed.

Verify processing and make publisher visible

In a heroku console (heroku run rake console), you want to do something like the following:

Event.where(publisher_id: 45).count

If you get any positive value, you are ready to proceed.

Publisher.find(45).update_attribute(:visible, true)

Verify city page looks acceptable

Inside citygram.org, navigate to the datasource in question, and verify that markers appear where events in the feed are present.  If they are...

Celebrate for real

Publish via Code for Charlotte Twitter, and social media channels.  Slack.  Wherever you like.

Bringing a new city online

The process of bringing a city online is small set of additional steps and verifications above bringing a new dataset online.  Sequence-wise, they must come before you go-live with the datasource-- odd but true.

PR's of this variety go to codeforamerica/citygram.  You absolutely need a sponsor from the city in question to bring it online, if for no other reason than to select the city banner image.  Here's an example.

Adding city graphic

If there's a city that got the right resolution the first time, I still haven't seen it.  (Target should be 2400px x mumbling sounds) It needs to go in the directory with all the other city images.

Adding to cities.geojson

A GeoJSON file itself, the structure is easy to copy.  (Note that for when multiple cities are a part of the same region, the RTP bundle has that composition.)  Ask the sponsor for the geographic center (or google maps for City Hall or equivalent.)

Random pull requests

They aren't flowing in like they used to, but they still happen.