WI Senate’s “budget repair bill” now available on OpenGovernment

Over the past few days, national news has focused on events in the Wisconsin state legislature, where 14 senate Democrats are protesting a controversial new budget bill introduced by Republican Governor Scott Walker and supported by 19 senate GOPers.

Because the bill was introduced in a special session, it’s taken a while for us to be able to obtain the legislative data, but thanks to our partnership with the Open States Project, we now have a page for it here on OpenGovernment:

LRB 1383, or Special Session SB-11: state finances, collective bargaining for public employees, compensation and fringe benefits of public employees, the state civil service system, the Medical Assistance program, sale of certain facilities, granting bonding authority, and making an appropriation.

Shorter direct link to that bill’s profile page on OG, for copying-and-sharing: http://bit.ly/WIbill_sb11_page

… on that page, you can find its official information, a link to view its full bill text (link to 144-page .pdf document), view all its official actions, and check out summaries of the two committee votes that preceded bringing the bill back to the full chamber (sorry, no member-by-member roll call info is available at this time — but we’d love to be able to provide this crucial detail, and we can in the near future… if you help us grow). We’re showing the full extent of the information we have available for this special session bill — the roll call details are currently sparse, but please keep in mind this bill is a special case and we’re working to standardize the data as best we can.

In addition, you can get engaged with this newsworthy piece of legislation: comment on it, subscribe to an RSS feed of its every action (or major actions only), and most importantly, on the right-hand side of the page, input your five-digit Wisconsin zip code to find your state senator and contact him or her about it (over phone, email, postal mail, and more).

On the OpenGovernment – Wisconsin homepage, you can browse all members of the WI state senate, browse all bills in the 2011-2012 session, those bills most in the news, a dozen major issue areas, and all publicly-available information about campaign contributions made to WI state senators and assemblymembers. OpenGovernment brings together all this info, for the first time all in one place, alongside news & blog coverage, social media mentions, free public participation tools, and more, in our distinctively welcoming web design. Finally, a version of OpenCongress for state legislatures.

(For more background: Madison.com, JSOnline, NYT, Memeorandum, TPM, Russ Feingold micropublishing updates).

We’ll have more to write about this soon, but in short — this is a tremendous illustration of the utility of OpenGovernment as a free public resource about what’s happening in state legislatures. This timely story is driving national news discourse and political blogging across the country, and the public needs objective information about the actual substance of the bill being discussed – and then user-friendly ways to organize their communities around it. We seek to bring much more data and many more powerful features to OpenGovernment and roll out this proven model of transparency to all 50 U.S. state legislatures — see how you can help us grow and obtain additional funding support from charitable foundations & philanthropists. In short, events like these are exactly why we need OpenGovernment for your state and local government!

Don’t forget — if you use the Twitter commercial micro-publishing service, tweet #WIbill #sb11 and your updates will appear there on the bill page shortly. We’re going to focus development resources on making this even more inclusive so that we can bring in other hashtags surrounding this bill, e.g. #WIunion, #solidarityWI, etc. Short link to this blog post, for copying-pasting-sharing, if you like: http://bit.ly/WIbill_sb11

Posted in Uncategorized | Leave a comment

CloudFront custom origin server with Rails, Jammit, and Apache

This is the first in a series of more in-depth technical posts about OpenGovernment. Our project is open source, so feel free to check us out over at github.

Here’s what you want to be able to do:

  • Serve your CSS, Javascript, and images from a CDN that’s near your users,
  • Serve gzipped versions of CSS and JS files if possible,
  • Make these files as small as possible by minifying them,
  • Serve them via SSL when desired,
  • Have them cached on the browser for as long as possible,
  • And serve as few of these files as possible (as Steve Conover at Pivotal Labs put it: “We send down exactly one .js and one .css file. If you are sending down more than one of each of these to the browser, you have a performance problem.”)

Our history with S3

On OpenCongress, we’d been using S3 for content distribution for a while, but we weren’t following many of these rules until pretty recently. We weren’t compressing our .css and .js, in some cases we weren’t minifying them either, and it was resulting in slow page loads and adding an unnecessary premium to our S3 bills.

My first step in speeding things up was to install Jammit, the excellent asset packager gem. Jammit reduces the number of CSS and JS files you need to serve, and it minifys and gzips them for you. Unfortunately, at the time I installed Jammit there wasn’t any support for S3, so I had to adjust our already bandaided S3 sync tool to handle Jammit’s packaged assets.

Why did it need patching? Well, Jammit creates minified assets with extensions .css, .css.gz, .js, and .js.gz. Because S3 does not do any content negotiation, if you put these files on S3 and reference the .css file in your stylesheet, compatible browsers will never receive the gzipped stylesheet.

Our initial fix for that was to change which assets we referred to based on whether the browser loading our site supported gzip or not. So if we saw an Accept-Encoding: gzip header, we’d serve up, say, default.css.gz as the stylesheet. Of course, we also had to make sure that S3 would deliver the .gz file with Content-Encoding: gzip, so browsers would know to unpack it before interpreting it.

Unfortunately, even with the proper Content-Encoding header, Safari barfs at this arrangement because of the .gz extension, and the result is a mangled page without any styles or javascript at all.

So the final fix was to rename the files before uploading them to S3. If they’re called .cssgz and .jsgz, they’re handled fine by browsers. The .css and .js versions still exist, of course.

Our site loaded much faster, so we had a working-yet-kludgy solution in place.

Fast forward a few months. Prior to the launch of OpenGovernment.org, I looked at the situation again and discovered that we probably should have been using CloudFront all along (it’s faster than S3), and that a new feature on CloudFront presented a much simpler solution.

The way better solution: CloudFront custom origin server

CloudFront now supports a Custom Origin Server option. With this option, instead of having CloudFront sync to an S3 bucket, you can just have it mirror your own server. So if we reference http://ourcloudfronthost.cloudfront.net/assets/common.css and someone hits our site, CloudFront will issue the same request to our server if needed, cache the result, and send it back down to the browser.

This actually simplifies a lot of things. It means that if you can get the headers right on your server, as they should be, then CloudFront will serve up the same things under the same circumstances. No extra gems are needed, and there’s no need to ever upload or sync things with S3.

To set this up on CloudFront is pretty easy, but you can’t do it via the AWS management console. You need to send an XML request to AWS. Download cfcurl.pl and create a file called ~/.aws-secrets with your AWS secrets, eg:

%awsSecretAccessKeys = (
 # PPF on AWS
 'ppf' => {
     id => 'your aws key',
     key => 'your aws secret',

Now you’ll need a little XML file. Here’s the one I used to create OpenCongress’s CloudFront:

<?xml version="1.0" encoding="UTF-8"?>
<DistributionConfig xmlns="http://cloudfront.amazonaws.com/doc/2010-11-01/">
<Comment>OpenCongress Remote Origin</Comment>

Use cfcurl.pl to send this to AWS using your keys:

perl cfcurl.pl --keyname ppf -- -X POST -H "Content-Type: text/xml;charset=utf-8" --upload-file opencongress_cf.xml https://cloudfront.amazonaws.com/2010-11-01/distribution

You should get a response back with your new CloudFront distribution information. You should also see the new distribution in the AWS management console.

Now you’re ready to add your new CloudFront hostname to your production.rb. This is ours:

# Enable serving of images, stylesheets, and javascripts from CloudFront
config.action_controller.asset_host = Proc.new {
   |source, request| "#{request.ssl? ? 'https' : 'http'}://d20tbjzc77cxpv.cloudfront.net"

Using CloudFront as your asset host does introduce a couple problems, and we’ll cover them next.

Cache Busting

CloudFront and S3 ignore Rails’ typical cache-busting strategy of appending a timestamp as a URL parameter on asset URLs. Normally with S3 sync you’d be able to simply upload new files and those would be reflected more or less immediately via CloudFront. But when you deploy under this new Custom Origin strategy, your changes will stay cached on CloudFront for up to 24 hours unless you give your assets new pathnames.

So I set up a new cache busting strategy that uses an Apache Rewrite rule and a subtle adjustment to Rails’ asset_path setting. In httpd.conf:

# Cache-busting rule for CloudFront.
RewriteEngine on
RewriteRule ^/r-.+/(images|javascripts|stylesheets|system|assets)/(.*)$ /$1/$2 [L]

And in production.rb:

# Use the git revision of this release
RELEASE_NUMBER = %x{cat REVISION | cut -c -7}.rstrip

config.action_controller.asset_path = proc { |asset_path|

This assumes you’re using Capistrano and you have a REVISION file in our application’s root folder containing the git revision of the release.

Deflate static assets

That covers the CloudFront side of things. The only issues left are on the Apache side: making sure that we’re gzipping content when possible (using content negotiation or mod_deflate), and setting far-future expiration dates.

Apache and Jammit don’t play too well together when it comes to content negotiation. With MultiViews turned on, I believe Apache will not serve any variants unless the requested file itself is not found. So if there is a common.css, for example, Apache will always serve it. I’ve found that only if I rename common.css to common.css.css, will Apache present the gzipped variant properly when common.css is requested.

Another option is to leave common.css and common.css.gz alone, but change the stylesheet link tag to refer to simply “/assets/common”. This is more fragile than the first option, though, because when /assets/common is requested, Apache will serve the smaller of common.css and common.js if both files exist.

So the solution was to set our deploy scripts up to rename the files after jammit runs. In deploy.rb:

desc 'Compile CSS & JS for public/assets/ (see assets.yml)'
task :jammit do
  run "cd #{current_release}; bundle exec jammit"
  # For Apache content negotiation with Multiviews, we need to rename .css files to .css.css and .js files to .js.js.
  # They will live alongside .css.gz and .js.gz files and the appropriate file will be served based on Accept-Encoding header.
  run "cd #{current_release}/public/assets; for f in *.css; do mv $f `basename $f .css`.css.css; done; for f in *.js; do mv $f `basename $f .js`.js.js; done"

And, of course, in httpd.conf:

<Directory "/web/opengovernment.org/current/public/assets"> 
  # MultiViews allows gzipped assets to be delivered
  Options MultiViews

Also in httpd.conf, by default I think there are AddType entries for .gz and other zipped files. You’ll want to comment these out, because they’ll override the content negotiation behavior we want.

Deflate dynamic content

Our static assets are now gzipped when possible, but the pages themselves are not. Assuming you’ve compiled mod_deflate into Apache, this is a trivial step. In httpd.conf:

# Deflate whatever hasn't already been deflated via MultiViews.
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/javascript text/css application/x-javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio

… and if you want logging:

LogFormat '"%r" %{outstream}n/%{instream}n (%{ratio}n%%)' deflate
CustomLog logs/opengovernment.org-deflate_log deflate

Far-future expiration

The last piece we want is to set the Expires: headers so that browsers will cache our assets without the hassle of issuing a request and getting back a 304 Not Modified. In httpd.conf:

ExpiresActive On
<Directory "/web/opengovernment.org/current/public">
   # We're already invalidating assets via the above cache-busting rule, so no ETags.
   FileETag None

   # Far future expiration, made possible via cache-busting as well.
   ExpiresDefault "access plus 1 year"

Finally, we have a setup that meets the basic goals, and I’m happy to report that our Google Page Speed score is over 90/100 for most pages on OpenGovernment. Yes, we still have some inefficient CSS rules to fix, but that’s a story for another day.

Caveats and room for improvement…

  • You need to use image_path or image_tag across your project, never plain old <img> tags. You also need to refer to images via their relative URLs from across your CSS, eg: background: url(../images/bg.gif) instead of /images/bg.gif. If you notice that some images aren’t being cache-busted properly, this is probably the reason.
  • It would be nice if the cache were only busted when the individual files changed. In the above configuration, every new deployment will require all of our regular users to reload assets once after each deployment, even if those assets haven’t changed. Having the asset_path reflect the individual asset’s github file revision rather than the github repo revision would fix this for images but not jammit assets. Using the modification time of the file doesn’t work because Capistrano seems to always update mtimes. So for now I don’t see a feasible way to do this. Do you?
  • File attachments and other items in /public/system are not accounted for here. You may need an exception in your asset_host and/or Expires settings for these items. Or use Paperclip’s S3 support, for example.

Thanks to Jeff Cleary and James Abley for their earlier posts on CloudFront and Rails.

Posted in Uncategorized | 6 Comments

OpenGovernment named a Finalist at SxSW Interactive – Accelerator

We’re very excited to announce that OpenGovernment was selected to be one of 8 finalists in the News category at South by Southwest Interactive’s Accelerator contest, March 14-16th in the gem-of-a-town Austin, TX. Looking forward to meeting & trading ideas w/ the other finalists & attending SxSW in general. We’re honored to have been chosen and excited to show our work. Continue reading

Posted in Uncategorized | Leave a comment

OpenGovernment on NCSL’s “The Thicket” and more

Hi everyone, plenty more press mentions & positive buzz to report: Wisconsin Watch, a great piece by Kate Golden; Frank Hecker’s Maryland-focused blog; Federal News Radio (with whom I was fortunate to do a phone interview earlier today about OG’s mission, audio piece forthcoming probably tomorrow, thanks @cdorobek); a quick mention at Library Stuff; good coverage in the GovLaunch forum on GovLoop, the social networking site for government community & state employees; a bite-sized item on Spread Drupal (big ups open-source); Texas Watchdog, another great post that uses one of our eye-catching badges; Resource Shelf; Law Librarian Blog (very cool); and many more. Plus there’s the micro-publishing machine, churning without rest, a Terminator with scant little merciful filtering of content, always a compelling distraction. We haven’t yet had time to fire up a social networking page on the leading commercial service, but that’s still in the works, as we patiently await further Diaspora development and other open-source solutions for connecting with people in open standards on the semantic Web.

To highlight one mention in particular, we’re pleased that OpenGovernment was featured on The Thicket, the blog of the National Council Conference of State Legislature (NCSL – ed. — corrected, Conference not Council, thanks Pam for the catch), the indispensable umbrella group for state-level legislatures across the country. We think highly of the Thicket & the work that they do, and they’re listed on all state homepages on OG (e.g., CA), towards the bottom of the page, included in the helpful resources about how state government works.

Pam Greenberg, in her great write-up, perceptively surveys our site’s offerings & tools. Towards the end, she took slight issue with an off-the-cuff quote I gave in a recent interview, and I wanted to be sure to clarify that we’re productively building a public resource with and for state legislatures — my notes as below:

Continue reading

Posted in Uncategorized | Leave a comment

Launch Day Round-Up

It’s been a fulfilling launch day for all of us at OpenGovernment — thanks to everyone who checked us out & helped spread the word. We build this site as a public resource, and are continually tweaking it in response to your feedback and actual usage patterns, so please don’t hesitate to hit us with your opinion. We’re interested and input from one person can have an impact to make the site a better tool for everyone. (That said, there is a lot we have planned to come, too, about which more below.)

Today’s launch received a nice wave of reactions on the leading micro-blogging service. To highlight a few: Alex Howard of O’Reilly Media (the wide-ranging @digiphile); Nick Judd of TechPresident; plus the blog post announcement from Ellen Miller, the Executive Director of our remarkable partners, the Sunlight Foundation; and another by Tom Lee, the Director of Sunlight Labs. This is as good a time as any to remind everyone reading to check out the community-driven Open States project, the data from which will form the legislative backbone of OpenGovernment for all 50 states. Continue reading

Posted in Uncategorized | Leave a comment

OpenGovernment (beta) launches today!

We’re very excited to announce today’s launch of the beta version of our next major project: OpenGovernment.org.

learn, track, share

Free & open-source, OpenGovernment is a non-partisan public resource for transparency at any level of government: state, city, local, international, and more.

It’s based on OpenCongress.org, the most-visited not-for-profit website in the country for tracking the U.S. Congress, with over a million visits per month and many millions of requests for data served every day. Finally, a version of OpenCongress for state legislatures.

Dedicated to building public knowledge and combating systemic corruption in government, OpenGovernment is a joint project of two 501(c)3 non-profit organizations, the Participatory Politics Foundation and the Sunlight Foundation.

This beta version is launching with information for five state legislatures: California, Louisiana, Maryland, Texas, and Wisconsin. Over the next year, we seek non-profit funding support to roll out OpenGovernment to all 50 U.S. states, dozens of major cities, other countries, and beyond. Official government information comes from the community-driven Open States Project coordinated by Sunlight Labs. As with OpenCongress, OpenGovernment brings together this official government information with social wisdom from around the open Web, campaign contribution data, and free public participation tools. OpenGovernment offers user-friendly web pages to track and share everything in these five state legislatures: bills, votes, actions, legislators, committees, issues, and much more (what’s most-viewed, most-in-the-news, key votes, etc.). Quick links: read more about us, our data sources, our developer hub, and how you can help us grow.

This is indeed a beta version of the site, so keep in mind that we expect there to be a few kinks, and much more data & features are forthcoming. In that context, our small non-profit development team is incredibly excited to get this new project out onto the open Web and in front of people — we’ve been working on it for three-fourths of the past year in partnership with Sunlight, and we’re excited to roll it out to all 50 states and to foster a diverse (in terms of skills, time, and other) community of volunteer developers around it to remix it. There are a lot of factors involved here that really motivate us: fighting systemic corruption, liberating public data, advocating comprehensive electoral reform, facilitating peer-to-peer communication about our government, empowering citizen watchdogs — but one of the primary ones is creating user-friendly interfaces for this baffling and arcane world of legislative data.

With OpenCongress, we took the hopelessly clunky and outmoded data on THOMAS (via our longtime data partners, GovTrack), as seen here on their page for the major health-care reform bill from the last Congress:

… and made the data more accessible, with tools at your fingertips to track its status, see what people are saying about, share it with your online community, link to specific sections of its full bill text, and easily contact your members of Congress about it. Here’s the overview page of our version (more detailed info available by diving into tabs on that page, e.g. for the Money Trail or Wiki narrative):

… keep in mind that when we first conceived of OpenCongress, back in 2004, THOMAS didn’t offer some basic technical features: RSS feeds, permalinks, social sharing tools, lists of most-viewed info, helpful search tips, and more. While the Library of Congress has  somewhat caught up with the times, we believe that their main focus, as the primary source for all federal legislative data, should be to move aggressively to comply in full with the community-generated Principles of Open Government Data. Nothing short of this standard is sufficient for public transparency in an accountable representative democracy. Public data can and should be available to the public, immediately and in full, at every level of government, full stop.

In the same way, we’ve taken the painfully user-unfriendly information from a hilariously dreadful mishmash of official state .gov websites and turned it, in this beta version, into something more widely accessible to today’s casual web surfer:

Old & busted official .gov bill page from CA:

… and a glorious official CA state leg. vote page:

… compared to the new & non-busted OpenGovernment (beta) bill page:

… another view of an OG bill page, with intentionally-glaring annotations of the disparate info it brings together on the overview page (click to enlarge, or of course click around the site here at will to explore more of our info design in this beta version):

… crucially, our pages don’t only put a better-designed style and superior information display on valuable government data (though they do that). They also doesn’t just bring together bills with news coverage, blog posts, social media mentions, campaign contribution data, videos, and more (though they do that too). OG also facilitates self-organizing communities to take action directly on our bill pages:

Track – subscribe to RSS feeds for bill actions and votes by members

Share – one-click access to social sharing tools makes it easy to spread the word on Facebook & Twitter & Reddit & StumbleUpon & others to come (e.g., Identi.ca & Diaspora).

Contact – simply enter your zip code to view a handy pop-up window with all the official contact information for your state legislator, directly from that bill page. We’re talking phone, email, district & capital office addresses, and more. This is the foundation of a fully non-commercial, open-source toolkit for constituent communication, all of which will be made available in full out to the public commons via a free API and open data downloads. Perfect for political bloggers, issue-based groups, and community advocates looking to contact their elected officials about what matters to them.

There’s a lot more to say, and we’re interested in hearing your feedback on this beta version, but for now, a quick look ahead to the function of this blog and its relation to our other projects. In the process of building out this beta version, we’ve run into a lot of idiosyncratic & amusing tidbits about the workings of state legislatures — e.g., in Louisiana, “Members of both houses of the State Legislature are free from arrest, except for felony, during their attendance at sessions and committee meetings of their house and while going to and from them. No member shall be questioned elsewhere for any speech in either house.” (cit. Wikipedia). Or rather, our aggregation of bill data from state houses so far has revealed some interesting top-level patterns, which we’ll be blogging about here and encourage the public at large to submit their findings too as they poke around our site and dig into our open data.

A few notes of introduction to each of the five beta state sites. Each state page has categories of info as listed in the left-hand nav bar: bills, people, issues, and the money trail. From bill pages, you can find news & blog coverage, comment forums, and more. From people pages, you can find recent bills sponsored & official actions, roll call results, personal campaign contribution data, comment forums, and more.

California: The 2011-2012 session began on January 3rd, 2011 and is scheduled to run until September 9th, 2011. Last year, the vast majority of bills were introduced in February, followed by smaller peaks in March and July.

Louisiana: The 2011 session begins on April 25th, 2011, and is scheduled to end on June 23rd, 2011. Last year, the vast majority of bills were introduced in April.

Maryland: The 2011 session began on January 12th, 2011 and is scheduled to end in early April. Last year, the vast majority of bills were introduced in February.

Texas: The 82nd Texas Legislature began on January 11th, 2011, and ends on May 30th, 2011. Last year, the vast majority of bills were introduced in March, followed by February and April.

Wisconsin: The 2011-2012 session began on January 11th, 2011 and is scheduled to continue regularly (with some holiday breaks) until December 8th, 2011. Last year, most bills were introduced in February, followed by March and April.

… we’ll be writing lots more about patterns we see in the data, but let us know what you’re finding too and we’ll be happy to surface it for all to see. Contact us anytime.

In the news, you hear often about the dysfunction of state houses. Let’s build public knowledge about bills in state legislatures together. Thanks to our terrific partners at the Sunlight Foundation, Sunlight Labs, and the awesome volunteer community at Open States.

Foundations and philanthropists: we are actively seeking funding partners to roll out this proven model of engagement to all 50 U.S. states, cities, and towns across the country — then to other countries, as OpenGovernment becomes a global platform. We’re working to revolutionize government transparency at the state and local level, and we hope you’ll help us grow. Let us know what you think!

Posted in Uncategorized | 6 Comments

Towards 50 states and beyond

OpenGovernment is a free and open-source public resource website for government transparency and civic engagement at the state and local levels. The site is a non-partisan joint project of two 501(c)3 non-profit organizations, the Participatory Politics Foundation and the Sunlight Foundation. OpenGovernment is independent from any government entity, candidate for office, or political party. The information contained on OpenGovernment pages, wherever applicable, is cited to a primary source– while we aggregate many different data sources, we do not edit or manipulate government data in any way before presenting it here. Read more about us, our data sources and the community-driven Open States Project from Sunlight Labs.

OpenGovernment is currently in early open-source development, with a planned public beta launch later in 2010 and updates continuing to roll out over 2011. Ever since we conceived of OpenCongress in 2004, we’ve realized that the model of combining government data with social wisdom in order to facilitate civic engagement can and should be applied to other levels of government: state legislatures, city councils, neighborhood associations, international institutions, the other branches of the federal government (Executive and Judicial), public-mission institutions such as schools & hospitals, foreign countries with more-or-less democratic systems, and more. We’re working towards a future in which the public at large can conveniently access the best available info about all public actions at every level of government, then organize civic actions of their own in response and in dialogue with their elected officials.

Towards this end, we’re working with our partners at Sunlight Labs on the community-driven Open States Project, with the goal of establishing a data standard and collecting machine-readable data streams for all 50 U.S. State Legislatures. These data streams will provide official government info to GovKit, the open-source application that combines it with other publicly-available data sources and social wisdom from around the open Web. GovKit, in turn, will power the OpenGovernment website: essentially, free and non-partisan versions of OpenCongress for all fifty state legislatures and a dozen major cities, with even more local versions planned. We’ll continue to encourage volunteers to remix the code for city, county, or municipal governments. Along the way, we’re working to make our code more modular and better-documented, in order to make it possible for volunteers to make their own versions of OpenGovernment with an emphasis on the issues they and their communities care about.

Our code has always been 100% open, but now we have an even better answer to the question, “How do I make an OpenCongress for my (state, city, town, or country)?” OpenGovernment is the new solution. Overall, we have a good start on a more transparent government, but we’ve just begun — we need more resources to take OpenGovernment to the next level and enable more powerful civic engagement. If you are in a position to contribute to our non-profit work, or for more detailed information, please contact us.

Posted in Uncategorized | Leave a comment