==== Swiss parliament minutes scraping ====
[[http://parlement.letemps.ch]]
Is the Swiss parliament really useful ? Once elected, what are our councilors talking about ? Who is answering to whom ?
Goal of this project is to answer some of these questions and many more. To do this, we are planning to:
- Scrape the parliament website in order to retrieve councilors bio, topics discussed and minutes
- Structure them in session, intervention (with rank/order), author and text
In order to perform some analysis, as
- from keywords, who talks about what, by parties, cantons and people
- person vs. vocabulary
- dialogue order
- gender
Our NEW [[https://github.com/douglas-watson/parl-scraping|github]] (careful, we had to fork it at the beginning of day 2).
=== Data ===
Raw data are available as one single JSON file, and its .csv counterpart. We had size problems, thus exploring ways to produce several .csv.
**The [[https://github.com/douglas-watson/parl-scraping/tree/master/data|final folder for our data is on github]], the [[https://github.com/douglas-watson/parl-scraping/blob/master/data/with-bio-split-csv.zip|.csv files split by session are here]].**
== Structure of the main JSON (on Giovanni's side, complement with the bio data from the Parliament API) ==
list of interventions, with the following fields:
* Link_subject: link to the page of the subject under discussion (cf. CuriaVista)
* Surname: of the person speaking
* Description: of the subject under discussion
* Bio: link to the page of the person speaking
* Canton: of provenance
* Subject_id: of the subject under discussion (still to be understood: typology)
* Date: of the intervention (DD.MM.YY)
* Group: political group of the person speaking at the moment of the intervention
* Session_title: title of the session (Séance)
* Data: transcript of the intervention
* Name: of the person speaking
== Structure of the Parliament API data via Yannick ==
* [[http://ws.parlament.ch/councillors/823?format=json|Very rich example fields-wise]] (thus almost complete) for one councillor
* [[http://ws.parlament.ch/councillors|All councillors]]
* [[http://www.parlament.ch/e/dokumentation/webservices-opendata/Documents/webservices-info-dritte-e.pdf|API's doc]]
* [[https://github.com/douglas-watson/parl-scraping/tree/master/biography_retrieval|Code on github]]
* From those data, building of a file (light one) describing members of parliament ([[https://github.com/douglas-watson/parl-scraping/tree/master/biography_csv_extract|script on github in the same folder]])
Fields are: id cantonName council firstName lastName party active birthDate gender language maritalStatus militaryGrade partyId salutationLetter workLanguage
== Structure of the final files from Jérémie ==
1 JSON file + its .csv counterpart for each Parliament session of the National Council from 1995.
The same datasets are also available split as one JSON/CSV file per legislative session.
== Graph data for Yannick ==
graph.csv: edgeless with Source (bio url as id of person replying to) - Destination (bio url as id of person talking before) - Subject (id of subject under discussion) - Date (of intervention, YY.MM.DD)
nodes.csv: nodelist with bio id - name - surname - canton - political group
=== Results visualization ===
* Kibana Dashboard iframe: