Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
| project:chparlscraping [2015/09/05 15:56] – shalf | project:chparlscraping [2015/09/07 16:23] (current) – yrochat | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ==== Swiss parliament minutes scraping ==== | ==== Swiss parliament minutes scraping ==== | ||
| - | Is the Swiss parliament really useful ? Once elected, what are our councilors talking about ? Who is answering to who ? | + | |
| + | [[http:// | ||
| + | |||
| + | Is the Swiss parliament really useful ? Once elected, what are our councilors talking about ? Who is answering to whom ? | ||
| Goal of this project is to answer some of these questions and many more. To do this, we are planning to: | Goal of this project is to answer some of these questions and many more. To do this, we are planning to: | ||
| Line 8: | Line 11: | ||
| In order to perform some analysis, as | In order to perform some analysis, as | ||
| + | - from keywords, who talks about what, by parties, cantons and people | ||
| - person vs. vocabulary | - person vs. vocabulary | ||
| - dialogue order | - dialogue order | ||
| Line 15: | Line 19: | ||
| === Data === | === Data === | ||
| - | Raw data are available as one single JSON file, and its .cvs counterpart. We had size problems, thus exploring ways to produce several .csv | + | Raw data are available as one single JSON file, and its .csv counterpart. We had size problems, thus exploring ways to produce several .csv. |
| + | **The [[https:// | ||
| - | == Structure of the main JSON (on Giovanni' | + | == Structure of the main JSON (on Giovanni' |
| list of interventions, | list of interventions, | ||
| - | * Link_subject: | + | * Link_subject: |
| * Surname: of the person speaking | * Surname: of the person speaking | ||
| * Description: | * Description: | ||
| * Bio: link to the page of the person speaking | * Bio: link to the page of the person speaking | ||
| * Canton: of provenance | * Canton: of provenance | ||
| - | * Subject_id: of the subject under discussion | + | * Subject_id: of the subject under discussion |
| * Date: of the intervention (DD.MM.YY) | * Date: of the intervention (DD.MM.YY) | ||
| * Group: political group of the person speaking at the moment of the intervention | * Group: political group of the person speaking at the moment of the intervention | ||
| Line 32: | Line 37: | ||
| * Data: transcript of the intervention | * Data: transcript of the intervention | ||
| * Name: of the person speaking | * Name: of the person speaking | ||
| - | |||
| - | == Graph data for Yannick == | ||
| - | |||
| - | graph.csv: edgeless with Source (bio url as id of person replying to) - Destination (bio url as id of person talking before) - Subject (id of subject under discussion) - Date (of intervention, | ||
| - | nodes.csv: nodelist with bio id - name - surname - canton - political group | ||
| == Structure of the Parliament API data via Yannick == | == Structure of the Parliament API data via Yannick == | ||
| - | [[http:// | + | * [[http:// |
| + | * [[http:// | ||
| + | * [[http:// | ||
| + | * [[https:// | ||
| + | * From those data, building of a file (light one) describing members of parliament ([[https:// | ||
| - | [[http:// | + | Fields are: id cantonName council firstName lastName party active birthDate gender language maritalStatus militaryGrade partyId salutationLetter workLanguage |
| - | [[http:// | + | == Structure of the final files from Jérémie == |
| - | [[https://github.com/ | + | 1 JSON file + its .csv counterpart for each Parliament session of the National Council from 1995. |
| + | The same datasets are also available split as one JSON/CSV file per legislative session. | ||
| - | From those data, building of a file (light one) describing members of parliament ([[https:// | + | == Graph data for Yannick == |
| - | Fields are: id cantonName council firstName lastName party active birthDate gender language maritalStatus militaryGrade partyId salutationLetter workLanguage | + | graph.csv: edgeless with Source (bio url as id of person replying to) - Destination (bio url as id of person talking before) - Subject (id of subject under discussion) - Date (of intervention, |
| + | nodes.csv: nodelist with bio id - name - surname - canton - political group | ||
| === Results visualization === | === Results visualization === | ||
| - | Kibana Dashboard iframe: | + | * Kibana Dashboard iframe: |
| - | < | + | < |
| - | Example of a Kibana Dashboard: | + | * [[http:// |
| <pic sylvain> | <pic sylvain> | ||
| + | * Example viz graph "who talks to who": | ||
| + | <pic yannick> | ||
| + | |||
| + | * Semantic distance between members of parliament: <viz pa> | ||
| + | |||
| + | * A simple gender gap visualization for the current Parliament that kind of summarizes it all: < | ||
| ===== Team ===== | ===== Team ===== | ||
| * Giovanni Colavizza, [[https:// | * Giovanni Colavizza, [[https:// | ||
| Line 65: | Line 77: | ||
| * [[http:// | * [[http:// | ||
| * Fabrice Hong, [[https:// | * Fabrice Hong, [[https:// | ||
| - | * Jérémie Knüsel [[https:// | + | |
| - | * Sylvain | + | |
| + | * Sylvain | ||
| * [[user: | * [[user: | ||
| * [[http:// | * [[http:// | ||
| * Douglas Watson, [[https:// | * Douglas Watson, [[https:// | ||
| - | * Jan Iwaszkiewicz, | + | |
| ===== Links ===== | ===== Links ===== | ||