Updates - PhilaDAO Data Dashboard
Updates
In the course of our work at the District Attorney’s Office, we often discover new and better ways to look at, organize, and interpret the data. We make changes and updates to the Public Data Dashboard to keep it as accurate and informative as possible. We also want to be clear to users of this site about the changes we’ve made. This page is a timeline of updates, fixes, and changes that we’ve implemented.
We learned from a member of the public about an issue with the arrests and offense category classification on the dashboard. After investigating, we are reporting on the fix for this issue and how implementing the fix affects the data.
Arrest offense category classification is derived from charge codes used by the police. We use additional information about the arrest to differentiate between “Aggravated Assault" and "Homicide: Other” in our categorizations.
What was likely a change in an external arrests database broke this distinction, moving what was previously in the “Homicide: Other” category into the “Aggravated Assault” category. The fix properly reclassifies these records. Please see before/after numbers below as an example of the effect of this fix. This issue was identified 6/10 and resolved 6/14.
Pre/post fix example, arrests, violent, year-end report:
The how to use this site page has been updated to show instructions for the home page, report pages, and the new downloads page. Menu items on the homepage were updated and lightly reorganized. Carousel was updated to include the 2023 Racial Injustice Report.
The data download page and the data available to the public has changed significantly.
- There is a new user interface for selecting and downloading DAO datasets. This new UI allows a user to select dataset, geography, and data format. The intent behind this change is to make it easier for users to navigate the many datasets we make available, make it easier for us to add new datasets in the future and to offer data in additional formats. The downloads page will also display the columns for the selected dataset and the time when it was last updated, so users can easily view the data about the data.
- Our data is now hosted on the city’s open data infrastructure; previously we self-hosted CSV files for each dataset. This change puts DAO data in the city’s data catalog and offers many more ways for the public to query and obtain our data.
- Data format has changed to from a “wide” format (where columns previously had counts by offense type) to a “long” format, with columns for matching geographies, categories, sub-categories, date/time and defendant race. This change provides much more transparency into the quantifiable operations of the criminal legal system.
Fixed some pre-2015 incidents and arrests classifications which resulted in zero-counts for some categories. Thanks to the folks at DVIC for their quick work updating the incidents dataset. Additionally, downloadable data now includes statistics back to January 1, 2010; previously historical data was only available back to 2014.
We have added four more years of historical data to all dashboard pages. Data previously only went back as far as 2015, now we show back to 2011. The dashboard tables have been modified with a few features that will allow us to display more data while also improving the dashboard for users with smaller screens and mobile devices:
- The category columns retain their position as a user scrolls horizontally through the tables. This allows a user to scroll across many columns of data and easily compare numbers for different years and categories. * Sub-category navigation is condensed to one column. This frees up some screen space to display data while also navigating into sub-categories of data.
We made two significant updates:
- We updated the Case Outcomes pages to collapse Guilty and Guilty Plea into a single category that can be clicked on to expand and view sub-categories. The subcategories include Guilty at Trial, Open/Negotiated Guilty Plea, Nolo, and Other Guilty Plea. We did this to reduce the complexity of this page for most users, but allow for additional granularity for users who want to know more. We also rewrote the underlying structure of our tables to use divs and bootstrap classes rather than the traditional kable structure. This allows us far more control over page layout as well as sped up double expand-collapse levels, like on the outcomes page. The update should be unnoticable to the average user (except for speed improvements), but took a lot of work under the hood.
- We have included new Key Metrics on our homepage and Case Outcomes page. The Key Metrics include Arrest Rates for Homicide: Shootings and Non-Fatal Shootings, as well as our Trial-Ready Conviction Rates for Homicide: Shootings, Non-Fatal Shootings, Rape, and Carjackings.
We updated our outcomes to properly identify cases that have completed the ‘Alternative Felony Disposition’ (‘AFD’) program and mark them as ‘Diversion’. In our data systems, AFD-completed cases result in a case outcome of ‘Nolle Prossed’ or ‘Withdrawn’, but by using the court room they are disposed in and the docket entries written by the court clerk, we are now able to distinguish these cases that have completed this relatively new diversion program. We also further re-classified a small number of retail theft cases that were resolved by a ‘satisfaction agreement’ as Diversion. For more information about the DAO’s approach to diversion, check out the DAO website.
We have adjusted how we classify Homicide: Shooting and Non-Fatal Shooting cases in all of our data.
- For Incidents, we now count the number of shooting victims rather than the number of shooting incidents. Previously, if the police classified multiple shootings as belonging to the same shooting incident, we only counted the shooting incident once. We felt it was appropriate to county shootings by individual impacted both because of the gravity of the crime and the availability of data. It should be noted that no other incident type has victim-level information available, so other incidents are still counted by the overall incident, not each individual offense associated with the incident.
- For arrests and court cases, we updated our classification system to be more accurate. Previously, if there was a shooting incident that included both fatal and non-fatal shootings, we classified the arrest and court cases as non-fatal shootings. This was incorrect. We now classify the arrest and cases as fatal shootings if at least one of the shootings in the incident was fatal and the person was charged with (or the police recommended charging with) a homicide.
We have added the DAO excerpts of the 100 Shooting Review Committee Report as a standalone page. This can be accessed through the Research portal.
We have added the ability to directly link to items on the Research page. Each item also now contains a Copy Link button to allow for easy sharing of direct URLs to each item.
We added a story describing how Philadelphia sought accountability and justice following the 2020 civil unrest that immediately followed George Floyd and Walter Wallace’s murders.
We have made several content and user experience updates to the dashboard. This is the first in a series of updates over the next few months to improve the dashboard.
- We have added Withdrawn in the Interest of Justice as a category on the Case Outcomes report and in the downloadable data.
- We have removed several of the columns comparing time periods beyond year-to-year. While these comparisons were useful at the launch of the dashboard, they have become less so as the time period has increased. The COVID-19 pandemic changed how the court systems functioned, making direct comparisons misleading.
- We have replaced the color scale for charts on all report pages. There are only so many shades of blue, and we were running out of them.
- We have changed how the home page pulls data stories and data snapshots. We are now pulling from the new District Attorney's Office website and providng both data stories and snapshots as separate categories. The Data Stories link in the menu bar has also been updated to link to the new website.
We have added ‘Illegal Dumping/Littering’ as an offense category.
We made a number of small improvements:
- We added additional ways to highlight that the criminal legal system has operated differently during the pandemic. There is a popup that appears notifying users, we have highlighted the graph and table for periods during the pandemic, and we have added a note to the table explaining the pandemic's impact.
- We added links from the offense categories and case outcomes to the definitions page so that users can understand exactly what each phrase means. We hope this will create a broader understanding of the work we are presenting on the dashboard.
- The case outcomes includes a row for 'cases opened at the end of the time period.' The goal is to help people understand how the backlog of cases has changed during the pandemic, particularly in more serious cases.
We have added ‘Victim/Witness Intimidation & Retaliation’ as an offense category.
Major Update!
- Offense Categorization: We have changed the way offenses are categorized on the dashboard for arrest and case data. In October 2021, the Philadelphia Police Department ended support for FBI UCR categorization of offenses for cases received by the District Attorney's Office. This meant that we could no longer use UCR categorization on the Public Data Dashboard. In this update, we have introduced a new classification system that is based on the Pennsylvania statutes of the most serious offense suggested by police for the Arrests and Summary Arrests and the most serious offense charged by the District Attorney's Office for the Charges, Bail, Outcomes, Case Length, Future Years of Incarceration, Future Years of Supervision, Summary Charges, Summary Case Outcomes, Summary Case Length Reports. The Incidents report is unaffected. While this change means categorizations of an event may change between incident, arrest, and case charging, it allows for a more accurate and complete view of each stage in the process. It also allows us to categorize more offenses than we could previously. For more information about these classifications see this data story and the updated Definitons page. The Limitations page has also been updated to reflect these changes.
- Technical Updates: We have rewritten the internals of the layout to make use of Bootstrap 5. This will help us maintain a more modern and multi-device compatible layout. We have also made several behind-the-scenes, ease-of-use, and bug fix updates that will improve the overall experience of the site.
We have improved our ability to categorize non-traditional diversion programs in our ‘Diversion’ outcome category. In particular we are now capturing cases where a person charged with a misdemeanor (usually drug possession) pleads to a summary offense in lieu of entering a diversion program. Many people in this program are ineligible for other diversion programs. This change was retroactive, so it restated outcomes for prior years. In 2019, for example, it increased the diversion rate for drug possession cases by approximately 6 percentage points.
We have added a DAO Publications section to the Research Page. This section includes reports on internal studies at the District Attorney’s Office. We have also reorganized the menu to simplify it and make it more clear which reports are year-to-date and which are year-end.
We have added a Research page detailing the work of the DAO DATA Lab and our partners. This page will keep you updated on new research work, publications, and grant partnerships. From this page, you can also subscribe to the DAO DATA Lab email list.
We have added a line denoting March 2020 to all reports. This was the month that the COVID-19 pandemic forced a shutdown of many essential government functions in Philadelphia, including the court system. It also lead to several policy changes in the Philadelphia Police Department and at the District Attorney’s Office. The purpose of this addition is to enable users of the site to more easily view the data in the context of the pandemic. We have also changed the text of two firearms labels to more clearly state what the data shows: ‘All Firearms Offenses’ has become ‘All Firearms Possession’ and ‘Possession of Firearms’ has become ‘Illegal Firearms Possession’.
We have added quarterly data summarized by Census tract to the Downloads section.
We added in ‘Non-Fatal Shootings’ as an offense category. We identify these cases based on the Open Data Philly Shooting Victims dataset.
We are now able to identify cases in which the defendant has been exonerated or won their case on an appeal. This is reflected in the Case Outcomes report. Previously, these cases would appear as ‘Dismissed/Withdrawn/Etc.’ This is counted at the time that the exoneration or resolution of the appeal occurs. As these cases are resolved, you may see minor adjustments to cases in previous years. Details on exonerations can still be found on the Exonerations page.
We improved the quality of bail information that is on the dashboard. In particular, we can now distingiush between monetary bail and unsecured bail (SOB), we can distinguish ROR from held without bail, and we have very few unknown bail amounts. This was made possible through a large batch of updated data that the DAO received from the Administrative Office of Pennsylvania Courts.
We added a report detailing the exonerations that have occurred since 2016. You can find basic information about each exoneration and links to more information about exonerations in Philadelphia.
We added a report that shows the effects of the COVID-19 pandemic on the criminal justice system in Philadelphia.
“Updates and fixes, including:\n\n* You can now download daily data by ZIP\ \ Code Tabulation Area and weekly data by Census Tract. This was one of the most\ \ popularly requested features. Please see the Limitations\ \ page for more information on these datasets.\n* We have added 8 new offense\ \ categories to the misdemeanor/felony reports: Other Assaults, Sexual Assault\ \ and Other Sex Offenses, Fraud/Theft of Services, Embezzlement, Possession of\ \ Firearms, Prostitution/Sex Work, Patronizing Prostitutes, and Threats of Violence.\ \ Similar categories were added to the Incidents report, although the incidents\ \ data available on OpenDataPhilly is not as fine-grained as our other data so\ \ the new incident categories are a bit more general.\n* We have added two new\ \ tabs: Firearms and Other.\n* We have now added reports on summary offenses for\ \ Arrests, Charges, Case Length, and Case Outcomes. These help to fill a hole\ \ in the original dashboard, which only reported on felony and misdemeanor offenses\ \ and therefore missed an important part of the system. Many of the offense categories\ \ reported on are the same as for felony and misdemeanors, although several are\ \ not reported on (e.g. violent offenses tend to not be summaries) and others\ \ have been added (e.g. Disorderly Conduct). In all, we report on the 9 most\ \ common summary offenses. \n* You can now view year end data as well as year\ \ to date data. Year end reports are available through the navigation menu at\ \ the top of the page. Rather than just viewing what has happened since January\ \ 1 of each year, these reports allow you to view years as a whole.\n* A small\ \ error was found in how we have been connecting arrests to case information.\ \ This caused a small number, about 0.5%, of cases to be duplicated in the Charges,\ \ Bail, Outcomes, Case Length, Future Years of Incarceration, and Future Years\ \ of Supervision reports. This error has been fixed, and the total count of cases\ \ in these reports have decreased slightly as a result.\n* The line chart on the\ \ Violent Offenses tab on the Bail Report was reporting by quarter. This has been\ \ changed to report by month.\n* The Arrests report inadvertantly excluded arrests\ \ where there was no date of birth listed for the defendant. These cases are now\ \ included.\n* A bug causing some Safari users to experience issues when interacting\ \ with the report tables has been fixed.
“Hotfix: Fixed the calculation of 18-present year averages related to the start of the new year. Data downloads were not affected by this bug.”
Hotfix: Fixed a calculation issue that would cause the dashboard build to fail at the start of the new year. Users should not notice a difference in functionality.
Added Updates page.
We discovered that due to server issues in May and June of 2019, we were missing some sentencing information from that time. This caused artifically low numbers to appear in the Future Years of Supervision and Future Years of Incarceration reports for these months. We have updated how we connect cases to their sentence information, and are now capturing this missing data. We are also capturing more sentencing information in other time frames, though the impact is not as noticeable.
Various bug fixes, including:
- Fixed header scaling on small screens.
- A broken link to our GitHub Repository on the How to Use This Site page has been repaired.
- Fixed how some ‘Key Metrics’ arrow symbols render on iOS devices.
- Changed how many significant figures are shown on the Rates tab on the Future Years of Supervision and Future Years of Incarceration reports.
The Philadelphia DAO Public Data Dashboard Released! See the District Attorney’s statement here: Philadelphia District Attorney’s Office launches public data tool.