360Giving offers a convenient way for developers, researchers and others to access grants data published in the 360Giving Data Standard: the 360Giving Datastore.
The Datastore is updated every day by fetching data from every file listed on the 360Giving Data Registry. GrantNav is then updated based on this latest data. The data update process also produces an enhanced JSON version of the 360Giving Data Registry showing metadata about each file.
Using the Datastore
For users with more complex data needs, it is possible to directly access the Datastore and run queries on it. Potential uses of the Datastore include:
- Creating a filtered dataset as raw data for a research project.
- Producing a bespoke dataset that can be regularly imported into a database powering a web app.
The Datastore is designed to facilitate one-off or repeating data transfers covering a large part of the 360Giving data corpus. It does not include a general purpose API, and doesn’t support API-style individual access to the data.
Contact the 360Giving team if you are interested in direct Datastore access. Access is usually through a Google Colab python notebook, while we can supply database credentials that can run SQL queries against the Postgres database that holds the data.
As part of the process of importing data to the Datastore, some additional data is added to enhance the published data. These additional fields include:
- For charities registered in England and Wales, the charity number is used to look up the registered name of the organisation with the Charity Commission.
- For records with a grant recipient postcode, details of the region, local authority and ward where the recipient is located are added.
Data included in the datastore
Data listed on the 360Giving Data Registry is included in the datastore. A number of checks are performed on the data before it is imported into the Datastore. These checks ensure that the data is:
- Openly licensed;
- Valid according to the schema used for the 360Giving Data Standard.
If a dataset linked from the 360Giving Data Registry becomes invalid according to the 360Giving Data Standard schema or temporarily unavailable, for example because the location of the dataset has changed, the Datastore will use the last valid downloaded version of the dataset. The Datastore keeps a backup version of each dataset for 30 days. If a dataset remains invalid or unavailable the data will disappear from the Datastore after 31 days. If a takedown request is received and acted upon, data may be removed before this date.
Datastore Code of Conduct
The Datastore is a new service from 360Giving and is offered as a resource to help users make the most of the data published by grantmakers. To make sure that the data is useful and available we ask that Datastore users follow the points set out below. If the code of conduct is not followed then it may be necessary to revoke access to the datastore.
Make appropriate requests
The Datastore is designed for one-off or repeating bulk data requests from researchers and other technical users. Users should design their queries to be efficient with the resources, and to not make large numbers of requests for bulk data in a short space of time, or perform complex queries repeatedly. Talk to us about your plans for the data, and we’ll help you design an access pattern that meets your needs and is appropriate for the service.
Data is fetched every day but most 360Giving publishers publish new data less frequently than that. For most purposes checking for new data once a week would be sufficient.
The Datastore is not designed as a general purpose API, and so large numbers of small requests (for example for one grant or based on one recipient organisation) are not an appropriate use of the service. For this kind of use it would be better to use Datastore access to extract a set of data on a regular basis and then build your own API around that data.
Be mindful of security
The database credentials you will use will be specific to you, and should be kept private. The credentials will only give read-only access to the database.
Respect licence conditions
360Giving publishers include an open licence for their data. This licence will allow a range of different uses of their data, but there may be conditions attached. Most commonly, users of the data will need to attribute and link to the data source (the publisher themselves).
Information about the licence under which each data item has been published and the source of the data is available in the Datastore itself and should be used to adhere to any licencing conditions, such as source attribution.
Work with data publishers
The 360Giving Datastore is not the original source of the data, and should not be treated as such. For queries about individual published grants or the content of the data it is generally most appropriate to contact the publisher of the original data. Information about the publisher of each item is included in a table in the database itself.
Respect take down requests
360Giving has a take down policy for managing requests from publishers or others that data or a portion of it, is removed from the 360Giving Data Register and 360Giving online platforms.
The take down policy sets out how 360Giving will deal with these requests. Users of the Datastore should also consider how they would respond to take down requests. Registered Datastore users will be informed by 360Giving if a take down request is received.
Keep in touch
The Datastore exists to help people use 360Giving data, so please keep us informed of how you’re using it. We’d love to help you explore new ideas about what you can do with the data, and promote exciting and innovative projects that use the data. We also want feedback to make the Datastore and our other services better.
360Giving releases all the code we create under an open source licence to help others. The code for the Datastore can be found on Github.